phi(x,y,z) = f(x,y,z) - kg(x,y,z)
Then f(x,y,z) will have a stationary point subject to constraint
g(x,y,z) = 0 when part(d(phi)/dx) = 0, part(d(phi)/dy) = 0,
part(d(phi)/dz) = 0 and g(x,y,z) = 0.
This gives four equations to find x, y, z and k.
k is the Lagrange multiplier and phi is the auxiliary function.
Applying these ideas to our problem, we have
f(x,y,z)= x^2 + y^2 +(z-c)^2
and
g(x,y,z) = x^2/a^2 + y^2/b^2 -z^2
The auxiliary function is
phi(x,y,z) = f(x,y,z) - kg(x,y,z)
= x^2+y^2+(z-c)^2 - k(x^2/a^2 + y^2/b^2 - z^2)
Then:
part(d(phi)/dx) = 2x -k(2x/a^2) = 0 ......................(1)
part(d(phi)/dy) = 2y -k(2y/b^2) = 0 ......................(2)
part(d(phi)/dz) = 2(z-c) -k(-2z) = 0 ......................(3)
g(x,y,z) = x^2/a^2 + y^2/b^2 - z^2 = 0 ............(4)
Now we must solve (1), (2), (3) and (4) for k, x, y and z.
From (1) and (2) x and y can take any values and we could have:
From (1) 1 - k/a^2 = 0, so k = a^2
then from (2) y = 0
From (3) z-c + kz = 0, so z(1+k) = c, z = c/(1+k)
z = c/(1+a^2)
From (4) x^2/a^2 + y^2/b^2 - c^2/(1+k)^2 = 0
x^2/a^2 + 0 - c^2/(1+a^2)^2 = 0
x^2/a^2 = c^2/(1+a^2)^2
and taking square roots
x/a = c/(1+a^2)
x = ac/(1+a^2)
So a point on the cone nearest to (0,0,c) is
x = ac/(1+a^2)
y = 0
z = c/(1+a^2).
And this was simple...