The electric field at a point is just the force that a unit electrical charge at that point would experience.

So if we have an isolated conductor, any charges inside the conductor are free to move, and they will move until they don't experience any more force. At that point, the electric field inside the conductor is, by definition, zero. Since like charges repel each other, the charges will try to get as far from each other as possible, which is what pushes them to the outside of the conductor. This is why the charged sphere doesn't behave the same way as a spherical arrangement of charged particles -- the particles would at once start moving away from each other, due to their mutual repulsion, and go on moving apart forever.

This shows that the field inside the metal itself is zero, but we have not yet shown that the field inside an empty cavity enclosed by the metal is zero.

It is essential that the cavity be empty; if you had a charged object inside the cavity, supported by an insulating stalk, then there would be an electric field around that object. So we'll insist that the cavity really is empty.

Suppose there were a non-zero electrical field inside the cavity. Then we could imagine taking a charged particle somewhere on the interior surface of the sphere, at a point 'A', say, and move it along a path through the field to some other point on the interior surface, 'B', say. The particle would experience a force as it moved along the path, so we would either have to do work on it to make it move along the path, or we could extract work from it as it moved along the path. Let's suppose we extract work from it -- we can change the charge on our imaginary particle, if necessary, to make this true.

We can now get the particle back from B to A by moving it through the conductor, where we know there isn't any field. So the trip from B to A doesn't involve any work. Now we have a loop, A to B to A again, and each time we go round the loop, we can extract work. So, if there were a field inside the cavity, we could use it to get energy from nothing. But this violates the principle of conservation of energy, so our initial assumption, that there was a non-zero field inside the cavity, must have been wrong.

This argument works for ANY closed conducting shell -- it doesn't have to be spherical. It does have to be complete, though; if we use a wire mesh, as in a Faraday cage, the field inside the mesh may be non-zero at a distance from the mesh up to two or three times the spacing of the wires making up the mesh.