Your calculation is flawed because lots of people share ancestors, so the number doesn't go up but rather down - to penultimate Eve or Adam...
Or think of it this way. You're right, at first the number goes up: you have 4 grandparents, and they have 16, those in turn have 64, and so on. Imagine (or draw) this as a cone balancing on its point (that's you), growing larger as the generations of your ancestors go upward. Now imagine Adam and Eve: they're directly above your point, way up above the cone you're building, and since they're so small compared to everyone in all those generations, they're a point, too. They also start a cone -- of all human beings -- that gets larger going downward, toward your ancestral cone. Now, as your cone grows upward, there is a moment, a generation, where it intersects the cone coming down, forming a disk where they meet. That disk represents the last generation at which every human being on earth, living or dead, was a direct relative of yours. The next generation up, of course, doesn't follow your expanding cone -- there were fewer people then (Adam and Eve's! cone of all humans gets narrower going up). So above the disk, your cone takes the narrowing shape of Adam and Eve's cone, until it reaches them. Everyone above the disk is a direct relative of yours. Interestingly enough, when you think about it, that means that (since the "disks" of everyone now living are at essentially the same level), everyone above the disk is a relative you share with every other person you will ever meet. We are all cousins.
making a small donation to science.ca.