Certainly most of the molecules and atoms inside us do constantly move into and out of our physical form, such as the air we breathe (and exhale) as well as the water and foods that we eat and eliminate. Other parts of us also change. For instance our skin sheds a few layers of cells every day. You are correct that apart from a very few components such as teeth, virtually all of the constituent molecules and atoms that make up a person are replaced over the course of a lifetime, some many times over. However, there is a continuum: some are replaced every few seconds, while others last a lifetime. Most of our bodies can be replaced yet we retain our integrity because: a) it happens very slowly, and b) molecules and atoms are unimaginably small, so we can't notice the changes. Obvious changes such as breathing, drinking, urinating, etc. show you how new atoms and molecules come in and old ones go out. There are a vast array of metabolic (life-giving) pathways within our body's cells that manage the process of constant renewal. Of course aging occurs as well, which is basically a preprogrammed failure of this renewal process over a long period of time.
It's important to emphasize the extent of the thousands of renewal precesses always going on in (nearly) every cell of your body at all times. Take the molecule DNA, for instance. DNA is a very long molecule but it's made of atoms of only a few elements: carbon, hydrogen, oxygen, nitrogen and phosphorous. There are minuscule molecular mechanisms that constantly check the DNA for integrity to make sure that the code of life, the instructions that make you what you are, stay perfectly correct. Just by random chance, in every living cell of your body (and there are trillions of them) the DNA in each of those cells may have up to 10,000 mutations per day. Mutations are errors in the code that come by chance. A bit too much to drink, a bump on the elbow, or maybe just too much sun might have caused the mutation. Actually it could be a million things. All such errors must be fixed. The mutation consists of a molecular error involving the disarrangement of a number of atoms in the DNA molecule. There are other molecules (teams of them) that constantly scan the DNA in every cell looking for these errors and when they find one, they then enlist the help of other molecules for correcting the errors. In the process some atoms of hydrogen, carbon, etc. may be replaced. That's just for DNA. The same thing happens for nearly EVERY component of every cell. This might involve swapping out some atoms for other ones in the cell's thousands of molecular structures.
We are not rocks, we are living entities, hence everything is in flux. The beauty of *life* is that it has attained a level of self-organization that includes countless automated self-repair processes that go on ALL THE TIME and at ALL LEVELS. For instance, inside your bones, there are cells that are constantly checking and renewing the bone tissue. In looking up the answer to this question I read (on Dawkins' site--link below) that this process causes our entire skeleton to be renewed every 140 days. That means even the bones have all new molecules/atoms a couple times a year. This is necessary to repair hairline cracks, and to modify the skeleton in case a person takes on a job requiring heavier or lighter use of certain parts of the body.
A key concept to understand all this is the idea of equilibrium. In nerves for instance, the signals are caused by waves of electrical potential which are achieved by changing the concentration of sodium and potassium atoms inside and outside of parts of nerve cells. There is a natural equilibrium across the nerve cell membrane--a balance of numbers of atoms. The individual atoms are always changing as they come and go with the food we eat and excrete, but the *balance*, the equilibrium, of these atoms and molecules is maintained by hundreds of life's molecular systems. If you go way down to the sub-cellular level, down to the level of molecules, it becomes apparent that everything is in constant flux with atoms and molecules coming and going constantly. Hence everything is always being swapped out, but it doesn't matter, since it's all in perfect balance, so everything keeps working just fine.
An analogy that might encourage understanding is city traffic. Think of a car as an atom. At any given time of day there will be about the same number of cars on a particular street. However, they are always changing. You never know exactly which cars are going to be there. It's the same in the body. At any time there's the same balance of atomic and molecular components, but they are coming and going like cars on a busy street. You never know which ones are going to be there, and over the course of a lifetime they change quite a bit. But the essential *structure* of the street and the traffic remains more or less the same. It does not really change. Yes, it *does* change somewhat, and this we experience as aging.
There is a long discussion of your question here on Dawkins' website: How much matter passes through a human body in a lifetime?
As for part 2, the answer is yes. Due to the incredibly strong force of the Earth's gravity, pretty much any molecule that ever gets here, stays here. Some extremely light atoms and molecules such as helium and hydrogen do escape to space, but virtually everything else stays here on Earth. They may have been here at the creation of the Earth, or they may have arrived by cosmic dust or meteors falling to Earth. We are a self-contained environment (actually to me, this is the key reason why we should watch how much oil we are burning if you think it through). However it is possible that an atom in your left hand came from a meteor that hit the earth 2 billion years ago, as a result of some collision on Mars. It may have also been part of a dinosaur. It's impossible to be sure, but one thing for sure: it has probably been on this planet for millions, if not billions of years.
It should be emphasized that these ideas are not unique to humans. The molecular processes of life occur in all life forms on the planet, from the simplest virus, through bacteria and algae, within all plants, and within all animals, including humans. When I look out upon the Earth and see the rich blanket of life covering every corner of this planet, I cannot help but be awed by all those countless biological processes that achieve a state of constant balance and renewal at so many different levels of existence. To me, it's one of the most marvellous things to contemplate about being alive. We humans are just one small part of this very big complicated thing called Life.