There are a number of factors to consider. First of all, the sun is a radiative heat source. That means we get the heat by just one method: radiation. (The other methods are conduction, like when you touch a hot piece of metal, and convection, like when you feel a warm breeze on your face.) You are indeed correct that radiative energy decreases with distance from the source. It decreases quite rapidly the further away you are. However, over the 150 million Kilometers (93 million miles) between the Earth and the Sun, the extra difference of about 8,000 meters (25,000 ft) (i.e. the difference between the highest mountain and the lowest desert) is essentially nothing. Just work out the fraction 8/150,000,000 on your calculator. You should find the answer is 0.0000000053 which for our sake we might as well call zero.
So if it's not distance what DOES cause the cold at the top of a mountain and the heat of a desert?
The answer has to do with air pressure. Put simply, sunlight heats the Earth's surface by radiation only. Then that heat is transfered to the air by conduction, and finally it moves around the planet by convection. The air on the planet is like a big blanket that is really thick at the bottom (by the desert) and very very thin at the top (up the mountain). Thick air has high pressure. Thin air is low pressure. When the air is thick that means its more dense, meaning it has more molecules. This nice thick air has lots of molecules to pick up the heat from the hot ground that was warmed by the sun. At the mountain top the thin air has so few molecules it can barely pick up any heat, so it feels colder.
But there's something else going on. Have you ever noticed when you let the pressure out of something (like a spray can) the thing feels colder? (Also, the gas coming out gets colder.) This is because of something called "The Gas Law" that says: as gas pressure is reduced the temperature of a gas goes down. (It's a bit more complicated than this, but this is the part we need to focus on.) Also, maybe you have also heard of the idea that warm air rises. This is how hot air balloons work. Think of the air on the Earth as existing in huge packets. One packet moving to another spot is basically experienced by us as wind. So when a packet of thick high pressure warm air on the desert floor rises, it has to lose pressure because of the gas law, and when a gas loses pressure it has to lose temperature. By the time it has reached the height of a mountain top it has lost a lot of pressure and so it is freezing cold. This is another reason why it's colder at the tops of mountains. There is a more scientific explanation of this at the Mad Scientists Network.