Some of the differences have to do with the history of battery standards. In addition, there's a hidden factor: the work a battery can do, or its amp-hour rating. Six volt batteries are probably bigger than 9 volt batteries for historical reasons. Usually the larger the battery, the more work it can do, or the higher its amp-hour rating. Sometimes you can see this rating on rechargeable batteries. For instance, the Nickel Metal Hydride AA batteries in my digital camera are rated at 2300 mAhr or 2.3 Amp-hours. That means they would be drained completely in 2.3 hours if they gave 1 amp of current constantly over that time. To put it in perspective, a typical 12V car battery can deliver about 300 amps to start a car engine. A home electric heater probably draws about 15 amps. An electric shock will be felt with a current of about 0.02amps (or 20 milliamps). Here is a table showing the Amp-Hour ratings of common alkaline batteries:
D -- 14,250 mAh
C -- 7,100
A -- 2,450
AA -- 1,120
AAA -- 550
9V -- 565
One of the big six volt batteries you describe would probably have a very high milliamp hour rating. The point is: the size of the battery shows how "long it can go" all other things being equal. The bigger, the longer. Maybe those big old 6V flashlight batteries were designed to last for many hours and that's why they are big.
Another consideration is the voltage. A single cell of the typical type used in household batteries is 1.5 volt. When you put two together in series you get 3V. A 9V battery actually has 6 small 1.5V cells inside it. D, C, AA, and AAA are all single cell batteries at 1.5V. I found a lot of this explained at the site of Steve Uhrig.