In fact the process of pruning connections and brain cells makes you smarter, kind of like pruning a wild bush makes it into a beautiful garden specimen. Of course like everything in your body, brain cells get old and die and eventually you do lose brain function, but this doesn't happen for a long time. It turns out that you have the maximum number of brain cells at around one year of age. After that things get organized slowly for 20-40 years and you actually get smarter, not dumber.
At least that's what scientists think now. The brain is still one of the biggest mysteries and new discoveries are always being made.