No, you are wrong. Magnets do not become less massive over time. The magnetic fields and associated magnetic energy of a magnet are altered when the magnet attaches to a refrigerator door, or if a magnet picks up a paper clip, or otherwise interacts with a piece of iron or steel. However, that change occurs in reverse when you remove the magnet from the refrigerator door or remove the paper clip from the magnet. The magnet then recovers all the magnetic fields and magnetic energy it had before. While it is attached to the refrigerator door, no energy change is required for it to continue to hold there - the friction between the magnet and door is sufficient to keep it from sliding down in response to gravity. No changes in mass are involved. Not all changes in energy involve changes in mass.

[Editor: for instance, chemical reactions such as the detonation of dynamite produce a lot of energy, but no net loss of mass occurs. The energy comes from converting one type of chemical bond to another lower energy type. Magnets and their associated magnetic fields and magnetic energy derive from the alignment of electron spins -- and that energy was introduced in the process of magnetizing the magnet, which took energy input, usually in the form of an electromagnet. The kind of conversion of mass to energy you are thinking about happens in nuclear reactors or nuclear bombs, but energy can be transfered in many other ways without loss of mass.]