The problem is really is there a center of the universe? Well, not really. Imagine a balloon...now draw a picture of yourself on it. Then blow it up, more and more. You see you can't point to the center because the center is actually in 4D space time. As time moves on, the universe (balloon) expands spatially in time.
[In addition, Physicist Doug Hube answered a similar question here about the "location" of the centre of the universe. According to him, the centre is everywhere. --Editor]