It weighs on your mind, doesn't it?

weight on one's mind : to cause someone a lot of worry, concern, or anxiety, especially for a long period of time.