High memory use is usually due to a shell extension, plugin, video codec or similar 3rd-party code leaking memory. See here, here and here for the most relevant FAQs (and here).
(It can also happen if you have a ridiculous number or size of images in your config for backgrounds etc. as they all get loaded into memory.)