I'd be surprised if an iconset caused such a large memory leak. (Not impossible, but not something I'd expect.)
Memory leaks are often (although not always) caused by shell extensions something else has installed. If it comes back, this guide can help track things down (although it is on the technical side of things):