It’s up to Windows how it loads and caches parts of the .exe and .dll files that make up the program.
For portable installs, there is always the dongle option which lets you copy the files to a faster medium and run them from there.
Maybe one/some of the favorites entries is/are slow to process. Check for any old entries that point to network locations that no longer exist, and so on.
Antivirus can have a big effect there, since it may scan .exe files even when a program just asks for their icons. Microsoft’s antivirus, the one bundled with Windows these days, is especially, inexplicably bad in this department, and will also not remember it has already scanned a file which has not changed.
Windows itself has an icon cache, although it is not always that good.
Beyond that, you have the option of creating an icon set if you want cached icons. That will also avoid accessing the .exe/.dll files they come from entirely.