Without a way to reproduce this locally it's tough to know exactly what's happening or where the toolbars are coming from.
I have a couple of ideas that might work but they could be wild goose chases. I've asked a friend who knows a lot about office add-ins if he has any ideas. Failing that I might code-up a possible solution and send it to you to try.
Alternatively, there's the MultiView plugin which shouldn't have this problem if you get the 3rd party viewer DLLs that it requires.
Office 2007 also comes with viewers which will work with the ActiveX plugin (the same one that you are using now) and which won't have this problem, but I imagine if you wanted Office 2007 you'd be using it already so that's not really a solution.