Just for the record - I wrote in my earlier post
[quote="BetterRed"]
I am not experiencing the sort of issues you are facing - presumably because I am not running massive unarchive operations, I only need to do that when TPTB lose the main database, and I'd drive that process via existing bespoke scripts and programs. I'd have Opus shut down along with just about everything else - including the phone 
[/quote]
However I do see parallels in the sort of structures, volumes and processes chrisp and I deal with.
My interest is should I keep my existing standalone scripts and programs or should I consider integrating the processes into Opus and let it serve as the glue.
So I value chrisp's work and the effort he puts into documenting his findings. Often by documenting ones issues one can arrive at solutions - just a few insightful words from a 3rd party can give one a different lens thru which to see the world.
Again for the record - leo, I cannot tell you the nature of the information with which I am dealing. It would require multi-national NDA's. You have no need to know, so my principals would not agree to an NDA anyway - sorry.
Maybe the thread should be moved to a "The Problems of Managing Large Data Volumes" Forum.