Since the RAR archive doesn't provide a global index file (large archives take really long to open) I started to switch over my workflow to use 7zip instead.
Unfortunately the creation process for 7zip archives takes 6 times longer than for RAR archives within DOpus (e.g. packing my Firefox profile using RAR takes 9 seconds using 7zip takes 54 seconds on my system).
I noticed that RAR utilizes all 8 cores of my system (average CPU load of 94%) but 7zip / LZMA2 seems to use only 1 core (average CPU load of 16%). Certainly I enabled the multithreaded option for both formats in the archive creation dialog as well as in the preferences.
Is this a problem or does 7zip actually use such a bad (not noticable) multithreaded implementation?
Ok thanks for (upcoming) investigation. I just wanted to flag this because the current performance is not applicable for our use-cases (took too much time for creating). However the separate index is a big advantage of 7zip for handling large archives.
BTW: I talked to Eugene Roshal (author of WinRAR) and he is actually thinking about the introduction of an optional index to the RAR format specification.
From what you said in the other thread, it could be that this isn't a bug, in that case.
Opus either tells 7z.dll to use one core or to use multiple cores (and decide for itself how many is best). Maybe LZMA2 selects 2 cores by default because using more than 2 can affect the filesize.
If so it might make sense for Opus to let specify the number of cores, although I didn't really want to make the UI that complicated. (In particular, I think specifying a specific number of cores is bad because you might take your configuration to a computer with a different CPU.)
I would prefer having a simple dialog providing just two options for the multithreaded setting (on vs off).
Since there is already a separate option for the compression method (normal, best, etc.) the multithreaded option should be single threaded (1 core) versus multithreaded (all cores). Thus whoever takes care of the last 1% efficiency can explicitly set singlethreaded but (I assume) for the majority the compression performance is much more important.
Unfortunately the current implementation (which doesn't use all cores) is a productivity blocker for me. The 31 GB virtual machine was packed within 7 minutes using WinRAR 4.2 but DOupus using 7zip took about 40 minutes on my system. That's not acceptable for my daily work so I have to wait for Leo's next steps.
I just installed the standalone 7zip GUI on my home system and compressed my firefox profile again using ultra compression mode.
Hmm, the same folder compressed by DOpus using best compression mode results in an 7zip archive of 34.624.319 bytes which is much more than the result of the 7zip standalone application in ultra mode which leads to an archive of 30.831.404 bytes.
Doesn't DOpus uses the ultra profile if "best" mode is configured?
Make sure you turn on solid compression. It is on by default in 7-Zip and off by default in Opus. (As well as using the same compression method, etc. Most of the settings you can't change in Opus will be the same as the defaults in 7-Zip.)
You should also compare against 7-Zip 9.20 as that's the version of the DLL which Opus uses (since it's the most recent stable version). (You can try replacing Opus's 7z.dll with a newer version but it's not tested or guaranteed to work.)
Strange. Didn't know that solid = on is the default in 7zip. That have to be considered if some comparisons are made between different compression tools using default settings.
I just noticed that 7zip decompression feels a little bit slow and seems to use 1 core only.
Is it possible to make this multithreaded also or is this by design of the 7zip algorithm and can not be configured in the 7zip API (of the DLL)?
Compare with 7-Zip itself. If 7-Zip can't do it (even with the obscure command-line-only options) then we probably can't either. If 7-Zip can do it then we can make Opus do the same (including obscure options; we can pass them to 7z.dll if you find an option that speeds things up, although it'd be strange for such an option to be off by default).
Ok, I had a look and there is no option to configure that. However the 7zip benchmark said that decompression should utilize all of my cores but in a real world scenario I can't verify that. Thus DOpus obviously can not control that.
BTW: The ultra mode takes 3 GB of my system's memory for a single compression task (same in DOpus if using BEST mode). Thus this should be used with care otherwise you will get out-of-mem errors when creating 2-3 archives in parallel.
This happens to me very often. I always tell Opus not to use multicore because it will say there is not enough disk space (which is false, actually it's out of RAM). Funnily enough, I noticed that Opus decompresses 7zip files a lot faster than 7zip filemanager (try to enter 7zip archive with Opus and copy then paste content out).
LZMA (which is the current default in DOpus) is much more memory friendly than LZMA2. Anyway I switched to LZMA2/normal/solid:64MB since this configuration provides better results than WinRAR/best under all conditions. Even if I use 7zip and enable all 8 cores of my processor the memory consumption is slightly below 1 GB. Since multithreaded compression is so much faster I'm willing to spend so much memory for a single compression task. If somebody really wants ultra (= DOpus best) mode --- which takes significantly longer --- he should know what he is actually doing.
[quote="leo"]Confirmed, LZMA2 mode does not seem to be using as many CPUs as it should.
I need to look into this more but won't have time to investigate for a few days.[/quote]
Are there any plans to investigate this topic in one of the next versions? You would make me happy to see all cores working when DOpus creates 7zip archives. This is a real blocker for us -> that prevents us from creating 7zip archives within DOpus. And to be honest the user interface of the original 7zip gui is a bit too much "straight forward" for our demands.
