Slow Network Access

Just wanted to check other users experiences, can DOpus be a little slow in accessing network resources. Have just moved over from XYPlorer to DOpus and in normal operation they are roughly equivalent in speed (Moved to DOpus as it is a lot more customizable and has many more options - and it's looks better !).

DOpus however is considerably slower when accessing network drives and folders (admitedly some of the network folders have hundreds of sub folders), but where XY takes approx 5 secs to display the resource DOpus can take up to 45-60 secs to display the same resource.

Are there some settings I am missing to help speed up network access (have already set Generic Icons etc.).

Have already seen the FAQ re. disabling plugins - but I am just using the standard plugins and I'm not sure how they would affect drive access speed.

Any suggestions would be appreciated (using registered version of DOpus 9 if that makes any difference).

Also minor cosmetic issue with the animation used when DOpus 9 accesses network resources, DOpus seems to fit in extremely well with both Vista and XP looks wise but the animation used is VERY Vista (and looks a bit strange on XP) - just a minor thing.

Thanks

Phat

Many file display fields such as: Size, MD5 checksum, Total files (in subfolder) and others require Opus to crawl through all files in a subfolder recursively to display relative meta data. Others (like Description, Image Size, MP3 bit rate) cause Opus to reference all files in the listed folder.

On local fixed drives such file display fields are awesome, but across a network they can quickly erode Opus performance. As a general rule, it is much better to keep your Network Default Formats as simple as possible, so that you can quickly traverse Network folders. once you are listing the actual folder you are working in you can apply a Content Type or Favorite Format with more detailed meta data fields.

See the FAQ on Folder Formats (the FAQ index is linked in my signature) for how to understand and edit Folder Formats.

1 Like

In addition to what Ken said, it's probably not that Opus is slower at reading the directory listings (almost all programs read them in the same way) but that some of your columns are causing Opus to open all the files to inspect them and that is clogging up the network.

The Description column is often an unexpected cause of file access (since all the plugins may look inside the file to see if they can generate a description), as is any other column which causes Opus to read the actual data out of files. Turn on the Network Drives folder format and give it a minimal set of columns (e.g. just Name, Size, Date.)

Also, make sure you have automatic folder size calculation turned off (this can also be done via the Network Drives folder format) as if it is on it will cause all of the subdirectories to be read in, which will slow things down. (The calculation is done on a background thread so it is normally not an issue but over a slow network it can clog up the request to change directories enough that things appear sluggish.)

Finally, the Total File Count and simlar columns, which count the number of files in sub-folders, also cause sub-folders to be enumerated and should generally be avoided on network drives.

In addition to the essential Folder Formats FAQ which Ken already mentioned, you may find the FAQ on folder-size auto-calculation useful:

[Calculate Folder Sizes Automatically)

Guys,

Thanks for the input - I downloaded the minor update of DOpus 9 (exported to USB) and today have absolutely no issues accessing any network resources (they are all lightning quick).

Am beginning to think this may have been a network issue (although this doesn't explain why XYPlorer was so much quicker than DOpus yesterday ?).

Also I only have basic columns showing (non of the ones that would require calculation - as far as I know).

Anyway issue is solved for the time being - so thanks again for your suggestions.

Cheers

Phat

I am also evaluating the switch from XY to Opus, and am experiencing the same thing.

From what I have seen so far, XY is orders of magnitude faster to browse a network folder than Opus.

I have tried reducing the folder format field to JUST Filename, and also have set "Get Folder Sizes" to "off", with no noticable difference. Is there anything else I can try?

Another point of interest is that if, in the tree, i click on a folder in the network drive which itself contains many subfolders/files, the contents of that folder are shown slower than XY but the delay is bearable. But if instead I dont click on the folder itself but on the "plus" to expand it, Opus takes foreeever to expand the tree. In XY it is instantaneous.

I observerd task manager to see what CPU usage was like, and it was at 3% so it does not seem to be a CPU issue.

Then I observed the bandwidth usage on the networked machine when I expanded a folder in Opus, and it was sky high! Obviously, Opus is doing alot more than just requesting the file names!

Any thoughts on this?

Can't throw any light on the differences between clicking on the plus and the folder itself, and apologies if I seem to be contradicting myself now, but I am having absolutely no problems accessing network resources with DOpus anymore.

In fact in many respects it is quicker than XYplorer (particularly flicking between tabs) - although as noted I have now discovered the description field (which was a feature I requested in XYplorer but was never implemented) - with this turned on and with multiple entries (I find it incredibly useful !!) it can slow down Network drive access, but even this is no longer noticible as the file structure appears and then the description fields seems to added fairly quickly.

Sorry I can't throw any light on your network issues (mine just dissapeared into the ether).

Phat

So, your issues went away by....upgrading to which version? I am using the most recent 9.0.0.3.

It is pretty apparent from the network usage that Opus is spidering more than it its being told to.

ryanagler,

Are there any zip files in the drive you are browsing? Maybe all the network traffic is Opus inspecting the zip files to put them in the folder tree or something like that?

Also, what's at the other end of the network connection? A windows machine, samba server, or something else? It's possible there's a problem that only gets triggered with certain devices.

There are no zip files at all -- but around 40 folders (which dont contain zip files either, but that should be irrelavent because Opus shouldnt be looking whats inside anyway)

The other end(s) are Samba servers, some Fedora, some Centos 4.4, but all exhibit the same behavior. I can email you my smb.conf if you like.

You might want to try running Process Monitor to see what is being access on the network drive when you navigate to it with Opus. Maybe it will reveal what's going on; either some operations that are unexpected (like reading some files to get information out of them) or some operations which seem to trigger a lot of network traffic or take a lot of time.

On the network here navigation with Opus, via the tree or otherwise, is as fast as in anything else on the network drives I have available. One of those is a Samba drive which seems okay, except for certain huge (100,000 files) directories which are very slow in Explorer as well. There are lots of variables and also lots of versions of Samba though, so it'll take some investigation to work out what's causing the slowdown for you.

Ok, so i I launched Opus and then launched the windows vista "Resource Monitor", and viewed the "Network" counter to a machine with a samba share. I reset the counters so they were at zero. Then and expanded the network drive with the "plus" and total b/min became 413,700. Then i expanded a subfolder which itself contained 170 subfolders, each of those containing 70 files and the b/min went to 3,495,000. thats three megabytes of traffic to list the names of 170 folders!

then i followed the same procedure, but this time with XY. expanging the root caused 15,000 b/min and the big subfolder changed it to 35,000 b/min. 35k versus 350k is about right, Opus takes about 10 times longer to expand network folders than XY.

is there another way of diagnosis you recommend? i guess i could install ethereal on the samba machine to see the result on that end, but i might not have time today.

Not Resource Monitor, but Process Monitor. It's a tool from Microsoft/SysInternals which will tell you which operations Opus is doing to the network drive. (It also monitors registry access etc. but you'll probably want to turn that off for this as it'll just get in the way.)

I've come to a new realization -- in Opus, if i click the "Plus" to expand a folder, it knows if each of the subfolders has a sub-subfolder (as evidenced that they have a plus next to them or nothing at all). Which means, that Opus is indeed crawling each subfolder to see if they have sub-subfolders! I would much rather do without this for my network drives, it really throws a wrench in my productivity when I have to wait for each folder to expand.

Compare this to XY which, for network drives, shows all folders with Pluses next to them. Meaning, its not crawling each subfolder to see if it should show a plus or not.

Ok, so I just did a Process Monitor capture (as you clarified with registry off), and it confirms my suspicion above. Expanding the large folder in Opus produced 2,088 events. Expanding the same folder in XY produced just 19 events! I saved both results as PML files. would you like me to email them to you?

Try setting these options as I have them displayed and see what happens:


shockingly, changing to those exact settings actually made it seem to load longer. I could literally just watch Process Monitor traverse through each of the 140 folders as I waited nearly 20 seconds (or more) for Opus to finish.

Here is an exerpt of just 1 of the 140 directories Opus spidered, if this will help you diagnose the issue:

3089 12:23:04.7427357 AM svchost.exe 1148 FileSystemControl \;U:0000000000026320\192.168.1.45\a SUCCESS Control: FSCTL_NETWORK_GET_CONNECTION_INFO
3090 12:23:04.7433889 AM svchost.exe 1148 FileSystemControl \;U:0000000000026320\192.168.1.45\a SUCCESS Control: FSCTL_NETWORK_GET_CONNECTION_INFO
3145 12:23:04.7463088 AM svchost.exe 1148 FileSystemControl \;U:0000000000026320\192.168.1.45\a SUCCESS Control: FSCTL_NETWORK_GET_CONNECTION_INFO
3339 12:23:04.7648452 AM dopus.exe 344 CreateFile \192.168.1.45\a\thebeast\databases\bte SUCCESS Desired Access: Read Data/List Directory, Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, OpenResult: Opened
3340 12:23:04.7679629 AM dopus.exe 344 QueryDirectory \192.168.1.45\a\thebeast\databases\bte SUCCESS
3444 12:23:05.0283078 AM dopus.exe 344 QueryDirectory \192.168.1.45\a\thebeast\databases\bte SUCCESS
3447 12:23:05.0330592 AM dopus.exe 344 QueryDirectory \192.168.1.45\a\thebeast\databases\bte SUCCESS
3448 12:23:05.0348910 AM dopus.exe 344 QueryDirectory \192.168.1.45\a\thebeast\databases\bte NO MORE FILES
3449 12:23:05.0352042 AM dopus.exe 344 CloseFile \192.168.1.45\a\thebeast\databases\bte SUCCESS
3455 12:23:05.0386113 AM svchost.exe 1148 FileSystemControl \;U:0000000000026320\192.168.1.45\a SUCCESS Control: FSCTL_NETWORK_GET_CONNECTION_INFO
3456 12:23:05.0395544 AM svchost.exe 1148 FileSystemControl \;U:0000000000026320\192.168.1.45\a SUCCESS Control: FSCTL_NETWORK_GET_CONNECTION_INFO
3459 12:23:05.0500007 AM svchost.exe 896 CreateFile \192.168.1.45\a\thebeast\databases\bte SUCCESS Desired Access: Read Attributes, Synchronize, Disposition: Open, Options: Synchronous IO Non-Alert, Attributes: n/a, ShareMode: Read, Write, Delete, AllocationSize: n/a, Impersonating: PENTIUM\a, OpenResult: Opened
3460 12:23:05.0531830 AM svchost.exe 896 FileSystemControl \192.168.1.45\a\thebeast\databases\bte PATH NOT FOUND Control: CSC_FSCTL_OPERATION_QUERY_HANDLE
3461 12:23:05.0534797 AM svchost.exe 896 CloseFile \192.168.1.45\a\thebeast\databases\bte SUCCESS

Out of interest, are you using XP or Vista?

Vista.

here is the Process Monitor output from clicking the plus next to the exact same filder in XY. Only 13 events, as opposed to 140 in Opus. Actual directory/file names have been masked for privacy.

575 12:37:19.7327451 AM XYplorer.exe 4132 FileSystemControl \192.168.1.45\IPC$ DFS UNAVAILABLE Control: 0x60194
576 12:37:19.7329856 AM XYplorer.exe 4132 FileSystemControl \192.168.1.45\IPC$ SUCCESS Control: FSCTL_NETWORK_DELETE_CONNECTION
577 12:37:19.7332295 AM XYplorer.exe 4132 CloseFile \192.168.1.45\IPC$ SUCCESS
579 12:37:19.7384427 AM XYplorer.exe 4132 CreateFile \192.168.1.45\a\thebeast\databases SUCCESS Desired Access: Read Data/List Directory, Synchronize, Disposition: somedir, Options: somedir, Synchronous IO Non-Alert, Attributes: n/a, ShareMode: somedir, Write, Delete, AllocationSize: n/a, OpenResult: Opened
580 12:37:19.7415635 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases* SUCCESS Filter: *, 1: .
581 12:37:19.9360466 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases SUCCESS 0: .., 1: somedir, 2: somedir, 3: somedir, 4: somedir, 5: somedir, 6: somedir, 7: somedir, 8: somedir, 9: somedir, 10: somedir, 11: somedir, 12: somedir, 13: somedir, 14: somedir, 15: somedir, 16: somedir, 17: somedir, 18: somedir, 19: somedir, 20: somedir, 21: somedir, 22: somedir, 23: somedir, 24: somedir, 25: somedir, 26: somedir, 27: somedir, 28: somedir, 29: somedir, 30: somedir, 31: somedir, 32: somedir, 33: somedir, 34: somedir, 35: somedir, 36: somedir, 37: somedir
1219 12:37:19.9755774 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases SUCCESS 0: somedir, 1: somedir, 2: somedir, 3: somedir, 4: somedir, 5: somedir, 6: somedir, 7: somedir, 8: somedir, 9: somedir, 10: somedir, 11: somedir, 12: somedir, 13: somedir, 14: somedir, 15: somedir, 16: somedir, 17: somedir, 18: somedir, 19: somedir, 20: somedir, 21: somedir, 22: somedir, 23: somedir, 24: somedir, 25: somedir, 26: somedir, 27: somedir, 28: somedir, 29: somedir, 30: somefile 31: somefile 32: somefile 33: somedir, 34: somedir, 35: somedir, 36: somedir
1876 12:37:20.0035320 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases SUCCESS 0: somedir, 1: somedir, 2: somedir, 3: somedir, 4: somedir, 5: somedir, 6: somedir, 7: somedir, 8: somedir, 9: somedir, 10: somedir, 11: somedir, 12: somedir, 13: somedir, 14: somedir, 15: somedir, 16: somedir, 17: somedir, 18: somedir, 19: somedir, 20: somedir, 21: somedir, 22: somedir, 23: somedir, 24: somedir, 25: somedir, 26: somedir, 27: somedir, 28: somefile, 29: somefile, 30: somedir, 31: somedir, 32: somedir, 33: somedir, 34: somefile, 35: somedir, 36: somedir
2455 12:37:20.0312814 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases SUCCESS 0: somedir, 1: somedir, 2: somedir, 3: somedir, 4: somedir, 5: somedir, 6: somedir, 7: somedir, 8: somedir, 9: somedir, 10: somedir, 11: somedir, 12: somedir, 13: somedir, 14: somedir, 15: somedir, 16: somedir, 17: somedir, 18: somedir, 19: somedir, 20: somedir, 21: somedir, 22: somedir, 23: somedir, 24: somefile, 25: somefile, 26: somefile, 27: somefile, 28: somedir, 29: somedir, 30: somedir, 31: somedir, 32: somedir, 33: somedir, 34: somedir, 35: somedir, 36: somedir
3017 12:37:20.0544307 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases SUCCESS 0: somedir, 1: somedir, 2: somedir, 3: somedir, 4: somedir
3103 12:37:20.0580007 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases SUCCESS 0: somedir, 1: somedir, 2: somedir, 3: somedir, 4: somedir, 5: somedir, 6: somedir, 7: somedir, 8: somedir, 9: somedir, 10: somedir, 11: somedir, 12: somedir, 13: somedir, 14: somedir, 15: somedir
3376 12:37:20.0707635 AM XYplorer.exe 4132 QueryDirectory \192.168.1.45\a\thebeast\databases NO MORE FILES
3377 12:37:20.0709562 AM XYplorer.exe 4132 CloseFile \192.168.1.45\a\thebeast\databases SUCCESS

Can I just check you don't have the 'show Zip files in folder tree' option on?