Long file list performance vs Explorer (Owner column)

Hello, I'm working at an animation studio and use Directory opus on a daily basis (love it) but have one problem with listing directories that contain a lot of files.

On this particular case, I have a list of 35k simulation cache files (each of them is about 100mb or so) that take almost two minutes to list. Windows explorer lists the same directory in about 15 seconds. I'm not sure what the cause of this is and wonder if there is something on by default that would be causing this to happen.

My other question is about listing file sequences as single files and if such option exists in dopus. For example a list of files named:
soil_box_v08.switch_v04_version_4.0000000001.bgeo.sc
soil_box_v08.switch_v04_version_4.0000000002.bgeo.sc
soil_box_v08.switch_v04_version_4.0000000003.bgeo.sc
…..

could be listed as:
soil_box_v08.switch_v04_version_4.####.bgeo.sc

Most animation or compositing software on the market does this in their open and import dialogues so the user does not need to scroll through many files looking for the ones that are the start of the sequence, especially if multiple sequences live in a directory. It would be amazing if dopus could provide this type of functionality.

The Folder Format/Options > Display > Sorting Options can have a small impact on how long folders take to display, but only a small one. Even in extreme cases like 35,000 files which trigger numeric sorting (if turned on) like your example files, the difference should only be a second or so (assuming a good machine).

If it's taking multiple minutes to list the folder, something strange is going on. (Even 15 seconds seems quite a long time, in the faster Explorer case.) It takes about 1-2 seconds for me to list a folder with 35,000 empty files using the same names as your examples, on a local SSD.

What type of drive are the files on? Is it a network drive; if so what kind of server/OS is at the other end?

Does turning on Preferences / Folders / Folder Display / Show generic icons for... and setting it to apply to all folders make a difference? If so, it's most likely either antivirus or a shell extension involved in generating the icons which is slowing things down for some reason. (Windows Defender is especially bad here, but usually with exe files, not data filles, unless it has some reason to scan them.)

Which file display columns are turned on? Some columns may have an impact on speed as they can cause the disk to be accessed while the folder is being read, or antivirus to start blocking things due to opening files.

Are you sorting by Name or another column/columns?

In Task Manager, is CPU usage high during the delay? If so, is it coming from dopus.exe or another process (e.g. antivirus)?

I did try to adjust some of those settings and the issue still persists. I'm connected to a Linux based file server over a 10gig connection. When I look at the cpu utilization of directory opus while it's trying to list the files it's about 0.2% at the most. The machine I'm running is a 24 core (48 thread) dual xeon gold 6136. I'm not sure if the threads could be creating some latency that would not happen on a low core count machine.

I'm including a screen shot of what the directory listing looks like as far as columns go.

The columns look fine and shouldn't cause any trouble. CPU shouldn't be an issue either if the usage is so low.

It's possible the Linux server at the other end doesn't support the FIND_FIRST_EX_LARGE_FETCH mode which we use to speed up directory reading on network shares, in which case there would be a network roundtrip for each file, instead of reading them in batches.

That mode is also only supported on Windows 7 / Server2008 R2 and above, which could be a factor if the machine is an earlier version, or if compatibility settings have been set on the Opus EXE to make it think it's on an earlier OS.

You can check that via Help > About Directory Opus and then click the Copy button. The text it puts in the clipboard will include the OS version it believes it is running on:

Directory Opus Pro 12.11.1 (Beta) Build 6955 x64
OS 10.0 (B:17763 P:2 T:1) SP 0.0

Path lengths may also play a part in how the local OS and remote server talk to each other. If the file paths are over 255 characters long (including the filenames) then it might be worth checking if the same issue happens with a similar number of files but with shorter names and paths.

When you compared Opus and Explorer, did you try Opus after Explorer or before? Directory listings may be cached if another program has read them recently, so if you test two programs, one after the other, the second one may be much faster due to caching.

If none of the above leads anywhere, it might be worth generating a couple of Process Monitor logs, one of what Explorer does when reading the folder, and another of Opus. (No need to log the full 2 minutes; the first 30 seconds or so should give us plenty.) If you do that, please let us know the path of the folder so we know what to look for in the logs. Here are instructions on how to make Process Monitor logs:

I generated the logs and put them up on we transfer. Where should I send the link ? I tried to do it via private massage but it says I cannot send one to you. Does crashdumps@gpsoft.com.au accept links ?

You can send anything to it; please go ahead.

I sent the e-mail. Hope it will shed some light on the issue.

I have also tried running the test on a few other machines in the studio that don't have directory opus. Basically they all load the directory fast (on par with my machine's windows explorer) and it's not a directory they have accessed before. I have asked our systems administrator if our server supports that command you have mentioned but have not received an answer yet.

The actual directory listing seems to complete in just a few seconds, but after that it looks like the Owner column is being displayed by the lister and is taking a long time to populate.

Does that make sense? It doesn't match the screenshot above, so maybe it is masking another problem, but most of the log activity, up to the end of the period that was recorded, looks to be caused by populating that column.

If the Owner column was on, but you still get the same problem with it off, could you generate a similar log with the column not being shown? (Make sure it is off before changing folders. You can use the Format Lock icon on the Status Bar to prevent the current columns from being modified when changing folders, if needed.)

This is a huge revelation, I did remove the owner column from the public image, but now I have tried another window with the owner column turned off and the performance is as expected. Thank you so much for finding this. Which part of the log shows this behaviour ?

I wonder why windows explorer with the owner column on does not have the same issue. At the moment i'm just happy that removing that column makes everything behave normally.

Fantastic tech support by the way (rare these days), thank you.

1 Like

It looks like at the moment we query the owner on the UI thread, which obviously for 35,000 files is not a great design :slight_smile: We'll look at moving this off to a background thread in the next update.

Which part of the log shows this behaviour ?

It's all the GetFileSecurity calls toward the end of the log.

Does each opus window get it's own thread or do all of them share one ?

This is probably not the place to do it (let me know where to put it), but I have two feature suggestions that would be a great for the animation/vfx business:

  1. Group file sequences as one file. Almost everything we output is a sequence of files.

  2. Variables in folder aliases. Often times we jump between projects so I have a bunch of aliases set up to go between common folders quickly. When the project changes right now I have to go and change the path in the settings to update the part of the path that's different (usually the project name). Would be nice to just have this part of the path be a variable that can be changed from one location. At the moment I resort to doing a find and replace in one of the settings files as I have a lot of aliases, but this is probably a risky move and should be something that can be done in the UI.

  1. Could be done via a VFS plugin. The hard part would be how to make it decide what is and isn't a sequence.

  2. The Favorites command can modify aliases and be run from scripts, so you could have a script which changes aliases according to any rules that you want to define. We can help with that if you tell us what you want in more detail.

A separate thread would be best if you want to discuss either in more detail.