DO 11 delayed start and freezes when browsing in a lister

Help & Support
1

I recently installed DO 11.18 x64 on a new Dell Inspirion 15 laptop running Win 10 and am having a major delay problem with DO that I have never experienced before. I am running the current install in eval mode and am planning on buying a license soon. But I have been a DO user for a long time and have a licensed copy running on my other computer and have never experienced anything like this before.

Here's what I'm seeing.

  1. On Windows startup DO loads in the system tray in a timely manner... actually quite fast as I am running on an SSD. However, when I open a new lister it takes up to 60 sec to open. This only happens when I open the first lister. After that I can close and reopen a new lister and that comes up in an acceptable amount of time. So it's only happening on the first lister opened.

  2. After opening a lister and navigating to folders, depending on the type of folder I navigate to I am also experiencing a freeze in DO for 30+ seconds where the mouse cursor spins and the lister display stays blank until it finally loads the folder. Not quite sure of this, but am pretty confident that this only seems to happen on folders with a lot of executable files. For instance if I navigate to the folders where I have portable versions of the Nirsoft and Sysinternals program suites where there a lot of exes. After those folders load, I don't have any problems re-navigating back to those folders again unless I do something like open a new lister or close the current lister and open a new one. Then I see the same freeze issue again.

I browsed the DO forum and did find Leos post on diagnosing delay issues and tried the suggestions for disabling Windows shell extensions to try and isolate the problem, but those procedures don't work here. I use ShellEx View x64 to disable the suggested shell extensions and also the shell extensions that I am using in DO. I've also disabled all the shell extensions I know I am using in DO by blacklisting them in DO preferences but again with no luck. I've also tried disabling all the DO plugins but that doesn't work either. I am only running one script... aussieboykie's Beyond Compare plugin v 2.4AB and have also tried removing that with no luck.

I'm hoping that someone can suggest a debug process I can go through to isolate this, such as tracing DO execution for the two problems I described above.

Please advise. TIA Mark.

2

Do you have, by any chance, some special columns activated in those folders, like file or folder counts?

3

This sounds similar to something I've encountered in the past. Assuming you are using Details mode, what is the state of Preferences -> File Display Modes -> Details -> Display icons? If on, turn it off and see if it makes any difference.

4

I am currently experiencing this exact phenomenon.

I.e., with icon display on there is "very long lag" in opening listers. With it "off" it opens quickly.

Any advice on debugging this issue to determine the "root-cause" to fix-it?

Hmmm. Revised comments. It is a "partial" fix to turn off icon display. It is not the only factor.

5

Using my psychic powers, I bet you are:

  • Reading a directory with a lot of large installer .exe files inside it.
  • Using Microsoft's terrible antivirus.

Is that correct? :slight_smile:

6

Lol

Yes to the anti-virus. But disabling or uninstalling showed it was not the culprit.

No to the large set of "large" installer files.

The "apparent" culprit was actually the "DOpus Favorites".

The "apparent" cause is that DOpus was confused by symlink chains (multiple reparse-points) that indirectly point to UNC (network) mount-points.

I.e., an absolute path "Junction" (/J reparse-point) to a drive-letter "\\?\C:\..." is a local-device auto-mount-point (reparse). Similarly, an absolute path to a "SymLink" (/D reparse-point) whose target is a UNC network drive path "\\dns-or-ip-etc\..." is a remote-share auto-mount-point [authenticating through your credential-store as needed].

If that, in turn, is indirectly referenced via a relative "SymLink" (reparse) then "apparently" DOpus is not performing or figuring out via a SymLink true-path resolution that the true target path is on a NetworkDrive.

I.e., reparse-points can be 31 levels deep. It appears that DOpus is not looking up the true-path of each reparse-point in the chain to determine whether the true-path is on a network drive (UNC mount-point).

As a result, on paths where the first reparse-points are relative but subsequent or final reparsepoints are UNC mount-points, the DOpus UX then ends-up blocking / waiting for "time outs" on those slow or unavailable network paths.

When I removed those "paths" from the "Favorites" my problems stopped almost instantly. Even with all the DOpus debug settings on and watching all the system DebugView trace output.

fyi, in other places where DOpus is aware of things being network-drive paths, it behaves "well" / "efficiently" using some form of "futures" for non-blocking UI responsiveness.

I should add that I worked this out by enabling lots of debugging and tracing, monitoring file activity etc. Then, in DOpus turning off icon display (sped some things up) but was intermittent. Turned of "folder-tree", sped more up, but was intermittent. However, 100% repro in all situations was that clicking on the favorite button had long-long-lag.

While I don't think it has any bearing on this issue, I am accessing the machine in question via a RDP (remote desktop session). That remote machine is running Windows 10 17XX; and with RDP usage, had to be tweaked after the latest Windows updates to correct / work around some COM+ "local activation" issues. Mostly relating to Cortana etc. However, none of that proved to be related to this "delay/lag".

On a related note I use the favorites in DOpus because they are really convenient. However, I have long been wondering why DOpus does-not share (or integrate with favorites) the windows "explorer" and "file-picker" Quick Access VFS, or even let you display / use them in the lister "folder-tree".

7

If you have things pointing to network drives that are not available then that will be slow, as it takes a long time for Windows to decide a request to inspect them has timed out (at least the first time).

Whether there are symlinks or other forms of indirection in between is largely irrelevant. If something in the chain is inaccessible (or very slow) then it will slow things down.

8

Hmm.

From the "settings:preferences" panel [which is really well designed from what I see] of DOpus, it "appears" clear that parts of DOpus handle network drive paths differently from other paths.

In Windows apps that are "Network Drive" aware, it is common practice to resolve file paths, segment by segment, to obtain the "true" path. I.e., not let Windows ntfs file-api's automatically resolve the path to its final directory location (because then you can suffer "time-outs" or "lag" while it attempts to auto-mount the UNC/SMB/CIFS paths).

If, while resolving the segments, a given segment is a "reparse-point", the path is recomposed from the current segment and the process repeats resolving the segments until one of two things is true.
a) The last segment has been processed and the "true-path" is a local drive path.
b) A network (UNC) mount point is encountered and then the "true-path" is treated as a network drive.

You see this in quite a few Windows tools, including Visual Studio and numerous scripting language systems and package managers. The result on Windows (which *nix apps tend to avoid) which can be annoying when it is not 100% transparent, is that you provide a path which happens to have symlinks (aka reparse-points on Windows ntfs), and the resulting reference to that path is set / recorded as the "true-path" rather than leaving it as the path you provided.

On *nix (linux etc) systems the function to obtain a "true" path is realpath, which is a part of the standard *nix libraries.

On Windows you have to build your own true-path resolver (which is not hard) using DeviceIoControl with FSCTL_GET_REPARSE_POINT to read the reparse-point data [via SE_BACKUP_NAME privilege - noting that reading reparse-point target paths does-not require elevation]. The only related Windows library function is GetFullPathName, which is similar, but not the same since it does not actually traverse and replace all symlinks stopping at UNC path expansion to avoid "auto-mount" timeout issues.

I assumed DOpus already was using this file-path analysis technique to avoid timeouts when cache-gathering file-path target information. The desktop explorer uses this in some but not all places. Many development tools and language engines, such as parts of the .NET facilities upon which PowerShell is built, use this by necessity because it is both a security issue and a performance requirement.

References

Reminder:
Repeated from other posts I made here on the DOpus forums.

Junctions (/J) reparse-points are resolved-server-side locally (transparently) on a SMB Network-Share Server before it shares the path. Which means they should almost never be used, except when you want the server-side to unify sharing of multiple server-side local-paths on single shared-volume to make them seem like a single server-side drive volume to remote (SMB - Samba, CIFS) share-clients.

Whereas, SymLinks (/D) reparse-points are resolved-client-side according to the group policy (gpedit) of the (SMB - Samba) share-client machine's fsutil behavior query SymlinkEvaluation settings.

SymLinks on Windows (other than absolute "\\UNC\path forms") behave like:
   - Apple OSX (Darwin-BSD) and Linux (various *nix filesystem) symlinks.

Junctions on Windows behave like:
   - Apple OSX (Darwin-BSD) and Linux (various *nix filesystem) local-device mount-points.

Symlinks on Windows that refer to "\\UNC\network-share-path" forms behave like:
   - Apple OSX (Darwin-BSD) and Linux (various *nix filesystem) network-share mount-points.

Link usage patterns fall into three categories: (whether on Windows or Linux, OSX etc)

  1. Links that are internally "relative" to their container-directory and point to a file or directory path within their directory container (e.g., inner, downward).
  • These are always sandbox/untrusted storage safe, secure, and immune to container-relocation issues.
  1. Links that are externally "relative" to their container-directory and point to a file or directory path outside their directory container (e.g., outer, upward).
  • These are locally safe but subject to breaking when their container is relocated. They should not be used in sandbox container directories unless sandbox is true-path link-aware.
  • The correct approach is to use volume-relative absolute paths such as "\full-path" with no-drive-letter.
  1. Links (Windows auto-mount point style) that are absolute paths to a mount-point which can be:
  • A server-side-resolved local-device-volume (drive)
    • defined by using a Junction reparse-point (/J) whose target is a full local-drive-based path.
    • e.g., \\?\C:\path
  • A client-side-resolved network-share-volume (drive)
    • defined by using a SymLink reparse-point (/D) whose target is a UNC path
    • e.g., \\machine.domain-name\path