NTFS Junctions, Circular References, Folder Size Calculation

Help & Support
1

I have found a great deal of frustration attempting to resolve a Directory Opus v10 issue surrounding:
[ul]1) Automatic folder-size calculation, and
2) NTFS Junction Points, where
3) A circular-reference occurs.[/ul]
Windows 7 has enabled a much higher level of utility with the extension of NTFS Junction Point functionality. I have found, with the strategic creation of a few junction points replicated in several places around my system drive, I can navigate around my hard drive with tremendous efficiency gains.

Simultaneously, I have found the combination of Directory Opus relative size, and size columns, quickly offer a great deal of insight surrounding the nature and makeup of folder contents. My laptop has a RAID-0, striped-volume, 128Gb SSD. Access to files, copying, duplicating, etc, are refreshingly fast... yet the 128Gb storage ceiling is quickly encountered. (With 8Gb of RAM, a matching-size swap file & a couple of virtual machines, I've already used 25% of the capacity and haven't even addressed the OS & application installations). The storage limitation has been a tremendous influence toward increased organization and efficiency. I find I am regularly pruning lower priority files from my drive, to maintain system usability.

My frustration stems from the seeming inability to prevent Directory Opus from attempting to calculate folder-sizes that have circular recursive-references. Should I leave a lister window open, that happens to show a folder with such a reference buried inside, DO will get sucked into a futile exercise attempting to calculate the folder-size.

The recursive, circular-references, created by my custom Junction Points, as well as the native Windows 7 OS junction points that exhibit similar attributes, are currently not being recognized or understood to have circular-references, by DO. This results in Directory Opus sapping all my otherwise available CPU cycles, as it loops through the circular references tenaciously attempting to demonstrate an accurate calculation for infinity.



The online help files for Directory Opus v9 http://www.gpsoft.com.au/help/opus9/default.htm?turl=WordDocuments%2Fnewsupportforlinksjunctions.htm suggest: "Opus now includes support for links/junctions [...] GetSizes doesn't include the size of linked folders in its calculations, etc."

While that might have been true in DO 9, somehow GetSizes no longer exhibits this behavior. Arguably, it has become unusable in the face of circular-folder-references, which are quite often the product of NTFS junction points. Is there some way to prevent Directory Opus v10 from spidering a circular-reference and engaging the perpetual folder-size calculation that fully consumes one of my CPU cores?

If not, is there an alternative workaround? This is a major impass for me, as the integrated folder-size feature DO offers, was the primary draw to the product.

I hope to see this issue resolved, as the rest of Directory Opus has been a joy to use!

Best regards,
J.Adik

2

Creating circular junctions seems like a spectacularly bad idea. Opus's folder size calculation will be far from the only thing that goes wrong.

3

Leo,
Perhaps it would be helpful for you to recognize that circular junctions exist in the default Windows 7 configuration, inherently, and without modification.

The very nature of junctions brings forth more than just an opportunity for circular junctions to exist...

  • Given Windows 7's utilization of junctions, serving to maintain compatibility with the legacy file-system structure, one must make no further investment in spectacularly bad ideas to drive the same results.

It should be relatively trivial to catch a circular reference the second time around... and, to your point, folder-size calculations will be far from the only thing that gets fixed. I would expect an increasing number of users are experiencing similar problems.

4

Could you give an example to back up that claim? (See next post.)

If I go to C:\ and click Calculate Folder Sizes, it does not get stuck in an infinite loop.

Microsoft themselves recommend that you do not create junction cycles. (Although that is in an article written circa Windows 2000, which does not seem to have a newer version. Perhaps they have changed their stance, but I couldn't find any newer advice from them on the subject in either direction.)

It seems like common sense to avoid cycles because most Windows software is not aware of junctions at all, and even software that is junction-aware is probably still going to assume, in places, that the filesystem is not cyclic.

To do otherwise requires every piece of code -- including in third-party libraries, plugins, shell extensions, etc. -- which might recurse a directory to keep track of every path it has visited so far in order to detect cycles. That's not impossible, but it also adds complexity and overheads and is difficult to retrofit to the entire codebase -- some of if which we have no access to whatsoever -- of a platform where the concept of cycling directory structures is alien.

If I create a cycle and then run "dir /s" on it at a command prompt, it stops when the path becomes too long, not when it detects a cycle. That should tell you that the Windows operating system is not designed with cycles in mind. Same problem affects 7-Zip and WinRAR, which also stop when the path becomes too long, not when a cycle is detected. Almost nothing on Windows tries to detect cycles, and creating them is a very and idea.

5

Okay, yes, there are cycles by default (e.g. a trivial one is "C:\ProgramData\Appliation Data" which points back to C:\ProgramData), but they're fine because Microsoft -- knowing that cycles will cause huge problems, and only creating these ones for compatibility reasons -- permissioned the junctions so that they would not be recursed.

Those don't cause a problem with Opus or any other program. So if you want to create more cycles then you should permission them in the same way Microsoft did their ones, for the same reasons.

6

Until now I hadn't bothered trying Opus's folder size calculation against the test cycles I had created, as I assumed if you said there was a problem there was one (and it seemed reasonable that Opus didn't check for cycles as creating cycles seems wrong to me)...

But I just tried it anyway, to see what would happen. Opus seems to work fine for me; it stops counting when it hits the junction.

Here is the directory structure I created:

bf585e64dfacabd69dd7ea26fdfb5b77470773c0.png746×399 104 KB

I created a cycle with both a junction and a softlink, in case there was a problem with one type and not the other.

As you can see, Opus's Flat View mode also knows to stop at a junction/link, and the folder size column only shows the size of File B.txt and has not recursed into the link or the junction.

This is using Opus 10.0.2.0 x64 on Windows 7.

7

I also get no problems with a more complex cycles, where the loop goes via another folder (in case Opus was just checking that links don't point at a parent of the current folder). Opus stops at the point the cycle is detected:

4d9811cdf672c5d634bd78ff1b753fb70cd32f0a.png746×688 158 KB

So these parts of Opus, at least, seem fine to me with cycles.

I still think cycles are a really, really bad idea because most other software (and probably some other parts of Opus, and definitely some parts of shell extensions and other software which plugs into Opus, and including parts of Windows itself) won't detect them.

8

Leo,
Let's take a step back, and look at this exclusively from the default, native Windows 7 x64 file-structure.

I've removed all of the junctions that aren't original Win7 x64 native, and what seems clear is that the unwanted behavior (recursively calculating folder sizes with cycles) continues to exist. While I'm not experiencing a perpetual-calculation process, the reported values are inflated & inaccurate, and during the calculation process my CPU is being over-taxed. (In this instance, the process errors out quickly as a result of the path-length limitation and the length of the native-Windows-7 junction names.)

393b5122055d45f5dfc0c7c35fee18941a63cb48.png973×549 180 KB

The root of the problem appears tied to DOpus' treatment of junctions. While, in some instances, treating junctions identically to actual folders has advantages, it should be clear that this treatment is incompatible with the folder-size calculation feature.

The help file http://www.gpsoft.com.au/help/opus9/default.htm?turl=WordDocuments%2Fnewsupportforlinksjunctions.htm for DOpus v9 explicitly states "Opus now includes support for links/junctions [...] GetSizes doesn't include the size of linked folders in its calculations, etc."

I have two questions:
[ul]
Has the DOpus v9 feature/functionality stated above been dropped in DOpus v10?
How can I direct DOpus v10 not to consider junctions in folder-size calculations?
[/ul]
Thanks for your assistance!

9

I agree that what's shown in your screenshot looks wrong, but it's not what I see. I don't know what's happening on your machine; could the permissions on the junction be non-standard or anything like that?

I have tried exactly the same setup on three different Windows 7 machines, 32-bit and 64-bit, UAC on and off, admin accounts and standard user accounts. In every case, Opus 10.0.2.0 never even tries to calculate the size of the Documents and Settings junction and the size column next to it stays blank. That''s the case both when automatic sizing is on (and/or the File and Folder Count columns you have on in your screenshot, which trigger automatic sizing even if it's turned off) and when it's off and I manually trigger the calculation. Opus never calculates the Documents and Settings junction's size for me.

That's one possible explanation. but do you have any basis for picking that explanation over all others? Most (though not all) of the code in Opus can handle arbitrarily long paths (beyond the 256 character limit of some Win32 APIs) and if Opus was looping forever through the directory until the path became too long then that would also happen in the other cases which you said went on calculating forever...

[quote]I have two questions:
[ul]
Has the DOpus v9 feature/functionality stated above been dropped in DOpus v10?
How can I direct DOpus v10 not to consider junctions in folder-size calculations?
[/ul]
Thanks for your assistance![/quote]

  1. As far as I know, no feature/functionality related to junctions was ever dropped.

  2. I don't know; it seems to happen automatically for me in all of my tests so far.

10

Take Ownership of the junction-file with full access. Than create a new User Everyone and give him no rights at all.
Folder-size should no longer be calculated for the junction.

11

The only thing you need to do is to create the User Everyone and deny the right to List Folders / Read Data like in my (german) Screenshot.


12

(Ignore what I posted a few minutes ago; the permissions do seem to be a factor, and Kundal is quite right. If I create a new junction without special permissions, Opus will calculate the size of it, but only once, even if it points to its own parent folder.)

13

Kundal, thanks for the insightful tip. While it can get a bit time-intensive to add this additional special permission to each junction (should you elect to add more than a few), it does stop DOpus from engaging circular-referenced calculations.

As a point of clarification for others who may attempt this: Windows has a built-in user group called 'Everyone', which (since XP SP2) includes all authenticated users. To modify the special permissions of a junction, adding this 'Everyone' group, right-click on the junction in either DOpus or Windows Explorer, and select 'Properties'. Then click the 'Security' tab along the top of the Properties dialog. Click the 'Advanced' button, then click 'Change Permissions' and then click 'Add'.

A new dialog will appear (as shown below). Type 'Everyone' in the large text box and click 'OK'

Another new dialog will appear (as shown below). Select the check box in the 'Deny' column, labeled 'List folder / read data', then click 'OK'. Close the three remaining dialog boxes by clicking 'OK' on each, and you will have successfully added the Special Permissions to which Kundal referred.

Once these Special Permissions have been added, DOpus will no longer perpetually cycle through the circular references. Since the permissions to traverse the junction have not been changed, double clicking your custom junctions will still support your ability to navigate, and thereby will retain the original intent of increasing file-system accessibility.

Leo, thank you for your extensive research on this thread. It would be nice if there was an option to globally manage how DOpus interfaces with junctions, in future DOpus releases. Perhaps you could make a recommendation?

14

You can do it for several selected files with a single click on a Button:

@runmode:hide icacls "{filepath|noterm}" /deny jeder:(RD)

15

I'm still wondering why my manually created, not specially permissioned, junction cycles don't cause an infinite loop. Maybe Opus detects infinite loops but there are cases where it doesn't (e.g. very complex cycles, or ones involving multiple drives, or something?).

By the way, if the junctions are just to make it easier for you to navigate around the place (and not also used by other programs/scripts/etc.) then you could avoid the whole issue while still keeping the same convenience by using basic folder shortcuts instead of junctions. (Opus has an option to sort folder shortcuts among folders, too.)

16

I believe we've found what was causing the problem you were seeing, and the fix will be in Opus 10.0.3.1-beta which will be released tomorrow.

The issue was unrelated to junctions and happened if you did certain types of Find operation (e.g. with folder size as a criterion) in locations that had sub-folders which were permissioned to deny access.