Issue with reading file size of soft links in a script

Help & Support
21

Ah, ok, happy holidays!

22

Hey, Leo, just checking if you've had a chance to look at this issue?

23

Bump.
Was testing with the latest 12.20.6 Beta and it's still unusable performance-wise compared to the default size column.
However, I've discovered another strange thing: the extremely slow walking over the folder content (as attached above) disappears whenever I move it/change its name a bit (the content stays identical)!
Alternatively, if I keep it in place, but change its content (remove all old folders and paste just 10 new folders), I get the same abysmal speeds!
But even at the old path it only affects the folders (so calculating size info for 10 folders is slower than 100 files)!
Have no clue what is so magical about the current path

Given all these troubles, I'd like to repeat a request to just expose whatever you already have in the default size column as a raw number to scripting (with an extra option to follow/exclude links so that I could have both numbers in one view), so that I could just apply the format that I like instead of having to replicate everything at a lower speed and encounter weird magical bugs.

24

Have you tried using this:

That change has been released for a while now and will let you go back to having Opus itself do the recursion instead of your script doing it.

25

But I'm not using any recursion in my test script (var folderEnum = DOpus.FSUtil.ReadDir(folder, false);), so this shouldn't change anything (and doesn't, I've removed the old boolean to have var folderEnum = DOpus.FSUtil.ReadDir(folder); and the bug with slow folder iteration remains)
By the way, even if it worked, it wouldn't have access to the Opus size cache that is used in the default column, right (and as such would still be underperforming)?
With the boolean removed, I've compared the performance within a folder with 4 subfolders that in total contain 2k files and 4k folders, 1G in size.

  1. "Native" tab: 4 "native" columns (name, size, # of folders, # of files).
    Refresh takes ~1 second to fill all the info
  2. "Script" tab: 2 columns (one "native" name and my script's size)
    (debug output on) Refresh takes ~3 seconds to just LIST the folders and then another ~15 seconds to list the size of all the four folders (I see the debug output scrolling very fast, not as slow as in the path described above)
    (debug output off) Now the LISTING of the four folders is immediate, not 3 seconds, but the total refresh time is similar: ~16 seconds

FYI these are the info about the four folders I'm testing with:

  1. TestScript (no folders, 1000 files) (each file the size of 100bytes)
  2. TestScript (1000 folders, no files)
  3. TestScript (2000 folders, no file)
  4. TestSize (1000 folders, 1000 files) (ech folder has only single 1Mb file in each)
26

I've tried with the script, on the (extracted) zip of 1679 folders.

I turned off debug output. (Testing speed with debug output turned on is pointless, since the debug output itself will become the bottleneck if there is a lot of it.)

It takes about 9 seconds for the script column to refresh on my machine, which is the same amount of time it takes for the native size column to calculate for the same folder.

I tried both in the folder with 1,679 folders below it, as well as in the parent of that (so there was only 1 folder directly below), and the results were always the same.

27

You might find this way of doing this has a bit less overhead:

function OnInit(initData)
{
	initData.name          	= "Column.Test";
	initData.desc          	= "Columns test";
	initData.copyright     	= "(C)";
	initData.url           	= "";
	initData.version       	= "1.0";
	initData.default_enable	= true;
	initData.min_version   	= "12.5"
}

function OnAddColumns(addColData)
{
	var col = addColData.AddColumn();
	col.name     	= "SizeTest";
	col.method   	= "OnColumnsTest";
	col.label    	= "Size.Test";
	col.header   	= "SizeTest";
	col.justify  	= "right";
	col.autogroup	= true;
	col.multicol 	= false;
}

function OnColumnsTest(scriptColData)
{
	var colName = "SizeTest";
	var item = scriptColData.item;
	var folderSize = DOpus.FSUtil.NewFileSize(0);

	if (item.is_dir)
	{
		var folderEnum = DOpus.FSUtil.ReadDir(item, "rl"); // Recursive. Skip Links.
		
		if (!folderEnum.complete)
		{
			var vecItems = folderEnum.Next(-1);
			var itemCount = vecItems.size;
			
			for (var i = 0; i < itemCount; ++i)
			{
				var itemChild = vecItems(i);
				if (!itemChild.is_dir)
				{
					folderSize.Add(itemChild.size);
				}
			}
		}
	
		scriptColData.value = folderSize;
	}
}
28

I've used your script as is and while it does seem to have similar performance for a single folder (so the recursion seems on par with the native one!!!, even though I lose the ability to get separately the size/count of symbolic links that I wanted to use in my more complicated script, but that maybe another ask for ReadDir — to give a separate list of all "skipped" elements in the "rl" which I could iterate over only when another column is present).

However, when I use this script inside a folder containing 10k empty folders, the performance drops:
~1m2s to get to the end (there are only two columns, name and script's size) vs
~13s for the "native" size column (again, the only other column is name)

29

That's probably the overhead of calling the script 10,000 times. But in a real situation, most of the time will be spent reading directories, not calling the script on a huge number of empty directories.