Practical limitations on file collections size and sub-collection depth

I’m interested in knowing what the practical limitations are for file collections, so I can assess what route to take in achieving what I want.

I experimented with adding a folder with deep subdirectories to a file collection and choosing the option to add all as sub-collections. This operation caused the program to crash, and the resulting folder and its subfolders were only partially added to the collection(hence the bug report tag).

What is the practical limit for number of file collections, and number files within?
I’m interested in knowing this info in regards to hierarchical depth file collection directories as well.
Is the crash I experienced likely to be avoided by batching up the operation instead(so is the limit being on computation of adding to the collection and not on the collection structure itself)?

If there is expected to be a limitation to the size and depth of the collection structure, I would likely solve this by having a separate database for my file collection groups- then creating/destroying the directory opus file collections needed.

This raises the question of the practicability of creating ad-hoc file collections on the fly –the intended use case would be selecting a file in one lister in duel lister pane, and then in the other lister a file collection for a set of associated files loads.

So, to summarize, what are the known/expected limitations to file collection size/depth and creation/modification speed.

  • How ridiculous is it to have, say, 100k file collections, somewhere in a logically organized file collection hierarchy?

Thanks!

I don't know what the limits are but I wouldn't advise creating collections that large or nested, as they aren't really designed for that and performance won't be great. (Although 100k items shouldn't make anything crash, even if it's not a good idea. Please send us the crash reports for that so we can take a look.)

What are you trying to use them for? There may be a better way to do the same thing.

1 Like

File collections are simple text files stored in

/dopusdata\Collections

Your backups will be larger, and you might experience a delay when Opus starts.

Think of file collections more as workbenches than as storage or archives.

1 Like

From the sounds of your answer, I should indeed create and destroy the file collections as needed, loading a list of files from my separate db into a new collection “workbench” when needed.

The need to group have so many file collections comes from two primary uses I have, but for now the main one is is to keep all identical files in a collection, to quickly view all copies and work on them as desired.

I have pre-calculated for each file a list of its duplicate copies, so for each unique file, there is an associated collection with all copies.
I want to be able to edit the metadata of my files – comments, tags, labels, ratings – with the change affecting the file’s duplicates as well, so my idea is that a script could load a selected file’s associated collection, which contains all copies of that file, then I can just select all files in the collection to edit at once (or sometimes just a subset, in cases where I don’t want to apply the same metadata to some copies)
Any solution that doesn’t quickly allow me to pull up the list of identical files fast and easily will not work for me.

So the outline of a script to solve my problem would be:

Script that clears a file collection, then adds to it a new list of files, then reloads the lister viewing that collection.
The script will read a DB that has two indexed columns: Filepath and ID. FP should be unique, but all files in the same group should have the same group ID.

So the script will:
Retrieve the group ID for a file by querying the database using the selected file’s fullpath.
Query the db again to retrieve all files with that ID
Add all those files to the collection
Reload the lister that has the collection

Using this method – the limitation on file collection size comes down to the DB and scripts I’m using, using, collection as a "workbench" only.

Does this sound sensible? Is there any advice you have before I go about making this script? I assume that the scripting languages available in dopus have a way to read sqlite databases?
Thanks!

If it can help, I did this by writing a small sqlitedb wrapper. You'll need sqlite3.exe so you can call it from "command line" (see: Command Line Shell For SQLite).
It's in an include file that references another include file ... so both are joined here in the package.
inc_sqlliteWrapper.opusscriptinstall (4.9 KB)

For reference, code for the wrapper :

// sqlLiteWrapper
// (c) 2024 Stephane

// This is an include file script for Directory Opus.
// See https://www.gpsoft.com.au/endpoints/redirect.php?page=scripts for development information.



// Called by Directory Opus to initialize the include file script
function OnInitIncludeFile(initData)
{
	initData.name = "sqlLiteWrapper";
	initData.version = "1.0";
	initData.copyright = "(c) 2024 Stephane";
//	initData.url = "https://resource.dopus.com/c/buttons-scripts/16";
	initData.desc = "";
	initData.min_version = "13.0";
	initData.shared = true;
}

// Returns an object with 2 properties : result = the commande line result, lines = a Vector of strings
// Each entry of the vector is a string  with the result (column values) of the select (each column is separated by |, so .split(/\|/); can make this an array of columns
function ExecuteSelectSQL(sqlitePath, sqlDbPath, sqlStatement) {
	var fsu 	= DOpus.FSUtil;
	var sql 	= fsu.Resolve(sqlitePath);
	var db 		= fsu.Resolve(sqlDbPath);

	var fso 	= new ActiveXObject("Scripting.FilesystemObject");
	var shell   = new ActiveXObject("WScript.Shell");

	var tmpFileBase = "DO.Temp.";
	var tmpFileExt = ".txt";
	var tmpFileStdout = ExtSystem.GetTmpFileName(tmpFileBase, tmpFileExt, fso).fullname;
	var tmpFileStderr = tmpFileStdout+".err"+tmpFileExt;
	
	var cmdLine = '%comspec% /c "echo ' + sqlStatement + ' | ';
	cmdLine += '"' + sql + '" "' + db + '"';
	cmdLine += ' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""'

	//dout ("Built command line = '" + cmdLine + "'");

	var result = {};
	result.cmd = cmdLine;
	dout("Command line SQL Wrapper = '" + cmdLine + "'");
	result.returncode = shell.Run(cmdLine, 0, true);
	result.stdout = ExtSystem.ReadFile(tmpFileStdout, fso); fso.DeleteFile(tmpFileStdout);
	result.stderr = ExtSystem.ReadFile(tmpFileStderr, fso); fso.DeleteFile(tmpFileStderr);

	if (result.stderr.length > 0) dout ("Some errors occured ...", true);
	//dout("Result errors : " + result.stderr);
	// dout("RAW internal Result : (" + result.stdout.length + ") : '" + result.stdout + "'");
	var lines = (result.stdout).split(/\r?\n/);
	// dout(" #" + lines.length +" results (lines in result)");

	var vResult = { "result" : result, "nbLines" : lines.length, "lines" : DOpus.Create.Vector };
	for (var e = new Enumerator(lines); !e.atEnd(); e.moveNext())
	{
		var l = e.item();
		// var detail = l.split('');
		// for (var i = 0; i < detail.length; i++) {
		// 	dout(detail[i]);
		// }

		if (typeof e.item() == "string") {
			dout (">> type=" + typeof l + " (" + l.length + ") : " + l);
			vResult.lines.push_back(e.item());
		}
	}

	return vResult;
}



function ExecuteSelectSQLJSON(sqlitePath, sqlDbPath, sqlStatement) {
	var fsu 	= DOpus.FSUtil;
	var sql 	= fsu.Resolve(sqlitePath);
	var db 		= fsu.Resolve(sqlDbPath);

	var fso 	= new ActiveXObject("Scripting.FilesystemObject");
	var shell   = new ActiveXObject("WScript.Shell");

	var tmpFileBase = "DO.Temp.";
	var tmpFileExt = ".txt";
	var tmpFileStdout = ExtSystem.GetTmpFileName(tmpFileBase, tmpFileExt, fso).fullname;
	var tmpFileStderr = tmpFileStdout+".err"+tmpFileExt;
	
	var cmdLine = '%comspec% /c "echo ' + sqlStatement + ' | ';
	cmdLine += '"' + sql + '" -json "' + db + '"';
	cmdLine += ' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""'

	//dout ("Built command line = '" + cmdLine + "'");

	var result = {};
	result.cmd = cmdLine;
	// dout("Command line SQL Wrapper = '" + cmdLine + "'");
	result.returncode = shell.Run(cmdLine, 0, true);
	result.stdout = ExtSystem.ReadFile(tmpFileStdout, fso); fso.DeleteFile(tmpFileStdout);
	result.stderr = ExtSystem.ReadFile(tmpFileStderr, fso); fso.DeleteFile(tmpFileStderr);

	if (result.stderr.length > 0) dout ("Some errors occured ...", true);
	//dout("Result errors : " + result.stderr);
	// dout("RAW internal Result : (" + result.stdout.length + ") : '" + result.stdout + "'");

	if (result.stdout.length == 0) {
		dout("No row returned.", true, true);
		return { "result": result, "nbLines": 0, "json" : { } };	
		// return { "result": result, "nbLines": 0, "json" : { "sqlRequestReturn" : "empty" } };	
	} 

	try {
		//dout("out length = " + result.stdout.length);
		var jsObj = JSON.parse(result.stdout);
		var vResult = { "result" : result, "nbLines" : jsObj.length, "json" : jsObj };

		return vResult;
	}
	catch(e) {
		dout("Error parsing JSON ...",true,true);
		dout("RAW Out :\n" + result.stdout);
	}
}

The "commons" include file :

// commonSAL
// (c) 2024 Stephane

// This is an include file script for Directory Opus.
// See https://www.gpsoft.com.au/endpoints/redirect.php?page=scripts for development information.



// Called by Directory Opus to initialize the include file script
function OnInitIncludeFile(initData)
{
	initData.name = "commonSAL";
	initData.version = "1.0";
	initData.copyright = "(c) 2024 Stephane";
//	initData.url = "https://resource.dopus.com/c/buttons-scripts/16";
	initData.desc = "";
	initData.min_version = "13.0";
	initData.shared = true;
}


///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
// Dialog Helpers
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////
function MsgDialog(title, message, sourcetab) {
	var dlg = DOpus.Dlg;
	dlg.window = sourcetab;
	dlg.title = title;
	dlg.message = message;
	dlg.buttons = "OK";

	dlg.Show();
}


///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
// Execute and get output
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////


///////////////////////////////////////////////////////////////////////////////
function RunHiddenEx(exe, params, tmpFileBase, tmpFileExt, shell, fso) { //v1.2
	if (!fso)   fso      = new ActiveXObject("Scripting.FilesystemObject");
	if (!shell) shell   = new ActiveXObject("WScript.Shell");
	if (!tmpFileBase)   tmpFileBase = "DO.Temp.";
	if (!tmpFileExt)   tmpFileExt = ".txt";
	var tmpFileStdout = GetTmpFileName(tmpFileBase, tmpFileExt, fso).fullname;
	var tmpFileStderr = tmpFileStdout+".err"+tmpFileExt;
	var cmd = '%comspec% /c ""'+exe+'" '+params+' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""';
	//dout ("CMD='" + cmd + "'");
	var result = {};
	result.cmd = cmd;
	if (exe.match(/^([A-z]:\\.+|\\\\(.*?)\\.+)$/)) { //test path to exe if given
		if (!fso.FileExists(exe)){
	        var msg = "E   Executable not found ["+exe+"]";
	        DOpus.Output(msg);
	        throw msg;
         }
		 //else dout (">> '" + exe + "' exists");
	}
	result.returncode = shell.Run( cmd, 0, true);
	result.stdout = ReadFile(tmpFileStdout, fso); fso.DeleteFile(tmpFileStdout);
	result.stderr = ReadFile(tmpFileStderr, fso); fso.DeleteFile(tmpFileStderr);
	return result;
}

///////////////////////////////////////////////////////////////////////////////
function ReadFile ( path, fso ){
	fso = fso || new ActiveXObject("Scripting.FilesystemObject");
	var content = "";
	if (!fso.FileExists(path)){
		return content;
	}
	var file = fso.OpenTextFile( path, 1, -2); // Read, UseDefaultEncoding
	if (!file.AtEndOfStream)
		content = file.ReadAll();
	file.Close();
	return content;
}

///////////////////////////////////////////////////////////////////////////
function GetTmpFileName(prefix, extension, fso) {
	fso = fso || new ActiveXObject("Scripting.FilesystemObject");
	var tFolder = fso.GetSpecialFolder(2); //2 = temp folder
	var tFile = fso.GetTempName();
	if (prefix!=undefined) tFile=prefix+tFile;
	if (extension!=undefined) tFile+=extension;
	return {
		path	: tFolder.Path,
		name	: tFile,
		fullname: tFolder.Path+'\\'+tFile
	};
}

if (typeof ExtSystem === "undefined") {
	var ExtSystem = (function () {
		var my = {};
		///////////////////////////////////////////////////////////////////////////////
		my.RunHiddenEx = function RunHiddenEx(exe, params, tmpFileBase, tmpFileExt, shell, fso) { //v1.2
			if (!fso)   fso		= new ActiveXObject("Scripting.FilesystemObject");
			if (!shell) shell   = new ActiveXObject("WScript.Shell");
			if (!tmpFileBase)   tmpFileBase = "DO.Temp.";
			if (!tmpFileExt)   	tmpFileExt = ".txt";
			var tmpFileStdout = this.GetTmpFileName(tmpFileBase, tmpFileExt, fso).fullname;
			var tmpFileStderr = tmpFileStdout+".err"+tmpFileExt;
			var cmd = '%comspec% /c ""'+exe+'" '+params+' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""';
			//dout ("CMD='" + cmd + "'");
			var result = {};
			result.cmd = cmd;
			if (exe.match(/^([A-z]:\\.+|\\\\(.*?)\\.+)$/)) { //test path to exe if given
				if (!fso.FileExists(exe)){
			        var msg = "E   Executable not found ["+exe+"]";
			        DOpus.Output(msg);
			        throw msg;
		         }
				 //else dout (">> '" + exe + "' exists");
			}
			result.returncode = shell.Run( cmd, 0, true);
			//result.returncode = shell.Run( cmd, 1, true);
			result.stdout = this.ReadFile(tmpFileStdout, fso); fso.DeleteFile(tmpFileStdout);
			result.stderr = this.ReadFile(tmpFileStderr, fso); fso.DeleteFile(tmpFileStderr);
			return result;
		}
		
		///////////////////////////////////////////////////////////////////////////////
		my.ReadFile = function ReadFile(path, fso){
			fso = fso || new ActiveXObject("Scripting.FilesystemObject");
			var content = "";
			if (!fso.FileExists(path)){
				return content;
			}
			var file = fso.OpenTextFile( path, 1, -2); // Read, UseDefaultEncoding
			if (!file.AtEndOfStream)
				content = file.ReadAll();
			file.Close();
			return content;
		}
		
		///////////////////////////////////////////////////////////////////////////
		my.GetTmpFileName = function GetTmpFileName(prefix, extension, fso) {
			fso = fso || new ActiveXObject("Scripting.FilesystemObject");
			var tFolder = fso.GetSpecialFolder(2); //2 = temp folder
			var tFile = fso.GetTempName();
			if (prefix!=undefined) tFile=prefix+tFile;
			if (extension!=undefined) tFile+=extension;
			return {
				path	: tFolder.Path,
				name	: tFile,
				fullname: tFolder.Path+'\\'+tFile
			};
		}

		return my;
	}());
}

///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
// Common types extension methods
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////
if (!Array.prototype.contains) {
	Array.prototype.contains = function(obj) {
	    var i = this.length;
	    while (i--) {
	        if (this[i] === obj) {
	            return true;
	        }
	    }
	    return false;
	};
}

///////////////////////////////////////////////////////////////////////////
if (!Array.prototype.indexOf) {
	Array.prototype.indexOf = function(item) {
		for (var i=0; i<this.length; i++) {
			if (this[i] == item) return i;
		}
		return -1;
	}
}

///////////////////////////////////////////////////////////////////////////
if (!String.prototype.padStart) {
	String.prototype.padStart = function padStart(targetLength,padString) {
        targetLength = targetLength>>0; //truncate if number or convert non-number to 0;
        padString = String((typeof padString !== 'undefined' ? padString : ' '));
        if (this.length > targetLength) {
            return String(this);
        }
        else {
            targetLength = targetLength-this.length;
            if (targetLength > padString.length) {
                padString += padString.repeat(targetLength/padString.length); //append to original to ensure we are longer than needed
            }
            return padString.slice(0,targetLength) + String(this);
        }
    };
}


///////////////////////////////////////////////////////////////////////////
if (!String.prototype.repeat) {
	String.prototype.repeat = function(count) {
	    'use strict';
	    if (this == null)
			throw new TypeError('can\'t convert ' + this + ' to object');

	    var str = '' + this;
	    count = +count;
	    if (count != count)
			count = 0;

	    if (count < 0)
			throw new RangeError('repeat count must be non-negative');

	    if (count == Infinity)
			throw new RangeError('repeat count must be less than infinity');

	    count = Math.floor(count);
	    if (str.length == 0 || count == 0)
			return '';

	    // Ensuring count is a 31-bit integer allows us to heavily optimize the
	    // main part. But anyway, most current (August 2014) browsers can't handle
	    // strings 1 << 28 chars or longer, so:
	    if (str.length * count >= 1 << 28)
			throw new RangeError('repeat count must not overflow maximum string size');

	    var maxCount = str.length * count;
	    count = Math.floor(Math.log(count) / Math.log(2));
	    while (count) {
	       str += str;
	       count--;
	    }
	    str += str.substring(0, maxCount - str.length);
	    return str;
	};
}



///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
// DOpus functions wrappers & helpers
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////
// Helper dout function
function dout(msg, error, time) {
	if (error == undefined) error = false;
	if (time == undefined) time = true;
	DOpus.output(msg, error, time);
}


///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////
// Misc helpfull methods
///////////////////////////////////////////////////////////////////////////
///////////////////////////////////////////////////////////////////////////

///////////////////////////////////////////////////////////////////////////////
function humanFileSize(size) {
    var i = (size == 0) ? 0 : Math.floor(Math.log(size) / Math.log(1024));
    return (size / Math.pow(1024, i)).toFixed(2) * 1 + ' ' + ['B', 'kB', 'MB', 'GB', 'TB'][i];
}

///////////////////////////////////////////////////////////////////////////
function FindIndex(array, item) {
	for (var i=0; i<array.length; i++) {
		if (array[i] == item) return i;
	}
	return -1;
}

///////////////////////////////////////////////////////////////////////////
function isEmptyOrSpaces(str){
    return str === null || str.match(/^ *$/) !== null;
}

///////////////////////////////////////////////////////////////////////////////
// Helper function to build a string with an input string repeated n times
/**
 * @param {string} mystring
 * @param {number} times
 */
function repeatStringNumTimes(mystring, times)
{
  var repeatedString = "";
  while (times > 0) {
    repeatedString += mystring;
    times--;
  }

  return repeatedString;
}


///////////////////////////////////////////////////////////////////////////
function sanitizeFilename(input, replacement) {
	// inspired from node sanitize filename : https://github.com/parshap/node-sanitize-filename/blob/master/index.js
	if (typeof input !== 'string') {
		throw new Error('Input must be string');
	}

	//var illegalRe = /[\/\?<>\\:\*\|"]/g;
	var illegalRe = new RegExp('[\/\?<>\\:\*\|"]', "g");
	var controlRe = /[\x00-\x1f\x80-\x9f]/g;
	var reservedRe = /^\.+$/;
	var windowsTrailingRe = /[\. ]+$/;

	var sanitized = input
		.replace(illegalRe, replacement)
		.replace(controlRe, replacement)
		.replace(reservedRe, replacement)
		.replace(windowsTrailingRe, replacement);
	return sanitized;
}

And finally a small excerp of what final call looks like :

	var sqlStatement = "select distinct ksi.vendor, kcp.alias from k_content_path kcp join k_sound_info ksi on kcp.id = ksi.content_path_id where ksi.vendor IS NOT null AND kcp.path='" + (""+srcDir).replace(/^[kK]:/, "D").replace(/&/, '^&') + "'";
	dbSearch = ExecuteSelectSQLJSON(Script.Config["sqlite3 Path"], Script.Config["Maschine komplete3.db path"], sqlStatement);
	var res = dbSearch.json;
	logger.debug("#" + res.length + " results returned");
	if (res.length>1) {
		logger.warn("Multiple aliases found !!!");
		for (var i = 0; i < res.length; i++)
			logger.warn(" >> Alias = '" + res[i].alias + "' | Vendor ='" + res[i].vendor + "'");
	}
	if (res.length == 1) {
		logger.info("1 result found in komplete3.db => Product='" + res[0].alias + "' & Vendor='" + res[0].vendor + "'");

		if (res[0].alias != uiCtxt.Product) logger.warn("Product inferred from path (" + uiCtxt.Product + ") is different from the one in DB (" + res[0].alias + ")");
		if (res[0].vendor != uiCtxt.Vendor) logger.warn("Vendor inferred from path (" + uiCtxt.Vendor + ") is different from the one in DB (" + res[0].vendor + ")");

		if (res[0].alias != null && res[0].alias.length > 0) uiCtxt.Product = res[0].alias;
		if (res[0].vendor != null && res[0].vendor.length > 0) uiCtxt.Vendor = res[0].vendor;
	}

The return from the ExecuteSelectSQLJSON has a json property which is an array of javascript object. Each item in the array will have a property coming from the way you construct your sql statement.

1 Like

I realized the wrapper code needed some cleanup :slight_smile: , new version below
inc_sqlliteWrapper.opusscriptinstall (4.6 KB)

And maybe a more straightforward example of usage :

	sqlStatement = 	"select DISTINCT kcp.PATH, kcp.alias, ksi.vendor from k_sound_info ksi JOIN k_content_path kcp on kcp.id = ksi.content_path_id ";
	sqlStatement += "WHERE ksi.vendor IS NOT NULL ";
	sqlStatement +=	"AND (kcp.path LIKE 'D:\\MUSIQUE_PROD_EXT\\SAMPLES\\%' OR kcp.path LIKE 'K:\\MEDIA\\MUSIQUE_PROD\\SAMPLES\\%' OR kcp.path LIKE 'B:\\Mes Documents\\Native Instruments\\User Content%');";
	dbSearch = ExecuteSelectSQLJSON(Script.Config["sqlite3 Path"], Script.Config["Maschine komplete3.db path"], sqlStatement);
	var res = dbSearch.json;
	DOpus.Output("#" + res.length + " results returned");
	var allVendors = DOpus.Create.Map();
	for (var i = 0; i < res.length; i++) {
		if (!allVendors.exists(res[i].vendor))
			allVendors(res[i].vendor) = DOpus.Create.Vector();
		allVendors(res[i].vendor).push_back( { "alias": res[i].alias, "path": DOpus.FSUtil.NewPath(res[i].path) });
	}

1 Like

Thanks a ton for your help!
I am very keen on using your wrapper in future. I Intend on upgrading this script to query directly the SQLite db, and to also use it in future projects.

But for now I’ve taken a stop-gap approach which pre-processes the db into a csv file for each directory. I already have working scripts in dopus that parse csv files, so this gets me to a working solution faster.
I might need to pre-load the file collections for each file on the event of a folder expanded or opened to make it work fast enough for me.
Soon I may have some polished scripts to share w/ the community. Until then, if anyone would find it helpful to have a working example of a custom column script which takes its data from a csv file, please lmk and I’ll share it.
Thanks again!