Column based on Descript.ion-style file parsing

The problem

I'm using an external program to gather some specific information from file objects, archives etc which I need to display in a column so the first thing I thought of was dumping it in a descript.ion file and use the metadata.other.userdescription. The file was written in the proper format and was working however I've ran into some problems with its behavior:

  • Dopus wouldn't reread the descript.ion file every time that one was written to displaying older cached values, even when forcing refresh via F5
  • ..or worse sometimes it decided to randomly overwrite my descript.ion with the old cached values for no apparent reason
  • Those things were triggered by navigating to and away from the folder

The solution

That's why I started this mini-project which illustrates how to add your custom descript.ion like column.
Here are the gains from such an approach:

  • Comments are Read-only and will never be changed or deleted on dopus' whims
  • Changes are picked up the moment the description file is modified.
  • Results are cached for the session speeding up access - even between listers and tabs
  • Coexistence with NTFS comments enabled as userdescription.
  • Virtually unlimited columns based on different descript.ion style files
  • Possibility to extend as multi-column so you can store more data ala csv in a single description file (not illustrated here)

Getting started

First let's look at a descript.ion file dopus generates when setting comments on files and NTFS comments are disabled. Those are saved in UTF-8 encoding without BOM (having a byte order mark doesn't hurt as opus ignores it when reading). You might want to stick to this format when generating your files:

<encoding#> UTF-8
item_without_space.rar Item description bla bla bla
"Item With Spaces.doc" Description of this thing as well

It resides in the directory which CONTAINS the files and folders to be described.

  • Now we look at the add-in code which initializes our new column: DescriptionX in this case:
// Set the script type to JScript to use this script
// The OnInit function is called by Directory Opus to initialize the script add-in
function OnInit(initData) {

    // Provide basic information about the script by initializing the properties of the ScriptInitData object
    initData.name = "DescriptionX";
    initData.desc = "Descript.ion-like data loader";
    initData.copyright = "Devocalypse";
    initData.default_enable = true;

    // Create a new ScriptColumn object and initialize it to add the column to Opus
    var col1 = initData.AddColumn();
    col1.name = "DescriptionX";
    col1.method = "DescriptionX";
    col1.label = "DescriptionX";
    col1.autogroup = false;
    col1.namerefresh = true;
    col1.justify = "left";
    col1.match.push_back("Yes");
    col1.match.push_back("No");
}
function DescriptionX(scriptColData) {

    //define file path - changing the last part allow you to have as many lists as needed
    var infoFilePath = DOpus.FSUtil.NewPath(scriptColData.item.path + "\\descript.ion");

    //abort if no file
    if (!DOpus.FSUtil.Exists(infoFilePath)) return;

    //load or use available
    var map = DOpus.Vars.Exists(infoFilePath) ? DOpus.Vars.Get(infoFilePath) : LoadData(infoFilePath);

    //force reload if description file has been modified since last load
    //trivia: comparator fails if both are not cast as strings
    if (String(DOpus.FSUtil.GetItem(infoFilePath).modify_utc) != String(map('~DateTimeUtc'))) {
        DOpus.Output("Cache is invalidated. Description source was modified");
        map = LoadData(infoFilePath);
    }

    //get data for item if present in the list
    if (map.exists(scriptColData.item.name)) {
        scriptColData.value = map(scriptColData.item.name);
    } else {
        //DOpus.Output("Not in map: " + scriptColData.item.name);
    }
}

function LoadData(descriptionFile) {

    DOpus.Output("Reloading: " + descriptionFile);
    var file = DOpus.FSUtil.OpenFile(descriptionFile);
    var map = DOpus.Create().Map();

    if ((file) && (file.error == 0)) {

        var blob = file.Read(file.size);
        file.close();
        var str = DOpus.Create().StringTools().Decode(blob, "utf-8");
        var ar = str.split("\n");

        //comment out the following lines if you don't write the <encoding#> UTF-8 header
        //ar.splice(0, 1); //remove unicode designation: <encoding#> UTF-8
        ar.shift(); //remove unicode designation: <encoding#> UTF-8

        //map for hashtable style dictionary lookups
        ar.forEach(function (key, val) {
            if (val.length < 1) return;
            if (val.startsWith('"')) {
                //filename contains spaces and is quoted
                var parts = val.split('"');
                map(parts[1]) = parts[2];
            } else {
                //filename has no spaces and is not quoted
                var parts = val.splitLimit(' ', 1);
                map(parts[0]) = parts[1];
            }

        });
        DOpus.Output("Added descriptions: " + map.count);
    }

    //add a timestamp for last updated
    map('~DateTimeUtc') = DOpus.FSUtil.GetItem(descriptionFile).modify_utc;

    //cache in global vars
    DOpus.Vars.Set(descriptionFile, map);

    return map;
}

Prototypes and helper functions

// <editor-fold desc="Prototypes">

String.prototype.splitLimit = function (delim, count) {
    var parts = this.split(delim);
    var tail = parts.slice(count).join(delim);
    var result = parts.slice(0, count);
    result.push(tail);
    return result;
};

Object.prototype.forEach = function (func, context) {
    var value;
    context = context || this;  //apply the function to 'this' by default
    for (key in this) {
        if (this.hasOwnProperty(key)) {  //to rule out inherited properties
            value = this[key];
            func.call(context, key, value);
        }
    }
};

String.prototype.contains = function (str, ignoreCase) {
    return (ignoreCase ? this.toUpperCase() : this).indexOf(ignoreCase ? str.toUpperCase() : str) >= 0;
};
String.prototype.startsWith = function (str, ignoreCase) {
    return (ignoreCase ? this.toUpperCase() : this).indexOf(ignoreCase ? str.toUpperCase() : str) == 0;
};
String.prototype.endsWith = function (str, ignoreCase) {
    return (ignoreCase ? this.toUpperCase() : this).slice(-str.length) == (ignoreCase ? str.toUpperCase() : str);
};
// </editor-fold>

Extras

Button script to force a reload even if the data is fresh

function OnClick(clickData) {
    var command = DOpus.NewCommand;
    var descfile = clickData.func.sourcetab.path + "\\descript.ion";
    DOpus.Output(descfile);

    if (DOpus.Vars.Exists(descfile)) {
        var map = DOpus.Vars.Get(descfile);        
        DOpus.Vars.Delete(descfile);
		DOpus.Output("Cleared " + descfile + " data. Had " + map.length + " items.");
        command.RunCommand("Go REFRESH");
    } else {
        DOpus.Output("Variable " + descfile + " is not set.");
    }
}

Download full add-in

DescriptionX.zip (2.3 KB)

4 Likes

Thank You very much !!! :slight_smile:
Now I have for my Work-Folders new 8 columns which I have so long needet!

Regards,
Mani

1 Like

I am a bit lost on how to use this. I would like to modify the descript.ion file without the associated problems.
For example, my descript.ion file contains:
test.txt 1

I then edit it to contain
test.txt 2

However, the "User Description" column still shows "1" and the "DescriptionX" column shows nothing.

Is the DescriptionX column supposed to show "2"? How?
Also, will I need to ignore the "User Description" column, which is not properly updated?

It should work out of the box unless:

  • Your DOpus version is not high enough
  • You messed up in creating the descript.ion file

Latter should be utf-8 encoded. Remove the ar.shift(); line if your file descriptions start at the first line.
And yes, DescX column should show "2". and user description reads via the built-in method so ignore it.

What min DOpus version is required?
How do I get multiple columns (Mani above posted he had 8 new columns)?

I have renamed the DescriptionX.js file to "Column1.js", second copy of to "Column2.js" third to "Column3.js" etc.
Each of the File has become a own comment file like "columnn1.ion", "column2.ion", "column3.ion"... in a Folder in which i want to add the columns.
In file "Column1" I have modified the

//define file path
    var infoFilePath = DOpus.FSUtil.NewPath(scriptColData.item.path + "\\column1.ion");

and (carefully) other things, like name of the column etc...

OK. Thanks Mani.

It seems like DOpus always creates descript.ion files that starts at the first line.

Why would it be advantageous to change it to include the <encoding#> UTF-8 line? If I keep the ar.shift(); line in the DescriptionX.js, then it would not be compatible with the DOpus generated descript.ion files...

Also, how do I create the Button to force a reload (and when would I need it)?

Sorry peter2,
but this is to difficult and complex for me, to answer in the english language.... :frowning:

I will no make mistake...

@apocalypse

This looks good. Is it still supposed to work correctly in DOpus v.13? Because at first sight it doesn't show the discript.ion content (but it could be that my manually created files is somehow flawed, although I'm sure I use the right syntax). I do not normally use descript.ion files with DO, but as a surplus to usercomment, I find it interesting.

To my surprise it seems to be working without issue currently on DOpus 13.16.2.

You can open your descript.ion and check if:

  • the encoding is Unicode (UTF8) and not some UTF-16 etc.
  • the filenames should be relative and not absolute since the descript.ion is in the folder we're describing and expects them as something.jpg and not C:\Users\Blabla\something.jpg
  • make sure the first line is <encoding#> UTF-8 otherwise it might skip the first item.

Here's how it looks on my side:

@apocalypse
Just to save you time: I'm an IT guy so don't bother too much about the nerdy details, that's okay. And I found a slight typo, the wrong space in the wrong place caused the problem I suppose. I also removed the two lines to skip the 'utf line' (which I don't use).

I still have a problem: changes do not reflect in the DO column if I don't click the extra reload-button you added in the end. But this could be related to something out of your script's control. I'm simply editing the .ion file in Notepad++. When I save it, it could be that for some reason DO does not notice it. Perhaps an issue with the date - but I can test it (using DOpus.Output etc) so I may perhaps find out in a while. It could also be something in the DO settings that I have changed somewhere, which somehow blocks the refreshing of your DescriptionX column.

I'll try to figure out tomorrow. Thanks for your script anyway, because even with the manual refresh, it is still very useful to me.

That saves a lot, thanks.

I ran past the code again and couldn't find anything game-breaking.
In my case I just edit the descript.ion, save, then hit F5 in Dopus (without even using the cache invalidator button) and the latest comments get reloaded. Even if a mismatch occurred when comparing the times it should just reload them from disk.

Just to make sure, in your dopus settings > FIle Operations > Metadata >> Save descriptions to internal file metadata is CHECKED, right? Could be that if its unchecked dopus reads the descript.ion itself and tries to overwrite its format if it doesn't like it or something.

@apocalypse
Indeed F5 works.

From your description "Changes are picked up the moment the description file is modified" I expected instant visibility of changes. But then I realized that the script has no FSUtil.WatchChanges or OnFilesystemChange implemented for the *.ion file. I also think that would require a different kind of implementation, more complex, because you need to add such a monitoring the very moment one enters a folder, and also remove the monitoring when leaving. So using F5 actually makes sense - unless I'm mistaken in my interpretation of your code.

Bad wording on my part. Normally you need to refresh to load the changes to any column.
And yes making it watch the files will over-complicate things. DOpus already has a hard time monitoring events across 10 listers with 30 tabs each and we wouldn't like to add even more overhead to that.

1 Like

@apocalypse
There's no need for such complexity either - I wasn't thinking. Because I will save the description from a DO dialog anyway, where I can easily force a refresh after having saved. Your script is just fine, it solves the problem that DO only allows NTFS OR descript.ion, not both. Managing descript.ion comments independently if DO's built-in functionality solves the dichotomy.

I'm also thinking of using descript.ion as a duplicate, backing up all NTFS metadata (and Pdf Subject) in descript.ion, and then only showing your DescriptionX column, because DO shows the latter quicker than NTFS and Pdf Subject data in columns.

1 Like