EBook Columns

This Script AddIn provides six new columns to display Ebook metadata in DOpus. Currently only epub, mobi, azw3 and pdf are supported and you must have the ebook-meta.exe program which is installed by default with Calibre.


Warning: This script runs slowly when a book is not in the cache, it's got a lot to do for each file. You can prevent the column from continuing to process the remaining files by clicking the spinning circle in the location field and selecting "abort".
abort

The script expects that you have installed Calibre into the default location: C:\Program Files (x86)\Calibre2 . If not, there's a configuration option (calibre_path) where you can change this.

The script uses the ebook-meta.exe file to extract the metadata for the ebook files and outputs it to a temporary file in "..\AppData\Local\Temp". This file is automatically deleted once the metadata has been read.

Once a book has been parsed the first time, it's particulars are added to the cache file (in "/localappdata\EbookColumns") resulting in faster read times when the columns are shown for that book. This cache can be turned off with the use_cache configuration option and the actual path used to store the cache can be changed with cache_path.

Note:

Series data is handled horribly by ebook formats. Epub3 allows for the series information to be embedded into the ebook file but other filetypes don't seem to. Because of this the ebook-meta.exe program will not read or write series information for those files. Calibre cheats and adds metadata to a separate file for each ebook and the ebook-meta.exe program ignores this data as far as I can tell (Happy to learn otherwise if anyone knows more).

Because of this, the series column will only display data for .epub files.

History:

Older
  • 1.0 (6/4/20)
    • Initial Release.
  • 1.2 (14/4/20)
    • Added Tags column.
    • Merged Series and Series Index into one column.
    • Fixed "Title sort" data overwriting the "Title" column.
  • 1.4 (24/4/20)
    • Now using multicol columns correctly so the data for each file is only read/extracted once.
    • Added: Calibre path is now a configuration item.
  • 1.4.1 (1/5/20)
    • Fixed bug which occurred when the script was used in the Advanced Rename dialog.
  • 1.5 (14/6/20)

    • Added: Year column which extracts the year from the "published" metadata item.
  • 1.5.1 (22/6/20)

    • Added: Uses a cache file to store ebook metadata and uses that to populate the columns if possible. The cache location can be configured, and even turned off if required.
    • Fixed: Parsing of the metadata failed if other fields used a keyword like Author etc in their content.
    • Fixed: Added ALL to the delete command for the temporary file to override user preferences requiring a prompt for all deletes.
  • 1.5.2 (4/8/20)

    • Fixed: Columns now work correctly in the rename dialog.

Installation:

  1. Download: Ebook Columns.js.txt (13.6 KB)
  2. Drag the .js.txt file to Preferences / Toolbars / Scripts.

Script Config:

A couple of settings are available in the normal Script Addin configuration editor:

  • directories: Enter a string to be used for directories (can be blank).
  • non_ebook_files: Enter a string to be used for non ebook files (can be blank).
  • series_show_index: Include the series index in the series column.
  • calibre_path: The path to your Calibre installation.
  • cache_path: The path used to store the cache file.
  • use_cache: Toggle cache usage on and off.

Usage:

To use the column (once the script is installed) simply choose your config options and then turn on any or all of the following columns:

Script / Ebook Columns / Ebook Author
Script / Ebook Columns / Ebook Title
Script / Ebook Columns / Ebook Series
Script / Ebook Columns / Ebook Series #
Script / Ebook Columns / Ebook Tags
Script / Ebook Columns / Ebook Year

Script:

// EBook Columns
// (c) 2020 Steve Banham

var pathCalibre;
var arrLines = [];
var arrCache = [];
var cacheResult = "";
var cachePath;
var busyAbort = false;

scriptName = "Ebook Columns";
scriptVersion = "1.5.2";
scriptDate = "4/8/2020";
scriptCopyright = "(c) 2020 Steve Banham";
scriptMinVersion = "12.20";
scriptDesc = "Display author, title, series, tags and year of ebooks in DOpus columns.";

function OnInit(initData) {

    initData.name = scriptName;
    initData.version = scriptVersion;
    initData.copyright = scriptCopyright;
    initData.desc = scriptDesc;
    initData.default_enable = true;
    initData.min_version = scriptMinVersion;
	initData.url = "https://resource.dopus.com/t/ebook-columns/35114";

    initData.config_desc = DOpus.Create.Map();
    initData.config_groups = DOpus.Create.Map();
    var configName = "";

	configName = "directories";
	initData.Config[configName] = "<dir>";
	initData.config_desc(configName) = "String to show for any directory (Can be blank).";
	
	configName = "non_ebook_files";
	initData.Config[configName] = "---";
    initData.config_desc(configName) = "String to show for non ebook files (Can be blank).";
    
    configName = "series_show_index";
    initData.Config[configName] = true;
    initData.config_desc(configName) = "Include the series index in the series column.";

    configName = "calibre_path";
	initData.Config[configName] = "C:\\Program Files (x86)\\Calibre2";
    initData.config_desc(configName) = "The path to your Calibre installation.";

    configName = "cache_path";
	initData.Config[configName] = "/localappdata\\EbookColumns";
    initData.config_desc(configName) = "Storage location for the cache file.";
    
    configName = "use_cache";
	initData.Config[configName] = true;
	initData.config_desc(configName) = "Use a cache file to store ebook metadata to speed up future access.";

    var col = initData.AddColumn();

    col.name = "EbookAuthor";
    col.method = "onEbookCol";
    col.label = "Ebook Author";
	col.header = "Ebook Author";
    col.autogroup = true; 
    col.autorefresh = true;
    col.justify = "left";
    col.multicol = true;

    var col = initData.AddColumn();

    col.name = "EbookTitle";
    col.method = "onEbookCol";
    col.label = "Ebook Title";
	col.header = "Ebook Title";
    col.autogroup = true; 
    col.autorefresh = true;
    col.justify = "left";
    col.multicol = true;

    var col = initData.AddColumn();

    col.name = "EbookSeries";
    col.method = "onEbookCol";
    col.label = "Ebook Series";
	col.header = "Ebook Series";
    col.autogroup = true; 
    col.autorefresh = true;
    col.justify = "left";
    col.multicol = true;
    
    var col = initData.AddColumn();

    col.name = "EbookSeriesNum";
    col.method = "onEbookCol";
    col.label = "Ebook Series #";
	col.header = "Ebook Series #";
    col.autogroup = true; 
    col.autorefresh = true;
    col.justify = "center";
    col.multicol = true;

    var col = initData.AddColumn();

    col.name = "EbookTags";
    col.method = "onEbookCol";
    col.label = "Ebook Tags";
	col.header = "Ebook Tags";
    col.autogroup = true; 
    col.autorefresh = true;
    col.justify = "left";
    col.multicol = true;

    var col = initData.AddColumn();

    col.name = "EbookYear";
    col.method = "onEbookCol";
    col.label = "Ebook Year";
	col.header = "Ebook Year";
    col.autogroup = true; 
    col.autorefresh = true;
    col.justify = "left";
    col.multicol = true;
}

function onEbookCol(scriptColData) {

    if (busyAbort) return;

    if (scriptColData.tab != 0) {
        var busy = DOpus.create.BusyIndicator();
        busy.abort = true;
        busy.Init(scriptColData.tab, "Extracting metadata...", true);
    }

    var fso = new ActiveXObject("Scripting.FilesystemObject");
    cachePath = DOpus.FSUtil.Resolve(Script.Config["cache_path"]);
    
    if (Script.Config["use_cache"] == true) {
        if (!DOpus.FSUtil.Exists(cachePath)) {
            if (!fso.CreateFolder(cachePath)) {
                DOpus.Output("Unable to create cache file, check settings.");
                return false;
            }
        }
        cachePath = cachePath + "\\EbookColumns.cache";
    }
    
    

    pathCalibre = Script.Config["calibre_path"];

	if (pathCalibre.substr(-1) == "\\") {
		pathCalibre = pathCalibre.substr(0, pathCalibre.length - 1);
	}

    if (scriptColData.item.is_dir) {
        scriptColData.columns("EbookAuthor").value = Script.Config["directories"];
        scriptColData.columns("EbookTitle").value = Script.Config["directories"];
        scriptColData.columns("EbookSeries").value = Script.Config["directories"];
        scriptColData.columns("EbookSeriesNum").value = Script.Config["directories"];
        scriptColData.columns("EbookTags").value = Script.Config["directories"];
        scriptColData.columns("EbookYear").value = Script.Config["directories"];
        return;
    }

    if (scriptColData.item.ext == ".epub" || scriptColData.item.ext == ".azw3" || scriptColData.item.ext == ".pdf" || scriptColData.item.ext == ".mobi"){

        var itemHash = scriptColData.item.name + "-" + scriptColData.item.modify; //DOpus.FSUtil.Hash(scriptColData.item.realpath);
        itemHash = itemHash.replace(/\+/g, ".");

        if (Script.Config["use_cache"] == true) {
            if (readCache(scriptColData, itemHash)) {
                parseCache(scriptColData, itemHash);
            }
            else if (readCalibre(scriptColData)) {
                parseCalibre(scriptColData, itemHash);
            }
        }
        else {
            if (readCalibre(scriptColData)) {
                parseCalibre(scriptColData, itemHash);
            }
        }
    }
    else {
        scriptColData.columns("EbookAuthor").value = Script.Config["non_ebook_files"];
        scriptColData.columns("EbookTitle").value = Script.Config["non_ebook_files"];
        scriptColData.columns("EbookSeries").value = Script.Config["non_ebook_files"];
        scriptColData.columns("EbookSeriesNum").value = Script.Config["non_ebook_files"];
        scriptColData.columns("EbookTags").value = Script.Config["non_ebook_files"];
        scriptColData.columns("EbookYear").value = Script.Config["non_ebook_files"];
    }

    if (scriptColData.tab != 0) {
        busy.Destroy();
        if (busy.abort) {
            busyAbort = true;
            return;
        }
    }
}

function parseCalibre(scriptColData, itemHash) {

    for (i=0; i < arrLines.length; i ++) {

        if (arrLines[i].indexOf("Author(s)           :") > -1) {
            var strAuthor = arrLines[i].substring(22);
	        var hasSortField = strAuthor.search("\\[");
		    if (hasSortField > -1) {
		        strAuthor = strAuthor.substring(0,hasSortField);
            }
            strAuthor = strAuthor.replace(/^\s+|\s+$/gm,'');
        }
        if (arrLines[i].indexOf("Title               :") > -1) {
            if (arrLines[i].search("Title sort") == -1) {
                var strTitle = arrLines[i].substring(22);
                strTitle = strTitle.replace(/^\s+|\s+$/gm,'');
            }
        }
        if (arrLines[i].indexOf("Series              :") > -1) {
            
            var strSeries = "-";
            strSeries = arrLines[i].substring(22);
            var hasSeriesNum = strSeries.search("#");
            if (hasSeriesNum > -1) {
                var strSeriesNum = "-";
                strSeriesNum = strSeries.slice(hasSeriesNum);
                strSeries = strSeries.substring(0,hasSeriesNum);
			    strSeriesNum = strSeriesNum.replace("#","");
            }   
            else {
                strSeriesNum = " ";
            }
            if (Script.Config["series_show_index"] == true) {
                strSeries = strSeries + " #" + strSeriesNum;
            }
        }
        if (arrLines[i].indexOf("Tags                :") > -1) {
            var strTags = arrLines[i].substring(22);
		    strTags = strTags.replace(/^\s+|\s+$/gm,'');
        }
        if (arrLines[i].indexOf("Published           :") > -1) {
            var strYear = arrLines[i].substring(22,26);
		    strYear = strYear.replace(/^\s+|\s+$/gm,'');
        }

        if (strAuthor == undefined) strAuthor = " ";
        scriptColData.columns("EbookAuthor").value = strAuthor;
        if (strTitle == undefined) strTitle = " ";
        scriptColData.columns("EbookTitle").value = strTitle;
        if (strSeries == undefined) strSeries = " ";
        scriptColData.columns("EbookSeries").value = strSeries;
        if (strSeriesNum == undefined) strSeriesNum = " ";
        scriptColData.columns("EbookSeriesNum").value = strSeriesNum;
        if (strTags == undefined) strTags = " ";
        scriptColData.columns("EbookTags").value = strTags;
        if (strYear == undefined) strYear = " ";
        scriptColData.columns("EbookYear").value = strYear;
    }
    
    if (Script.Config["use_cache"] == true) {
        var fso = new ActiveXObject("Scripting.FilesystemObject");
        
        var cacheData = fso.OpenTextFile(cachePath, 8, true, 0);
        cacheData.WriteLine("[" + itemHash + "] Author=\"" + strAuthor + "\" Title=\"" + strTitle + "\" Series=\"" + strSeries
            + "\" Num=\"" + strSeriesNum + "\" Tags=\"" + strTags + "\" Year=\"" + strYear + "\"");
        cacheData.Close();
    }
}

function parseCache(scriptColData, itemHash) {

    var authorStart = cacheResult.search("Author=");
    var titleStart = cacheResult.search("Title=");
    var seriesStart = cacheResult.search("Series=");
    var numStart = cacheResult.search("Num=");
    var tagsStart = cacheResult.search("Tags=");
    var yearStart = cacheResult.search("Year=");

    if (cacheResult.search("Author=")) {
        
        var strAuthor = cacheResult.substring(authorStart + 7, titleStart - 2);
        strAuthor = strAuthor.replace(/\"/g, "");
    }
    else strAuthor = "";

    if (cacheResult.search("Title=")) {
        
        var strTitle = cacheResult.substring(titleStart + 6, seriesStart - 2);
        strTitle = strTitle.replace(/\"/g, "");
    }
    else strTitle = "";

    if (cacheResult.search("Series=")) {
        
        var strSeries = cacheResult.substring(seriesStart + 7, numStart - 2);
        strSeries = strSeries.replace(/\"/g, "");
    }
    else strSeries = "";

    if (cacheResult.search("Num=")) {
        
        var strNum = cacheResult.substring(numStart + 4, tagsStart - 2);
        strNum = strNum.replace(/\"/g, "");
    }
    else strNum = "";

    if (cacheResult.search("Tags=")) {
        
        var strTags = cacheResult.substring(tagsStart + 5, yearStart - 2);
        strTags = strTags.replace(/\"/g, "");
    }
    else strTags = "";

    if (cacheResult.search("Year=")) {
        
        var strYear = cacheResult.substring(yearStart + 5);
        strYear = strYear.replace(/\"/g, "");
    }
    else strYear = "";

    if (strAuthor == undefined) strAuthor = " ";
        scriptColData.columns("EbookAuthor").value = strAuthor;
    if (strTitle == undefined) strTitle = " ";
        scriptColData.columns("EbookTitle").value = strTitle;
    if (strSeries == undefined) strSeries = " ";
        scriptColData.columns("EbookSeries").value = strSeries;
    if (strNum == undefined) strNum = " ";
        scriptColData.columns("EbookSeriesNum").value = strNum;
    if (strTags == undefined) strTags = " ";
        scriptColData.columns("EbookTags").value = strTags;
    if (strYear == undefined) strYear = " ";
        scriptColData.columns("EbookYear").value = strYear;
}

function readCache(scriptColData, itemHash) {

    var fso = new ActiveXObject("Scripting.FilesystemObject");

    if (Script.Config["use_cache"] == true) {
        
        var cacheData = fso.OpenTextFile(cachePath, 1, true, 0);
        if (!cacheData.AtEndOfStream) {
            var content = cacheData.ReadAll();
            arrCache = content.split(/\r\n|\n/);
        }
        cacheData.Close();
        
        for (i=0; i < arrCache.length; i ++) {
            
            if (arrCache[i].search(itemHash) > -1) {
                cacheResult = String(arrCache[i]);
                return true;
            }
        }
        return false;
    }

}

function readCalibre(scriptColData){

    var cmd = DOpus.Create.command;

	cmd.SetType("msdos");
	cmd.SetModifier("runmode","hide");

	var fso = new ActiveXObject("Scripting.FilesystemObject");

	var tmpFolder = fso.GetSpecialFolder(2);
    var tmpName = fso.GetTempName();
    var tmpFile = tmpFolder + "\\" + tmpName;

	cmd.RunCommand("\"" + pathCalibre + "\\ebook-meta.exe\" " + "\"" + scriptColData.item.realpath + "\"" + " > " + tmpFile);
	
	var tmpDataFile = fso.OpenTextFile(tmpFile, 1, 0);
	
	if (!tmpDataFile.AtEndOfStream) {
		var content = tmpDataFile.ReadAll();
		arrLines = content.split(/\r\n|\n/);
	}

    tmpDataFile.Close();

    cmd.RunCommand("DELETE ALL NORECYCLE QUIET \"" + tmpFile + "\"");

    return true;
}

function OnScriptConfigChange() {
	
	Script.RefreshColumn("EbookAuthor");
	Script.RefreshColumn("EbookTitle");
    Script.RefreshColumn("EbookSeries");
    Script.RefreshColumn("EbookSeriesNum")
    Script.RefreshColumn("EbookTags");
    Script.RefreshColumn("EbookYear");
}

function OnAboutScript(aboutData){
    dlg = DOpus.Dlg;
    dlg.window = aboutData.window;
    dlg.title = scriptName + " " + scriptVersion;
    dlg.message = scriptName + " v" + scriptVersion + "\t\t\t\t" + scriptDate + "\n\n" + scriptDesc + "\n\n" + scriptCopyright;
    dlg.buttons = "Close";
    dlg.icon = "info";
    dlg.show;
}
4 Likes

Neat!!!
I'm still relying on columns from the file name convention method but I see the value of getting the data from the file itself.

Two thoughts (not requests, these are just thoughts about usability):

  • "Year" could be a valuable addition. When I have a whole collection of books by an author, I often sort by that column.

  • Not sure about the exact mechanism, but it could perhaps make sense to have a setting with a toggle for something like "Don't update metadata unless file name changes", which would allow you to cache the metadata and do a quick lookup instead of refreshing when a folder is shown again.

Kudos on the cool script. I'm in the process of updating the DearOpus pages, will make sure to add this one, DopStack and other wonderful goodies you've been baking. : )

1 Like

1.5 now provides a Year column - there's no actual Year metadata item, so the column just extracts it from the Published date.

I don't think it will be much quicker to cache the info, it's still going to have to parse and read a text file to get the info in the same way that it does now. I guess you could store it in DOpus variables but with a large ebook collection that would be a HUGE number of variables stored.

Hi Steve, thank you for explaining this! Wishing you a fun Sunday.

@playful
Looks like I was very wrong. :slight_smile:

I've been playing around with this and have a (very) beta version if you'd like to try it. It uses a cache file in /localappdata/EbookColumns/. The cache file relies on the md5 hash of the file and any time an ebook is shown in the columns which is not in the cache, it will be added.

With caching switched on, the script will generate a hash number for each file, search for it in the cache and populate the columns from it if found, defaulting back to using calibre if not. This seems to result in speeding up subsequent views significantly - I haven't create a huge cache file yet, so not sure what the speed will be like when the cache grows huge.

You can configure the cache location and turn caching off if you want.

As I said, please consider this a beta version, it doesn't do anything destructive to your files though.

Ebook Columns.js.txt (13.0 KB)

Hi Steve,

I'm as surprised as you are! I didn't want to suggest an MD5 in my original post because of the computation time involved.
Regarding the cache, my assumption was that it would be faster to look up twenty file names in a single sorted file (or sqlite database etc) than to unpack and process twenty individual ebook files, especially having to fire up some unknown heavy machinery that might come with Calibre.

The speed of your MD5-based cache suggests that a great amount of scaffolding does get erected every time you fire up Calibre!

Tested the new version on:

  1. folder with 5 books, folder with 20 books
    Very comfortable refresh time.

  2. flat view of ~ 800 books.
    The challenge there (testing-wise) was to force Opus to compute the metadata for the whole folder — only the columns for the shown files are computed. Thought there was a setting for that but can't recall. If one wanted to generate metadata for their whole library in one swoop, they could use such a setting to create a "Force compute columns" button.
    Anyhow, to force compute, I paged down the lister to the bottom of the flat view, that seemed to force Opus to compute everything. I let it run the first time around.
    After that initial run, the refresh time was more than acceptable when paging down the flat view (~ 1 second per page with 24 files.)

I assume it would be even faster if the cache were based on file names, but of course there is no guarantee that users would change file names every time they edit metadata. Given how well the script is running with a hash, I would go with that. : )

Another great script by you!
I'm sure a lot of people will be very happy to have this new facility for books! : )
:thumbsup:

1 Like

I didnt think basing it on filenames, whilst faster, was a good idea as people might have an ebook in two places with different cover art, different editions, etc but with the same filename and different metadata.

FWIW, the general thumbnail cache uses full path + modified date. Don't know if that makes sense here though.

Might be worth a try, thanks Leo.

the general thumbnail cache uses full path + modified date.

For a large PDF, that should be much faster than an MD5.
Love that the script handles PDFs, by the way.

It does? It shouldn’t. It specifically ignores anything other than .epub, .azw3 and .mobi.

edit: So it does. :slight_smile: I will permanently include .pdf files to save you editing the script.

This version uses Leo's tip re. filename/modified date. Testing shows it's faster as we knew.

I tried this with my collection of ebooks - 2800 in author folders, using flatview (worked fine btw) which resulted in a 500kb cache file - by the time the cache reaches this size it's much slower but still faster than reading the data from the actual file with calibre.

Ebook Columns.js.txt (13.2 KB)

Terrific. Please advise when you make that the official version, the one updated by ScriptWizard.

As I'm sure you're aware, from a certain collection size a database could be more efficient because of the log(n) factor of indexed lookup vs. scanning the whole file. Depending on the implementation of the database issuing SELECT queries may also be more efficient than parsing text line.

But IMO for this application it's overkill.
Really thrilled that you made this, I think I'll start using it.

Wishing you a terrific rest of the week. : )

Does ScriptWizard still work?

I see it looking for updates every time I launch Opus, but haven't checked if it updates anything. : )

None of my scripts are using Script Wizard. I thought I had read in its thread that it no longer worked.

Good to know, thank you.

Using this version, and i get this delete dialog appear for every book that is scanned.
image

Also there's an error when fetching authors name, see image

Can you either send the file to me or, using epub-meta.exe with that file send a screenshot of the command line output it generates?