DOpus-Scripting-Extensions project. Wild idea

Some time ago, I made a comment on the topic:
Script to run external commands hidden and capture their output without temporary files
where I "foolishly" stated:

What I wanted to say is: @Leo, you could add an extra method in the DOpus scripting API to cover this use case properly. It should be easy to do that from C++ (especially if you use Boost :slight_smile: ). A method like var res = Dopus.Utils.run("prog.exe args") and res will contain exitCode , stdOut , stdErr

Little did I know how difficult it would be to implement such a class :slight_smile:
Everything is difficult in C++. It is the ultimate test of your sanity.

I managed to implement it, and it led me to the idea of creating a project that would contain helper classes extending JScript functionality that can be used for writing DOpus extensions:
PolarGoose/DOpus-Scripting-Extensions

Currently, it only contains a ProcessRunner class.

I started this discussion in case someone finds the idea of this project useful and sees how it can be extended further.

7 Likes

Thanks very much for sharing that.
That ProcessRunner idea (avoiding temp files) is a great one.
I'll let you know if I can think of extensions for this project.

Note: I tried to look at the github repo, and realized ... that it's really been years since I touched C++ :slight_smile: and never for that kind of application, so I won't probably be of any help on that part.

Thanks for this. It's truly apreciated.
In a quick demo test, it works on Windows 10 22H2.

I'm guessing it won't work in a portable environment because the DLL needs to be registered first, right?

I recall reading about a way to "declare and use" a library without needing registration—I think it's called Registration-Free COM. I hope that helps.

Again, thanks. We definitely need more initiatives like this.

@PolarGoose, here's some feedback with time benchmarks from quick tests I did using a script that calls Mediainfo utility:

  • With the "old method" (using WScript.Shell Run, then reading the resulting temp file): 300–330 ms
  • With DOpus-Scripting-Extensions: 60–75 ms

This is definitely great news! :partying_face:

Thanks again.

Something that would be extremely useful is the ability to directly call functions from a DLL library. For me, that would be like the "holy grail" when it comes to scripting in Opus.

Since I'm not a programmer and just an enthusiast, I wonder if it's possible to include this functionality.

2 Likes

Interesting numbers. This opens the ability to build wrappers of many kinds, since these numbers (gap between "old" and this method) will grow higher with the quantity of output (avoiding the disk write at execution and the read after that to get the data).
I think I have a couple of scripts that include a sqlite wrapper: I need to test this !! :slight_smile:

@errante, @PassThePeas,

Thank you very much for your interest.

I'm guessing it won't work in a portable environment because the DLL needs to be registered first, right?

Yes, it will not work in a portable mode. You have to register the DLL and then unregister it when it is not needed. That is why I had to create an msi installer.

I think it's called Registration-Free COM . I hope that helps.

Thank you for the hint. Unfortunately, I don't see how it can be used for the JScript use case. You need to control the manifest of the main application that consumes the COM object.

Something that would be extremely useful is the ability to directly call functions from a DLL library. For me, that would be like the "holy grail" when it comes to scripting in Opus.

Similar to AutoHotkey DllCall. Unfortunately, JScript doesn't support that.

One of the ideas I had for this project was speeding up my File MIME type column by creating a custom COM class that will use libmagic to determine MIME type and it will be much faster because you need to load the magic file only once when DOpus starts instead of doing it for every file by calling file.exe
MediaInfo is another use case, it is possible to use MediaInfoLib to get the media information, it should be much faster.

Hi.
I have a 'small' issue :slight_smile:
I want to execute a command that includes a pipe.
The sqlite3.exe requires a command line call in the form echo <sql statement> | sqllite3.exe [-json] <sql db location>.

What would be the syntax to call the ProcessRunner to achieve that?

FWIW, here's the actual code relying on WScript.Shell:

	var fso 	= new ActiveXObject("Scripting.FilesystemObject");
	var shell   = new ActiveXObject("WScript.Shell");

	var tmpFileBase = "DO.Temp.";
	var tmpFileExt = ".txt";
	var tmpFileStdout = ExtSystem.GetTmpFileName(tmpFileBase, tmpFileExt, fso).fullname;
	var tmpFileStderr = tmpFileStdout+".err"+tmpFileExt;
	
	var cmdLine = '%comspec% /c "echo ' + sqlStatement + ' | ';
	cmdLine += '"' + sql + '" -json "' + db + '"';
	cmdLine += ' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""'

	var result = {};
	result.cmd = cmdLine;
	result.returncode = shell.Run(cmdLine, 0, true);
	result.stdout = ExtSystem.ReadFile(tmpFileStdout, fso); fso.DeleteFile(tmpFileStdout);
	result.stderr = ExtSystem.ReadFile(tmpFileStderr, fso); fso.DeleteFile(tmpFileStderr);

With ExtSystem defined as follow:

if (typeof ExtSystem === "undefined") {
	var ExtSystem = (function () {
		var my = {};
		///////////////////////////////////////////////////////////////////////////////
		my.RunHiddenEx = function RunHiddenEx(exe, params, tmpFileBase, tmpFileExt, shell, fso) { //v1.2
			if (!fso)   fso		= new ActiveXObject("Scripting.FilesystemObject");
			if (!shell) shell   = new ActiveXObject("WScript.Shell");
			if (!tmpFileBase)   tmpFileBase = "DO.Temp.";
			if (!tmpFileExt)   	tmpFileExt = ".txt";
			var tmpFileStdout = this.GetTmpFileName(tmpFileBase, tmpFileExt, fso).fullname;
			var tmpFileStderr = tmpFileStdout+".err"+tmpFileExt;
			var cmd = '%comspec% /c ""'+exe+'" '+params+' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""';
			//dout ("CMD='" + cmd + "'");
			var result = {};
			result.cmd = cmd;
			if (exe.match(/^([A-z]:\\.+|\\\\(.*?)\\.+)$/)) { //test path to exe if given
				if (!fso.FileExists(exe)){
			        var msg = "E   Executable not found ["+exe+"]";
			        DOpus.Output(msg);
			        throw msg;
		         }
				 //else dout (">> '" + exe + "' exists");
			}
			result.returncode = shell.Run( cmd, 0, true);
			//result.returncode = shell.Run( cmd, 1, true);
			result.stdout = this.ReadFile(tmpFileStdout, fso); fso.DeleteFile(tmpFileStdout);
			result.stderr = this.ReadFile(tmpFileStderr, fso); fso.DeleteFile(tmpFileStderr);
			return result;
		}
		
		///////////////////////////////////////////////////////////////////////////////
		my.ReadFile = function ReadFile(path, fso){
			fso = fso || new ActiveXObject("Scripting.FilesystemObject");
			var content = "";
			if (!fso.FileExists(path)){
				return content;
			}
			var file = fso.OpenTextFile( path, 1, -2); // Read, UseDefaultEncoding
			if (!file.AtEndOfStream)
				content = file.ReadAll();
			file.Close();
			return content;
		}

		///////////////////////////////////////////////////////////////////////////////
		my.ReadFileByLines = function ReadFileByLines(path, fso){
			var vOut = DOpus.Create.Vector();
			fso = fso || new ActiveXObject("Scripting.FilesystemObject");
			var content = "";
			if (!fso.FileExists(path)){
				return vOut;
			}
			var file = fso.OpenTextFile(path, 1, false, -2); // Read, do not create, Unicode
			while (!file.AtEndOfStream) {
				// DOpus.Output("ext system : reading 1 line");
				vOut.push_back(file.ReadLine());
			}
			file.Close();
			return vOut;
		}		
		
		///////////////////////////////////////////////////////////////////////////
		my.GetTmpFileName = function GetTmpFileName(prefix, extension, fso) {
			fso = fso || new ActiveXObject("Scripting.FilesystemObject");
			var tFolder = fso.GetSpecialFolder(2); //2 = temp folder
			var tFile = fso.GetTempName();
			if (prefix!=undefined) tFile=prefix+tFile;
			if (extension!=undefined) tFile+=extension;
			return {
				path	: tFolder.Path,
				name	: tFile,
				fullname: tFolder.Path+'\\'+tFile
			};
		}

		return my;
	}());
}

EDIT:
I found a way to avoid the pipe ... there's an alternate syntax in case of a single sql statement (sqlite3.exe [-json] <sql db location> <sql statement>).

But I have two issues:

  1. Calling Run method leads to a cmd window flashing which was avoided with shell.Run(cmdLine, 0, true);
  2. The execution time is surprinsgly longer than with previous method : about 1800 ms vs. about 800 before. Could be related to the window "flash" ... but I'm not sure.

I was wondering about that.
You had me looking into my DOS7 manual.
Interesting though is the example used in the manual for pipes uses the DOS Sort command which allows a sort on character n of a filename.

Your post is beyond my understanding but I get parts of it.

The last time I used a DOS pipe was to get a Cout left by a programmer in an old Exif command line program . That was long ago, but I still have the resultant button. Directory Opus exceeded that goal long long ago.

Pipes are meant to redirect the output from a command so it is considered as the input of another command (the one after the pipe). So sorting the output of a command is actually a common good example:

  • Normal sort behavior: sort <some list of text>
  • When used to sort the result of some other command that outputs text: someCommand <some input> | sort (example: find /search/root/dir -name "somePattern*" | sort)

In unix, this can be sometime avoided by using the back-quote:

sort  `someCommand <some input>`

That way the shell executes someCommand first and its output is used directly as an input for sort.

@PolarGoose: Since I succeeded to avoid the pipe, it's not so important/urgent, but should still be interesting to know if there's a way to do that with ProcessRunner. As of now, I'd be more interested in having the ability to run the command with the command window hidden (and check if it has an impact on performances).

Running it via cmd.exe should let you use the pipes:

cmd.exe /c echo <sql statement> | sqllite3.exe [-json] <sql db location>

(the /c tells the new cmd.exe to exit after it finishes, which is vey important if the cmd window is being hidden. Otherwise you can get lots of hidden cmd.exe processes building up which can only be seen/closed in Task Manager.)

Thanks Leo.
This is actually pretty close to what I am currently doing (see my post above):

	var cmdLine = '%comspec% /c "echo ' + sqlStatement + ' | ';
	cmdLine += '"' + sql + '" -json "' + db + '"';
	cmdLine += ' >"'+tmpFileStdout+'" 2>"'+tmpFileStderr+'""'

The only twist is that I need to retrieve the output of the command (the sql statement being just a SELECT). Currently I'm using temp files to which the output is redirected and I am trying to use the ProcessRunner class provided by PolarGoose and was struggling with the way to use the pipe with this class.

Would it be a future feature to have such an object built in Opus, allowing to execute external code but with the ability to retrieve output without reverting to using temp files ? (and one allowing pipes :slight_smile: )

Some of the characters might need some extra escaping, maybe. Not sure though. What to escape and when with DOS/BAT/CMD has always seemed very arcane to me.

A built-in method would probably have the same problem as piping is a feature of cmd.exe not Windows as a whole (at least to my knowledge).

I have released a new version of the library Releases - PolarGoose/DOpus-Scripting-Extensions.
Now, it shouldn't create a console window. I have tested it in a DOpus script.

Speaking of pipes, I have added an example of how to use the pipe syntax:

res = processRunner.Run("C:/WINDOWS/system32/cmd.exe", ["/c", "echo test string| findstr test"])

@PassThePeas, could you please check if it works for you and has a good performance?

1 Like

Many thanks for this quick answer. I will try the new release as soon as I can (probably over the week-end) and report back on performance impact.

@Leo : Sorry, I must have been unclear. The current code is working fine (and I did struggle a bit to build the command line). I just saw an opportunity with PolarGoose ProcessRunner to avoid using temporary files. Anyway, no much more time to spend on this for you, there's quite an activity on the forum these days with many minor glitches or documentation clarifications requested and a lot on your hands to address and answer.

EDIT:
I just tried the new release (v2.0):

  • No more window flashing :+1:
  • Performance is greatly improved on that use case:
    • Previous method (based on temp files) : Approx. 700 ms per call without pipe
    • ProcessRunner: Mostly between 160 and 200 ms (first call was 1200 ms but maybe some initialization of the component).
  • I also tested with pipe, but performance are largely impacted by that:
    • Previous method (based on temp files) : Approx. 800ms per call using pipe
    • ProcessRunner with pipe: Approx. 800ms per call too !!

This is only one use case with pipes, so I can't say for sure that using pipes will always impact performance "so much".
Anyway, I'll keep the new way (ProcessRunner and no pipe :slight_smile: ) !
Once again, thank you very much !

1 Like

Good ideas! I hope you'll find the time to add those. Exiftool could also be a good candidate.

It would be great if it were possible to "extend", so to speak, the current Opus Scripting API, so that these libraries could be accessed directly from Opus without needing prior registration, don't you think?

I found out you have to be very careful about the way you define the array of parameters especially when you have switches that take a parameter and both are separated by a space: you need to define the switch as one parameter in the array and the value as a second one.

In the example below (ImageMagick convert.exe call), I had to have the params array defined as this to work:

	var paramsArray = 
		[ 
			"-background",
			 "#ff5722",
			"-fill",
			"#000000",
			"-size",
			"227x47",
			"-pointsize",
			"22",
			"-gravity",
			"West",
			"-font",
			"\"" + font + "\"",
			"caption:"+ caption,
			"\"" + destFile + "\""
		];

Note that in that example, in order to have leading spaces at the begining of the caption string, it is defined as such: var caption ="\ The caption string".
In command line call, you have to make it caption:"\ the caption string" but with ProcessRunner you actually have to make the parameter as above ("caption:\ the caption string" omitting the double quotes). I guess ProcessRunner is doing something behind with each maramater.

Maybe this could be useful to explain that on the github page.

@PassThePeas,

I have released a new version: Releases. I have improved the performance even more. I have found the main root cause of performance issues:
Boost issues - [Windows] Setting the capacity of the underline pipe
I have created a workaround, and now it is very fast:
The number from my performance test:

ProcessRunner_test_executable_with_small_output
WScript.shell: 207ms
ProcessRunner: 88ms

Running test: ProcessRunner_test_executable_with_1mb_output
WScript.shell: 215ms
ProcessRunner: 88ms

ProcessRunner_test_executable_with_30mb_output
WScript.shell: 1050ms
ProcessRunner: 223ms

Good ideas! I hope you'll find the time to add those. Exiftool could also be a good candidate.

Thank you, good proposal. I need to look into it.

It would be great if it were possible to "extend", so to speak, the current Opus Scripting API, so that these libraries could be accessed directly from Opus without needing prior registration, don't you think?

Yes, ideally, it would be great. I think if your Registration-Free COM thing can be implemented, then DOpus could have a folder like Plugins where you could place extra COM DLLs that then will be picked up when you start DOpus and get available in JScript.
On the other hand, I really don't know, is portable DOpus such a big thing? You can only carry it on a flash drive. You can't just copy-paste DOpus on your machine and use it like you can do with SublimeText or another SW that offers a portable option. If you don't need portable DOpus, then installing a COM dll via an installer is not such a big deal.

2 Likes

First of all: I haven't found the time to try it yet, but this project is a brilliant idea, thank you very much!

Thinking of the ProcessRunner, another application that would stand to profit is ffmpeg.exe.

Which leads me to an idea: ffmpeg (like many command line tools) does not accept a list of files to run on. So would it make sense to add looping functionality to the ProcessRunner? Pass a command with a placeholder and a list of files, then collect all the output and pass it back in another list when finished or, alternatively, if one of the calls returns an error? Would that speed things up or is looping in JS and making individual ProcessRunner calls just as fast?

Thanks ! Works great.

In the example below (ImageMagick convert.exe call), I had to have the params array defined as this to work:

You don't have to fiddle with " sign. This code works:

var font = "Consolas"
var caption = "test"
var destFile = "some name with spaces.png"
var paramsArray =
  [ "-background", "#ff5722", "-fill", "#000000", "-size", "227x47",
    "-pointsize", "22", "-gravity", "West", "-font", font,
    "caption:"+ caption, destFile ];
var res = processRunner.Run(
  "C:/Program Files/ImageMagick-7.1.1-Q16-HDRI/magick.exe", paramsArray)

Parameters are passed into the executable as is even if they contain spaces, they are still considered as 1 parameter. It is also how Python subprocess works.