Script @include topics

[tbone]

Hi there! o)

I was playing with the new include feature for external scripts/snippets. I think it is a nice addition to the scripting functionality (and I'm quite curious how you solved this technically, but offtopic here o)!

Some things came up my mind while looking closer, please read on.. o)
The inc_ Prefix:
I would prefer if included files could be separated into a folder, instead of mixing them into the 100+ script addins I already have in /scripts. Is the prefix really necessary? Could we add support for a path to the file to be included, supporting relative and absolute paths and aliases like /scripts as well? So we get the usual DO experience? o)

"@" vs. WSH:
The "@" character is reserved in MS JS in connection with conditional compiling. Any JS script using that "@include" statement at the top will give following error if run with the MS JS interpreter.

>cscript.exe inc_runHiddenExAIO.js
..\inc_runHiddenExAIO.js(2, 9) Microsoft JScript compilation error:
Conditional compilation is turned off

The MS JS interpreter would handle @if @end etc. if conditional compiling is enabled, you can enable cc by adding /@cc_on @/ at the top of the JS file. It still does not know about @include statement though. It will give following error message with cc enabled:

>cscript.exe inc_runHiddenExAIO.js
..\inc_runHiddenExAIO.js(3, 10) Microsoft JScript compilation error:
Expected ';'

"@" vs. nodejs:
Also nodejs will complain if @include is to be found in the script:

>node inc_runHiddenExAIO.js
..\inc\inc_runHiddenExAIO.js:3
@include inc_myinclude.js
^
SyntaxError: Invalid or unexpected token
←[90m    at internalCompileFunction (node:internal/vm:73:18)←[39m
..
Node.js v18.15.0

Nodejs is using a function called require() to include external files and modules, it also handles some "return value" from inside the included script. The script would just assign a value to "module.exports" and in the parent script you can pass that value on.

• Node.js Export Module
• Requiring modules in Node.js: Everything you need to know

var foohandler = require('/path/to/foohandler.js');

Using require() without "module.exports" will just run the external code and not ex/import any members.

And now?!:
I'm all in for an "include" of some kind, don't get me wrong, but we should make sure, it does not interfere with existing interpreters, editors and syntax highlighters and is not to be confused with @-modifiers, at first I was thinking "modifier!", but it's not to be mixed up with DO modifiers using the same syntax, right? Anyway, adding "@" to the JS files will break them, it is basically the same thing I tried to bring to your attention in 2016, where you added resources to a JS file, by injecting "== SCRIPT RESOURCES" straight into valid JS code - it was a very "quick" idea from my point of view (which still can be enhanced, no problem! o).

• DO12 - dialog resources

EASY FIX!? o)
As a first measure, until we might have a syntax friendly and nodejs compatible require() function one day, adding a simple "//" to the front of @include would make things work for now, so.. can we switch to this syntax?

//@include inc_myfile.js

Same for "== SCRIPT RESOURCES", separating the script resources from the JS body by using "//"..

//== SCRIPT RESOURCES
//..
//== SCRIPT RESOURCES - END

..would not destroy the JS syntax and the best thing:
This has no technical limitation or drawback at all (I think)! o)

The greater picture I have in mind is..

• keep/restore compatibility with jscript + nodejs syntax
• do not create a "walled garden" with new DO JS script syntax
• support for GIT repos in the /scripts folder in the future (getting more and more important)
• introduce proper require() like function (allow any rel/abs path)
• keep things future proof, without adding "hacks" which are hard to work around

Thank you! o)

[Leo]

Why do you want to run Opus scripts outside of Opus? None of the Opus-specific objects will be available, so the script won't work (unless it doesn't do anything involving Opus itself, in which case there's no reason for it to be an Opus script; it's just a normal script outside of Opus at that point).

[tbone]

Well, you can run test and tools on those files and you can use mockups for the DO objects (which I actually did) and in my scripts are plenty of functions not relying on DO objects. You simply can't source / include a script file containing "@include" anywhere, you cannot run auto-formatters, linters, autodocs or anything which expects a proper JS syntax.

Maybe you want to use an online repo at some point, what file type is a DO script addin for Github? It's not a *.js file anymore. Every web / code editor on earth supporting JS and syntax highlighting will also choke on the @include and == SCRIPT RESOURCES injection.

Basically anything that can be used to handle a regular JS file will fail - that's not good! o)

I think the question rather is:
Why would DO break the general JS syntax, when there is no reason to do so?

Thank you for taking the time! o)

[Leo]

@tbone

Why would DO break the general JS syntax, when there is no reason to do so?

Because there are plenty of benefits in being able to combine resources etc. into a single file, while there is no reason for most people to want to run Opus scripts outside of Opus.

[cyilmaz]

I agree that running DOpus scripts outside DOpus is not a common or necessary task, but I second the request (which I have also requested before) that @include, ==SCRIPT RESOURCES, or any custom statement should be JavaScript-syntax compatible, so we can at least use external editors with proper linters & syntax checkers. I had to open a thread to figure out loadResources() just to get rid of ==SCRIPT RESOUCES block. As much as quick scripts can be edited with internal editor, there's no substitute for a proper editor like VSCode, Sublime, etc.

Double-slash or TypeScript-style triple-slash or JSDoc-style would work perfectly fine, too, e.g.

/// @include...
/** or simple /*
==SCRIPT RESOURCES
*/

[Jon]

We aren't going to change this, as it would break everyone's existing scripts. Opus scripts are not designed to be used outside of Opus.

[tbone]

@Leo

Because there are plenty of benefits in being able to combine resources etc. into a single file

Hu? Where did I say, that dialog resources and things should not be added to a JS file?
They just need to be "escaped" or hidden in a comment from the regular JS syntax, so external tools / editors / code formatters etc. don't fail. Yes, @cyilmaz like this! o)

@Leo

while there is no reason for most people to want to run Opus scripts outside of Opus.

True, but please do not ignore all the facts I listed, it's a huge list of drawbacks and it's against industry standards as well I guess, to inject foreign statements into an existing language / syntax. If that has to be done, they will always be wrapped into comments, to keep things compatible.

If you wrap @include and ==SCRIPT RESOURCES into comments, nobody will complain. It will not make a difference to anybody but people trying to use JS related tools and functionality. Everybody can be happy and standards are raised, it's a win win?! o)

To prove the point, I will give some examples of injected directives and statements out there:

This is how Microsoft added they CC_ON directive, it's in a JS comment:

Microsoft again, telling IE8 to load CSS file, in a HTML comment:

This is Linux, telling what interpreter is to be used, in a comment:

This is something for the GO language, in a comment:

This is the ECMAScript (es) linter, being disabled and enabled in a portion of the JS file:

Again "eslint", single line instruction this time, hidden in a JS comment:

This is c-sharp, injecting tooltip for the IDE (in a comment AND using a tag syntax on top):

This is documentation added to JAVA code (javadoc):

They all do this, to keep the code compatible to the related tools around, makes sense to me!

[tbone]

@Jon

Opus scripts are not designed to be used outside of Opus.

Why is everybody ignoring the big list of drawbacks in conjunction with JS related tools?

@Jon

We aren't going to change this, as it would break everyone's existing scripts.

I think it would not, since..

/** or simple /*
==SCRIPT RESOURCES
*/

will parse just as before I think, even if it does not, you know how to handle a little difference in strings you are loading and parsing! I know that you can do it! o)

And.. I have not seen a single script using @include up until now, so now is the time!
It's never more easy than doing it now and back in 2016 you also had the chance (just saying.. o).

See you! o)

[Leo]

@tbone

Why is everybody ignoring the big list of drawbacks in conjunction with JS related tools?

To be blunt, we don't see that as a drawback that matters, especially compared to breaking existing scripts and making it harder to include the script and its resources in a single text file.

Opus scripts tend to be a few lines long, or a page or two at most. You simply do not need linters, mock objects, or any of that stuff to write and debug such simple scripts. If a script is more complex than that, it should probably be split into separate components for the Opus-specific part and the rest of the script, which the Opus-specific script could launch or talk to in various ways.

[tbone]

@Leo

especially compared to breaking existing scripts and making it harder to include the script and its resources

You are trolling me, are you? o)

I think you guys managed much harder tasks than wrapping an xml string into a JS comment or trimming some "//" from the front - for sure! o) Writing thousands of lines of code for the script GUI editor and creating hundreds of objects and properties including lots of documentation comes to my mind e.g.! o) Man, you guys are highly skilled C++ developers, what are you trying to tell me? o)

Unfortunately, sometimes you tend to play down the role of the scripting functionality in your own application - I really wonder why. It's the best in the world in a file manager, please realize that! o)

If all the scripts are "a few lines long, or a page or two at most", then why was @include invented?
To include even smaller snippets? C'mon! o)

I have 30 (of 120) scripts ranging from 30kb to 60kb, up to 350kb+, these are not "single pagers". I could actually make use of //@include and proper external JS tooling, because nearly all of my scripts use variants of Xlog(), DumpObject(), ArgsMagic(), OnAboutScript() etc.

I was thinking on moving to an online repo with all of this. Handling GIT repos inside the /scripts folder does need some work from your side though! Another topic of course.. o)

Whatever the future brings in that regard, we could try to adhere to existing standards to not block roads unnecessarily. I wonder how you would react if somebody inserts some random statements from the ruby language into your C++ code and tells you: "It's ok, C++ is not used outside Visual Studio, we can invalidate the syntax!" - uhm what!? o)

Maybe consider a prefixed "//@include .." variant at least. I think that would already help to not offend the experienced scripting gurus around. o) Just take a minute and think again, I'm out for now, made my point I guess, will not argue any longer, will wait and see.

With all the things you implemented to make DO scripting fun and useful, the scripting API is not some kind of fifth wheel on the wagon, it's a first class citizen in DO - don't leave it scar faced! o)

cu! o)

[MartO]

While I think DOpus is a fantastic product and its configurability is beyond any other software I've seen (and sometimes beyond belief ), the real star for me is the scripting. Over the years I have written loads of scripts for simple, every-day tasks plus some rather more complicated ones that all revolve around file handling in DOpus. And I've built up quite a collection of modules I keep reusing.

The script development features (including the extremely useful dialog editor) included with DOpus are more than sufficient for most users, I'm sure. On the other hand, just like many users love DOpus and - unless forced - will never use anything else for their file management needs, in developer circles it's quite normal to have a favorite editor. So the impulse to use it for DOpus development as well will come quite naturally. This is by no means a complaint about DOpus' abilities!

As a developer myself I know the feeling when a customer's wishes don't adhere to my own vision for my product - it can be highly frustrating. But perhaps the DOpus team can take it as a testament to the versatility of their product that we even have those wishes?

Enabling us to include external modules and script resources without clumsy workarounds already went a long way to making life easier for us. In fact, for me the addition of loadResources() was just what I needed to solve that issue. I wrote myself a little script to translate the files between pure XML and the version needed by the DOpus dialog editor, so I can enjoy the best of both worlds.

So the last issue, for me at least, is the @include line. It just hurts my feelings to see VSCode accusing me of having errors in my files... Accepting both @include and //@include would solve this issue and this

@Jon

it would break everyone's existing scripts.

wouldn't be an problem. What do you think?

I understand and agree with tbone's arguments that as a matter of principle breaking a language with new syntax should be avoided wherever possible. But I also understand that the ideas proposed here just aren't what the developers have in mind, so arguing with the integration into a node.js environment must sound rather exotic to them.

Apart from all that, let me repeat: I'm quite happy with the scripting already - and with the support we're getting from the team!

Just my two cents...
Have a great day!

[Leo]

@MartO

Accepting both @include and //@include would solve this issue and this

So if you comment out an include to test something, it'll look like it has been commented out but it'll still be applied. That seems a worse problem than the one you're trying to solve.

[tbone]

Good morning everyone! o)

@Leo

That seems a worse problem than the one you're trying to solve.

I would think the problem on how to add "bells and whistles" to an existing language or syntax or set of statements is already solved. I gave several examples on how other projects do this, and I like to give another one from the hydrogen project.

// ==PREPROCESSOR==
// @name "Playback Buttons - Menu"
// @author "marc2003, modified by Defender"
// @import "%fb2k_component_path%samples\js\lodash.min.js"
// @import "%fb2k_component_path%samples\js\common.js"
// ==/PREPROCESSOR==

var b_single= 1;	// 0=Select multiple buttons
if (b_single==0) {
	b_menu					= 1;	//	1		Main Menu				- Help Menu

In general you have to options (I think):

1. Add "preprocessor" like statements, which the interpreter does not care about in comments (like in the example(s) above).
2. Add a real function or object to the scope, like you already did with the "DOpus" object e.g. or like any browser binds to the JS functionality, by adding the setTimeout() function, the "window" object etc.

Both options do not invalidate the actual JS syntax, which is seemingly important to not break with an existing world.

@Leo

So if you comment out an include to test something, it'll look like it has been commented out but it'll still be applied

Well yes, but NO! o)
You cannot use valid JS syntax (comments), to comment out something, which is not part of the language. That's why something like @include is always in comments. If you want to disable the @include, you have to come up with a different syntax, you can't mix them up.

If you want comments to work for the include functionality, you have the option to chose #2 for your extension, which means, add an include() method to the scope, which can be commented out easily.

The current @include seems to combine some negative side effects of a language extension.

This seemingly valid code, is also not valid.

There are some more variations and new restrictions on how to format code and where the @include statements needs to go. It is kind of unpredictable it seems? Probably yes, it has to be. It is some kind of preprocessing statement with invalid language syntax in the interpreted scope, where only valid syntax should live - this calls for trouble.

This is not working, I thought it would, since it would keep the JS syntax valid at least, but it does not. It's totally up side down of how these things work in other projects / environments.

Thank you all! o)

[Jon]

We aren't adding anything to existing languages.

Opus scripts are not javascript. They may contain embedded jscript (or vbscript, or python, or whatever) but they are not javascript themselves.

[pro]

It shouldn't be a "fix" as much as it'd be an addition. It's basically a feature request, opposed to an implied bug report.

I also edit scripts externally. The thing is that when I started doing so, I didn't expect the Opus syntax to be as compatible as it turned out to be. I expected it to throw errors all over, so it came as a pleasant surprise it did not. It seems that some have gotten so used to this, that when it shows errors for them it's bothering them, which I totally understand.

All the proposed syntax alluding to "just accept commented out code" isn't great though. All the examples given aren't just comments, they are meticulously constructed to not interfere with normal code comments.

[tbone]

@Jon
Of course we are talking JScript here, which is a Microsoft implementation of ECMAScript 5 specification (iirc), which is known as Javascript to the world by now. Current Javascript is ECMAScript 6+, which added a lot of new statements, but JScript is still compatible to that generic Javascript interpreter. DO scripts are JScript(s) to me or VBScript or Python etc., because you make use of the WSH under the hood, right?

Is that the reason for the @include, to make it work with all languages supported by WSH?
That gives some background, but still.. I think we should not implement something, which "just works" for all the different interpreters, when it messes up the specific language syntax.

[cyilmaz]

Comment directives are quite common in JavaScript world, ESlint being the 800-pound gorilla in the room using these, TSchecker being its little sibling, but other languages use them as well. In fact, TypeScript, which is a superset of JS and by no means a niche language, uses triple-slash /// exactly for this purpose, to include files: ///<reference path="./mydir/myfile" />

I find you guys' opposition very unfortunate. Yes, we are very happy that you are listening to users and adding new features, but some of these additions break language rules. There is nothing remotely similar to @-directives or ==SCRIPT RESOURCES in JS or JScript world. Remember this thread about maps? I had requested the change because it broke JS rule that one cannot use RHS statement as LHS value. This is the same problem all over again. @pro is right about it, if the added feature is relatively new, it can be still adjusted.

If //@include looks weird to you (to me it doesn't, not one bit), then why not use more JS-like syntax like: var mylib = require('mylib.js) or import * as mylib from "mylib.js"?

[Jon]

You're missing my point. The language syntax of Opus scripts is not messed up. @include is fully compliant with Opus script syntax, as is ==SCRIPT RESOURCES, as is @script:jscript as are all the other modifiers that Opus buttons let you use around the embedded script code, which can be in any supported language.

The fact that you're choosing to run an Opus script through a linter or whatever else that's designed to accept javascript is entirely up to you

How would adding support for //@include help someone who wanted to run an Opus script through a vbscript linter? (answer: it wouldn't). What you're asking for is language specific, but Opus scripts by design are non-language specific.

[tbone]

@cyilmaz

If //@include looks weird to you (to me it doesn't, not one bit), then why not use more JS-like syntax like: var mylib = require('mylib.js) or import * as mylib from "mylib.js"?

I think Jon was thinking of all the other languages, which the Windows Scripting Host is able to run, he came up with something, which "worked" for all of them. I think this approach is to be respected and is in general a good idea! o)

[cyilmaz]

Alright, this is unproductive. We're at a deadlock as far as I'm concerned. I for one will never use @import or ==SCRIPT RESOURCES (EDIT: that's what I'd call a dead-on-arrival feature), a linter is 10x more important to me than some feature for which there are workarounds.

To anybody else, @tbone, @pro @MartO and others in the future, who want to use modularization sticking to JS rules, superior IDEs, linter, etc. tools. I cannot recommend using TypeScript and transpiling enough. I had posted about this, but can summarize my most recent findings in a separate thread again.