Script wizard (split discussion)

Off-Topic
1

Hello,

I'm new to Directory Opus community and I"m still learning the ropes of Directory Opus and I'm been loving it.

One of the scripts that is listed pretty much everywhere is this one. It's very well crafted and, considering the limits of JScript 5.6, it's even more impressive! Excellent job @tbone!

Sadly, it doesn't work with the (new?) forums. So I decided to try to upgrade it as an exercise to Directory Opus scripting

@tbone I hope you don't mind it. If this bothers you at all, please, tell me and I'll remove the script right away.

If you do like it and would like to use it as a base to keep improving it, you're more than welcome to do so. It is your script, after all :slight_smile:

On my tests, it seems to be working nicely. Keep in mind I couldn't stress the code too much, since I'm new to DOpus and don't have many scripts installed.

Here is the file. Don't worry, it won't auto-update anyone, my modifications only check (on purpose) the first post of a forum topic :wink:

Command.Generic_ScriptWizard.js.txt (355.3 KB) v0.6.0 unofficial
(link removed per author's request)

A (terribly written) changelog:

  • Updated most links with the new Directory Opus Resource Centre forum address scheme
  • Added JSON parsing support (https://github.com/douglascrockford/JSON-js/blob/master/json2.js)
  • It now uses the UID when checking if a downloaded script is valid
  • Versions are compared using the number specified in the script itself (before, it was the forum text)
  • Download function completely rewritten to support redirects and automatically download JSON when needed
  • Download function now relies on WinHttp.WinHttpRequest to support and catch redirection URLs
  • New Function: FetchScriptsData(), responsible for downloading each script candidate and organizing them in a sorted array
  • GetVersionsOfBestMatchingResource() simplified, since the forums scheme changed

Some notes on the code:

  • I come from a C++ background. The programming style is a bit different from the one used on the base code, but I couldn't stick to the one you used. Hopefully that won't be a problem. (Mostly, the curly braces are on different lines).
  • I removed the check from your own website. It was already a daunting task to go into a language I'm not used to, on someone else's code, without the tools I'm used to work with. It's commented out, should be easy to restore.
  • I did not remove the XML parsing utilities for HTML pages. Usually I'd remove, but IDK if you have other plans for them
  • I hope you are not offended for me to upgrade your script, this was never my intent. If you are okay with this, I'd love to get your feedback if I screwed up in something or if I got things right :smiley:
  • I suggest you use Beyond Compare tool to check the differences between versions. IDK if DOpus has a tool for that, I'm sorry if it has.
  • I tried using GitHub to keep the files, but apparently git does not work well with UCS-2 LE BOM files.
  • I tried to avoid saving the files when downloading them to check versions, but I couldn't figure out a way to convert the stream to unicode/parseable format while in memory.
  • It is probably possible to use the content downloaded to save the files, without needing to download them twice (as it does now). I think major changes would be required to make that happen, though. (I didn't explore this possibility, it was already complex enough for my crash course :laughing: )

And please, to whomever is willing to attempt to use this script, please backup your scripts! While it should be safe, I'm not a veteran in JScript!

I hope to hear feedback from you guys! :smiley:

Edit: I removed the link per author's request :slight_smile:

1 Like
2

Hi Zoc! o) I really appreciate what you did and I'm quite impressed as well! o)

But I like to suggest that you remove your version for now, which does not mean all of your enhancements will be useless. There are 2 main reasons why I recommend you remove your update now o):

  • My updated SW version is done and tested for some time, I can upload any minute and I will.
  • Since you removed the fallback download location, SW will be cut off on the next change to the forum. We had that in the past and it's quite annoying to manually update SW, it's in strong contrast to it's actual aid. You aways lose some people whenever that happens, which is probably not what we want after spending time and brain on this. o)

I will look at your work for sure and I will see if I can merge some bits back into my version. I really don't mind you contributing to SW at all, but next time we really need to sync to prevent work being done twice, hu? o)

3

I just looked at the code and it seems the support for phpBB threads has been removed completely? We still need this for the german forum and the fallback download. Did you try to run the Upload-Preparation-Wizard in your version? I think it still misses support for the Discourse url-pattern and thereby cannot work. So please, take your version down and also remove the changelog and code notices so people do not get irriated. I saved your posting to still be able to read it in private.

I'm sure we will find opportunities to cooperate on this in the future, but for now, this seems a bit too short. Let's share intermediate/beta versions on other channels as well, not here. This thread should only be used for problems and help on officially released versions to keep the noise down. Feel free to contact me directly at mb_scriptwizard a@t steax (.) net. Thank you! o)

4

Hello @tbone!

I had no idea you had a a ready version! I thought I would be a helping hand, since it was a while an update was posted here :slight_smile:

I removed the previous download link :slight_smile:

About removing the fallback location... I had no idea of your plans on it, so, I gone the easy route :smiley:
(As a plus, this test version would require manual updating, so, it wouldn't break people script accidentally :smile:)

The support for phpBB is not on the main functions, indeed, but it's easy to restore. Since I didn't knew it was necessary and didn't have a forum with multiple examples to test, removed the support.

What is... Discourse?

Ah, I forgot to run the preparation wizard... the MD5 and Date at the end of the file are outdated :frowning:

Is there a place that is appropriate for this kind of stuff (script development discussions)? :slight_smile:

I'll send you an email :smiley:

Cheers!

5

Hi Zoc, Discourse is what runs this forum now, it was phpBB before.
I think we can chat in the coffe-shop area, I mean in the off-topic section as it's called now. o)

Thanks for your email and offer to merge things together. We'd need to find out what makes sense first, since some additions you implemented have been done by me as well (following http redirects e.g.) and because you changed some other basic behaviour I think we still need to look closer and check the pros and cons.

I will upload my version in the not yet existing thread in the other forum section, you are invited to take a look.

@Leo, can you cut off everything after your last post here and move it into a new thread over there?
Thank you! o)

6

Wooh! Warped over here, thank you very much @Leo! o))

@Zoc
This is my current work in progress (v0.6.2), it works quite ok as far as I can tell.
Command.Generic_ScriptWizard.js.txt (377.9 KB)

//v0.6.2 - 2017.04.12
//- urls for ScriptWizard ABOUT updated to Discourse forum

//v0.6.1 - 2017.04.12
//- support for the new DO forum and its downloads (discourse)
//- support for old redirected DO resource center forum urls
//- fix detection of scripts being enabled or not for DOv12 (in columns e.g.)
//- resolving redirected urls requires knowledge of potential proxy server in use,
//  there is support for reading HTTP_PROXY env-var and a new proxy script config
//- fix detection of script packages being enabled or not (for DOv12 at least)
//- fix "modified" status of scripts being wrong in rare cases

I went a slightly different route handling redirects I think. I first resolve the forum/script url, see where things ended and then I decide what parser to use. That way, old links and new links still work and I know quite exactly what to download and whether there is the JSON output available or not. Since Leo put efforts in keeping the old urls intact which is a really nice thing, I attempted to support this in SW as well.

Did I get this right, your version downloads every attachment and then starts looking into the files if there is an updated version? I still only use the first post information to parse version information, I actually think it's the more elegant way, especially if you use the script detail columns, where update checks happen for dozen of scripts at "once".

Thanks everybody! o)

7

Thanks for the splitting, @Leo! :slight_smile:

@tbone, I've read your script and, great job! It seems there are many improvements in comparison to the previous version! :smiley:

I will focus on the part that I have edited: downloading, parsing JSON and getting version files. The XML files from phpBB are beyond my knowledge for the little time I spent with the script :slight_smile:

About JSON/Discourse:

Identifying:
When attempting to match the URLs after the redirection I came up with the following RegExp:

var JSONRegExp = /(?!.*\.json$)^https\:\/\/resource\.dopus\.com\/t\/.*/i;

Translating from cough voodoo cough, I mean, RegExp (:smile:) , It means it matches the URL IF it starts with https://resource.dopus.com/t/ AND does not end with .json
Might be a good check to add ~line 1738:

	this.GetVersions = function(resource){
	///////////////////////////////////////////////////////////////////////////
		out.trc("GetVersions():",1);
		var down = this.DOResourceUpdater.Download(resource.urlHome+".json");

before the .json suffix is actually appended :slight_smile:

Parsing:

for parsing, I found it this is the line that does the magic (~line 1763, inside GetVersionsFromDownloadedContent() ):

			eval("var post = "+json+";");

Well... I'm kinda paranoid to that kind of eval - it's a direct execution of data retrieved from web, without prior sanitizing - Malicious people can do a nasty damage if they really want to. Sanitizing that seems to be a daunting task.

I recommend using something like this public domain JSON parser for old JavaScript clients. At the bare minimum, it will prevent execution of malicious code.

(Sorry, I'm kinda paranoid with security, and that eval something scary for me! :frowning2:)

Getting latest version
In the same function ( GetVersionsFromDownloadedContent() ), after using eval to put JSON data into post variable, it is added to a <dummyRoot> tag and then run through the XML parser.
This is what I could understand from that code, but since I don't have a memory debugger (IDK if that exists in JavaScript... I came from a C++ background, which gives some hardships but have some extra goodies :blush:), I didn't bother attempting to dissect this part of the code. I could use a rundown if if you have the time and patience :slight_smile:

From now on, there's a lot of XML parsing stuff. Some questions:

On GetPotentialScriptingAttachmentsFromPost, I see that you search for a bunch of extensions. Do you try to match .js.txt right there (aka: is there a way without downloading the file?), or just checking for one of the possible extensions there (without checking specifically for .js.txt) ?

On GetResourcesFromXMLDoc, you convert the retrieved data to a DOResource().
Then, at some point, it calls ValidateExtension() and it validates the extension in the format of .x.txt, where x must be one of the following: js, vbs, pyt, osp. Are you able to fetch that data from the server before downloading the file ?

Then there's a ParseVersion(), it's quite different from _GetVersion() one. I don't see you downloading the file there. Do you rely on some (hidden?) data on the forum posts?
I couldn't find a pattern among the scripts posts I visited when I was trying to update version 0.5.9 to a working version. So, to have something reliable, I downloaded all links after passing them through a filter and getting from it. The largest of the scripts were around 350kb, most of them has less than 100kb, so I thought it would be fine.

Last but not least, since I downloaded the files, I could match the UID of the scripts with the one being updated. This felt reliable, and worth the extra data downloaded.

Aaaaand.. wow, this post got really long! Enough questions and comments I think :confounded:(I'm sorry!)

Looking forward to your reply! Cheers! :smiley:

8

I added that lookahead piece. I'm not sure it has an effect right there, but it could be useful when validating the url entered when using the upload wizard. Thanks! o)
From my perspective, it does not make any sense in GetVersions() (around line 1738) though, since I explicitly add *.json to get the json data, hu? o)

Regarding the json parsing.. Yes, I use eval. It's not the best choice I know, but it saves around 36kbyte code. I actually don't have any security concerns here. That json data is not entered by user, it's rendered by the forum which should make sure it's safe for us and.. In the end, you are dealing with DO script addins here, which are not safe to use anyway?! o) The moment you drop one of them into the DO script addins folder, whether its manual or by SW, hell can break lose, so I think using eval() instead of a decent json-parser doesn't makes matters worse.
If you have a(nother) killer argument against the use of eval(), keep it coming, I'm still open for facts. o)

Yes there is. How depends on what parser is in use (Discourse or phpBB), but I look for attachments with specific extensions and also the version information is parsed beforehand by just downloading the initial post, no attachments yet.

Script UID is checked at the end, to really make sure we do not overwrite/update one script with something totally different. Downloading all existing attachments to check for an update seems reasonable and safest, as long as you don't try to fetch updates or the status for a good collection of scripts like this:

Clipboard Image.png846×460 24.7 KB

Sure, most people don't have that many addins, I guess most max out at around 20-30, but even then. If each of the installed addin has recent versions for download, a button to showcase what can be done, maybe an icon-set as well, then you get to a hundred downloads very quickly if you use the columns SW offers to see what's old and what's not. Downloading all the files and then looking into them, not sure exactly, but it seems overkill to me, hu? Detecting the available versions for download is quite easy with the initial forum post alone, if people stick to some very basic scheme to tag their uploads. It worked very fine in phpBB, since every upload could be "shipped" with a comment. In Discourse this is different so I parse the information within the download-link (where the script name is) and if there is nothing version-related to be found, I check the text right above the download link. Seems to do the trick, the latter method is a fallback. Ideally version information should be entered where the script filename is and then it's very similar to what phpBB offered, where there was a direct relation between an upload and some piece of metadata floating around.

9

v0.6.3 - 2017-05-28 - regex enhancements for parsing version from Discourse uploads
Command.Generic_ScriptWizard.js.txt (378.6 KB)

Tests with DO v11 are yet to be done, if you or anyone else likes to give it a try, just do so. o)

10

I'm sorry to have "vanished" suddenly. I have been sick and I couldn't dedicate time to this :frowning_face:

Previous your updates, I had split you script in a way that made sense to me (I'm used to work with different files, separated by their specific functionalities, in C++), and built a small script that would put all of them together. The result is the original script.

If you want, I can send it to you. Maybe you have some use for it (since the current script is long and hard to navigate for someone that's not used to it, I thought it could be a nice addition :slight_smile: )

Sadly, downloading all the scripts on the first post is really not the fastest way to do it. I don't know if MS JScript supports multi-threading, so it can indeed be slow for a large list, so I'm not sure if accuracy is a good trade-off for speed here.

If it was possible to build a Discourse addon/plugin (no idea if it support these) which could output the script data in a reliable, easy to parse way (i.e. together with the download link, in an expected format), it would be the fastest and accurate way.

I wish I could help more, but sadly I need to recover. This illness "drained" my money pool right now, so I'm not even sure if I'll be able to buy the upgrade from light to pro (I still have 30 days of evaluation to go).

If there's some specific problem I can tackle, please, let me know. I would like to help with something - and it could be a good exercise while I"m recovering :slight_smile:
(Otherwise, I'll probably port my open source updater from C to C++ for ease of use :slight_smile: )

11

Discourse lets you output things in XML fairly easily I believe. Leo can probably explain how.

12

Hello @Jon,

We're using JSON to be able to parse it easily. (I tried parsing the regular outputted page, JScript didn't really like parsing through it).

On @tbone solution, he does some wizardry to parse it. While I thought it was insecure, it works and he firmly believes that no script developer would add a nasty file in there that could harm the end user.

On my solution, which adds several lines to end the script file, I added a fully featured JSON parser (from this link) with comments removed. Still, it adds around ~200 lines to the script.

Anyway, JSON works for both of us. What I meant is, when an user uploads a script file, it would try to parse the script on server side, and, if it detects it's there's some key script information in there, it would add those in the forum topic in the same place as the download link. This way, we would be able to detect it without either needing to parse the script (needing to download all of them, in my solution) or relying on proper user formatting of the post (in @tbone solution).

Probably due to my background, I'm almost always a bit concerned about security and accuracy, but it might indeed be an overkill here, as @tbone pointed out to me.

(PS: I'd like to apologize if my post seems confuse or my English is unclear right now - like I said, I'm not in the best shape. Hopefully I could explain what I meant more clearly :slight_smile: )

13

Okay, now, my health is finally starting to improve! yay!

So, I upgraded from dopus light to pro, when my trial period ended. I decided to start fresh, delete everything and customize from scratch. (While the "Dear Opus" toolbar is an awesome starting point, it does not looks really pretty in my opinion :frowning: )

I installed the latest beta from this thread.

While it did attempt to upgrade scripts when dopus was launched (I configure it this way), I couldn't use any of it's functions. About menu for instance didn't work (failed silently)

I decided to try an older version and re-install the menu button. Everything worked (except updating, of course).
Put the latest beta in place (0.6.3), everything is working.

I"m not sure if the latest script is failing to setup something, but I thought this would be good to report.
I don't dare try to reproduce this because... I resetted opus a few days ago and now things are finally starting to look how I want them, I don't want to break anything again until I have a solid configuration. I'm sorry :frowning:

Oh, btw @tbone... Before getting sick, I made a small utility to split the files into different sections and put them together nicely, aiming to make the script support GitHub nicely.

I would love if you could take a look at it (even if you think you don't won't use it), maybe it could help you in some way.
If it isn't useful to you... Well, it's fine I guess... It's already done, so it's not like I'd spend more time on it. :slight_smile:

Thank you very much for your awesome work! If you need a hand in a specific feature, just ask! :smiley:

1 Like