Vector persistency issue

Help & Support
1

Hi all!

I tried to make use of the persistent variable (vector) feature and failed.
After a restart of DO, the vector saved into DOpus.Vars exists, but is of zero length.

Any hints appreciated, in case it's me being the faulty part. o)
Thx!

Demo-Code:

//DOpus.Vars.Delete("MyVector"); if (DOpus.Vars.Exists("MyVector")){ DOpus.Output("MyVector exists"); var myVector = DOpus.Vars.Get("MyVector"); DOpus.Output(" Retrieved Vector persistency: " +DOpus.Vars("MyVector").persist); DOpus.Output(" Retrieved Vector length: " +myVector.length); DOpus.Output(" Retrieved Vector(0): " +myVector(0)(0) + " - "+ myVector(0)(2)); //DOpus.Output(" Retrieved Vector(0): " +myVector(0)); DOpus.Output(" Retrieved Vector(1): " +myVector(1)); DOpus.Output(" Retrieved Vector(2): " +myVector(2)); //DOpus.Vars.Delete("MyVector"); } else { DOpus.Output("Creating MyVector (now run again)"); var v = DOpus.Create.Vector(); var subV = DOpus.Create.Vector(); subV.push_back( "subVal1"); subV.push_back( "subVal2"); subV.push_back( "subVal3"); v.push_back( subV ); v.push_back( "mystring2"); v.push_back( "mystring3"); DOpus.Vars("MyVector") = v; DOpus.Vars("MyVector").persist = true; DOpus.Output(" Vector persistency: " +DOpus.Vars("MyVector").persist); }

2

Thanks, this will be fixed in the next update. The issue is the recursive vector (i.e. persistent vectors are fine, but it currently can't handle vectors as members of other vectors).

3

Wonderful! o)

Regarding the sub-vector, I tried the same with a vector containing just strings and it also wouldn't save correctly, so I'm a bit suprised.
The reason I left the subvector in the demo code, was just to enrich the test-case. o)

Thank you!

4

Hello! o)

I still have problems with vector persistency (I make use of a 3 dimensional vector here).

Please look at the output before I restart DO:

  • the vector (named "dc" here) has a length of 69
  • each contained sub-vector has a length of 4 (the fourth element is another vector of strings, the 3rd dimension)

After I restart DO:

  • the vectors length of 69 is truncated to 23 elements (exactly 1/3rd?)
  • each contained sub-vector has 3 elements now instead of 4 (the forth element, the sub-vector of strings is gone)

Another restart does not change the data in the reloaded vector anymore, so it stays at 23 elements, not loosing more information.

Thanks! o)

5

Test script?

6

Correction:
It's not the 3rd-dimension vector that's missing, its index just decreased by one 4->3, so the "recursiveness" seems to be fully maintained (looking into uservars.oxc confirms that).
So despite the root vector being truncated, each subvector misses the 3rd element, which was a regular string (output of Date())

7

Could you post a test script please :slight_smile:

8

Of course Jon. o) I will post again once I managed to make the test script show the same issues I see with my actual use case.

Right now the test script works as expected, but it does things differently compared to my usage of the vector in my column code.
The main difference probably is, that the test script fills the vector in one go, while the columns add data in chunks various seconds apart.
Additionally I might need to make use of some more weird data types than just strings to make DO skip a vector field when saving out to disk.

Till then! o)

ps: This is what I got so far, some code to create a 3 dimensional array and dump it (after reload), but that won't trigger the effects I see.

varName = "Script.VectorTest4";
//DOpus.Vars.Delete(varName);

if (!DOpus.Vars.Exists(varName)){
	DOpus.Output("Creating root vector..");
	var doCache = DOpus.Create.Vector();
	DOpus.Vars(varName) = doCache;
	doCache = DOpus.Vars(varName);
	doCache.persist = true;
	doCache = DOpus.Vars.Get(varName);

	for(var indexOne=0;indexOne<100;indexOne++){
		DOpus.Output("    "+indexOne+" vector");
		doCache.push_back(DOpus.Create.Vector());
		//DOpus.Output("        filling vector..");
		doCache(indexOne)(0)="String"; DOpus.Output("        0 "+doCache(indexOne)(0));
		doCache(indexOne)(1)="String"; DOpus.Output("        1 "+doCache(indexOne)(1));
		doCache(indexOne)(2)=Date(); DOpus.Output("        2 "+doCache(indexOne)(2)); 
		doCache(indexOne)(3)=DOpus.Create.Vector(); DOpus.Output("        3 vector"); 
		//DOpus.Output("           filling vector..");
		for(var indexThree=0;indexThree<10;indexThree++){
			DOpus.Output("           "+indexThree+" String");
			doCache(indexOne)(3)(indexThree)="String"+indexThree;
		}
	}
} else {
	var doCache = DOpus.Vars.Get(varName);
	DumpVector(doCache,"");
}

function DumpVector( v, pad){
	for(var vIndex=0;vIndex<v.length;vIndex++){
		try{
			if (v(vIndex).count == undefined) throw "no vector";
			DOpus.Output(pad+vIndex+" vector["+v(vIndex).count+"]");
			DumpVector( v(vIndex), "    "+pad);
		}
		catch(e){
			DOpus.Output(pad+vIndex+" "+v(vIndex));
		}
	}
}
9

Ok, no progress with the test script, I just cannot reproduce what I see if I append data to the vector by a column function, even with some delays thrown in.

IIRC a column functions context, a script-"session" or it's thread, keeps active for a while and then dies. It seems, the first time the vector is created by a column function, all data put into the vector within that lifespan of the corresponding thread, is saved to disk. If the column is called some time later, any further data appended to the vector will not be written out to disk, but will be there while DO is not shutdown.

An example: If a lister has 40 files, 20 of them visible, data for 20 files will be saved to disk. If you scroll through the lister to trigger fetching of further column values for the other items, this data will not make it into the uservars.oxc, if the thread for that column is not the same as for the first 20 files. Does that make sense to you? o)

I'd like to suggest, that you look at what happens with the addin in progress, where I like to implement that persistent column cache.
Open the "script addins" folder, place the script there and add at least one of the columns provided by the add-in to get the column and cache magic going. You probably need to put some more add-ins in there as well, or at least put it so, that not all column values for all items will be fetched in one go (you need to have that column thread shutdown before fetching column data for a second chunk of items).

All data is stored into the global variable "Script.ColumnCache.ScriptDetails". You can use the "DumpVector()" function to compare quite easily what's in-memory and what's in uservars.oxc.
Command.Generic_ScriptWizard_test.js.txt (227 KB)

var doCache = DOpus.Vars.Get("Script.ColumnCache.ScriptDetails"); DumpVector(doCache,"");

To start from scratch, delete the "Script.ColumnCache.ScriptDetails" vector and retrigger the columns (F5 in "script addins" e.g.).

Thank you! o)
If there's more information required, please do not hesitate to ask, but actually this is as much as I can think of right now. o)

10

It must be possible to narrow down the script to something (a lot) less than the 3,000-line example you've given. It can't all be essential to reproducing the problem, and simplifying it to the smallest possible script that still reproduces the problem may reveal what the problem is along the way.

11

But I think it's more about the thread ending/restarting theory I described, than the actual code.
Without knowing exactly what's going on for scripts that get a context and die, I tried this:

[code]varName = "Script.VectorTest6";
//DOpus.Vars.Delete(varName);

if (!DOpus.Vars.Exists(varName)){
DOpus.Output("Creating root vector..");
var doCache = DOpus.Create.Vector();
DOpus.Vars(varName) = doCache;
doCache = DOpus.Vars(varName);
doCache.persist = true;
doCache = DOpus.Vars.Get(varName);
}

doCache = DOpus.Vars.Get(varName);
DOpus.Output("Start length: "+doCache.length);
var start = doCache.length;
DOpus.Output("/////////////////////////////////////////////");
DOpus.Output("ADDING 10 ELEMENTS");
DOpus.Output("/////////////////////////////////////////////");
for(var indexOne=start;indexOne<(start+10);indexOne++){
DOpus.Output(" "+indexOne+" vector");
doCache.push_back(DOpus.Create.Vector());
//DOpus.Output(" filling vector..");
doCache(indexOne)(0)="String"; DOpus.Output(" 0 "+doCache(indexOne)(0));
doCache(indexOne)(1)="String"; DOpus.Output(" 1 "+doCache(indexOne)(1));
doCache(indexOne)(2)=Date(); DOpus.Output(" 2 "+doCache(indexOne)(2));
doCache(indexOne)(3)=DOpus.Create.Vector(); DOpus.Output(" 3 vector");
//DOpus.Output(" filling vector..");
for(var indexThree=0;indexThree<10;indexThree++){
DOpus.Output(" "+indexThree+" String");
doCache(indexOne)(3)(indexThree)="String"+indexThree;
}
}

DOpus.Output("/////////////////////////////////////////////");
DOpus.Output("DUMPING");
DOpus.Output("/////////////////////////////////////////////");
var doCache = DOpus.Vars.Get(varName);
DumpVector(doCache,"");

function DumpVector( v, pad){
for(var vIndex=0;vIndex<v.length;vIndex++){
try{
if (v(vIndex).count == undefined) throw "no vector";
DOpus.Output(pad+vIndex+" vector["+v(vIndex).count+"]");
DumpVector( v(vIndex), " "+pad);
}
catch(e){
DOpus.Output(pad+vIndex+" "+v(vIndex));
}
}
}[/code]
It seems as if this snippet behaves the same as when used in columns. Run this snippet once, watch uservars.oxc (10 elements get saved).
Run the snippet again (a new context/thread?), it will append 10 items for each run to the existing in-memory vector, which won't get saved (within some minutes at least).
So the in-memory vector will grow, but the one saved to disk after the first run, will not update -> unless I create another persistent vector (change the varname at the top and run snippet again).

Additionally: If DO is exited, a vector which was already saved to disk and to which elements have been added, won't be updated on disk (I just tried to force dumping of the vector data to disk by restarting DO).

12

So, does this last snippet meet your requirements in terms of shortness and isolation?
Now that it does not seem to be related to columns, I could probably reduce it some more, but I guess you already got the idea.

Just let me know if there's something else I can add to the mix to get that persistency going strong! o)

13

Ok, thanks, I have been able to reproduce it. It's happening because even though the contents of the vector is being modified, the variable itself isn't (e.g. you aren't going through DOpus.Vars when you modify it, you just modify the vector directly) and so Opus isn't noticing that the value of the variable has changed. We'll fix it in the next update.

14

Top notch and.. phew! o)

I think I understood your explanation, but if DOpus.Vars.Get("MyVectorVar").push_back("lala") is the unsupported way,
what's the currently supported way to modify the vector through the variable?

Looking forward to the fix, yippie! o)

15

New beta is out which should fix this, so you shouldn't need to look for a workaround.