Creating a simple data visualizer

Hello :slight_smile:
I'm willing to create a plugin or script (I'm not sure which is the appropriate one) to view the data in a target file.

The data I'm willing to see is the patch data contained inside one of the patch files for my application updater.

The file format is as follows:

Header [1 entry]:
7 bytes		"Magic String"
1 byte		"Version Number"
1 byte		LZMA2 properties

File Entry [1..N entries]:
1 byte		Operation (Delete, Add, Replace, Add Directory)
8 bytes		Target file name length
? bytes		Target file name 
8 bytes		Compressed size [ Patch file version 2 ONLY! ]
? bytes		Compressed data

How should I proceed to attempt to create a file visualizer for that ?

Ideally, I'd output a list with something close to this:

ZPatch File Version X
[Add    ] /data/test.txt
[Replace] /data/test2.txt
[Delete ] /data/test3.txt
[Add Dir] /data/newdir

Thank you very much in advance! :smiley:

A viewer plugin probably makes the most sense, at least if you're comfortable writing C or C++. (Other languages are possible but will be a lot more work due to lack of examples.)

The plugin SDK is here:

http://www.gpsoft.com.au/DScripts/download.asp

If you want a plugin that displays simple text data about a file, see the section on Returning Text Data (or search for "DVPFIF_ReturnsText"). That will be the easiest way as you just have to implement a few functions and return the text which Opus will display for you using the built-in text viewer. That way you don't have to write your own custom viewer UI (although you can if you want more control over how the text is presented, of course).

This page has more plugin example sourcecode, although none use the text mode. (The NFO plugin may be of interest if you want to implement your own text-based UI, however. It uses a standard edit control.)

https://pretentiousname.com/pnop_source/index.html

Thank you Leo!

I'll give it a try :smiley:

Hey @Leo

I ran into some issues while trying to create a plugin. Can you, please, give me some help? :slight_smile:

This is what I have right now:
ZPatchViewer_a1.zip (18.6 KB)

I didn't manage to make it run on DOpus. Apparently, I should create a Viewers folder in /dopusdata, but it didn't work, and I have no idea why.

If it worked, I expected it to crash.

On the function DVP_LoadText(), I was expected to create a IStream* object (and I took a while to notice this wasn't std::istream). Apparently, I'm not used at all to COM stuff.
How can I create this IStream object?

Here is what I have (and I expected it to crash, since I didn't create an object):

	DWORD bytesWritten;
	const wchar_t* test = TEXT("Random test\nTest test.");
	lpLoadTextData->lpOutStream->Write(test, sizeof(test), &bytesWritten);

Last but not least, I noticed some errors in the PDF manual:

page 17
DVPFIFITypeHint_PlainText, DVPFIFITypeHint_RichText and DVPFIFITypeHint_HTML have the wrong prefix (the actual prefix is DVPFITypeHint_*)

page 24
DVPCVF_Slow and DVPCVF_NoRandomSeek have the wrong prefix (the actual prefix is DVPSF_*)

Thank you very much and I hope it isn't much of all the hassle for you :blush:

Viewer plugins go in the Viewers folder which already exists under /home (the main program files folder).

They don't go in /dopusdata (the user profile folder).

Make sure your DLL is compiled as 64-bit if you're using 64-bit Opus, and check that your plugin appears in the list of viewer plugins in Preferences before anything else, since it has to appear there for anything else to work.

Thank you very much!

Any advice about creating that IStream* object? I couldn't find and example :frowning:

Okay, I finally was able to try and... No success.

I copied the dll, fully closed opus (near the windows clock, right clicked the icon and selected "Exit Directory Opus", opened it again, the plugin doesn't show.

This is the file: ZPatchViewer.zip (12.8 KB) - Compiled file from the same supplied project.

Any idea of what's wrong? :slight_smile:

You aren't exporting the functions properly. They're being exported with a C++ naming convention, which adds lots of extra noise to the ends of function names (which can vary from compiler to compiler):

They should look like this:

Export the functions like this:

extern "C"
{
	__declspec(dllexport) BOOL DVP_InitEx(LPDVPINITEXDATA pInitExData);
	__declspec(dllexport) void DVP_Uninit(void);
	__declspec(dllexport) BOOL DVP_USBSafe(LPOPUSUSBSAFEDATA pUSBSafeData);
	// etc.
};

You should probably implement and export DVP_InitEx and DVP_Uninit as well; that seems to be missing from the exports. Remember that they are refcounted, so you need to count the number of times InitEx is called vs Uninit and only free your plugin's extra resources when the count goes back to zero.

Thank you a lot!

I've been able to progress a bit with the plugin :smiley:

Sadly, I still have questions - and sorry to keep asking them, apparently it has been a long while I've messed with platform code :frowning:

I don't get the point to actually use DVP_InitEx and DVP_Uninit - I'm not initializing anything, I'll just read the target file and print the useful part of the contents for me. Right now, I'l just returning true -
Maybe I'll improve it in the future..? :smile:

Ok, now to the stuff that's been bothering me all the night:

After attaching the debugger to DOpus.exe, I keep getting this error, when I press F8 to preview the file contents:
Exception thrown at 0x00007FFA0A829E08 in dopus.exe: Microsoft C++ exception: Exiv2::BasicError<wchar_t> at memory location 0x000000B7766FEB20.

I think it's related to IStream (Microsoft's version, not the standardized one).
IStream is painful to work with. Or maybe the documentation is terrible for those who aren't used to it.

This is what I managed to do so far:

__declspec(dllexport) BOOL DVP_LoadText(LPDVPLOADTEXTDATA lpLoadTextData)
{
	// Apparently, while the documentation says this is the only flag available, it is never set :(
// 	if (lpLoadTextData->dwFlags != DVPCVF_FromStream)
// 		return false;

	if (lpLoadTextData->dwStreamFlags & DVPSF_NoRandomSeek)
		return false;

	lpLoadTextData->iOutTextType = DVPText_Plain;		// TODO: Format text

	ULONG bytesWritten;
	std::wstring test = TEXT("Random test\nTest test.");

	// Create and verify our IStream object
	LPSTREAM txtStream;
	if (CreateStreamOnHGlobal(NULL, true, &lpLoadTextData->lpOutStream) != S_OK)
		return false;

	// Write our test data
	lpLoadTextData->lpOutStream->Write(test.c_str(), test.size() * sizeof(wchar_t), &bytesWritten);

//	MessageBox(NULL, TEXT("DVP_LoadText"), TEXT("Function Ran"), MB_OK | MB_ICONINFORMATION);

	// TODO: Actually read the file here!

	return true;
}

I tried writing a '\0' after the contents of the wstring, but still didn't work :confused:

Any idea on how to fix that? :frowning:

Thank you very much for all your patience and support! :slight_smile:

Unless you're using exiv2 in your code, that exception is probably coming from Opus itself, and it's an exception that Opus will handle by itself, but the debugger is probably configured to break as soon as an exception is thrown (whether it is an exception that will be handled or not).

You can configure the debugger not to break on exceptions, or to ignore specific types of exceptions, if that is the problem.

I'm not using exiv2 lib.

I don't know what else I'm missing

If the problem isn't on the way I'm creating and handling IStream*, I have no idea of what's wrong.

Just in case, is there an example plugin that uses DVP_LoadText() so I can take a look at it?

(At this point, it would probably be easier to just remove DVP_LoadText() from the code and take a peek at the NFO plugin code on how to create a custom window, but I really would like to know what's wrong.)

Here's the current project file: ZPatchViewer_a2.zip (17.8 KB)

The only mention of DVP_LoadText() I found while searching is this file that uses it. I didn't try compiling it, but the only difference I saw from my code is the inclusion of the UTF-16 BOM. I tried adding it, nothing changed.

	const wchar_t wcUTF16BOM = L'\xFEFF';
	lpLoadTextData->lpOutStream->Write(&wcUTF16BOM, sizeof(wcUTF16BOM), &bytesWritten);

Am I missing something? Is the output supposed to be seen somewhere else than the preview window?

Thank you very much for your patience and support! :slight_smile:

The BOM will definitely be needed for a UTF-16 stream. Easiest way is to change one line in your code:

std::wstring test = L"\uFEFFRandom test.\nTest test.\nThis is UTF-16.";

That's the good news. The bad news is that it turns out DVP_LoadText is semi-broken at the moment.

We've found where it's broken and have a fix coming.

For the time being, there's a kludge you can add to the plugin which will make it work with the current Opus version. It's probably not something you want in a finished release version but it lets you get on with things while you wait for us to fix our side.

(Basically, the text viewer needs to verify it can handle the data, but instead of looking at the stream your plugin provides, it is looking at the original file's data that was already cached in reaction to another plugin. Since patch files are not plain text, the text viewer is rejecting the file. The kludge is to overwrite the cached buffer with text data -- "aaaaa..." will do -- to trick the text viewer into accepting it. Strictly, this isn't allowed as that buffer is read only to plugins and could conceivably be needed for something else in the future, but we can get away with it for now, as a temporary workaround.)

Here's a version that works with Opus 12.6, with comments in the code. Only the main .cpp and .h were changed so if you diff them you can see the changes:

ZPatchViewer_a2_leo.zip (18.4 KB)

Hey Leo!

Thank you very much for taking your time and going through my code! I’m glad this actually led to a fix! :smile:

Now I’m split between waiting to 12.7 beta/release or keep going as is (Spoiler alert: I’ll probably keep going :smiley: )

Since we’re on the subject, can you explain why DVP_IdentifyFile() was being called before DVP_IdentifyFileBytes() ? According to the documentation, DVP_IdentifyFile() function is always called unless:

  1. DVPFIF_CanHandleBytes flag is set or;
  2. DVPFIF_CanHandleStreams flag is set and the file is not disk-based (i.e. it is inside a FTP server or a ZIP file)

And in this case, DVPFIF_CanHandleBytes was set, but upon setting Alerts (MessageBox – wish there was a possible console output), I never see the ones I added to DVP_IdentifyFileBytes being called, if DVP_IdentifyFile() is set.

Maybe there’s a mistake either in the documentation or in the code? :slight_smile:

Thank you very much, again, @Leo ! :grinning:


For anyone visiting this thread in the future, to use DVP_LoadText() prior Opus v12.7, you need to add a workaround, implementing only DVP_IdentifyFileBytes() among the possible Identify functions, and add in its body:

for(UINT i = 0; i < uiDataSize; ++i)
{
	lpData[i] = 'a';
}

Thought only use this if it’s absolutely necessary to support an older version of Directory Opus (v11, etc.).

Thanks for all the help @Leo!

I managed to get the bare bones of the plugin working :smiley:

Is there a way to toggle word wrap on it?
If there's not, Can I make this a feature request? :slight_smile:

Without it, it looks a bit odd:

Thank you very much for all your help!

PS: I still don't get what's the priority order to call IdentifyBytes when the regular Identify function is implemented :frowning:

If you right-click the text viewer, there is an option to toggle Word Wrap.

That will affect all uses of the text viewer, of course.

That do the trick! Thank you very much! :smiley:

Hi @Leo, my apologies for doing some necromancy on this.
I decided to give this a go again, but I'm having some trouble updating the plugin. It just doesn't run.

ZPatchViewer-2021.7z (15.8 KB)
test.zpatch.zip (23.7 KB)

I attached the code I'm running and a sample zpatch file for completeness, but this is the main function for the test:

ZPATCHVIEWER_API BOOL DVP_LoadText(LPDVPLOADTEXTDATA lpLoadTextData)
{
	if (lpLoadTextData->dwStreamFlags & DVPSF_NoRandomSeek)
		return false;

	lpLoadTextData->iOutTextType = DVPText_Plain;		// TODO: Format text

	ULONG bytesWritten;
	std::wstring test = TEXT("Random test\nTest test.");

	// Create and verify our IStream object
	if (CreateStreamOnHGlobal(nullptr, true, &lpLoadTextData->lpOutStream) != S_OK)
		return false;

	const ULONG targetWriteSize = (ULONG)(sizeof(test[0]) * test.size());
	HRESULT hr = lpLoadTextData->lpOutStream->Write(test.c_str(), targetWriteSize, &bytesWritten);
	if (hr != S_OK || targetWriteSize != bytesWritten)
	{
		lpLoadTextData->lpOutStream = nullptr;
		return false;
	}

	return true;
}

It's supposed to be as simple as it gets, but I'm not being able to get this to display properly in DOpus :frowning:

And to make things harder, sometimes it randomly fails with with HRESULT getting values I can't find on tables online - but I can't trigger that consistently.

Can you give me some help on how to make this work? I'm not a Windows programmer, so it's possible I'm struggling with some obvious COM issues, but I'm stuck and I don't know where to look next.

Thank you, again! :slight_smile:

Which part doesn't run? It sounds like DVP_LoadText is being called when you say:

Which function call is failing? CreateStreamOnHGlobal? lpOutStream->Write? Something else?

What is the HRESULT value that is returned when it fails?

Are you working from my sample code (above in the thread) or is it from scratch? I haven't had a chance today to verify the sample still works but as far as I'm aware it should.

Hi Leo,

My apologies for the long time for a reply, I've been quite busy at work.

I downloaded, removed the block with the comment that was a kludge for old version of dopus and tried to use the resulting plugin. Unfortunately, it's not working. :frowning:

I got this to work years ago, so I'm wondering if anything changed or if a new bug was introduced.

Could you please look into this for me, please? :slight_smile:

Thanks in advance!

I am following this conversation, because I'm curious. But, when you say it's not working, you're not specifying how it's not working. Is nothing happening? Is something happening and you are getting an error. Is something happening, you aren't getting an error, but it looks wrong? Etc.