Alternate data streams (ADS) support for MetaData

I was experiencing some issues with setting Metadata on zip files that are stored in VeraCrypt container.

I store my photo sets in zips on an encrypted external drive. I recently replaced one drive with another. When doing this I created the new drive as exFAT instead of NTFS.

When setting a rating on a zip, if I used the SetAttr META=rating:1 command it silently failed. If I use the Metadata panel I received this error An error occurred setting Metadata 'xmas-2016.12.25.zip': The file name, directory name, or volume label syntax is incorrect. (123). Took me a while to figure out the issues was due to not creating a NTFS file system, which is needed to support ADS.

Regarding this I have some suggestions to improve usability:

  1. Can the SetAtt META command be improved to not silently fail. Perhaps using the yellow panel across the tops of the Lister.
  2. Could the error message from the Metadata Panel be improved to indicate the issue is the file system not being NTFS? Or could the Metadata panel make read-only or hidden any properties that are not valid due ADS not being supported
  3. When copying files could DOpus check if any ADS would be lost in the copy and warn the user?

I was disappointed to find I had lost all my ratings. I might update my process to also save the rating to a text file within the zip, then I could recover the data if needed.

Regards

1 Like
  1. I'd like to have that too (a proper error message I mean). I have a button/menu to apply ratings to selected files. If files are locked or unaccessible otherwise, you have no clue that setting the rating failed. To really make sure the rating went into the file, I always double check by toggling the meta panel on/off, which does not really speed up the process since scrolling down the meta data panel is required as well. o)
  1. If a file is currently in use by another process, don't fail silently (as happens now), ask if the user wants to retry.

It seems, for inaccessible files, it's not only the SetAttr META command which fails silently.
DO also does not give any hint for errors while setting the rating via the native rating column.
You click and rate like a pro, then you hit refresh and.. all the stars are gone -> sad! o)

I also noticed that for some files (video files), DO seems to load the whole file and seems to create a temporary version of it. I tried with some season movie files on the network (ntfs / windows server) and network IO most of the time seems to skyrocket every time. Well, not everytime actually. I did not yet figure out when and why this is.

I asked about this before. The answer was that sometimes rewriting the file is necessary. I guess it is a limitation of some component being used under the hood.

I observed this: for a video file which needed to be rewritten for metadata modification, if I created a version of it using ffmpeg, the new version would not need to be rewritten.

I'm not sure of the exact detail but if adding or changing the metadata requires data be inserted (not just overwritten) near the start of the file, then there's no way to move all the rest of the data along the file (even if the file format allowed it); you can only read all the data after that point in and write it all out again if it needs to be moved to make room before it.

One reason why some metadata formats are at the end of files rather than the beginning.

I made a quick test with an hex viewer and two mp4 files.

One mp4 is the original and Opus needs to rewrite it do change the metadata, even to only remove all metadata.

The other is a remux of the original, created with:

var remuxCmdLine = "ffmpeg -xerror -avoid_negative_ts make_zero -fflags +genpts -hide_banner -y -i " + inputPath + " -map_metadata g -threads 6 -map 0:v -map 0:a? -map 0:s? -c:v copy -c:a copy -c:s copy " + outputPath;

The remux does not need to be rewritten.

Both appear to have the metadata written at the end of the file.

Thanks you @andersonnnunes, interesting find which fully applies to what I experienced.

As we/you found out, for some video formats a full rewrite of the file is necessary. You also know that video files these days are huge. All my files are located remote and if I set a rating the 1gb ethernet is maxed out for minutes for nearly every single rating I apply. This takes just too long, especially if you start a "rating session". The files being locked during the processing is another heavy drawback. You cannot really work fluently with these heavy delays in between.

Now - Idea time! o):
Since ratings in photos and smaller file types should better be done correctly and in a way that generic software is able to make use of them, this does not really apply for video files (for me at least o). So I really wouldn't care if video ratings won't be picked up by.. don't know.. any video catalog software out there.

Could there be an option to specifiy all the file extensions for which DO only uses the plain, simple and excellent ADS storage for the ratings? I tested and changed extensions of huge video files into something unknown and applying the rating works instantly with these. What do you think?

DOpus integration with something like this could be a start.