Steps to plan a whole strategy

Help & Support
1

Hi,
I am starting an evaluation for a NFP I am involved with to see if Directory Opus can replace NTFS and Windows File Explorer.
What I would really like now are some early pointers from you all on where to start and in what order - before I open the 'advanced features' box and get myself all tangled up.

(Trying) to cut the story short: we have a collection of about 50,000 files of which 90% are in image format as well as lower proportions of PDF, DOC etc.
Since 2019 these have been undergoing a 'Managed Asset' approach and are in the process of being 'tagged'. About 20% have been completed.
For this a handful of Windows attributes have been selected to hold our basic 'Who', 'What', 'Where', 'When'
then a tool, FileMeta by Dijji (GitHub - Dijji/FileMeta: Enable Explorer in Vista, Windows 7 and later to see, edit and search on tags and other metadata for any file type) has been installed that allows the File Property Handlers to be modified to include a standard pattern across all the file types we are interested in.
Then Windows File Explorer standard functionality is used to modify the attributes - in the details pane.
Now Windows File Explorer will search for entries in all these attributes as well as the file name (and file content if so configured).

All was going well for Windows 7 and 10 and even 11 for a while. A recent regular update of Win 11 has removed the ability to edit attributes (without climbing down into lower 'properties' windows).
Faced with that, and the achilles heel of the whole approach in that it relied on a NTFS file system; I am looking for alternatives.
(Personally, I think Microsoft has abandoned all Metadata and features available for over 20 years, since XP, are just being washed out without even any advance notice. They never say 'you lose all your attributes when your files go to one drive')

So after a quick look at DOpus I can see
* Has Multiple Panes
* Has Multiple Tabs
* Has no trouble with 40,000 file folder
* Has Preview Pane
* Has Properties view - dont know about configurable
* seems to cache its own thumbnails
* Maybe has Plugins
* Does have 'scripting' - not explored yet
That ticks most of my evaluation boxes :slight_smile:
I am not so sure about 'search' ability but as I said that was just a quick look.

So I can see the metadata in the Metadata pane but how can I configure it to show the attributes I want?
In this case they are all Extended Attributes (I think).

Thank you for your time
JC
FileAssociation_manager

Museum_Who_What_Where_When_w10
Museum_Who_What_Where_When_w10965×586 200 KB

1 Like
2

Hello,

you cant show this Extended Attributes in the Metadata pane or configure the Metadata pane (perhaps in the future)....

...and even if it were possible, the Extended Attributes (not all of them) are still dependent on NTFS resp. NTFS Alternate Data Streams. Some metadata such as author and titel are part of the file type itself and not dependent on NTFS ADS.

However, the metadata can be displayed in columns. Editing but only to a limited extent.

2024-03-08_175958
2024-03-08_175958796×561 31.6 KB

3

If I may add a couple points in addition to Hardkorn, just to give you some more ideas.

Personally I think ADS are great, I use them for multiple purposes, mostly to store data from my own DOpus scripts. Then again, my files rarely leave NTFS. As a side note: one "easy" way to keep the ADS on a different FS or via email is compressing them into RARs, which has an option to store and extract ADS.

Second, I know the pain all too well and I personally think format-specific data belong into format-specific metadata, e.g. if it's a music file it should use id3v2 (.mp3) or Vorbis (.flac), or exif (.jpg) etc. instead of a chaotic disarray of non-standard metadata. DOpus tries its best (and I have yet to see another file manager which does better) to put the metadata into file-specific fields where possible but some formats just don't support the same data a related format does, and a comment field for an .mp3 is stored as id3, or for a .docx in the file itself, but for a .txt or .md in ADS just as an example. IMHO it'd be much more easier to convert all your files to a handful of well-supported format like PNG (or TIF, JPX, JXL, AIF, HEIC, whatever suits your needs) than to find the common denomianator for all of BMP, PNG, JPG, TIF, GIF, WMF, DNG, RAW, etc. all with varying degrees of metadata and support across many programs. But if you still go the latter route, I think ADS are indeed the better way, you can at least completely standardized metadata.

For my own purposes, the decision was easy: FLAC for original audio with embedded tags (and quick transcode to temporary MP3/MKA/M4A if necessary), MKV for movies (AVI, FLV, WEBM, MP4... never again!), PNG & JPG for images, and rarely PSDs for complex projects. Luckily many good programs like Foobar2000 support audio in a RAR/ZIP file or DOpus multiple images in a CBR/CBZ file (plain old RAR/ZIP files with renamed extension). All my image albums are stored now as CBR files with 5-15% recovery data. As a side-bonus, archives can have comments however you please and greatly reduce disk clutter. CBRs even support a cover image for DOpus thumbnails mode.

Unfortunately there is no universal metadata format or storage, but there are great many formats and tagging mechanisms like id3, vorbis or exif. Hope these give you some ideas.

2 Likes
4

Hardkorn
Thanks for that snippet about using columns - I will explore that too

1 Like
5

cyilmaz
I see you have followed the same path of metadata thinking and representation as I have.
In the post above I only told half the plan.
Part A. (Semi Urgent) Find a tool to replace Windows Explorer - or replace the functionality recently removed from that.
Part B. (Longer Term) Find or develop a tool that can store user defined metadata INSIDE the file itself. Thus removing the NTFS dependancy

To get to part B. I had concluded differently from you that using NTFS ADS was problematic today and probably had no future.
Problematic today because of the daily pushes from Microsoft to 'use one drive', 'backup to cloud'

I too have rationalised the set of file types 'allowed' in the collection.
I am developing a design/architecture that melds the ideas of FileMeta MetaTags (MetaTag)
with https://schema.org/ and the abilities of exiftool to read and write metadata into a file.

If I succeed here maybe that could become a Plugin for Dopus.

Regards
John C

6

That's going to depend on the file type, and will be impossible for some types.

E.g. There is no way to add metadata to a plain-text file (unless it's OK to modify the text itself, and also guaranteed nothing else will modify the text in a way which breaks the metadata).

In some cases the only options will be NTFS ADS metadata, sidecar files (where the metadata is stored in a file next to the real file, or similar), or a database separate from the file (the database could also contain the file, of course).

Where tags can be stored inside a file -- i.e. where the file format provides that functionality -- Opus already allows that for many common types, e.g. PDF.

FWIW, Opus has a VFS plugin API which would let you define a custom virtual filesystem which could be useful if you want to go the database route, and store the file data itself in the database. (A bit like an archive format, but with your custom metadata.)

7

Leo,
Great - I am aware how big this mountain is. I have been chasing Metadata for 30 or 40 years on and off. I just think I might have to do something about it myself.

Where tags can be stored inside a file -- i.e. where the file format provides that functionality -- Opus already allows that for many common types, e.g. PDF.

That would be excellent - is there a list of these ? If it could cover PDF, JPG or similar, and MS Office stuff I would have over 99.5% of the collection covered.

Can you provide a link to the resources for the SDK and VFS plugins? All google can find is from a post in 2007 but the link there (Directory Opus - GPSoftware)
just takes you to the opening gpsoft.com.au page.

Thanks in anticipation
JC

8
9

Excellent Thank you!

10

Well, for most common files I use the established tagging systems indeed (MKV/Vorbis being my favorite duo, amazing tagging possibilities), but also wholeheartedly agree with Leo there: Some files like .TXTs (or my favorite in recent years .MD) simply do not allow tagging. Other formats require expensive software. Even though I'd welcome a universal tagging, and even if you can come up with a good universal plan which is format-agnostic (different formats' file layouts can be fundamentally different), it will depend on critical acceptance and 3rd party apps support obviously, but you seem to be aware of the challenge. So yeah, I explored all the possibilities Leo described myself: side-files, DAM/external indexes, naming conventions. The reason why I liked ADS after all those experiments and only for specific use-cases is much more pragmatic tho: They eliminate the need for external indexes, FS-watchdogs & expensive software and also they follow the file, and regardless of filename, date & location changes. They're as close as it gets to embedded tags. So I'd rather stay in NTFS-land and package them in RARs or encryptied containers, etc. than lose the info in favor of distributed, heterogenous FS like cloud or Linux/NAS servers.

But admittedly my use-cases are quite niche and more importantly entirely format-independent; like storing hash checksums. They're super-resistant against superficial changes and since DOpus copies ADS between NTFS, it's never been a problem. Before Dopus v13 introduced Mediainfo support, I even built my own Mediainfo script which supported more formats than previous version and stored in ADS. Now that DOpus/Mediainfo supports MKVs and countless other formats much better, my script is almost obsolete, but that didn't keep me moving away from junk formats like AVI or MP4. Vorbis is far superior to all others I evaluated. Image formats are too much scattered for my taste, EXIF, XMP, IPTC, vendor-specific metadata, but at least there are a few established ones, too. Text-based formats are a mess on their own, I am still in search, there's not a best-of-the-bunch, at best an acceptable one for one usecase, but ^not another. There are a couple academic standards, but they're nearly unknown/unsupported in consumer world.

tl;dr: I use few best-of-breed formats for audio, video, image, etc. with good tagging mechanisms where available, and ADS when it's filetype-independent info, like checksums.