Hard-links handling at all levels

[Magnetik]

Hi everyone,

This is my first topic on this forum.

My native language is french so pardon my english mistakes if you find some

Also my computer/informatics knowledge is not very deep therefore some of my questions and comments may seem pointless to some of you, if that happens I apologize in advance.

First, I would like to thank the development team for the great program that is Directory Opus and all the possibilities it offers, it is already huge and continues to improve. I already have been using some evaluation versions of Directory Opus in the past years, but I never used it regularly nor at full potential. Only recently I decided to buy a Pro licence.

There are already many useful and unique features that I began to use for files managing. But there is one aspect of common file systems (NTFS in particular) that main file managers (like Directory Opus) seem to not handle with full potential, it is hard-linking, or using multiple hard-links per file. I know it is already possible to create multiple hard-links per file in Directory Opus with commands like “makelink=hardlink” (I didn’t look into how to use commands yet). But to my knowledge there is no real integration in the main user interface to handle all the possibilities offered by multi hard-linking.

For me one of the main reasons to use multiple hard-links per file is for classification. Multi-hierarchical organizations to handle any kind of files groups are very powerful, logic and permissive systems for end users. It is not limited to mono-hierarchical structures that are very restrictive for classifications and force users to make illogic, constrained choices and make files finding less efficient. There are systems that use tags or labels to classify files with multi-hierarchies, some softwares propose but often this solution is not based at the file system level, is software/vendor locked, modify file names, don’t handle nested tags/labels, don’t hold in backups, transfers, etc.

I know there are few limitations inherent to hard-links, like the inability to span them across different volumes, but it is not really a problem in most contexts. I already use hard-links for some years to classify many of my personal files, for that I use especially Link Shell Extension which provides many useful tools to manage all types of NTFS links. But I am still limited in many ways, notably to navigate efficiently through those multi-hierarchy classifications and analyze them, or to find particular files. For example there is no tool to find files relatively to their multiple locations in the classification, which kind of defeats one of the big purposes of using hard-links. There is no program with handy user interface (like Directory Opus) to find files by recoupment with ensemble-logics like unions or intersections across different sets of folders. There are also some problems that arise by using multiple hard-links for some non-aware programs, like calculating sizes of folders which contain multiple hard-links per file.

I don’t know why hard-linking and multi-hierarchies are not already widely used in files classifications, at user level, handled by common files managers, because it is a feature very powerful and already supported by most file systems like NTFS. Maybe some already use this, or maybe there are other ways to use multi-hierarchies for files classification, that I am not aware of. So maybe what I am asking for here is features-request and I don’t post in the correct place, or maybe It is already possible to do it In Directory Opus in some ways, in this case I would like to know how.

So, if not already implemented in Directory Opus, I think it would be very beneficial to implement full hard-linking management, at all levels (creation, visualization, moving/copying, recouping/finding, size calculations, etc.). I can give here already some examples of implementations in Directory Opus that I think about:

1. To find a file referenced in multiple folders with multiple hard-links (multi-hierarchy classification). We could use the find mode and have options to include multiple locations (folders) linked with logic operators like “and”/“or”. We can do that already with the advanced find tool but only for things like names, dates and other file attributes, not for locations. For example if you multi-class your photos by dates, by locations, by events, etc., with hard-links, you can find easily all the photos that are of one particular date AND one particular place AND one particular event. You could also exclude some folders, to find all photos that are from one particular event AND from all dates BUT one. And many other possibilities and for all types of files. It is what you can do with tagging systems but with less efficiency, and not tied to file system level.

2. To create cross-locations (cross-folders) directly in navigation tabs. We could combine multiple folders with “and”/“or” operators in the navigation tabs (by using folder trees and files lists) and visualize/manipulate the resulting files directly in navigation level without using find tools.

3. To calculate folder sizes accordingly to hard-links counts. For that Directory Opus could scan inside a directory for all hard-links pointing to the same files and counting only one time the size of multi hard-linked files.

I already searched for queries about hard-links in Directory Opus, in this forum also, and found some, like for calculating folder sizes accordingly to hard-links, but there seems to be no solution at present. For all the possibilities and operations mentioned above I think it is easy to find at NTFS level the hard-link count and also in which places (folders) hard-link are located, for each file. Link Shell Extension do that already.

What do you think about all those things? Is it possible to implement those features in Directory Opus in near future?

Thanks in advance for your replies.

[Leo]

Soft-links can cross volumes if you ever need to do that (although I think they may be one-way, i.e. you can't look at a file and find all the soft-links that point to it, like you can with a hard-link).

The default toolbars come pre-made with a menu for creating most types of junctions and links: Making Links and Junctions. I think that can replace a lot of what Link Shell Extension does, although I haven't looked at LSE in a while so I'm not familiar with absolutely everything it does.

Finding files based on which hardlinks point to them is not built-in, but may be possible via a script-column. (I am not sure if Windows provides scripts a way to query for hardlinks, as I've never needed that before in a script. We could add something to the Opus scripting objects if needed.)

I think it's more common to use tags and other metadata for the type of thing you are using links for. Opus allows you to assign one or more arbitrary tags to any file type (provided the files are on NTFS; on other filesystems, it's limited to file formats that can store tags in the main file data, such as JPEG and MP3). You can also use the Opus labels system to do the same thing. (Labels are usually just one per file, but you can adjust things so they stack.)

You can configure Opus to either include or ignore the size of junctions (etc.) when calculating folder sizes. There isn't currently a way to make it "only count each hardlink once", however. (I think that would slow calculation down at least a bit, and also leads to some questions about when to count/not count things, and what to show at the folder level for the size of folders below it... there are no real right answers here as it depends what you plan to use the size values for and what's being measured.)

Libraries already provide a way to get a view which merges together multiple folders.

[Magnetik]

Thank you very much for the fast reply.

Yes I know for soft-links, they can be useful for cross volumes but I generally don’t need that, and as you said they are one-way and not at the same level as the source file (you can’t do multi-hierarchies with equal importances and behaviors). Also even if we use soft-link there is also no program to search by multi soft-links per file, same problem as hardlinks.

Ok, I just saw for the toolbar menu to make links, although I don’t see it in my own toolbar (I don’t see the advanced link option). So yes I think it can replace LSE for some actions (like for creation).

For the finding mode, I am interested to know how to do via script-column as you said. More globally I think it is “easy” to add it as normal feature in the program, to let the user choose the different folders of interest in which the hardlinks of the wanted file(s) are located, and telling the relations to take into account between these locations, with logic operators. For example all files that are classified in [Folder A] AND [Folder B] AND [Folder C], or files that are classified in [Folder A] OR [Folder C], or files that are in [Folder A] AND [Folder B] AND NOT in [Folder C], etc. And it would only take into account hardlinks and not duplicate files, which are not technically same files.

For tag classing as you said it is only for some file types, or for all files with the NTFS tag system but there is no possibility to create hierarchies of tags (tag trees) like we do with folders. So it is limited to flat classifications. I used photos as example but I also use hardlinks for many other kinds of files.

For size calculations, for each file we can get the informations about hardlinks count and where they are in the folder structure, in the MFT table, like for file size, timestamps and other attributes. I think (not sure) we can get general informations about the file in $STANDARD_INFORMATION attribute and informations about each hardlink in $FILE_NAME attribute.
So maybe it can slow down calculations but very slightly. And we could choose to use this option or not when calculating sizes. If we choose option to not count multiple hardlinks it would count only once the size of each file with several hardlinks within the folder and all its sub folder. Also if you implement a feature to see graphically relatives folders/files sizes in one volume, like WinDirStat do, you could choose to represent only one hardlink per file, for example the one with the most little path name. Or represent all hardlinks but divide the size of each one by the total number of hardlinks of the file. So if a file with a real size of 10 mb has 5 hardlinks within the folder/subfolders analyzed, each hardlink will be considerer as 10/5 = 2 mb. I think there are many simple ways to implement those features.
The downside of the slightly increased calculation times would be easily offset by the possibility to have real "physical" size of folders/files structures. As for WinDirStat (Windows) it seems it is also not hardlink-aware yet, whereas KDirStat (Linux) and TreeSize (Windows) seem to take hardlinks correctly into account for sizes.

As for libraries I didn’t really look into the possibilities, but it seems it is like virtual hardlinks to folders, somewhat. I think it is a top layer system not handled at the filesystem level. At least it seems we can have nested libraries of folders (library trees with multiple hierarchical levels).

Also there is another new possible feature that I think about, it is the deleting of files with multiple hardlinks. If we want to delete totally one file with all it’s hardlinks at once, it should be possible to do so by a simple command that will automatically scan for all the other hardlinks and delete them, instead of manually locating and deleting each one until the last.

For hardlinks actually it is all the ways we create and use computer files classifications that can and should be improved, in all environments, to pass from mono-hierarchies-only to multi-hierarchies structures. And it can change many things in the way we can handle classifications. We could still use basic mono-hierarchy structure if we don’t mind, but we shouldn’t be limited to that. I use this method for many years now to store my files, but I lack real and comfortable tools take advantage of this to navigate, find, analyze etc.