Move or clone links/junctions as links/junctions

Help & Support
1

I have CLI tools to do this. But, given the richness of DOpus I was trying to figure out how to do it with just DOpus.

I am looking to accomplish two similar operations. A command to move and/or a clone a collection of one or more files and directories, which treats all links as literal things without attempting to resolve their targets.

The goal is to "move" or "clone" the items literally as their NTFS file-system entities without reparse-point interpretation.

So a /J (junction) or a /D (symlink) gets moved and recreated in its destination exactly as it was in its source.

  • No modifications to the target definition in the reparse-point (link-target).
  • If elevated privilege is required, then prompting occurs once, as needed, for the entire move/clone.
  • Moving or cloning a directory tree with symlinks (incl junctions) just recreates the tree with no effort to validate, resolve, or extract the symlink target's (effectively treating the symlinks the same way a .lnk file would have been treated).

Thus a relative-link or absolute link moves without any attempt to modify or fix what it's target says. I.e., Exactly what a backup-archive tool might do in wim/tar recording it.

2

I have an Opus button which runs xcopy for this, where the /B switch makes it copy symlinks at least:

(Bearing in mind what Xcopy means by "copy symlinks" may not always be what you actually want. It's a complex topic.)

(As the button name suggests, this also duplicates the permissions from source to destination. Something Opus can do itself -- Preferences / File Operations / Copy Attributes / Copy security permissions. I like to use XCopy for a particular combination of esoteric cloning modes when moving from one data HDD to a replacement. I made this button many years ago, possibly before Opus had the option to copy permissions, which may be why it was named after that when that isn't the main reason for using it these days. It is buried deep in my toolbars and gets used about once every 5 years here. :slight_smile: )

@admin
@leavedoswindowopen
@nofilenamequoting
xcopy "{filepath$|noterm}" "{destpath$}{file$}" /O /X /E /H /K /I /B /J

200dpi_wc9665_dopus.png1306×789 36.8 KB

It's not something built-in to Opus. If you ask Opus to copy folders containing junctions and/or symlinks, it will copy them as if they were normal folders, creating new, standalone copies of them.

What Opus does is usually what you'd want if you were copying just those folders to a disk (e.g. to give someone else a copy of some data), but not what you'd want when cloning a whole disk (e.g. to replace one physical disk with another). Opus doesn't really aim to be a cloning tool; there are better tools for that, although often just running XCopy does the job fine if it's a data drive and not a system drive.

There's also the question of what is meant by "copying a junction/symlink". Should it point to the same thing the original pointed to? (Won't always be possible with a junction, of course.) Or should it be converted into a path relative to the original, even if it wasn't actually a relative symlink? The answer will vary, and it's a fairly technical and esoteric thing, needing specialist knowledge, and also very rarely needed, so it's not something we have tried to tackle. If the whole drive is not being cloned then junctions and symlinks are often fiddly things that need to be re-created manually, where an automated tool may not do the right thing.

Opus has built-in support for creating all types of junctions and links, on the other hand. Just not copying them, since copying them often requires human decision making.

3

Most of the time the reason for cloning is because one has a directory structure that is self-referential from the root level and you want to copy it or move it around. Which is something we do all the time since that is how we design the structure of installed packages (for various reasons - because blending various tools requires it, or it is efficient for aggregating various forms of critical and non-critical data for backups and replication).

In other words, nothing inside the package-root references anything outside the package root without using a \... absolute path reference to the same drive.

So it extremely common for our setups on Linux, OSX, and Windows. More so with use of disk-images [VHDX, DMG, IMG] and VMs, containers/dockers.

I.e., literally cloning [symlinks] as TAR and WIM can do, and as GIT will on all systems except windows, is a commonly used devops feature. We also allow clone selective symlink include, where the rule is to include archive-marked-symlinks.

That's why in my other posts I commented that we use the "A" archive bit on the SymLink itself to tell the clone to archive the symlink's target rather than just recording the symlink.

The pattern for cleanly and clearly using links can be consistent and simple. The right usage of junctions, relative links, and absolute links has clean rules one can follow. In point of fact, one of the features I have in my notes I would like to see for DOpus is a way to use the "label" system to be enabled to set "label" style settings based on attribute flags and symlink forms (junction, symlink, absolute path, relative internal path, relative external path).

Using SymLinks is an extremely efficient and powerful way to deploy software packages and reduce both quantity of code and configuration nightmares.

A analogy for symlink usage that might help is to think of them the same way the SQL uses VIEWS versus TABLES. Or similarly, depending on your programming language backgrounds, the way namespaces, packages, and perspective types are used for abstraction and composition.

Because Windows has very poor "google" information as to how junctions, symlinks, etc work it is frequently misunderstood and improperly used (including within Microsoft where I worked for a long time as an Architect).

And frankly, DOpus is the best tool I've ever seen or worked with for data and file management. It is so amazingly powerful and useful that with some modest investment it can replace dozens of other tools. It works very well for coordinating managing many machines and devices (linux, windows, osx, and iot devices).

4

If you use relative symlinks (which Opus can create) and the xcopy button above, that should do the trick.

5

More news for those that are using SymLinks extensively for DevOps with DOpus.

I will follow with posts on some "challenges" using DOpus (and Microsoft's cmd.exe) with SymLinks. But Microsoft made the following announcement to further underscrore the importance of symlinks for modern Windows devops work.

See article: https://blogs.windows.com/buildingapps/2016/12/02/symlinks-windows-10/

If you want to try using "mklink" without elevated-admin requirements, you will need to either call the CreateSymlink api yourself with the new flag SYMBOLIC_LINK_FLAG_ALLOW_UNPRIVILEGED_CREATE, or (for now) turn on windows developer-mode in your settings (aka like Android has) using the information in this link "https://docs.microsoft.com/en-us/windows/uwp/get-started/enable-your-device-for-development".

6

I need to leave my +1 here! o)

It's a pity DO does not recreate links while copying things around.
I wish for a new option to be able to choose the desired behaviour (resolve links or not).

BUT: With all the options available to the copy/move operation already, like copy metadata or not, copy permissions or not, set archive flag, clear readonly, preserve timestamps etc., it's time for something like a Copy-Wizard mode which asks what kind of (complex/rare) copy process you like to run. Toggling DO preferences around and settings them back afterwards is no good solution. Flipping the manual to build a new button each time is neither.

A nice addition would be a dialog in the middle of the process if DO encounters a link. You could choose what to do (resolve or not), just like the exists/replace dialog. There should be an option in the "unattended mode" dialog as well, so you can tell DO what to do with symlinks on the fly. All symlink-like things that failed to copy/recreate should create an error entry in the log/unattended log or as well ask what to do while in the process.

Having to revert to using external tools and xcopy for this (which surely has other restrictions) is probably not the reason why people buy into pro level filemanagers. No offense, but really! o) I think there is still a lot of potential in DO's capabilties when it comes to file handling and the different situations you encounter while doing non-regular copy/move ops.

1 Like
7

Ideally DOpus would implement a copy command that copied the "ReparsePoints (Junctions and Symlinks)" themselves, unaltered, rather than trying to resolve them and copy their final-(true)-path "targets".

I would much rather have a "broken" reparse-point, than a duplicated file structure containing no Reparse-Points.

It would also be a "faster" copy. Additionally, it might be "nice" to have the copy command support a "preflight" phase that would build a list of dirs and files it planned to copy and reported broken-links that might result if it were allowed to proceed.

Advanced power-user usage: Better still, if a given file being copied was on the same volume, if it would have the option to create "hard-links" for files instead of copying; thus de-duping for free and being faster.

Same principles apply for move in cases where dirs and files are crossing onto different volumes.

1 Like
8

Not for most people/situations, since it would slow down copying everything else by having to check if things are junctions or not. Even more so with your preflight phase idea.

I'm not against it as an option, but copy speed is something it would harm, not improve, if turned on.

9

Are you sure of that???

The FileAttributes DWORD of Windows has a specific flag FILE_ATTRIBUTE_REPARSE_POINT that is provided in any File/Dir activity that obtains the attributes.

File Attribute Constants (WinNT.h) - Win32 apps | Microsoft Learn.

The copy operation must already the attribute DWORD so it has that information for free.

To retrieve the reparse point tag, use the FindFirstFile function. If the dwFileAttributes member includes the FILE_ATTRIBUTE_REPARSE_POINT attribute, then the dwReserved0 member specifies the reparse point.

Additionally, assuming it got the attributes using appropriate NT/OS APIs, it also got the ReparseTag for free at the same time. So it knows if it is a IO_REPARSE_TAG_SYMLINK (symlink) or IO_REPARSE_TAG_MOUNT_POINT (junction).

P.s., assuming DOpus is using the faster and more efficient:
OpenFileById and GetFileInformationByHandleEx class FileIdExtdDirectoryRestartInfo and FileIdExtdDirectoryInfo calls rather than FindFirstFile etc, all the information is all free and much more.

I have high speed tools that traverse drives and manage reparse points, etc. The performance differences can be 10X or better using FileIds, avoiding redundant: ACL/SACL validation, path-folding for ., .., and illegal chars etc.

You can comfortably scan an entire loaded machine's "C:" drive in a couple seconds looking at everything, glob matching files, attributes, etc.

Additionally, walking the $MFT directly using SCTL_GET_NTFS_FILE_RECORD, turns out to be about 40% slower than walking the directory tree using FileId's because they more efficiently cache/check the ACL/SACL attributes.

1 Like
10

Hmm, possibly, not that you mention that attribute (I forgot about that). I would have to look at the code.

What are you doing that involves needing to replicate directory hierarchies with junctions so frequently? What's the use-case for this request, so we can understand how it would help and who else it might benefit?

(Keep in mind that any changes to the main copy code need to be tested extensively, and to work with what is already a complex function with a lot of options. There's a higher cost to changing things here than in other parts of the program.)

11

I totally understand how carefully you need to test any changes in this area. DOpus is an amazing body of code, and it really does so many things. I can't use windows without it, and I've gotten many friends and my company to use it.

I develop language engines and edge cloud and IoT client and server devops software.

The activity where links gets used so heavily is with devops work, often involving npm/nodejs. I happen to have to also do a lot of work involving Linux and Android (also Linux) devices.

But many of my friends and power users I know need to copy / restore (backup, update) directory trees. It is now very common for them to contain fairly complex link path chains if they've been installing open source (Linux ported/portable) apps.

Little background on me: I was an architect at Microsoft for many years on languages, tools (.Net runtime). I owned JavaScript for a while, then was lead architect on PowerShell before I left.

1 Like
12

+1

Are you the one who came up with the idea to reduce arrays with only 1 element to that element when returning arrays from functions? o)

13

That makes sense, thanks!

At the moment the best method would be to set up a button that runs xcopy for this type of thing. (Or drag & drop event, e.g. alt + drag & drop, or use add something to the right-click d&d menu.) But I agree that isn't as nice as having it work with the Opus copy command.

14

No, that was not me :wink: