V.13.12 x64
I was moving a huge set of folders (kareoke songs) from a local drive to a remote share, both on Windows.
As part of the process I selected unattended, and told it to overwrite.
During the process the destination drive filled up, so the dialog showed me the hundreds of files that failed.
I cleared space on the destination drive, then selected Retry on the dialog.
I was then prompted, asking if I wanted to flatten the directory, or recreate the structure. I thought this was an odd question, but specified recreate.
It then proceeded to create a folder with the partial name of every file, apparently getting confused by quotes in the filenames, and has now removed all of the source files so I can't recover them.
Here is a sub-list of the 582 folders it created before I realised something was wrong and cancelled the process. I'm sure you can guess some of the filenames:
'll be in my heart
's a Sin
't Lose My Number
- Because The Night
- Born To Be Alive
- Crazy [CO].jpg
- Crazy.avi
- Crazy.mp3
- Crazy.txt
- Happy
- Torn
- Tu amor.avi
- Tu amor.jpg
- Tu amor.mp3
- Tu amor.txt
- Undertow
- World Long Gone
- Zombies on your Lawn
- Ain't Nobody
- Babies.jpg
- Babies.mp3
- Babies.mp4
- Babies.txt
- But It's Better If You Do
- Hopelessly devoted to you
- I Write Sins Not Tragedies
- Magic
- Nine in the Afternoon
- Physical
- Souvenir.jpg
- Souvenir.mp3
- Souvenir.mp4
- Souvenir.txt
- Time To Dance
& Alcohol
& Cliff Richard - Suddenly
(US 12'' Extended)
And Forever
And Gold
Are You
back
back in anger
bangs
Beach
Blues (Don't Chu Worry 'Bout Me)
Cold Man
Dishes
Don't need a Man
Dont talk to strangers
Everybody's Eyes On You
Evolution
Freak
get your Love
guys have all the luck
Had An Angel
Hallelujah
Hallelujah (Shrek version)
Hearts
here) On my own
Hurts
I Remember Love
Is Over
Jane
Ledbetter
love
Man
Me On My Way
Me To Your Heart
More Night
night long
of Gold
of Life
Pharaohs - Wooly Bully
Power Generation - Cream
If you select files that are below multiple locations (e.g. via expanded folders, or from find results), it will (by default) ask if you want to:
Copy them while keeping the hierarchy.
So if SubDir1\File1.txt was selected, it'll be copied to SubDir1\File1.txt in the destination.
Flatten them, copying all of the files directly into the destination.
So if SubDir1\File1.txt was selected, it'll be copied to File1.txt in the destination.
That can be configured under Preferences / File Operations / Copying Files / Confirmation, "When copying nested items..." to always do one or the other without asking.
The undo log may be able to restore the files' original names (while moving them back to their original location, so make sure there is still space).
Alternatively, the File Log may tell you the original names, depending on the level of detail it's configured to store. That would allow them to be restored without having to move them back, or if the Undo log doesn't still have the action.
The names changing should not happen, other than the parent paths being removed if the option to flatten things was chosen.
Can you give more detail on what some of the original names were?
Was the copy done using the normal Copy Files button on the toolbar, drag & drop, or another method? Are any custom commands involved?
Some of the filenames still have ' characters in them, and " quotes aren't possible in Windows filenames. Or was it another type of quote character? (Or files on a non-Windows drive?)
This was a simple drag of a long list of folders, so there was no hierarchy prompt. It was only when 'Resume' was chosen on the Unattended dialog that the prompt appeared.
Also, single quotes are valid in Windows filenames, it's double-quotes that are not.
I did the undo operation, and some files appeared to be copied back, but given that most of the destination files had been changed into empty folders I don't see how this could possibly work.
Some example names were probably something like 'll be in my heart was You'll be in my heart - Born To Be Alive was Patrick Hernandez - Born To Be Alive - Ain't Nobody was Rufus & Chaka Khan - Ain't Nobody - Babies.jpg I'm guessing was a subfile under folder Pulp - Babies
Another example folder (containing 4 files) Rockwell - Somebody's watching me has become a single empty folder y's watching me, which makes no sense, so there's not really a pattern to the problem.
I think the point is that the resume decided to take all of the files it couldn't copy, then in some weird way re-process the path, assuming that single quotes and spaces were delimiters, then 'copy' those corrupted filenames as folders, discarding the contents, but still deleting the source files.
I've used DOPUS for fifteen years, so I'm aware of how to copy and move files around, I've just never used the 'Unattended' mode (and probably won't ever again!).
I've been going through the 582 newly created folders to try and work out what they might have been originally and try and re-source them.
I just managed to reproduce a similar issue:
I created a folder with lots of subdirectories containing files
Copied a few of those folders to a target network share
Selected all of the folders (including those already copied) and Move-dragged them to the target
When the dialog appeared, clicked 'Unattended' (and left default action in dropdown)
When the completion dialog appears showing the uncopied files, click 'Retry'
It then copied the 'clashed' files in a flat structure into the root of the target
I've not reproduced the weird name splitting thing, and perhaps it only happens when you have hundreds of files clashed, but there's definitely something wrong with the unattended feature.
Were the files really there in the root of the destination after the operation finished, or was Opus just displaying "ghost" placeholders that didn't really exist?
Doing the same thing you describe above, I can confirm there is a bug with the placeholders appearing in the wrong place during the move, but the files being moved do end up in the correct place in the end, and it's just a visual error.
Once the move finishes, the placeholders go away and the files are all where they should be, at least in my tests so far.
Placeholders appear as translucent/faded items in the locations that you're copying to, to help see what the directory will look like when the copy/move completes, and keep track of what's already copying when selecting additional things to copy.
They can be turned off via Preferences / File Operations / Copying Files / Show ghost files for queued copies. If you turn that off and OK/Apply Preferences, do you still see a problem when doing a similar test?
I've made a note to look at the placeholders bug in more detail, since that shouldn't be happening either, but if there is still a way to cause data loss (or files going to the wrong place) I haven't been able to reproduce that yet. We'd definitely like to repro and fix that if it is happening.
Hi Leo,
I made a copy of the folder contents for reference, and they are all empty folders created in the target folder, not placeholders.
I think it may also be related to whatever made it ask me if I wanted to flatten the structure. I wasn't starting a new copy, just performing Retry when more disk space had been cleared. Could it be related to it not stopping the copy, but copying lots of smaller files after the ones that failed, then something got out of sync?
Each source folder contained a mix of typically approx 40MB, 70K, 6MB, 10K, so when the 40MB failed to copy it would succeed on many of the smallest files, leaving a long list of large ones available to retry. Perhaps try and reproduce the destination full in this way, so you have a collection of sparsely populated leaf nodes rather than a block?
Files fail or succeed, but the reason for failure isn't normally a factor in what Retry does, so it shouldn't matter if the error is because the drive filled up or something else (e.g. the files already exist in the destination).
Did you try what I mentioned above? Is it actually putting files in the top-level folder, or just showing temporary placeholders there which vanish when the operation completes?
Hi Leo,
No, I haven't tried the ghost option as I'm tied up with work at the moment.
I understand that the reason for failure seems unlikely related to the issue, but it does make a difference to the process;
If you have created an internal list of 100 files, start copying them, and then network fails, you may have the first 40 files copied, and the last 60 not. Simple case, and retry is straightforward.
If you have a disk-full issue, then you may copy the first 40, the 41st fails as it's too big, then 10 of the remaining 49 may still fit on the target. Hence why you are left with a non-contiguous list of 40 files still eligible for retrying. It may be unrelated, or it may not.
As I already explained, there were empty folders physically in the target folder, some with filenames, some with corrupted names. I'm not sure if Opus creates those as 'placeholders' or if Opus just uses a visual representation on the GUI, but they were left there after the operation.
First, many thanks for taking the time to report this to us and all the detail you've provided above!
No need to do any more tests on your side. I've been investigating this all day and now have a handle on what happened, although I've still not been able to reproduce exactly what you saw. (In the situations I found where it failed, I'd get an error message instead of anything being copied/moved.)
Some of it depends on Preferences settings (or related command arguments that do the same), and path lengths or number of errors can also be a factor. While it doesn't matter if the errors are due to a full disk or something else, it can matter if the errors happens to something at the top-level or something under one of the selected folders.
In the short-term, we're going to remove the Retry button from the Unattended dialog entirely, until we can be sure it works reliably in all situations.
(Normal Skip/Retry/Abort handling won't be affected by any of this. The change will only be to the Unattended UI.)