I've been copying a lot of files from a folder on an exFAT drive into a folder on an NTFS drive. The NTFS folders is mapped to OneDrive and the disk has only about 120GB of capacity. I am copying about 500GB into it, which I expect to fail (since the target drive's capacity is only 120GB). That's OK. after the first ~100GB copies and OneDrive finishes sync'ing it, I tell OneDrive to free up space. After a while I get >100GB free space again (since OneDrive leaves empty stubs behind and stores the actual file in the cloud for access "on demand"). Rinse and repeat. I've done it 3 or 4 times so far. Each time I Export the error listing.
I happened to notice that after the first run, there were 7080 lines (files), each reporting "There is not enough space on the disk." This seemed odd since there are 63,000 files to copy and this was after the first about 20%. Second time: 15,503 lines. Third time: 9,944. I'm now doing the fourth iteration and I see it's got about 12,000 files left.
This is not a matter of great concern to me, but I thought you might want to know about the anomalous and inconsistent behavior in reporting the files that do not succeed in each copy attempt.
Something more than just numbers of files and an assertion the numbers are wrong when we can't see any other details.
I'm not sure I understand what's being done, either. If the files are being copied rathe than moved, won't you be copying the same ones each time? Or are you selecting different files/folders each time?
As you can see, it has a total capacity of about 120GB and right now it's almost full (I've just completed my final copying of files from the source). As I write this, OneDrive is busy uploading files to OneDrive in the cloud. As it verifies a file is safely in the cloud, it deletes the file's contents from local OneDrive folder, while leaving a stub behind. This is because I have set it for "Free up space"
The option is grey here, because I've already set the entire folder for that - there's nothing left to apply it to. In an hour or so, it will show ~115GB free, once it completes uploading & cleaning up.
Meanwhile, here are the current properties of the destination folder on E:
As you can see the numbers of files & folders match that of the Source properties above.
However, the "Size on disk" is much lower, because OneDrive has already sync'ed and cleaned up the first three batches of files.
Now to the point:
I did the exact same operation each time, namely dragging the folder from the source to the destination. this initiates a Copy operation. I immediately set it to "Skip identical" (or if DO is too fast for me, I reply with that to the prompt on the first collision).
The first time I did it, it copied about 15,000 items. It should have given me an error for each of the remaining ~48,000 items. It didn't. (At least they didn't show up in the exported listing.) It only showed errors for about 7,000.
On the second time I did it, it went through the first ~15,000 files and discovered they were identical on the OneDrive destination, so skipped them. When it finally got to a file that wasn't in the destination yet, it commenced copying and proceeded to copy about 15,000 more until the E: drive filled up again. Then it started logging error . At this point the error log should have shown 30,000+ items. It didn't - only about 15,000.
You get the idea.
Here's an example of a line (anonymized) from the Error Log:
Copy,20190612_203900.mp4,E:\XXXXX Takeouts 2023-07-03\XXXXX For OneDrive\XXXXX\pictures\2019-mar-sep,There is not enough space on the disk.
Overall, DO did a great job. As you can see above, every single file & folder is accounted for, despite my having to do it in four stages. Just the error log was wrong each time. As I said, it's not a great concern to me (this time), and everything worked out in the end.
That said, I thought that you would want to know about this bug.
It's possible Skip Identical is skipping more things than you're expecting, maybe?
Or, if you're expecting Skip Identical to add an error line, it won't. Those files will just be skipped without reporting anything (since it's not an error).
There might also be small files which can fit in the destination and get done even through some of the larger files before them error, which makes the totals harder to consolidate.
There are the following distinct populations of files before trying to copy:
those that were already copied previously (and are in the destination)
those that are not yet copied
When copying, the Copy operation will encounter the following distinct groups:
those identical to ones in the destination. They will be skipped without further ado or notification.
those that are present in the destination, but not identical - they will be recorded as error (since "Treat as an error" was selected)
those not present in the destination - they will be copied
those that failed to copy because of an error, such as "out of space" or other issue. they will appear in the error listing.
I am talking about only group 4. I think you can agree with me that when progressively copying the same selection of files, as I described, where space in the destination is reclaimed between copy attempts, group 4 must become smaller with each iteration.
For example, suppose there are 1,000 files and room on the destination for only 300 each time.
the first time
is empty
is empty
300 are successfully copied (the first 300)
700 are errors due to "out of space"
second try
300 are identical and skipped
for simplicity, we assume it is empty (in practice it was, since this is a fresh destination and each file copy appears to be atomic - either it succeeded or it didn't)
300 more are successfully copied
400 are errors due to "out of space"
third try
600 are identical and skipped
for simplicity, we continue to assume it is empty
300 more are successfully copied
100 are errors due to "out of space"
fourth try
900 are identical and skipped
for simplicity, we continue to assume it is empty
100 more are successfully copied (the final 100)
no errors due to "out of space"
As I reported, the errors reported did not fit this scenario. DO appears to have a bug.
I'm also not sure Skip Identical makes sense after OneDrive has removed the old files from disk (which may mean they are no longer considered identical).
We need to see some examples which show the files which should have been copied (or in the error list) but weren't. Only then can we hope to understand, reproduce or explain what's going on.
And it sounds like it's working to me, if it's copying more files each time until they're all copied. You will not see a full list of all files in the error log if most of them were skipped due to already being in the destination.
OneDrive doesn't "remove the old files from disk". If one selects "Free up space" on the OneDrive context menu (see example in my previous post), OneDrive keeps the contents of the file in the cloud, but the file is still completely present in the file system, vis-à-vis location, name, size and other attributes. If you check the "Size on disk", however, it will say "0" (zero). Here's an example:
When accessing the file, OneDrive transparently streams the contents from the cloud. It works surprisingly well.
In practice, "Skip identical" does work for such files, in my limited experience. Same name, same size ==> identical. (Does DO use anything else to determine "identical"?)
As I wrote above, it did seem to perform correctly in terms of copying the right files. My purpose was to report the bug regarding the listing of errors being incomplete.
(sigh) I've already answered this earlier. I am fully aware that "skipped identical" files are not listed in the error log. As explained in excruciating detail in my last post, I'm talking about actual errors that were not logged ("group 4" in my example above).
@Leo - I love your responsiveness and detailed knowledge of DO. Also, I am very grateful for all the times you've helped me in the past, sometimes going above and beyond the call of duty. I wish every product I use had a Leo supporting customers.
In this thread, I'm not looking for help. On the contrary, I'm trying to help Team DO by reporting incorrect behavior. I'm so used to DO's thoroughness that seeing this behavior surprised me.
All I would suggest you do at this point is record what I've reported in your bug database and someday maybe it will get fixed. It's not a critical bug. The copying did work as expected.
If it is your practice to not record non-critical bugs - then I suppose we are done.
OK, I should have said "the file contents". It leaves empty files containing no data on the disk.
That may still complicate things, was my point.
That's one filename and some size totals. It doesn't give us enough information about what's actually going on.
We need to see details like which files are copied, which are skipped, which are in the error list and which are not to make sense of what's going on.
And we've yet to see the error list or list of files being copied, or examples of something that should be in the error list but isn't.
We'll keep this thread in mind if we get any similar reports, but so far we cannot reproduce what you describe via similar actions, and can't confirm this is actually a bug with the information we have.
I've had a look at the code as well, and can't see any obvious problems. (E.g. If the errors were placed in the list via PostMessage, they could start to be lost if the list didn't process them fast enough and the message queue filled up. But I checked for that and they aren't being sent that way. I also checked with folders containing a large number of files and all the expected errors appeared in the list.)
