GP SoftwareTwitter
Opus FAQsManualCommandsObjects

Errors occurring during a batch move

#1

I frequently need to change metadata within picture files. To do this I use a third party program (Exif Date Changer) which reproduces all the modified picture files in a sub-directory under the original picture directory named "Adjusted". When I am happy with the modifications I move the modified files back to the original location and overwrite the original files. I would normally do this on a file count of, say, between 100 and 300 files.

During the move operation I regularly get a DO error message as below;
"An error occurred moving 'filename': The process cannot access the file because it is being used by another process. (32)
Options - Retry, Skip, Abort"

I always use the retry option and the process continues happily until the next error. So I sit there with my finger above the Enter key tapping away until the whole move is completed. Not a major hassle but annoying.

I have observed that the happens more often with larger files. 2-3 MB files never cause a problem, 10 MB files do occasionally, 25 MB files give the message up to about 20% of the time. I shudder to think how it will react to newer 40 MB picture files! The files are sitting on an old NAS which is connected by gigabit ethernet but is probably a bit slow.

Have you seen this issue before?

#2

It means something has the files open, which blocks them being replaced.

It could be another program or part of Opus, or a shell extension something else has loaded into Opus.

If you can't work out what is locking the files, a process monitor log can be sent to us and we can usually see from there. Instructions are in the FAQs area, but please link your account first if you want us to do this.

#3

Thanks Leo

I have now linked my account to the forum account. I am surprised I had not done it before as I have been here in prior times.

I have reproduced the problem whilst running Process Monitor and sent you the zipped log file at crashdumps for review.

Thanks for the help
Bob

#4

Thanks for the log file.

It indicates that the thread in Opus which is getting metadata about the file is getting in the way of moving the file.

This is unusual, and I think the real problem is that the file is taking over 2 seconds just to have its metadata extracted. (It should usually only take a fraction of a second.)

Would it be possible to send us one of the files that the error dialog is appearing for? I'd like to see if it also takes a long time here.

It's also possible that antivirus is involved and slowing things down, perhaps because it is delaying access to the file due to finding something about it suspicious and needing to do a deep scan of it. I can see BitDefender in some of the callstacks, so it might be worth seeing if the problem still happens if you temporarily disable its realtime scanning.

It also looks like the network drive the files are on may be under heavy load from another operation? Some of the ReadFile requests, even for small amounts of data (43KB) are taking a very long time compared to normal. That could also be due to antivirus rather than the network or server/drive.

#5

Thanks for your analysis Leo. That is very interesting and suggests some lines of research for me to do here.

My sole purpose for doing all this bulk moving is to get metadata changes into the picture files before I work on them in Adobe Lightroom. Specifically I am changing capture time data in the files to allow for the many time zone changes encountered while travelling through different countries. I remember to change my watch but forget our cameras sometimes.

After a trip I am confronted with about 40 folders of daily picture files which I work on one at a time, as necessary, using Exif Date Changer. This program works its way through all the photos in a folder and created new photo files in a new directory with adjusted time values in the metadata. On completion of this process I copy the new files over the old and this is when I sometimes get the error message. Exif Date Changer is still in memory but not running at the time. I now wonder if it still has its hooks into some of the metadata processes used by Dopus. There has been some anecdotal evidence of this and I made sure to mimic actual work flow conditions whilst generating the Process Monitor log file for you.

Now I would like to see if changing my workflow might ease the condition. I will get back to you on this.
I will also look at anti-virus effects. I believe the network is fairly heavily loaded whilst these operations are going on but I don't believe there other big things going on at the same time. Wife on the internet perhaps. All the photo files are being generated and moved to networked drives. I will see if is better to move files to a local drive (SSD even) before starting my processes and then transferring files to the NAS only on completion.

Still, it is interesting that the problem is down to timing issues on different threads and I would think there will be some programming opportunities there for Dopus.

Thanks again
Bob

#6

OK I have now done some testing.

My thoughts about Exif Data Changer being still resident in memory and somehow slowing metadata access were unfounded. I was still getting file open errors in Dopus after the metadata program had been dismissed.

I then concentrated on Leo's comments about the slow file access times showing up in the Process Monitor log file. My original data was sitting, and being processed, on a 10 year old Buffalo LinkStation NAS which I knew was a bit of slow beast. So I decided to try the process on some faster equipment:
. a newish fast NAS, a Thecus N5550
. a local standard hard drive, a WD black caviar
. and a fast solid state drive, a Samsung Pro 970

For each of the tests I used a folder containing 137 photo files which was reproduced on all the drives. The Bit Defender antivirus was left running. The photos were processed by Exif Data Changer which produced a copy of the photo files with changed capture time metadata into a sub-directory below the original file directory. Exif Data Changer was shut down and then the modified photo files were moved back to their original locations, overwriting the original files. This was where I had been getting the file open error previously reported. These operations were timed and are reported below.

Whilst using a faster NAS I was able to reduce processing times by about a half but I did not eliminate the Dopus file open errors, however their frequency was lower.

Moving the data files to the local PC environment, however, made a tremendous change to the operation times. Exif Data Changer was 15 to 30 times faster and Dopus performed its file movements 50 to 100 times faster. Importantly there were no file open errors.

As far as my work flow is concerned, processing data locally on a PC beats processing on a NAS hands down. However it still takes me a further 8 minute per directory moving the data to and from the NAS and there is not much I can do about that. I am still on a winner and I don't get the errors.

There are still issues for Dopus doing data moves on slow or fast NASs. Would you still wish to get your hands on one of the files that has caused a file open error? It would be a 24 MB file which I would need to upload to Google Drive or similar.

1 Like
#7

Yes please, it would be good to look at the file to see if there is a reason it's taking a long time to process, or if it was just because the network or NAS were very slow.

#8

I have sent a link to your "Pretentious Name" contact.

#9

Many thanks for sending the image!

I did some tests using the Description column and the metadata is read very quickly here, processing a folder full of copies of the file in a very short time.

The file-in-use error would only occur if the file was open right at the point Opus was trying to move it, and stayed open for at least half a second after that. In my local tests, each file was processed in a much shorter time.

That rules out anything particular about the file causing Opus to take an extra long time processing it, which was my main worry.

It looks like the NAS (or network, or antivirus, etc.) are what was slowing things down so much.

Testing here was done with a basic gigabit LAN and a Windows server at the other end.