Even if you manage file copying yourself and do not use CopyFile at all, my suggestion should work.
If FILE_FLAG_WRITE_THROUGH is used but FILE_FLAG_NO_BUFFERING is not also specified, so that system caching is in effect, then the data is appended to the system cache but the thread will not return until the previously submitted data is written too.
No idea where this CopyFile stuff is coming from nor where you are getting your ideas from about which flags we do and don't pass. See my reply above and please stop assuming we do something with no evidence while urging us to do what we are already doing.
This is getting a little ridiculous, with ideas from out of nowhere about how things work and too many misunderstandings not just about Opus but about Windows in general.
The most useful thing you can do on this subject is to try different threshold values and let us know which seem to work best with your setups, so we can get an idea of the default value that will work the best for the most people.
Leo, two postings above you explicitly asked for some ideas and details how to flush the system caches and you got some ideas. I would assume, that you appreciate that, even if they were wrong.
My suggestion doesn't depend on how you fill the system cache.
In any case you need a way to synchronize your copy thread of the progress dialog with the state of the file system cache becasuse currently the copy thread hurry ahead the actual progress.
And the easiest way I know is to just write the next file in write-through mode using FILE_FLAG_WRITE_THROUGH without FILE_FLAG_NO_BUFFERING since this suspends your thread until the caches are flushed. This should prevent the copy thread from hurry ahead the actual progress and leads to a more reliable progress.
PS: If I am completely wong, please ignore everything I wrote. 
Unfortunately I have no further suggestions for a good default value except that I wish to try values less than 1 MB to see if that may affect my freezing issues.
I asked for ways to flush files which were copied using buffering and had already been closed. That seems to be what you're asking for, but Windows does not seem to provide a way to do it, at least for non-admin processes.
Opening other files without buffering seems unrelated to the caching of the earlier files, as my video showed: the cache usage increases whether that is done or not.
Write-through will slow things down if on for all small files, and seems to make no real difference if only used for every 100th small files. It is the main factor in write speed and buffer latency on the write end.
BTW, if you want to test what Opus would do if it turned off buffering every 100 files, just set up 99 small files followed by a 1.1MB file, then 99 more small files, etc. and set the threshold to 1 so it triggers that pattern.
Yes, and my answer was related to this. But we misunderstood each other.
Actually I do not know any way to do this except flushing the whole volume which needs admin priveleges. Thus I proposed to indirectly flushing the file system cache by writing the next file a little bit different. Obviously that kind of indirectly flushing can only be done if there are further files to be copied. But that's exactly our situation, we want to synchronize the copy thread with the file system cache each 100th file to provide a more accurate progress when copying thousands of small files.
My suggestion was to flush the data of closed files (that were written buffered) indirectly by writing the next file buffered and direct. Since such a write does use the file system cache the data to be written is appended to the file system write-queue and the thread is suspended until the previous data (from the closed files) as well as the current data is written. In other words opening a file for writing using the flags FILE_FLAG_WRITE_THROUGH without FILE_FLAG_NO_BUFFERING provides an option to indirectly flush all previously written (and already closed) files by just using the trick of writing the next file a little bit different.
Yes, that's correct. That's why I say to write files buffered (data is fully passed through the caches) but direct by only using the FILE_FLAG_WRITE_THROUGH without FILE_FLAG_NO_BUFFERING. The current implemention of DOpus as well as of xcopy uses write-through only when buffering is turned off (as I guess).
Is should be if write-through is used buffered.
I tested that too, and it doesn't improve anything. But as you said, this turns off buffering every 100 files (similar as in the xcopy example) but we need to keep buffering on but switch over to direct writing.
There is no way to simulate a write-through with buffering mode turned on by using xcopy since /j explicitly writes everything unbuffered. To be honest I have no idea, why the xcopy example performs so much better on my system. I guess that it depends on the fact that each file is copied by an newly launched process. But it doesn't matter, it uses the wrong write mode and it can not help us.
How do you know that? Where is the documentation which says the filesystem or cache manager in Windows behaves that way, queuing things up as you describe with that combination of flags?
But you said before that doing exactly that with xcopy made everything work great. Now you're saying it's not the answer and what Opus needs to do is some other thing, for some reason.
This feels like a wild goose chase.
Question: Does any program copy files in an optimal way for your system? Have any of the ideas you're proposing been tried by anyone and proven to work better than what we now have?
I thought the last postings were related to improve the progress dialog because the progress doesn't reflect the actual state since the data is buffered in the file system cache. Dylan said "Forget the freezes, let's improve the progress by introducing some extra flushes". You ask for an idea how to flush the caches and you get an idea. Nothing more. If you think the idea is bullshit than just ignore and forget it. 
Anyway the progress often shows very inaccurate throughput values depending on the system cache state. Obviously it would be fantastic if this can be improved without degrading the copy performance.
Since you already have the new unbuffered copy operation maybe you can falsify / verify the idea of indirectly flushing by just changing the flags if you think it's worthwile. Otherwise we will wait for somebody else having another idea. 
How can I globally enable this new feature to let DOpus always write unbuffered (e.g. for internal copy actions performed during a synchronization etc.)?
The advanced threshold does not except "yes" as the argument of the copy command does.
You can't at the moment. 1MB is the lowest threshold value you can set right now.
Is the new unbuffered copy operation already fully tested (similar as it will be done for the next major release) or is the new copy operation just for testing purposes but probably produces incorrect data? Thus it's not recommended to use it all day long?
It is unlikely to cause any problems but it is, obviously, brand-new code which we have only been testing for a very short amount of time.
I've spent some time to find out a good threshold value for my system. I tried to configure different hotkeys that use several nonbuf threshold parameters below 1 MB.
Files having a size below 400 KB are faster when copying buffered but above 400 KB there is no significant delay when copying unbuffered. Since DOpus is much more reactive when copying unbuffered and the progress is more reliable I want to use 400 KB as my threshold for the unbuffered copy mode.
Unfortunately 1 MB is the smallest value that can be configured in the preferences. 
Can you please improve this to use kilobytes for this value to allowing fine-tuning of this impressive feature?
I think kilobytes would be more consistent in respect to the nearest neighbour (copy_buffer_size), which uses kilobytes also.
The reason we used megabytes for the units was that this thread started out only talking about "very large files"...
The Copy NONBUFIO lets you specify KB. This is undocumented and might change in the future, but works in 10.0.5.1:
Copy NONBUFIO=400KB
(You can't do the same in Preferences, however.)
Obviously, but nobody could say that the new copy mode performs so much nicer for files below 1 MB too. Personally I like the smooth progress and the reactive dialog. In the past I could click the abort button and sometimes I have to wait 10-20 seconds until the dialog disappears. When using the new unbuffered mode DOpus reacts to my clicks near instant all the time.
Since there is no performance gain for files above 400 KB on my system, I think that this would be a very good threshold. Unfortunately this was not predictable without actually testing it.
So please switch over from megabytes to kilobytes. Since your current default is 0 this should not affect too much users.
Optionally you can provide a unit to the threshold argument of the copy command (if the unit is omitted MB is used but 100kb is possible as well as 1gb). But I would perfer the foce kilobytes solution to be consistent to the other tuning parameters. That was me who initially configure 65536 since I expected KB. 
Nice. Thanks for the hint!
I just tested the newest beta and set the threshold to 100 MB. Copying a single 8 GB file to an USB stick was very smooth with the newest beta and the progress bar didn't jump anymore. Then I dig out my 16x 500 MB files used in my last test and started a copy to the USB stick and it performed like a dream (no hiccups as before). Then I created 16.000 x 500 KB files and started the copy again. Unfortunately there were short freezes (1-5 seconds) when accessing the stick in parallel with DOpus. But after a few seconds the device was refreshed and everything is quick and fast again until the next hiccup. This happened multiple times before the overall progress reached the end and the dialog disappears.
Anyway, since I never copy so much small files I can live with that. Thanks for this great work guys.
Do these freezes look similar as in my video clip posted some days ago?