i just digged the forum for topics like that. I already checked the copy buffer setting and tried different values (512 - 8192k).
This thing bothers me quite a while already, so while boosting other network components, I am now back to DOpus performance.
If I copy a larger file (around 1GB) from an network location (UNC-pathed Win2003 Server) onto my local disk (Win7-Pro), the
operation takes progress with about 5-10MB/sec. That is quite slow for a GBit-Ethernet. If I copy the same file with Explorer.exe
the operation is 2-3 times faster and steady.
And now, what really makes me wonder is, if I copy that file with DO, abort the operation at 50%, restart the copy,
suddenly the file "flies" in! I can see it on a local "network-meter" and on the Win2003-Eth-Diagram, the transfer-rate
is then about 40-60MB/sec, but only up to the "point" (these 50% in this example), where I aborted the previous transfer, then
it is slow at 5-10MB/sec again. Now what the heck is that ?!.. o)
So these are 2 issues, which might belong together, i don't know.
Explorer is 2x-3x faster in general, it makes no difference how often i copy the same file.
DOpus "sleeps" while copying over network, but a single aborted transfer on the other hand, is awesome fast - up to the
point, the last copy stopped. Watch the attachment,it shows the network-throughput from the Win2003-Box.
I have no anti-virus or special firewall software installed on any of my systems. I use the windows built-in fws.
Data transfering quicker the second time implies it is being cached, presumably on the server side as the copy is still using the network. So the question seems to be why is the server being so slow to read the file and send it over the network the first time it is read?
Is the server fully patched with Windows updates?
When comparing with Explorer, did you copy different (but similarly sized and same types of) files to ensure they weren't already cached?
@ tbone
Try disabling any of the following settings for your network adapter: "IPv4 (TCP / UDP) checksum offload" (or in German "IPv4 Prüfsummen abladen"). That gave my gigabit network a tremendous boost in speed.
I would like to chime into this topic as I am having a similar strange behavier of DOpus (Version 10.0.0.1) regarding network file transfers.
My setup:
Router: D-Link DGL4300 (Gigabit) connected to 100/10 MBit cable modem
Switch: Cisco SML2008 (Gigabit) connected to router
All devices are connected to the Cisco switch, Jumbo frames are enabled on all devices.
PC1: laptop HP Elitebook 8540w (i7 Q740, 16 GB, 500 GB HD, Intel Gigabit), Windows 7 Ultimate 64 Bit
external Stardom 3,5" RAID-5 over eSATA
external Raidon 2,5" RAID-5 over USB3.0
PC2: tower (Core2Duo Q6600, 8 GB, 3.2 TB HD, dedicated Intel Pro Gigabit), Windows Vista Ultimate 64 Bit
When doing large file transfers from the interal HD of PC1 to PC2, there are no problems.
When doing large file transfers from any of the external HDs of PC1 to PC2 it first goes well, then suddenly it slows down, then goes a bit up and later on the transfer stalls. This is the time when Windows shows me it has lost connection to the network.
Transfer speeds with Dopus: ~ 51 MB/s
The stalling is repeatable but sometimes it happens earlier, sometimes later during the transfer process.
Transfering the same file from PC2 back to PC1 works flawlessly.
Transfering the same file from PC1 to PC2 by using Win Explorer it works flawlessly but much faster.
Transfer speeds with Win Explorer: ~ 108 MB/s
So something seems to be odd with the way Dopus does its file copy over networks.
If Windows is telling you you've lost connection to the network while Opus is reading data from an external (USB?) drive and writing data to a network drive, then I'd look into updated drivers for the motherboard (USB controller, bus etc.) and network card.
Changing the copy buffer size in Opus may also help, as some USB devices/controllers are sensitive to the size of each read/write request.
@leo
Ok, you got me.. that windows 2003 server machine is not fully patched, but i'll try
to put it uptodate now. I stopped installing updates after a special KB-somewhat, never
went through and several updates reconfigured that whole machine without my permission.
I'll be back again when this is done.. o) Thx so far!
@freezer
Thanks for the hint, i remember having played with several tcp-ip settings, but cannot
recall if i touched this one too (and i reset those settings, after thinking they do not improve
anything).
Occassional "network-drops" while copying big files from that win2k3-box to the win7-box
I had also! I changed the active NIC to get around this, as the server has 2 GBit-adapters.
That's no solution, only a workaround.. you could do the same, just to clarify what end of
your network is the primary problem.. ?!
That bad networking and windows behaving "odd", is so time consuming.. o((
@ Leo
All drivers are absolutely the latest ones.
I was reading data from an external eSATA RAID-5 and writing over network to an external 4lane SAS RAID-5 (connected to a post pro workstation).
I will try playing with the copy buffer size in Opus.
@ tbone
I found this setting to be the number 1 reason for slow GB networks during my researches all over the web.
Unfortunately I cannot change the NIC on my laptop workstation.
Just wanted to note that changing the copy buffers of Opus did not help. It is getting quite annoying that I am not able to copy files over LAN via Opus.
Any other suggestions what to try? Is there anything I can log so the Opus developers can trace/reproduce the problem?
I can't think of anything that could be logged, unless you're seeing very high CPU usage during the copy (which could indicate something is going wrong, and could be traced using Process Explorer; shout if that is happening).
Since the network connection is actually dropping sometimes it seems like an issue with the network hardware and/or drivers (or firewall/av) becoming overloaded/confused to me, since Opus would not be disconnecting the network.
No - CPU load is normal.
As I wrote above - copying through Windows Explorer works without any network disconnections. So I would assume there is something in the way how Opus does the file transfers which results in a network drop.
I don't doubt that Opus is doing something different which is triggering the problem, but I can't see how it's doing anything but triggering a problem in something else, since the network connection itself is dropping when all Opus is doing is reading and writing to files, and since this isn't a problem everyone else is seeing.
My network is 100% GigE but DO was only averaging 30-40MBs on 25GB file transfers to and from a large raid 60 2008 server. Windows 7 Ent could hit and sustain 130MBs. After discovering this thread (thank you!) I bumped the copy buffer up to 16348KB and my transfers jumped up to an 112MBs average on DO, but increasing DO's buffer beyond 1634KB doesn't get anymore throughput.
The server is using an Intel quad NIC server card teamed for 4.0GB, the workstation is using an Intel dual NIC server card teamed for 2.0GB. Using a LAN speed test I've recorded transfers that can hit around 150MBs at peak w/ 130MBs on average.
Only issue I see now is that about 2/3 of the way through the throughput starts coasting down to between 70 & 80MBs, and I don't why. The server isn't doing anything that I can tell, but the slowdown is always about the 2/3 mark.
@BadKarma
I wish I had your problems.. o) 30-40MB/s is what I want to achieve.. i'm still stuck at about 5-10MB/s.
Ok, this is no teamed GBit-Ethernet on my side, but.. o)
Did you test your transfer speeds with Explorer or another filemanager as well ?!That might give you
some different results or further hints where to look. I checked the web for any filecopy-caching
mechanisms one could alter in Windows-Server-OSs, but it looks like are no options for that.
Regarding my tranfers-speed-topic:
In the meantime I fully patched my Problem-System to the latest Microsoft-Patchlevels, but that did
not help much. But at least I seem to have figured out that strange caching behaviour, it seems it is
related only to truecrypt volumes. I'll continue my research.. thanks so far.
ps: Further tests with that CopyBuffer-Size let me go back to 32KB.
Sasa, disabling jumbo frames drops the speed back down under 100MBs. Starts at just under 100MBs and steadily falls to around 55-80MBs.
TBone, Yes , Explorer can sustain anywhere from 125-130MBS all the way through the transfer (with small slowdowns that last only 10-15 seconds) . I'm not sure why but DO just starts slowing down after 2/3 -1/2 way into the transfer. It's still an ok speed but it just makes me wonder what's happening.
I’ve experienced similar issues: I edit HD video via HD video network server. Just using the fastest network components didn’t give me the required speed at first. I found that different brands and types of network cards and routers played a big part. (Software seems to vary in communication methods via these network devices) I just had to research, test and find what was the most compatible with the software I use. Some of the software works better with different network cards of the same speed. To address this, I had to install multiple cards on the same systems and set the software to use the card it’s most efficient with. Because of these issues, systems like Avid and Autodesk Flame Premium use propriety dedicated lan built to their system to maximize the performance, thus reducing driver, hardware and software compatibility issues.
My troubleshooting checklist:
Driver updates
Use a different computer with the same (i.e. opus software on the same network to test)
Check the cables and connections (bad cables can reduce the functions network cards try to use. Some software like Opus might try to use some of the more advanced features crippled by the bad cabling)
Remove router or switch from chain to test
Swap network cards around if possible
Change network card types brands, etc. (laptops use an external to test)
Just switch things around and using a simple process of elimination always works great for me.
One surprising find, was another device on the network can affect the performance of another or all devices, even though it’s not being actively used to move files, etc. Just the way the router communicates collaboratively with all network devices can play a big part; the managed routers a most prone to this. One port with identical settings can work better than the other with various software; very strange!
This might not be the cause, but maybe it can help. Cheers
@ tbone
Did you try my hint with disabling "IPv4 (TCP / UDP) checksum offload" (or in German "IPv4 Prüfsummen abladen")?
Before I did that my network was stuck to <10 MB/s.
@ all
Which security software are you using on your computers (AV & firewall)?
I am using MS security essentials and just saw that it has realtime network inspection - so I will do some tests if that might interfere with Opus.
Am I just blind or is there no possibility to edit my own postings?
Anyway, @ tbone again: You might consider buying an Intel NIC - the Intel Gigabit CT is quite cheap but very fast (I am using that on 2 of my computers)
-> geizhals.at/deutschland/351749
@tbone
I don't use per workstation security software. I choose to hardware firewall the WAN with a Fortinet FortiGate 60C (Integrated Threat Management for Front Line Locations). Catches everything! I've never had a virus on my network since for 4 years now. No slowdowns on workstations via bloated AV software. Before this though, I used Panda Security; very low overhead and never had virus’s escape quarantine for two years on 12 research workstations. Other low overhead AV's are: VIPRE and Online Armor ++. I have also used Paket Fence and Network Security Task Manager; for real-time monitoring and reporting across the network. Through testing: Norton, AVG and AdwarePro, interfered the most with performance issues for all 30 AV's we tested; Panda Security and Vipre performed the best across the board for network and single workstation setups.
@freezer
I did that now (again?), i cannot really recall if I did this change to the NICs anytime before,
I also checked for the latest drivers for the NICs and put some blank unfragmented hardrives
into the involved systems. I'm now at nearly 60MB/s. I guess your hint was the most efficient
change I did so far, although I think I already played around in this corner, but nonetheless,
thank you for stressing this setting again!.. o) Sidenote: I disabled ANY checksum offloads,
for UDP as well as TCP and enabled those JumboFrames @8K again.
BUT!..
DOpus is still behaving weird. I can copy the same file over and over again back and forth with
Explorer.exe and always get these 60/MBs, sometimes even more when this strange caching,
(or what is is) kicks in, then I can even get around 100MB/s shortly. This sudden speed up or
boostet filetransfer is NOT related to truecrypt volumes. I was wrong with that, this behavior
occurs even with freshly formatted partitions.
DO on the other hand is not capable of reliably copying with 60MB/s. It would do this one time,
and the next two attempt it falls back to 20MB/s. If I stop and restart the transfer with Explorer.exe
it is 3 times faster @ those 60MB/s. CopyBuffer is at 8MB right now. I noticed that it might have
something to do with HOW I copy the file. Drag&Drop vs. Copy&Paste or the CopyButton. As
I don't think that there is an internal difference on how DO copies files.. I must forget about this, it
might just have something to do with the abnormal situation, where a filecopy is fast, and the
next is slow.
So whatever I change.. settings, harddrives, patchlevels.. it is still DO being slower than Explorer
in most of the situations.
@kgibbs
I am not a fan of security-software as well, with the latest MS-Update, that Microsoft-Security-
Essentials got onto my Windows7-machine.. I let it be right now, as I did not find any slowdowns,
but I will pay further attention.. o)
To sum things up:
I still think there is something "in" DO, that keeps us from getting a steady and reliably fast file copy.
It cannot be the CopyBuffer-Setting anymore, because the same setting is fast.. and then slow for the
next try.
Are you sure nothing else is going on with the folder (or network) while this is happening?
For example, if Opus is configured to show media duration in the status bar (i.e. the total playtime of all the files in the folder), or set to show thumbnails, or anything like that, then it will be working through all the files in the folder(s) you are viewing, using the network (and disk etc.) in the background.
That is one possible explanation for the speed varying when you do the same copy each time in the same program. There are many others, of course.
Looking at the various things the Resource Monitor tool (comes with Win7) shows being used may reveal where the bottleneck is, too. (e.g. The disk, network, memory, etc. may be hitting 100% usage when the slowdown occurs, which could point towards what to investigate. The tool gives a detailed breakdown of e.g. which process and file is responsible for the disk access. Of course, it may be worth looking at this on both the client and the server machine.)
