Recent Changes - Search:


Code:


Social:


My journals will take the place of a blog. If you go to the Journal page there is an RSS feed to subscribe with.

(:blogcal group=Journal:)


Sites I take responsibility for

West Volusia, Florida

Documents

Technical

General

Me

Living

Places I frequent

Hobbies

Humor

Items for sale:


Clipboard

edit SideBar

The keys to fastest and global wide area network file transfer speed

Include our styles below Infobox - invoke as >>infobox<< ... >><<

Codebox: - invoke as >>codebox<< ... >><<

warnbox: - invoke as >>codebox<< ... >><<

editingbox: - invoke as >>codebox<< ... >><<

noticebox: - invoke as >>codebox<< ... >><<

Page bread crumbs: Main - MyPublicTechnicalNotes - LargeFileTransfersAcrossWideAreaNetworks

Pages by tags: (:listtags:)
Subscribe to this wiki: RSS Feed RSS or subscribe to this page for changes: RSS Feed RSS
496 articles have been published so far. Recent changes
(:addThis btn="custom":)

2016-03-22: This site is being moved to my main site at https://kevininscoe.com/wiki as part of a consolidation to one domain.


The keys to fastest and global wide area network file transfer speed are:

  1. UDP rather than TCP protocol
  2. Concurrency (multiple simultaneous transfers) with file management
  3. Less than 80% circuit utilization to avoid congestion control
  4. Less software layers and more straight file copy, avoid Subversion or rsync and instead archive single files for transfer
  5. Avoid encryption over the wire, instead encrypt and shrink before transfer
  6. Shortest path
  7. Mirror files or edge delivery (Content Delivery Network) at locations closer to receiver
  8. Use WAN acceleration
  9. Delta changes - block level changes rather than whole file synchronization
  10. Use cloud or distributed file systems where possible

Useful links:

http://udt.sourceforge.net/

After reading this government web site (folks who apparently shift terabytes of data around every day)

http://fasterdata.es.net/

http://fasterdata.es.net/tuning.html

http://fasterdata.es.net/tools.html

I have come to the conclusion our current tool sets are inadequate. Given the tuning they suggest has been applied in several scenarios now with little improvement (mainly TCP congestion control and increased buffers) I am soliciting your input on improved toolsets to test and evaluate for possible adoption in the future.

As a global company with large amounts of content shifting around the world on an almost daily basis we have to find a way to improve our current situation.

This encompasses all of our operating system in use at xxxx today (Windows, Apple OS X, Solaris and Linux).

Here are some of my personal notes on the subject of they are of help.

Troubleshooting large file transfers across wide area networks

One thing I have learned it that much of the inefficiency is in the software itself and usually not in the underlying network or operating systems. Thatís not to say we donít have process improvements of course (such as moving data closer to clients, using delta transfers rather than whole collections, etc..) but it clearly shows our classic tool sets are woefully inadequate for the job at hand.

There are times when back-grounding data transfer is the best approach for link congestion concerns (ala BitTorrent - Mass Transit?) however there are other times when a near immediate file transfer has to take place. Mirroring technology seems best suited for this task but sometime the amount of data burst to the mirrored locations are almost too much for the mirror copies to stay current. One thing I have observed is short small transfers (many of them like in collections or an rsync) is highly inefficient. Favored for speed are larger single non-encrypted (on the wire) transfers in parallel (which get better utilization of the link capacity) This I believe will become the new paradigm in our bulk file transfers. I still favor remote mirroring (it is essential) but the way in which we transfer the mirrored data from one location to another in a timely fashion is our new challenge.

Buffers and tuning:

Bufferbloat from Jim Getty


Kevin's Public Wiki maintained and created by Kevin P. Inscoe is licensed under a
Creative Commons Attribution 3.0 United States License.

Back to my web site - http://kevininscoe.com

Edit - History - Print - Recent Changes - Search
Page last modified on December 16, 2010, at 08:59 AM EST