FTP Exports Slow

Joshua Pickering's Avatar

Joshua Pickering

20 Jun, 2014 08:30 PM

I am a computer consultant working on behalf of NUGS.net and Mike Elberg. We're trying to solve an issue that just started happening last Friday. When they do exports of TransloadIT jobs to their server (cryptical.nugs.net/209.234.243.31 - hosted in an Equinix colo) via FTP, the uploads go very slowly (30kbps - see attachment). The FTP server is running on a 1Gb connection at Equinix and all other uploads to that server are very fast. Also, FTP exports from the TransloadIT servers to their other FTP server (in a DataPipe colo) are going at normal speeds. So the problem seems to be specific to FTP exports going from TransloadIT to Equinix.

We think there is some kind throttling happening on the Equinix side or possible some peering issues happening between the Amazon EC2 network and Equinix, but we need more information.

  1. What EC2 zone are the TranscodeIT servers located in?
  2. Would it be possible for you to do a traceroute from one or more of your EC2 servers to cryptical.nugs.net so we could see the path the traffic takes?
  3. Do you have any other customers reporting similar export speed issues?
  4. Do you see any errors in the FTP exports that might indicate some kind of network transmission or FTP protocol issues?

Thanks very much for your help.

  1. Support Staff 1 Posted by Kevin van Zonne... on 23 Jun, 2014 01:53 PM

    Kevin van Zonneveld's Avatar

    Hi Joshua,

    Sorry for the late reply. Depending on the endpoint you use (or if you are really close to one particular cluster), you're serviced from Ireland, Switzerland, or (most likely) US east.

    Here's a traceroute to the host you requested from a US east machine

    $ traceroute cryptical.nugs.net
    traceroute to cryptical.nugs.net (209.234.243.31), 30 hops max, 60 byte packets
     1  ip-10-46-240-2.ec2.internal (10.46.240.2)  0.401 ms  0.426 ms  0.492 ms
     2  100.64.145.224 (100.64.145.224)  0.961 ms 100.64.145.0 (100.64.145.0)  0.771 ms 100.64.145.96 (100.64.145.96)  0.988 ms
     3  100.64.139.117 (100.64.139.117)  0.725 ms 100.64.138.35 (100.64.138.35)  0.977 ms 100.64.138.105 (100.64.138.105)  0.700 ms
     4  100.64.142.146 (100.64.142.146)  1.056 ms 100.64.142.100 (100.64.142.100)  0.988 ms 100.64.128.30 (100.64.128.30)  1.187 ms
     5  ip-10-1-173-104.ec2.internal (10.1.173.104)  0.458 ms ip-10-1-173-86.ec2.internal (10.1.173.86)  0.474 ms ip-10-1-3-12.ec2.internal (10.1.3.12)  0.527 ms
     6  100.64.39.139 (100.64.39.139)  2.908 ms 100.64.37.209 (100.64.37.209)  1.191 ms 100.64.39.219 (100.64.39.219)  0.886 ms
     7  216.182.224.74 (216.182.224.74)  1.366 ms 216.182.224.94 (216.182.224.94)  1.458 ms 216.182.224.80 (216.182.224.80)  1.369 ms
     8  100.64.14.11 (100.64.14.11)  12.292 ms 100.64.8.213 (100.64.8.213)  16.736 ms 100.64.14.167 (100.64.14.167)  17.172 ms
     9  100.64.0.62 (100.64.0.62)  1.374 ms 100.64.0.84 (100.64.0.84)  1.606 ms 100.64.0.64 (100.64.0.64)  1.430 ms
    10  100.64.16.203 (100.64.16.203)  0.703 ms 100.64.16.9 (100.64.16.9)  0.523 ms 100.64.16.5 (100.64.16.5)  0.727 ms
    11  72.21.220.225 (72.21.220.225)  0.905 ms 72.21.222.130 (72.21.222.130)  0.939 ms 72.21.222.34 (72.21.222.34)  0.681 ms
    12  72.21.220.28 (72.21.220.28)  1.134 ms 72.21.220.44 (72.21.220.44)  1.043 ms 72.21.222.34 (72.21.222.34)  0.647 ms
    13  xe-7-3-0.was14.ip4.tinet.net (173.241.128.49)  1.261 ms 72.21.220.36 (72.21.220.36)  1.112 ms 72.21.220.44 (72.21.220.44)  1.222 ms
    14  xe-7-1-2.was14.ip4.tinet.net (77.67.68.153)  1.415 ms xe-9-1-0.lax20.ip4.tinet.net (89.149.181.166)  70.752 ms xe-7-2-0.was10.ip4.tinet.net (216.221.156.185)  0.747 ms
    15  highwinds-gw.ip4.tinet.net (77.67.68.30)  63.858 ms  63.989 ms  65.483 ms
    16  209.197.4.6 (209.197.4.6)  68.277 ms highwinds-gw.ip4.tinet.net (77.67.68.30)  65.409 ms  65.407 ms
    17  209.197.4.6 (209.197.4.6)  64.441 ms te8-2-3580.bbr1.lax1.bandcon.com (69.16.190.214)  66.791 ms 209.197.4.6 (209.197.4.6)  64.720 ms
    18  * * *
    19  * * *
    20  * * *
    21  * * *
    22  * * *
    23  * * *
    24  * * *
    25  * * *
    26  * * *
    27  * * *
    28  * * *
    29  * * *
    30  * * *
    $
    

    We haven't had other customers complain. One thing to keep in mind is that since our platform is distributed and jobs are processed by many machines, as soon as their required input files are available, in parallel.

    It's possible some hosts consider that unnatural traffic and start rate-limiting, or even rejecting. We can't really throttle this without implementing global locks / SPOFs, which we want to avoid.

    So how we ended up dealing with rejects, is retries with exponential backoffs. This however seems not to be the case. So the traceroute, host throtteling, or bandwidth issues are more likely.

    Hope this helps,

    Kevin

  2. 2 Posted by Joshua Pickerin... on 26 Jun, 2014 10:15 PM

    Joshua Pickering's Avatar

    Kevin,

    Thanks for all of this information - it's been helpful, but we're still experiencing the issue. One additional question - do you require passive or active FTP on the destination server?

    JP

  3. Support Staff 3 Posted by Kevin van Zonne... on 27 Jun, 2014 04:36 PM

    Kevin van Zonneveld's Avatar

    Hi Joshua,

    For FTP imports you can configure passive_mode (true or false) to control passive behavior. For exports that is not implemented yet.

    I just checked the code and we don't have passive turned on atm. If you need that we can implement it via a -p flag which we could make available under the same passive_mode configuration option.

    Please let me know if that's required.

    Kind regards,
    Kevin

  4. 4 Posted by Joshua Pickerin... on 27 Jun, 2014 09:17 PM

    Joshua Pickering's Avatar

    Thanks - let's leave passive mode disabled for the moment.

    Would you please do one more traceroute from one of your servers in Virginia to cryptical.nugs.net? I am gathering information for a conference call with Equinix for further troubleshooting.

    Thanks very much for all your help with this..

  5. Support Staff 5 Posted by Kevin van Zonne... on 28 Jun, 2014 11:36 AM

    Kevin van Zonneveld's Avatar

    Hi Joshua,

    Sure, I'll attach two, from different servers. Looks like something could be up with that xe-9-3-0.lax20.ip4.tinet.net / xe-9-0-2.lax20.ip4.tinet.net router.

    Kind regards,
    Kevin

  6. Support Staff 6 Posted by Kevin van Zonne... on 28 Jun, 2014 11:42 AM

    Kevin van Zonneveld's Avatar

    After 683 packages we're seeing a total loss of 26.1%. That seems significant.

  7. 7 Posted by Joshua Pickerin... on 12 Aug, 2014 05:23 PM

    Joshua Pickering's Avatar

    Hello,

    Thanks for all your earlier help with this. The problem ended up being a peering issue between Amazon and Equinix. They have a direct peering connection but, for some reason, a temporary route was put in place by an Amazon peering engineer that circumvented that direct peering connection. Once they removed the temp route, everything went back to normal.

    Unfortunately, the problem has reappeared as of a few days ago and Transloadit FTP uploads are going slow again. Would you please do another traceroute and send it along at your earliest convenience? I will send this information to Equinix so they can use it for troubleshooting.

    Thanks,
    JP

  8. Support Staff 8 Posted by Kevin van Zonne... on 13 Aug, 2014 12:21 PM

    Kevin van Zonneveld's Avatar

    Hi JP,

    Ok, here goes:

    ubuntu@misba:/srv/current$ traceroute cryptical.nugs.net
    traceroute to cryptical.nugs.net (209.234.243.31), 30 hops max, 60 byte packets
     1  ip-10-46-240-3.ec2.internal (10.46.240.3)  0.354 ms  0.440 ms  0.460 ms
     2  100.64.145.162 (100.64.145.162)  1.041 ms 100.64.145.194 (100.64.145.194)  1.025 ms 100.64.145.34 (100.64.145.34)  0.920 ms
     3  100.64.138.31 (100.64.138.31)  1.079 ms 100.64.138.179 (100.64.138.179)  1.107 ms 100.64.138.155 (100.64.138.155)  0.794 ms
     4  100.64.128.244 (100.64.128.244)  0.828 ms 100.64.129.176 (100.64.129.176)  1.077 ms 100.64.142.114 (100.64.142.114)  1.114 ms
     5  ip-10-1-3-94.ec2.internal (10.1.3.94)  0.460 ms ip-10-1-3-20.ec2.internal (10.1.3.20)  0.449 ms ip-10-1-173-52.ec2.internal (10.1.173.52)  14.884 ms
     6  100.64.39.219 (100.64.39.219)  14.919 ms 100.64.36.147 (100.64.36.147)  13.926 ms 100.64.39.75 (100.64.39.75)  10.057 ms
     7  216.182.224.88 (216.182.224.88)  1.780 ms 216.182.224.92 (216.182.224.92)  1.546 ms 216.182.224.86 (216.182.224.86)  1.328 ms
     8  100.64.2.87 (100.64.2.87)  7.764 ms 100.64.14.169 (100.64.14.169)  14.885 ms 100.64.2.17 (100.64.2.17)  4.827 ms
     9  100.64.0.26 (100.64.0.26)  1.144 ms 100.64.0.180 (100.64.0.180)  2.131 ms 100.64.0.92 (100.64.0.92)  1.711 ms
    10  100.64.16.229 (100.64.16.229)  0.732 ms 100.64.16.141 (100.64.16.141)  0.929 ms 100.64.16.75 (100.64.16.75)  0.653 ms
    11  72.21.222.148 (72.21.222.148)  0.550 ms 205.251.245.64 (205.251.245.64)  0.772 ms 72.21.222.148 (72.21.222.148)  0.789 ms
    12  72.21.220.36 (72.21.220.36)  2.342 ms 72.21.222.34 (72.21.222.34)  0.850 ms 72.21.220.36 (72.21.220.36)  2.294 ms
    13  unknown.hwng.net (209.197.0.52)  2.420 ms 72.21.220.36 (72.21.220.36)  2.242 ms 72.21.221.115 (72.21.221.115)  0.846 ms
    14  3-1.r1.at.hwng.net (69.16.191.126)  67.081 ms 2-3.r1.dc.hwng.net (69.16.190.25)  0.968 ms unknown.hwng.net (209.197.0.52)  2.582 ms
    15  3-1.r1.at.hwng.net (69.16.191.126)  67.231 ms 2-3.r1.dc.hwng.net (69.16.190.25)  11.399 ms 3-1.r1.at.hwng.net (69.16.191.126)  66.995 ms
    
    16  * * *
    17  * * *
    18  * * *
    19  * * *
    20  * * *
    21  * * *
    22  * * *
    23  * * *
    24  * * *
    25  * * *
    26  * * *
    27  * * *
    28  * * *
    29  * * *
    30  * * *
    

    Let me know if it's ok like that!

    Kind regards,
    Kevin

  9. Support Staff 9 Posted by Tim Koschützki on 28 Apr, 2015 01:19 PM

    Tim Koschützki's Avatar

    This discussion has been inactive for 30 days and has been marked as resolved automatically.
    Feel free to reply to this ticket to re-open it and we will continue the investigation.

    Kind regards,
    Tim

    Co-Founder Transloadit

  10. Tim Koschützki closed this discussion on 28 Apr, 2015 01:19 PM.

Comments are currently closed for this discussion. You can start a new one.

Keyboard shortcuts

Generic

? Show this help
ESC Blurs the current field

Comment Form

r Focus the comment reply box
^ + ↩ Submit the comment

You can use Command ⌘ instead of Control ^ on Mac

Recent Discussions

20 Nov, 2016 02:52 AM
29 Oct, 2015 05:01 AM
27 Oct, 2015 05:01 AM
27 Oct, 2015 05:01 AM
26 Oct, 2015 05:03 AM

 

26 Oct, 2015 05:02 AM
26 Oct, 2015 05:02 AM
26 Oct, 2015 05:02 AM
26 Oct, 2015 05:02 AM
26 Oct, 2015 05:02 AM
26 Oct, 2015 05:01 AM