28

I'm trying to copy a file from one of my local machines to a remote machine. Copying a file with size upto 1405 bytes works fine. When I try to scp a larger file, the file gets copied but the scp process hangs up and doesn't exit. I have to hit Ctrl-C to return back to the shell.

I have observed the same behavior with FTP as well. Any ideas about what might be causing this?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
Chandranshu
  • 383
  • 1
  • 4
  • 7
  • 3
    The disk is nearly full and only has 1405 bytes left? Your firewall only allows one packet to pass (assuming 1500 MTU)? What does rsync do? Can you create the file manually? Does it work in the other direction? – Konerak Mar 29 '11 at 13:15
  • I don'thave an answer per se, but the size suggest to me that the network and packet size somehow is involved. I have found Wireshark and/or tcpdump good tools for troubleshooting in the past. –  Mar 29 '11 at 13:18
  • 1
    You say the file does get copied. Have you checked the contents to verify the whole file has been copied, or do you base the conclusion solely on the appearance of a file with the correct size in the target path? –  Mar 29 '11 at 13:18
  • @Konerak: The disk has lots of free space. The MTU is indeed 1500 but there is no firewall. The machine is directly behind a router. I can copy to the server from other machines. So, I think that is equivalent to be able to create the file manually. I can't check its working in the other direction as the client machine doesn't have a public IP. –  Mar 29 '11 at 13:21
  • @JenEriC, I have tried capturing network packets using tcpdump and have all the logs available but am no good at making sense out of them. Should I update the question with my logs? –  Mar 29 '11 at 13:23
  • 4
    If you have a clean log, it certainly won't hurt to have it included. If you can mess around with the MTU size, try to decreae it and see if the problem decreases with the same amount; it might help someone to figure out whats going on. –  Mar 29 '11 at 13:32
  • @JenEriC I saw Stephan's answer below before I saw your comment and reducing the MTU worked. Thanks much. –  Mar 30 '11 at 05:45
  • This is definitely an MTU issue. See also [Can't access select https sites on Linux over PPPoE](http://unix.stackexchange.com/questions/4261) and [How to diagnose a reliably unreliable connection?](http://unix.stackexchange.com/questions/11107). – Gilles 'SO- stop being evil' Jun 02 '11 at 18:26

5 Answers5

36

This definitely sounds like MTU problems (like @Konerak pointed out), this is how I would test this:

ip link set eth0 mtu 1400

This temporarily sets the allowed size for network packets to 1400 on the network interface eth0 (you might need to adjust the name). Your system will then split all packets above this size before sending it on to the network. If this fixes the scp command, you need to find the problem within the network or make this ugly fix permanent ;)

törzsmókus
  • 184
  • 1
  • 9
Stephan B
  • 475
  • 3
  • 5
  • 1
    This fixed it. Could you please point out why setting MTU to a lower value works? Just a pointer would be enough. –  Mar 29 '11 at 13:32
  • 1
    See the Wikipedia page on [MTU](http://en.wikipedia.org/wiki/Maximum_transmission_unit) (look out for PPPoE) and maybe [RFC 2516](http://tools.ietf.org/html/rfc2516) wich suggests an MTU of 1492 when behind a home router. –  Mar 29 '11 at 13:40
  • 1
    Thanks for the pointers. I found another article [here](http://www.mynetwatchman.com/kb/adsl/pppoemtu.htm) that nicely explains why lower MTUs might be better. –  Mar 29 '11 at 14:09
  • 2
    @Chandranshu I'd like to point out that matching your network MTU to some ancient ATM header size is no longer important with current internet backbones, so if 1492 works for you - go for it ^^ –  Mar 29 '11 at 14:17
  • @Stephan I found another article which suggested pinging a remote machine with the DF bit set with increasingly larger packet sizes to figure out the optimal MTU. Using that, I have set my MTU to 1472. Anything larger and my old problem resurfaces. –  Mar 30 '11 at 05:43
  • 3
    If his solution worked, you might want to accept his answer as valid so he gets the bounty. :) – Shadur Jul 16 '12 at 20:58
  • @Chandranshu this should not be necessary. The IP protocol is designed to accommodate for mismatched MTU sizes. Your'e not doing something horrible like blocking all ICMP packets are you? – phemmer Jul 18 '12 at 12:24
  • 1
    @Patrick, IP fragmentation is horrible for performance, so TCP disables it and relies on getting ICMP feedback when a router finds the packet too large and will start sending future packets smaller until they fit. Stupid routers ( the kind that use PPPoE, which steals 8 bytes for its header ) often don't bother sending those messages. – psusi Jul 22 '12 at 01:49
  • @psusi and setting a lower MTU for **ALL** traffic is better than having traffic for a single destination require frame size negotiation? – phemmer Jul 22 '12 at 08:43
  • @Patrick, yes... if the problem is a single ( or few ) destination(s). In this case, the problem router is on his end, so it makes for a reduced MTU to ALL hosts, and also prevents the auto sizing mechanism from working, thus, you need to manually lower the MTU. – psusi Jul 22 '12 at 22:34
  • 1
    @Shadur: I had originally asked this question on stackoverflow and it was shortly moved here thereafter. So, I was delayed in accepting this answer well by over an year. – Chandranshu Jan 07 '13 at 08:23
  • 2
    @psusi, shortly after I had reduced the MTU on this machine (which was at a client's office), we got to know that they were facing this issue while connecting to a number of hosts. It also began showing up for some other clients in the same area. It turned out that the common culprit in this case was the ISP who had setup a blackholing router by error as part of infrastructure 'upgrade'. – Chandranshu Jan 07 '13 at 08:31
3

Consider that the MTU is typically 1500 B (due to the limitations of ethernet). Of those 1500 B not all are used for 'data'. So-called protocol overheads take a hefty chunk out of the 1500.

  • SCP (uncompressed) requires a 65 B header.
  • TCP requires a 20-60 B header.
  • IP requires a 20-60 B header.

In light of this, it's not so surprising that your payload is limited to 1405 B.

P.s. Have a play with wireshark and check the IP headers. Are they allowing packet fragmentation?

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
  • 1
    The ssh protocol overhead goes into the TCP byte stream, not each individual packet. TCP obviously is aware of, and accounts for the size of its and IP's headers. – psusi Jul 22 '12 at 22:38
0

Sounds like bad hardware to me. If it's only one machine on the LAN, it might have a bad Ethernet card; if it's all machines on the LAN, I'd check the hub or router.

Ernest Friedman-Hill
  • 121
  • 3
  • 1
    I disagree. If faulty hardware were in play, Chandranshu would very likely have difficulties in lots of other areas as well. Unless he's using some sort of h/w encryption acceleration device for the SSH connection, or something like that.. –  Mar 29 '11 at 13:20
  • @Mels: I am indeed facing problems while uploading files through even FTP. I also tried using HTTP to POST the file to a web server and the connection there hangs as well. May be I'll try with another network card. –  Mar 29 '11 at 13:25
  • @Chandranshu: but you are able to browse the web using that machine? That's essentially the same thing. Is it always the same file you're trying to upload? Perhaps some kind of Deep Packet Inspection firewall is false-positive'ing on the contents of the file, thinking it's malware? –  Mar 29 '11 at 13:33
  • @Mels: To answer your question, I don't have a web browser on the machine. However, I could successfully issue GET requests and copy files from other remote machines. The problem used to appear only when large traffic originated from this machine. Stephan's answer above has helped me track down this problem and resolve it. –  Mar 30 '11 at 05:48
0

Sounds like a bad router configuration, data packet transfer limit should be less amount I think

abab
  • 146
  • 3
0

There is some kind of tunneling going on between the source and target machines involved. TCP will transmit an MSS when opening the connection (that will be used for ssh/scp), and this tunnels along the way (that add encapsulation, and thus reduce the maximum available MTU) should compensate by transparently modifying said MSS en-route to the destination (and vice versa).

Some tunnel (VPN?) is not doing its job. Or the MSS on your machine is misconfigured.

I am guessing you'll see the same behavior if you ssh to the machine from which you're trying to copy, and just do an ls in an empty dir. That should work fine. But if you "cd / ; ls -lR" (and thus get large packets), this will hang as well.

Nitzan Shaked
  • 133
  • 5