27

I'm trying to ping a remote host, but I get an error.

# ping 192.168.80.1
PING 192.168.80.1 (192.168.80.1): 56 data bytes
ping: sendto: No buffer space available
ping: sendto: No buffer space available
^C
--- 192.168.80.1 ping statistics ---
2 packets transmitted, 0 packets received, 100% packet loss

It works for other hosts:

# ping 192.168.16.1
PING 192.168.16.1 (192.168.16.1): 56 data bytes
64 bytes from 192.168.16.1: icmp_seq=0 ttl=254 time=0.442 ms
64 bytes from 192.168.16.1: icmp_seq=1 ttl=254 time=0.402 ms
^C
--- 192.168.16.1 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.402/0.422/0.442/0.020 ms
#

The message "No buffer space available" seems to indicate some sort of memory error. And indeed, when I check with Netstat, the "mbuf clusters" number looks VERY wrong:

# netstat -m
11780 mbufs in use
4294966716/32768 mbuf clusters in use (current/max)
0/3/6656 sfbufs in use (current/peak/max)
1785 KBytes allocated to network
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

What is going on here? Is there away to fix this without downtime, or do I need to reboot the host or restart the network interface?

Stefan Lasiewski
  • 19,264
  • 24
  • 70
  • 85
  • 1
    Can you post your dmesg output? – Mike H Aug 10 '10 at 20:00
  • I have pasted my own. In my case, it was a backup that was clogging some queue somewhere, it's unclear to me why - but suspending then resuming the backup (rsync --bwlimit=40) fixed the problem for us. I have also tried doubling `net.inet.tcp.sendbuf_max` and `net.inet.tcp.recvbuf_max`, didn't change anything. – anarcat Jun 26 '13 at 15:57

1 Answers1

11

Since the problem seems isolated to just one interface -- I assume you're using plain old class C masks for these two networks -- I'd just quickly bounce it:

# ifconfig en0 down
# ifconfig en0 up

Obviously you need to substitute the correct interface name here for en0.

You may have to restart any servers listening on that interface, and any established TCP connections using it will drop when you do this. It's brief, though, so I don't really view such a test as "downtime".

Be sure not to do this while ssh'd in to the box on the interface you're bouncing. It's best to log in on the console when you do this, if you can. If the server is remote, a modem connection is best, since bouncing the network interfaces won't affect serial gettys. If you must do this while logged in over the network, be sure your connection is coming in over a different interface.

Warren Young
  • 71,107
  • 16
  • 178
  • 168
  • 4
    Well, turns out this interface had a loose Ethernet cable. I tightened the cable, and the "No buffer space available" error has gone away. Now, I need to figure out if I need to worry about the 'mbuf' number, or will it clear on it's own? – Stefan Lasiewski Aug 10 '10 at 22:01
  • Unfortunately, bouncing the interface didn't fix it (there are multiple interfaces on this machine, I tried each one in sequence, and then all at once, hoping that something would free the space). I rebooted the box, and hopefully this problem won't happen again after we fix the cable. – Stefan Lasiewski Aug 10 '10 at 23:31
  • 1
    An update 4 years later. Yes, I do need to worry about the mbuf error. This problem occasionally pops up on my FreeBSD boxes and is almost always associated with high network load, and possibly with a buggy cable. – Stefan Lasiewski Dec 29 '14 at 23:41
  • Bouncing fixed it for me. – gravitation Mar 14 '15 at 07:42
  • I came here specifically for OpenBSD 5.8 AMD64 running on Cloudstack running CentOS6.5 AMD64. Every time the physical box gets interrupted this vm cannot get out. I tried to bounce the interface and that did not work. Just thought I'd share here in case someone else went down a similar path. UPDATE- less than one minute later it is working, so maybe this does fix it. – danno Jun 06 '16 at 17:52