25

I'm running CentOS 5.7 and I have a backup utility that has the option of dumping its backup file to stdout. The backup file is rather large (multiple gigabytes). The target is an SSHFS filesystem. To ensure that I don't hog the bandwidth and degrade the performance of the network, I would like to limit the speed with which data is written to the "disk".

How can I limit the ability of stdout based on a byte number? For example, limiting a process's ability to write to about 768Bps.

maxschlepzig
  • 56,316
  • 50
  • 205
  • 279
Wesley
  • 13,963
  • 12
  • 35
  • 49
  • 6
    @msw this question is about rate-limiting `stdout`, the previous question was about limiting network traffic. Different beasts... – voretaq7 Mar 14 '12 at 19:14
  • 7
    @msw Last I checked, the point of the SE network is to collect a massive amount of knowledge in specific areas. In this case, it's about *nix. Limiting the output of `stdout` is different than limiting bandwidth. Just because the underlying problem is the same doesn't mean that future visitors won't find each question useful independent of the other. Remember that the Qs&As are just as much for googlers as for the actual askers, which is why deleting answered questions is such a cardinal sin here. – MDMarra Mar 14 '12 at 19:27

3 Answers3

28

You can add a rate limiting tool to your pipeline.

For example there is pv which has a rate-limiting option:

-L RATE, --rate-limit RATE

Limit the transfer to a maximum of RATE bytes per second. A suffix of "k", "m", "g", or "t" can be added to denote kilobytes (*1024), megabytes, and so on

An alternative is the tool buffer which has:

   -u microseconds

After every write pause for this many microseconds. Defaults to zero. (Surprisingly a small sleep, 100 usecs, after each write can greatly enhance throughput on some drives.)

maxschlepzig
  • 56,316
  • 50
  • 205
  • 279
3

I would be tempted to dump this on a local disk instead of writing it directly to the remote volume. This way you can use rsync with --bwlimit to easily limit the transfer.

One of the benefits to this approach is that if there is a connectivity hiccup to the remote machine, the transfer is interrupted, but the backup itself doesn't explode in a blaze of glory.

MDMarra
  • 478
  • 1
  • 4
  • 12
  • Noted. I do so wish that `rsync` was a viable solution in this rather limited situation I find myself in. =| – Wesley Mar 14 '12 at 19:33
3

pv (pipe view)

echo asdlfkjasdf | pv -q -L 12
Ole Tange
  • 33,591
  • 31
  • 102
  • 198