Transfer large files without hogging the bandwidth (is there a BITS equivalent for Linux?)

Question

Is an implementation of Microsoft's Background Intelligent Transfer Service (BITS) available for Linux systems?

I'm looking at my options for transferring large files to a remote Linux server over the internet and I don't want it eat all of my (limited!) upstream bandwidth.

I've successfully used BITS on Windows systems in the past but this time I'll need to be transferring to and from Linux servers.

If it makes any difference both systems are likely to be running Ubuntu based systems although ideally I'd like a solution that is distro independent.

Answer 1

First, the easy way: rsync has a --bwlimit parameter. That's a constant rate, but you can use that to easily throttle it down.

Now, if you want the adaptive rate, there is the linux traffic control framework, which is actually fairly complicated. There are several references I'm aware of:

• Linux Advanced Routing & Traffic Control
• Traffic Control HOWTO
• A Practical Guide to Linux Traffic Control

Personally, when I have to set this up, I use tcng to simplify the task. Here is an example:

dev office {
    egress {
        class ( <$ssh> ) 
            if ip_tos_delay == 1 && tcp_sport == PORT_SSH ;
        class ( <$kyon> )
            if ip_dst == 172.16.1.62; // monitoring host
        class ( <$fast> )
            if ip_tos_delay == 1;
        class ( <$default> )
            if 1;

        htb() {
            class ( rate 1440kbps, ceil 1440kbps ) {
                $ssh = class ( rate 720kbps, ceil 1440kbps ) { sfq; };
                $kyon = class ( rate 360kbps, ceil 1440kbps ) { sfq; };
                $fast = class ( rate 180kbps, ceil 1440kbps ) { sfq; };
                $default = class ( rate 180kbps, ceil 1440kbps ) { sfq; };
            }
        }
    }
}

In that example, traffic being sent out over the office interface is being classified into several classes: ssh, kyon, fast, and default. The link (a T1, when this was in use) is capped at 1440kbps (this must be slightly lower than the actual link rate, so that buffering happens on the Linux box, not a router). You can see that ssh is assigned 720kbps, kyon 360, etc. All can burst to the full rate (the ceil). When there is contention, the 'rate' acts as a ratio, so ssh would be given 1/2, kyon 1/4, etc. The 'sfq' says how to handle multiple ssh sessions; sfq is a form of round-robin.