3

I'm trying to use tc to shape traffic on a system with 10 Gbps NICs and I find that I can't get anywhere near 10 Gbps through any qdisc. When I do:

tc qdisc add dev $ifc root handle 1: htb default ffff
tc class add dev $ifc parent 1:0 classid 1:1 htb rate 32Gbit
tc class add dev $ifc parent 1:1 classid 1:ffff htb rate 1Gbit ceil 16Gbit burst 1G cburst 1G

My throughput gets capped around 3 Gbps. I've tried variations with CBQ and HFSC. No matter what I do I can't seem to get around that.

Adding just the qdisc does not cause the problem (as I previously said). I've spent days reading everything I can find that mentions tc and qdisc and "10G". There seems to be a lot of mailing list activity 6-10 years ago (perhaps on the cusp as 10G became common, taking over from 1G) but no resolution. Am I missing something? Is it impossible to shape multiple gigabits/second on Linux?

Chris Nelson
  • 221
  • 1
  • 2
  • 7
  • What are the specs of the machine you're running this on? Just a thought but it could be a CPU and/or memory bottleneck. Over 1 gigabyte per second having to traverse whatever internal algorithm is involved with tc and qdisk could certainly be sensitive to general CPU and RAM performance. – bgp Jun 03 '22 at 19:23
  • 1
    Thanks for the follow up. The machine is overspec'd or lightly loaded depending on how you look at it. 32 cores with a load average around 5, 56 GB with around 5 used. – Chris Nelson Jun 03 '22 at 20:09

1 Answers1

3

This is due to the qdisc locking problem (it sticks to one CPU core by default). The known solutions are XDP-redirect, XPS, and HTB offload.

The XDP-redirect solution is xdp-cpumap-tc which creates an independent HTB qdisc for each CPU core, and then filters IP traffic into the appropriate CPU / top level HTB using XDP-redirect.

We develop an application called LibreQoS which uses xdp-cpumap-tc and we've found XDP is a very good solution for our use case at least. There are some notable limitations, such as that you need to be able to divide up your traffic to about 4-6 Gbps per CPU core. That can be a lot of throughput total (20+ Gbps easily) but if you need a single HTB leaf to shape more than 4 Gbps that will not be possible until HTB's qdisc locking problem is solved in the kernel.

HTB offload is a potential solution as well. Throughput is very good. Drawbacks are that only Mellanox/Intel NICs have support for it, and its max leaf depth is 3 levels.

lutelife
  • 31
  • 3