Separating Enqueued and Actual Traffic States

For a long time, LibreQoS has only gathered traffic data at “enqueue” time - when packets enter TC. This has led to LibreQoS’s monitoring occasionally looking inaccurate - and fails to highlight just how tight the CAKE/fq_codel + HTB traffic shaping really is. A recent update changed this.

Let’s start by understanding the path taken by packets through the Linux kernel, and how LibreQoS interacts with them.

1. A packet arrives at a network interface port.
2. The packet is immediately (before any kernel processing) shunted into XDP.
3. The LibreQoS XDP system analyzes the traffic and makes sure it’s on the correct CPU. We also gather enqueue-time data: flow membership, packet type, the ISP-side device with which the packet is associated.
4. The packet CPU shunt occurs (if necessary), and either the XDP or Linux bridge shunts the packet to the egress interface.
5. TC-Classify, another eBPF program fires. It usually picks up meta-data from the first run to avoid having to look anything up again, and the packet is allocated to an appropriate TC entry-point.
6. TC - The Linux Traffic Control System - enqueues the packet with the labeled qdisc. This will be a CAKE or fq_codel queue, somewhere in the large HTB tree representing the network as a whole.
7. HTB - the Hierarchical Token Bucket system that represents the whole network tree - allocates bandwidth to parts of of the network. It does this by “spending” tokens (which refresh each tick) - first up to circuits’ “minimum” rate, and then up to their “ceiling” (ensuring that everyone gets a fair share).
8. Once HTB has allocated tokens, qdiscs (Queue Disciplines) that represent the individual ISP customers get to “de-queue” traffic. CAKE and fq_codel have different strategies for this - both built around reducing bufferbloat and enhancing fairness. Packets that have been waiting too long are dropped.
9. TC transfers the packet to the interface driver for transmission. This is where a kprobe can help - giving a chance to inspect the packet on the way out.

So in our LibreQoS kernel interface, we are able to define another function:

SEC("kprobe/dev_hard_start_xmit")
int kprobe_xmit(struct pt_regs *ctx) {
    ...
}

In this function, we very quickly associate the packet with the sender - and increment “actual” traffic counters. We can now include “actual throughput” as well as “enqueued traffic” on circuit displays:

This improvement flows through a lot of the LibreQoS display. We can now display all dashboard items with real traffic levels, submit more accurate data to Insight, and finally show just how much bandwidth CAKE is saving you on a daily basis.

Thanks to NLnet for sponsoring this project.