Some aspects of linux have the reputation of being hard. Traffic control via queueing disciplines for bandwidth management for example. Even the title is enough to strike fear into the heart of a seasoned system admin.
Which is a pity really, as the things outlined in chapter 9 of the lartc are very useful in practise. The problem is the documentation is very descriptive - which is good once you know roughly what you're doing - but which has quite a steep learning curve if you don't. In fact it's pretty vertical if you don't already know quite a lot about networking. A few more worked examples would help over and above those in the cookbook.
Instead, like most people in a rush, I have relied on attempting to bash together snippets of code that are on random blogs to make /sbin/tc do what I want it to do, without really understanding what is going on.
This time, when presented with a problem for which this is the exact tool, I found I needed to dive deeper, and actually understand it, as none of the precanned recipes worked. It was a case of "if all else fails try the manual".
So now I think I've got a vague handle on what is going on, I'm documenting what I ended up doing because I'm sure I will need a worked example when I come back to this in the future. If its useful to you too, so much the better.
The Problem
We have a need to test the loading speed of our web page and trading platform under a set of network conditions that approximate the following;
- Local LAN, unrestricted
- "Europe", 20ms round trip latency, limit of 512kbit/sec in and out
- "SE Asia", 330ms round trip latency, limit of 128kbit/sec in and out.
In practise thats quite generous, particularly in the case of the south east asia profile. There was no way I was getting 128kbit on the wifi in shakey's on Rizal Boulevard in Dumaguete earlier this month. Which was better than the hotel wifi.
The Solution
Background
We have selenium to run the tests via webdriver/remotedriver to two windows virtual machines, one running chrome and one running IE. They run on a Linux host system, and can see a loadbalancer behind which lies one of our performance test environments. We need to add latency and bandwidth restrictions to their connections, effectively to put them into each of the traffic classes above depending on which test our CI system asks them to run.
The load balancer has been set up with three virtual servers, all listening on the same IP address but different ports.
- Local: 9090
- Europe: 9092
- SE Asia: 9091
Each virtual server has the same webserver pool behind it, so they're all the same from the point of view of the load balancer, but we'll use the different destination ports to switch the traffic between the different sets of network latency and bandwidth restriction we need to simulate the different customer locations.
The linux virtual machine host has the guests vnet network devices attached to a bridge. In turn the bridge is attached to the network, via a bonded interface. In our case bond0.30.
To make this work for both machines, we'll apply the traffic management on the bond0.30 side of the bridge.
Ascii art diagram of that;
IE Windows VM - vnet0 eth0
\ /
host bridge 30 - bond0.30
/ \
Chrome Windows VM - vnet1 eth1
Qdiscs and Classes
There are three creatures we're dealing with here;
- qdisc - a Queueing Discipline. These are the active things we're going to use to control how the traffic is managed. qdiscs can be classless or classful. We're going to use a classful qdisc called htb
- classes - We'll use these to separate the traffic into its constituent flows and to apply different constraints on each flow.
- filters - Similarly to iptables, these allow us to specify which traffic ends up in which class.
Chapter 9 says that you can only shape transmitted traffic, which is not 100% accurate, as we can do things to inbound traffic too, however our options are very limited.
So, looking at the default qdiscs, classes and filters
[root@vm01 ~]# tc -s qdisc show dev bond0.30
qdisc pfifo_fast 0: bands 3 priomap 1 2 2 2 1 2 0 0 1 1 1 1 1 1 1 1
Sent 47844819829 bytes 140593932 pkt (dropped 0, overlimits 0 requeues 22)
rate 0bit 0pps backlog 0b 0p requeues 22
[root@vm01 ~]# tc -s class show dev bond0.30
[root@vm01 ~]# tc -s filter show dev bond0.30
[root@vm01 ~]#
The "-s" option shows the statistics. So, but default, we have a queue discipline called pfifo_fast, which just passes traffic.
Each device has a default root which we use to build upon. We can also attach handles to classes and qdiscs to allow us to relate each part to the others and build up chains to process the packet stream. "root" is shorthand for a handle of 1:0, or the top of the tree.
One of the most useful pages I found is here; http://luxik.cdi.cz/~devik/qos/htb/manual/userg.htm
Things worth repeating from that link are;
- tc tool (not only HTB) uses shortcuts to denote units of rate. kbps means kilobytes and kbit means kilobits
- Note: In general (not just for HTB but for all qdiscs and classes in tc), handles are written x:y where x is an integer identifying a qdisc and y is an integer identifying a class belonging to that qdisc. The handle for a qdisc must have zero for its y value and the handle for a class must have a non-zero value for its y value. The "1:" above is treated as "1:0"
The whole page is worth reading carefully.
The Design
.
The Pentagons are filters, the circles represent qdiscs, and the rectangles are classes. One important point is that this diagram in no way implies flow. This is hard to get, and I had problems understanding the comments in section 9.5.2.1 "How filters are used to classify traffic" - particularly;
"You should *not* imagine the kernel to be at the apex of the tree and the network below, that is just not the case. Packets get enqueued and dequeued at the root qdisc, which is the only thing the kernel talks to."
.
The way I squared it in the end was to think of it as an order of application for traffic flowing through the root qdisc.
So in the above we have the root qdisc, which is an instance of the HTB qdisc. From that depends each of the classes we set up to handle the three different classes of traffic. We use htb to limit the outbound bandwidth for each of the classes (1:10, 1:11, 1:12). When we define the root qdisc we specify that class 1:10 will be our default class for the bulk of the traffic we don't want to delay.
Setting up the root qdisc;
INTERFACE=bond0.30
tc qdisc add dev $INTERFACE root handle 1:0 htb default 10
"root" is a synonym for handle 1:0. $INTERFACE is defined in the shell script to make the porting from machine to machine easier. This installs the htb qdisc on the root for our bond interface, and tells it that by default all traffic should be put in a class called 1:10.
Now we add classes for each of the types of traffic, along with the bandwidth limits we want to enforce on each of the traffic classes.
# default class
tc class add dev $INTERFACE parent 1:0 classid 1:10 htb rate 1024mbit
# "europe" traffic class - outbound bandwidth limit
tc class add dev $INTERFACE parent 1:0 classid 1:11 htb rate 512kbit
# "se asia" traffic class - outbound bandwidth limit
tc class add dev $INTERFACE parent 1:0 classid 1:12 htb rate 128kbit
We now attach the network emulator qdisc, netem, which we will use to introduce latency into each of the classes;
# network emulation - add latency.
tc qdisc add dev $INTERFACE parent 1:11 handle 11:0 netem delay 20ms 5ms 25% \
distribution normal
tc qdisc add dev $INTERFACE parent 1:12 handle 12:0 netem delay 330ms 10ms 25% \
distribution normal
This attaches the emulator instances to their parent classes, with handles that match the parents Y value, for ease of tracing.The netem parameters break down as follows.
- delay 20ms - This is pretty self explanatory.
- 5ms - this is a jitter on the latency to give a bit of variation
- 25% - this indicates how much the variation in the latency of each packet will depend on its predecessor
- distribution normal - how the variation is distributed.
The netem module is described completely here: http://www.linuxfoundation.org/collaborate/workgroups/networking/netem
One thing that could be improved here is that we're adding all the latency on the outbound leg. Ideally we'd add 165 ms on the way there and on the way back for the SEAsia traffic (and 10ms for the EU traffic). To do that means applying latency to the outbound interfaces in both directions. In our case that would mean applying 165ms of latency to both of the vnet interfaces as well as the bond0.30 interface. However that is tricky to do simply as the virtual machine interface names may change as they get rebooted. Instead this way we end up with the same result for far less faffing about.
Now, all we need to do is add the filters that classify the packets into their classes
SEASIAIP=172.16.10.10
SEASIAPORT=9091
EUIP=172.16.10.10
EUPORT=9092
# filter packets into appropriate traffic classes.
tc filter add dev $INTERFACE protocol ip parent 1:0 prio 1 \
u32 match ip dst $SEASIAIP match ip dport $SEASIAPORT 0xffff flowid 1:12
tc filter add dev $INTERFACE protocol ip parent 1:0 prio 1 \
u32 match ip dst $EUIP match ip dport $EUPORT 0xffff flowid 1:11
The action is mainly in the second line of each command, where we match the target IP of the load balancer, and the ports we've setup. The flowid is the class handle for the appropriate classes. We don't need to set up a filter for the "normal" traffic, as it is covered by the "default 10" part of the original htb root qdisc declaration.
And that takes care of the outbound traffic shaping and latency.
We now need to handle inbound.
For this we use the special ingress qdisc. There's very little we can actually do with this qdisc. It has no classes, and all you can really do is to attach a filter to it. Usefully we can use the "police" key word to restrict (by packet dropping) the inbound flow. Its not exact, but its good enough for our purposes.
# inbound qdisc.
tc qdisc add dev $INTERFACE handle ffff: ingress
# attach a policer for "se asia" class.
tc filter add dev $INTERFACE protocol ip parent ffff: prio 1 \
u32 match ip src $SEASIAIP match ip sport $SEASIAPORT 0xffff \
police rate 128kbit burst 10k drop flowid :1
# attach a policer for "europe" traffic class.
tc filter add dev $INTERFACE protocol ip parent ffff: prio 1 \
u32 match ip src $EUIP match ip sport $EUPORT 0xffff \
police rate 512kbit burst 10k drop flowid :2
The handle ffff: is a synonym for the inbound traffic root. All you can do is attach ingress to it as shown. To be frank I've not dived into exactly how the burst keyword affects things. Essentially the above filter rule is the same as the one we used on the outbound side except we now match the source ports and IPs rather than the destination ports and IPs. Then rather than using the flowid argument we use police to instruct the kernel to drop packets from each of our loadbalancer ports if they exceed the stated rates.
Cleanup
To clean up after all of this, its sufficient to just remove the root and ingress qdiscs. Removing the top of the tree removes all the other configuration.
# remove any existing ingress qdisc.
tc qdisc del dev $INTERFACE ingress
# remove any existing egress qdiscs
tc qdisc del dev $INTERFACE root
Which cleans up all classes and filters.
Conclusion
There's an init script that encapsulates all of the above which can be downloaded from here.
[root@vm01 ~]# chkconfig latency on
[root@vm01 ~]# /etc/init.d/latency
Usage: /etc/init.d/latency {start|stop|restart|condrestart|status}
[root@vm01 ~]# /etc/init.d/latency start
[root@vm01 ~]# /etc/init.d/latency stop
[root@vm01 ~]# /etc/init.d/latency status
Active Queue Disciplines for bond0.10
Active Queueing Classes for bond0.10
Active Traffic Control Filters for bond0.10
[root@vm01 ~]#
And thats it.
This mainly suits a static configuration, as is the case with our load balancer and continuous integration environment. However for web development use, this approach lacks flexibility, particularly if you don't have root access. For our developers, I looked at ipdelay but eventally settled with charles which was adequate for our purposes.
HTH.
It’s widely recognised that adding people to a team often doesn’t result in increases in velocity and can even reduce the velocity of the team due to the extra communication overhead. Instead, teams look to increasing levels of automation and tools to improve their productivity without adding extra people.
here, which is nice because we like that at LMAX.
on the train in that morning and couldn't help but wonder if he's read the same thing - that was talking about how you should treat someone with suspicion if they suggest you can apply science and logic to anything that is… well, actually to anything other than actual science (by which he meant physics I believe). By saying it's a science you suggest it's predictable and follows rules. And if it was predictable, it wouldn't be hard. Big Data is anything other than predictable - the data could be corrupt (it's safest to assume some of it is); it's generated by people (and we all know how fickle they are); and if you're collecting lots of it practically randomly, then cause/effect/correlation are not guaranteed. So Martin suggests the term Data Journalist. There was a storm on Twitter which suggested a certain amount of disagreement with this term, but I like it. But then, I like to pretend I'm a writer and not a programmer.
. If you have a publication or example code, I will glance at those. If you've worked for a company I've worked for in the past, I'll look you up on 
