By default, the perfSONAR Toolkit installation follows the advice of ESnet’s “fasterdata” knowledge base for test and measurement hosts:
http://fasterdata.es.net/host-tuning/linux/test-measurement-host-tuning/
These tuning settings should be fine for most uses of perfSONAR, but there are a couple of cases where you might want to adjust these.
- 40G hosts (or even 2x10G bonded hosts)
- hosts where much of your testing is over paths with more than 100ms RTT
For such cases, you may want to increase the max buffer settings in /etc/sysctl.conf, for example:
#allow hand tuning up to 256MB buffers net.core.rmem_max = 268435456 net.core.wmem_max = 268435456 # allow auto-tuning up to 128MB buffers net.ipv4.tcp_rmem = 4096 87380 134217728 net.ipv4.tcp_wmem = 4096 65536 134217728
Note that any time you upgrade the perfSONAR Toolkit, you should make sure your modified settings are still in place.
In case you didn’t use toolkit installation an optional perfsonar-toolkit-sysctl package is available that configures the system with perfSONAR settings. For more information on optional script installation see Installation.
CentOS7 and Debian now support “Fair Queuing” and kernel-level packet pacing. Pacing can be done on a per socket basis, or you can set a global maximum rate for each NIC on the host.
This can be very helpful for cases where you want to ensure that your perfSONAR host never uses all of your bandwidth. For example, to ensure your 10G host never exceeds 7Gbps, run this command:
tc qdisc add dev $ETH root fq maxrate 7gbit
Where $ETH is the name of your host NIC. Add that command to /etc/rc.local to run at system boot time.
You can also configure your test mesh to tell iperf3 to reduce it’s bandwidth as well. Details on how to do that are coming soon.
More details on host pacing are available at: https://fasterdata.es.net/host-tuning/packet-pacing/