perfSONAR

Low-cost Hardware

BEFORE YOU GO FURTHER READ THIS

There are numerous hardware platforms that have emerged that are an attractive option for use in network performance measurement. The perfSONAR collaboration does not recommend, nor support, the use of perfSONAR on low-end, ARM-based hardware such as the Raspberry PI. It has been shown that it is extremely difficult to distinguish network issues from host issues on these devices. In particular, we do not recommend these devices for testing throughput with the BWCTL, NDT, or other related tools. Use of latency based tools (Ping, OWAMP) is possible provided that an accurate clock source (direcly connected) is available.

As with evaluation of any hardware platform, early adopters should be sure to note how the architecture of the machine will impact "end to end" performance. E.g. the measurement tools are only as good as the host that initiates the test, and it is probable that a host without sufficient system resources can report a lower/suspect network performance number due to:

  • Tests being initiated to a remote resource with a larger or smaller network interface speed (e.g. a 1G LIVA node will show performance abnormalities when testing to a 10G server due to the behavior of the NIC hardware, and TCP)
  • The CPU's clock speed not being sufficient to test a single stream of TCP or UDP
  • Lack of system memory to support testing
  • Contention for the system bus, which manages the interactions between CPU, Memory, and NIC
  • Inability to report the differences between 'local' congestion (e.g. within the host) versus remote congestion on the network

When evaluating any hardware, be sure to consider the platform limitations, as they will have an eventual impact on the measurements that perfSONAR collects. 

Low Cost perfSONAR Nodes

There are several efforts underway in the performance measurement community to explore how well small form factor computers (e.g. single-board or embedded  types) can perform at routine measurement and monitoring tasks.  Many of these devices use the ARM RISC architecture, others use more traditional CISC chips such as the x86.  

The perfSONAR Endpoint Node Project was formed to explore a standard perfSONAR configuration for hardware and software. 

Porting perfSONAR to these alternative devices is an emerging area of community interest, and normally features some of the following concerns:

  • Cross compiling binaries for use on alternative architectures (e.g. x86 to ARM)
  • Evaluating performance of hardware versus advertised expectations (e.g. ensuring the CPU, memory, system bus, and network card can support tools such as OWAMP and BWCTL)
  • Assessing stability over time, particularly under load or via external mechanisms such as the ability to maintain system clock performance
  • Reducing perfSONAR toolkit functionality to a smaller set to fit the use case of a measurement beacon versus a fully functional platform
  • Evaluating deployment strategies for distributed deployments

This wiki page outlines known projects from this area of interest, and summarize some of the progress that has been made. 

  • Raspberry Pi
  • BeagleBoard/BeagleBone
  • Cubox
  • Intel NUC
  • MiraBox
  • LIVA

The vision for much of the work described below is to build network assessment and NOC alerting tools using 'swarms' of the small nodes deployed on a campus.  Additional goals include self-help testing functionality in limited circumstances.  

Documentation describing installation procedures for low cost nodes can be found here: http://docs.perfsonar.net/low_cost_nodes.html

Known Problems

As the community experiments with these nodes, a number of problems have been discovered.  Since the efforts are ongoing, please note that things may change rapidly in this environment with regards to hardware and software.  The best location to learn of problems (and to ask about new observations) is the perfSONAR user mailing list and archive

Ubuntu Versions

It has been suggested that older versions of Ubuntu desktop (12) do better than newer (14).  It is not known at this time what causes this differnce in performance. 

Ubuntu Power Management

Ubuntu-based operating systems also enable an "ondemand" init script by default which puts the CPU in "powersave" mode.   In testing on certain platforms, it has been observed this can reduce TCP throughput by between 5% and 10%, and induce UDP packet loss.  You can disable the script with the following which will leave the CPU in performance mode:

update-rc.d -f ondemand remove

IPTables Performance

Security features, in particular those provided by the "perfsonar-toolkit-security" package (e.g. use of IPTables) can cause problems on small machines.  This is due to a CPU bottleneck in many cases - applying network filtering at the same time as a network test will cause a reduction of throughput in most cases.  Disabling this package via the following steps has been known to increase performance, but does decrease security posturing:

apt-get remove perfsonar-toolkit-security
apt-get autoremove
cd /etc/iptables/
rm rules.v4 rules.v6
reboot

IO Performance

Some users have also reported abnormalities on their small nodes related to I/O activity (e.g. iostat reports long w_await times - sometimes measured in multiple seconds).  These coincide with intervals of testing, in particular related to OWAMP.  

Deeper investigation found that there is too much I/O going on: syslogd and systemd-journald processing syslog messages from "owampd, bwctld, and powstream” in “/var/log/messages”, sometimes up to 30-40 syslog messages per second depending on the testing configuration of a host.  Given that small nodes are based on flash memory, changes should be made to ensure a more balanced approach to logging:

  • Do journaling on memory by editing “/etc/systemd/journald.conf”.
    • Make option "Storage=volatile” instead of the default “Storage=auto”. Make sure to limit the maximum usage of memory for journaling. You can do this by fiddling with “RuntimeKeepFree” and “RuntimeMaxUse” options.
    • Don’t *restart* the journaling service (i.e., don’t do “systemctl restart systemd-journald”). Do an *OS reboot* instead.
  • Turn down owampd syslog level.
    • For perfSONAR before 3.5rc2, the only way is to comment out the “verbose” option in “/etc/owampd/owampd.conf” (so that it defaults to non-verbose. For perfSONAR 3.5rc2, I believe you can set the logging level in the configuration.
    • Restart owampd service

Headless Operation

Some versions of "Desktop" Linux (e.g. Ubuntu Desktop) will not allow the operating system to boot unless a monitor is conneccted to the system.  This can make "headless" operation challenging.  Users of LIVA hardware will experience this via a notice on their web site: http://www.ecs.com.tw/ECSWebSite/Product/Product_LIVA_FAQ.aspx?DetailID=1593&LanID=0

There are several known solutions, the first is to aquire a "VGA Dummy" plug that tricks the system into thinking there is a monitor attached: http://acquisitionsyndrome.com/2014/07/headless-ecs-liva/

Other homeade solutions revolve around inserting electronic resitence into the port (similar to how the dummy port would function):

Note that the perfSONAR project does not assume any lability for damage for these approaches, and suggersts using a linux operating system that will support headless operation.