Released April 17, 2017



  • pScheduler is a new component responsible for managing the execution of network measurements, or more generally tasks, in perfSONAR. It replaces the BWCTL and perfsonar-regulartesting components of previous version. When you want to run a network measurement in perfSONAR you ask a pScheduler server, either through the pscheduler command-line client or through the pScheduler API. See the introduction to pScheduler here for more information.
  • Some major benefits of pScheduler include:
    • Increased Extensibility - pScheduler provides a plug-in architecture for writing new tests, tools and archivers
    • More Diagnostics - pScheduler maintains a database of its schedule. This schedule can be queried to see what has run, is running and will run. It also lets you see errors and output from tools when things do not work as expected.
    • Greater Access Control - pScheduler has a revised limits system allowing you to define limits on more parameters and leaves room for further extension in the future.
  • pScheduler is a completely new codebase and protocol. It does NOT speak the BWCTL protocol natively but it does provide a backward compatibility strategy with BWCTL. In general you should not have to take any extra steps to continue testing to sites running BWCTL. Likewise, tests running to you should continue to operate normally as well. For more information see the document here.

New Graphs

  • This release contains new graphs based on the react-timeseries-charts library. The new graphs now stack different metrics for easier comparison without overloading axes. They also include a number of performance improvements over previous iterations of the graphs
  • The graphs can now be found at the URL http://your-host/perfsonar-graphs and visting http://your-host/perfsonar-graphs/graphWidget.cgi should redirect to the new location.
  • For more information on using the graphs see the document here

CentOS 7 Support

  • perfSONAR now officially supports CentOS 7. The Toolkit ISOs are only available as CentOS 7. We will continue to support CentOS6 RPMs for this release, so existing CentOS 6 users will be able to auto-update.
  • CentOS 7 drops support for i386/i686 architectures and as a result there are only x86_64 versions of the CentOS 7 perfSONAR packages available

Debian Toolkit Support

  • Debian 8 Jessie, Debian 7 Wheezy and Ubuntu 14 Trusty now support the perfsonar-toolkit package installed as a bundle
  • perfSONAR support of Ubuntu 12 has been discontinued since the operating system has reached its end-of-life

MaDDash 2.0

  • MaDDash has been integrated with the MaDAlert project developed by the University of Michigan as a subproject of the PunDit project. This means MaDDash now supports customizable patterns that can be used to highlight common problems and generate alerts based on those.
  • MaDDash now natively supports email notifications that are generated based off of the alerts above. You can define the notifications you want sent in your maddash.yaml file.
  • A new more color-blind friendly palette is now the default for MaDDash. A visitor may also change the colors they see by using a new pull-down in the settings menu. This pulldown will save the choice in a cookie so the user sees the same colors on future visits in the same browser.

Expanded Role of MeshConfig Agent

  • In previous releases perfSONAR contained a regular-testing component for executing locally defined tests and a meshconfig-agent for reading remote meshes. In 4.0, these roles have been combined in the new perfsonar-meshconfig-agent daemon to streamline test management. The new daemon will monitor both remote meshes and a local configuration file for changes, and submit tasks to pScheduler as needed.
  • Restarts are no longer required when new tests are added and there is no longer a nightly cron job required to detect changes to remote meshes. Local changes will automatically be detected within a few seconds and remote mesh changes will be picked-up within an hour by default.
  • The /etc/perfsonar/regulartesting.conf file has been moved to /etc/perfsonar/meshconfig-agent-tasks.conf file as part of these changes. For upgraders, this file should be moved to its new location automatically and any previously-defined tests or archives should be maintained in the transition.

MeshConfig Admin GUI Beta

  • An all-new GUI for developing mesh configurations has been developed which doesn’t require hand-editing of the configuration file. The new system produces configuration specific to each host. It is currently still considered beta, but you can find information about testing it here.

NDT and Web100 No Longer Included by Default

  • Starting with this release, new installs on all supported operating systems will not get NDT and the Web100 kernel by default. We will continue to provide Web100 kernels for perfSONAR 3.5.1 while it is still in its support window (6 months after the final release of 4.0). At that time we will discontinue maintenance of the web100 kernel and since NDT requires web100, that will not be supported either. We will be providing more communication on this decision and timeline through our user lists and on the web site.

Important Upgrade Notes

  • All your data and tests should be preserved during the upgrade of an existing 3.5 host. If for whatever reason this does not happen, please contact our user list if you need assistance.
  • Debian/Ubuntu users of perfSONAR 3.5.1 will not get this update automatically. See our Debian upgrade guide for information on updating your host
  • For CentOS users running auto-updates, it may take up to 72 hours for your system to get the release depending on when mirrors sync and yum-cron runs.
  • You may want to reboot your host after update. While not strictly required, this may help with occasional issues we have see with httpd processes not restarting properly and similar
  • If you were pointing at the staging repository on a production host, you should remove it now. You can do so by running yum remove Internet2-repo-staging
  • perfSONAR 4.0 requires more CPU than previous versions as a result of some of the increased functionality. See our hardware requirements page for more information.
  • We HIGHLY recommend you start using the new pscheduler command instead of the bwctl for all command-line tests since doing so has the potential to create scheduling conflicts. See the page here for a guide on how to convert BWCTL commands to pScheduler.
  • Web100 kernels are not provided for CentOS 7. The plan is to continue to provide Web100 kernels for CentOS 6 until October 17, 2017.
  • NDT and NPAD are no longer installed for new installs since they require Web100. They will NOT be removed from existing systems automatically, but require a web100 kernel to operate so will either require an outdated kernel or cease to function after October 17, 2017
  • The perfsonar-regulartesting package is gone ion perfSONAR 4.0. This means your /etc/perfsonar/regulartesting.conf file has been moved to /etc/perfsonar/meshconfig-agent-tasks.conf. It also means you no longer have a perfsonar-regulartesting daemon running
  • The /usr/lib/perfsonar/bin/generate_configuration script and associated cron are gone and now handled by the new perfsonar-meshconfig-agent daemon
  • The /usr/lib/perfsonar/bin/generate_gui_configuration script and associated cron are gone and now handled by the new perfsonar-meshconfig-guiagent daemon. This daemon will now create traceroute and ping dashboards by default if you have the tests defined.

Detailed Bug Fixes and Changes

Bundles and Shared Libraries

  • Added pScheduler to bundles
  • Removed NDT from bundles
  • Packaging support for CentOS 7
  • Added pScheduler Perl client
  • Install a default limits file as part of testpoint bundle
  • Fixed memory rounding errors in memory detection libraries
  • Detecting OpenStack, QEMU and BHYVE virtualization technologies and marking as VM in host info used by toolkit web page and lsregistrationdaemon
  • Fixed issue where toolkit page would not load because calls made in wrong order
  • perfsonar-core package now responsible for adding localhost MA to meshconfig-agent-tasks.conf instead of having set for all bundles
  • Added default ulimits file on CentOS 6 that increases allowed process count for httpd and pcheduler from 1024 to 4096. This matches the system default for CentOS 7.


  • CentOS 7 packaging and systemd scripts


  • On CentOS 6, esmond now runs under Apache 2.4 from CentOS SCL with a reverse proxy from the standard apache to the custom setup. This removes the need for a custom python27-mod_wsgi module that was the source of numerous headaches.
  • Esmond now requires the selection of an esmond-database package. Thiis can either be esmond-database-postgresql (what was supported in previous versions) or esmond-database-postgresql95 which support postgresql 9.5 as provided by SCL. The latter will be used on toolkit systems also running pScheduler. Any existing metadata will be automatically migrated.
  • Added support for the pscheduler-raw event type
  • Fixed IP authentication when behind a web proxy
  • Added support for pscheduler-run-href event type


  • CentOS 7 packaging
  • Updates to support modern autoconf
  • Fix that defines SYSLOG_NAMES for only one source file

LS Cache Daemon

  • CentOS 7 packaging and systemd scripts

LS Registration Daemon

  • Support for pScheduler registration
  • CentOS 7 packaging and systemd scripts
  • Fixed issue where web service (e.g. oppd, ma, pscheduler) based records were losing track of their parent host record
  • Added ability to track whether perfSONAR was installed with NeInstall or FullInstall ISO


  • CentOS 7 packaging
  • Added support for mapped parameters
  • Updated many library dependencies to latest version, including requiring java 1.7
  • Added packaging so new perfSONAR graphs get used in place of old graphs
  • Added support for detecting most configuration changes without needing to restart
  • Added newline to the end of default configuration file to make meshconfig-guiagent happy
  • Defaulting to /usr/lib64 for nagios checks in default config


  • CentOS 7 packaging and systemd scripts
  • Added perfsonar-meshconfig-agent startup script that detects changes faster and consolidates old regulartesting functionality
  • pScheduler support
  • Added support for all missing tool options that are supported by pscheduler but were not yet in meshconfig
  • Fixed issue with ping packet_interval where it should have been read as floating point not integer
  • Removed all dependencies on the Toolkit ConfigDaemon
  • GUIAgent is now a daemon instead of a cron job. It reads the mesh once an hour and detects local config file changes to meshconfig-guiagent.conf without need for restart. It also no longer restarts maddash since a restart is not required to pick-up config changes.
  • Added support for the address_map option that allows a host to communicate with different hosts using different addresses even when in the same mesh. This is especially useful in layer2 causes where a host communicates with each remote endpoint using a different /30 or similar.
  • Related to the previous bullet, added the exclude_unmapped option which makes it easier to create “sparse” meshes where only a few of the hosts test to each other.
  • Fixed logging permission issues for meshconfig-guiagent
  • Fixed issue where perl considered maddash.yaml filename tainted
  • Updated all test parameters to match pscheduler
  • Properly set reverse URL in disjoint case with combined graphs
  • Added dependency on perl-Mo
  • Updated default path to nagios checks
  • Improved detection of changes to MA authentication settings
  • Removed explicit setting of default summaries
  • Added support for setting custom reference parameters
  • Added created_by field in meshconfig-agent-tasks.conf so clear where test came from
  • Fixed uninitialized concatenation warning when no mesh description provided
  • Fixed issue where did not fallback properly to IPv4 mesh URL when IPv6 failed
  • Updated meshconfig-guiagent defaults to more sane values
  • Updated agent to define measurement archive only for tests it creates, not all tests in file
  • Now support defining multiple MAs of same type
  • Can now set configure_archives at mesh level in meshconfig-agent.conf so only individual meshes have their archives configured
  • Added support for simplestream tests and dashboards. Should only be used by developers for debugging.
  • Added support for ping and traceroute dashboards to guiagent by default
  • Fixed issue where client was not properly timing-out
  • Fixed issue where agent was not setting correct start and end times on new tasks
  • Made retry policy a configurable parameter for measurement_archive blocks
  • Removed localhost MAs from default config file. Now will get added by perfsonar-core bundle.
  • Added support for bind_address,lead_bind_address and pscheduler_address options that can be useful in certain setups with private default routes


  • The Open perfSONAR Perl Daemon (OPPD) is deprecated now. The OPPD packages are still installed by a regular perfsonar-toolkit installation, but the service is not started.


  • CentOS 7 packaging and systemd scripts
  • Don’t call setpgid() if we are already the process group leader. This allows owampd -Z to run under OSX launchd with a plist file.
  • Fix invalid struct timeval values that caused errors on Mac OS X
  • Removed unnecessary logging

Regular Testing

  • This component no longer exists. The perfsonar-meshconfig-agent package obsoletes it and will migrate its configuration file to /etc/perfsonar/meshconfig-agent-tasks.conf. The tasks contained within will then be submitted by the meshconfig-agent to pScheduler.


  • CentOS 7 packaging and systemd scripts
  • Updated service listing to show pScheduler and meshconfig-agent. Also sorted them
  • Firewalld now used on CentOS 7
  • Added additional host details to main page
  • Added support for interface aliases in interface selection screen
  • List interfaces in UNKNOWN state such as loopback interfaces in Toolkit web pages’s interface selection menu
  • Added support for throughput “zero_copy” parameter
  • Update migration scripts for CentOS 6 to 7
  • Fixed broken links to reverse traceroute and reverse ping tools
  • Fixed issue where lo interface was selected as primary
  • Changed cassandra process detection to use process name since cassandra package has inconsistent use of pid file on CentOS 7
  • Added htpasswd support to manage_users script
  • Fixed broken links on toolkit homepage
  • Added pscheduler support to backup scripts
  • Default limits file will also allow 50m UDP tests
  • Removed code that was altering wheel group since perfsonar no longer uses that group
  • Fixed bug preventing use of subinterfaces in mod_interface_route script
  • The configure_sysctl script now detects OS, interface speed, and maximum MTU to determine how best to tune host
  • Removed NDT checks from configure_nic_parameters
  • Updated labels on web interface containing the phrase “RPM” to be less OS-specific
  • Changed owamp-server regular restart to once a week
  • Created symlink in /var/log/perfsonar to /var/log/pscheduler
  • Created symlink to web admin configuration files in /etc/perfsonar/toolkit/web
  • Fixed bug that prevented users from getting added by user management script
  • Made traceroute the default over tracepath in GUI
  • Fixed recursive symlinks created by .spec file
  • Moved user prompts out of .bashrc and into /etc/profile.d
  • Fixed call to tracepath in reverse_traceroute.cgi script

Raw changes

Updated components: