perfSONAR

Version 4.2.0

Release Date

August 19, 2019

Features

 

New pScheduler Disk-to-Disk Test

  • pScheduler supports a new test type called disk-to-disk that includes GridFTP and curl tools. For more information on using this new feature see the document here.

pSConfig Web Admin (PWA) RPMs

  • pSConfig Web Admin is now available as RPM packages. This provides an alternative to the existing docker containers. See the document here for details.

perfSONAR Ansible Roles

  • Supported Ansible roles are now available to help automate your perfSONAR deployment and customize it to meet your needs. See the document here for details.

pScheduler Task Priorities

  • pScheduler is now able to apply priority to tasks. The primary use case for this feature is to allow tests run from the command-line to more quickly get a time slot in cases where the schedule is full. See the document here for details.

MaDDash ServiceNow Integration

  • MaDDash is now able to create and manage records in ServiceNow. See the document here for details.

pSConfig Enhancements

  • The pSConfig MaDDash agent can now create dashboards for HTTP and DNS tests using default settings.
  • The psconfig validate command now does more thorough validation of pSConfig templates. This includes validating references and asking pScheduler to validate archivers, tests and contexts.
  • Added a new psconfig agenctl command to make interacting with the pSConfig pScheduler and MaDDash agent easier.

Lookup Service Enhancements

  • Added support for bulk renewals. This should significantly decrease messaging on clients.
  • Added support for signing records to provide a mechanism for validating who registered a particular record.

End of BWCTL pScheduler Support

  • This release marks the final phase of BWCTL deprecation in perfSONAR. This version removes remaining support for backward compatibility with BWCTL. Older systems with tools supporting BWCTL installed will receive upgrades that makes them inert. See docs.perfsonar.net for full details of this process.

Debian 8 and Ubuntu 14 Deprecation

  • Debian 8 and Ubuntu 14 are not supported for version 4.2. See the documentation here for a full discussion of Debian 8 and Ubuntu 14 support and perfSONAR.

Bug Fixes and Changes

Esmond

  • Added command delete_user_ip to remove user IP addresses.
  • Upgraded to Django 1.11.20
  • Fixed RPM to properly enable httpd
  • Updated python library to properly set timeouts
  • Added options to control SSL settings to python libraries

Graphs

  • Removed centos 6 backwards-compatibility hacks. Specifically, node-jsdom is no longer needed, along with its dependencies, which were significant.
  • Made the different graph areas more distinct from each other; make them brighter white, or add lines in between, or something, to divide up the space on the page more
  • Added a note explaining about negative latency to the graphs page, if there are negative latency values in the graph
  • Added a /etc/perfsonar/graphs.json file where you can set options such as enabling SSL host verification
  • Added an advanced settings panel with addition details about the subjects in the graph
  • Added ability to specify start or end date, instead of requiring both

LS Registration Daemon

  • Updated to work with SELinux enabled

MaDDash

  • Fixed database connection pool exhaustion that occurred when using notifications and caused dashboard to not display after a few hours
  • Added configurable privacy notice link to web interface
  • You can now add ?refresh=SECONDS where SECONDS is the refresh interval to a MaDDash url to pre-set the refresh time.
  • Disabled the HTTP OPTIONS method in the default Apache configuration
  • Upgraded c3p0 library to version 0.9.5.4
  • The command property can now be a map keyed on row and column
  • Cleaned-up default YAML file to remove references to outdated services and prefer https in URLs

Nagios

  • Added check_esmond command for generically checking against fields in esmond.

OWAMP

  • Updated to work with SELinux enabled
  • Updated to work with AppArmor
  • Added cron job to delete old session files
  • Reduced the number of reverse DNS lookups
  • Added SO_BINDTODEVICE option sockets to better support VRFs
  • Fixed segfault when entering a password
  • Fixed issue with TWAMP where it would not send above a particular rate

Perl Shared Libraries

  • Fixed inherit warning in Lookup Service client libraries

pScheduler

  • Added a new netreach test that determines whether or not a given network has hosts that are reachable, optionally excluding the gateway.
  • Migrated from python requests library to PycURL
  • Context information is now passed to archiver plug-ins
  • Fixed troubleshoot command to better handle a mix of IPv4 and IPv6
  • Improved the diagnostics of the simplestreamer command
  • Updated RabbitMQ archiver to keep connections open between archivings to improve performance
  • Fixed NUMA detection to consider a specified source when given
  • Adjusted behavior of the iperf3 tool to not use NUMA if the selected node cannot run a program
  • Fixed use of traceroute's -s command option on Debian
  • Added an --output switch to the task command for redirecting result(s) to file(s)
  • Fixed issue where the watch command added extra text to output when --format option was used
  • Improved output of messages when task is forbidden by limits
  • Added --skip-single-test option to troubleshoot command
  • Updated latency and latencybg JSON schema to be consistent withdraft 7
  • Fixed SSL warning that was sometimes displayed on Debian systems
  • Updated HTTP libs to accept HTTP success codes other that just 200
  • Made logging levels between components more consistent
  • Updated troubleshoot command to exercise the database through API calls
  • Handling errors found by troubleshooter better
  • Added a new header to the API that allows clients working on behalf of other hosts (e.g., pShooter) to securely ask that limits be enforced accordingly.
  • Added updated python2-pyasn1-modules packages in support of SNMP tests
  • Added output to the clock command that gives better information about error state.
  • Upgraded python-jsonschema to 3.0.1
  • Guarded troubleshoot command against return values of None by ntp_adjtime()
  • Upgraded jq library to 1.6
  • Added the linuxvrf context for changing VRFs. (Currently functional only on distributions running Linux kernel >= 4.0.)
  • Fixed issue with task rewriter failing on default limits file
  • Updated esmond archiver to store disk-to-disk results
  • Added diagnostics to the iperf3 and iperf tools on failure
  • Reduced lead time for first run of single-participant tasks. (Makes some ad-hoc testing happen sooner)
  • Improved debugging in the API service
  • Migrated from ipaddr to ipaddress library to make components consistent
  • Added Debian suppor to version command
  • Corrected a jq scripting problem where certain large integers were being rewritten as floats.
  • Added a headers item to HTTP archiver that passes provided headers
  • The pScheduler account is created as a system user (new installs only), login is disabled and the home directory is pointed at a temporary directory.
  • Added a local-subnet identifier to the limit system that identifies requesters on subnets connected to local interfaces.
  • Fixed issue when a run fails and there's an error present at the top of the JSON, result doesn't show it.
  • Added the ability to send failed runs to archivers.
  • Split the Background column in schedule plots to show background and background-multi runs separately.
  • Added parameters to the RTT test to allow fragmentation control.
  • Added a version command that lists the version number of all pScheduler packages.
  • Improved diagnostic messages when no time is available on the schedule for a run of a task.
  • Fixed issue where the filestring() function in the Python library overshoots the end of an empty string when looking for a leading at-symbol.
  • Added a new url-fetch limit that allows determination of pass/fail by querying an external server.
  • Adjusted web server configuration to eliminate failed requests due to process turnover
  • Adjusted Apache configs to use Require syntax
  • Allow RabbitMQ archiver's routing key to be determined using a jq script
  • Improved CPU utilization by reducing the work done for certain types of tasks changes
  • Added additional slop time to idle test to increase chance of success on slower/high-load hosts (such as when running the troubleshoot command)
  • Updated troubleshoot command to support alternate ports
  • Fixed tracepath test (and troubleshoot command MTU test) to work better in high latency situations
  • Fixed potential hostname lookup error in ping test

pSConfig

  • Fixed issue where pScheduler agent incorrectly set schema number when using contexts
  • Fixed an issue where the flip template variable incorrectly was set when using a mix of agent and no-agent hosts in a group
  • Fixed issue where MaDDash agent ignored the unidirectional setting of disjoint groups
  • Fixed issue where the archives defined in include files are being ignored
  • Fixed issue where pScheduler agent did not set long enough expiration for tasks over 24 hours
  • Added ability to cache templates when connectivity goes away. This prevents tasks from being deleted due to temporary connectivity issues.

pSConfig Web Administrator

  • Made pSConfig format the default output
  • Fixed issue where users could create non-unique usernames when using sca-auth module
  • Fixed issue with throughput tests would not schedule when ip-version set
  • Fixed format of duration, repeat interval and omit_interval
  • Improved handling of host expiration
  • Updated default url path to /pwa so can run more easily with other services
  • Made the port of the publisher URL configurable
  • Fixed issue where hosts were sometime duplicated in the configuration
  • Fixed issue where additional archives were only added to one but not all hosts
  • Made New test button more prominent
  • Added instructions for using the plus-sign buttons
  • Fixed issue where you are asked to abandon unsaved changes when none are made
  • Fixed automatic generation of display URL in pSConfig format
  • HTTPS is now the default for archive URLs
  • Upgraded bootstrap to 4.1.2
  • Improved date/time picker
  • Made Slip configurable
  • Changed default config file paths
  • Changed default PWA URLs
  • Simplified deployment
  • Simplifed SSL cert generation

Toolkit

  • Updated default Apache cipher settings to follow Mozilla's modern profile. This is stricter than the Intermediate profile that was previously followed.
  • Updated to work with SELinux enabled
  • Removed prompt to add a sudo user if a user in the group pssudo already exists.
  • Fixed issue where commas in lat and long values for host info got converted to an array in JSON service
  • Ping was renamed in GUI to "round trip" time
  • Removed link to external traceroute visualization interface that no longer functioned
  • Added force_toolkit_name option to web_admin.conf to force toolkit_name to be used in display instead of auto-decting addresses
  • Added disable_ls_lookups to web_admin.conf to disable lookups in cases where interface hangs
  • Numerous small clean-up items on GUI to make things clearer
  • Added TCP tuning settings to JSON service
  • The time range displayed on the graph matches the main page when you click the "Details" link for a graph on the main toolkit page
  • Cleaned-up whitespace on web page header
  • Added lock file to esmond data cleaner lockfile tto prevent two instances from running at same time

Miscellaneous

  • Updated group IDs in all packages to be system group IDs
  • Moved mod_interface_route from toolkit to testpoint bundle
  • Dropped -v /var/run from the suggested docker command in documentation since it does not appear to be needed

Raw changes

Updated components: