August 15, 2018
- pSConfig is a new component that replaces the old MeshConfig component. It is a template framework for defining the tasks to be run by hosts. The templates can be used to not only configure the tasks on each host through pScheduler but also to display their results through MaDDash. See the introduction to pSConfig here for more information.
- Some major benefits of pSConfig include:
- Increased Extensibility - pSConfig can take full advantage of the extensibility provided by pScheduler. It supports any test, archiver or context plug-in installed on your pScheduler server. Best of all, as more plug-ins get added, no software update is required for pSConfig!
- Consolidated Configuration Format - pSConfig defines all the tasks to be run in a JSON format it refers to as a template. There is no conversion between various file formats required as there was in MeshConfig. No longer do you need to edit an Apache Config-like format and convert it to JSON nor edit yet another format referred to a meshconfig-agent-tasks.conf for you local agent. Now there is only one format required no matter where the template lives.
- Enhanced Control - pSConfig gives increased control over what each agent does. This includes not only familiar operations like defining archives locally, but also allows for agent-defined transforms of any aspect of the templates after they are downloaded. See the documentation here for more information
- Better Diagnostics - pSConfig formats all logs of its agents in a parsable format. It also provides command-line utilities to help extract useful information from the logs. See the documentation of the pscheduler agent and maddash agent for examples.
- The pSConfig Web Administrator (PWA) is a separate component that provides a web interface for creating pSConfig templates. It is a revision of the former MeshConfig Administrator component that was introduced with version 4.0. As such, it can also generate the old MeshConfig format for tests using options supported by both formats. The Web Administrator component is still considered beta software as we would like more user feedback before removing the beta label. For more information see the documentation here.
- pSConfig uses a completely new JSON format to define templates. That being said, it is capable of reading files in the old MeshConfig format. This should give some flexibility in terms of when you migrate. For more information on migration see the document here.
- The Two-Way Active Management Protocol (TWAMP), described in RFC 5357, is an extension of the One-Way Active Management Protocol (OWAMP) that supplies two-way or round-trip measurements instead of unidirectional capabilities. TWAMP measurements are helpful because round-trip delays do not require host clock synchronization but can still provide additional accuracy compared to tools such as ping. It has the additional advantage that many routers are capable of running a vendor's implementation of a TWAMP server.
- A pscheduler plug-in is available named pscheduler-tool-twping for running tests of type latency or rtt. Periodic measurements can be setup via the pScheduler CLI, pSConfig Web Administrator or through pSConfig templates setup by hand by specifying tool twping
- Docker containers will be supported in 4.1 final release for tools and testpoint bundles. See the documentation here for more details.
- Added support for task re-writing to make on-the-fly changes to tasks at ingest to supplement the limit system.
- Added support for "single-ended" throughput tests that allow for testing to a server not running pScheduler.
- Improved the pscheduler troubleshoot command to check the maximum transmission unit (MTU) as well as enhanced options for checking on IPv4 vs IPv6.
- Added an http test and associated tools for measuring various HTTP performance metrics
- BWCTL is no longer included in new installations of perfSONAR but will be left untouched on existing installations. For a complete discussion of BWCTL support including timelines and options see the upgrade notes.
CentOS 6 Deprecation
- CentOS 6 is not supported for version 4.1. See the documentation here for a full discussion of CentOS 6 support and perfSONAR.
Debian 7 Deprecation
- Debian 7 is not supported for version 4.1. See the documentation here for a full discussion of Debian 7 support and perfSONAR.
Important Upgrade Notes
- See the documentation here for a full list of important notes about the upgrade process.
- After upgrading to perfSONAR 4.1, toolkit users will notice that tests not setup by the Toolkit Configuration UI (such as those from a remote mesh) will NOT appear on the configuration listin web page, whereas in previous versions they had appeared "grayed-out". This is not a bug but a side-effect of the approach taken to transition the Toolkit UI to pSconfig in the 4.1 timeframe. Ultimately the Toolkit UI needs a large overhaul on the backend which was simply out of scope for 4.1. Future versions will likely re-visit this work. In the meantime, your tests are still running even if not displayed in the UI and can be viewed by running psconfig pscheduler-tasks on the system.
Detailed Bug Fixes and Changes
- Removed BWCTL dependency from pscheduler-tools bundle
- Created package perfsonar-bwctl-compat as an option for installing BWCTL packages and associated pScheduler plug-ins.
- Replaced dependencies on packages perfsonar-meshconfig-agent and perfsonar-meshconfig-guiagent with perfsonar-psconfig-pscheduler and perfsonar-psconfig-maddash respectively.
- Added dependency on perfsonar-psconfig-publisher to perfsonar-centralmanagement bundle.
- perfsonar-centralmanagement now sets-up the esmond cleaner cron job on install
- This component has been removed from all new installations. See the upgrade notes.
- No changes this update
- Fixed duplication of paths in esmond URLs
- Added reverseurl GET option for specifying a separate archive to contain reverse results
- Added new displaysetsrc and displaysetdst options that allow optimization of querying results from multiple addresses
- Updated labels to better distinguish TWAMP, OWAMP and RTT results
- Stopped using deprecated sys_errlist
LS Cache Daemon
- No changes this update
LS Registration Daemon
- Removed support for registering deprecated or unused services: BWCTL, BWCTL MP, OWAMP MP, NDT, NPAD, Mesh, Phoebus and REDDnet
- Added support for pscheduler-tests field on interface records. This allows advertisement of types of tests supported by an individual interface
- Disabled test auto-discovery in esmond registration since these records generate substantial LS traffic but are unused.
- Added support for registering TWAMP service type
- Added new template variable maUrlReverse that maps to maURL if row and column are swapped
- Better error handling when asking for a grid that does not exist
- Fixed template variable substitution issue where similarly named variables could non-deterministically overwrite each other
- Fixed issue where undefined template variables where left as-is in output. They are now replaced with an empty string.
- Moved "the scheduler shutdown" message to warning level in logs since is usually caused by MaDDash reloading configuration. Also added message to indicate as much
- Updated default maddash-server.yaml file to included the added_by_psconfig option. This means that psconfig will automatically remove default structures and no longer requires a tedious manual process to do so.
- This component has been removed and replaced by pSConfig
- Renamed check_delay to check_rtt
- Fix uninitialized string error caused by version not being set
- This component is no longer included with new installs. It's functionality is replaced by pScheduler.
- Extended source code to support TWAMP
- Fixed issues where execute permissions not set on startup script
Perl Shared Libraries
- Added new package libperfsonar-psconfig-perl that provides libraries in support of pSConfig clients
- Fixed incorrect logging statements in various libraries that were calling warning instead of warn
- Removed unused references to MySQL in code
- Remove BWCTL references from code
- Consolidated toolkit_version and toolkit_rpm_version in Toolkit data service to just toolkit_version
- Added support for singled-ended throughput tests in libraries that support toolkit GUI
- Added TWAMP to service-related calls sued by the toolkit
- Updated pScheduler library to use new run-started field if available to determine if a background-multi task needs to be recreated
- Improved diagnostics when restoring configuration and data.
- Attempt to put old data back in place if database load fails
- Added MTU diagnostics to the troubleshooter.
- Troubleshoot command tests archiving
- Dropped required placeholder when importing tasks using the CLI and not changing parameters.
- Made debugging of REST API server more robust.
- Added initial support for command completion when using the Bourne Again SHell (BASH).
- Added IPv6 support to the simplestream test and simplestreamer tool.
- Added IPv6 support to the troubleshooter.
- Improved handling of decisions about how to contact the local host.
- Improved propagation of errors into CLI diagnostics.
- Made clock sync information available in the CLI when a survey shows a pair to have large differences.
- Added pscheduler-tool-twping
- Added pscheduler-test-http
- Added pscheduler-tool-psurl
- Added support for singled-ended parameter to throughput tests
- Moved BWCTL tools to pscheduler-bundle-extras package
- Cleaned-up owner of files in /var/run
- Added protocol field to pscheduler-test-rtt in support of TWAMP
- Updated WSGI settings to use 10 processes and restart every 2.5 million requests
- Improved diagnostics when no tools in common
- The reference field of a task is now passed to archiver plugins
- Updated esmond archiver to register displayset fields if present
- Backward compatibility checks no longer fail the task if they cannot connect to pScheduler or BWCTL. This prevents issues where special binding is needed and also move code closer to post-BWCTL behavior
- No longer require lead bind address to be on local machine during participants check
- Completely new component to replace MeshConfig. See documentation.
pSConfig Web Administrator
- Renamed from MeshConfig Administrator to pSConfig Web Administrator
- Added support for pSConfig templates
- Added a way to specify central MA(s) for a Config
- Added a way to archive to end hosts
- Changed authentication model - registration disabled by default. It can be enabled or users can be created via CLI
- Added support for twping tool (pSConfig-only)
- Fixed switch breaking hostgroup by switching static to dynamic host groups and then back
- Removed dependency of traceroute tests on a traceroute service in LS
- Fixed issues where certain host details were not saved properly
- Added ability to override the global MA in favor of one or more local MAs on a host
- Added ability to add addresses to host that are not in LS
- Added a way to override the MA URL for hosts
- Fixed the mesconfig importer
- No changes this update
- Updated pscheduler limits to use rewriter to enforce bandwidth limits
- Added TWAMP rules to firewall
- Removed references to BWCTL
- Moved logrotate rules to include file to /etc/logrotate.d/perfsonar-toolkit
- Updates toolkit web interface to save configuration in pSConfig format
- Added psconfig-pscheduler-agent to toolikit services listing and removed meshconfig-agent
- Removed meshconfig-agent from configdaemon
- Removed no longer used psb_to_esmond script
- Updated JSON service to to replace get_meshes action with get_templates and return pSConfig template URLs
- Updated configure_owamp_testports to produce pSConfig transform file
- Removed no longer relevant MySQL and web100 references
- Removed OPPD firewall rules
- Updated configure_firewall to check for existence of file it is editing
- Cleaned-up enable_auto_updates script to prevent unnecessary stderr output during RPM install
- Fixed issue where login message displayed incorrect version
- Added script to find tests in archive that are running BWCTL
- Added psconfig files to backup and restore scripts
- Added support for single-ended throughput tests in test configuration web interface
- Added TWAMP to default lsregistrationdaemon.conf file
- Better error message on main toolkit web page when unable to reach esmond
- Removed NTP servers that are no longer functional and added a few that are.
- Fixed error with saving communities when community list is unavailable
- Upgraded reverse_traceroute to version 7.7 from SLAC
- Reload firewalld when installing rules to make sure they take effect