- perfSONAR SOAP
- SSL Drown
- glibc and IPTables
- SSL Alternative Chains Certificate Forgery
- OpenSSL Vulnerability
- FREAK Vulnerability
- GHOST Vulnerability
- POODLE Vulnerability
- Shellshock Vulnerability
- NTP Amplification
httpd, July 2016
An important CVE was announced regarding the Apache web server package. Details from RedHat, Debian, and Ubuntu can be found below:
It does NOT appear that perfSONAR is directly affected by this issue. The vulnerability affects apache web servers running CGI scripts in certain languages. Perl (which is the language perfSONAR uses in its CGI scripts) does not appear to be affected. For languages that are affected, the attacker can set a proxy in the HTTP header of a request and force the host to forward information to remote server of the attackers choosing.
perfSONAR SOAP, July 2016
Updated perfSONAR packages were published to address security concerns. A special thanks to Luke Young for taking the time to find, document and provide a few patches for the items detailed below. The updates address the following issues:
- It was possible to generate a carefully crafted SOAP message that goes to the OPPD service that would allow an unauthenticated user to read arbitrary files from the filesystem as the 'perfsonar' user. This was done by exploiting a feature of LibXML that processes external entities. The ability to do so has since been disabled.
- The second issue allowed someone logged-in to the host via SSH as an unprivileged user to escalate to root privileges using a combination of the Toolkit’s ConfigManager and BWCTL’s posthook feature. ConfigManager did not actually need access to the BWCTL config file anymore, so access to this file (and thus the posthook feature) has been removed.
nss-util, March 2016
Red Hat has released a new CVE for nss-util.
While we do not believe that anything in perfSONAR is explicitly making use of this library, the “nss-util” package is included by default on CentOS installations. It is possible that software somewhere along the entire operating system stack is making use of it or could make use of it in the future. As such, to be safe we recommend upgrading to the latest version of the “nss-util” library posted in the CVE.
Machines with auto updates enabled should have already installed this update, but you should consider rebooting the machines to ensure any processes making use of it have the patches applied. Any servers without auto updates should be upgraded with “yum upgrade nss-util”.
SSL "DROWN", March 2016
Red Hat has released a new CVE today for a vulnerability called "DROWN":
Our read of this CVE is that it only impacts the SSLv2 protocol, which has been turned off by default in the toolkit for some time now. If you are running a current instance of the perfSONAR toolkit and have not made changes to the apache configuration, you should be fine.
If you have made changes to the apache configuration, you should review the CVE and make sure that you either disable SSLv2 or upgrade the openssl package, and either reboot the machine or restart any processes such as apache that use openssl to ensure all processes have the updates applied.
glibc and IPTables, February 2016
Version 184.108.40.206 of the perl-perfSONAR-Toolkit packages was released on Feburary 16th. This package contain an important fix to the default firewall configuration. For "clean" installs of version 3.5, the default firewall rules were not setup to properly reject packets. If you instance was upgraded to 3.5 from a previous version, the rules should be setup correctly. If you have automatic updates enabled you should get this update automatically, otherwise run yum update perl-perfSONAR_PS-Toolkit-security to get the change. No further action is required beyond the yum update.
Additionally, an important update to glibc was released that has pretty wide impact to software beyond just perfSONAR. See https://rhn.redhat.com/errata/RHSA-2016-0175.html for details. Updated packages for CentOS should be available shortly. Again, if you have automatic updates enabled you should get this update automatically when they are available, otherwise run yum update glibc to get the latest version.
SSL Alternative Chains Certificate Forgery, July 2015
On July 9th 2015 an OpenSSL vulnerability was announced:
We are contacting the list to let you know that we believe the the operating systems supported by perfSONAR are NOT affected. We do not believe you should not need to take any further action to protect your host from the vulnerability. For reference, the current list of supported perfSONAR operating systems are:
- CentOS 6
- Debian “wheezy"
- Ubuntu 12 and 14
All of the above ship with versions of OpenSSL that are not reported to be vulnerable. For details concerning the vulnerability as relates to specific operating systems see the following:
- CentOS/RedHat: https://access.redhat.com/security/cve/CVE-2015-1793
- Debian: https://security-tracker.debian.org/tracker/CVE-2015-1793
- Ubuntu: http://people.canonical.com/~ubuntu-security/cve/2015/CVE-2015-1793.html
SSL Logjam, May 2015
A new SSL vulnerability was announced on May 20th 2015, dubbed Logjam: https://weakdh.org/sysadmin.html. perfSONAR hosts of 3.4 lineage should NOT be vulnerable to this attack. The Apache configuration installed by the Toolkit disables the cipher suites in question by default. See the resource above for more details. There are no further actions required.
Interested parties may visit https://weakdh.org/sysadmin.html to check a host (perfSONAR or otherwise) if there is worry that a machine may be vulnerable.
Cassandra Security Announcement, April 2015
There was a CVE released April 1st 2015 for cassandra, which is used by the perfSONAR measurement archive software, esmond. You can find more information here: http://seclists.org/bugtraq/2015/Apr/0. If you are using the perfSONAR Toolkit distribution NO further action is required to protect your host. The summary of the issue is that by default cassandra listens on ports for JMX connections which allows remote execution of java code. Since inclusion of cassandra on the Toolkit last year, the Toolkit has a script that automatically turns these ports off in the cassandra configuration. Furthermore, the default iptables that the Toolkit installs block these ports had anything been listening on them.
If you are running a standalone esmond instance you need to update and restart cassandra. A few users have installed esmond separately from a Toolkit host as a central measurement archive or similar. If you are one of these users you need to run the following:
yum update cassandra20 /sbin/service cassandra restart
Note the restart of cassandra; auto-updates alone aren't enough to solve this issue.
OpenSSL Vulnerability, March 2015
RedHat has put up a page detailing how the recent OpenSSL updates affect RedHat 6 (which forms the basis for CentOS 6).
From our read through of the vulnerabilities, none seem particularly worrisome for Toolkit deployments. The problem bugs in this set of CVEs were for bugs that allowed for Denial-of-Service attacks (i.e. an OpenSSL service could be crashed). However, given the specifics of these DoS scenarios, it's unlikely that perfSONAR Toolkit deployments will be affected.
RedHat has yet to release updated OpenSSL RPMs. Once they release them, the CentOS team will have to vet the RPMs to their satisfaction, and then they will be released. There is no time table for the updates, but I would imagine they will be out within a day or so.
For those who have auto-update enabled, you will automatically get the updated RPMs, once they are released. However, you may need to restart Apache and OpenSSH so that they use the updated library. It may be easiest to reboot the host.
"FREAK" Vulnerability, March 2015
Some may be aware the recent announcement regarding the "FREAK” vulnerability, more information can be found here:
The perfSONAR development team has reviewed the report, and has determined that there are no risks that require intervention. The changes implemented after one of the prior vulnerabilities reduced the risk, e.g. disabling the export ciphers in the mod ssl configuration.
GHOST Vulnerability, January 2015
A vulnerability in glibc was announced in January 2015 going by the name of GHOST. You can find more info here:
The problem can be triggered remotely through the gethostbyname function. This problem is not specific to the perfSONAR Toolkit and has the potential to impact many common Linux services. The RedHat announcement can be found here:
Users may run 'yum update' to get the package or if you have auto-updates enabled you will get the package sometime in the next 24 hours. Note that even if auto-updates are enabled it is HIGHLY recommended to manually reboot the host. glibc is a rather ubiquitous library so rebooting is the best way to ensure there are no services running still using the old library.
SSLv3.0 "POODLE" Vulnerability, October 2014
On October 15th 2014 a vulnerability in the SSLv3 libraries was disclosed. A write up is available here:
And the full CVE from Redhat is here:
The best way to summarize the risk is that someone attempting a man in the middle could steal authorization headers from HTTP traffic, and gain entry to a server. This naturally impacts all servers implementing SSLv3 protocols, including the perfSONAR Toolkit. There are no reports of perfSONAR servers being victimized by this vulnerability, but the risk is a danger for any communication that uses the vulnerable libraries.
The CentOS project released openssl-1.0.1e-30 on October 16th 2014. The perfSONAR team has taken additional steps to modify the Apache configuration on the toolkit to disable use of SSLv3 within the 3.4 release of perfSONAR, and the 3.3.2-11 build of the LiveCD. A new package is available in our yum repository that addresses this. We are recommending that netinstall users:
- Check your logs to see if the package has been automatically downloaded yet. The package names are perl-perfSONAR_PS-Toolkit-3.4-29.pSPS and perl-perfSONAR_PS-Toolkit-SystemEnvironment-3.4-29.pSPS
- If you don't see it automatically downloaded, 'yum update' by hand.
Users of the 3.3 release series, that have not updated to 3.4 yet, can enable the perfSONAR Vault (http://software.internet2.edu/vault/3.3), or manually install packages using 'yum localinstall'.
Shellshock Vulnerability, September 2014
The perfSONAR Toolkit, and all systems that feature a vulnerable version of the bash application, are susceptible to the risks of the recently discovered shellshock vulnerability. More information from the upstream operating system vendor on this vulnerability can be found here and here.
There were a total of 6 vulnerabilities announced:
These link back to two specific errata pages that apply to the version of CentOS the perfSONAR toolkit is based on:
CVE 6271 was addressed with the 1st RPM. CVEs 7169, 7186, and 7187 were addressed with the 2nd RPM. The two unaccounted for CVEs (6277 and 6278) have this statement on their pages as of Oct 1st, 2014:
Technical details of this flaw are currently not public. Red Hat believes that changes introduced via updates RHSA-2014:1306, RHSA-2014:1311, and RHSA-2014:1312 that prevent Bash from defining new functions based on arbitrary environment variables sufficiently mitigate this issue. This statement will be updated once more details are available.
This does not rule out additional patches from the upstream providers, operators should remain cognizant of the risks and keep their system patched.
Packages to address the bash vulnerability are available from the upstream CentOS repositories. Additional mitigation from the perfSONAR project to prevent software from exposing the bash vulnerability has also been released. This mitigation involved searching for libraries and applications that invoked bash variables - our search found this to be limited to CGI scripts served through the Apache web server. Tools like OWAMP/BWCTL and others were not exposed to the bash risk. F5 Networks has developed a helpful graphic (source: https://f5.com/solutions/mitigation/mitigating-the-bash-shellshock-cve-2014-6271-and-cve-2014-7169-vulnerabilities) to illustrate the risk to systems via HTTP:
We recommend that perfSONAR operators take the following steps immediately:
- If you are running the netinstall (version 3.3 or 3.4), please run 'yum update' to pull in the latest packages, and reboot the system
- If you are running a LiveCD, please download the release prepared after these security events (version 3.3.2-10 or greater)
- The following additional steps, to limit external access to the GUI, may also be used for any of the use cases:
- Open the configuration file: /etc/httpd/conf.d/apache-toolkit_web_gui.conf
- Search for instances of these permissions:
Allow from all
- Modify the permissions to look like this (replacing the obvious fake address with a real one, multiple 'Allow' lines are permitted):
Allow from AAA.BBB.CCC.DDD/16
Deny from All
- restart httpd (sudo /etc/init.d/httpd restart) or reboot
For those that have not done so, please consider configuring yum to automatically update the system. Instructions on how to do this can be found here.
Please subscribe to the perfSONAR user or announcement mailing lists for additional information as it becomes available. Patches from the upstream vendors may become available as more is understood about the impact of this vulnerability.
JOWPING, July 2014
sudo rm -rf /opt/perfsonar_ps/toolkit/web/root/gui/jowping/
This will result in a broken link on the left sidebar, but removes the software and the risk. A future update to the 3.3.x series of the pS Performance Toolkit will remove JOWPING completely, and this tool was already earmarked to not be present on the upcoming 3.4 release due to lack of a maintainer.
The development team would like to thank John Parker from NOAA, who found this vulnerability through routine use of the skipfish tool (http://code.google.com/p/skipfish).
CACTI, April 2014
The perfSONAR project was made aware of two CVEs related to the cacti
software. More information is available by following these links:
Additionally, another issue was found with the Cacti configuration on all perfSONAR Toolkit nodes. The issue allows someone to access a settings web page unauthenticated from which they can change titles and other display values on the Cacti graphs. The extent of the harm that can be done appears to be limited to defacing the Cacti web pages, and unfortunately this was exploited in a few cases. Yesterday we posted manual work-arounds to correct this issue but today we have updates that will automatically apply the necessary fixes. The updates will 1) clear out any defaced fields and 2) require authentication to ANY cacti page, including just viewing the graphs. We recommend ALL users update as soon as possible by taking the following steps:
- Login to the command-line of your host and run 'yum update'
- Run ' /sbin/service httpd restart'
- Download and create a new CD from the relevant images found here: http://software.internet2.edu/pS-Performance_Toolkit/
Heartbleed, February 2014
On April 7th 2014, the perfSONAR project received notice from the CentOS of a CVE related to the OpenSSL package:
This alert was triggered by the recent exploit entitled "", which targets TLS activity. perfSONAR is recommending that all users run yum update to download the latest packages from the CentOS repositories. Additionally, we recommend re-generation of the host's certificate information after the packages are updated, as it was generated with the vulnerable implementation of OpenSSL. The following steps will accomplish this goal:
# build a new key /usr/bin/openssl genrsa > /etc/pki/tls/private/localhost.key # create a new self-signed cert cd /etc/pki/tls/certs make testcert # restart apache sudo /etc/init.d/httpd restart
If you have installed a customized certificate, refer to instructions specific to that process.
New versions (3.3.2-2) of the LiveCD and LiveUSB product have been generated, and are available .
NTP Amplification, January 2014
In early 2014 the perfSONAR project was made aware of a recently highlighted risk of NTP amplification (http://www.symantec.com/connect/blogs/hackers-spend-christmas-break-launching-large-scale-ntp-reflection-attacks)), and in some cases others may have already had a host that was used in an attack. A sample notice received may look something like this:
You are running a public NTP server that participated a very large-scale attack against a customer of ours today, generating UDP responses to spoofed requests with bogus timestamps that claimed to be from the attack target. Your server was particularly active in the attack, sending a significant percentage of the attack traffic we saw.
Please consider reconfiguring your NTP server in one of these ways:
- To only serve your customers and not respond to outside IP addresses. If your NTP server runs as a standalone installation, setting the service to ignore all queries would work well for this. With ntpd, that can be done with "restrict default ignore" in /etc/ntp.conf; other servers should have a similar configuration option. A firewall rule to block UDP to the public IP address on port 123 would also work for this.
- To rate-limit responses to individual source IP addresses
- To limit queries to TCP-only
- To ignore particularly unlikely queries, such as those representing dates far in the future or past
- To limit the size of allowed responses
Starting with the 3.3.2 release of the pS Performance Toolkit, configuration changes have been implemented to correct the risk of NTP amplification. We highly recommend updating hosts to this version to mitigate the risk of being a part of future protocol abuses. For those that have not updated to the latest version, it is possible to make manual modifications to the local /'etc/ntp.conf' file to accomplish the same goal:
# by default act only as a basic NTP client restrict -4 default nomodify nopeer noquery notrap restrict -6 default nomodify nopeer noquery notrap # allow NTP messages from the loopback address, useful for debugging restrict 127.0.0.1 restrict ::1
Additional restrict lines can be added to allow trusted subnets, e.g.:
restrict a.b.c.d mask 255.255.0.0 nomodify notrap nopeer
Any changes to NTP should be followed by a restart of the service. More information on protection for hosts (and routing devices) can be found here: