perfSONAR

Version 3.3 FAQ

Mailing List Archives

Version 3.3 Frequently Asked Questions


  1. Can the OWAMP test port ranges be added via the GUI?
  2. [BWCTL and OWAMP] Graphs fail to open on my browser. I am using IE 8/9. Is this a problem with my toolkit or my browser?
  3. My host is only connected on R&E routed networks, and I have no ability to access commercial web sites that host packages for the perfSONAR Toolkit. Yum is failing as a result, what steps can I take to correct this?
  4. My (OWAMP|BWCTL) measurements have stopped, and I notice mysql errors in the logs. What should I do?
  5. I have a limited amount of disk space in /var/log, and I have noticed that perfsonarbuoy_ma is writing a lot of INFO entries in the /var/log/perfsonar/perfsonarbuoy_ma.log which is filling up this disk space rapidly. Is there a way to suppress all INFO level logs?
  6. How do I migrate an existing LiveCD installation to a NetInstall installation?
  7. As of RHEL-5.10, Redhat is no longer going to issue security updates for MySQL-5.0. They advise upgrading to MySQL-5.5, with 5.1 as a required intermediate step. How can I do that on the perfSONAR Toolkit?
  8. Is there a Javascript library that can be used to fetch perfSONAR data?
  9. My perfSONARBOUY MA is listed in the "Running" state, but returns no data. How can I debug this?
  10. Why is my LiveCD not running the default perfSONAR iptables configuration?
  11. Can I add my own rules to iptables/ip6tables?
  12. After updating to perfSONAR Toolkit v3.3.2, maddash stopped working. What should I do?
  13. I am not able to start the perfSONARBUOY Web Service, and there are MYSQL errors. What should I do?
  14. My Cacti instance was vandalized/could be vandalized due to guest access being enabled. What can I do to stop this?

Q:Can the OWAMP test port ranges be added via the GUI? A:

The pSPT development team recommends not limiting measurement tools to certain ports; this action may cause unexpected or unpredictable behavior. If selecting a port range is still required, please see this section for instructions on how to alter the testing port range.


Q:[BWCTL and OWAMP] Graphs fail to open on my browser. I am using IE 8/9. Is this a problem with my toolkit or my browser? A:

If you are using the most recent version of the perfSONAR Toolkit, the graphing package has been converted to dygraphs, which is HTML5 and JavaScript based. Not all browsers (namely recent versions of IE) fully support HTML5, and as such there will be compatibility issues outside of our control and support structure.

A write up on known problems and possible solutions here:

http://dygraphs.com/#ie

The perfSONAR Toolkit project suggests using other browsers as a short term fix (e.g. Chrome, Firefox), as these support the required technology just fine. If you require the use of IE (8, 9, previous versions) the above page has suggestions on organizations using the perfSONAR Toolkit can modify HTML to fool the browser (Web Content is located in '/opt/perfsonar_ps/toolkit/web' and '/opt/perfsonar_ps/serviceTest' directories). Our projet has not tested these items to date; caveat emptor.


Q:My host is only connected on R&E routed networks, and I have no ability to access commercial web sites that host packages for the perfSONAR Toolkit. Yum is failing as a result, what steps can I take to correct this? A:

Version 3.2.x of the perfSONAR Toolkit (CentOS 5 based) requires access to 4 major repositories:

  • CentOS
  • CentOS updates
  • EPEL
  • Internet2

Of these 4, only the last one is consistantly available via both R&E and commercial routing. The first 3 are available via both methods, but the defaults on CentOS will attempt to fetch a mirror list that might use a commercially routed server only; resulting in timeouts and other unexpected behavior.

To edit the existing repository choices, you need to swap out the mirrorlist with a baseurl in the relevant .repo files:

  • For CentOS-Base (/etc/yum.repos.d/CentOS-Base.repo):
    • set the 'updates' baseurl to one in http://software.internet2.edu/rpms/mirrors-Toolkit-CentOS-Updates
    • For 'base', you can take that same mirror for updates, and replace "/updates/i386" in the URL with "/os/i386" (e.g. http://linux.mirrors.es.net/centos/5/updates/i386/ -> http://linux.mirrors.es.net/centos/5/os/i386/ ). You'll probably need to disable the 'extras' with a "enabled=0" in it.
  • For EPEL (/etc/yum.repos.d/epel.repo), set the 'epel' baseurl to one in http://software.internet2.edu/rpms/mirrors-Toolkit-EPEL

Examples of what these files will look like after modification appear below:

  • For CentOS (/etc/yum.repos.d/CentOS-Base.repo):
  • [base]
    name=CentOS-5 - Base
    baseurl=http://linux.mirrors.es.net/centos/5/os/i386
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
    
    #released updates 
    [updates]
    name=CentOS-5 - Updates
    baseurl=http://linux.mirrors.es.net/centos/5/updates/i386
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-5
    

and

  • For EPEL (/etc/yum.repos.d/epel.repo):
  • [epel]
    name=Extra Packages for Enterprise Linux 5 - $basearch
    baseurl=http://linux.mirrors.es.net/fedora-epel/5/i386
    failovermethod=priority
    enabled=1
    gpgcheck=1
    gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL
    

Q:My (OWAMP|BWCTL) measurements have stopped, and I notice mysql errors in the logs. What should I do? A:

If you see the following types of messages in the logs:

2013/06/06 17:00:35 (4864) ERROR> SQL.pm:218 perfSONAR_PS::DB::SQL::query - Query error "DBD::mysql::st execute failed: Table 
'./owamp/20130605_DATA' is marked as crashed and should be repaired at 
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/../lib/perfSONAR_PS/DB/SQL.pm line 213.
" on statement "(select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130530_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130531_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130601_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130603_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130604_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130605_DATA) union (select distinct send_id, recv_id, tspec_id, case when tspec_id=1686456867 then '0' end as tid from 
20130606_DATA) order by send_id, recv_id, tspec_id;".
2013/06/06 17:00:35 (4864) FATAL> perfSONARBUOY.pm:1303 perfSONAR_PS::Services::MA::perfSONARBUOY::createStorage - Query error, 
aborting.
2013/06/06 17:00:35 (4864) FATAL> perfSONARBUOY.pm:304 perfSONAR_PS::Services::MA::perfSONARBUOY::init - Couldn't load the store 
file - service cannot start
2013/06/06 17:00:35 (4864) ERROR> daemon.pl:380 main:: - Failed to initialize module perfSONAR_PS::Services::MA::perfSONARBUOY on 
8085:/perfSONAR_PS/services/pSB

It is likely that your mysql instance has crashed and will need to be repaired. Follow these steps. Note that this assumes repairing both the latency and bandwidth functionalities at the same time - if you are running only one of the services, you can omit the comamnds for the one you are not using.

  1. Stop essential services:
    sudo /etc/init.d/crond stop
    sudo /etc/init.d/owampd stop;sudo /etc/init.d/perfsonarbuoy_owp_master stop;sudo /etc/init.d/perfsonarbuoy_owp_collector stop
    sudo /etc/init.d/bwctld stop;sudo /etc/init.d/perfsonarbuoy_bw_master stop;sudo /etc/init.d/perfsonarbuoy_bw_collector stop
    sudo /etc/init.d/perfsonarbuoy_ma stop
    
  2. Kill anything left over (its always good to check to see if things wedged):
    ps auxw | grep powstream
    sudo kill -9 $PID
    ps auxw | grep owampd
    sudo kill -9 $PID
    ps auxw | grep powmaster
    sudo kill -9 $PID
    ps auxw | grep powcollector
    sudo kill -9 $PID
    ps auxw | grep bwctl
    sudo kill -9 $PID
    ps auxw | grep bwctld
    sudo kill -9 $PID
    ps auxw | grep bwmaster
    sudo kill -9 $PID
    ps auxw | grep bwcollector
    sudo kill -9 $PID
    
  3. OPTIONAL, note that if you do this, you nuke results that may not be in the DB yet:
    sudo rm -frd /var/lib/perfsonar/perfsonarbuoy_ma/owamp/*
    sudo rm -frd /var/lib/perfsonar/perfsonarbuoy_ma/bwctl/*
    sudo rm -frd /var/lib/owamp/catalog/*
    sudo rm -frd /var/lib/owamp/hierarchy/root/regular/*
    
  4. Stop the database:
    sudo /etc/init.d/mysqld stop
    
  5. Run a rebuild:
    sudo su -
    sudo myisamchk -r /var/lib/mysql/owamp/*MYI
    sudo myisamchk -r /var/lib/mysql/bwctl/*MYI
    
  6. restart the database:
    sudo /etc/init.d/mysqld start
    
  7. Backup old data (usually a good idea), in this case it saves 3 months:
    sudo mkdir -p /var/log/BACKUP/bwctl
    sudo /opt/perfsonar_ps/perfsonarbuoy_ma/bin/clean_pSB_db.pl --mysqldump-opts="--skip-lock-tables" --dbtype=bwctl --maxmonths=3 
    --owmesh-dir=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/ --dumpdir=/var/log/BACKUP/bwctl
    
    sudo mkdir -p /var/log/BACKUP/owamp
    sudo /opt/perfsonar_ps/perfsonarbuoy_ma/bin/clean_pSB_db.pl --mysqldump-opts="--skip-lock-tables" --dbtype=owamp --maxdays=90 
    --owmesh-dir=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/ --dumpdir=/var/log/BACKUP/owamp
    
  8. restart services:
    sudo /etc/init.d/crond start
    sudo /etc/init.d/owampd start;sudo /etc/init.d/perfsonarbuoy_owp_master start;sudo /etc/init.d/perfsonarbuoy_owp_collector start
    sudo /etc/init.d/bwctld start;sudo /etc/init.d/perfsonarbuoy_bw_master start;sudo /etc/init.d/perfsonarbuoy_bw_collector start
    sudo /etc/init.d/perfsonarbuoy_ma start
    
  9. check data sometime soon after (~5-10 minutes for OWAMP, several hours for BWCTL) on the web pages.
  10. As an additional step, you can add the following to the /etc/my.cnf file, in the [mysqld] section:
        set-variable = table_cache=12000
    

    Run /etc/init.d/mysql restart after this change.

Q:I have a limited amount of disk space in /var/log, and I have noticed that perfsonarbuoy_ma is writing a lot of INFO entries in the /var/log/perfsonar/perfsonarbuoy_ma.log which is filling up this disk space rapidly. Is there a way to suppress all INFO level logs? A:

The perfSONAR-BUOY MA (and other perfSONAR services) use log4perl controls to set the logging level. You can make a change to the specific configuration file:

 /opt/perfsonar_ps/perfsonarbuoy_ma/etc/daemon_logger.conf

And change the 'INFO' in 'log4perl.logger.perfSONAR_PS=INFO, A1' to be any of the following:

  • OFF
  • FATAL
  • ERROR
  • WARN
  • INFO
  • DEBUG
  • TRACE
  • ALL

WARN is normally sufficient.


Q:How do I migrate an existing LiveCD installation to a NetInstall installation? A:

The toolkit provides scripts for backing-up important configuration files and data then restoring them to a new host. These can be used to migrate a LiveCD to a host running the NetInstall. Follow the steps below to perform the migration:

  1. Login to the host you wish to migrate via ssh
  2. Download the backup script here. You can use the following command to download it directly to your host:
    wget https://github.com/perfsonar/toolkit/blob/master/scripts/ps-toolkit-migrate-backup.sh
    
  3. Run the following command to make the script executable
    chmod 755 ps-toolkit-migrate-backup.sh 
    
  4. Run the script giving it the name of the gzipped tarball you would like generated as the only parameter. In the command below, all backups will be stored in the home directory in a file call ps-toolkit-backup.tgz:
    ./ps-toolkit-migrate-backup.sh ~/ps-toolkit-backup.tgz
    
  5. Remotely copy the backup file to your new host NOTE: The backup script saves non-root user accounts/password, test configurations, test data and a few other key service configuration files. It does NOT backup every file on the system, so data in home directories or other changes made outside of the Toolkit GUIs will be not be saved. Please verify you have all your important files, especially if you choose to replace the existing partition in the next step. You may use a tool such as scp to accomplish the copy task, for example:
    scp ps-toolkit-backup.tgz user@remotehost.example:ps-toolkit-backup.tgz
    
  6. Login to your new host via SSH
  7. Run the restore script with the following command, replacing ps-toolkit-backup.tgz with the file path to your backup file generated in previous steps:
    $ /opt/perfsonar_ps/toolkit/scripts/ps-toolkit-migrate-restore.sh ps-toolkit-backup.tgz
    
  8. Reboot the host
  9. All your user accounts, service configurations and performance data should have been migrated. You will need to update your administrative information by following the instructions here.
  10. Your host should now be successfully migrated!

Q:As of RHEL-5.10, Redhat is no longer going to issue security updates for MySQL-5.0. They advise upgrading to MySQL-5.5, with 5.1 as a required intermediate step. How can I do that on the perfSONAR Toolkit? A:

MySQL 5.1 and 5.5 are new packages included with RHEL-5.10, that can be installed concurrently with MySQL-5.0. More info and tips for migrating here:

https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Deployment_Guide/ch-Migrating_from_MySQL_5.0_to_MySQL_5.5.html

The process:

yum install mysql51 mysql55
service crond stop  # so service_watcher cronjob doesn't restart mysql later

Then follow the 'dump and restore' method at the above site, with the exception of running
/opt/rh/mysql55/root/usr/bin/mysql_secure_installation
after starting mysql55.  Since a password may be reset, use '-p' with the mysql commands 
in single quotes in the last 2 steps.

Edit .conf files in:
/opt/perfsonar_ps/toolkit/etc/backingstore.conf.d
and change references to /var/lib/mysql to
/opt/rh/mysql55/root/var/lib/mysql
so that you see something like:
# grep /var/lib/mysql *
cacti.conf:store      /opt/rh/mysql55/root/var/lib/mysql/cacti
mysql.conf:store      /opt/rh/mysql55/root/var/lib/mysql/mysql
perfsonarbuoy.conf:store  /opt/rh/mysql55/root/var/lib/mysql/bwctl
perfsonarbuoy.conf:store  /opt/rh/mysql55/root/var/lib/mysql/owamp
pinger.conf:store     /opt/rh/mysql55/root/var/lib/mysql/pingerMA

Edit /opt/rh/mysql55/root/etc/my.cnf and add this line:
skip-networking
to the [mysqld] section.

Edit /opt/perfsonar_ps/toolkit/scripts/service_watcher and set
my $mysql_pid  = "/opt/rh/mysql55/root/var/run/mysqld/mysqld.pid";
my $mysql_init_script  = "mysql55-mysqld";
($mysql_pname doesn't seem to matter)
then change
system("pkill -f mysqld");
to
system("pkill -f mysql55-mysqld");
and change
system("pkill -9 -f mysqld");
to
system("pkill -9 -f mysql55-mysqld");

chkconfig mysqld off
chkconfig mysql55-mysqld on
With mysql55 stopped, I moved /var/lib/mysql out of the way just to be sure it wasn't 
getting used.  The directory will get re-created because the mysql socket still gets 
created in /var/lib/mysql, didn't try to change that, since it might affect subsequent 
updates to mysql55.


Q:My perfSONARBOUY MA is listed in the "Running" state, but returns no data. How can I debug this? A:

The MA will not be running and when you look at the logs you will see a MySQL connection error. If you run '/etc/init.d/mysqld status' it will be listed as 'stopped' and there will be no mysql processes in the process list. Attempting to restart mysqld yields the following:

# /etc/init.d/mysqld restart
Stopping mysqld:                                           [  OK  ]
Another MySQL daemon already running with the same unix socket.
Starting mysqld:                                           [FAILED]

There are RHEL and MySQL issues for this particular problem, with not ETA (as of January 2014) for a fix:

The way this problem has been addressed by others, is by manually running "rm /var/lib/mysql/mysql.sock" and then re-running " /etc/init.d/mysqld restart".


Q:Why is my LiveCD not running the default perfSONAR iptables configuration? A:

As of version 3.3.2 of the Toolkit, a default set of iptables is loaded on all installations of the perfSONAR Toolkit. On the LiveCD, the iptables will only be installed if you have setup a disk to persist configurations. If you do not see the rules, run /opt/perfsonar_ps/toolkit/scripts/nptoolkit-configure.py and follow the prompts after selecting option 1.


Q:Can I add my own rules to iptables/ip6tables? A:

Yes. Version 3.3.2 and later install a default set of iptables rules in /etc/sysconfig/iptables and /etc/sysconfig/ip6tables. You may modify these files with your own rules and they will not be overwritten as long as you leave intact the line # perfSONAR Toolkit Firewall Config v1.0.

If your ran a version of the Toolkit prior to 3.3.2 and manually defined iptables rules, your manual rules can be found in /etc/sysconfig/iptables.bak and and /etc/sysconfig/ip6tables.bak. You will need to manually merge these rules if there are any you wish to include in the updated rule set.

Q:After updating to perfSONAR Toolkit v3.3.2, maddash stopped working. What should I do? A:

The quickest way to correct the issue is to run:

rm -rf /var/lib/maddash/maddash

After doing this, restart maddash. This will allow maddash to rebuild the tables correctly. The downside is you'll lose the last 7 days of alerting data (not the last seven days of measurements, just whether checks returned green/yellow/red/orange status).


Q:I am not able to start the perfSONARBUOY Web Service, and there are MYSQL errors. What should I do? A:

The following is a possible error when attempting to restart the perfSONARBUOY Web Service:

[root@perfsonar ~]# sudo /etc/init.d/perfsonarbuoy_ma restart
/etc/init.d/perfsonarbuoy_ma stop: perfSONAR perfSONAR-BUOY MA Service (no pid file) not running
waiting...
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/daemon.pl 
--config=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/daemon.conf --pidfile=perfsonarbuoy_ma.pid 
--piddir=/var/run --logger=/opt/perfsonar_ps/perfsonarbuoy_ma/etc/daemon_logger.conf --user=perfsonar 
--group=perfsonar
DBD::mysql::st execute failed: Can't find file: 'DATES' (errno: 2) at 
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/../lib/perfSONAR_PS/DB/SQL.pm line 213.
Can't use string ("-1") as an ARRAY ref while "strict refs" in use at 
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/../lib/perfSONAR_PS/Services/MA/perfSONARBUOY.pm line 1044.
/etc/init.d/perfsonarbuoy_ma start: perfSONAR perfSONAR-BUOY MA Service could not be started

It is suggested that the user try to rebuild the database by deleting, and re-creating things in the event that MySQL is corrupted:

/opt/perfsonar_ps/perfsonarbuoy_ma/bin/owdb.pl -x root
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/owdb.pl -i root

If this fails with an error like the following:

Unable to drop database owamp at /opt/perfsonar_ps/perfsonarbuoy_ma/bin/../lib/OWP/DB.pm line 118, 
 line 2.

It will be necessary to take a manual action against the database storage system. Do the following:

sudo mv /var/lib/mysql/owamp /var/lib/mysql/owamp.back
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/owdb.pl -x root
/opt/perfsonar_ps/perfsonarbuoy_ma/bin/owdb.pl -i root

This step will backup the correct database directory, and allow the user to start over. You can remove the backup copy once the operation is compete.


Q:My Cacti instance was vandalized/could be vandalized due to guest access being enabled. What can I do to stop this? A:

In June of 2014, the perfSONAR project was made aware of Cacti instances on toolkit nodes being valndalized via the guest account. It is possible for someone to change the name of graphs, but it is important to note that they were not granted any access to the system itself. More information is available on the cacti website, and the steps they took to stop this problem: http://www.cacti.net/downloads/docs/html/guest_access.html.

Running yum update on 3.3.2 systems will pull in all required fixes (e.g. build 16 of the perl-perfSONAR_PS-Toolkit package and above). The major change from this update is adding the following lines to /etc/httpd/conf.d/apache-toolkit_web_gui.conf:

RewriteRule ^/toolkit/gui/cacti(.*) 
https://%{SERVER_NAME}/toolkit/admin/cacti$1 [R,L]

Once this is implemented, guest access to the tool will no longer be allowed.  If you have noticed your instance has been vandalized, the following steps can correct the error in MySQL:

mysql cacti -e "update settings_graphs set value='' where name in ('title_font','legend_font','axis_font','unit_font') and user_id = (select id from user_auth where username = 'guest')"