PI Forum Project Notes

Entries are made by date, with the most recent at the top starting just after the Issues table.

ISSUES

2018-05-18

Work on Missing DokuWiki Page Tools Icons

 Found the missing menu for the Page Tools hiding under a 3 or 4 pixel wide div - mouseover and the menu pops up. So this column of tool icons is what is missing:

  Instead there is a 3 px vertical line that on mouse-over it displays the full floating menu:

This is a known and logged issue (#2366 at https://github.com/splitbrain/dokuwiki/issues). Not a show stopper. 

2018-05-14

Installed Latest Stable Release of DokuWiki

Installed updated version of DukuWiki, 2018-04-22a "Greebo", aka Version 50.1 and reconfigured settings appropriately from the Admin interface.

The default template appears differently from previous release. It has no edit menu, to allow for editing of existing pages or creation of new pages. I have started researching the cause for this and have yet to determine whether this is due to the new default template page or some obscure permissions issue.

2018-05-11

Created Table pifgroups

Created a table in the het1pi database to support the authentication interface to DokuWiki using the native `authpdo` plugin. 

CREATE TABLE `pifgroups` (

  `pifgroups_id` mediumint(8) unsigned NOT NULL AUTO_INCREMENT,

  `user_id` smallint(6) DEFAULT NULL,

  `pifgroup`  varchar(2) DEFAULT NULL,

  PRIMARY KEY (`pifgroups_id`)

) ENGINE=InnoDB AUTO_INCREMENT=1 DEFAULT CHARSET=latin1;

• Populated initial table with all existing user ids and with a pifgroup value of 'user'
• Inserted entries for each RA (Resident Astronomer) with a pifgroup values of 'ra' and 'admin'
• Inserted entries for each TAC Chair with a pifgroup values of 'tac_admin'

2018-04-18

Configure authpdo Plugin Settings

Since all authentication is based on the application's original auth mechanism, only the minimum required authpdo plugin options and functions are configured at this time.

To set or modify the authpdo plugin settings, click

• Debug is checked and should be turned off after testing is completed - FOLLOWUP

• DSN is set to mysql:host=localhost;dbname=het1pi;charset=utf8

• Database connection user is set to pi (see source for pw)

• authpdo.select_user() is set to

  SELECT user_id AS "uid", username AS "user", CONCAT_WS(" ", first_name, last_name) AS "name", password AS "hash", email AS "mail" FROM user WHERE username = :user

• authpdo.select-user-groups() is set to SELECT CONCAT("user") AS "group"

• All other options are left blank.

2018-04-17

Continuing integration of DokuWiki auth mechanism

DokuWiki provides multiple means to for authenication using application backends. Since we use MySQL, I had initially planned on using the authmysql plugin. As I studied the code and documentation, I discovered that this is deprecated and it is recommend to use the authpdo plugin, which is bundled with the release. This plugin allows to configure your wiki to authenticate against an existing database using PHP's PDO extension. 

When configuring DokuWiki to use authpdo, I discovered a significant security issue with the original auth mechanism. Currently passwords are encrypted using the sha1() function. These calls need to be replaced with the password_hash() and password_verify() prior to moving to production. This has been noted in the "ISSUES" table at the top of this page.

2018-04-03..11

Corrected self-injected defect

In an earlier commit, I had converted the deprecated regeg string substitution function ereg_replace() with it's replacement preg_replacement(). To my determent, I only ran a few test cases to ensure that it was working, but those cases did not cover Tab names with blanks in them (e.g, 'Phase I', PI Forum'). I chased the rabbit down many holes thinking that the fault inject was surely introduced by debugging code I added or something about how DokuWiki was being called. I finally found the error and corrected it.

Deployed DokuWiki in PI Forum Tab as iframe

After many attempts and reviewing a great deal of the Original Application code and the DokuWiki source, it became clear that the only means of display integration was either by opening another window or browser tab or by utilizing an iframe in the PI Forum Tab. I chose the latter in order to preserve application's navigation framework and sharing of global variables. Initial usability testing was successful.

PI Forum Tab Demo Stub

Began Integrating User Auth Mechanisms

The Astronomy Proposal System application has it's own authentication mechanism that will be used to drive the DokuWiki auth mechanism. Work has begun on scoping this effort by studying the DokuWiki documentation and source code. Sticking with the original application's auth mechanism is one of the fundamental requirements for the PI Forum.

2018-04-04

OTHER WORK: INC0292065 - Delete media

Updated photo for user on Astronomy People page for Research Scientist Judit Ries and Resolved ticket.

Continued Work to Resolve DokuWiki Display Issue

When DokuWiki is embedded in the application under the PI Forum Tab, DokuWiki thinks is displaying to a mobile device and sets display width to small percentage of current browser page. Used wtf_backtrace() calls to follow flow control and display $_SESSION and $_SERVER globals. Trying to limit modifications to DokuWiki source to defect corrections, an approach that may result in PI Forum being display in a new browser tab instead of the application's "PI FORUM" Tab.

2018-04-03

OTHER WORK: INC0291173 - MCDO Schedule Updates

Resolved request from UT TAC Char to update Observing Schedules on McDonald Site (www.as.utexas.edu/mcdonald/schedules/) - running on affirmed.as.

Wrote detailed directions on How to make these updates and posted on the wiki.

2018-04-02

DokuWiki Integrated

Installed DokuWiki under new application directory DocumentRoot/piforum and linked it new PI Forum Tab.

Integration passed proof of operation, functional testing including launching wiki from PI Forum Tab, manually (via DokuWiki auth) creating a user account, creating new pages, and random testing of basic wiki features.

Placed DokuWiki source source including active config files under source control and committed modified original application files.

04/02/18 11:14:57: git commit -a -m "Completed initial implementation of 'PI Forum' application tab." 

These changes were merged in the local dev branch then pushed to the UT GitHub repo

2018-03-30

Completed New "PI Forum" Tab

Created a new application Tab for PI Forum as seen below.

2018-03-29

Continued Learning the Tab Related Application Code

Determined that the het1pi.tabs table is never used. Checked old backup files which confirmed that this empty table has not been used since at least October of 2017. It may have never been used for production purposes. See previous day's entry on Tab for more details.

Attended CNS-Web/Apps-Dev weekly meeting in which nothing other than the plan to restart the httpd service so the security tweaks made by Eric Rostetter would take effect. Sent email to RA's at HET and got a green light from Jim Fowler. Restarted httpd at 3:30 then successfully tested the website.

Spent some time updating the CNS-IT wiki pages for Astronomy Proposal System.

2018-03-28

Continued Code Cleanup

Installed Missing Website Directories

Installed the directories that were found and added to repo yesterday under endorphin:/home/hydra.

Creating New Application Tab for PI Forum

Tabs Overview

Tabs play a central roll in site navigation for the Astronomy Proposal System. A few screen shots of the application's Tabs are shown below. Navigation and access to the various pages of the application website are managed by the code on a per user basis. The user's roles and various status attributes associated with the user determine which Tabs are displayed and what functionality is allowed for that user. For example a Principal Investigator (PI) would never be presented with the TAC, UT Chair, or TAC Admin tabs.

Application Tabs Elements

The tabs presented to a user are based on user role access (PI, TAC Member, Chair,) and project status attributes associated with the user's account combined with and stored in a global PHP array named $tabs, which is stored in the PHP session array $_SESSION and NOT stored in the tab table in the MySQL database het1pi. [ISSUE: Low priority: Delete hep1pi.tabs table]. The application files controlling tabs are

index.html and inc/sec_nav_proc.inc and inc/sec_nav.inc

I do not believe that any other files are involved, but I am loath to say for certain as the original code utilized food programming paridigms including the two worst from the starch patterns, Pretzel and Spaghetti. Heavy use of globals only one object Class.

2018-03-27

More Code Cleanup

Application Directories Outside of DocumentRoot

Determined all file required by the website that are not located under the site DocumentRoot and tar'ed them up on hydra, copied the tar file to endorphin and extracted these files and directories into the local git repo on endorphin, added them to my clean_up branch then merged them into the dev branch and pushed it the UT Github repo. The following directories were added to the repo and map to the installed location of /home/hydra and contain a total of 10,061 files and sub-directories:

• bin/

• charts/

• mcdonald/

• mtlocke_uploads/

• ph1_uploads/

• ph2_uploads/

• plan.db/

With the exception of bin/, all these directories hold various uploaded files whose names are correlated to primary keys for related database tables such as prop_id field in the props table for the mtlock_uploads/ directory:

/home/hydra/mtlocke_uploads/all_6179.pdf

Source of PHP Log Noise

Most of the warnings in the /var/log/httpd/ssl_error_log file are caused by the use of unquote strings used as keys to index associative arrays. For example

[Tue Mar 27 10:46:11 2018][error] [client 128.83.161.210] PHP Notice: Use of undefined constant display - assumed 'display' in /home/hydra/www/pdflib/ph1_pdf.php on line 181, referer: https://endorphin.as.utexas.edu/?a=tac&b=prop

Identified 1206 lines of source with this particular coding error. Although these are treated as errors and logged as such, the interpreter assumes the identifiers to to strings and does not cause any functional errors but still creates unnecessary noise in the log files.

FOIA / Open Records Request Distraction

Paused my regular work for two hours to provide answers to Bob Davis for questions from the requestor of UT Austin ORR No. 146647.

2018-03-26

Learning more about the application code

Spent most of the day learning more of the Astronomy Proposal System code and with few goals in mind:

• Learning how the code implements the application navigation TABs, (i.e. Home, TAC, Phase I, etc…). - this is still a work in progress as the code is:

  • Poorly structured

  • Does not utilize much of anything one could call a naming convention

  • There is no documentation to indicate the functionality of individual .php and .inc files

  • Very limited use of inline comments

  • Produces a great deal of noise in the /var/log/httpd/ssl_error_log file

  • Utilizes directories outside of the DocumentRoot, /home/hydra/www, at least they are all under the same parent directory, /home/hydra

• Created maps of .php, .html and .inc files to the files they include.

2018-03-21

PDFLib+PDI License Info

Researching licensing on the PDFLib website, I determined that that hydra is one major release behind the company's current release (8.0.4 and 9.1.2 respectively.) and requested a PO search from Lana Beraneck in the ASTR/MCDO Business Office. She forwared a copy of the only PO found, which is for the current licensed copy of PDFLib+PDI running on hydra- click here for a pdf copy of the PO .

Ugrading to 9.1.2 has a list price of $2,390.00 USD. If the same 15% discount is applied, that would come to $2,130.50. In the past, MCDO has not purchased the support. The support fee is 20% of the product's list price per year, and includes technical support as well as all minor and major updates.

MCDO has purchased this product without support three (3) times, the initial purchase plus 2 upgrades, over the course of 12 years. Estimating past purchase prices based on extrapilation of the 2012 price to previous prices by MCDO comes out to a total cost of approximately $3,039.00 without support. Had support been purchased with the initiall license and maintained, the total cost would have been $2,177.00for the same time period providing a savings of $862.00.

Table of Cost to MCDO for Each License Purchased

The version of PDFLib+PDI we are using now, 8.2.2, reached end-of-life in February 2015.

According to CSV Details, an Intenet Security Issue tracking site, 4 CSVs have been reported since 2004. While this number is low, running this PDFLib without the support option leaves hydra open to future vulnerabilities with no means of resolution other than to pay full price for a new release containing patched code. The probability of future CVEs may be low, but a correction would be costly without support. We know there is at least one aggravating issue in the 8.2.2 where PDGLib fails to process some special characters. Corrections require manual intervention by programming staff unless the PI can re-submit their proposal substituting the offending characters with friendly characters (who knows what those are?) . Given the special character issues plus the the dependency of the UT TACs using the HET and MtLocke proposal systems, and the potential long term savings on licensing fees, the purchase of a new license for PDFLib+PDI with support appears to be a wise use of funds that would include vendor support and access to minor and major releases for no additional charges.

Vendor Informaion

PDFlib GmbH

Franziska-Bilek-Weg 9 80339 München, Germany 

www.pdflib.com](http://www.pdflib.com/)sales@pdflib.com](mailto:sales@pdflib.com)

phone +49 89 452 33 84-0

fax +49 89 452 33 84-99

2018-03-16

Installed DokuWiki

Installed DokuWiki (2017-02-19e) on endorphin under /home/hydra/www/piforum. Got it running and put admin credentials in stache.

2018-03-15

Apche 2.4 install and backout

Attempted to get Apache 2.4 installed and working, which was a yummy learning experience.

Unistalled httpd (2.4), installed 2.2 and corrected problems with /etc/httpd/modules configuration after scrapping attempt the 2.4 effort.

Missing allocation database

Discovered a previously unknown (and undocumented) dependency on the NOSS MySQL database allocation when testing the HET Observing Support pages after the httpd version rewind. Created the database from backups on hydra (see /root/cloud/work/data_from_hydra/ for the .sql files).

2018-03-11

Set up mysql backups for endorphin

1. Created directories /opt/hydra/{bin db_backups doc etc lib man mysql_dumps src}

2. Copied backup scripts and config files from hydra to endorphin

3. Created crontab entry 15 17 * * * /opt/hydra/etc/automysqlbackup -bc "/opt/hydra/etc/automysqlbackup-hydra.conf"

4. Add backup dirs, scripts and config file to wiki.

5. Add notes to wiki scripts about backup scripts.

2018-02-28

Create Clone/dev Databases

Copied mysql database backup files from hydra to endorphine which are located here:

/root/cloud/work/data_from_hydra/het1pi/het1pi.sql

/root/cloud/work/data_from_hydra/mtlocke/mtlocke.sql

Created both databases on from the cli then sourced the .sql files.

Testing browser access, https://endorphin.as.utexas.edu, failed during login (mastontc1). Blank page was displayed and following errors were found in `/var/log/httpd/ssl_error_log:

File does not exist: /home/hydra/www/favicon.ico, referer: https://endorphin.as.utexas.edu/

PHP Fatal error: Class 'mysqli' not found in /home/hydra/www/incs/help_doc.inc on line 16

File does not exist: /home/hydra/www/sdk

File does not exist: /home/hydra/www/nmaplowercheck1519768623

File does not exist: /home/hydra/www/HNAP1

PHP Fatal error: Class 'mysqli' not found in /home/hydra/www/classes/form_proc.inc on line 88, referer: https://endorphin.as.utexas.edu/?t=1&a=login

Issue of mysqli Error Resolved

Although het1pi and mtlocke databases are the only onese written to by code on hydra, the het_help database is required for the site to function. Copied the daily backup from hyrdra to endorphin and installed/imported it after creating the table first:

mysql -p het_help </root/cloud/work/data_from_hydra/het1pi/het_help.sql

2018-02-27

Time to create the MySQL databases

NOTE: The following fields are found in mysql.user table only on hydra so the associated values in the INSERT commands below were removed prior after receiving error messages when attempting to load the user table data from hydra on endorphin

ERROR 1136 (21S01): Column count doesn't match value count at row 1

1 Create_tablespace_priv 2 plugin 3 authentication_string

INSERT INTO `user` VALUES ('localhost','pi','*8795CBC85AFA70BADC614ED394EC3599DF6FEB6E','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0),('hydra.as.utexas.edu','pi','*8795CBC85AFA70BADC614ED394EC3599DF6FEB6E','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0),('endorphin.as.utexas.edu','pi','*8795CBC85AFA70BADC614ED394EC3599DF6FEB6E','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','N','','','','',0,0,0,0)

The Real Problem with Mismatched Columns

The real problem was that the old version of mysql was not removed prior to installing the IUS version of 5.6. So, the old version was removed.

 yum remove mysql-server-5.1.73-8.el6_8.x86_64 

 yum remove mysql-5.1.73-8.el6_8.x86_64

 yum shell

 > erase mysql-libs

 > install mysql56u mysql56u-libs mysql56u-server mysqlclient16

 > run

 yum info mysql56u

 Loaded plugins: product-id, refresh-packagekit, rhnplugin, search-disabled-

               : repos, security

 This system is receiving updates from RHN Classic or RHN Satellite.

 Installed Packages

 Name       : mysql56u

 Arch       : x86_64

 Version     : 5.6.39

 Release     : 1.ius.el6

 Size       : 36 M

 Repo       : installed

 From repo   : ius

 Summary     : MySQL client programs and shared libraries

 URL         : http://www.mysql.com

 License     : GPLv2 with exceptions

 Description : MySQL is a multi-user, multi-threaded SQL database server. MySQL

             : is a client/server implementation consisting of a server daemon

             : (mysqld) and many different client programs and libraries. The

             : base package contains the standard MySQL client programs and

             : generic MySQL files.

2018-02-26

Install Missing Packages

Continued reviewing the configuration on endorphin and found the following packages to be missing so I installed installed them:

• php56u-pdo-5.6.33-1.ius.el6.x86_64

• php56u-gd-5.6.33-1.ius.el6.x86_64

• php56u-mysql-5.6.33-1.ius.el6.x86_64

• php56u-mysql-5.6.33-1.ius.el6.x86_64

• php56u-mysqlnd-5.6.33-1.ius.el6.x86_64

• php56u-odbc-5.6.33-1.ius.el6.x86_64

• php56u-mysqlnd-5.6.33-1.ius.el6.x86_64

• /usr/lib64/php/modules/libpdf_php.so - PDFLib+PDI was not available as an rmp, so for now a copy from hydra is being used.

2018-02-26

SSL Certs

In order to keep the config files the same on the dev system (endorphin) as they appear on the production system (hydra), I copied the recently created certs from /etc/bacula/ssl to /etc/pki/tsl/{certs,private} and created links to match the hydra config files in /etc/httpd/conf.d/ssl.conf:

/etc/httpd/conf.d/ssl.conf:SSLCertificateFile /etc/pki/tls/certs/hydra.as.utexas.edu.crt

/etc/httpd/conf.d/ssl.conf:SSLCertificateKeyFile /etc/pki/tls/private/hydra.as.utexas.edu.key

/etc/httpd/conf.d/ssl.conf:SSLCertificateChainFile /etc/pki/tls/certs/incommon-new-interm.crt

Problems with http Access

Updated /home/hydra/www (working development tree) from local git repo branch clone_site_to_endorphin at /root/cloud/work/hydra-web/home_hydra_www/.

Also updated /etc/... as follows:

etc/sysconfig

etc/logrotate.d

etc/php.d

etc/ld.so.conf.d

etc/httpd

etc/httpd/conf

etc/httpd/conf.d

etc/bash_completion.d

etc/rc.d

etc/rc.d/rc1.d

etc/rc.d/rc3.d

etc/rc.d/rc2.d

etc/rc.d/rc5.d

etc/rc.d/rc0.d

etc/rc.d/rc6.d

etc/rc.d/rc4.d

etc/rc.d/init.d

etc/php-zts.d

Tested browser access to https://endorphin.as.utexas.edu which responed with Forbidden. Appeared to be file permissions on .htaccess file. Updated all files in repo with chmod -R o+r and all directories with chmod -R o+rx. This did not resolve the issue. I found that I also could no longer ssh to endorphin. iptables looked fine, so I set SELINUX=permissive still had access problems. A halt followed by a cold boot resoved the access issues.

2018-02-25

Installed PHP Project Tree

Open source PHP development productivity tool. Here's the link to the wiki page

This is a simple PHP application that parses your PHP source tree and builds an dependency tree based on included|required files (not by class usage/inheritance). Using this information it can answer simple question like:

• what are the .php files that this specific file depends on (directly/indirectly)

• what are the files that includ/require directly or indirectly a specific .php file

Copied these from repo to system locations:

• bash_completion.d/

• httpd/

• ld.so.conf.d/

• logrotate.d/

• php.d/

• php.ini php-zts.d/

• rc.d/

• sysconfig/

2018-02-23

Used following to test if the gd extension is installed:

<?php

var_dump(gd_info());

?>

Answer is yes:

array(13) {

 ["GD Version"]=>

 string(26) "bundled (2.1.0 compatible)"

 ["FreeType Support"]=>

 bool(true)

 ["FreeType Linkage"]=>

 string(13) "with freetype"

 ["T1Lib Support"]=>

 bool(true)

 ["GIF Read Support"]=>

 bool(true)

 ["GIF Create Support"]=>

 bool(true)

 ["JPEG Support"]=>

 bool(true)

 ["PNG Support"]=>

 bool(true)

 ["WBMP Support"]=>

 bool(true)

   ["XPM Support"]=>

 bool(true)

 ["XBM Support"]=>

 bool(true)

 ["WebP Support"]=>

 bool(false)

 ["JIS-mapped Japanese Font Support"]=>

 bool(false)

}

Back out yum changes made for IMagick

package-cleanup --orphans

package-cleanup --problems 

package-cleanup --dupes

package-cleanup --leaves

yum history undo 18

Undoing transaction 18, from Thu Feb 22 12:12:26 2018

   Install php-pear-1:1.9.4-5.el6.noarch @rhel-x86_64-workstation-6

yum history undo 17

Undoing transaction 17, from Thu Feb 22 12:07:59 2018

   Install     php56w-5.6.33-1.w6.x86_64         @webtatic

   Dep-Install php56w-cli-5.6.33-1.w6.x86_64     @webtatic

   Dep-Install php56w-common-5.6.33-1.w6.x86_64   @webtatic

   Install     php56w-devel-5.6.33-1.w6.x86_64   @webtatic

   Install     php56w-gd-5.6.33-1.w6.x86_64       @webtatic

   Install     php56w-intl-5.6.33-1.w6.x86_64     @webtatic

   Install     php56w-mbstring-5.6.33-1.w6.x86_64 @webtatic

   Install     php56w-mcrypt-5.6.33-1.w6.x86_64   @webtatic

   Install     php56w-mysql-5.6.33-1.w6.x86_64   @webtatic

   Install     php56w-opcache-5.6.33-1.w6.x86_64 @webtatic

   Dep-Install php56w-pdo-5.6.33-1.w6.x86_64     @webtatic

   Install     php56w-xml-5.6.33-1.w6.x86_64     @webtatic

Undoing transaction 19, from Thu Feb 22 12:44:53 2018

   Install ImageMagick-perl-6.7.2.7-6.el6.x86_64 @rhel-x86_64-workstation-optional-6

2018-02-22

yum install ImageMagick-devel

pecl install imagick

had to edit /usr/bin/pecl

• removed the -n flag to get rid of the parsing problems

before the above will work, we need to install php 5.6 properly php -v reports: PHP 5.6.33 (cli) (built: Jan 14 2018 08:26:10) yum info php reports: Version : 5.3.3

rpm -Uvh https://dl.fedoraproject.org/pub/epel/epel-release-latest-6.noarch.rpm

rpm -Uvh https://mirror.webtatic.com/yum/el6/latest.rpm

both of the above packages were already installed ????

now we remove the current (old version)

yum remove php-common

and install the new

yum install -y php56w php56w-opcache php56w-xml php56w-mcrypt php56w-gd php56w-devel php56w-mysql php56w-intl php56w-mbstring

tried to install imagick again and it failed with the error:

The install failed.

pecl install imagick

had to edit /usr/bin/pecl

• removed the -n flag to get rid of the parsing problems

same error so let's try it manually

cd /usr/src/

wget http://acelnmp.googlecode.com/files/imagick-3.0.1.tgz

tar -xvf imagick-3.0.1.tgz

cd imagick-3.0.1

phpize

./configure

make

make install

More problems

 wget http://acelnmp.googlecode.com/files/imagick-3.0.1.tgz

 --2018-02-22 12:23:41-- http://acelnmp.googlecode.com/files/imagick-3.0.1.tgz

 Resolving acelnmp.googlecode.com... 64.233.180.82, 2607:f8b0:4003:c0b::52

 Connecting to acelnmp.googlecode.com|64.233.180.82|:80... connected.

 HTTP request sent, awaiting response... 404 Not Found

 2018-02-22 12:23:41 ERROR 404: Not Found.

/usr/lib64/pkgconfig/

/usr/local/php56/bin/phpize -> /usr/lib64/pkgconfig/

 export PKG_CONFIG_PATH=/usr/lib64/pkgconfig/

 ln -s /usr/include/ImageMagick-6 /usr/include/ImageMagick

 ./configure --prefix=/usr --with-imagick=$HOME/local --with-php-config=/usr/php56/bin/php-config

Abandonded IMagic install and will rely on PHP GD extension