| Date | Service Issue | Notes |
|---|---|---|
| 08/05/16 | Solaris Security Updates | 2:00 PM After some update problems were resolved all systems and services have been restored. 12:00 PM (Noon) Our Solaris systems will be rebooted to apply Quarterly updates that include security patches and to release a hung I/O device. The following systems and services will be unavailable and offline for hopefully no longer than 20 minutes.
|
| 07/12/16 | Solaris Quarterly Updates | 12:24 PM All services restored. 12:00 PM (Noon) Our Solaris systems will be rebooted to apply Quarterly updates that include security patches and to release a hung I/O device. The following systems and services will be unavailable and offline for hopefully no longer than 20 minutes.
|
| 12/12/15 | Solaris Security Updates |
12 December @ 9:41 AM - Having trouble with some of the zones' boot environments (BE), but moving forward.
|
| 12/02/15 | cerberus RAID fileserver | 7:00 AM - Cerberus, is going to require a reboot to update the systems software. This update is required so we can install a patch to the system. The patch may require a reboot as well. We will know as soon as the Oracle engineers evaluate a resolution to a potential security issues identified by ISO. |
| 11/19/15 | Astro email Unscheduled Maintenance | 2:59 PM - Maintenance completed. Services restored. 2:50 PM - astro email services will be taken off-line today to adjust performance tuning parameters. email services are expected to be back on-line within 20 minutes. |
10/21/15 - 11/03/15 | DocuShare Sever Emergency Maintenance | 11/03 02:14 PM - DocuShare is back online in WRITE mode. Log files indicate that our backups are running without any problems. 11/02, 9:37 AM - The DocuShare server, ds.as.utexas.edu, has been recovered via the indomitable and resolute efforts of Paul Morris and Patrick Goetz (CNS Systems), to whom we owe prodigious gratitude for their invaluable work. DocuShare is currently online in READ-ONLY mode while we configure, initiate and test a cron based backup to cerberus, our Oracle RAID. 7:00 AM - ds.as.utexas.ed was taken offline, it's internal disks removed and copies were made. Booting from a Redhat DVD, the two mirrored disks were reviewed and found to be inconsistent. Recover efforts continue. 03:18 PM - Server attempted to run a disk check and hung at 11.4%. Rebooted and it hung again at 11.4% during the disk check. Will allow the process to run overnight. And re-evaluate in the morning. 02:10 PM - ds.as.utexas.edu, DocuShare server rebooted. 1:45 PM DocuShare server database engine crashed. Rebooted server. |
| 10/13/15 | Solaris Sever Maintenance Updates | Security and maintenance updates will be applied to gemini, our Global Solaris server that hosts astro and ftp virtual machines. 7:15 AM - Maintenance completed. All services restored. 7:00 AM gemini zones shutdown; global zone rebooted Services: Email, Webmail, astro logins, anonymous FTP, bigbubba logins, individual user web pages |
7/15/2015 7/13/2015
| Astro email Critical Security Maintenance | The SSL security certificates must be updated for mail services on astro to provide compatibility with utexas.edu mail services. SSL Certificate configuration continues to be a recondite undertaking with a plethora of methodologies and a poverty of guidance. In short, this may take awhile. 12:45 PM astro email back online. The SSL certs were updated and tests indicated success installation and that astro email service now meets the higher standard of security required by Gmail and orher mail related services by properly presenting the intermediate SSL certificates. 12:15 PM astro email services will be taken off-line at and a new verified set of SSL certificates will be installed and tested. 9:50 PM astro email is back on line using the original configuration. Dovecot rejected the second configuration like a bitter pill being pushed down a sick cat's throat. 9:20PM. astro email services will be taken off-line at and a new convolution of contentious Secure Socket Layer certificates will be installed with great hope of success and triumphant exodus from this afflictive drama. 9:14 - After much research and many attempts to get astro's SSL certificates configured in a format that may be acceptable by Dovecot, the IMAP and POP email server software as we often say here in Texas, "I'm fixing to give another go at fixing the &$*^$%$#@ thing." 6:12 PM The recommended Certificate configuration did not work. Inserting the Intermediate CA cert per recommendations for Dovecot (our IMAP and POP service) resulted in an Cert / Private Key Mismatch. astro email is back on-line with the old configuration while I make another attempt with newly downloaded Root and Intermediate Certs from InCommon. astro email will be taken offline again as soon as I have can fetch the certs from the Certificate Authority and install them as a new configuration. If you have any patience left, please send it my way as my stockpile is running low. 5:00 PM astro email and webmail will be taken off-line for the SSL certificate updates. |
| 5/13/2015 | Solaris Sever Maintenance Updates | Maintenance updates will be applied to gemini, our Global Solaris server that hosts astro and ftp virtual machines. Services: Email, Webmail, astro logins, FTP. 5:48: Updates completed. All system services restored. 5:30 System reboot. |
| 5/13/2015 | VM Server Emergency Security Patch | Services: All departmental virtual machines will be power cycled once the patches are in place for the exploit in VENOM to be fixed. |
| 5/9/2015 | Astro email RAID Mounts Emergency Maintenance | Services: All astro email services including webmail will be unavailable while correct action is being taken to scrub the RAID pool for astro inboxes (/var/mail). Status shows the scrub should be completed around 12:45 PM today. 12:37 All services restored.
|
| 3/13/2015 | EMERGENCY RAID MAINTENANCE | Services: /opt/local, DHCP (laptop connections), Mathmatica, IDL, astro Webmail, etc. NOTE: Regular astro mail should continue work but may be slow. 14:10 All services restored 12:45 Determined system hangs were caused by the failure of one of the two systems disk which are mirrored. Paul Morris worked with Oracle Support and removed the failed disk from the mirror, rebooted, ran integrity tests and brought the RAID back online. Replacement disk is being shipped and will be installed next week. 9:25 RAID (cerberus) failed and went off-line. Oracle technical support is working the issue |
| 2/25/15 | Astro RAID Mounts Emergency Maintenance | Services: All astro services including email, FTP and webmail will be unavailable for a short time starting at 6:45AM. This work is expected to take from 15 to 20 minutes. 6:50 astro and all related services back online 6:45 astro rebooted. |
| 02/03/15 | Astro Email Scheduled Maintenance | Services: All astro email services including webmail will be unavailable for a short time starting at 5:30 PM today. This work is expected to take from 15 to 20 minutes. 5:38 all email services are back online, maintenance completed 5:30 astro email services taken off-line |
| 01/07/15 | Astro and Galactica Maintenance: Solaris Updates Email, Webmail, FTP /sans/* RAID shares | Services: All astro services including email, webmail, ftp along /sans/* partiion shares will be unavailable for a short time while astro and galactica are taken off-line to apply regular updates 9:10 FTP is now online; Maintenance completed. 8:23 All services expect FTP back online 7:30 astro and galactica taken off-line |
| 12/16/14 | Astro Email - Unscheduled maintenance | Service: All email services including webmail 7:36 all email services restarting and available 7:30 astro email restart to clear out zombie processes |
| 12/10/14 | Services | Service: printers and calendars. 10:25 - System functional again. 9:52 - Disc IO errors due causing system to hang. |
| 12/9/14 | Services | Service: printers and calendars. 14:30 - Temporary system in place. 11:10 - System became unresponsive due to a hardware failure. Migrating system to new hardware. |
| 11/16/14 | cerberus RAID | Service: /opt/local, /san and other cerberus RAID mounts NOTICE: Server is unreachable. 11:31 - Service restored. 11:05 - . RCA in progress. |
| 11/16/14 | Astronomy Web Site | Service: All Astronomy web sites hosted on dept web server, affirmed NOTICE: Sites unreachable. Server load 257.0 (should be in single digits or low double digits) 11:36 - Service restored. 10:32 - System hung after reboot. 10:20 - Preparing to reboot. 09:59 - Access to www.as.utexas.edu times out. Load excessively high. |
| 11/16/14 | Web Mail | Service: Astronomy web mail NOTICE: Server is unreachable. 11:32 - Service restored. 10:25 - Cannot access webmail.as.utexas.edu. RCA in progress. |
| 11/07/14 | Astro Email | Service: Astro IMAP and POP email (reading) NOTICE: Emergency maintenance: SSL 3.0 patch 07:52 - Patch successfully applied; Service restored 07:50 - Inbound and stored mail reading off-lined |
| 11/05/14 | Astro Email | Service: All astro email NOTICE: SSL 3.0 patch failure. |
| 05/09/14 | Astro Email Blacklisting | Service: Outbound astro email NOTICE: The de-blacklisting of astro may take hours to a few days to propagate 22:38 - Outbound email restored 21:55 - astro scheduled for reboot 21:07 - De-blacklisting request made 20:45 - Root caused identified, clean up initiated 19:15 - Outbound mail service off-lined |
03/14/14 | Astro Server Upgrade | Service: All astro services including email 03:14 - Mail Services resumed. SSH keys have changed. Please report any problems you might have. 07:00 - System taken off-line |
| 01/01/14 | DocuShare Server Emergency Maintenance | Service: All DocuShare services 21:49 - UT ISO Notification of ds Bot/Worm compromise 22:00 - Border Quarantine set by Networking 15:16 - Efforts to reestablish services continue |
| 11/27/13 | File Server (cerberus) Scheduled Maintenance | Service: Email and all other astro services 07:00 - cerberus taken offline and began updates 08:30 - cerberus hung on last update; Oracle Support contacted 10:00 - Oracle Support engineer finally gets involved 12:45 - Oracle Support engineer still working to resolve upgrade issues 14.30 - Maintenance completed. |
| 11/02/13 | Astro Scheduled Maintenance | Service: Email and all other astro services 07:00 - astro taken offline. 09:45 - Solaris and email updates completed. 09:55 - astro is back online. |
| 08/12/13 | Astro IMAP Emergency Maintenance | Service: Mail 13:06 - Services resumed. 11:46 - services being restarted again. Please do not use your email client. 11:30 - IMAP services being restarted due to system load issue. |
| 08/10/13 | Solaris OS Scheduled Maintenance | Service: All Suns running Solaris OS 07:00 - Will begin rebooting systems. 08:48 - All workstation systems are back online. 08:15 - astro is back online. |
| 08/10/13 | Astro Empergy Maintenance | Service: Email (astro) 07:00 - Begin applying recommended patches; Email and other services unavailable. 08:15 - Email and all other astro services are back online. |
| 07/01/13 | Webmail Host Server Upgrade | Service: Webmail 9:15 - Maintenance finished. 9:00 - Server hosting Webmail will be down for a few minutes for a memory upgrade. |
| 05/28/13 | Webserver Emergency Maintenance | Service: Webserver 19:59 - Partial functionality returned. 13:30 - Drive reconfiguration failure. 8:50 - Service functionality returned. 7:00 - Hardware failure. |
| 05/09/13 | Astro Emergency Maintenance | Service: Mail 18:10 - Mail service operational. 16:29 - Mail is temporarily offline while allocation issues for the service are being resolved. 14:43 - Troubleshooting mail service errors. |
| 05/03/13 | Vault Scheduled Maintenance | Service: Vault 8:00 - Server hosting Vault brought down to rebuild new drives. |
| 04/29/13 | Astro Emergency Maintenance | Service: Mail 13:55 - Mail service operational. 10:09 - Mail service interruption |
| 04/09/13 | Astro Emergency Maintenance | Service: Mail 11:45 - Mail service operational. 11:15 - Mail service interruption |