Skip to end of metadata
Go to start of metadata

 

 

DateService IssueNotes
08/05/16Solaris Security Updates

2:00 PM After some update problems were resolved all systems and services have been restored.

12:00 PM (Noon) Our Solaris systems will be rebooted to apply Quarterly updates that include security patches and to release a hung I/O device.

The following systems and services will be unavailable and offline for hopefully no longer than 20 minutes.

  • astro  - logins, all email services including webmail, webpages based located in astro home directories
  • gemini - exported RAID files systems /san/dw (Mike Montgomery (ix.as.), /san/ne2  and /san/ne3 (Neal Evans' hick machines) /san/psg (austral.as.) 
  • ftp - anonymous ftp
  • bigbubba
07/12/16Solaris Quarterly Updates

12:24 PM All services restored.

12:00 PM (Noon) Our Solaris systems will be rebooted to apply Quarterly updates that include security patches and to release a hung I/O device.

The following systems and services will be unavailable and offline for hopefully no longer than 20 minutes.

  • astro  - logins, all email services including webmail, webpages based located in astro home directories
  • gemini - exported RAID files systems /san/dw (Mike Montgomery (ix.as.), /san/ne2  and /san/ne3 (Neal Evans' hick machines) /san/psg (austral.as.) 
  • ftp - anonymous ftp
  • bigbubba
12/12/15

Solaris Security Updates

  • 12 December @ 01:05 PM www.as.utexas.edu web sites passed basic testing (i.e., mounts from astro including home web pages work fine)
  • 12 December @ 12:24 PM Anonymous FTP test failed. ftp.as.utexas.edu did not remount the /ftp/pub share from astro on reboot. Manually mounted and successfully tested.
  • 12 December @ 12:23 PM test emails start showing up. 
  • 12 December @ 11:57 AM email testing began. 
  • 12 December @ 11:56 AM astro, ftp, bigbubba, solmaster, and gzone03, 
  • 12 December @ 11:56 AM gemini rebooted successfully with recommended and security patched applied
  • 12 December @ 10:39 AM - Zones BEs conflicted with global updates, requiring pkg update to be re-run on gemini (global zone).
  • 12 December @ 9:41 AM - Having trouble with some of the zones' boot environments (BE), but moving forward. Sorry for the delay 
  • 12 December @ 9:00 AM Most of our Solaris systems rebooted to apply updates that include security patches and new firewall features that we need to avoid quarantines by ISO. The following systems and services will be unavailable and offline for hopefully no longer than 20 minutes.
    • astro  - logins, all email services including webmail, webpages based located in astro home directories
    • gemini - exported RAID files systems /san/dw (Mike Montgomery (ix.as.), /san/ne2  and /san/ne3 (Neal Evans' hick machines) /san/psg (austral.as.) 
    • ftp - anonymous ftp
    • galactica - should have no user impact
    • gandhi - may take longer than 20 minutes to deal with Solaris 10 OS updates, which take longer than Solaris 11

12 December @ 9:41 AM - Having trouble with some of the zones' boot environments (BE), but moving forward.

  • 12 December @ 9:00 AM Most of our Solaris systems will be rebooted to apply updates that include security patches and new firewall features that we need to avoid quarantines by ISO. The following systems and services will be unavailable and offline for hopefully no longer than 20 minutes.
    • astro  - logins, all email services including webmail, webpages based located in astro home directories
    • gemini - exported RAID files systems /san/dw (Mike Montgomery (ix.as.), /san/ne2  and /san/ne3 (Neal Evans' hick machines) /san/psg (austral.as.) 
    • ftp - anonymous ftp
    • galactica - should have no user impact
    • gandhi - may take longer than 20 minutes to deal with Solaris 10 OS updates, which take longer than Solaris 11
12/02/15cerberus RAID fileserver7:00 AM - Cerberus, is going to require a reboot to update the systems software. This update is required so we can install a patch to the system. The patch may require a reboot as well. We will know as soon as the Oracle engineers evaluate a resolution to a potential security issues identified by ISO.
11/19/15

Astro email Unscheduled Maintenance

2:59 PM - Maintenance completed. Services restored.  

2:50 PM - astro email services will be taken off-line today to adjust performance tuning parameters. email services are expected to be back on-line within 20 minutes.

10/21/15 -

11/03/15

DocuShare Sever Emergency Maintenance

11/03 02:14 PM - DocuShare is back online in WRITE mode. Log files indicate that our backups are running without any problems.

11/02, 9:37 AM - The DocuShare server, ds.as.utexas.edu, has been recovered via the indomitable and resolute efforts of Paul Morris and Patrick Goetz (CNS Systems), to whom we owe prodigious gratitude for their invaluable work. DocuShare is currently online in READ-ONLY mode while we configure, initiate and test a cron based backup to cerberus, our Oracle RAID.

7:00 AM - ds.as.utexas.ed was taken offline, it's internal disks removed and copies were made. Booting from a Redhat DVD, the two mirrored disks were reviewed and found to be inconsistent. Recover efforts continue.

03:18 PM - Server attempted to run a disk check and hung at 11.4%. Rebooted and it hung again at 11.4% during the disk check. Will allow the process to run overnight. And re-evaluate in the morning.

02:10 PM - ds.as.utexas.edu, DocuShare server rebooted.

1:45 PM DocuShare server database engine crashed. Rebooted server.

10/13/15Solaris Sever Maintenance Updates

Security and maintenance updates will be applied to gemini, our Global Solaris server that hosts astro and ftp virtual machines.

7:15 AM - Maintenance completed. All services restored.

7:00 AM gemini zones shutdown; global zone rebooted

Services: Email, Webmail, astro logins, anonymous FTP, bigbubba logins, individual user web pages

7/15/2015

 7/13/2015

 

Astro email Critical Security Maintenance

The SSL security certificates must be updated for mail services on astro to provide compatibility with utexas.edu mail services. SSL Certificate configuration continues to be a recondite undertaking with a plethora of methodologies and a poverty of guidance. In short, this may take awhile.

12:45 PM astro email back online. The SSL certs were updated and tests indicated success installation and that astro email service now meets the higher standard of security required by Gmail and orher mail related services by properly presenting the intermediate SSL certificates.

12:15 PM astro email services will be taken off-line at and a new verified set of SSL certificates will be installed and tested.

9:50 PM astro email is back on line using the original configuration. Dovecot rejected the second configuration like a bitter pill being pushed down a sick cat's throat.

9:20PM. astro email services will be taken off-line at and a new convolution of contentious Secure Socket Layer certificates will be installed with great hope of success and triumphant exodus from this afflictive drama.

9:14 - After much research and many attempts to get astro's SSL certificates configured in a format that may be acceptable by Dovecot, the IMAP and POP email server software as we often say here in Texas, "I'm fixing to give another go at fixing the &$*^$%$#@ thing."

6:12 PM The recommended Certificate configuration did not work. Inserting the Intermediate CA cert per recommendations for Dovecot (our IMAP and POP service) resulted in an Cert / Private Key Mismatch. astro email is back on-line with the old configuration while I make another attempt with newly downloaded Root and Intermediate Certs from InCommon. astro email will be taken offline again as soon as I have can fetch the certs from the Certificate Authority and install them as a new configuration. If you have any patience left, please send it my way as my stockpile is running low.

5:00 PM astro email and webmail will be taken off-line for the SSL certificate updates.

5/13/2015Solaris Sever Maintenance Updates

Maintenance updates will be applied to gemini, our Global Solaris server that hosts astro and ftp virtual machines.

Services: Email, Webmail, astro logins, FTP.

5:48: Updates completed. All system services restored.

5:30 System reboot.

5/13/2015VM Server Emergency Security PatchServices: All departmental virtual machines will be power cycled once the patches are in place for the exploit in VENOM to be fixed.
 5/9/2015Astro email RAID Mounts Emergency Maintenance

Services: All astro email services including webmail will be unavailable while correct action is being taken to scrub the RAID pool for astro inboxes (/var/mail). Status shows the scrub should be completed around 12:45 PM today.

12:37 All services restored.

 

3/13/2015EMERGENCY RAID MAINTENANCE

Services: /opt/local, DHCP (laptop connections), Mathmatica, IDL, astro Webmail, etc. NOTE: Regular astro mail should continue work but may be slow.

14:10 All services restored

12:45 Determined system hangs were caused by the failure of one of the two systems disk which are mirrored. Paul Morris worked with Oracle Support and removed the failed disk from the mirror, rebooted, ran integrity tests and brought the RAID back online. Replacement disk is being shipped and will be installed next week.

9:25 RAID (cerberus) failed and went off-line. Oracle technical support is working the issue

2/25/15Astro RAID Mounts Emergency Maintenance

Services: All astro services including email, FTP and webmail will be unavailable for a short time starting at 6:45AM. This work is expected to take from 15 to 20 minutes.

6:50 astro and all related services back online

6:45 astro rebooted.

02/03/15Astro Email Scheduled Maintenance

Services: All astro email services including webmail will be unavailable for a short time starting at 5:30 PM today. This work is expected to take from 15 to 20 minutes.

5:38 all email services are back online, maintenance completed

5:30 astro email services taken off-line

01/07/15

Astro and Galactica Maintenance: Solaris Updates

Email, Webmail, FTP

/sans/* RAID shares

Services: All astro services including email, webmail, ftp along /sans/* partiion shares will be unavailable for a short time while astro and galactica are taken off-line to apply regular updates

9:10 FTP is now online; Maintenance completed.

8:23 All services expect FTP back online

7:30 astro and galactica taken off-line

12/16/14Astro Email - Unscheduled maintenance

Service: All email services including webmail

7:36 all email services restarting and available

7:30 astro email restart to clear out zombie processes

12/10/14Services

Service: printers and calendars.

10:25 - System functional again.

9:52 - Disc IO errors due causing system to hang.

12/9/14Services

Service: printers and calendars.

14:30 - Temporary system in place.

11:10 - System became unresponsive due to a hardware failure.  Migrating system to new hardware.

11/16/14cerberus RAID

Service: /opt/local, /san and other cerberus RAID mounts

NOTICE: Server is unreachable.

11:31 - Service restored.

11:05 - . RCA in progress.

11/16/14Astronomy Web Site

Service: All Astronomy web sites hosted on dept web server, affirmed

NOTICE: Sites unreachable. Server load 257.0 (should be in single digits or low double digits)

11:36 - Service restored.

10:32 - System hung after reboot.

10:20 - Preparing to reboot.

09:59 - Access to www.as.utexas.edu times out. Load excessively high.

11/16/14Web Mail

Service: Astronomy web mail

NOTICE: Server is unreachable.

11:32 - Service restored.

10:25 - Cannot access webmail.as.utexas.edu. RCA in progress.

11/07/14Astro Email

Service: Astro IMAP and POP email (reading)

NOTICE: Emergency maintenance: SSL 3.0 patch

07:52 - Patch successfully applied; Service restored

07:50 - Inbound and stored mail reading off-lined

11/05/14Astro Email

Service: All astro email

NOTICE: SSL 3.0 patch failure.

05/09/14Astro Email Blacklisting

Service: Outbound astro email

NOTICE: The de-blacklisting of astro may take hours to a few days to propagate

22:38 - Outbound email restored

21:55 - astro scheduled for reboot

21:07 - De-blacklisting request made

20:45 - Root caused identified, clean up initiated

19:15 - Outbound mail service off-lined

03/14/14

 
Astro Server Upgrade

Service: All astro services including email

03:14 - Mail Services resumed. SSH keys have changed. Please report any problems you might have.

07:00 - System taken off-line

01/01/14DocuShare Server Emergency Maintenance

Service: All DocuShare services

21:49 - UT ISO Notification of ds Bot/Worm compromise

22:00 - Border Quarantine set by Networking

15:16 - Efforts to reestablish services continue

11/27/13File Server (cerberus) Scheduled Maintenance

Service: Email and all other astro services

07:00 - cerberus taken offline and began updates

08:30 - cerberus hung on last update; Oracle Support contacted

10:00 - Oracle Support engineer finally gets involved

12:45 - Oracle Support engineer still working to resolve upgrade issues

14.30 - Maintenance completed.

11/02/13Astro Scheduled Maintenance

Service: Email and all other astro services

07:00 - astro taken offline.

09:45 - Solaris and email updates completed.

09:55 - astro is back online.

08/12/13Astro IMAP Emergency Maintenance

Service: Mail

13:06 - Services resumed.

11:46 - services being restarted again. Please do not use your email client.

11:30 - IMAP services being restarted due to system load issue.

08/10/13Solaris OS Scheduled Maintenance

Service: All Suns running Solaris OS

07:00 - Will begin rebooting systems.

08:48 - All workstation systems are back online.

08:15 - astro is back online.

08/10/13Astro Empergy Maintenance

Service: Email (astro)

07:00 - Begin applying recommended patches; Email and other services unavailable.

08:15 - Email and all other astro services are back online.

07/01/13Webmail Host Server Upgrade

Service: Webmail

9:15 - Maintenance finished.

9:00 - Server hosting Webmail will be down for a few minutes for a memory upgrade.

05/28/13Webserver Emergency Maintenance

Service: Webserver

19:59 - Partial functionality returned.

13:30 - Drive reconfiguration failure.

8:50 - Service functionality returned.

7:00 - Hardware failure.

05/09/13Astro Emergency Maintenance

Service: Mail

18:10 - Mail service operational.

16:29 - Mail is temporarily offline while allocation issues for the service are being resolved.

14:43 - Troubleshooting mail service errors.

05/03/13Vault Scheduled Maintenance

Service: Vault

8:00 - Server hosting Vault brought down to rebuild new drives.

04/29/13Astro Emergency Maintenance

Service: Mail

13:55 - Mail service operational.

10:09 - Mail service interruption

04/09/13Astro Emergency Maintenance

Service: Mail

11:45 - Mail service operational.

11:15 - Mail service interruption

  • No labels