Recently in “Outages” Category

Update: The migration went as planned, and we finished early. We may experience momentary server hiccups as we adjust the memory usage according to the server load.

We will be relocating the Movable Type blog installation to a brand new server location and moving the database to an external MySQL Enterprise server the morning of Tuesday, July 13. 

This will require a scheduled outage between 8 AM and 11 AM on Tuesday. We will try to maintain website access during most of that time. However, the need to place a freeze on file and database changes will require that we disable the admin interface. 

If all goes as planned, the entire three hours will not be necessary.

Update: 3/16 10:30 AM - The blog server is back up after the power outage. 

I had hoped to add more content to this weblog other than outage notices, and I will soon be posting updates on a planned move to a better server box and plans for migrating older blogs over to the Movable Type 4 Publishing Platform. However, we have two more OIT-wide outages to get through first. 

Outage #1: OIT has rescheduled February's power outage for Saturday, March 15 at 6 PM through March 16 at 10 PM. Most of servers at the main University data center, including the blog server, will be shut down for the installation of new electrical equipment. 

Outage #2: OIT will interrupt all campus networking services on Tuesday, March 18, from 5:00 AM to 8:00 AM. This outage is due to various maintenance activities being performed.

Both the statically-published blog sites and the database-driven MT admin interface will be inaccessible during these times. As usual, you can monitor the left column of the OIT page for the latest outages: http://oitweb.princeton.edu/

The power outage on February 9 at the main University data center has been postponed. The Department of Facilities has identified a problem that cannot be resolved in time to proceed with the scheduled installation of the new electrical equipment.

The Princeton University Blog Service servers should remain active throughout the weekend.

At some point in the future, Facilities will need to complete the work on the electrical upgrade to the data center. We will post an update to this blog they announce the new outage dates.

The administrative interface for the MT blog server was down for most of the day on Sunday, September 9. Static files were not affected.

The system partition for the server ran out of space, and that caused the database server to go down. I will be working with our Collaborative Services Group (CSG) over the next week or so to make sure this doesn’t happen again. We may have to schedule an outage within the next month to correct the disk space issue. Watch this space for updates.

I apologize for the length of this outage.

Update: As of 11 AM Sunday, power had been restored, and the web server and database server were started back up.

The Department of Facilities requires an electrical power outage at the main University data center to install new power cables from the data center to an electrical substation. This work is necessary to improve the electrical infrastructure and capacity at the data center.

To minimize the impact on the University community, the electrical work has been scheduled for Sunday, February 11, from 12:01 a.m. to noon.

This outage will directly impact the Princeton University Blog Service. All weblogs and the Movable Type administrative interface will be inaccessible during this time. Blog service should be restored some time between 6 AM and noon EST.

Outage Information: Most central computing services unavailable

OIT Unix Systems will reboot the blog server soon after 8 AM on Wednesday, August 23. This server reboot is necessary for operating system patches and minor changes to its network configuration. Service interruption should be minimal.

From the OIT Outage Announcement mailing list:

An extended power outage is planned in the 87 Prospect machine room from 7:00 p.m. on Friday, June 9, through 6:00 a.m. on Monday, June 12, 2006 to install a new electrical transformer.
 
OIT’s 87 Prospect computing facility, which is currently at maximum electrical capacity, will require installation of this large additional transformer to meet the University’s growing computing needs.
 
This machine room power outage will affect all OIT computing services. In addition, departmental services provided by servers hosted in the machine room at 87 Prospect Avenue will be affected. The campus community should expect central computing services to be unavailable throughout the outage.

The power outage will affect this weblog server. All blogs and the Movable Type admin interface will be unavailable during this time. We will post more information if it becomes available.

Update 3/30: The database migration went seamlessly, so now both the web server and the database server are running off of the same server box.

There will be an emergency outage of the blog server from 8 AM to 9 AM on Thursday, March 30.

Currently, the web server runs on a box named “Communitas,” and the database server runs on a box named “Gravitas.” Hackers recently compromised the web server on Gravitas; fortunately, this security breach did not affect the database.

We have decided to move the database over to Communitas. This will allow us to better monitor security because we only have to worry about one box.

The migration should take considerably less than one hour to complete. We will take the web server offline at 8 AM, then make a backup of the database. After we import the backup, we will bring the web and database servers back as soon as we can.

The blog server was down for at least twelve hours on Saturday. A required Apache server module had been accidentally deleted from the server, and the web server crashed some time after midnight.

I apologize for the inconvenience.

If the blog server goes down again for any extended period of time, you can check the Current Outages list on the OIT Help Desk main page for updates.

You can also subscribe to the new Help Desk Outage RSS feed .

Update: 12/28, 9:30 PM The upgrades and changes to the server installation proved more daunting than originally planned.

There was one last issue, related to the display of high-ASCII characters (ex. Asian or Cyrillic characters, accented characters, curly quotes) that should now be resolved.

You may need to empty your browser’s cache and restart your browser if you still see any scrambled characters on your blog pages.

It seems that there is a bug in the Apache 2 web server software that we had to compensate for by adding a line to the httpd configuration file (detailed explanation).

I apologize that the high-ASCII character problem persisted for an entire week. If you should come across any other problems related to your weblog, please bring them to our attention by e-mailing blogs@princeton.edu.