September 2010 Archives

Amok time (resolved)

Update: After Tuesday morning's system configuration changes, the time on the blog server does not appear to be drifting.


For the past couple of weeks, the system time on the blog server has been drifting away from actual time at a rate of approximately 1.5 minutes for every minute of real time.

We suspect the issue has something to do with the interaction of the Linux kernel with the virtual machine (VM) server software. We are uncertain when this problem actually began. We are still waiting for an official response to a support request sent last week to VMware, the VM software vendor.

As a workaround, the server is now regularly resetting its system time against a network time server.

The problem has likely manifested itself in the following ways:

  • Inaccurate entry post date & times and inaccurate comment post dates & times
  • Scheduled posts that either post too early, too late, or not at all
  • Inaccurate dates in RSS feeds
  • Periodic page rebuilds not executing properly

Until this problem is fully resolved, I recommend the following workarounds:

  • Manually backdating entries according the the correct date and time
  • Not relying on scheduled publishing and manually verifying that items were published as scheduled
  • Clicking the "Publish Site" button in the admin interface if you notice that one of your pages has not updated with new content (new comments, for example).

I apologize for the inconvenience and hope that we will have a resolution to this problem soon.