Database Outage

We’re very sorry to report that we had a database outage over the weekend. We filled the disks on the database server, which meant that we were unable to store new location data as it came in.

The problem started at around 1PM Pacific time yesterday, and was resolved this morning at around 8:30AM. During that time we lost any location data that was sent to us. The database is the “life-blood” of Whereoscope — without it we can’t store new location data, can’t sign up new users, etc. Basically every aspect of the system was affected in some way.

To make matters worse, we didn’t find out about it for some time due to a problem with our monitoring.

So first of all, let me sincerely and personally apologize to everyone out there relying on Whereoscope. I dropped the ball on this, and it’s something we’re taking very seriously.

Secondly, let me assure you that we’re putting in place new measures to ensure that this never happens again. I have been working closely with our sysadmin team on this. We’ve got the system back on the air for now, and have scheduled a disk upgrade for tonight. We’ve also resolved that issue with our monitoring, so we should get notified about this before we fill the disks next time, so that we can take steps to avoid the problem. Because of the disk upgrade, there will be a short downtime this evening at 1AM Pacific time, lasting about half an hour.

Once again, we’re really sorry about this, and I thank you all for your patience while we work through these issues.

James Gregory,
CTO & Co-Founder.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: