Database Outage

We’re very sorry to report that we had a database outage over the weekend. We filled the disks on the database server, which meant that we were unable to store new location data as it came in.

The problem started at around 1PM Pacific time yesterday, and was resolved this morning at around 8:30AM. During that time we lost any location data that was sent to us. The database is the “life-blood” of Whereoscope — without it we can’t store new location data, can’t sign up new users, etc. Basically every aspect of the system was affected in some way.

To make matters worse, we didn’t find out about it for some time due to a problem with our monitoring.

So first of all, let me sincerely and personally apologize to everyone out there relying on Whereoscope. I dropped the ball on this, and it’s something we’re taking very seriously.

Secondly, let me assure you that we’re putting in place new measures to ensure that this never happens again. I have been working closely with our sysadmin team on this. We’ve got the system back on the air for now, and have scheduled a disk upgrade for tonight. We’ve also resolved that issue with our monitoring, so we should get notified about this before we fill the disks next time, so that we can take steps to avoid the problem. Because of the disk upgrade, there will be a short downtime this evening at 1AM Pacific time, lasting about half an hour.

Once again, we’re really sorry about this, and I thank you all for your patience while we work through these issues.

James Gregory,
CTO & Co-Founder.