News

Downtime Today – December 6, 2009 RESOLVED

This article is filed under: News

Update 11:26 12/6 EST (below) Today some OK Public customers have been experiencing some unexpected downtime for which we sincerely apologize.

This is unscheduled and was caused not by issues with our servers but by a mistake by the datacenter that we use. The gyst of the issue is that the datacenter launched a change to their internal infrastructure and in doing so, they incorrectly altered the DNS settings for domain names that resolve to servers within the datacenter. So basically OK Public’s servers and your sites are up and running just fine right now but, because of the DNS issue, people viewing your websites via the web aren’t being properly directed.

We are in contact with technicians at the datacenter and, although we don’t have an estimate time for when all sites will be back up, we will keep you posted here with any new information.

All the Best,
Eliot

Update 2:07 EST

Just received an ETA of 60-90 minutes from a datacenter administrator. Will add more info as it comes in.

Update 3:47 EST

I wanted to update everyone as it’s been 100 minutes since my last update in which I mentioned the datacenter had given me an ETA of 60-90 minutes to resolve the issue. Alas, there are still OK Public users experiencing issues.  We’ve been hounding them for info but unfortunately we have nothing new to report right now. I’ll post here as soon as I have any new information.

Update 8:08 EST

Unfortunately there is no update of significance at this time. At 5:10 today our datacenter said that the issue had been resolved. Here is the exact communication:

At this time, DNS issues have been resolved. Customers may still see some truncated data in their customer portal and this will be corrected within the next 24 hours.

We apologize for this disruption and thank you for your continued patronage and understanding.

That was at 5:10 and 3 hours later the issues aren’t resolved.

And here is the latest aknowledgement from an admin that I discovered in a forum, which was posted at 7:50:

Hey folks, we realize there are still issues with those customers who used to have geodns as slaves to their master servers. We’re still working on identifying and correcting the source of these issues. Sorry about the slowness of updates.

I know this doesn’t make it better as many of you are still having issues, but please know we’re doing the best we can on our end. I’ll post again with any additional information.

Update 11:26 EST

The DNS issues have been resolved by the datacenter and your sites should be coming online shortly if they aren’t already. I am confident that there shouldn’t be any additional issues caused by the overlying DNS problem but I will post again here if any arise. I will also post any additional information that I receive from the datacenter as to why this occured and plans for the future.

Thank you all for your patience and understanding. Again, I sincerely apologize for the downtime.

All the Best,
Eliot

Leave a Reply

Feel free to add any additional thoughts or bring up any questions.