slideshowpro.com status

Investigating latency issues (resolved)

Brad Daily — Wed, 18 Jan 2012 17:54:49 +0000

FINAL UPDATE – 7PM ET – We have monitored all afternoon and the issue has not popped up again. Marking this one as resolved. Sorry for the downtime.

Original Post

We are seeing intermittent latency issues across SlideShowPro.com that is causing some pages, slideshows and API requests to run slowly or fail altogether. We’re looking into it and will provide updates as we have them.

UPDATE – 1:20PM ET – We’ve rebooted our main web server but are now having issues with our storage array remounting. We’ve taken the Director backend offline while we fix that.

UPDATE – 1:30PM ET – The storage has been remounted and we’ve reopened Director as well. Response times are back to normal. We will continue to monitor the situation over the next few hours to make sure it is completely resolved.

DNS outage affecting SlideShowPro.com (updated)

Brad Daily — Sun, 08 Jan 2012 05:39:25 +0000

UPDATE 12:20AM ET – All accounts seem to be up again. EasyDNS appears to be getting a handle on things, we’ll continue to monitor their progress.

Our DNS provider has been fighting off a denial of service attack today, and as such we are seeing issues with all SlideShowPro.com accounts. If you are not seeing your slideshows or the Director login, this is why. According to our DNS provider, the attack has been mitigated and they are restoring service as fast as they can. Unfortunately, all we can do is wait. We’ll post more information when we have it, but hopefully all accounts will be back online in the next hour.

Investigating upload issues (resolved)

Brad Daily — Tue, 03 Jan 2012 21:40:33 +0000

UPDATE: Uploads appear to be working again. Please contact support if you are still having trouble uploading.

We’ve heard some reports of uploads failing and are investigating.

Emergency maintenance: 3PM ET (complete)

Brad Daily — Tue, 20 Dec 2011 19:29:45 +0000

Sorry for the short notice, but our one remaining issue for the transfer is the database. As it is causing inconsistent and poor performance, we need to put SlideShowPro.com in read only mode why we try to retune, that way we can ensure that there will not be any data loss.. We’ll do that at 3PM ET (30 minutes from the time of this post). Again, sorry to do this in the middle of the day, but it’s a must.

Your slideshows should remain only during this time, only the Director backend will be unavailable. Hope to be done in 60-90 minutes.

3PM ET – Underway. Hope to be done in 60-90 minutes.

3:55PM ET - SlideShowPro.com accounts are back open. We’ll continue monitoring to see if this bit of tuning has helped. Thank you for your patience.

Transfer issues remain (updated)

Brad Daily — Tue, 20 Dec 2011 16:01:39 +0000

We are still dealing with some unforeseen issues from last night’s transfer. We are currently focusing our efforts on two particular things:

1) The database configuration needs some tweaking, as it is running much slower than the former setup. This is causing slowdowns in many places, and is our highest priority.

~~2) We also have sporadic reports of thumbnails not generating, and are also looking into that.~~ (The thumbnail issue is now resolved)

More updates when we have them. We are working as fast as we can to resolve these issues.

New servers still warming up

Brad Daily — Tue, 20 Dec 2011 12:20:17 +0000

While the transfer went off without a hitch overnight, the new servers had no cache to work with, so the onslaught of traffic this morning has caused some slowdowns here and there. We’re temporarily bringing some additional servers online to help alleviate the issue, and it should get better as the day goes on and the cache rebuilds. As always, stay tuned here for updates.

Maintenance underway (complete)

Brad Daily — Tue, 20 Dec 2011 04:01:20 +0000

3AM ET – The maintenance action is now complete. For a rundown of what happened, read the play-by-play below. Accounts should be reopening in the next 30-45 minutes. If you see any issues, please contact us.

————————————-

11PM ET – Our planned maintenance action is now underway. Updates will be posted here as the maintenance progresses. We expect SlideShowPro.com to be in read only mode for 3-4 hours. Your slideshows will remain available during this time.

11:20PM ET – We’ve waited a safe amount of time to ensure that all requests that were in process before the window began have now completed. Time for a final dump of the database, then we’ll transfer it to the new cluster.

11:40PM ET – The database has been transferred. We now enter into the longest portion of the maintenance window, as content is synced from the old system to the new.

12:30AM ET – Transfer is progressing, but a little slower than expected. Will probably be at least another 2 hours.

1:30AM ET – Things are progressing well, looks like the transfer will be done in the next hour or less. Once it is complete, we’ll do a few spot checks and then re-open all SlideShowPro.com accounts.

2:30AM ET – Still a few things transferring. Getting close now, hope to be be reopened in the next hour.

3:00AM ET – Transfer is complete. We’re doing some final checkups, and will be throwing the DNS switch soon.

3:20AM ET – DNS records have just been updated. All SlideShowPro.com accounts should be open in the next 30-45 minutes, as DNS propagates.

Maintenance window: December 19th, 11PM ET

Brad Daily — Mon, 19 Dec 2011 13:43:56 +0000

We’ve been working for the last few weeks on moving the SlideShowPro.com system to an updated hardware stack to alleviate some of the performance issues we’ve experienced in the last few months. We are now ready to move to the new hardware and will need a maintenance window in order to properly transfer the data. Here’s the plan…

At 11PM ET (what time is that for me?), all SlideShowPro.com accounts will be put in “read only” mode. You won’t be able to login, but your slideshows/API applications will still function, so your website will be unaffected. Once we are sure data is no longer being written to the system, we will do a final dump of the database and transfer it to the new setup. We’ll also do a final sync of the content files (images, videos, etc) from the old setup to the new. We do anticipate that this step may take some time, perhaps up to an hour and a half.

Once the sync of data and content is complete, we’ll perform checks to ensure that things have transferred as expected. Assuming that is successful, we will then modify the DNS for SlideShowPro.com so that it points at the new stack. Our DNS TTL (time to live) is set to 20 minutes, so once we make that change most accounts should be open back up in less than 30 minutes.

If all goes well, SlideShowPro.com should be back to normal in no more than 3 hours. It’s quite possible that it won’t take that long, but we want to provide a sufficient amount of time to ensure that everything is done correctly.

Stay tuned to this status blog for updates on our progress once the maintenance window has begun.

Connectivity issues (resolved)

Brad Daily — Tue, 06 Dec 2011 19:48:51 +0000

UPDATE: 3:20 PM ET – We rebooted the storage cluster and things are back to normal. Thanks for your patience.

We are again experiencing issues with the storage portion of our cluster, which is in turn causing some intermittent connectivity issues for all SlideShowPro.com accounts. We are on the issue and will update this post when things are back to normal.

Incident report: Connectivity issues (Nov 27-28, 2011)

Brad Daily — Tue, 29 Nov 2011 16:48:04 +0000

This post is a summary of the connectivity event that began on November 27th and ended on November 28th. We’ll cover what went wrong and what we are doing to ensure it doesn’t happen again.

What went wrong

Around 12PM ET on November 27th, our monitoring alerted us to higher than normal error rates and response times across the entire SlideShowPro.com network. We quickly determined that our storage array was faltering and was the root of the issue.

SlideShowPro.com is responsible for storing a large number of files. We currently manage over 5,000,000 photos and videos, totaling more than 4TB. In order to serve those files in an efficient way, we use a large storage appliance that connects to all of the servers in our cluster. While that storage appliance has served us well over the years, it has become obvious that it is showing its age from a hardware standpoint, as this incident further proved.

Isolating the issue was only the beginning. Next, we had to work with our provider to figure out exactly what was wrong with the storage appliance and how we could get it back online. Unfortunately, due to the size of the appliance, lack of specifics as to why it was faltering, and the timing of the incident — it was a holiday weekend here in the US — it took until noon on Monday to get a handle on the problem. Once we did, maintenance began on the storage appliance and it was finally rebooted around 2PM ET. Once the reboot was complete, response times returned to normal and errors ceased.

Going forward

After speaking with our provider, we believe that the fixes put in place yesterday will prevent this from happening in the near to medium future. Long term, it is time to upgrade our hardware, particularly for storage. Our provider has a roadmap and new hardware to move us in that direction, and we are already putting a plan together to do just that. We hope to be complete with the transition before the end of the year, and will provide updates here on the status blog as work progresses.

Finally, we want to apologize. We know that you depend on our service to power some of the most important portions of your website, and that’s a responsibility we take very seriously. We let you down yesterday and will work tirelessly to reduce the chances of this happening in the future. If this incident caused you inconvenience, please contact us and we’ll be glad to extend your subscription an extra month free of charge.