State of the Grid 2010
Hi, my name is Matt Jones and I’m on the Product Team here at (mt). I want to close out 2010 by making sure everyone who wants to stay informed about our (gs) Grid-Service has the latest information on the status of the platform. If you’re unfamiliar with the (gs), it is a cluster-based hosting platform that we launched in late 2006. The (gs) is home to over 400,000 domains serving up more than half a billion web hits per day. At launch, the (gs) gave birth to a new way of thinking about on-demand web hosting and was a big departure from traditional shared hosting technology. We set expectations high and in some cases over-delivered but the system itself has not been without its share of problems.
In the past we have communicated a great deal of transparency about the Grid, conveying its success and its shortcomings, but in 2010 we fell short on the frequency of our communications. Users of the Grid deserve to have all the information – whether good, bad, or ugly and I’m committed to delivering the details. Read on to see progress this year with the (gs) and upcoming improvements for 2011.
2010 in Review
Despite the wide adoption of the (gs) platform, as we came into 2010 the stability of web, email, and MySQL were improved, but not up to the (mt) standard. We hear the sentiments on Twitter and blogs, and we surveyed all of our customers this year, and while the vast majority of users remained happy, there were far too many that experienced frustrating downtime, security and service interruption issues. The top three issues our users asked us to focus on were Uptime, Stability and Security. With these as our prime targets, we’ve continued to make improvements to the (gs) platform in 2010. The following is a sample of some of the biggest changes this year:
Security Improvements
2010 brought a drastic increase in malicious attacks on hosting providers – more than we’ve ever seen. While many large web hosts felt similar pain, we found our (gs) platform to be a particularly attractive target due to its size and unique architecture. Many of our customers suffered and, frankly this is just not acceptable to us at (mt). To mitigate and repair the issues, we formed an internal, multi-disciplinary security team who led efforts to drastically improve security on the platform.
These improvements, many of which we’ve talked about before included: Ongoing security scans and greater automated detection of the most common exploits across the entire (gs), enhanced blocking of internet addresses that try to hack the Grid, and several infrastructure and configuration level fixes. We’re happy to mention that though attacks continue, as they do at every hosting company, we’ve seen a 77% drop in affected domains since October. Exploit counts continue to drop and we estimate about 0.7% of (gs) domains are experiencing problems. The best way to keep your websites safe is to follow good practices like keeping your software up-to-date and changing your passwords, but (mt) will be watching your back.
Uptime & Stability
We have always monitored server health, but at this scale traditional ‘best practices’ become unmanageable. We needed to step up and anticipate problems before your sites were affected, so we completely rebuilt our internal monitoring systems to give us the greatest possible visibility into every aspect of the Grid architecture. We’ve built in specific alerts for leading indicators to performance issues – giving us the visibility we need to minimize or prevent downtime from occurring. Today, we’re continuously identifying and fixing issues that you will never notice.
Updated Email Spam Filters
With over half a million mailboxes on the (gs) system, spam email was becoming an issue that not only annoyed our users, but created noticeable performance issues on the (gs). We worked with Cloudmark, the industry’s fastest, most accurate and most scalable spam, phishing and virus protection system, to update all user mailboxes. Filtering is now better than ever, but if you are still experiencing problems please contact support and let us know immediately.
That’s not everything that changed in 2010 but it definitely represents the highs and lows. Now we want to share with you some highlights that are on our roadmap for the Grid for the early half of the year.
2011: Coming Soon
Faster Web and Email
(mt) Engineers have been working hard to optimize responsiveness and reliability for the most important services: Web and Email. By changing our load balancing scheme and email architecture we expect mail to be between 1.5 and 3 times faster. We don’t have numbers on the web response yet, but we will be discussing the geeky details in the user forums. Incremental roll outs will begin in early January and we will share more specific details once they get started.
In addition, we are working to decrease page load times by adding Solid State Drives (SSD) to all storage segments. The SSDs will let us serve more content from cache on disks that respond many times faster than normal hard drives. This has already decreased load times for the newest clusters and we are evaluating the best way to roll this out to all clusters.
Beyond making these performance improvements we’re also actively investigating ways to separate our email and hosting architectures. To make a long story short, we’re convinced this is the way to go to improve the reliability and performance of the Grid and offer a world-class email solution at the same time. Again, we’ll provide many more details as research and discovery phases conclude.
Easier Software Updates
One of the best ways to stay secure on the internet is to keep your software up-to-date.
The Grid’s custom Linux architecture has made updating software without breaking user sites or creating customer downtime very difficult. Many users have asked for newer versions so they can use the latest and greatest applications, and because they are as concerned as we are about the security benefits of up-to-date software. The next cluster will launch with completely updated OS, PHP, and MySQL and we’re continually working to update software across all clusters.
What About (cs)?
In 2009 we announced that we were working on the successor to the (gs)… A product called the (cs) Cluster-Server. This was exciting news to users on the (gs), but its now almost 2011, and the (cs) still hasn’t been rolled out. Understandably, our users are wondering what the hold up is.
Put simply, we chose to dedicate our resources to improving the hosting experience for our existing (gs) customers, numbering in the tens of thousands before we were comfortable working on something new. That said, many of the (gs) improvements you’ve felt in 2010 as well as the upcoming enhancements for 2011 were originally slated to be a part of (cs). We felt the benefits to providing improvements for existing users far outweighed the benefits of creating a brand new system, and overall Grid performance and Stability improved in 2010 because of it.
We haven’t yet delivered on our promises with regards to the (cs). We want to make it quite clear that there isn’t anything more important to us than launching (cs) and continuing to make the Grid a more reliable, secure, and feature-rich service. In the past couple months we have realigned and strengthened our teams and shifted focus to make this happen. We’re in much better shape than ever before to make these promises a reality. We know that actions speak louder than words, so stay tuned. We’re excited for what’s in store for 2011.
You’ll Be Hearing From Us…
There are many other projects in progress that we’re excited to talk about, but it’s a bit too soon to share details. Expect product updates on the weblog on launch days and more detailed conversations in the user forums. Support agents, engineers, and managers are in the forums every day. Don’t be shy!
As we head into 2011 I want to re-emphasize that I’m committed now more than ever to improving the Grid product and to launching (cs). In fact, it’s one of my resolutions for the New Year! (Well, that, and finish my first Ironman Triathlon.)
I would love your feedback on anything mentioned above or questions about anything else that’s on your mind. You can leave a comment here on the blog, find me on Twitter @ehmatt, or join me in the (gs) User Forums.
Thank you for your continued loyalty in (mt) Media Temple.
Happy New Year,
Matt Jones
