Inside American Eagle Outfitter's 8-Day website nightmare

What happens when a business suffers an 'unprecedented outage'

HANG IN THERE. That was the first plea American Eagle Outfitters offered its nearly 1.5 million Facebook friends three days after its trio of e-commerce sites had crashed.

The online retail outlets -, and, which target teens and kids-s uffered what American Eagle Outfitters (AEO) termed a "major incident" on July 19. "This has caused all of our sites to go down," a Facebook update said on July 22, at 1:26 p.m. "We're working together 24/7 to get them up and running as soon as possible. Hang in there....we'll make it up to you. Check back soon."

Those back-to-school shoppers who checked back too soon at the sites received this message: "We're making updates to our sites." This was no planned maintenance, however. Behind the scenes, AEO and its external hosting provider IBM were scrambling for several days as the sites stayed dark.

The reasons for the four-day outage (and subsequent four more days of technical aftershocks) related to backup and disaster-recovery technologies and processes that failed, according to several articles on retail tech site

As the articles describe, atypical and concurrent failures with IBM's hosting servers and backup plans as well as with Oracle's Data Guard utility program ultimately proved to be the sources of problems. (Stephanie Oschwald, a spokesperson for Pittsburgh-based AEO, did not return several messages seeking comment. IBM and Oracle declined to comment for this article.)

[ Read about e-tailer's $1.6 million pricing mistake ]

On July 23, at 5:39 p.m., AEO announced on Facebook that the sites were A-OK. "We had a few glitches but we're back! We missed you while we were gone. Free shipping now thru Monday, July 26 11:59 PST. Automatic on your entire purchase (US and Canada only)."

However, noted, four more days of technological healing lie in wait for the AEO site, "when it limped along with various parts not functioning until Tuesday afternoon, July 27th."

An Unprecedented Outage

The shock of a 100-hour e-commerce outage now over and the damage done to their critical back-to-school sales period, executives of the $3 billion AEO have to be left scratching their heads in frustration--and wondering just how much "teeth" they had written into their service-level agreements (SLAs) with IBM. (For the record, reported that AEO had recently filled the position of "Manager--Business Continuity & Disaster Recovery," and not a moment too soon.)

AEO's websites weren't the first to go down unplanned, nor will they be the last. Dating back to the mid-1990s, history is littered with tales of site outages, complex IT fixes and lost revenue--embarrassing for the companies and career killers for those in charge. Any unplanned downtime, for instance, is still reported with as much zeal as a Jennifer Aniston "new boyfriend" sighting.

But the duration of AEO's site outages could be considered unprecedented for an e-retailing brand.

"In my career, I can't remember a mission-critical application that was down for four days," says Paula Rosenblum, an analyst at RSR Research and former retail CIO. "It's almost impossible to conceive--and I say that not as an analyst but as a former CIO."

Deals, Discounts and Disappointment

The timing couldn't have been worse for AEO, at the outset of its "back to school" summer push. In its stores, AEO's July performance was a "disappointment," according to a Zacks Investment Research report on retailers. The retailer had also lowered its quarterly earnings outlook in July.

The company not only lost online revenue during the outage, but also attempted to make up for the downtime by offering its customers deep promotions--such as this deal promoted on its Facebook page on July 30: "Thanks for hanging in there while our site was down...this Friday & Saturday (7/30 & 7/31) online only take 25% off your entire order & we'll throw in free shipping, too!... Thanks again for hanging in there!"

Brian Walker, a Forrester Research principal analyst who follows e-commerce, doesn't term the AEO outages unprecedented, but unusual. "Sites routinely will become slow and the customer experience and conversion rate may be affected," he says via e-mail, "but to be down for four days has a significant business and customer impact."

AEO's "direct businesses" (those three websites) rung up more than $340 million in sales in fiscal 2009, which accounted for approximately 12 percent of the retailer's overall sales.

Outsourcing E-Commerce Ops: Deal or No Deal?

Outsourcing e-commerce operations in retail is not a new proposition--and IBM and Oracle, for that matter, are not fly-by-night operations. "Using managed services or SaaS solutions for all or key parts of the e-commerce platform is routine today," says Forrester's Walker. "On average, eight hosts will be involved in supporting a typical e-commerce transaction."

Even with the site outages, the outsourcing proposition is still sound, Walker says. "There is no guarantee that doing this in-house is any better," he opines, "and in many ways can be much harder to do as resources, budgets and skillsets may be harder to have at the right levels."

However, it's important that a company outsourcing its operations not abdicate every decision and responsibility over to the hosting company. "Often in these situations there will be some shared responsibility between the business and IT-support staffs," Walker says. Trouble happens, he says, when something as simple as a failure to communicate an upcoming big promotion or event that will likely result in a spike in online traffic big enough to take down a website.

Interestingly, during the sites' outages, AEO's mobile site was still functional, according to RSR Research's Rosenblum says that's a "good news / bad news" situation. The good news: The mobile site was up, and AEO could have done a customer redirect from the online site that was down to the mobile site, if they chose. The bad news: It would appear that AEO is running separate, siloed systems (for customer databases and queries, for instance) for what are, essentially, similar e-commerce channels, Rosenblum says.

"Our research shows us that too many retailers are in a similar situation," she says. "They spend an inordinate amount of scarce resources--IT and human--forcing multiple channels into synch. The opportunity costs associated with these activities are incalculable."

Forrester's Walker advises other retailers to use AEO's outage as a wake-up call. "Get your responsibilities, processes and systems together now, test them, and correct as needed," he says. "Having a clear plan in and of itself will pay huge dividends in the event something happens, and with the technology and architectures we have available today this can be done at a reasonable cost."

"Consider the downside of having the business offline for a few days during the holidays," he adds, "and the ROI is pretty clear."

Do you Tweet? Follow me on Twitter @twailgum. Follow everything from on Twitter @CIOonline.

Join the CSO newsletter!

Error: Please check your email address.

Tags e-commerceUtilitiesbackupsoftwareAmerican Eagle Outfittersinternet

More about Amazon.comAmazon Web ServicesAmerican Eagle OutfittersFacebookForrester ResearchIBM AustraliaIBM AustraliaOracleSalesforce.comYahoo

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Thomas Wailgum

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts