​Census fail – a recipe of poor planning, poor risk management and irresponsibility

According to an IBM insider, who was involved in the e-Census project that spectacularly failed on 9 August 2016, the true tale of why things went horribly wrong is a combination of factors compounded by bureaucratic thinking that actually stopped the ABS from using a more secure solution.

The best analysis of the chain of events that lead to the shutdown of the e-Census website has come from Patrick Gray, the publisher of the Risky Business podcast. Not only did the Australian Bureau of Statistics (ABS) choose to not take up DDoS protection from the network provider, Next-Gen Networks, but they also failed to keep the rulesets on their two firewalls in sync resulting in a problem when they restarted a firewall following the initial attack – which was actually quite small.

A look at data from Digital Attack Map suggests attacks coming into Australia were insignificant in global terms.

Australian Information Commissioner, Tim Pilgrim, said in a statement yesterday that "ASD advised me that the incident was a denial of service (DoS) attack and did not result in any unauthorised access to, or extraction of, any personal information".

Our source at IBM says his understanding of event fits with Gray's and that "there wasn't a major DDoS attack. I think they got scared and pulled the plug".

According to my insider IBM SoftLayer, the company's cloud platform, was not used. This is because the platform does not have IRAP certification. This is an assessment of services that can be used for the storage of government data.

"Any state or federal government agency require all remote computer installations that are not on government-owned premises to be certified to a certification called IRAP".

Our source says IBM already provides cloud services to other governments around the world who require a higher level of security certification than IRAP. But as IBM's competitors in the cloud business have already achieved IRAP certification, they have been reluctant to point this out.

"IRAP is actually a lower standard than the US government. IBM's SoftLayer data centres around the world are certified to the highest standards the US government adheres to. But because we don’t have, yet, IRAP certification for our Australia-based SoftLayer data centres in Sydney or Melbourne, any state or federal government information can’t be housed on them".

Our source says IBM has been working towards getting IRAP certification but have been forced to "tap dance and stall" while waiting. Their Sydney data centre is close to certification with others expected to follow.

The clarification here is that many people have been saying IBM was hosting the e-Census website. According to our source, this is not strictly correct. IBM provided a content distribution network (CDN), running on SoftLayer, for static content such as fixed text and images. This is similar to the services Akamai provides with clustered nodes distributed across the world.

But the actual e-Census application, which operates dynamically is not hosted by IBM. Our source suspects the application is being hosted on ABS' own systems.

"We couldn’t have actually hosted it. Even if we got IRAP certification, it's only happened in the last few weeks and this thing has been ready for months and months. For the most part, when the denial of service attacks happened you still got all of the static content because that was hosted by the CDN".

Our source says they were part of the IBM team that put in the original end-to-end bid for the e-Census solution including their performance testing, functional testing and quality management tools. But they lost that part of the bid, which went to Revolution IT.

Revolution IT is both an IBM and HP business partner and chose to use HP's load testing tools according to our source.

"I know the guys at Revolution IT and they're a good partner. I believe that they would have done the best job that they could given the specs they would have been told by the ABS".

The challenge, says our source, is that full end-to-end performance testing is difficult to execute. It's not possible to do a full test through the ABS public master firewall system as it's unlikely anyone would have been given sufficient access to do that. While some testing from outside the ABS' firewall would have taken place, this is more likely to have been user acceptance testing rather than major performance testing. Performing DDoS-style performance testing is extremely rare – something our source says he's never seen in his 30 years working in the IT industry.

What we do know is IBM was paid $9,606,725.00 to "Design, development and implementation of eCensus Solution 2016". And a look at documents pertaining to the 2011 census are clear in stating the solution IBM built then was to run on the ABS' own hardware.

No doubt the finger-pointing and blame game will continue. What seems certain is the intransigence of agencies being forced to only use certified systems when better solutions are available, a lack of foresight in expecting a DDoS attack and having suitable contingencies in place, and configuration errors in the ABS' firewalls all contributed to the embarrassing outage.

The Prime Minister has said that "heads will roll". Given the budget cuts made to the ABS over recent years, all while the ABS' head commands a $700,000 salary (which is over $200,000 more than Mr Turnbull receives) we wonder whose head will roll. Will the government simply look for a scapegoat? Certainly IBM and Revolution IT are easy targets.

But they are the wrong targets. The responsibility lies with the government and the ABS.

Join the CSO newsletter!

Error: Please check your email address.

Tags ABSIBMIBM SoftlayerDoS attackscensus 2016Australian GovernmentIRAPcyber securityasdDDoS ProtectionddosNextgen Networkse-censuscensusus governmentCDNsAustralian census

More about Australian Bureau of StatisticsHPRevolution IT

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Anthony Caruana

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts