According to an IBM insider, who was involved in the e-Census project that spectacularly failed on 9 August 2016, the true tale of why things went horribly wrong is a combination of factors compounded by bureaucratic thinking that actually stopped the ABS from using a more secure solution.
The best analysis of the chain of events that lead to the shutdown of the e-Census website has come from Patrick Gray, the publisher of the Risky Business podcast. Not only did the Australian Bureau of Statistics (ABS) choose to not take up DDoS protection from the network provider, Next-Gen Networks, but they also failed to keep the rulesets on their two firewalls in sync resulting in a problem when they restarted a firewall following the initial attack – which was actually quite small.
A look at data from Digital Attack Map suggests attacks coming into Australia were insignificant in global terms.
Australian Information Commissioner, Tim Pilgrim, said in a statement yesterday that "ASD advised me that the incident was a denial of service (DoS) attack and did not result in any unauthorised access to, or extraction of, any personal information".
Our source at IBM says his understanding of event fits with Gray's and that "there wasn't a major DDoS attack. I think they got scared and pulled the plug".
According to my insider IBM SoftLayer, the company's cloud platform, was not used. This is because the platform does not have IRAP certification. This is an assessment of services that can be used for the storage of government data.
"Any state or federal government agency require all remote computer installations that are not on government-owned premises to be certified to a certification called IRAP".
Our source says IBM already provides cloud services to other governments around the world who require a higher level of security certification than IRAP. But as IBM's competitors in the cloud business have already achieved IRAP certification, they have been reluctant to point this out.
"IRAP is actually a lower standard than the US government. IBM's SoftLayer data centres around the world are certified to the highest standards the US government adheres to. But because we don’t have, yet, IRAP certification for our Australia-based SoftLayer data centres in Sydney or Melbourne, any state or federal government information can’t be housed on them".
Our source says IBM has been working towards getting IRAP certification but have been forced to "tap dance and stall" while waiting. Their Sydney data centre is close to certification with others expected to follow.
The clarification here is that many people have been saying IBM was hosting the e-Census website. According to our source, this is not strictly correct. IBM provided a content distribution network (CDN), running on SoftLayer, for static content such as fixed text and images. This is similar to the services Akamai provides with clustered nodes distributed across the world.
But the actual e-Census application, which operates dynamically is not hosted by IBM. Our source suspects the application is being hosted on ABS' own systems.
"We couldn’t have actually hosted it. Even if we got IRAP certification, it's only happened in the last few weeks and this thing has been ready for months and months. For the most part, when the denial of service attacks happened you still got all of the static content because that was hosted by the CDN".
Our source says they were part of the IBM team that put in the original end-to-end bid for the e-Census solution including their performance testing, functional testing and quality management tools. But they lost that part of the bid, which went to Revolution IT.
Revolution IT is both an IBM and HP business partner and chose to use HP's load testing tools according to our source.
"I know the guys at Revolution IT and they're a good partner. I believe that they would have done the best job that they could given the specs they would have been told by the ABS".
The challenge, says our source, is that full end-to-end performance testing is difficult to execute. It's not possible to do a full test through the ABS public master firewall system as it's unlikely anyone would have been given sufficient access to do that. While some testing from outside the ABS' firewall would have taken place, this is more likely to have been user acceptance testing rather than major performance testing. Performing DDoS-style performance testing is extremely rare – something our source says he's never seen in his 30 years working in the IT industry.
What we do know is IBM was paid $9,606,725.00 to "Design, development and implementation of eCensus Solution 2016". And a look at documents pertaining to the 2011 census are clear in stating the solution IBM built then was to run on the ABS' own hardware.
No doubt the finger-pointing and blame game will continue. What seems certain is the intransigence of agencies being forced to only use certified systems when better solutions are available, a lack of foresight in expecting a DDoS attack and having suitable contingencies in place, and configuration errors in the ABS' firewalls all contributed to the embarrassing outage.
The Prime Minister has said that "heads will roll". Given the budget cuts made to the ABS over recent years, all while the ABS' head commands a $700,000 salary (which is over $200,000 more than Mr Turnbull receives) we wonder whose head will roll. Will the government simply look for a scapegoat? Certainly IBM and Revolution IT are easy targets.
But they are the wrong targets. The responsibility lies with the government and the ABS.