The Case for Metadata Retention

The government's plan to force telecommunications providers to retain a set of metadata for every person has privacy advocates up in arms and police and security agencies telling us that this legislation is essential for fight crime in the 21st century.

So, who is right?

In this two part series we look at both sides of the argument.

Tim Morris is an Assistant Commissioner of the Australian Federal Police. With over 30 years of experience investigating and solving crimes, he can speak from a position of authority on what it takes to catch and prosecute criminals.

Morris was one of the keynote speakers at the Tech Leaders forum held in Sydney in February 2015. He discussed the reasons the proposed data retention legislation was necessary for the AFP to pursue criminals and resolve crimes.

He opened his presentation saying "metadata continues to play a central role in most successful crime investigations".

He cited statistics such as 92% of counter-terrorism, 87% of child protection and 79% of organised crime had metadata a central element of the investigations. However, he noted that the data retention processes in different industries and companies were inconsistent and that this hampered the police's efforts.

"The value of telecommunications data in protecting public safety is, from an AFP perspective, indisputable," he said.

In Morris' view we are now at a tipping point where the need to retain metadata is required to fight modern criminal activity.

Read more: The week in security: Data retention looms, Superfish gutted

According to Morris, the proposed bill does not add any new powers or access to police agencies. "The bill simply obliges companies to retain a limited dataset for a minimum of two years. The limited range of metadata is limited to information about a communication; the who, the where and the when. Not the content or the substance or the substance of a communication," he said.

As examples of what the AFP mean by metadata Morris noted that the phone numbers and duration of calls were included in the data set but not the content. Similarly, with email it would included the email addresses of the senders and recipients but not subject lines or body text.

"The AFP wants industry to also retain the IP address allocated to a user's device, which is a critical piece of data for law enforcement and security agencies," he added.

Morris also took the opportunity during his speech to address some of the concerns raised through the media and other channels. For example, there are some parties that say the proposed legislation places the entire population under mass surveillance.

"Data sitting on a carrier's network is not mass surveillance and we are not speculatively spying on people," he said. He also highlighted that the AFP's role was to investigate people suspected of having "committed individual criminal acts" and to not indiscriminately trawl through data hoping to find potential criminals.

"Agencies can only access this data in limited circumstances, on a case by case basis, where it's reasonably necessary for a lawful purpose," he said.

Access to the retained metadata would require an investigating officer to receive sign-off from a senior, commissioned officer - individual officers won’t have unfettered access to the metadata. And the existing warrant process, which Morris noted was rightly strenuous and under judicial oversight, would still be needed in order to access the actual content of any communication.

Requests for the metadata would be subject to audit by the Ombudsman, who would have increased rights of inspection, as well as ministerial oversight, Senate Estimates, parliamentary committee enquiries and other bodies.

Read more: Centrify expands identity management to protect big-data honeypots

Under the current metadata access regime, Morris said 54000 requests for data were made against about 44 million connected devices.

According to Morris, the new legislation represents a tightening in which agencies will have access to the data with only those who have a "clear operational investigative need".

The currently definition of agencies who can ask for access will be replaced by a short list of key agencies who will specifically have access with new agencies only added if they satisfy specific criteria and are approved by the Attorney-General.

On the question of cost, Morris told the audience that the proposed metadata retention regime is being imposed "for national interest reasons and, as such, the government is prepared to pay a reasonable proportion of the upfront cost associated with the data retention scheme".

Currently, the AFP pays telecommunications providers about $1.5M for access to metadata so that the cost of access to this data isn’t passed on to customers.

Addressing the issue of why metadata would need to be retained by telecommunications providers for a minimum of two years, Morris said, based on experience, there was no correlation between the length of time data was held and its usefulness. He also noted that the AFP only requested data it knew was available so, in many cases, data more than a year old was rarely requested as the AFP knew it would not be available so concerns raised about the two year period being in excess of current requirements were based on an incorrect assumption.

Many crimes are only identified many months, or in some case years, after they have occurred.

Morris highlighted the importance of the various elements of the proposed legislation by using three different cases to illustrate the AFP's position. Operation Pendennis, Operation Inca and Operation Drakensberg. These terrorism, organized crime and child exploitation cases were successful cases where metadata was used to successfully find and prosecute criminals.

He also noted, in Operation Drakensberg which was a multi-year international investigation that originated in the UK and it took two years for the UK police to send the case to the AFP. The metadata required to investigate 41% of the Australian offenders was not available as carriers did not retain the data for a sufficient period. As a result, none of those 41% of potential offenders were investigated.

We asked Morris whether the AFP would be selective in the crimes it would investigate using metadata and whether software and media piracy was a crime the AFP would investigate using metadata.

"One of the threshold tests in the Act is 'is it reasonably necessary?'. Let's say it's a trivial or minor offence. You've still got to pass that threshold test. Is the intrusion to get someone's data reasonably necessary? The AFP is not interested in someone sitting in their lounge room torrenting Game of Thrones. We're not going to have a taskforce come out to get you based on metadata that we've collected".

In his concluding comments, Morris noted that the AFP was committed to protecting the privacy of individuals but that "the AFP can not support the right of anonymity, especially when it becomes related to unlawful activity".

Image from

This article is brought to you by Enex TestLab, content directors for CSO Australia.

Upcoming IT Security Events

March 3rd, March 5th, March 9th 2015

Join CSO for the day@#csoperspectives and hear from @kimzetter @LeviathanSec

3 International Keynote speakers, 36 Key IT Security Industry Speaker, 21 Exhibitors, Security Analysts and many more.. Register today

Dont miss one of the biggest IT Security events in ANZ (registration is free, but seats are limited)

Join the CSO newsletter!

Error: Please check your email address.

Tags tech leadersAustralian Federal PoliceOperation DrakensbergAFPTim MorrisgovernmentOmbudsmanTelecommunicationsCSO Australiadatasetmetadata retention

More about Attorney-GeneralAustralian Federal PoliceCSOEnex TestLabFederal PoliceIT Security

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Anthony Caruana

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts