Inside Symantec's top secret Melbourne digital certificate authority

The 'Ghostbusters' jokes were flying thick and fast as the facility's dozen-strong group of assembled media entered the facility and descended to meet the 'Key Master' – yes, that is his real title – down fire stairs and through one, then two, then three ASIO-rated fireproof doors secured with a variety of biometric, card and physical controls backed by a dizzying series of access rules.

It's an intimidating welcome to Symantec's high-security digital certificate issuing facility, a nondescript building tucked into a Melbourne suburb from which billions of secure Internet transactions are validated every day. It is one of four such Symantec facilities around the world, employing around 80 staff in the task of validating the identities of Web sites and the people that want to protect them.

Security is the name of the game inside the facility, where staff regularly deal with door-access rules so complicated that that they would make even the Sirius Cybernetics Corporation blush.

No more than one person may pass through the door at a time, those rules dictate, and entrances and exits are tracked and logged so if anybody tailgates another person into a room, they cannot exit it without a physical rescue by the facility's guards. That rule alone had caught out more than a few employees who might, naturally, follow colleagues through an open door when going on a toilet break – only to find they couldn't get out without an embarrassing extraction.

We eventually entered a windowless room where the almost imperceptible rush of circulated air and a few optimistically placed pot plants offered a glimpse of green to a dozen workers in this Tier 3 high-security data centre environment.

Heat sensors in the ceilings continually count the number of bodies in the room, raising raise alarms if it doesn't match the number of people that have swiped in.

Alarmed vibration sensors in the walls, ceilings and floors raised the inevitable questions about 'Mission Impossible' styled infiltrations, although our guides – including Symantec senior principal systems engineer Nick Savvides and the Key Master, who like that dentist in the TV commercials cannot actually be named – point out that the air-venting system has been designed with so many roundabouts and discontinuous segments that you'd more than likely just get lost up there.

Its roof, too, is protected, with military-grade secure mesh built into the design and an array of roof-mounted sensors that raise alerts even when a pigeon alights. Security, data and power cabling throughout the facility is separated into three colour-coded networks running over physically separated trays, with all fibre runs required to be unbroken and exposed, with sensors able to detect if there is any attempt to tap into the networks.

These and other protections make the facility a textbook exercise in military-grade data-centre security, but the physical protections are only the setting for the everyday management of digital certificates – the basis of trust on the Internet for their use in the public key infrastructure (PKI) system supporting SSL-enabled security.

The consequences of a security breach on an operation like this have already been played out in graphic detail in the media: Dutch certificate authority DigiNotar ceased operations within a month after it was compromised in 2011, and others have met similar fates after hackers successfully breached their protections and stolen digital certificates that allow them to trick Internet users into believing they are legitimate certificate owners.

There is, unsurprisingly, zero tolerance for lax security in the Symantec facility – hence the high level of alarming and surveillance. "We can see when you go to the toilet," Savvides points out when one journalist asks the way. "Not actually in the toilet," he adds – only half-jokingly – "but we know when you're going down the hall."

The Key Master steps up. For all the physical protections built into the facility – part of a global Symantec network that processes over 14 billion authentication transactions every day – even higher procedural security is in place during the top-secret 'Key Ceremony' in which new public and private root keys – the foundation of PKI-based digital certificate hierarchies – are created.

Savvides won't say when or how frequently these key ceremonies are held, but when they're on he, the Key Master and others steward an exacting process that is carefully scripted, recorded on video and painstakingly executed – and can, we learn, run for up to two weeks.

The ceremonies are held in a nondescript room that lies behind two additional military-grade secure doors. For all its import, the white walls, floor and ceiling would look unremarkable in any office building in Australia. However, Savvides tells us, apart from the other existing protections, the several desktop computers and printers on the desk are not connected to the Internet and the only communication with the outside world is through the single phone extension.

Once staff enter the room for a key ceremony, they might not leave for an entire eight-hour day – and often come back the next day, and the next, until the key-activating script has been completed. Food is not allowed, although each person in the room can have one drink with them. Lunch and even toilet breaks pose additional complexities, with staff required to pack up and lock everything in the room before leaving it for any reason.

"We can end up with two people stuck in here for eight hours," Savvides says. "It can be pretty hard on us."

Those eight-hour shifts are filled with mind-boggling minutiae, carefully and sequentially executed according to printed scripts that can run to 600 pages or more. Written by the Key Master in conjunction with Symantec's clients, they document every single mouse movement, key press, and checkpoint required throughout the process of adding a new trusted domain to the Internet.

Even one mismatched serial number is cause for aborting the process, while subsequent discovery of any human error during the process will force Symantec to revoke the certificate and the process to be checked and started over. Documents must be painstakingly verified, serialised USB drives containing private keys carefully managed and triple-checked to preserve the integrity of the process.

The process ranges from 45 minutes to renew a digital certificate, to four hours to register a user and two weeks to set up a complete digital-certificate hierarchy for a government or other top-level entity. It may be thankless, painstaking work – but this is the price of Internet security, and it's happening day in and day out to ensure what happened to DigiNotar and other certificate authorities doesn't happen again.

As we leave the ceremony room, filing through one secure door after another, someone asks how staff find everyday life in such a locked-down, unforgiving environment.

That sort of thing, Savvides says, is addressed early in the interview process. "The people we select know what they're getting into," he laughs. "It does take a certain type."

This article is brought to you by Enex TestLab, content directors for CSO Australia.

Join the CSO newsletter!

Error: Please check your email address.

Tags symantecsecuritydigital certificates

More about ASIOCSOEnex TestLabGalaxySymantecTechnologyTier 3

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by David Braue

Latest Videos

  • 150x50

    CSO Webinar: Will your data protection strategy be enough when disaster strikes?

    Speakers: - Paul O’Connor, Engagement leader - Performance Audit Group, Victorian Auditor-General’s Office (VAGO) - Nigel Phair, Managing Director, Centre for Internet Safety - Joshua Stenhouse, Technical Evangelist, Zerto - Anthony Caruana, CSO MC & Moderator

    Play Video

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

More videos

Blog Posts

Market Place