Inside Symantec's top secret Melbourne digital certificate authority

The 'Ghostbusters' jokes were flying thick and fast as the facility's dozen-strong group of assembled media entered the facility and descended to meet the 'Key Master' – yes, that is his real title – down fire stairs and through one, then two, then three ASIO-rated fireproof doors secured with a variety of biometric, card and physical controls backed by a dizzying series of access rules.

It's an intimidating welcome to Symantec's high-security digital certificate issuing facility, a nondescript building tucked into a Melbourne suburb from which billions of secure Internet transactions are validated every day. It is one of four such Symantec facilities around the world, employing around 80 staff in the task of validating the identities of Web sites and the people that want to protect them.

Security is the name of the game inside the facility, where staff regularly deal with door-access rules so complicated that that they would make even the Sirius Cybernetics Corporation blush.

No more than one person may pass through the door at a time, those rules dictate, and entrances and exits are tracked and logged so if anybody tailgates another person into a room, they cannot exit it without a physical rescue by the facility's guards. That rule alone had caught out more than a few employees who might, naturally, follow colleagues through an open door when going on a toilet break – only to find they couldn't get out without an embarrassing extraction.

We eventually entered a windowless room where the almost imperceptible rush of circulated air and a few optimistically placed pot plants offered a glimpse of green to a dozen workers in this Tier 3 high-security data centre environment.

Heat sensors in the ceilings continually count the number of bodies in the room, raising raise alarms if it doesn't match the number of people that have swiped in.

Alarmed vibration sensors in the walls, ceilings and floors raised the inevitable questions about 'Mission Impossible' styled infiltrations, although our guides – including Symantec senior principal systems engineer Nick Savvides and the Key Master, who like that dentist in the TV commercials cannot actually be named – point out that the air-venting system has been designed with so many roundabouts and discontinuous segments that you'd more than likely just get lost up there.

Its roof, too, is protected, with military-grade secure mesh built into the design and an array of roof-mounted sensors that raise alerts even when a pigeon alights. Security, data and power cabling throughout the facility is separated into three colour-coded networks running over physically separated trays, with all fibre runs required to be unbroken and exposed, with sensors able to detect if there is any attempt to tap into the networks.

These and other protections make the facility a textbook exercise in military-grade data-centre security, but the physical protections are only the setting for the everyday management of digital certificates – the basis of trust on the Internet for their use in the public key infrastructure (PKI) system supporting SSL-enabled security.

The consequences of a security breach on an operation like this have already been played out in graphic detail in the media: Dutch certificate authority DigiNotar ceased operations within a month after it was compromised in 2011, and others have met similar fates after hackers successfully breached their protections and stolen digital certificates that allow them to trick Internet users into believing they are legitimate certificate owners.

There is, unsurprisingly, zero tolerance for lax security in the Symantec facility – hence the high level of alarming and surveillance. "We can see when you go to the toilet," Savvides points out when one journalist asks the way. "Not actually in the toilet," he adds – only half-jokingly – "but we know when you're going down the hall."

The Key Master steps up. For all the physical protections built into the facility – part of a global Symantec network that processes over 14 billion authentication transactions every day – even higher procedural security is in place during the top-secret 'Key Ceremony' in which new public and private root keys – the foundation of PKI-based digital certificate hierarchies – are created.

Savvides won't say when or how frequently these key ceremonies are held, but when they're on he, the Key Master and others steward an exacting process that is carefully scripted, recorded on video and painstakingly executed – and can, we learn, run for up to two weeks.

The ceremonies are held in a nondescript room that lies behind two additional military-grade secure doors. For all its import, the white walls, floor and ceiling would look unremarkable in any office building in Australia. However, Savvides tells us, apart from the other existing protections, the several desktop computers and printers on the desk are not connected to the Internet and the only communication with the outside world is through the single phone extension.

Once staff enter the room for a key ceremony, they might not leave for an entire eight-hour day – and often come back the next day, and the next, until the key-activating script has been completed. Food is not allowed, although each person in the room can have one drink with them. Lunch and even toilet breaks pose additional complexities, with staff required to pack up and lock everything in the room before leaving it for any reason.

"We can end up with two people stuck in here for eight hours," Savvides says. "It can be pretty hard on us."

Those eight-hour shifts are filled with mind-boggling minutiae, carefully and sequentially executed according to printed scripts that can run to 600 pages or more. Written by the Key Master in conjunction with Symantec's clients, they document every single mouse movement, key press, and checkpoint required throughout the process of adding a new trusted domain to the Internet.

Even one mismatched serial number is cause for aborting the process, while subsequent discovery of any human error during the process will force Symantec to revoke the certificate and the process to be checked and started over. Documents must be painstakingly verified, serialised USB drives containing private keys carefully managed and triple-checked to preserve the integrity of the process.

The process ranges from 45 minutes to renew a digital certificate, to four hours to register a user and two weeks to set up a complete digital-certificate hierarchy for a government or other top-level entity. It may be thankless, painstaking work – but this is the price of Internet security, and it's happening day in and day out to ensure what happened to DigiNotar and other certificate authorities doesn't happen again.

As we leave the ceremony room, filing through one secure door after another, someone asks how staff find everyday life in such a locked-down, unforgiving environment.

That sort of thing, Savvides says, is addressed early in the interview process. "The people we select know what they're getting into," he laughs. "It does take a certain type."

This article is brought to you by Enex TestLab, content directors for CSO Australia.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Follow our new CSO Australia LinkedIn
Follow our new social and we'll keep you in the loop for exclusive events and all things security!
Have an opinion on security? Want to have your articles published on CSO? Please contact CSO Content Manager for our guidelines.

Tags symantecdigital certificates

More about ASIOCSOEnex TestLabGalaxySymantecTechnologyTier 3

Show Comments

Featured Whitepapers

Editor's Recommendations

Brand Page

Stories by David Braue

Latest Videos

More videos

Blog Posts