SSH key mismanagement and how to solve it

In 1995, when I was a university student in Helsinki, I developed a security protocol to protect data-in-transit as it moved throughout our network. I named it the "secure shell," or SSH for short. Today, SSH is used by organizations of all types and sizes as a secure method to move data from machine to machine and provide remote administrator access. From the perspective of an attacker or malicious insider, SSH is an artery that carries vital organizational data.

[RELATED: Tatu Ylonen, father of SSH, says security is 'getting worse']

However, most organizations do not have control over how their SSH keys are created or managed, leaving them vulnerable to attack and at risk of costly fines for regulatory noncompliance. To remediate this issue, every organization should audit their SSH key management systems.

SSH: A Primer

SSH works by creating a cryptographic key pair - one for the server, and one for the user - and securing the data that passes between those two points. Unfortunately, most organizations have no process for managing, removing, and changing access-granting "keys" once they are created and deployed. As a result, thousands of keys are distributed at will throughout the network environment, typically without any way to find or remove them, creating untraceable access paths to sensitive data. Mismanagement of SSH keys, therefore, presents a significant security vulnerability.

The scope of the problem is as pervasive as it is shadowy. Our experience in working with major enterprises has shown that most organizations have on average eight to over 100 SSH keys configured granting access to each Unix/Linux server. Some keys even grant high-level administrative access.

"High-risk" insiders can use these poorly-managed SSH keys to create permanent backdoors to production servers, bypassing all normal privileged access management and session auditing systems. These insiders include anyone who has ever had access to a server, including former system administrators, consultants, and anyone with access to backups or decommissioned hardware.

This problem has flown under the radar because it is deeply technical and obscure, residing in the domain of system administrators. Each system administrator typically only sees a small corner of the IT environment, and therefore does not have a full picture. Administrators and their managers are often so busy that while they may recognize that there is a problem, they simply have not had time to analyze its scope, or its implications.

Unmanaged SSH Keys - An Attack Vector for Viruses

The chances of such a breach occurring are growing by the day. News reports on network breaches are commonplace as attacks become more widespread and sophisticated. Implementing SSH keys as an attack vector in a virus is quite easy, requiring only a few hundred lines of code. Once inside an organization, a virus can use improperly-managed SSH keys to spread from server to server.

In fact, the mesh of key-based access is so dense that it is highly likely that an attack can spread to nearly all servers in an organization, especially if the virus also utilizes other attack vectors to escalate privileges to "root" (high-level administrator) after penetrating a server. With so many keys, odds are the virus will infect nearly all servers in a manner of seconds to minutes, including disaster recovery and backup systems that are typically also managed using such keys.

In the worst case, a virus or cyberweapon using multiple attack vectors could spread globally, Internet-wide, enterprise-wide in a matter of minutes and, combined with destruction technologies, could wipe out massive amounts of data.

Compliance at Risk

Major organizations without proper SSH key management in place aren't just at risk from security breaches; they also appear to be non-compliant with mandatory security regulations and laws. Industry requirements such as SOX, FISMA, PCI and HIPAA demand proper control of access to servers and termination of that access. Furthermore, these organizations might also find themselves running afoul of internal security policies (including in some cases policies mandated by their customers).

How did we get to this point? The situation is not a result of any vulnerabilities or flaws in the SSH protocol itself or in the most commonly used implementations of the protocol. Rather, it is the confluence of a number of factors:

  • Years without clear guidelines or policies relating to SSH keys.
  • Lack of understanding of the scope and implications of the problem.
  • Insufficient time and resources to dig into the issue to gain understanding or develop solutions.
  • Reluctance on the behalf of auditors to flag issues for which they don't have effective solutions.

These masking symptoms are pervasive, yet the issue of SSH key mismanagement cannot be ignored forever. Without properly controlling, auditing or terminating SSH key-based access to IT systems and data, most enterprises, government agencies, and healthcare providers are leaving themselves vulnerable to attack.

Best Practices

Before the problem can be remedied, its existence must be recognized. Several teams within IT operations might be involved in the remediation project and proper support and endorsement within the organization is needed.

Remediation of the problem involves several steps:

" Discovering all existing trust-relationships (who can access what).

" Monitoring the environment in order to determine which keys are actually used and removing keys that are no longer in use.

" Enforcing proper processes for all key setups and other key operations.

" Automating key setups and key removals, eliminating manual work, human errors, and reducing the number of administrators from possibly several hundred to virtually none.

" Rotating keys, i.e., changing every authorized key (and corresponding identity keys) regularly, so that any compromised (copied) keys cease to work.

" Controlling where each key can be used from and what commands can be executed using the key.

Nearly all Fortune 500 companies and major government agencies appear to be non-compliant and facing major security issues (existential virus risk, hacking risk and rogue employee risk). Most of the next 10,000 companies (there are about 10,000 companies having over 10,000 employees in the world) are also exposed to the same issues. We have seen this issue in companies with as few as 20 servers, but when scaling to 100 servers and above, the benefits of automated systematic key management are clear.

It will take several years to fully address the issue and thousands of people must be involved and properly trained. Awareness must be created among CISOs, CIOs and enterprise IT risk management professionals so that they can work to ensue SSH user keys are properly managed in their organizations.

While SSH remains the gold-standard for data-in-transit security, the current threat landscape requires organizations to take decisive steps to improve how they are managing access to their SSH networks.

About the Author:

Tatu Ylönen is the CEO and founder of SSH Communications Security. While working as a researcher at Helsinki University of Technology, Tatu Ylönen began working on a solution to combat a password-sniffing attack that targeted the university's networks. What resulted was the development of the secure shell (SSH), a security technology that would quickly replace vulnerable rlogin, TELNET and rsh protocols as the gold standard for data-in-transit security.

Tatu has been a key driver in the emergence of security technology, including SSH & SFTP protocols and co-author of globally recognized IETF standards. He has been with SSH since its inception in 1995, holding various roles including CEO, CTO and as a board member.

In October 2011 Tatu returned as CEO of SSH Communications Security, bringing his experience as a network security innovator to SSH's product line. He is charting an exciting new course for the future of the space that he invented.

Tatu holds a Master of Science degree from the Helsinki University of Technology.

Read more about wide area network in Network World's Wide Area Network section.

Join the CSO newsletter!

Error: Please check your email address.

Tags compliancesecurityWide Area Networkdata protection

More about Helsinki University of TechnologyIETFLinuxSSHSSH CommunicationsSSH Communications SecurityTechnology

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Tatu Ylönen, CEO and founder of SSH Communications Security

Latest Videos

  • 150x50

    CSO Webinar: The Human Factor - Your people are your biggest security weakness

    ​Speakers: David Lacey, Researcher and former CISO Royal Mail David Turner - Global Risk Management Expert Mark Guntrip - Group Manager, Email Protection, Proofpoint

    Play Video

  • 150x50

    CSO Webinar: Current ransomware defences are failing – but machine learning can drive a more proactive solution

    Speakers • Ty Miller, Director, Threat Intelligence • Mark Gregory, Leader, Network Engineering Research Group, RMIT • Jeff Lanza, Retired FBI Agent (USA) • Andy Solterbeck, VP Asia Pacific, Cylance • David Braue, CSO MC/Moderator What to expect: ​Hear from industry experts on the local and global ransomware threat landscape. Explore a new approach to dealing with ransomware using machine-learning techniques and by thinking about the problem in a fundamentally different way. Apply techniques for gathering insight into ransomware behaviour and find out what elements must go into a truly effective ransomware defence. Get a first-hand look at how ransomware actually works in practice, and how machine-learning techniques can pick up on its activities long before your employees do.

    Play Video

  • 150x50

    CSO Webinar: Get real about metadata to avoid a false sense of security

    Speakers: • Anthony Caruana – CSO MC and moderator • Ian Farquhar, Worldwide Virtual Security Team Lead, Gigamon • John Lindsay, Former CTO, iiNet • Skeeve Stevens, Futurist, Future Sumo • David Vaile - Vice chair of APF, Co-Convenor of the Cyberspace Law And Policy Community, UNSW Law Faculty This webinar covers: - A 101 on metadata - what it is and how to use it - Insight into a typical attack, what happens and what we would find when looking into the metadata - How to collect metadata, use this to detect attacks and get greater insight into how you can use this to protect your organisation - Learn how much raw data and metadata to retain and how long for - Get a reality check on how you're using your metadata and if this is enough to secure your organisation

    Play Video

  • 150x50

    CSO Webinar: How banking trojans work and how you can stop them

    CSO Webinar: How banking trojans work and how you can stop them Featuring: • John Baird, Director of Global Technology Production, Deutsche Bank • Samantha Macleod, GM Cyber Security, ME Bank • Sherrod DeGrippo, Director of Emerging Threats, Proofpoint (USA)

    Play Video

  • 150x50

    IDG Live Webinar:The right collaboration strategy will help your business take flight

    Speakers - Mike Harris, Engineering Services Manager, Jetstar - Christopher Johnson, IT Director APAC, 20th Century Fox - Brent Maxwell, Director of Information Systems, THE ICONIC - IDG MC/Moderator Anthony Caruana

    Play Video

More videos

Blog Posts