As an ever-changing range of threats pummels corporate security defences, big-data strategists will redouble their efforts to apply data aggregation and analysis techniques even as the volume of collected security data pushes towards an exabyte per day, the market leader in this space has vowed.
The logistics of grappling with massive and growing volumes of security logs – generated with ever-increasing granularity from an ever-broader range of sources – had become unwieldy for many environments, Splunk CEO Doug Merritt told CSO Australia as the growth-minded company recently opened its Asia-Pacific headquarters in Melbourne.
“In the past five years we have dramatically opened up the types of data we can ingest,” Merritt explained. “Our most aggressive customers are really using Splunk as this end-to-end data layer and are already ingesting petabytes per day, and we have a whole cluster of customers in the hundreds of terabytes per day.”
That kind of growth had posed new challenges for the company, which has increasingly pushed into the security-analytics market after establishing itself as a broader big-data analytics provider. The move has worked, with Splunk’s own security capability complemented by a growing network of partners that value its big-data nous in keeping abreast of ever-changing security threats.
Yet with customers taking the company at its word – its biggest customer’s security data collection has jumped from 1 petabyte per day in February to more than 3 petabytes per day now – Splunk has inadvertently created a rod for its own back.
That same customer is entreating the company to plan to cater for an exabyte of new data every day within five years – pushing its engineers to fight to continue what Merritt calls “a never-ending journey to continue pushing elastic horizontal scalability whilst still having effective response time” that he believes will keep the company on its toes as customers accumulate “incredible” amounts of security-related data.
“The only way we will go from a billion dollars in revenue, to $5b or $10b, is if we can figure out a way to keep doing that,” Merritt explained. “That is the number one priority for the company, and we will be keeping the foot hard on the pedal through the next year and beyond.”
As well as opening its new regional headquarters in Melbourne – from which a growing range of clients and partners will be served by a staff that has more than doubled in size in the last 12 months – Splunk has committed to donate at least $US100m ($A133m) over the next 10 years in licenses, support, education programs and people.
The investment is designed to strengthen Splunk’s position as the de facto analytics platform within the corporate world as well as Australian academic institutions, research organisations, and not-for-profit organisations. This includes an increasing investment in machine-learning techniques as well as a range of initiatives is designed to help local engineers upskill in key areas such as security, with around $US2500 (A$3300) worth of training available through Splunk certification programs.
Six Australian universities are currently participating in the Splunk program but Merritt sees this expanding to “hundreds or even thousands of entities” within the next 12 to 24 months.
It’s the kind of investment that drove Victoria minister for small business, innovation and trade Philip Dalidakis to welcome Splunk’s investment – which follows other new Melbourne offices opened recently by companies such as LivePerson, Jitterbit, Mimecast, Check Point Software Technologies, and others – as being “great for growing our local talent pool and more high-skill jobs for the state.”
The choice of Melbourne “places us squarely in an ecosystem with likeminded companies”, Merrit said. That ecosystem has grown more vibrant this year on the back of the federal government’s formal Cyber Security Strategy and concrete steps like the recent appointment of industry figure Craig Davies to head the new investment-focused Australian Cyber Security Growth Network, which has underscored its commitment to fostering development of cybersecurity as a thriving local industry.
If he and his team have their way, Splunk will be the glue that holds together many of the most innovative security-related initiatives being developed across the industry in Australia and the Asia Pacific.
With CISOs “likely going to have the need to accommodate 5 to 10 new technologies a year,” Merritt said, he envisions the platform as being “the universal abstraction layer that allows them to put in and take out different technologies that still have that consistent, universal, flexible interrogation layer across that turbulent landscape.”
“We are building the product and its flexible schema – as opposed to having to architect your data layout for the time you are ingesting data,” he said. “A key focus will be to try and batten down new openings in the technology landscape, and to close openings that someone has figured out how to interrogate effectively from whatever operating place they are coming from.”