Big-data analysis helps QUT learn more about its security posture than ever
- 11 August, 2015 14:49
A massive big-data collection and analysis system is providing network infrastructure managers at the Queensland University of Technology (QUT) with unprecedented visibility into security events and other operational issues that arise daily when managing such an extensive and varied computing environment.
The big-data investment was instigated in order to provide a way of centrally storing and analysing operational logs from a broad range of systems, but in a university context it was also seen as an important tool to support researchers in a broad range of fields.
Splunk Enterprise big-data analytics software was initially deployed to allow IT staff to keep tabs on logs from the student management system and the university's QUT Virtual portal, helping pinpoint load-testing activities that ensured existing and potential bottlenecks were spotted early on and dealt with summarily.
In this way, QUT has been able to monitor, pinpoint and optimise the efficiency of its user experience.
The system has rapidly expanded to the point where it is currently collecting over 200GB of data every day. This includes data from authentication systems, security tools, virtualisation hypervisors, database management systems, operating systems, and physical hardware that also supports the helpdesk, lecture recording system, Microsoft SharePoint and Exchange, Blackboard learning management system, Media Warehouse, and other Web sites.
The Splunk environment provides instantaneous access to over 50 billion log entries through which IT staff can search and generate reports as necessary. “Our staff, students and researchers are eager to extract knowledge from data through visualisations,” a QUT systems specialist explained.
“By enabling our colleagues to correlate, contextualise and apply analytics to information from disparate sources, the Splunk platform offers significant opportunities for data-driven decision-making. They're able to analyse authorised machine data, gaining an unprecedented level of visibility and agility.”
This agility has proved particularly valuable in vetting the massive volume of security log information that is generated daily by the system's various infrastructure components to service the 45,000 students and 10,000 staff at QUT.
Root-cause analysis and performance monitoring form a critical benefit of the big-data environment, with log information highlighting persistent authentication, performance and other issues that may point to more sinister activity on the network. Specialised Splunk extensions for Microsoft Exchange allow the team to meaningfully monitor and analyse the performance of that environment, easily pulling out key metrics to ensure operational performance, simplify capacity planning and facilitate auditing of security events.
Because the data is being collected from a range of systems across the networking environment, cross-correlation of data provides a richer analytical toolkit than would otherwise be available.
The value of the environment, the systems specialist said, “is only constrained by the diversity of data given to it and the questions asked of it.”