Legacy data warehouse architectures may have been designed for large amounts of data – but even massive warehouses would struggle to manage, much less analyse, the 20 to 30 terabytes of operational data that nbn™ Australia collects on a daily basis.
That data – including network performance data, performance alerts, usage information, network alarms, and more – is vital to understanding how the company’s Australian wholesale broadband access network is performing.
But as network and systems became more complex and its data requirements more sophisticated, nbn has had to explore different approaches – which it did with the establishment of the Foresight Lab over a year ago.
Amongst their many tasks, the technical and data-analytic experts within Foresight scale have been charged with transitioning nbn from legacy data-warehousing technologies to a less well-defined, unstructured virtual ‘data lake’ – an evolution of big-data technologies that sidesteps past architectural constraints through the integration of web-scale technologies.
The virtual data lake is built and managed internally within nbn as a private cloud, offering tight control over data while providing the scalability benefits of web-scale companies that “have changed the way we look at data,” explains Arun Kohli, executive general manager for IT architecture and Foresight Lab with nbn.
“There were always metrics, KPIs, and data used to make the decisions, but the scale that we now call ‘big data’ was very difficult and very expensive. With cloud computing and storage, the technology of the storage has changed – which has helped us exploit the data at scale.”
A foundation for change
Coping better with scale has proved critical in fuelling two other key data paradigms that are driving R&D work.
The second, performance and capacity management, is gaining accuracy and relevance thanks to the detailed analyses it provides about the operation of the wholesale network that nbn operates across several transmission modes spanning across Australia.
Ongoing analysis of usage patterns and congestion spots helps the company prioritise target areas for its ongoing rollout, while correlations with factors such as time of day have revealed important patterns in consumer demand for services over the nbn™ access network and retail service provider (RSP) demand for nbn™ wholesale products and services.
The third area where nbn’s data collection is helping to drive a service revolution lies in the ability of such massive data stores to feed increasingly common artificial intelligence (AI)/machine learning (ML) algorithms.
By training AI tools with ML algorithms designed to better understand baseline performance and configuration specifications, it becomes much easier for the systems to automatically spot issues that may be causing performance problems or service interruptions.
Whatever the transmission medium, patterns of behaviour are built up over time – providing unprecedented predictive power that can help the company anticipate a performance problem before end users even notice it.
This might be an issue with a setting in an end-user router, or an issue with the way that a hybrid fibre modem is bonding multiple transmission channels. Under today’s models, such problems would only be identified when an end-user complains about their service and the RSP escalates the call – but in the AI/ML world, nbn can help to proactively identify looming issues and resolve them well before they become a problem.
Finding the right skills
Intelligent analysis techniques may be providing new insights into ever-larger quantities of nbn™ data, but they’re also creating new challenges as the company works to build out its base of skilled data specialists.
While establishing the Foresight unit over a year ago, the team faced a challenge to find people with the necessary domain expertise and data-analytics credentials. Instead, they identified internal data engineers for training and separately hired pure data-science specialists to help them transition into the more proactive, AI/ML-led environment.
This was not always straightforward. “Our domain experts are coming from an educational background and industry training,” Kohli explains, “where things are defined by engineering and mathematical formulas. They like to use supervised machine learning because it’s very close to the way they were always taught – but when we talk about unsupervised machine learning, it becomes difficult. It’s a conceptually different way of thinking.”
To resolve these differences, data-focused specialists were paired with internal domain specialists – who understand core nbn technologies including fixed wireless, satellite networks, hybrid fibre-coax (HFC), fibre-to-the-curb (FTTC), and dense wavelength division multiplexing (DWDM) backhaul – and tasked with training them in the disciplined application of data analysis techniques to their existing knowledge.
This collaboration between three disciplines spawned a process by which the company has been able to upskill its internal network specialists, technical architects, operational support system (OSS) and business support system (BSS) architects, and other specialists – bringing them together on a common purpose to transition nbn into a more data-focused organisation than ever.
In August, nbn revealed their updated corporate plan that highlighted their focus to improve the experience end-users and improve the time to resolve issues as quickly as possible.
“We are very much a metric-driven organisation,” Kohli explains. “Once we understand these issues, we can talk about leading indicators – and not just lagging indicators, which most of the industry does. The end user may still have a service, but we can now spot issues that may impact them in the future.”
Bringing this all together has been influential in helping nbn to spot those issues before they impact an end-user or business. Advances in data uses further enhance nbn’s ability to improve the experience of internet access when customers need it most.