Where the Metrics Are

In this story:
- What metrics CSOs use.
- What it means to their operations
- Why they talk to business leaders about what they are tracking



Metrics are measures that matter, providing evidence of performance both to experts and to interested observers.

That's why CSOs are hungry for them. It's not good enough to maintain a quiet, reliable security service until something goes wrong. Security executives want to understand how their operations are working and how they can improve. CEOs want to know how the security function is faring by looking at the department's data. And metrics can provide the hard numbers and context on the performance of the security function, proving that nothing happening was the direct result of an effective security management program.

Key metrics vary by CSO, organization and industry. What's important to energy provider Georgia Power (federal regulation compliance, for example) may not be important to coffee purveyor Starbucks (armed robbery statistics, for example). "Metrics resist uniformity," says Dennis Treece, director of security for the Massachusetts Port Authority. "What works here may or may not work elsewhere."

Moreover, CSOs say that metrics don't always have to be straight-up numbers. Impromptu conversations with key executives can sometimes have just as much punch as a glitzy, chart-and-pie-graph show in the boardroom. "Clearly, statistics on their own don't make a very good read," says John Hedley, head of group security for food maker Nestle. "You have to interpret them and put them into context."

Here is the story of four security executives in different industries who give a rare peek into the physical security metrics that are important to them, their CEOs and their organizations. Taken together, these data points and measurements help them keep a firm grip on the most important metric of all: How much confidence the rest of the organization has in the security department.

Starbucks Tracks Everything That Moves

Starbucks Metrics insight:
Rigorous tracking of processes leads to improvements and business value.


To Francis D'Addario, the connection between security metrics and how effective he is as CSO of Starbucks is simple: His mission to protect people, secure assets and contribute savings year over year is validated with key performance indicators.

Whether D'Addario, vice president of partner and asset protection at the $US5.3 billion coffee and food retailer, is talking about physical assets (stores and equipment), liquid assets (cash and coffee) or human assets (employees and customers), using metrics is how he judges the success of his security group.

First and foremost on the priority list, D'Addario says, is the safety of people. The frequency of armed robberies at retail outlets, for example, is an important metric at Starbucks and within the retail industry. He says that since 1996, when there were 46 incidents per thousand Starbucks stores, there has been a steady decrease to a best-in-class 11 per thousand in 2004. D'Addario says Starbucks' numbers compare favourably to historic trends at similar outlets, such as quick-service restaurants (which have averaged 45 armed robberies per thousand) and convenience stores (125 per thousand). He uses metrics from uniform crime reports and industry associations.

D'Addario says the decline in robberies at Starbucks has resulted from implementing better awareness campaigns to help employees anticipate problems. Technologies, including smart safes and an interactive system that confirms security events, also have played a role.

Other metrics D'Addario relies on include tracking the frequency and outcomes of background identity checks, employee access control compliance (which is measured by spot audits and credentials checks), and cash or asset protocol performance (including sales, deposit preparation and banking). D'Addario says those are continuously audited, and exceptions are investigated routinely. "Cash loss is monitored as a percent to sales on every business unit's P&L," he adds.

D'Addario says that some measures he takes for security are also valuable to Starbucks' quality assurance team. For example, tracking how well the company maintains the integrity of its food containers remains a critical interest for both his security group and quality assurance. Container integrity is the reasonable assurance that the contents shipped - via overseas and truck routes - are those that were ordered. The company performs auditable inspections on these processes, including checking the integrity of container seals, he says.

Because Starbucks is global, method-ologies for tracking these processes vary by region, depending on the infrastructure and technology available. But the measures are an essential component of quality assurance, D'Addario says.

Key performance indicators are tracked by period, quarter, year-over-year and five years running, he adds. "That enables cost and benefit impact assessments, risk-gap closure analysis as well as return on funds spent," he says.

The trend analysis that D'Addario documents allows him to test new security technologies and protocols against the trends to decipher if they are contributing to sales or net profitability.

Working in the retail industry, D'Addario also benchmarks his cash loss as a percentage of sales as well as inventory shrinkage numbers with reputable industry group figures. Those kinds of numbers (which he declined to share for publication) allow D'Addario to present security performance indicators to his bosses.

"Thoughtful prevention design with forecastable results for performance improvement are viewed as investment opportunities," he says. As an example, he says that a number of international markets adopted exception-based reporting after witnessing its performance for top-line and bottom-line contributions in the United States. D'Addario reports that the protocol has since delivered the same performance in the international markets.

The key to all of that, D'Addario says, is that those forecastable results "are baked into the operational budget process with return expectations." While that puts your security department on the hook for demonstrable results, it also can make the CSO look brilliant in the boardroom when he delivers.

Nestle Metrics Emphasize Prevention and Protection

Nestle Metrics insight:
Preparing to handle disasters can avert big losses in life, capital and prestige.


When there is civil war where your people are working, one physical security metric rises above all others: Keeping all of your employees alive.

For John Hedley, head of group security for Nestle in Vevy, Switzerland, this scenario played out in November 2004 at Nestle's operations on the Ivory Coast. The West African nation has experienced constant turmoil between the government and rebel forces for the past three years. Hedley's security staff, led by a regional security manager based in Abidjan, the commercial capital, set in motion an evacuation plan for the international Nestle employees when it was clear that the violence was escalating to a dangerous level. The Ivory Coast produces 40 percent of the world's cocoa, and Nestle is one of the biggest purchasers. The evacuation of Nestle's expatriate staff was accomplished "with a minimum of hardship," Hedley says. "While such an unplanned departure is distressing for all, at least we were able to set in motion some pre-evacuation plans." Hedley's group had reviewed those plans just three weeks before the evacuation happened.

For a global company such as Nestle, with 115 production facilities in 86 countries, Hedley says operations such as the Ivory Coast evacuation are a necessary and expensive undertaking. Metrics enter afterward, in judging how well the operation went, what went into the preparation involved and the results - such as whether there were injuries or deaths.

"We have not done a cost-benefit analysis of how much money we have saved because of the security plan in place," Hedley says, adding he was not sure of the evacuation's cost. "We had more important things on our mind," he says. "Having a plan in place and revisiting it once a quarter or year may be the most important metric of all.

"However, the costs can be reduced by effective contingency planning - the emotional cost for the staff concerned as well as the financial cost," he adds. "Getting everyone out safe and sound means that there are no staff replacement issues. Keeping the factories and other buildings properly protected ensures continuity or early restart of production. These benefits could be measured if required."

Hedley says he can't apply blanket security and preparedness metrics around the world. "The ability to equate performance in one country, in one region, with another is difficult," he says. "For example, our security officers in New Guinea are armed (but with bows and arrows), whereas in most places they are unarmed."

Even with those impediments, Hedley does employ physical security measurements wherever he can. The areas most important to him are Nestle employees, distributors and consumers; company property; and the strength of Nestle's reputation and brand.

Hedley says he focuses much of his attention on Nestle's brand and reputation among consumers. "We have a broad brand protection strategy, in which we work in close collaboration with the intellectual property department," he says. "There's a very strong argument that brand and reputation are worth more than physical assets." Hedley points to the difference in measuring hard physical assets versus intellectual property and brand assets. "You can measure the number of burglaries you suffer and the amount of shrinkage," he says. But in the order of priorities for his group, he looks to condensed milk as an example. "Stolen boxes of condensed milk can be replaced," he says. "But if someone keeps them past the 'sell by' date, and then someone consumes it and gets an upset stomach, it's not so much the actual value of condensed milk but the effect that the inappropriate distribution and handling of such products can cause to people." And consumers' upset stomachs tend to give him an uncomfortable feeling as well.

The bottom line is also important to Hedley and his bosses. "We [in security] are judged by our overall contribution to the profitability to the group," he says. As an example, Hedley tells of how he grapples with trying to plan for the unforeseen. "Having the ability to reduce the number of events that are unforeseen is a very valuable metric," he says. When he is able to do this, it grabs the attention of senior management. "If you can tell a story that says, We were able to preempt a problem that was going to affect us, and, Oh by the way, had we not done this, this would have been the cost - that is a very good story to tell."

CSOs can estimate the damage that was not predicted or planned for by comparing to previous events or ones that hit other companies, Hedley says. You can say, If we hadn't taken the action we did, then the probability effect would have been X. "The downside, however, is that you can't say, This is the money we would have saved, and go put it back in the bank account," he says.

Utility Uses Government Rules to Build Metrics Georgia Power Metrics insight:
Scorekeeping on government regulations compliance yields valuable performance measures.


Margaret Levine, corporate security manager at Georgia Power, has found ways to convert the necessary burden of regulation into a bounty of physical security data for the electric utility.

Levine must demonstrate that Georgia Power, the largest subsidiary of Southern, the $US11.3 billion regional utility based in Atlanta, complies with federal regulations. Her security group does that by completing security audits to make sure that the protected areas at plants and substations are indeed protected.

"We have reports documenting that the people who have access to those areas have legitimate reasons to be there," Levine says.

Tracking results of these and other reports yields a measure that allows Georgia Power to compare its performance to itself in past years. It's a conscious management decision to turn the "play by the rules" portion of the operation into a performance measure.

"You need to find a meaningful purpose other than just pushing paper," she says. Security executives, she adds, can "take the next step and think, How can I use this report and statistics in a way to improve my security program or to better educate me about my customers' business?"

A second metric for Levine comes from a combination of readiness reviews and penetration testing.

Readiness reviews are planned events and are a key component of Georgia Power's business continuity program. The reviews assess whether employees and site security professionals at a particular facility understand that facility's threat plans and know what to do when the threat level is raised or lowered. Readiness reviews also include interviews with local managers about facility security; an audit of procedures and documentation related to security requirements; an evaluation of the facility's physical security program; and a review of its emergency action plan.

At the end of each review, Levine says, her office writes a report for the facility manager that highlights findings, best practices and recommendations.

For readiness reviews, Levine sends a pre-announced team of security professionals to do security audits of all critical facilities and operations (though she declines to list what types of facilities those are).

In addition, penetration testing attempts to breach security - procedurally, technologically or physically - to determine whether the security program is functioning as it should, she says. "We may have someone try to walk through a facility without wearing a badge to see how far they can get before being challenged," Levine says. "Or we may have someone see if they can talk their way around our delivery processing requirements."

Results Reports

Results are reported in two ways. First is what Levine calls the "objective, scenario, outcome": Here's what Georgia Power was testing (for example, the effectiveness of visitor management personnel); here's how security tested it (use of outdated or fake identification credentials); and here's what happened. "The results are reported by comparing the test outcome with the test objective, in addition to including a description of how the test was carried out," Levine says.

Second are the lists for "did well" and "areas for improvement": These are reported along behaviourally based criteria (for example, clarity of communications with "outsider" or whether incident notification procedures were followed) as well as results-based criteria (penetration foiled or speed in which penetration was detected).

After collecting results, Levine's group tracks the physical and technical security measures at each location to ensure that they are functioning properly. Physical security measures include perimeter barriers, lighting, locking devices and key controls, and signage. Technical security measures include intrusion alarms, closed circuit television and other monitoring devices, access control and visitor management systems.

"We would want to make sure that the security folks onsite knew what to do in the event of raising the threat level or a breach of security," Levine says, "and also have a good awareness of security protocol and who they could go to if a breach did occur."

Tracking Trends

Incident trends and loss trends are next on Georgia Power's metrics list. Levine says that it's critical to be able to demonstrate that a CSO's security program is a significant mitigating factor in preventing increased incidents and losses. Levine can compare incidents by quarter, year-to-year and across multiple years. She can note the changes in the number and frequency of incidents by type of incident (for example, thefts, threats against employees or sabotage), by line of business (generation, transmission, distribution, staff services) or by location. She follows the same process for tracking losses; she says she tracks property and monetary losses. The key, she says, is if you're not able to prevent losses, then "you can demonstrate an ability to quickly pinpoint where the weakness was and put in place the appropriate stopgap measures."

Levine adds that metrics must be more than in-house security tools; they have to be relevant to the people she supports - business executives, plant operators, substation engineers, customer service managers. She says her reports must contain information that is important to them, not just to security managers. Doing this, Levine says, "also enables us to educate them about things that are important from our perspective, and in that give-and-take process we're able to validate the measures that we're using." Depending on the type of data and compliance requirements, Levine reports her metrics monthly, quarterly or yearly.

Levine considers two other factors when collecting data for metrics. The first is how Georgia Power compares to other utilities. And the second is data quality.

Levine says Georgia Power collaborates on metrics reviews with other security managers from within Southern's 12 operating companies. (Besides Georgia Power, there are four electric utilities and companies in wholesale power, power generation management, natural gas, nuclear power and energy services. Southern also owns a wireless company and a fibre optics business.)

As for data quality, Levine says that it's important to watch out for the equivalent of scorekeeping changes. She says Georgia Power recently transitioned from a 10-year-old case management system to a new system developed last year by Southern's security managers. The case management system is a database that records all the details of incidents that are reported to corporate security. This includes an incident narrative and summary; victim, witness and reporting party names; losses; investigative activity; and case resolution.

Building the new system required a review of incident definitions so that a year-to-year comparison made sense, she says. For example, the old case management system had separate incident categories for burglary, larceny, fraud and robbery. But in the new case management system, all of those crimes are categorized as financial matters. "To make an apples-to-apples comparison between the old and the new, we have to select a specific subcategory (for example, larceny) in the new system," Levine says. "Otherwise, the analysis - larceny versus financial matters - would show that we'd had a crime wave at Georgia Power." And that's the last thing that Levine and her executives want to hear.

SIDEBAR: Five Metrics That Matter

George Campbell, former CSO of Fidelity Investments and now a security consultant, says there are hundreds of security metrics available for CSOs, who need to identify those relevant to their organization. Here are five important ones.

1. Risk analyses. The risk analysis process, a constant activity for security executives, incorporates several metrics: assets, loss events, vulnerability assessments (how easy would it be to do X, Y or Z?), likelihood of an event, probabilities, and options to mitigate vulnerabilities and their cost and benefits.


2. Value indicators. Cost-benefit analyses yield relevant metrics. "If you've got an investigation function that costs X amount of dollars, and it recovers twice that in losses, that's a positive return on investment," he says. But the value indicators will be unique to each business segment within a corporation.

In the financial world, much is based on reputation. In businesses where there's a lot of intellectual property, the value will be based on stopping some-one from counterfeiting or stealing any proprietary processes.


3. Process performance. Response times and recovery procedures produce metrics. How long does it take to recover a critical business process lost to a natural disaster or cyberattack? What is the average time for a security officer to respond to a critical alarm or injured person? What is the time needed and cost of a background or business conduct investigation? "Every CSO develops annual objectives that must be measurable if they are to devote resources to their accomplishment - and be willing to be held to them," Campbell says.

4. Integrity scorecard. Campbell says this is where the CSO tracks what keeps business executives awake at night. These include risk awareness; security breaches resulting in losses; hiring people with bad backgrounds; higher than normal accident rates; and failure to address known vulnerabilities. "You maintain a scorecard on what makes that business unit tick from an integrity standpoint," Campbell notes. "You understand where to allocate your resources, and you can show the CEO where the problems could be in the business."

5. Confidence measures. These allow the CSO to see how well the security function is delivering services. Through internal customer satisfaction surveys and post-mortems on investigations, CSOs can measure the confidence the business has in the security department. "You can look at how well you did and what the problems are," Campbell says.

Campbell adds that communicating the goal of metrics is a key activity. "If you're going to track metrics on integrity or by a scorecard, you'd better pre-sell the process at various levels and be very careful to ensure accuracy of information

SIDEBAR: Airport Keeps Records to Build Credibility


Dennis Treece's boss, the CEO of Massport, has said that Treece must derive "effective security metrics," ASAP.

Treece, director of security at Massport, the agency that runs Boston's Logan Airport and several other transportation facilities, says that CSOs work very hard to show that nothing has happened, nothing has gone wrong. "CSOs need to know how to report nothing in context that makes sense," he adds.

The three physical security metrics that matter most to Treece are uptime, performance and violations - all of which he tracks quarterly. With uptime (or availability) measures, he is looking for shortcomings in staffing levels and equipment availability compared with what's required for an effective security program. Treece's performance metric is simple: If the required performance standard is X, then is that being achieved? For example, if the baggage-screening equipment is supposed to process 500 pieces of luggage per hour, he must track whether his operation meets that target, and if not, why not.

Tracking violations of security policies is usually measured by failures to comply with government regulations. Treece says employees who break the rules need to be disciplined, motivated, trained - sometimes a combination of the three. Keeping track of trends in such violations helps a CSO keep track of the nature and extent of a problem.

Massport is subject to inspections by the Department of Homeland Security and the Transportation Security Administration, Treece says. If the inspectors cite the agency for failure to conform to regulations, "that is something you will want to know, track and work on so it doesn't happen again," he adds.

Treece says the tracking takes time, but it helps justify security operations. Not doing so means depending on others to make decisions about his operation. At the end of the day, though, Treece longs for one metric that remains elusive. "At any given point of the year, are we better off because of anything that we did?" he asks. "That would be nice to have."

Join the newsletter!

Error: Please check your email address.

More about AvertFidelity InvestmentsHISSpeedStarbucksTransportationVIA

Show Comments

Featured Whitepapers

Editor's Recommendations

Solution Centres

Stories by Thomas Wailgum

Latest Videos

More videos

Blog Posts

Market Place