The bottlenecks in measuring security performance

There are bottlenecks throughout the cyber security risk management process. UK Government surveys suggest that directors are invariably unclear about the business implications of the cyber security reports they receive. Conversely, despite the challenges associated with the massive volumes of ever-changing security data, security teams believe their communications to the business are clear.

In many of the conversations with customers and others we hear stories about the challenges organisations are having with their current interview/questionnaire based cyber risk management and reporting activities. They talk especially of the cost and disruption of gathering and providing the cyber resilience status reports to boards, cyber insurance providers and even key customers.

It seems that the overhead of “security reporting” is becoming a major burden; tens of thousands of hours per year, rising frequency and even then, accuracy and clarity is increasingly coming into question. No wonder there are calls to standardise the process.

At the heart of why this is so difficult, is that there really are lots of obstacles that hamper the availability and assessment of cyber security report and performance data.

The number of sources of data to interrogate

Cyber security is a broad discipline. While Newton’s 3^rd law of physics does not apply – it is true to say that for every digital initiative there is a cyber security implication! There are so many ways to attack systems and steal data; and so many settings, controls and mitigation strategies that need to be in place to effectively protect an organisation’s cyber security posture.

Because of these variables, reporting on security performance of IT assets and systems can be a bigger task than almost any other part of business. Rather than a single CRM, finance or HR application, the security team must aggregate data from a plethora of systems: firewall configurations, anti-virus updates, privilege access management, IDS/IPS technologies, security analytics platforms, patch management, backup solutions and the network itself. And the list goes on….

So the task of collecting and reporting cyber security performance across IT systems and assets is a complex activity; made more difficult by the need for its increasing frequency.

The number of separate indicators to report on

Another issue, having gathered the data from across sub-systems, IT environments and security controls, is how to present the information reliably and with clarity.

Evidence that underpins the management and report activities must be available if required; but KPIs relating to the key security controls should provide clear and relevant information to inform the risk management process.

Most security standards contain a large number of controls or requirements, but often they are categorised into a small number of “groups”:

• • The 5 framework functions of the NIST Cyber Security Framework;
  • The 8 controls in the Australian ACSC Essential Eight;
  • The 5 control groups of the UK’s NCSC Cyber Essentials;
  • The 4 control groups of ISO 27002:2022; or
  • The 12 requirements of PCI-DSS 4.0

So typically, there is no shortage of cyber security data but it is the relevance and clarity of the information that enables effective management. Once dashboard measures or KPIs go beyond a reasonable number, the level of understanding can diminish for some users and the information gets crowded out with detail. This is particularly important for people who may not be familiar with the absolute detail but need precise information upon which to base important decisions.

A shift from eminence and opinion to evidence-based risk assessment

Like any profession, security decisions often rely on the opinions or eminence of experts when it comes to what’s acceptable and what’s not. Their judgements can have significant cyber security risk implications so it’s important that, wherever possible, they are verifiable and evidence-based. New technologies are increasingly available to automatically measure and reliably report risk assessments and so increase the confidence levels of a cyber risk management process and better inform non-technical stakeholders.

The move towards more evidence-based objective measurements, and hard quantitative KPIs, is becoming overwhelming. Subjective anecdotal risk assessments still hold sway in some organisations but a cultural change towards evidence-based risk decisions is underway; driven by auditors and risk managers. Recently the Australian Cyber Security Centre changed its recommended risk assessment methodology noting that evidence-based judgement and opinion is far more reliable than any other objective measure.

The frequency of reporting

As press reports all too frequently reveal, your cyber security posture can change overnight. A system can be secure (or at least “fully patched” and configured correctly) one minute and a new vulnerability render it vulnerable, and exploited, by a zero-day attack the next.

This is made worse with our IT environments constantly changing – configurations, software versions, files and data, user accounts. Risk assessment and reporting practices must be able to keep up with constant change in our risk environment.

Security teams need risk assessment and management solutions that can address the velocity of these changes.

Decision makers need their information in a timely manner to ensure that the cadence of their risk assessment and reporting practices adequately meet the risk management needs of their enterprise. The greater the lag between the identification of a cyber risk and its subsequent reporting, the less chance of its effective management. Equally importantly, the less reliable the cyber security reporting to the executive and the board.

The translation of technical material into comprehensible business information

The details in security reports can be highly technical. For example, the patching performance reports might list servers, software versions, applications, vulnerabilities/CVE numbers, patches, severities, mitigations.

Detailed information is critical to security operations teams; but to be frank, relatively meaningless to all but technical risk management teams. Any business risks emerging from the patching assessment, however, may need to be translated into a clear and accurate business risk information as it might have significant business implications.

There is ongoing discussion whether this type of security information needs to be more clearly articulated in non-technical terms to be more easily understood by executives or whether those executives and directors should be more cyber literate. There is no single answer, except to say that technical information coming from security systems and controls must be adequately summarised and concise to reliably inform security risk management decision makers.

Then, there is the question of what does this technical risk information mean to the business in terms of the impacts, their effort to understand and address issues, the potential costs, the impacts on service levels, customers or even insurance premiums? To be fair, those creating the reports may simply not fully know. Yet, these factors are likely to be of particular interest to the business risk team and will require careful reconciliation of the state of the security controls and their potential impact on key business critical IT assets and systems.

Removing bottlenecks

As boards, stakeholders, customers and cyber insurers demand greater visibility, clarity and frequency of security information reporting, these bottlenecks need to be considered and resolved in any cyber risk management process.

Whether the audience is internal and expecting a periodic report, an insurer demanding evidence of security controls to set cyber insurance premiums, or a customer seeking confirmation of your cyber security posture, the demand for timely cyber security information will only increase.

The effort and expertise needed to manage the reporting process, from data gathering to interpretation and then presentation, is significant and requires a common risk management process to support the efforts of each of the multiple interdependent stakeholders. Finding ways to automate these processes in a highly dynamic risk environment is vital for a systematic, accurate and timely cyber security decision making and oversight.

If time can be saved in the data collection and reporting processes, it might just allow work to be undertaken that can enable concerted “risk management wide” focus on finding, mitigating and reporting on performance improvements.