Read the article " Security Controls that Work" by Dwayne Melancon below write a report that answers the following questions.

4. What metrics can an IT auditor use to assess how an organization is performing in terms of change controls and change management? Why are those metrics particularly useful?

Security Controls That Work By Dwayne Melanon, CISA Ask the average IT or security manager what measures his/her organization takes to secure its networks, systems, applications and data, and the answer will most likely involve a combination of traditional perimeter protection solutions (such as firewalls, intrusion detection, antivirus and antispyware) together with patch management, business continuance strategies, and access control methods and policies. All of these measures make sense at first glance, yet the deluge of intrusions, data thefts, worms and other attacks continues unabated, with organizations losing productivity, revenue and customers every year. There are many reasons for this gap in controls and effectiveness. Access controls can be taken only so far before they run into legitimate resistance from employees who find their productivity hampered by the very controls designed to protect it. Traditional perimeter protection and access control are not as effective at blocking attacks from inside organizations as they are at blocking external hackers, which says a lot, since the latter manage to breach thousands of company networks every year. And, as the number and frequency of zero-day attacks continue to grow, the effectiveness of patch management and traditional signature-based intrusion detection, antivirus and antispyware solutions is increasingly in doubt. All of this begs a host of questions: How is it possible to determine whether an organizations security controls actually work? Of all the hundreds of practices and objectives within Control Objectives for Information and related Technology (COBIT), IT Infrastructure Library (ITIL) and the other frameworks an organization may implement, which ones are truly the most effective at helping the organization block and respond to attacksand which ones merely sound good but do not accomplish all that much in practice? Why are some organizations vastly better than others at preventing and responding to attacks? On which controls should auditors focus to verify that the infrastructure is genuinely protected? Come budget approval time, where should the company concentrate its security money, and how can it be demonstrated to senior management that those proposed investments will actually do the job? These are the types of questions the IT Process Institute (ITPI) set out to answer when it was founded in 2000. One of the results of ITPIs work, the IT Controls Performance Benchmark Study,1 proves with empirical evidence that not only are some organizations vastly better than the rest of the pack at preventing and responding to attacks, but also that the difference between these and other organizations effectiveness boils down to just a few foundational controls. And the most significant within these foundational controls are not rooted in access control, but in monitoring and managing change. According to Gene Kim, cofounder with Kevin Behr of ITPI, Security executives often whine that the business does not value security controls, viewing them as bureaucratic and burdensome. What the IT Controls Performance Benchmark Study benchmarking proves is that no matter how many access controls you have, you wont get the performance or security breakthroughs you really want until you tackle change. Pareto in Practice In more than six years of research, the IT Controls Performance Study examined 98 IT groups across multiple industries to determine whether the Pareto Principle, otherwise known as the 80/20 rule, applies to IT controls. The Pareto Principle states that, for many phenomena, 80 percent of the consequences stem from 20 percent of the causes. As part of its research, ITPI was able to identify a small group of very high-performing IT organizations that had the following outstanding characteristics: Superior service levels, measured by the mean time between failures and low mean time to repair The earliest and most consistent integration of security controls into IT operational processes, measured by control location, security staff participation in the IT operations life cycle and number of security incidents resulting in loss The best posture of compliance, measured by the fewest number of repeat audit findings and lowest staff count required to stay compliant High efficiencies, measured by high serverto-system administrator ratios and low amounts of unplanned work (i.e., new work that is unexpectedly introduced when a change is made) Further benchmarks and survey results led to some truly eye-opening observations regarding security. When it came to preventing and responding to security incidents, the high performers, which represented 13 percent of the survey respondents, outperformed their lower-performing peers by a factor of five to 10. When these high performers experienced a breach, they were markedly better at response than their lower-performing peers, for example: High performers typically detected breaches within minutes vs. hours for medium performers and even days for low performers Why are some organizations vastly better than others at preventing and responding to attacks? I NFORMATION S YSTEMS C ONTROL J OURNAL , V OLUME 4, 2007 High performers were far more likely to detect breaches using existing automated controls. Medium performers were 60 percent less likely to detect breaches this way, and low performers were 79 percent less likely to detect breaches with such controls. High performers were 29 percent less likely than companies classified as medium performers to experience financial loss or loss of customers and reputation and 84 percent less likely than companies classified as low performers The corresponding performance gap in operations was similarly dramatic. Compared to medium and low performers, high performers: Completed eight times as many projects Managed six times as many applications and IT services Authorized and implemented 15 times as many changes Achieved server-to-system administrator ratios 2.5 times higher than medium performers and 5.4 times higher than low performers Experienced one-half the change failure rate of medium performers and one-third the change failure rate of low performers Experienced 12 percent less unplanned work than medium performers and 37 percent less than low performers Another interesting finding was that top performers allocated three times more budget to IT as a percentage of their total operating expenses than their lower-performing brethren. This may seem counterintuitive at first, but this finding actually reflects higher IT satisfaction ratings in the business and, therefore, more willingness on the part of senior management to spend a higher percentage of the budget on IT and IT security projects. After all, these organizations have proven they deliver more predictable results with the money they receive, so they can more easily justify funding for additional projects. Which Controls? After identifying high-performing organizations, researchers set out to determine whether there was some consistency in the types of controls most commonly implemented by the high performers compared to their lower-performing counterparts. This would, in turn, provide evidence as to which controls were actually the most effective in helping organizations prevent and respond to security incidents. To do this, researchers identified 63 COBIT control objectives within six ISO 20000 control categoriesaccess, change, resolution, configuration, release and service levels representing the places where high-performing organizations first implement IT controls. They then conducted a survey containing 25 performance indicators spanning audit, operations and security performance measures. These included security effectiveness, audit compliance disruption levels, IT user satisfaction and unplanned work. By analyzing relationships between control objectives and corresponding performance indicators, researchers were able to differentiate which controls are actually most effective for predictable service delivery, as well as for preventing and responding to security incidents. The study concluded that the Pareto Principle does apply. Study results showed that 20 percent of the controls provide 80 percent of the benefit. In this case, researchers found that 21 controls, three to four within each of the six control categories, had the same impact on performance measures as the full set of 63 controls. The next question, however, was whether using more of the 21 foundational controls actually resulted in better security and higher performance. To answer this question, researchers employed a statistical technique called clustering to group similar populations with similar control environments and performance. The goal of this exercise was to find a cluster that achieved the absolute highest levels of performance. Figure 1 shows a representation of the controls of the three clusters that emerged. Each wedge on the polar vector indicates one of the foundational controls, and the size of each wedge shows the percentage of the cluster members that responded yes to questions that mapped to that control. What is immediately apparent is that nearly all the members of the high-performing cluster used all of the foundational controls, while almost all the members of the low-performing cluster used none of them, except those that applied to access and resolution. 2 Figure 1Three Clusters: Low, Medium and High Performers Low Performers Medium Performers High Performers 5: Resolution 0: Access 1: Change 4: Svdlvl 3: Release 2: Config 5: Resolution 0: Access 1: Change 4: Svdlvl 3: Release 2: Config 5: Resolution 0: Access 1: Change 4: Svdlvl 3: Release 2: Config F-Cluster Low F-Cluster Med F-Cluster High I NFORMATION S YSTEMS C ONTROL J OURNAL , V OLUME 4, 2007 What does this mean exactly? Low-performing organizations rely almost exclusively on access controls, such as issuing and revoking passwords, and reactive resolution controls, such as trouble-ticketing systems, to prevent and respond to security incidents. The study further found that out of the 21 foundational controls high performers used, there were two used by virtually all the high performers and none of the low or medium performers. Both are highlighted in figure 2, which overlays the high performers cluster controls with those of the medium performers, indicated by the solid black line. Both of these controls revolve around change management: Are systems monitored for unauthorized changes? Are there defined consequences for intentional unauthorized changes? These two controls are very significant in that they are discriminant controls in this study, meaning that when they are absent from an organization, that organization is never a high performer. Rounding out the top six foundational controls were four change and configuration management controls identified as most present in the high performers and least present in medium and low performers: A formal process for IT configuration management An automated process for configuration management A process to track change success rates (the percentage of changes that succeed without causing an incident, service outage or impairment) A process that provides relevant personnel with correct and accurate information on current IT infrastructure configurations The study found that these top six controls help organizations manage risks and respond to security incidents by giving them the means to look forward, averting the riskiest changes before they happen, and look backward, identifying definitively the source of outages or service issues. Because they have a process that tracks and records all changes to their infrastructure and their success rates, highperforming organizations have a more informed understanding of their production environment and can rule out change as a cause very early in the incident response process. This means they can easily find the changes that caused the incident and remediate them quickly. Low performers lack the means to detect unauthorized change in their IT environments and, therefore, expose themselves to higher security risks and a decreased ability to respond to events quickly. In fact, the study showed that high performers have fewer security incidents, fewer audit findings and lower compliance costs than low and medium performers. Further bolstering the observation that change management is a major differentiator, the study found three things that all high-performing security and IT organizations never do: They never let developers make changes in production. They never let change management processes get bureaucratic. They never let users exceed their role in the change process. What This Means The most impressive aspect of the ITPI study is just how clear and definitive the results are. The organizations that are most successful in preventing and responding to security incidents are those that have mastered change management. Those that are least successful focus all their security resources on access management and reactive resolution controls, and none on change management. The implications are best described in the Visible Ops Handbook. For an organization to be a high performer, it must cultivate a culture of change management and causality throughout, with zero tolerance for unauthorized changes. As with any organizational culture, the culture of change management should start at the top, with leaders establishing a tone that all change must follow an explicit change management policy and process from the highest to the lowest levels of the organization, with zero tolerance for unauthorized change. These same executives should establish concrete, wellpublicized consequences for violating change management procedures, with a clear, written change management policy. Many of the studys high performers said their organization had instituted a policy of warn once, discipline on second offense, and involved top management in the warning process. Those that do not have this culture are likely to show a higher frequency of security incidents, longer and less-effective incident response, more unplanned work, lower service quality, and poorer compliance. One of the components of an effective change management policy is the establishment of a governing body, such as a change advisory board, that reviews and evaluates all changes for risk before approving them. This board reinforces the written policy, requiring mandatory testing for each and every change, and an explicit rollback plan for each in the case of an unexpected result. Postincident reviews are also crucial, so that the organization protects itself from repeating past mistakes. During these reviews, change owners should document their findings and work to integrate lessons learned into future operational practices. Perhaps most important for responding to changes is having clear visibility into all change activities, not just those that are authorized. Automated controls that can maintain a change history reduce the risk of human error in managing the 3 Figure 2High vs. Medium Performer Clusters 5: Resolution 0: Access 1: Change 4: Svdlvl 3: Release 2: Config F-Cluster High I NFORMATION S YSTEMS C ONTROL J OURNAL , V OLUME 4, 2007 process. They also allow IT to take measures such as preimplementation testing or more rigorous change review to improve change success rates and accurately measure the effectiveness of those processes and policies. The Role of Auditing High-performing organizations were able to provide proof that management audited actual practices and enforced accountability for process and policy adherence. Auditors can play a crucial role in moving an organization from the low- or medium-performing category to the high-performing category. By focusing heavily on the following metrics, an IT auditor can get a good picture of how the organization is performing: Amount of time devoted to unplanned workAn unplanned work rate higher than 20 to 25 percent is a sure indication of a lack of effective controls and a cultural problem within IT. It usually means too much time and resources are spent on troubleshooting and maintaining IT operations and not enough time is spent on improving the business. The Visible Ops Handbook indicates that high performers spend less than 5 percent of their time on unplanned work. Volume of emergency changesAlmost by definition, emergency changes are unauthorized changes that are often used as a way to circumvent the formal change management process or avoid disciplining employees for violating those processes. If an organization has a volume of emergency changes that exceeds 15 percent, auditors should take that as a warning sign that it is not taking change management seriously. The highest performers tend to have 5 percent or fewer emergency changes. Also, it is important to ensure that there is an actual process, albeit streamlined, for emergency changes. Number and causes of failed changesThe ITPI study found that high performers consistently maintained successful change rates of 95 percent or more, often as high as 99 percent. Successful changes are those that are implemented without causing an outage or unplanned work episode. Other things to look out for, which the study found in medium and low performers, include: A high frequency of security incidents, unexplained outages or other system availability events A lot of late projects and cost overruns due to unplanned or emergency work High employee turnover and low morale Auditors also should examine the automated controls used by the organization to gain visibility into all change activities, not just authorized changes, to determine if the change management technology successfully covers all the right foundational controls. Some of these technology types include: PreventiveThis is usually a change management or authorization system, such as an IT service or help desk, that can create an audit trail of authorizations, track the status of changes and guide the overall change process. DetectiveThis technology uses automated, independent detective controls or random change audits to monitor the production environment for changes, compare changes with authorizations, and detect undocumented changes that circumvent the change review and authorization process or violate policy. Called out of band changes, these also include extra changes hidden in an authorized work order. CorrectiveThis technology implements processes, such as provisioning or backup and restoration programs, that can revert unauthorized or troublesome changes and restore the system to a known, authorized, supported state. Look at the Numbers The results of the IT Controls Performance Study make a strong case, based on empirical evidence, that most of the value of IT security controls comes from implementing a small subset of COBIT or other controls centered around change management. Organizations that focus on access and reactive resolution controls at the expense of change management are guaranteed to experience more security incidents, more damage from security incidents, and dramatically longer and less-effective resolution. Organizations that foster a culture of disciplined change management and causality, with full support from senior management, and have zero tolerance for unauthorized change, will have a superior security posture with fewer incidents, dramatically less damage to the business from security breaches and much faster resolution of incidents when they happen. Change management is particularly effective at detecting internal security breaches, which many existing security strategies and technologies, such as firewalls and access controls, fail to address adequately. A recent Deloitte Touche Tohmatsu study found that almost half of all surveyed financial services companies had experienced an internal breach in the past year.2 Security is not the only benefit of a culture of change management. Organizations that foster a culture of change management also perform dramatically better than their less change-oriented counterparts in just about every way, from less unplanned work to more successful IT projects, higher number of successful changes and much more efficient use of IT resources. The security managers who are gaining responsibility and budget are those who are tackling the harder issues around change, said Gene Kim. Those who dont will continue to shrink in responsibility or have their air cut off. Endnotes 1 ITPI, IT Controls Performance Benchmark Study, April 2006, www.itpi.org 2 Deloitte Touche Tohmatsu, Global State of Information Security, 2005, www.deloitte.com Dwayne Melanon, CISA is the vice president of corporate and business development at Tripwire. He is a specialist in strategic partnerships and alliances, and developing professional services and support organizations. Melanon is certified on both IT management and audit processes, possessing ITIL Foundations.

actually the most effective in helping organizations prevent and respond to security incidents. High performers were far more likely to detect breaches using existing automated controls. Medium performers were 60 percent less likely to detect breaches this way, and low performers were 79 percent less likely to detect breaches with such controls To do this, researchers identified 63 CoBIT control objectives within six ISO 20000 control categories-access, change, resolution, configuration, release and service levels representing the places where high-performing organizations first implement IT controls. They then conducted a survey containing 25 performance indicators spanning audit, operations and security performance measures. These included security effectiveness, audit compliance disruption levels, IT user satisfaction and unplanned work. By analyzing relationships between control objectives and corresponding performance indicators, researchers were able to differentiate which controls are actually most effective for predictable service delivery, as well as for preventing and responding to security incidents . High performers were 29 percent less likely than companies classified as medium performers to experience financial loss or loss of customers and reputation and 84 percent less likely than companies classified as low performers The corresponding performance gap in operations was similarly dramatic. Compared to medium and low performers, high performers Completed eight times as many projects Managed six times as many applications and IT services Authorized and implemented 15 times as many changes Achieved server-to-system administrator ratios 2.5 times higher than medium performers and 5.4 times higher than low performers Experienced one-half the change failure rate of medium performers and one-third the change failure rate of low performers The study concluded that the Pareto Principle does apply. Study results showed that 20 percent of the controls provide 80 percent of the benefit. In this case, researchers found that 21 controls, three to four within each of the six control categories, had the same impact on performance measures as the full set of 63 controls .Experienced 12 percent less unplanned work than mediunm performers and 37 percent less than low performers The next question, however, was whether using more of the Another interesting finding was that top performers allocated 21 foundational controls actually resulted in better security and three times more budget to IT as a percentage of their total operating expenses than their lower-performing brethren. Thisemployed a statistical technique called clustering to group may seem counterintuitive at first, but this finding actually reflects higher IT satisfaction ratings in the business and, therefore, more willingness on the part of senior management to that achieved the absolute highest levels of performance. spend a higher percentage of the budget on IT and IT security projects. After all, these organizations have proven they deliver clusters that emerged. Each wedge on the polar vector more predictable results with the money they receive, so they indicates one of the foundational controls, and the size of each can more easily justify funding for additional projects. higher performance. To answer this question, researchers similar populations with similar control environments and performance. The goal of this exercise was to find a cluster Figure 1 shows a representation of the controls of the three wedge shows the percentage of the cluster members that responded "yes" to questions that mapped to that control. Which Controls? What is immediately apparent is that nearly all the members After identifying high-performing organizations, researchers of the high-performing cluster used all of the foundational set out to determine whether there was some consistency in the controls, while almost all the members of the low-performing types of controls most commonly implemented by the high performers compared to their lower-performing counterparts and resolution. This would, in turn, provide evidence as to which controls were researchersof the high-performing cluster used all of the foundational cluster used none of them, except those that applied to access Figure 1-Three Clusters: Low, Medium and High Performers Low Performers Medium Performers High Performers 0: Access 5: Resolution 0: Access US 4: Svdlvl 4: Svdlvl ge Hi Ig 1: Change Ch nge 2: Config 3: Release lg 1g INFORMATION SYSTEMS CONTROL JoURNAL, VoLUME 4, 2007