Calculating and Consolidating Compliance Metrics

Now that the basic concepts for compliance surveys and response sets have been presented, we can proceed to a discussion of how the compliance metrics supported by the system are calculated and consolidated. We want to investigate exactly how the Compliance Index and the Compliance Level – once obtained for the Compliance questions – are consolidated for the requirements associated with questions, for the authoritative documents that contain the requirements, as well as for the objects included in the scope of a compliance project. You will then be able to understand and interpret the indicators shown in the Compliance Analysis Report, in queries, and in charts from the Dashboard module.

When specifying the scope of a compliance project, a compliance survey, a response set (which defines the possible answers for the Compliance questions), an interviewee, and a reviewer (optional) are selected for each object to be analyzed. It is by means of these associations in the context of a project that a compliance interview is generated (see figure below). It is through these associations that a compliance interview is generated within a project. As the answers for the Compliance questions in the interview are processed, the compliance metrics are obtained directly based on the response set selected.

Each Compliance question answered by an interviewee when completing a compliance interview will have a certain value associated for the Compliance Index (%) and the Compliance Level. These values will be obtained according to a) the response option selected by the interviewee and b) the corresponding values for the response option, considering the response set associated with the interview in the scope of the project. Note that the response set is selected for the interview as a whole; thus, all Compliance questions for the interview will use this same set and will present all the same response options to the interviewee.

For example, consider the following compliance question: “How would you evaluate the maturity of the organization’s backup process for the servers?”

Suppose this question is part of an interview that was associated with the “Maturity” response set with its six options when the scope of a project is being specified (see figure below).

If the interviewee selects option 2 as a response, the value of the Compliance Index for this question, for this interviewee, will be 20%, and the corresponding Compliance Level will be “Not Compliant”.

Note that similar interviews (those based on the same survey using the same response set) may be sent to various interviewees. Thus, similar compliance questions may be answered differently by different interviewees, generating different compliance metrics.

Suppose the question: “How would you evaluate the maturity of the organization’s backup process for the servers?” is included in two interviews using the same response set, and they are sent to two interviewees, John and Joan. It may be that Joan selects option 2 and John selects option 5. In this event, the value of the Compliance Index for the question in John’s interview will be 80%, while in Joan’s it will be 20%. The values for the Compliance Level will also be different (“Compliant” for John, “Not Compliant” for Joan). For this reason, the responses for all the interviews sent during a compliance project must be consolidated in order for the compliance metrics to be meaningful. The system consolidates these automatically, and allows the status of each interview to be monitored (sent, partially answered, cancelled, etc.).

By sending compliance interviews to different people in the organization, the varied perceptions on the level of fulfillment of certain requirements can be measured. This allows more reliable results to be obtained that do not depend on the isolated opinion of one person alone.

To satisfy a greater number of use cases, the system allows interviews containing the same questions using different response sets to be sent. In the example shown in the figure below, interview I1 and I2 were sent to John and Joan that were generated based on the same survey. As the same compliance survey was used to generate the two interviews, they will have the same questions and the same associations between Compliance questions and requirements from authoritative documents. However, the response sets associated with each interview are different. For Joan, interview I1 was sent, which uses the response set R1, while interview I2 was sent to John, which uses the response set R2. Thus, both interviewees will answer the same questions but will have different response options.

As shown above, the Compliance Index and the Compliance Level for Compliance questions are generated directly from the response set. Suppose that three interviews (I1, I2, and I3) generated based on different surveys (containing different questions) were sent to John, Joan, and Maria (see figure below). The table below shows the responses obtained from the interviewees already containing the value of the Compliance Index and the Compliance Level for each question (obtained directly from the response set). In this scenario, the Compliance Index for question 37 in interview I1 is 40% and its Compliance Level is “Compliant”.

Note that each interview was sent to a different interviewee, although the system allows various interviews to be sent to the same interviewee. In addition, although in the example all the interviews use the same response set, it would also have been possible for each interview to use a different response set.

Once the compliance metrics are obtained for the questions, we can consolidate the results obtained for each question to show results for each requirement, for each authoritative document, and for each object, as explained in the table below.

Using the compliance metrics obtained for the questions, the first important consolidation can take place: the Compliance Index and Compliance Level for a requirement from an authoritative document may be obtained. In the example below, there are three compliance questions (37, 45, and 53) from different interviews associated with one requirement (R170) from authoritative document XYZ. Each question has its own Compliance Index and Compliance Level values obtained directly from the response set used.

Thus, the Compliance Index for question 37 is 40%, for question 45 is 60%, and for question 53 is 20%. What is the consolidated value of the Compliance Index for requirement R170 with which the questions are associated?

To calculate the Compliance Index for a certain requirement, the summation of the Compliance Index (%) values for the questions directly or indirectly related to the requirement is divided by the number of questions considered. In the example above, the Compliance Index for R170 is 40%. Although the calculation is simple, some considerations must be added to the explanation for calculating the Compliance Index of a requirement. For example, a question may be connected to a parent and child requirement at the same time. How will this affect the calculations?

Say we have a question associated with a child requirement and its parent requirement as well. For example, in the figure below, question Q4 is associated directly both with child requirement R3.1 and parent requirement R3. The Compliance Index for the child requirement R3.1 is (10% + 80%) / 2 = 45%. What is the Compliance Index for parent requirement R3?

Question Q3 is connected indirectly to parent requirement R3 through the child requirement R3.1, and contributes 10%. Question Q4, however, is directly connected both to the parent requirement as well as to the child requirement, and contributes 80%. The Compliance Index for R3 is obtained based on all the questions connected directly or indirectly to it. Thus, question Q3, connected indirectly to R3, counts once and contributes 10%, while question Q4, which is connected both indirectly and directly to R3, counts twice in the consolidation. The result would be obtained as follows:

(10% (Q3) x 1 + 80% (Q4) x 2) / 3 = 56.67%

What would the Compliance Index be if there were no questions connected directly to the parent requirement R3 (see figure below)?

In this case, both questions Q3 and Q4 are connected indirectly to R3 through the child requirement R3.1. However, Q4 is no longer directly connected to R3 in this example. Thus, the Compliance Index would be:

(10% (Q3) x 1 + 80% (Q4) x 1) / 2 = 45%

Note that question Q4 now only counts once in the calculation for the parent requirement R3.

To obtain the Compliance Level for a requirement, the worst result obtained for the Compliance Level is considered, taking into account all of the questions directly associated with it. In the example in the figure below, requirement 170 is related to three questions (37, 45, and 53), and the worst case for the Compliance Level is “Not Compliant” for question 53. Thus, by convention, the Compliance Level for requirement 170 will be “Not Compliant”.

Consolidating the Compliance Index for an authoritative document is based on the Compliance Index for the questions associated with all the requirements in the document, at all levels. In the example shown in the figure below, the consolidated Compliance Index for the authoritative document XYZ is 42.85%, using the calculation below:

CI AuthDoc = (40% + 60% + 20% + 20% + 60% + 20% + 80%) / 7 = 42.85%

In a more complex scenario in which there are questions associated at the same time with parent and child requirements, the Compliance Index for authoritative document XYZ is 37.14%, using the calculation below:

CI = (Q1 x 1 + Q2 x 2 + Q3 x 2 + Q4 x 2) / 7

CI = (40 + 40 + 20 + 160) / 7

CI = 37.14%

Note that question Q4, which is associated with the child requirement R3.1 and to its parent requirement R3, is counted twice in the consolidation. As for the Compliance Level, the worst-case scenario is again used. Thus, if one of the questions associated with the requirements from the document has a Compliance Level of “Not Compliant”, the Compliance Level for the document as a whole will be “Not Compliant”.

To calculate the Compliance Index for an object, the method used is to sum the Compliance Index for all the questions from all the finalized interviews associated with the object when the scope of the project is specified, and then dividing by the number of questions. The associations between the questions with the requirements are not considered in the calculation (*).

In the example shown in the figure below, assuming interviews 1 and 2 are finalized, the Compliance Index for Object 1 will be:

(40% + 60% + 20% + 20% + 60%) / 5 = 40%

*Each object can only be associated with one survey in the scope. However, this same object can be included again in the scope and associated with a different survey.

To consolidate the Compliance Level for an object, the worst-case scenario is again used among all the questions for all the interviews associated with the object. If at least one question is answered as “Not Compliant”, the Compliance Level for Object 1 will be “Not Compliant”.

Keep in mind that a Compliance Level of “Not Applicable” does not generate compliance metrics. If all the questions associated with a requirement are answered as “Not Applicable”, this requirement will not appear in dashboards or in queries.