Evaluation of the Registry - Final Report

November, 2021

Submitted by:
Samson

Table of Contents

• Executive Summary
• 1 Introduction
• 2 Overview of the Registry
• 3 Evaluation Methodology
  • 3.1 Interviews with Representatives from Institutions
  • 3.2 Interviews with OIC Personnel
  • 3.3 Validity of the Evaluation Methodology
• 4 Evaluation Findings
  • 4.1 The Registry is Considered to be Efficient and Effective by OIC Inside “Clients”
  • 4.2 External Institutions Are Satisfied with the Registry’s Performance, and Have Suggestions
  • 4.3 The Registry Continues to be Relevant
• 5 Conclusions
• Appendix A
• Appendix B

Executive Summary

This report sets out the findings and conclusions from the evaluation of the Office of the Information Commissioner of Canada’s (OIC) Registry function. This evaluation project was the first part of a multi-phased evaluation to address the relevance and performance of the OIC’s Investigations Program. The evaluation of the OIC’s Registry was conducted between April and August 2021.

The Registry serves as a point of entry for complainants, and manages the “production” aspects of files from start to finish, focusing on overall quality in efficiency and uniformity. The Registry is the gateway for formal correspondence between the OIC and its stakeholders. It is a key element in the flow of information in the investigation process and is an important contributor to the OIC’s processes and, hence, credibility.

The focus of the evaluation was on the relevance and performance of the Registry. The evaluation included interviews with key informants from institutions that worked with the Registry and with OIC senior managers and Registry management to obtain their views on the evaluation questions.

Evaluation Findings

Overall, the evaluation determined that internal personnel were very satisfied with the Registry’s effectiveness and, generally, its efficiency. They believe that that the Registry is effective at delivering on its mandate. With respect to efficiency, all internal interviewees noted there is a need to modernize the technology that supports the Registry. They said that the Registry functions well within the limitations of the automated tool it has. Also, OIC senior management was able to access all the management information needed to oversee and monitor the Registry’s performance.

External institutions that work with OIC were, overall, very satisfied with the Registry’s performance. In particular, they found the Registry’s current documentary and timing requirements clear. They also felt that they were able to raise emerging issues with the Registry and resolve them, which is an important feature in maintaining positive working relationships. There were also a number of suggestions from the institutions’ representatives:

• More information on the context of requests could enable them to respond more accurately, thereby increasing their efficiency
• Using the same information for multiple related complaints could increase efficiency
• Timeframes for response were too short, and
• Monthly activity reports could help in coordinating activities.

Finally, interviewees indicated that the current organizational model of the Registry is viewed as a relevant model to meet OIC’s needs.

Conclusions

The Registry continues to be relevant, it is effective in achieving its intended outcomes, and the positive working relationships enable the resolution of any emerging issues. The Registry’s efficiency would be increased if the automated tools were modernized. Even so, the Registry is viewed as being generally efficient. Overall, the Registry is performing well and will likely be even more successful with some technology modernization.

This report sets out the findings and conclusions from the evaluation of the Office of the Information Commissioner of Canada’s (OIC) Registry function.

1 Introduction

The OIC supports the Information Commissioner in her advisory role to Parliament and parliamentary committees on all matters pertaining to access to information.

This evaluation project was the first part of a multi-phased evaluation to address, as per the Policy on Results, the relevance and performance of the OIC’s Investigations Program. This first phase focused on the OIC’s Registry function. The second phase will evaluate the overall Investigations Program. Investigations is the OIC’s only program, and the Risked-based Audit and Evaluation Plan recommends it be evaluated every five years.

The evaluation of the OIC’s Registry was conducted between April and August 2021.

2 Overview of the Registry

The Registry serves as a point of entry for complainants, and manages the “production” aspects of files from start to finish, focusing on overall quality in efficiency and uniformity. In summary, the Registry receives and assesses the validity of complaints, obtains all relevant documents from institutions in order to conduct investigations, formats and sends investigation documents to external parties, and publishes Final Reports on the OIC’s external website. Overall, the Registry is the gateway for formal correspondence between the OIC and its stakeholders. It is a key element in the flow of information in the investigation process and is an important contributor to the OIC’s processes and, hence, credibility. When the Registry is effective, the following medium term intended outcomes are achieved:

• OIC’s case management database provides useful, complete, and accurate information about complaints
• OIC devotes its effort only to cases within its jurisdiction
• OIC’s information and document flow related to cases is efficient and effective in supporting the complaints process
• OIC reports are made available to Canadians in a timely manner
• OIC is viewed by stakeholders as credible, effective and efficient

A notable change was introduced to the Registry function in 2019. The Commissioner created the current Registry structure in 2019 in order to better manage the “production” aspects of complaint investigation files from start to finish, to ensure an improvement in the uniformity and quality of products, as well as an increase in the efficiency of processes. Prior to this centralization of the Registry function, the activities were carried out in separate investigation teams, with a less unified approach.

Finally, the Registry is responsible for the integrity of the data held in the OIC’s internal case management database, called Intrac.

As of April 1, 2021, the Registry had a resource base of 10 FTE and $732,000.

A logic model for the Registry function is attached as Appendix A.

3 Evaluation Methodology

The evaluation methodology was developed during a planning phase that included the review of documentation, internal interviews and observation of the Registry activities. A logic model was developed to clarify the activities, outputs, and intended outcomes. Interviews provided the information to define the evaluation issues and questions, identify interviewees, and define the parameters for applying the methodologies. An Evaluation Workplan was developed and approved by the Project Authority.

The focus of the evaluation was on the relevance and performance of the Registry. Therefore, the evaluation questions were designed to address those key issue areas. Given the nature of the Registry’s activities, the findings relied heavily on the findings from interviews with internal “clients” and external institutions that OIC works with. The interviews included open-ended questions to enable respondents to provide a full range of views on the Registry.

3.1   Interviews with Representatives from Institutions

The evaluation included interviews with key informants from external institutions that worked with the Registry. The following criteria were used to select the institutions for interview:

• The largest-volume external institutions were included -- Immigration, Refugees and Citizenship Canada and the Royal Canadian Mounted Police
• Other higher-volume institutions were included to reflect a diversity of sizes, and to include some that are “operational” and not “operational.” The following institutions were included: Canada Border Services Agency, Canadian Broadcasting Corporation, Transport Canada, Public Safety Canada, and Privy Council Office.

These institutions know the Registry function quite well and were in a position to provide useful information on the evaluation questions. The OIC works with almost all federal institutions. However, it was determined that these core external institutions were best positioned to provide knowledgeable and relevant information to support the evaluation.

3.2   Interviews with OIC Personnel

Interviews were held with OIC senior managers and Registry management to obtain their views on the evaluation questions. The OIC’s three Senior Investigators were interviewed as internal “clients” of the Registry. Selected Registry staff also provided views on the evaluation questions.

The list of internal interviewees is included as Appendix B.

3.3 Validity of the Evaluation Methodology

Overall, from an evaluation perspective, the methodology is sufficient to provide a high level of confidence in the accuracy of the findings and conclusions.

4 Evaluation Findings

The following sub-sections present the findings from the evaluation methodologies.

4.1   The Registry is Considered to be Efficient and Effective by OIC Inside “Clients”

Interviewed internal personnel, all of which understand the Registry well, believe that the Registry is effective at delivering on its mandate.

The interviewees indicated they believe that the Registry’s current roles and responsibilities are suitable. Some noted that the current centralized model (established in 2019) is a much improved structure over the one that was in place previously.

Interviewees believe that the Registry is discharging its assigned responsibilities effectively. In particular, there have been no concerns related to the Registry’s performance in assessing “validity”, which is an important screening role performed by the function. Furthermore, they report there have been few instances of inaccurate information in Intrac, and Registry resolved those quickly. Overall, interviewed investigators felt they were well supported by the Registry during the investigation process.

With respect to efficiency, all internal interviewees noted there is a need to modernize the technology that supports the Registry. They said that the Registry functions well within the limitations of the automated tool it has – Intrac. However, efficiency would increase if the system had features such as auto-fill, data capture, and formatting capabilities. There was no attempt to quantify this potential increase in efficiency.

Interviewed members of OIC senior management indicated that they were able to access all the management information they needed to oversee and monitor the Registry’s performance, which is an important consideration for any organization. Furthermore, when asked what “score” they would give to the Registry’s performance, they answered “8” and “9”, reflecting a high level of satisfaction. The interviewees indicated that the scores would be higher is the Registry’s automated tools were modernized, leading to improved performance.

Overall, the evaluation determined that internal personnel were very satisfied with the Registry’s effectiveness and, generally, its efficiency. This is an important finding because these interviewees are very familiar with the Registry, and would be impacted by any difficulties. The finding also suggests that Registry management works closely with internal operations to resolve outstanding or emerging issues, which is a positive finding on its performance.

4.2 External Institutions Are Satisfied with the Registry’s Performance, and Have Suggestions

Interviewed departmental contacts were very satisfied overall with their relationship with the Registry and how the Registry was functioning. To help provide a quantitative measure, when asked to provide a score of 1-to-10 on the performance of the Registry, where “10” is the highest score, the following were the results (some departments had multiple interviewees):

• Score of 9: 5 interviewees
• Score of 8: 3 interviewees
• Score of 7: 1 interviewee

Many of these interviewees provided a score of “9” or “8” indicating that there was always opportunity for improvement, even though they had no specific issue of concern.

This finding suggests that external institutions were, overall, very satisfied with the Registry’s performance. In particular, they found the Registry’s current documentary and timing requirements clear. They also felt that they were able to raise emerging issues with the Registry and resolve them, which is an important feature in maintaining positive working relationships. One interview summarized comments by saying: “If OIC wanted to invest funding into improving operations, Registry is not the place where I’d spend the money.”

Even though there was a high level of satisfaction, there were a number of suggestions identified by the institutions’ representatives:

More information on the nature of the requests: Two interviewees indicated that they would like to receive from OIC more information about the nature of complaints. One described the situation as follows: “The text that describes what they (OIC) are looking at and motivations aren’t clear. So I can’t tell what they are investigating.”  Both interviewees indicated that, if they had more information on context, they could respond more accurately to the request, and this would increase their efficiency.

Reducing redundancies: One interviewee wondered why the OIC couldn’t use the same information for multiple related complaints. They believed that doing so would increase efficiency by not requiring repetitive submissions of the same information.

Timeframes are too short: Several interviewees indicated that the timeframes for response were too short. One indicated that they could never meet the deadlines, and that requesting an extension had become the norm. Many found it frustrating that they had only a small number of days to provide documents, but that the investigation would often occur only months later, thereby not justifying the initial rush. It affected their efficiency because, months later at the time of investigation, in order to respond to further requests, they would need to once again become familiar with the file. Even though this comment came from several sources, they all indicated that, along with the OIC, they had found resolutions that worked, and that the operations ran smoothly.

Monthly reports: Two interviewees indicated that they would find it helpful to receive from the OIC a monthly report of outstanding files. Having that report would help coordinate activities on an ongoing basis.

Other small items reported included:

• It would be useful to receive timely information about changes in OIC processes that affect institutions
• In past years, the mailroom occasionally misplaced correspondence; however, that has now improved.

4.3 The Registry Continues to be Relevant

The evaluation did not include an explicit question about whether the Registry function was relevant. Given the nature of OIC’s mandate, it is essential to have a formal function that registers and manages correspondence flows.

As indicated above, there is general agreement that the Registry discharges all its assigned responsibilities effectively and, generally, efficiently.

It is noteworthy that interviewees who were familiar with past OIC operational models commented that the current Registry model worked “much better” than the previous model did.

As a result, the current organizational model of the Registry is viewed as a relevant model to meet OIC’s needs.

5 Conclusions

The evaluation findings support the conclusion that the Registry continues to be relevant. Internal clients and knowledgeable external institutions view the Registry as effective in achieving its intended outcomes, and they have a positive working relationship that enables the resolution of any emerging issues.

Internal interviewees indicated that efficiency would be increased if the automated tools – Intrac – were modernized. Even working within Intrac’s limitations, the Registry is viewed as being generally efficient. As a result, there is no compelling reason for this evaluation study to propose a recommendation to modernize the Registry’s automated tools. This is a known concern, and it could be expected that this investment would be made in due course by the OIC.

Overall, the Registry is performing well and will likely be even more successful once Intrac is modernized.

Appendix A

Office of the Information Commissioner

Evaluation of the OIC Registry

Logic Model

Appendix B

Internal Interviewees

Layla Michaud, Deputy Commissioner of Investigations
Allison Knight, Senior Director, Investigations and Registry

Mélanie Séguin, Manager of Operations, Registry

Eric Girard, Senior Investigator
Heather Rousson, Senior Investigator
Natasa Pushkar, Senior Investigator