Audit of Establishment-based Risk Assessment model

July 2025

On this page

• 1. Results in brief
• 2. Context
  • 2.1 Development of the ERA model
  • 2.2 How the ERA model works
• 3. About the audit
  • 3.1 Objective
  • 3.2 Scope
  • 3.3 Approach
  • 3.4 Statement of conformance
• 4. Findings and recommendations
  • 4.1 ERA model governance
  • 4.2 Data quality/management
• 5. Audit conclusion
• 6. Appendix A: Audit criteria
• 7. Appendix B: List of acronyms
• 8. Appendix C: Glossary
• 9. Appendix D: ERA model mathematical calculation
• 10. Appendix E: Inherent, mitigation and compliance risk factors
• 11. Appendix F: Internal audit recommendation priority criteria
• 12. Appendix G: Management response and action plan

1. Results in brief

The Canadian Food Inspection Agency (CFIA, the Agency) is dedicated to safeguarding food, which includes performing inspections of federally regulated domestic food establishments to verify industry is complying with the Acts enforced by the CFIA. The CFIA uses data and information to identify risk trends and support the program design, planning, compliance, and enforcement efforts. As such, the CFIA developed the Establishment-based Risk Assessment (ERA) model which is a key risk modeling tool used by the Agency to evaluate domestic food establishments based on the risk they represent to Canadians.

The objective of this audit was to provide assurance that there are effective governance, risk management^{Footnote 1} and internal control processes to support the Agency's ERA model. The scope period of this audit was from April 1, 2023, to December 31, 2024.

The audit concluded there are effective governance elements in place to support the Agency's ERA model. Specifically,

• Clearly defined roles, responsibilities, and processes are in place to support the governance and scientific validity of the model.
• Data sourcing processes for the model are in place and documented to support data extraction activities.

There are opportunities to strengthen oversight, obtain compliance with the Treasury Board Secretariat Directive on Automated Decision Making, and improve the effectiveness of data quality assurance activities and processes.

The audit recommends the CFIA:

• document the standard operating procedures for updating the algorithm and establish a centralized repository to document the evidence, status and approvals for each algorithm update
• review the Directive on Automated Decision Making and conduct the necessary activities to bring the ERA model into full compliance
• develop a formal data quality assurance program for the ERA model, leveraging existing tools and initiatives

2. Context

The Canadian Food Inspection Agency (CFIA) (the Agency) is dedicated to safeguarding food, animals, and plants, which enhances the health and well-being of Canada's people, environment, and economy. As part of this commitment, the Agency performs inspections of federally regulated domestic food establishments to verify that industry is fulfilling their responsibility to produce safe food and meeting Canadian standards and regulations.

The Safe Food for Canadians Act (SFCA) governs the Agency's food establishment inspection activities; the Safe Food for Canadians Regulations (SFCR) came into force in January 2019, modernizing CFIA's regulatory framework and strengthening Canada's food safety system by applying federal food safety oversight to all foods sold interprovincially. There are nine categories that a food commodity generally falls under: Meat, Fish and Seafood, Dairy, Fresh Fruits and Vegetables, Processed Fruit or Vegetable Products, Egg and Egg Products, Manufactured Foods, Maple, and Honey.

Risks to food safety continue to evolve with global trading patterns, innovation, and new technology. In this environment, the CFIA must continue to adapt to help protect Canada's resources which supports industry's ability to compete globally.

The CFIA uses data and information to identify risk trends and support program design, planning, compliance, and enforcement efforts. As such, the CFIA has developed tools to support risk management activities, including the Establishment-based Risk Assessment (ERA) model. The ERA model uses data to evaluate domestic food establishments based on the risk they represent to Canadian consumers and along with other factors, is used to inform program priorities and guide where the CFIA should focus inspection efforts (for example, establishments of highest risk). In 2024-25, the preventative control inspection- system verification (PCI-SV) inspection activity for domestic food establishments accounted for approximately 6.5% of the Agency's planned inspection work, and this is projected to increase to around 15% in 2025-26.

2.1 Development of the ERA model

The development of the ERA model (the model) started in 2013 and was a collaboration between the CFIA, academia, industry, and government partners. Model development considered scientific literature, leveraged modelling technology, and drew upon the experience of other countries that have used a similar approach to risk assessment.

Between 2013 and 2014, in the early stage of the ERA model development, a benchmarking exercise was conducted to better understand how countries with similar food safety systems implemented their risk assessment approach and allocated inspection resources; meetings were held with France, Belgium, the Netherlands, Australia, New Zealand, Ireland, the United Kingdom and the United States of America.

Key steps in the development of the model included:

• identifying key pathogens that have the most impact on food safety
• identifying, refining, and weighting risk factors
• performing risk attribution at the commodity (for example, dairy) and sub-product (for example, cheese made with pasteurized milk) levels
• designing the mathematical algorithm
• assessing the performance of the model by testing the agreement between model outputs and assessments done by CFIA senior inspectors

A preliminary version of the ERA model algorithm was piloted in 2014, with the final version being completed in 2016. Commodities were formally integrated into the model on a rolling basis between 2019 and 2023. This process was supported by scientific elicitations, literature reviews, and consultations with the Scientific Advisory Committee (detailed information provided in section 4.1.2).

Figure 1: Timeline overview of key events

This figure illustrates the timeline of key events in the ERA model development and integration.

• 2013: development of the ERA model started
• 2014: pilot of the preliminary version of the ERA model
• 2016: completion of the final version of the ERA model
• 2019: integration into the Dairy and Maple commodities
• 2020: integration into the Honey and Egg and Egg Products commodities
• 2021: launch of ERA model IT solution
• 2021: integration into the Fish and Seafood and Processed Fruit or Vegetable Products commodities
• 2022: integration into the Meat and Poultry commodity
• 2023: integration into the Fresh Fruit and Vegetables and Manufactured Foods commodities

An automated Information Technology (IT) solution for the ERA model was implemented in April 2021 to allow for automatic data extraction and execution, and improved reporting functions. Prior to implementation, a validation was conducted to ensure the automated IT solution results were consistent with the non-automated model. The Meat and Poultry commodity was integrated in July 2022. The final commodities, Fresh Fruit and Vegetables and Manufactured Foods, were integrated in April 2023.

2.2 How the ERA model works

• 2.2.1 Design of the ERA model
• 2.2.2 Inputs into the ERA model

2.2.1 Design of the ERA model

The ERA model evaluates food safety risk of individual establishments subject to the SFCA/R based on their potential impact on consumer health in Canada. The model assesses risk using scientific data, establishment-specific information gathered from regulated parties and an establishment's compliance history. This impact is based on an establishment's activities and production volume. It is measured in Disability-Adjusted Life Years (DALYs), a widely recognized measurement commonly used by science researchers and policymakers to estimate and compare the burden of disease. The DALYs are calculated for each of the 18 pathogens that cause over 99% of the microbial food-borne disease burden in Canada. The health impact of each pathogen takes into consideration the number of annual food-borne cases associated with the pathogen, the health impact per case of illness for the pathogen, and its attribution to specific food commodities and sub-products within the nine commodity groupings. To calculate the total DALYs for a particular sub-product, the DALYs for all applicable pathogens related to the sub-product are added together. The health impact is then allocated to individual establishments based on the type of activity and the volume of each sub-product type they manufacture resulting in the initial DALYs (Figure 2).

An establishment's initial DALYs are then adjusted by multiplying it by the establishment's mitigation and compliance risk factors to produce the final establishment risk result. This result is used to assign an establishment to one of seven risk categories, with Category 1 representing the highest risk and Category 7 representing the lowest risk. The assigned risk category is used as one of the inputs into the work planning process for inspection activities related to food establishments.

See Appendix D for the algorithm equation and related information sources.

2.2.2 Inputs into the ERA model

The ERA model uses three different types of risk factors to determine an SFCR licenced establishment's level of risk:

1. Inherent risk factors are the risks associated with a specific food commodity, operation or manufacturing process. Some of the inherent risk data is used in the initial DALYs calculation for an establishment, while the remaining inherent risk data is used in the risk adjusting variable.
2. Mitigation factors are the measures or strategies that a food establishment has implemented to control the inherent risks and reduce the overall risk of a food safety issue. Examples include the presence of a dedicated quality assurance employee and a microbiological sampling plan.
3. Compliance factors refer to a food establishment's track record on how well it has complied with its own preventive control plan and with regulatory requirements, and is assessed using information from inspection reports, complaints, recalls, and enforcement actions and control measures taken.

Figure 2: ERA model design

This figure illustrates the model design as an arrow. First, the initial DALYs are calculated by attributing the health impact to 4 factors: type of activity, commodity, type of products and volume. Then, the health impact is adjusted by the inherent risk factors, the mitigation factors, and the compliance factors represented by the first, second and third boxes, respectively. This then generates the health impact at the establishment level represented by the last box.

Inherent risk factors take into account:

• the processing steps
• the direct distribution to vulnerable population

Mitigation factors include:

• presence of additional processes
• food safety certifications
• third party audits
• control of incoming supplies
• quality assurance personnel
• presence of a sampling plan

Lastly, compliance factors are assessed using:

• inspection results and impact assessment
• history of enforcement actions
• recalls (class I, II, III)
• food safety confirmed complaints

Inherent risk factors and mitigation factors are sourced from Additional Establishment Information. Compliance factors are sourced from inspection data.

Inherent risk factor and mitigation factor data are collected through the Agency's client portal (My CFIA) from the Additional Establishment Information (AEI) questionnaire completed by the establishment. The AEI is stored in the Digital Service Delivery Platform (DSDP).

Compliance factor data is extracted from the CFIA databases and website. The model extracts data for inspection results from the DSDP, while recalls and confirmed food safety complaints are extracted from the Issues Management System (IMS). Enforcement actions and control measures come from data available on CFIA's external website and DSDP.

3. About the audit

3.1 Objective

The objective of the audit was to provide assurance that there are effective governance, risk management^{Footnote 1} and internal control processes to support the Agency's ERA model.

3.2 Scope

The scope included the governance and data quality processes that support ERA model for food, as well as the digital integration of information feeding into the model. The scope period was from April 1, 2023, to December 31, 2024.

The scope excluded the following: integration of ERA model results into work planning; other risk assessment models (for example, ERA model for hatcheries, renderers, and feed mills, Importer Risk Assessment Model (IRAM)); the coding of the ERA model algorithm; and inspector training.

3.3 Approach

The audit was conducted in accordance with the requirements of the Treasury Board Policy on Internal Audit and the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing. The audit was planned and performed to obtain reasonable assurance that the audit objective was achieved. A risk assessment was conducted during the planning phase of the audit to establish the scope and criteria (see appendix A), which were accepted by management. The audit findings were based on a comparison of the conditions in place at the time of the audit with the audit criteria.

The methodology used for this audit included various procedures to address the engagement's objective. This included policy and documentation review, interviews, sample file review, data analysis, and process walkthroughs.

3.4 Statement of conformance

The audit conforms to the Institute of Internal Auditors' International Professional Practices Framework, as supported by the results of CFIA's internal audit quality assurance and improvement program. Sufficient and appropriate evidence was gathered in accordance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing to provide a reasonable level of assurance over the findings and conclusions in this report. The findings and conclusions expressed in this report are based on conditions as they existed at the time of the audit and apply only to the areas included in the audit scope.

4. Findings and recommendations

4.1 ERA model governance

• 4.1.1 Roles, responsibilities and accountabilities
• 4.1.2 Processes to support scientific validity of ERA model
• 4.1.3 Internal governance
• 4.1.4 Compliance with the Directive on Automated Decision Making
• 4.1.5 Recommendations

4.1.1 Roles, responsibilities and accountabilities

Roles and responsibilities are well documented and understood by staff in Science Branch and Digital Services Branch, the main branches associated with the management of the ERA model:

• Science Branch has the overall authority and accountability for ERA model. It is responsible for the scientific design component of the algorithm, selects the Scientific Advisory Committee (SAC) members, and provides leadership level approval for the ERA model development, continuous improvement, maintenance and automation. They are the business owners of the model and are responsible for the algorithm from a food safety perspective and commodity updates within the model.
• Digital Services Branch plays an ongoing IT support function for the model, including the curation and integration of data, the reporting capabilities of the model as well as business continuity and transfer of knowledge.

Operations Branch has responsibilities related to the ERA model; specifically entering data for compliance history and reviewing the Additional Establishment Information, which is a key input into the ERA model.

The SAC is a committee created by the CFIA, comprised of Canadian food safety experts. In 2024 there were eight members external to the CFIA. They provide expert advice, literature review, and validation of risk factors for the model. Members are volunteers and can be from academia, CFIA, or other Government of Canada departments. Members are expected to provide advice once or twice a year and may be involved in other technical discussions as required.

4.1.2 Processes to support scientific validity of ERA model

To maintain the scientific validity of the ERA model, the Agency has several strategies in place.

Peer-reviewed scientific articles

The CFIA has published scientific articles to allow international food safety experts to evaluate the methodology and design of the ERA model. Four articles were published in peer-reviewed scientific journals between October 2018 and September 2019. These articles were authored by Agency staff and SAC members, and relate to the following steps of the model's development:

1. identification of factors associated with food safety risk
2. selection of risk factors for the ERA model
3. risk factors' criteria weighting for the ERA model and
4. risk attribution at the sub-product level

A fifth article, relating to the design of the model, is currently being prepared for submission to a peer-reviewed journal. Since 2013, the ERA model team has delivered 15 technical presentations relating to the ERA model at various international conferences.

Expert elicitations

Since 2013, CFIA has conducted six expert elicitations. These activities surveyed numerous Canadian food safety experts with a range of backgrounds (for example government, academia, and industry) to provide input on certain components of the ERA model. Expert elicitations were conducted to support:

• risk factor selection: evaluation of risk factors according to their impact on food safety to determine which risk factors to include in the model
• risk factor's criteria weighting: quantification of the relative importance of selected criteria used to measure the risk factors in the model
• risk attribution at the commodity and sub-product levels: attribution of foodborne illnesses in the Canadian population to different commodities/sub-products

Literature review

Scientific publications related to food safety and risk assessment are reviewed regularly. Upon review, if new information that could impact the algorithm is identified (for example a new pathogen affecting a specific commodity, updates of DALYs value for a specific pathogen), the ERA model team conducts additional in-depth analysis and brings the information to the attention of the SAC.

Scientific Advisory Committee consultations

SAC meetings have been ongoing since 2014, with one planned for 2025. At these meetings, various topics are presented for information and discussion, including new information from literature reviews, addition of new mitigation factors, proposed changes to the algorithm, and weighting of factors.

Algorithm updates

CFIA updates the algorithm's scientific design on an as required basis as new information becomes available; there have been 13 updates to the model since 2019. In general, the process is:

• Potential new or updated information to be considered in the model is identified
• Scientific literature is reviewed to determine if the inclusion of the new information is relevant and if so, to determine how to incorporate this information into the model
• The SAC is consulted as necessary on the proposed change for endorsement
• Proposed changes are presented to the Agency's oversight governance bodies
• The proposed change is implemented with the required parties

While the process for updating the algorithm is well understood, there is no documented Standard Operating Procedure (SOP) for the process. In addition, changes and/or updates are not centrally recorded to support continuity of corporate knowledge.

4.1.3 Internal governance

The ERA model has been presented to Agency governance committees on an as-needed basis, predominantly to the Food Business Line Committee (FBLC) and Food Business Line Management Board (FBLMB). These presentations were often for information only with little challenge function required. The FBLC and FBLMB have not met formally since Fall 2024 due to the ongoing Agency governance review initiatives. Without regular governance reporting to support robust discussions and identification of issues, there is a risk that senior management may not receive the information required to address risks and support decision making. When the governance review is completed, the Agency should determine the appropriate internal governance committee(s) to provide oversight of the model, and should consider membership, frequency, and reporting into senior management.

4.1.4 Compliance with the Directive on Automated Decision Making

The Government policy framework for the design and management of information technology solutions includes the Treasury Board Secretariat (TBS) Directive on Automated Decision Making(the Directive). The Directive is applicable to "any system, tool, or statistical model used to make an administrative decision or a related assessment about a client" and aims to ensure that solutions adhere to principles of transparency, accountability, legality, and procedural fairness.

The Directive outlines a list of requirements based on the impact level of the algorithm. The ERA model is in partial compliance with these requirements. Of note, the model has undergone an Algorithm Impact Assessment, a Privacy Impact Assessment, peer review through scientific journals, and provides clients a recourse option to express concerns about their risk results. However, during development of the algorithm, a risk assessment or gender-based analysis, or a formal test for bias were not conducted as required by the Directive. Additionally, the Agency cannot ensure that the data inputs into the model are accurate due to inconsistent quality controls. Data quality will be discussed further in section 4.2.

4.1.5 Recommendations

To support continuous improvement to the overall governance of the ERA model:

• The Vice-President of Science Branch (VP SB) should document the Standard Operating Procedure (SOP) for updating the algorithm and establish a centralized repository to document the evidence, status, and approvals for each algorithm update.
• The Vice-President of Science Branch (VP SB) should review the Directive on Automated Decision Making and conduct the necessary activities to bring the Establishment-based Risk Assessment (ERA) model into full compliance, including conducting a Gender-Based Analysis Plus (GBA+) analysis and validating that the data collected for and used by the model is relevant, accurate, and up-to-date, and in accordance with the Policy on Service and Digital and the Privacy Act.

4.2 Data quality/management

• 4.2.1 Processes to support data sourcing into the ERA model
• 4.2.2 Data input - Additional Establishment Information
• 4.2.3 Data input - inspection results
• 4.2.4 Data quality assurance processes
• 4.2.5 Recommendation

4.2.1 Processes to support data sourcing into the ERA model

Data sourcing processes

Processes for the ERA model data sourcing are documented and well understood by those using the model. These documented processes ensure that the same standards and methodologies are followed to support consistent data extraction and business continuity. This includes identification of source databases and the applicable parameters/criteria to be applied for each data input.

There is also a documented process for responding to the most common algorithm disruptions, such as network time-outs. Additionally, when other issues have occurred in the data extraction process, DSB and SB have demonstrated an ability to investigate and collaborate to fix issues in a timely manner.

Data stewardship

Source databases for the ERA model (DSDP, IMS, and the CFIA website) each have a data steward. As per CFIA's Standard on Data Stewardship, data stewards are responsible for activities surrounding monitoring and managing data throughout its lifecycle, including ensuring data quality, promoting availability, and escalating and solving data issues. Data stewards were consulted during the development of the model but have not always been consulted when subsequent changes have been made to the data used by the model. This may impact the Agency's awareness of potential issues relating to data quality and limitations.

4.2.2 Data input - Additional Establishment Information

Data for the inherent risk factors and mitigation factors is collected through the Additional Establishment Information (AEI) questionnaire (See first two boxes in Figure 2). Licence applicants are asked to complete the AEI questionnaire when applying for a Safe Food for Canadians (SFC) licence, and when the licence is amended or renewed. Also, licence holders are asked to update the AEI questionnaire at any time there is a change in their production processes that may impact its risk. The questionnaires are submitted via the Agency's client portal (My CFIA). Throughout the audit scope period, it was not mandatory for establishments to complete their AEI.

If AEI is not provided, an average risk category is assigned based on the commodity.

AEI completion rates currently range from 65% to 94%, depending on the commodity (See Figure 3). Generally, commodities with lower AEI completion rates were more recently integrated into the ERA model or more recently came under CFIA regulation.

Figure 3: AEI Completion Rates by Commodity

This figure illustrates the AEI completion rates by commodity as of December 2024 as a bar graph.

The AEI completion rates are:

• Dairy: 88%
• Maple: 89%
• Honey: 89%
• Egg and Egg Products: 85%
• Fish and Seafood: 88%
• Processed Fruit or Vegetable Products: 78%
• Meat and Poultry: 94%
• Fresh Fruit and Vegetables: 70%
• Manufactured Foods: 65%

As per Agency operational guidance, inspectors are instructed to review the AEI profile of the licence holder's establishment prior to the first planned Preventive Control Inspection-System Verification (PCI-SV) to confirm the accuracy of the identified inherent risk factors and mitigation factors. Guidance outlines what needs to be reviewed for each risk factor in the model, and inspectors are asked to identify obvious/major discrepancies. Inspectors do not have the authority to change an establishment's AEI if they find discrepancies. Instead, inspectors are expected to advise regulated parties to update their AEI. This expectation is not mandatory, and inspector review is not tracked, as there is no specific field in DSDP to document evidence of AEI review, other than in the inspection notes, which is an open text field.

Audit testing of 58 sampled inspection results from establishments with completed AEI questionnaires identified no documented instances of AEI review by inspectors.

Notably, audit testing of establishments without an AEI at the beginning of the year found that establishments that had been inspected were more than twice as likely to complete their AEI (21.4%) compared to establishments that were not inspected (10.0%).

During the course of the audit, to improve data completeness, the decision was taken to make completion of AEI questionnaire, including all the inherent risk and mitigation factors, mandatory prior to receiving, renewing or amending an SFC licence. To improve data quality as well as data completeness, the Agency has initiated a review to improve the accessibility of the language in the AEI questionnaire; for example, plain language questions would enable establishments to answer the questions accurately and consistently. The Agency also plans to establish a process to validate the AEI data. This validation activity should take into consideration the level of validation required, the level of impact of risk factors, and be considered as part of an overall data quality assurance process.

To demonstrate the importance of accurate AEI data, the audit team created a fictional establishment and had the ERA model team run it through the model test environment with changes made to the production volume.

4.2.3 Data input - inspection results

Data for compliance factors is collected from inspection results. This includes the food establishment's historical and current data related to inspections, control measures and enforcement actions taken, recalls, and confirmed food safety complaints (See third box in Figure 2 titled "compliance factors").

This information is currently collected from three distinct data sources:

• DSDP – inspection results and some control measures and enforcement actions (letter of non-compliance, meet with regulated party, seize, detain, removal from Canada, refusal to issue export certificate)
• IMS – confirmed food safety complaints and recalls
• CFIA external website – some enforcement actions (licence suspension, licence cancellation, refusal to issue, renew, or amend a licence, and prosecutions)

The audit identified issues with the data quality of some of the inputs into the model. In DSDP there are data quality issues due to non-mandatory fields and unclear field definitions, resulting in inconsistencies. For example, within DSDP it is not mandatory for an inspector to check that a non-compliance was observed, which impacts the ability of the ERA model to extract that information, and may impact the ERA model results, as demonstrated by the following example.

The DSDP data stewards are aware of these issues and are working towards solutions to support more accurate data entry, such as making the non-compliance check box a mandatory field.

The IMS has been in use at the Agency for more than 20 years and was not designed to align with the modern systems, such as DSDP and the ERA model. As such there are data and system limitations that prevent the linkage of an establishment's complaints and recall data to other establishment data from DSDP, and not all complaints and recall data can be extracted by the model. This gap has a greater impact on compliance information associated with Manufactured Foods and Fresh Fruits and Vegetables commodities.

• Audit testing of complaints data from April 2024 to December 2024 revealed that 76.0% of the complaint files were not incorporated into the ERA model due to a lack of common fields.
• Audit testing of recall data from April 2023 to December 2024 revealed that 77.5% of the recall files were not incorporated into the ERA model due to a lack of common fields.

IMS also has data quality issues due to free-text fields and unclear field definitions, resulting in inconsistent use and data entry errors. For example, what appeared to be IMS user IDs were entered into the Facility Registration Number field.

• Audit testing of complaints data from April 2024 to December 2024 found that 15.6% of the complaint files were not incorporated into the ERA model due to data entry errors.
• Audit testing of recall data from April 2023 to December 2024 revealed that 13.2% of the recall files were not incorporated into the ERA model due to data entry errors.

As a result, not all relevant complaint and recall information was able to be incorporated into the ERA model, which may impact the establishment risk result and/or risk category level.

During the course of the audit, a solution was implemented to improve the incorporation of recall data into the model. In March-April 2025, an IMS task force manually added the Establishment ID into each recall file from the last five years. Additionally, the Agency is conducting an IMS User Experience initiative to support the development of a longer-term solution for managing complaints and recalls data.

4.2.4 Data quality assurance processes

Data quality assurance activities and processes vary across the data inputs.

As previously reported in section 4.2.2, while there is guidance in place to support inspector review of AEI information, there is no formal process to document this review.

Data quality assurance of inspection results in DSDP rests with the inspection supervisors. For example, supervisors can require inspection results to undergo supervisory review prior to being completed, though this requirement is up to the supervisor's discretion. While there was no evidence of a formal process to support this supervisory review activity, two noteworthy initiatives were observed.

• 1. A Data Quality Monitoring Solution (DQMS) project was funded to define performance metrics and improve data quality for the Laboratory Information Management System (LIMS).
• 2. An Inspection Quality Assurance Program (IQAP) pilot for Meat was implemented, to assess performance of inspection activities and decision-making and to identify systemic issues to facilitate continuous improvement.

For recall data, the IMS data steward is responsible for performing quality control checks before changing the status to "closed". In doing so, they are attesting to the quality, thoroughness, and completeness of the food investigation and data entry. There is guidance to support the quality control of the review of data prior to closure. For complaints data, IMS Closers, generally inspection managers, supervisors, or their delegates are responsible for reviewing complaint files before changing the status to "closed". Guidance exists to support file closures, but it does not contain instructions to support data validation.

Complaint closure activities are decentralized and may be treated as a secondary duty. Of complaints files opened between April 2023 and December 2024, as of December 2024, 57% were closed. The remaining 43% remained open or were awaiting closure and had not been subject to review by an IMS Closer. As the ERA model pulls all confirmed complaints regardless of status, there is a risk that non-reviewed data is being entered into the model.

While some activities are being conducted, the lack of a formal data quality assurance process means that the Agency cannot provide assurance on the quality of the data collected for and used by the ERA model.

4.2.5 Recommendation

• To address the identified gaps and enable continuous improvements in overall data quality, the Vice-President of Science Branch (VP SB) in collaboration with the Vice-President of Digital Services Branch (VP DSB) and the Vice-President of Operations Branch (VP OPS) should develop a formal data quality assurance program for the Establishment-based Risk Assessment (ERA) model. This program should leverage existing tools and initiatives such as the Data Quality Monitoring Solution (DQMS) and the Inspection Quality Assurance Program (IQAP), as well as on-going User-Experience initiatives. As well, this program should identify the roles and responsibilities of the quality assurance reviewers for each data source; provide detailed guidance to support consistent application; define the frequency of reviews; and include a monitoring and reporting function to governance.

5. Audit conclusion

The CFIA-developed Establishment-based Risk Assessment (ERA) model is a key risk modeling tool used by the Agency to evaluate domestic food establishments based on the risk they represent to Canadian consumers. The ERA model uses data related to the inherent risks of an establishment based on the type and volume of the products manufactured, along with the preventive controls in place, and compliance with the regulations. Performance of the ERA model depends on its governance and the quality of industry and Agency data.

The audit concluded that there are effective governance elements in place to support the Agency's ERA model. Clearly defined roles, responsibilities and processes are in place to support the governance and scientific validity of the ERA model. Data sourcing processes for the ERA model are in place and documented to support data extraction activities.

The audit found opportunities to strengthen oversight, obtain compliance with the Treasury Board Secretariat Directive on Automated Decision Making, and improve the effectiveness of data quality assurance activities and processes.

The recommended implementation of a data quality assurance program for the ERA model will enable continuous improvements to data quality to support risk-based decision-making, and overall compliance.

6. Appendix A: Audit criteria

The audit criteria were developed based on a risk assessment and in consideration of the Office of the Comptroller General's Audit Criteria related to the Management Accountability Framework: A Tool for Internal Auditors (the Tool), and the COSO Internal Control, Integrated Framework 2013 and the COSO Enterprise Risk Management Integrating with Strategy and Performance Framework 2017. The audit criteria are organized under the lines of enquiry below.

Line of Enquiry 1: Governance of the ERA Model

• 1. The CFIA has clearly defined roles, responsibilities, authorities, and accountabilities regarding the management of the ERA model.
  • 1.1 Roles, responsibilities, authorities and accountabilities regarding the ERA model are defined and communicated.
• 2. The ERA model has an effective governance structure that supports performance, risk, compliance, and change management oversight and supports horizontal Agency initiatives.
  • 2.1 Governance processes address ERA model risks, performance, and change management.
  • 2.2 The ERA model team provides material and guidance to support the Agency's awareness and understanding of the model.
  • 2.3 The ERA model is managed in compliance with the Directive on Automated Decision Making and other applicable policies and standards.
  • 2.4 There are processes in place to support the scientific validity of the ERA model.

Line of Enquiry 2: Data Quality Processes

• 3. The CFIA has designed and implemented effective data quality assurance processes that support completeness, availability, relevance, and accuracy of ERA model data inputs.
  • 3.1 Processes and procedures are applied to incorporate accurate and verified Additional Establishment Information (AEI) into the model.
  • 3.2 Processes and procedures are applied to accurately incorporate unassessed establishments with incomplete Additional Establishment Information (AEI) into the model.
  • 3.3 Processes and procedures exist to support timely, accurate, and complete incorporation of compliance data into the model.
  • 3.4 Processes and procedures exist to support successful sourcing of data from internal source databases.
  • 3.5 All processes and procedures for risk factor data inputs have supporting documentation to maintain consistency and business continuity.

7. Appendix B: List of acronyms

AEI

Additional Establishment Information

CFIA

Canadian Food Inspection Agency

DALY

Disability-Adjusted Life Year

DSB

Digital Services Branch

DQMS

Data Quality Monitoring Solution

DSDP

Digital Service Delivery Platform

ERA

Establishment-based Risk Assessment

FBLC

Food Business Line Committee

FBLMB

Food Business Line Management Board

GBA+

Gender-Based Analysis Plus

IMS

Issues Management System

IQAP

Inspection Quality Assurance Program

IRAM

Importer Risk Assessment Model

IT

Information Technology

OPS

Operations Branch

PCI-SV

Preventive Control Inspection – System Verification

QA

Quality Assurance

SAC

Scientific Advisory Committee

SB

Science Branch

SFC

Safe Food for Canadians

SFCA

Safe Food for Canadians Act

SFCR

Safe Food for Canadians Regulations

TBS

Treasury Board Secretariat

VP

Vice-President

8. Appendix C: Glossary

Additional Establishment Information (AEI)

Information that an establishment provides related to its products and processes.

Algorithm

A specified mathematical process for computation.

Commodity group

Products and/or things that are regulated have been organized under each of the Canadian Food Inspection Agency's (CFIA's) three business lines (for example: fish and seafood under the food business line, live animal under the animal health business line and horticulture under the plant health business line).

Complaint

Producer complaint or commercial complaint (regarding a competitor's products) and directed inspections or follow-ups for imported products suspected of non-compliance as identified when reviewing import documents from the Import Service Centers (ISCs) or the Canada Border Services Agency (CBSA).

Disability-Adjusted Life Years (DALY)

A time-based unit of measurement that measures the overall burden of disease. One DALY represents the loss of the equivalent of one year of full health. DALYs for a disease or health condition are the sum of the years of life lost to due to premature mortality and the years lived with a disability.

Digital Service Delivery Platform (DSDP)

A tool that gives Canadian Food Inspection Agency staff the ability to issue export certificates and permissions (such as licences, permits, and registrations), capture inspection data, document compliance decisions, complete technical reviews, generate inspection reports, and manage risks.

Enforcement action

An action taken by CFIA in response to non-compliance.

Issues Management System (IMS)

The IMS is a database information tool designed to document, track, and communicate information relating to the food investigation of incidents within the Canadian Food Inspection Agency (CFIA)'s priorities and mandate. The IMS allows users to determine the status of a food investigation from the trigger to the final conclusion.

Inspection task

A task that an inspector conducts to verify a regulated party's compliance to regulatory requirements and/or permission conditions.

Non-compliance

A contravention of the applicable Acts and/or Regulations.

Pathogen

an organism that causes disease, such as a virus, bacteria, or fungus. For example, Salmonella.

Preventative Control Inspection - System Verification (PCI-SV)

An evaluation of a regulated party's preventive controls to achieve compliance with regulatory requirements and/or permission conditions. The inspection could include a regulated party's systems-based approach that focuses on prevention as a way to achieve compliance.

Permissions

Official consent granting legal authorization to a regulated party to conduct specified activities (for example: permits, certificates, licences and registrations).

Recall

Removal of a food from further sale or use, or the correction of its label, at any point in the supply chain as a risk mitigation action.

Risk category

A grouping of risk results within a defined range. The risk categories for the ERA model range from 1 to 7, with 1 representing the highest risk and 7 representing the lowest risk.

9. Appendix D: ERA model mathematical calculation

$$\mathrm{Re}=\sum _p\left[\sum _{\text{prod}}(\frac{\text{DALYs}}{{\text{cases}}_p}\times \frac{{\text{cases}}_p}{\text{year}}\times \frac{{\text{cases}}_{\text{comm}}}{{\text{cases}}_p}\times \frac{{\text{cases}}_{\text{prod}}}{{\text{cases}}_{\text{comm}}}\times \frac{V_{\text{prod}}}{{\text{Food availability}}_{\text{prod}}})\right]\times {\text{Adj(RF)}}_e$$

• $\mathrm{Re}$ – Risk assessment result for an establishment.

• $\sum p$ – The sum of all risk assessment results for all 18 pathogens.

• $\sum \text{prod}$ – The sum of all sub-product risk assessments for a pathogen.

• $\text{DALYs}$ - The DALY is based on the instance of illness per 1,000 cases of each pathogen. This component is calculated for each of the 18 pathogens considered in the ERA model, which represent over 99% of the total burden of microbial food-borne disease in Canada. Each pathogen has its own calculated DALY based on scientific literature.

• ${\text{Cases}}_p$ - Number of cases of illness for a specific pathogen.

• $\text{Year}$ - One year.

• ${\text{Cases}}_{\text{comm}}$ - Number of cases attributed to a specific commodity.

• ${\text{Cases}}_{\text{prod}}$ - Number of cases attributed to a specific sub-product.

• ${\text{Volume}}_{\text{prod}}$ - Volume of production of a specific sub-product at the establishment.

• ${\text{Food Availability}}_{\text{prod}}$ - Total volume of the specific sub-product available for consumption in Canada.

• ${\text{Adj(RF)}}_e$ - Based on the mitigation and compliance factors present in a particular establishment.

10. Appendix E: Inherent, mitigation and compliance risk factors

Inherent risk factors

Inherent risk factors are risks associated with a specific food commodity, operation, or manufacturing process. Inherent risk factors are collected from regulated parties through the My CFIA AEI questionnaire and are stored in DSDP.

Some of the inherent risk data is used in the initial DALYs calculation for an establishment, while the remaining inherent risk data is used in the risk adjusting variable of the ERA model equation (See Appendix D).

The following are inherent risk factor data inputs collected from the AEI questionnaire:

• 1. Type of activity – The regulated party identifies the type of activities performed by the establishment, such as domestic and international activities.
  1. Example: The establishment's domestic activities include preparing and storing.
• 2. Commodity – The regulated party identifies the commodity(ies) produced by the establishment. This information must align with their SFC licence(s).
  1. Example: The establishment produces products in the Egg and Egg products commodity.
• 3. Type of products – The regulated party identifies the sub-products produced by the establishment and the volume produced relative to total production volume for each commodity identified on their SFC licence.
  1. Example: The establishment produces the following sub-products within the Egg and Egg products commodity: 50% shell eggs and 50% dried egg products.
• 4. Distribution volume – The regulated party identifies the volume produced by the establishment for each commodity identified on their SFC licence.
  1. Example: The establishment produces 100,000 kg/year.
• 5. Processing steps – The regulated party identifies the processes and treatments applied to distributed products.
  1. Example: The establishment uses the following processing steps: slicing, dicing, shredding or grinding of ready-to-eat products.
• 6. Distribution to vulnerable population – The regulated party identifies the percentage of total volume produced that it is known to be distributed to vulnerable populations (for example daycare centres, hospitals or institutional care centres)
  1. Example: The establishment is aware that 50% of their products is distributed to vulnerable sub-populations.

Mitigation factors

Mitigation factors are the measures or strategies that a food establishment has implemented to control the inherent risks and reduce the overall risk of a food safety issue. Mitigation factors are collected from regulated parties through the My CFIA AEI questionnaire and are stored in DSDP.

The mitigation factor data is used in the risk adjusted variable of the ERA model equation (See Appendix D).

The following are mitigation factor data inputs collected from the AEI questionnaire:

• 7. Additional processes – Additional processes the regulated party has in place.
  1. Example: The establishment uses high pressure processing
• 8. Food safety certifications – The regulated party identifies the food safety certification scheme(s) that the establishment holds. The regulated party must select all that apply from a list of options presented in the questionnaire.
  1. Example: The establishment identifies that they hold schemes certified to Hazard Analysis Critical Control Points (HACCP) and International private certification schemes.
• 9. Third party audits related to food safety – The regulated party identifies if the establishment's Preventative Control Plan has been audited by a third party besides those conducted as part of the food safety certification. This section only considers third party audits for food safety and does not consider organic certification audits.
  1. Example: The establishment answers no; they do not have their Preventative Control Plan audited by a third party.
• 10. Food safety control of incoming supplies/materials – The regulated party identifies the food safety control activities performed by the establishment. The regulated party must select all that apply from a list of options presented in the questionnaire.
  1. Example: The establishment checks off that they have a letter of guarantee, review of certificate analysis, and use flock or farm treatment records as part of their food safety control of incoming materials.
• 11. Quality Assurance (QA) personnel – The regulated party identifies if the establishment has at least one dedicated quality assurance resource who is available full time on-site during production hours (for example food microbiologist, food quality management expert, HACCP coordinator, or a Quality Assurance technician).
  1. Example: The establishment answers yes; they do have a dedicated quality assurance resource.
• 12. Microbiological sampling plan – The regulated party identified any microbiological sampling plans in place at the establishment. The regulated party must select all that apply from a list of options presented in the questionnaire.
  1. Example: The establishment identifies that they have a microbiological sampling plan in place and perform trend analysis.

Compliance factors

• 13. Inspection results and impact assessments – data relating to an establishment's inspection history and level of compliance with regulatory policies enforced by the CFIA. Each identified non-compliance is assigned an impact level (direct, potential, or no impact) based on the impact posed to food safety. This data is sourced automatically from DSDP, where inspectors enter inspection result data. The ERA model considers the last four results of each Preventive Control Plan (PCP) Sub-Element (satisfactory or unsatisfactory results with their associated impact level).
• 14. History of enforcement actions and control measures – data relating to the history of enforcement actions and control measures taken on an establishment. This data is sourced manually from DSDP and the CFIA website. Information on prosecutions, licence suspensions, and licence cancellations are manually extracted from the CFIA website. The remainder of enforcement actions/control measures data used by the ERA model (letter of non-compliance, meeting with regulated party, seizure and detentions of products, removal from Canada, and refusal to issue export certificate) are manually extracted from DSDP. The ERA model considers enforcement data associated with an establishment from the last two years.
• 15. Recalls (Class I, II, III) – data relating to the number of recalls associated with an establishment, separated into three classes of recalls. This data is sourced automatically from IMS, where inspectors input issues (recalls and complaints) data. The ERA model considers recall data from the last five years.
• 16. Food safety confirmed complaints – data relating to the number of complaints associated with an establishment. This data is sourced automatically from IMS, where inspectors input issues (recalls and complaints) data. The ERA model only considers complaints that have been confirmed by an inspector. The ERA model considers complaints data from the last year.

11. Appendix F: Internal audit recommendation priority criteria

The CFIA uses internal audit recommendation priority criteria as follows:

12. Appendix G: Management response and action plan

Audit of Establishment-based Risk Assessment (ERA) model

Overall management response:

Science Branch management agrees with the audit report findings and recommendations and is proposing the action plan as described below.

Management response:

Science Branch management agrees with the recommendation and will implement the actions and deliverables outlined below.

Management response:

Science Branch management agrees with the recommendation and will implement the actions and deliverables outlined below.

Management response:

Management from Science Branch, Operations Branch and Digital Services Branch agrees with the recommendation and will implement the actions and deliverables outlined below.