Canadian approaches to research impact and its assessment

Abstract

1. Introduction

With increased global investment in research has come increased expectations from decision-makers to make a return on that investment. Hence, there is a greater demand to demonstrate and communicate the economic and societal impacts that benefit communities, both locally and globally. As a result, we are seeing increased calls from researchers, practitioners and other stakeholders to advance the science and practice of impact assessment.

Scholarly impact remains an important element of research assessment, since their measures dominate international rankings, institutional incentives and reward systems. But broader societal impact studies, including those of impacts on culture, health, economy and environment, have been gaining international attention—at least since 2014 (Williams and Grant 2018). To complement the research articles in this special issue, we describe the unique ways that Canada is contributing to global discussion on the societal impacts of research.

2. The Canadian context

Impact assessment is a practical science, as it is bourne out of the demand for its practical application. Canada lacks a systematic nationwide research impact assessment system, like the Research Excellence Framework in the UK, or the former Excellence in Research for Australia framework. Canadian researchers have tended to focus on ‘knowledge mobilization’, that is, on an intentional effort to support the societal impacts of research through co-production with user communities (Adams et al. 2022). We propose that the Canadian approach can be characterized as taking a holistic view to promoting impact, setting the conditions to foster an impact culture collaboratively across ecosystem players. It is ‘grass roots’ in nature and aspires to integrate adaptive learning systems to improve practices for realizing impact.

Drawing from Parks, Rodriguez-Rincon, Parkinson and Manville’s (2019) six ‘A’s of reasons to assess research are:

• Analysis: to understand who, how and whether research is effective, and how it can be better supported.

• Advocacy: to demonstrate the benefits of supporting research, and enhance the understanding of research and its processes among policy makers and the public.

• Allocation: to determine how to distribute funding across the research system.

• Accountability: to evidence that money and other resources have been used effificently and effectively, and to hold stakeholders to account.

• Acclaim: to compare and recognize the value of higher education institutions and the research conducted within them.

• Adaptations: to steer change in organizational structures, behaviours and cultures, and research activities and priorities.

We argue that Canada has made a distinctive and significant contribution to embedding positive practices around impact and its assessment by focusing particularly on two of these ‘A’s, namely Analysis and Adaptation.

In countries with national assessment systems, Analysis and Adaptation must compete with Allocation and Accountability. But by focusing on Analysis and Adaptation, a range of efforts and initiatives across Canada have contributed to learning how to grow impact for communities.

We can point to an intellectual line of enquiry in Canada that is focused around questions about how to better generate pathways to impact, in contrast to studies about what impacts have occurred. This study of impact process includes embedding methods of implementation science, as well as of knowledge transfer, translation and mobilization to optimize the impacts of research. Led by strong knowledge translation science as well as strong knowledge mobilization practice (Chia-Kangata, Lachance and Ungar 2020), Canada is seen as a world leader in a number of research fields where evidence is generated to inform policy and practice when it is co-produced with user communities (McLean et al. 2023). In the area of impact evaluation, the work of John Mayne on contribution analysis has led to the recognition that when assessing impact, the contribution that a program is making to outcomes must be considered as well as ex-ante theories of change. Canada has also more recently emerged as a leader in the health research assessment arena, with the publication of the Canadian Academy of Health Sciences report on a preferred impact framework and indicators (CAHS 2009; Frank and Nason 2009) that has widely applied a ‘fit for purpose approach’ to the Canadian contexts by different communities across Canada (AIHS and NAPHRO 2015) and international contexts (AAMRI 2021; Adams et al. 2022).

The following three examples illustrate these key characteristics and principles, driven by Analysis and Adaptation, where Analysis (i.e. learning) allows us ‘to understand why (why not) how and whether research is effective under what conditions, and how it can be better supported’ and, having learned about what works, Adaptation enables us to ‘course correct and steer change in organizational structures, behaviours and cultures, and research activities and priorities’. The Canadian approach to impact focuses on processes for making impact (which we call knowledge mobilization) and extending to impact assessment. This list is not intended to be exhaustive but simply illustrative examples of organizations in Canada that have committed resources to advance research and innovation impact and its assessment. They represent a provincial research and innovation organization (Alberta Innovates Impact Action Lab), a pan-Canadian knowledge mobilization network (Research Impact Canada) and a community network of public health researchers (Canadian Health Services and Policy Research Alliance, CHSPRA).

3. Alberta innovates impact action lab

Alberta Innovates created the Impact Action Lab¹ (IAL) in 2021 to support ecosystem organizations, researchers, and entrepreneurs to help activate and amplify impact across all sectors. The IAL takes a novel holistic systems approach and applies cutting-edge practices to advance the science and application of impact and its assessment. The focus is on co-creating and embedding a fit-for-purpose Performance Impact Management System (PIMS) across organizations and networks to understand how to generate and measure real world impact. The key is to design for impact from the outset (ex-ante) via impact planning and to improve knowledge mobilization activities along the pathways to impact [analysis from Parks et al. (2019)] with follow up assessments to allow enough time for outcomes to be realized. A formative approach that complements monitoring with realistic impact case studies is taken to understand what works, under what conditions, and for IAL partners across different jurisdictions to test and share leading practices of mutual benefit (Grant 2024). To this end, Alberta Innovates works with communities to foster an impact culture focused on building capacity and skills for greater impact [adaptation from Parks et al. (2019)].

4. Research impact Canada

Research Impact Canada² (RIC) is a network dedicated to building capacity for research impact literacy, or the ability to ‘identify appropriate impact goals and indicators, critically appraise and optimize impact pathways, and reflect on skills needed to tailor approaches across contexts’ (Bayley and Phipps 2019). The RIC network contains over 30 research organizations in a bilingual community of practice, sharing tools and services to build institutional capacity to support knowledge mobilization and broader societal (non-commercial) impacts of research. RIC supports building institutional capacity [adaptation from Parks et al. (2019)] that allows professional staff at member institutions to support impact for researchers. RIC members build capacity for knowledge mobilization to maximize research impact [analysis and learning from Parks et al. (2019)].

5. The Canadian Health Services and Policy Research Alliance (CHSPRA)

CHSPRA brings together organizations and individuals from across the health services community to collaborate on specific and common priority initiatives. One strategic pillar is impact assessment. The Impact Assessment volunteer working group is a ‘bottom-up’ approach from interested Health Systems Policy Research (HSPR) community members that has built out the CAHS impact framework to begin to articulate the value of HSPR to the health system beyond its traditional bibliometric approaches, to consider societal issues (e.g. equity, diversity and inclusion) and measures. The updated health impact framework developed is non-linear in nature and focuses on assessing better decision-making in health systems and policy. It highlights co-production and knowledge mobilization as promising practices for generating impact in Canadian HSPR. In addition to the assessment framework, CHSPRA also built a collaborative approach to testing, cataloguing, and providing framework user support [analysis from Parks et al. (2019)] for HSPR across different types of organizations in Canada [adaptation from Parks et al. (2019)] (CHSPRA 2018, 2022).

These three examples illustrate some of the key characteristics of a Canadian approach to research impact.

• Commitment to fostering an impact culture and awareness of societal impact and who benefits as a first principle in research impact.

• Highlighting the importance of designing and planning impact from the onset—starting with the end in mind.

• Support collaborative (i.e. co-production) bottom-up approaches to impact creation and assessment from within research communities.

• Advance research impact process practices in Canada by offering training and access to practical tools for implementation and mobilization.

We propose that advancing practices in ‘Analysis and Learning’ and ‘Adapting’, which are two of the six ‘A’s’ of research impact assessment from Parks et al. (2019) fits culturally with knowledge mobilization and research impact approaches in Canada. We refer to this as a Made in Canada approach. More than a descriptor of the Canadian context, the Made in Canada approach is a vision to maximize the societal impacts of research by addressing the challenge of bridging analysis (=learning about impact) to adaptation (=impact practice). These characteristics of research impact in Canada—planning ex ante, collaboration, learning, improving—are facilitated because Canadian institutions are not competing for impact rankings or for funding via a national impact assessment scheme. But we aren’t doing impact assessment in a vacuum. As exemplified by the articles in this special edition, all of which evolved out of an international workshop, Canada is but one country among many developing approaches to creating and assessing research impact. It shouldn’t be that surprising that Canada has its own special way of doing things.

Notes

References