Using contribution mapping to evaluate the research impact of Universities of Applied Sciences

Abstract

Research conducted by Universities of Applied Sciences (UASs) is frequently driven by professional practice where researchers are challenged with finding solutions to real-life problems. These real-life solutions are significantly enhanced by the participation of stakeholders. Through this inclusion and the resulting interactions, activities, and knowledge transfer, between the stakeholder and research(ers), impacts occur at a micro level. Micro impacts are the normal impacts that occur during the research process through interactions between researchers and stakeholders, that facilitate an unexpected and unplanned effect, be it positive or negative (Lykke et al. 2023, Mapping Research Activities and Societal Impact by Taxonomy of Indicators: Uniformity and Diversity across Academic Fields, Journal of Documentation, 79: 1049–70). Contribution analysis has been recognized as a viable method for evaluating micro impacts. One recognized contribution analysis framework is Kok and Schuit’s (2012, Contribution Mapping: A Method for Mapping the Contribution of Research to Enhance Its Impact, Health Research Policy and Systems, 10: 21) Contribution Mapping. It is also one of the frameworks acknowledged as conforming to several of the recommendations for evaluating UAS research impact. However, to do justice to the nature of Practice Oriented research, a new framework is needed. The purpose of this article is to test how Contribution Mapping works in real-life to answer the question: What can we learn from Contribution Mapping as an impact evaluation tool for a future UAS specific research impact evaluation framework? This article will examine the specificity of UAS research, the relevance of Contribution Mapping for evaluating UAS research, and the theoretical and practical implications of Contribution Mapping. Through inductive analysis conducted on information gleaned from interviews and focus groups, observations, challenges, and limitations are identified, and modifications suggested to take into consideration for a new framework.

1. Introduction

Evaluating the impact created by research done by Universities of Applied Sciences (UASs) has been a point of discussion for a considerable number of years. While initially founded to create impact through the education of future professionals, UASs have spent the last two decades professionalizing their research role. Today, research conducted by UASs is frequently driven by professional practice where researchers are challenged with finding solutions to real-life problems. These problems can be relatively small such as the optimalization of production processes within companies, or much larger, for example, by redeveloping a rural living environment. These real-life solutions are significantly enhanced by the participation of stakeholders. The nature of Practice Oriented research, therefore, is such that stakeholders play a profoundly important role in it.

Co-creation, co-production, societal and public engagement, and citizen science have become means of describing the participation of stakeholders in the different phases of the research cycle. Each of these examples provides a different level of stakeholder inclusion in the research process. Through this inclusion and the resulting interactions, activities, and knowledge transfer between the stakeholder and research(ers), impacts occur at a micro level (Budtz Pedersen and Hvidtfeldt 2024). These micro impacts are what UASs strive to make visible (Coombs and Meijer, in review).

One method of evaluating the micro impacts created by UAS research has been identified as Contribution Analysis (Coombs and Meijer 2021; Lykke et al. 2023). One recognized contribution analysis framework is Contribution Mapping created by Kok and Schuit (2012) and (Greenhalgh 2016). Work by both Coombs and Meijer (2021) and van Beest (2022) have stated that the specific context of Practice Oriented research requires a new framework for evaluating the impact of research conducted by UASs, one that focuses on the research process. Contribution Mapping is one of the frameworks acknowledged as conforming to several of the recommendations for evaluating UAS research impact (Coombs and Meijer 2021). It is one of the few established frameworks with a performative assumption (Greenhalgh 2016). An assumption is the lens through which research and its evaluation is viewed. A performative assumption is based on Actor-Network theory, bringing the actors, activities, outputs, and interactions into view. This impact framework is formative and is thus for learning purposes. It is also adaptable to real time in co-creation with stakeholders (see 2.2, Coombs and Meijer 2021). Its focus on learning in collaboration throughout the research process and its performative elements can potentially provide insight into methods for a new impact evaluation framework focusing on the context and process of UAS research. The purpose of this article then is to test how this framework works in real-life through the use of a case study, named GO! Noord Nederland. The GO! Noord Nederland project is an example of research typically conducted by UASs. It is namely Practice Oriented, aiming to change the living environment of the people involved. It is also a large collaboration between municipalities, health organizations, societal partners and research institutions. Like many UAS research projects there was also collaboration with students. However, this case study specifically looks at the work of the participating researchers and their cooperation with stakeholders and partners. While students made contributions, their work was not the core of this project nor of this article.

By using and analysing Contribution Mapping within this project we strive to answer the question: What can we learn from Contribution Mapping as an impact evaluation tool for a future UAS specific research impact evaluation framework? This article will examine the specificity of UAS research, the relevance of Contribution Mapping for evaluating UAS research, and the theoretical and practical implications of Contribution Mapping.

2. Theoretical framework

2.1 What are the specificities of UAS research?

The application of research to solve problems in practice is acknowledged as one of the key attributes of UAS research (de Weert and Leijnse 2010). The applied, problem-solving nature of research done at UASs is reflected in the two terms frequently used to describe Dutch UAS research, Praktijkgericht (Practice Oriented) and Toegepast (Applied). While Applied is perhaps the best known, there is academic discussion on similar but different definitions such as Practice Oriented research, that focuses more on the nature of UAS research (Borgdorff, van Staa and van der Vos 2007). Practice Oriented research is known for its practical application when attempting to tackle Grand Challenges. It emphasizes the importance of collaboration and co-production between researchers and stakeholders, with researchers working alongside practitioners to, among other tasks, identify problems, gather data, and develop interventions (de Weert and Leijnse 2010). Van Beest (2022) have referred to this type of research as practice-based (van Beest 2022), emphasize the importance of questioning and collaboration with practice. Brouns’ (2016) prefers the term, Praktijkgebonden (Practice Related). She suggests this term reflects the non-linear nature of UAS research as it follows practice. Additionally, she argues it emphasizes the continued role of professional practice throughout the research cycle, and the value of combining scientific knowledge of the researcher and these experiences. Regardless of the word used, for the scope of this paper the emphasis is on the most important element of this type of research, i.e. the stakeholder contributes to research that links the three components of the Knowledge Triangle of Education, Professional Practice and Research (Miedema, van der Sijde and Schuiling 2013).

Countries including Austria, Belgium, Croatia, Czech Republic, Denmark, Estonia, Finland, Germany, Greece, Ireland, Lithuania, the Netherlands, Norway, Portugal, Spain, Sweden, and Switzerland, all have a binary educational system that includes traditional universities as well as Universities of Applied Sciences (Universities of Applied Sciences for Europe 2017). While they are not homogeneous in how they express this binary system (Lepori 2022), UASs around the world participate in a similar type of research and produce similar output (Universities of Applied Sciences for Europe 2017). This research has been referred to as, among other things, Applied, Triple Helix, Third Mission, Entrepreneurial, Mode 2 or Edison’s Quadrant research (Carayannis and Campbell 2009; Leydesdorff and Zawdie 2010; Bornmann 2013; Carayannis and Pirzadeh 2014). Mode 2, a theoretical framework initiated by Gibbons et al. (1994) in their book, The New Production of Knowledge, suggests that in addition to traditional, fundamental research (Mode 1 research), a new form of knowledge production is being created, i.e. Mode 2. Mode 2 research is described as transdisciplinary, heterogeneous, heterarchical and transient (Gibbons et al. 1994). Among other characteristics, this knowledge production is generated within the context of applications and utilizes a broad range of theoretical perspectives to solve problems. The results of this knowledge production are shared through formal channels and informal interactions with participants (Gibbons et al. 1994). The Netherlands Association of Universities of Applied Sciences (NAUAS) has recognized that the research their institutions conduct can be viewed as Mode 2 (Netherlands Association of Universities of Applied Sciences 2008).

They have also stated that the applicable, problem-solving nature of UAS research is reflected in Stokes’ Pasteur’s Quadrants theory (1997) (van Gageldonk 2017). Stokes’ theory of technological transfer suggests that knowledge production is composed of four quadrants. He describes three of these quadrants as categories of research to illustrate the three different ways in which knowledge is produced: pure basic (Bohr); use-inspired basic (Pasteur); and, pure applied research (Edison). The fourth quadrant has been left undefined. It has been suggested that research undertaken by UASs falls into the Edison quadrant (de Weert and Leijnse 2010). This differs from the other quadrants as it is characterized by pure applied research that seeks to solve a specific problem rather than to understand any broader scientific phenomena resulting from what is being discovered (Stokes 1997). Whether this accurately encompasses the work done by UAS research has been questioned (Kyvik and Lepori 2010).

It has also been suggested that the research done by UASs is in fact comparable to Carayannis and Campbell’s (2009) concept of Mode 3, and Quadruple Helix Innovation Systems (Meister Broekema 2023). Mode 3 research builds upon Mode 1’s traditional academic knowledge production and emerging, collaborative, transdisciplinary knowledge production of Mode 2. Using a systems analysis approach, Mode 3 emphasizes the integration of different knowledge sources and the contextualization of knowledge in addressing real-world problems because knowledge production practice is a multi-layered, multimodal and multimodel system. Carayannis and Campbell’s work extends Etzkowitz and Leydesdorff’s (1998, 2000) concept of the ‘Triple Helix’ to that of a ‘Quadruple Helix’ model. They suggest that the Quadruple Helix model involves not only the collaboration between government, industry, and academia in the innovation process but also the involvement of civil society or the public. This model emphasizes the importance of involving diverse stakeholders to foster innovation. It is this interplay between the different types of knowledge in an ecosystem of diverse partners and stakeholders that reflects what occurs at UASs.

However, Gulbrandsen and Kyvik (2010) have stated that it can be difficult to separate the types of research done in practice. In reality, the research done at UASs encompasses different types of research at different moments because research projects require several types of activities, contributing to theory or practice, for completion (Kyvik and Lepori 2010). This is not to suggest that the research done by universities cannot be applied or vice versa but it provides a starting point from which UASs can establish how to evaluate the impact of the research they conduct (de Weert and Leijnse 2010). Contribution Mapping has been identified as one of the appropriate frameworks for evaluating the impact of the types of research conducted at UASs as it focuses on the various types of activities that can occur (Coombs and Meijer 2021).

2.2 Why contribution mapping is relevant for evaluating UAS research

The importance of the impact of research on society and how this can be evaluated continues to be an international point of discussion (Budtz Pedersen, Grønvad and Hvidtfeldt 2020; Smit and Hessels 2021). Several countries with a binary tertiary system such as Denmark, Finland, Belgium, and the Netherlands are currently examining how their research impacts the world at large. To provide a scientific basis for the necessary evaluation, previous research has suggested that there are several elements that require consideration to evaluate the impact of the practice related research done by UASs. As indicated in Raftery et al. (2016), often the evaluation approach is based on ‘philosophical assumptions’ made regarding the links between research and impact. The philosophical assumption on which the evaluation is based must, therefore, first be established. These assumptions assist in forming and enhancing the methods and tools used for evaluating (Raftery et al. 2016). Applicable philosophical assumptions for practice related research are either a realist assumption based on context-mechanism-output-impact or a performative one grounded in actor network theory (Greenhalgh 2016). Additional recommendations suggest that this evaluation should focus on formative, ‘real-time’ evaluation allowing for learning throughout the process of the research as well as in the future (Guthrie et al. 2013). This should be done in co-production with the stakeholders (Raftery 2016; van Drooge and Spaapen 2022). Finally, the nature of practice related research should not be constrained through the use of a logic model where a linear representation of the process does not do justice to the messy feedback loops of UAS research. Linearity suggests that impact is created through an immutable context-independent process (Kok 2021). Instead, these recommendations stress a need to understand the process and specific context of the research done at UASs to evaluate the impact of it and the importance of the stakeholder throughout the process (Coombs and Meijer 2021). These recommendations emphasize the need for process and context to be taken into consideration throughout the evaluation (Meister Broekema, Bulder and Horlings 2023). The research of Coombs and Meijer (2021) concluded that in order to accomplish all of these recommendations for evaluating the impact of UAS research, a new research impact evaluation framework is needed, one in which the process and context are central. This same research also indicated that there are three currently used research impact evaluation frameworks, ASIRPA (Joly et al. 2015), PIPA (van Drooge and Spaapen 2022) and Contribution Mapping (Kok and Schuit 2012), that present possible starting points for creating this new suitable approach (Coombs and Meijer 2021).

Contribution Mapping is part of a larger family of contribution analysis frameworks that focus on the research process. Other examples, including Morton’s (2015) Research Impact Assessment, build on the work of Mayne (2011), and are frequently from a realist assumption based in a ‘Theory of Change’ (Riley et al. 2018). They focus on programmatic activities rather than emphasizing external factors and events unlike Contribution Mapping does (Garcia Diaz Villamil, Santos Legaspi and Akoto 2023). Contribution Mapping, however, is based on Actor-Network Theory and thus begins with a performative assumption (Greenhalgh 2016). The process of research and the impact created throughout the process is central to the framework (Greenhalgh 2016). It is formative in nature and, while originally ex-post, it can be done in real-time. It stresses the importance of stakeholders as an active part of the evaluation process in co-production at the project level (Kok and Schuit 2012).

2.3 The use of the stakeholder in research and evaluation

Terms such as co-production, co-creation, societal engagement, public engagement, and citizen science, all speak to the inclusion of stakeholders throughout the research process (Cohen 2022). A systematic literature review conducted by Voorberg, Bekkers and Tummers (2015) has suggested that the terms ‘co-production’ and ‘co-creation’ have come to be not only related but interchangeable. Both terms involve collaboration between stakeholders to design and deliver services, products, or policies with active involvement throughout the process (Voorberg, Bekkers and Tummers 2015). They recognize that three specific forms of stakeholder participation are addressed in the literature on co-creation/co-production; co-implementer, co-designer, co-initiator. While much of the literature attributes these three roles to both co-creation and co-production, Voorberg, Bekkers and Tummers (2015) suggests that co-creation is perhaps best connected with the involvement of stakeholders as co-initiators and co-designers, and co-production is better defined as including stakeholders in co-implementation.

By including stakeholders, it has been suggested that synergy is created between the various types of contributors (Brandsen and Pestoff 2006). As an added benefit, it is believed that by using stakeholders the results will be used beyond the duration of the project as stakeholders develop a shared sense of responsibility for the outcome. Through this sense of responsibility, a power shift can occur where the stakeholders begin to lead in the project and its outcomes (Bovaird 2007). For this to occur, the stakeholders need to see the value of both the process and the outcome of the research (Talsma and Molenbroek 2012). This will also aid in the adoption process (Adam et al. 2018).

However, it is important to note that the use of stakeholders in research does not come without risk. As Oliver, Kothari and Mays (2019) have identified, stakeholder participation can result in extra costs. While these can be financial costs, also temporal, relational, reputational, and ethical costs are also a risk. These challenges are all potentially created by the human factor. The risk of bias and other issues potentially affecting scientific integrity can be caused through issues as human as disagreements or over-eagerness to be of assistance with analysis and resources. In addition, pressure on the researcher to produce a certain outcome or to withhold information to achieve a certain outcome beneficial to the stakeholder can be a source of significant stress.

Recent work by Pel et al. (2023) also, notes that while the use of stakeholders in research can lead to solidarity, it can also lead to an insular situation and the exclusion of important participants. This can be caused by the nature of the subject itself in which, for example, there is a certain associated socioeconomic level. This can also be because of the unintentional focus of the interests and perspectives of specific stakeholders in much the same way as the results of questionnaires need to be methodologically representative. They also speak of several unseen ‘costs’ that come with working intensely with stakeholders. These include, among others, information costs required for educating the stakeholder, negotiation costs associated with negotiating agreements among different stakeholders, and coordination costs involved in coordinating activities and efforts among the various actors.

The work of Boaz et al. (2021) has also suggested that the use of stakeholders throughout the research process presents a set of challenges to the researchers in ensuring that the stakeholders are in fact engaged in the research process as intended. While intentions and expectations for stakeholder inclusion may be initially set at a certain standard, time and energy on the part of the researchers may be required to sustain this. This is particularly true when stakeholders are unsure of the outcome (Talsma and Molenbroek 2012).

3. Method

3.1 Contribution mapping

Through interactions between the researchers and the stakeholders, micro impacts are created throughout the research process (Lykke et al. 2023). The aim of Contribution Mapping is to bring attention to these interactions, as well as actors, activities, alignment efforts, and outputs to illustrate the potential trickledown effect of knowledge to indicate potential impact. Like other forms of Contribution Analysis, this framework helps to identify and understand the many links, factors, and actors, be they humans, outputs or activities, that all contribute to creating impact in applied research. The approach focuses on the research process and how the actors, involved convert knowledge into ‘actor scenarios’ to make a contribution (Hegger et al. 2016). It systematically maps out what efforts are made to achieve additional contributions from the research to assist in solving a specific (social) problem. It focuses on the many connections between process, individuals, organizations and actors and other factors to create impact rather than on the linear output-impact concept (Boshoff and Sefatsa 2019). This mapping allows for the feedback loops of research to be taken into consideration and attempts to reveal the multiple mechanisms of impact (Beckett et al. 2018). Figure 1 presents the visual outcome of the Contribution Mapping process.

Contribution Mapping identifies four types of actors. The first type is the ‘investigator’, those who are directly involved in the research. Secondly, ‘linked actors’ are those with whom interaction occurs during the research. This can be through contribution to the research plan, participation in the implementation, or the interpretation of results. The third type of actor is the ‘potential key users’. These people are ‘linked actors’ who play a central role in relevant networks and seem most capable of translating the research into new ways of acting. They can for example include policy makers, representatives of patient associations. Finally, the ‘unlinked actors’ are those who are not connected to the research process, such as people in practice, but who become aware of it and create new action scenarios themselves.

Contribution Mapping recognizes four types of contributions. One type of contribution relates to changes in competency, behaviour, and relationships of researchers and linked actors created by activities that take place during the research. A second is knowledge products or outputs being added to ‘codified knowledge reservoirs’. These products include outputs such as protocols, publications, and methods. ‘Codified knowledge reservoirs’ can be understood as journals, scientific databases and repositories or other ways of disseminating output. A third form of contribution is new actor scenarios being created between the researchers and linked actors. Finally, applications of the knowledge by actors who are not involved in the research are seen as a fourth means of contribution. Contribution Mapping states that the conscious efforts made to generate more contributions from the research are ‘alignment efforts’.

The Contribution Mapping framework divides the research process into three phases. The first phase is Research Formation. This phase includes the exploration of possible research questions, the search for funding, discussions with potential actors, and the setting of priorities. Kok and Schuit have suggested this phase closes when funding is awarded. The second phase is called the production phase. This is the stage where knowledge production takes place. This includes the myriads of activities that take place while doing research such as recruitment, purchasing equipment, theory development, experimentation, and statistical analysis. Knowledge can be shared at this point in the process; however, this phase is closed when the researchers determine the final results of the study. The third phase, knowledge extension, makes the acquired knowledge available to potential users, and stimulates the application of the knowledge. This can be done through different forms of output such as presentations, publications, and data depositing. Linked actors can also play a role in this stage by applying the knowledge in practice.

The original method of Contribution Mapping involves a step-by-step mapping plan, in which the characteristics of the research being evaluated are mapped out with the contributions made, when, and by whom throughout the research project. There are four Stages divided into 10 steps in this plan beginning with Stage 0. This is understood as the setup stage where the research team conducting the mapping reviews what contribution mapping is, its purpose, roles, expectations, and benefits. Stage 1 involves interviewing the researchers to attain a first impression of the process and possible contributions. This is followed by Stage 2 in which potential key users and other relevant actors are interviewed to trace, explore, and coordinate possible contributions. The third Stage maps out and analyses the (possible) alignment efforts. Preliminary results are shared with relevant stakeholders for feedback and validation. The final mapping is then shared with the stakeholders in order for them to learn, improve, and be accountable.

3.1 The case study

To test the theory of Contribution Mapping, we used the project GO! Noord Nederland as a case study. GO! Noord Nederland is a collaborative project funded by a large health oriented Dutch funder. This particular project, which ran for four years, was selected because it is an example of the type of research that takes place in a UAS where there is a clear link to practice, is transdisciplinary, utilizes a variety of stakeholders throughout the process, and flows back into education in several ways. This type of research stresses the active role of the stakeholder throughout the research process whereby an impact is created in the profession and education through this knowledge transfer. This project appears to lend itself well to analysis through Contribution Mapping.

The GO! Noord Nederland project has been carried out by a large consortium of two Universities of Applied Science, five municipalities, three provincial public health departments, the Dutch National Institute for Public Health and the Environment, and a regional Safety and Security Organization. Their collective ambition was to identify and utilize opportunities to enhance the living environment in municipalities. The aim was to accomplish this by implementing adjustments to the public space in co-creation with residents.

While this case study focuses on the research project itself, it is interesting to note that a central element of the GO! Noord project is the GO! Method, that has been developed by the Dutch National Institute for Public Health and the Environment. The GO! Method is a six-step process that starts with data collection related to the area of interest in order to create an overview of important themes and social developments. This is followed by collecting statistical data on; the municipality population, housing, health, safety, facilities and green space. Using this information as a basis for conversation, dialogue with residents and local stakeholders is initiated to assess opportunities, threats, and needs for improvement. Based on this work, an overview of options is generated, in which different combinations of possible changes for creating a healthier environment for the municipality are presented. In consultation with the residents, the municipality can then select and implement the desired changes. Finally, the effects of these changes on the health of the residents are monitored and evaluated (Gezonde leefomgeving Noord Nederland 2023). The method itself focuses on collaboration and interactions with stakeholders in iterations and feedback, as well as implementation.

3.2 Application of contribution mapping

The purpose of this case study was to test the implementation and applicability of the Contribution Mapping method by following the designated phases as presented in the article by Kok and Schuit (2012) in ‘Contribution Mapping: A method for mapping the contribution of research to enhance its impact’. In doing so, we hoped to understand the theoretical and practical implications of this approach, and gain insight into what is needed for a new research impact evaluation for research conducted at UASs. While the recommendations for impact evaluation of UAS research suggests that an evaluation should take place in real-time, this study followed the initial instructions of Contribution Mapping and performed the evaluation ex-post. This allowed the research team to assess the model in its original form in the hopes of making informed decisions for any alterations that may be necessary to meet the requirements for UAS research impact evaluation.

The process of evaluation began with an analysis of documents including research proposals, project plans, and a project website. To facilitate the performance of this evaluation, we utilized a detailed summary of Kok and Schuit’s Contribution Mapping framework created by van Vliet (2021). This provided a step-by-step guideline distilled from Kok and Schuit’s (2012) initial article on Contribution Mapping. It was used primarily as a guide for the semi-structured interviews. These interviews correspondent to Stages 1 and 2, and focused on establishing the roles of the actors and the contributions made in the form of activities, outputs, and alignment efforts throughout the three phases of the project. Online interviews were conducted by two members of the research team (one that knew the project and one that had no prior knowledge of the project) over the course of 2 months. A total of 12 interviews were conducted, one with each member of the consortium and project participants (researchers, linked actors, and key users). Interviews were conducted in accordance with the ethical rules of the research institutions. Transcriptions of the interviews were verified and approved by the participant.

Using the description of the Contribution Mapping framework as a guide, the focus group was conducted online by two members of the research team with the members of the project team in correspondence with Stage 3 of the framework. This allowed for verification of the interview content as well as the receipt of feedback.

Each member of the research team took notes during the interviews and focus group. Following each of these events, these notes of observations, findings and points of interest were discussed and compiled. These were included in the verification process with participants. It also provided an opportunity to discuss the actual process of conducting Contribution Mapping and what was needed to be more successful in capturing the micro impacts of the research being evaluated.

Inductive analysis of the interviews and notes was conducted by members of the research team. The information contained in the interviews as well as the research team notes were placed per respondent in a matrix according to the stage in the contribution mapping process (Stage 0–2), and the stage in the research process (formation, production, and extension). Commonalities such as activities, outputs, roles, language use, and alignment efforts were colour coded. This resulted in a thematic categorization of data for both the assessment at hand and the larger question for evaluating the impact of UAS research.

In accordance with the final stage of Contribution Mapping, a visual representation of the actors, activities, outputs, and alignment efforts based on both the interviews and focus group, was constructed in idraw. This was initially done by one member of the team and then built upon by the remaining two members in an attempt to illustrate the areas of potential micro impact. This was further developed with a graphic designer and can be found in Figure 2.

The observations and conclusions of the mapping process (Stage 3) were then added to the matrix. Table 1 presents the analysis matrix prior to utilization.

The themes were grouped according to observations, challenges, limitations, and modification suggestions to assess the useability for a future research impact evaluation framework. Table 2 provides an overview of this data.

4. Results

As can be seen in Figure 2, this mapping of the research process for the project GO! Noord Nederland attempts to visualize the various actors, activities, outputs and alignment efforts that took place throughout the Go! Noord Nederland project. The three stages of the research process, Formation, Production and Extension have been set out, as have the involved actors, the activities that have taken place, and the new participates in the activities. As indicated in Table 2, the evaluation research team was not satisfied with the initial results. In an attempt to enhance the story and begin to provide context, information about the new actors/stakeholders was added after much puzzling as to how to make the mapping less flat and linear.

Table 2 presents the final results of the thematic analysis and includes the observations of the evaluation research team’s throughout the evaluation process. Based on the research team’s reflections, these have been clustered according to the stage in the Contribution Mapping framework and whether they are perceived as a challenge, a limitation, or a need for modification by the participants or the research team. The participants from which these themes have been gathered are indicated.

As can be seen in Table 2, the participants in a research project, be they researcher or stakeholder, appear key to a project. While often selected because of their contribution to a network, it is the person that becomes fundamental to the project rather than the institution they represent. Consequently, the role they come to fill within a project is not always what they are initially chosen for or what they are aware of themselves but is something they grow into throughout the course of a project. This can be a ‘translator’, ‘educator’, internal coordinator, external networker or something else entirely. They are not necessarily reflected in the terminology used by Contribution Mapping. When participants change, be they researchers or stakeholders, the momentum of a project is lost. The people leaving the project not only become difficult to replace but also to track as they take knowledge from the project with them. It is the commitment of the people in the project that appears to bring the extension phase of a project beyond the output of the project itself.

Table 2 also underscores the challenges with the implementation of the framework itself. A project manager is perhaps best suited to conducting this framework as it is clear that inside information from the start of the project is required for understanding how everything fits and flows together. The use of interviews is better done in a conversation-like, semi structured style rather than a structured one as it provides the respondents the freedom to state and discuss what they feel is required. The outcome of the mapping exercise, based on these interviews and focus groups, was very flat and does not show the context of the research. There are many different and interlinking routes within the mapping that are, as seen above, reliant on the participants. It is important to avoid becoming rigid in the linearity of the exercise as twists and turns throughout the project are unpredictable and lead to important indirect impacts fundamental to the long term impacts of the project. These long term impacts also appear reliant on the people involved.

5. Discussion

By implementing Contribution Mapping, the aim of this study was to learn from Contribution Mapping and assess how this can be utilized in creating an impact evaluation framework for UAS research. As one of the recognized frameworks for evaluating UAS research, Contribution Mapping accomplishes some of the requirements for an evaluation framework for evaluating the research impact of research conducted by UASs. It is formative, based on a performative assumption and conducted in coproduction with project stakeholders. As such, it provides a sufficient starting point for investigating how an impact evaluation framework could potentially be structured for UAS research projects. Contribution Mapping’s performative assumption clearly gives insight into the potential trickledown effect of impacts created through actors, interactions, activities, and outputs. It provides a visualization through which these potential outcomes can be traced. As a formative tool, these mappings allow you to see where the links in the Knowledge Triangle are functioning as well as where they need to be strengthened. It provides an opportunity to explain the reasoning behind the choices made. It is, however, not without its challenges, limitations. There is a need for modifications in the process, and results that need to be taken into consideration for a future UAS specific impact evaluation approach.

5.1 Challenges

Testing theory in practice does not come without its challenges and conducting Contribution Mapping in real life proved to have several. The method of Contribution Mapping is very structured and theoretical which created challenges in both the execution of the framework and the participation of the stakeholders.

The use of participant interviews and focus groups highlights the differences in experiences and goals of those being interviewed. For many participants, interviews gave them the opportunity to share their side of a larger story. They were open about their feelings and experiences. At the same time, focus groups allowed for a common narrative to be told. It became the job of the evaluation research team to plot these experiences together. By doing so it became evident that people were less open to reveal their opinions in the focus group session. In contrast, they could be more vocal in the interview. This confirms the importance of hearing each person’s interpretation of the story and confirms the theories behind using interviews in impact evaluation frameworks (Budtz Pedersen, Grønvad and Hvidtfeldt 2020). Fortunately, the research team was quick to realize that the questions to be asked were too structured. Because participants did not understand exactly how to answer the questions and did not see themselves reflected in the jargon, momentum was lost. The team switched to a semi-structured interview system with a focus on people, activities, interactions, outputs and ‘alignment efforts’. This resulted in interviewees being more comfortable and more likely to provide fuller responses which in turn garnered more information.

Who conducts the interviews would also appear to be important. To fully understand the results, inside knowledge of the project and participants is required. The research team was made up of both people from the project and people outside the project. It became clear that conducting this study as an outsider is more difficult because you lack the nuanced information of the project. The project manager may be best suited for this role as they know the full story and consequently can prioritize and manage the intersection of the people and the project. This is especially relevant given the time required to conduct this evaluation. However, the etic perspective of an outsider is also important to uncover participants biases.

The constraint of time is most definitely a challenge throughout this framework. Organizing focus groups and preparing, holding and analysing interviews is very time intensive. Proper application requires a commitment from both the participants and the funders to include sufficient time in planning and in the budget, as in this situation, time is literally money. Funders need to be aware that stakeholder participation requires additional resources such as time and money (Beckett et al. 2018). Funding is allocated in favour of the researchers. Stakeholders are often expected to give in-kind without financial incentive but with the promise that the project may result in useful and tangible results for them. While co-production in the research project and evaluation often leads to stakeholders feeling responsible for the research outcomes, proper financial recognition for their time could increase their commitment (Voorberg, Bekkers and Tummers 2015).

The language used in the framework such as ‘linked’ or ‘unlinked’ actors, ‘key users’, ‘alignment efforts’ and ‘codified knowledge reservoirs’ was frequently difficult for participants to grasp. The research team spent time explaining what each of these terms meant in the hopes that participants would be able to identify what role they felt they played. This point moves the discussion from challenges to limitations, as participants were unable to recognize their roles and functions within the research project in the terminology used.

5.2 Limitations or opportunities?

The suggested roles of participants, as ‘linked actors’ or ‘key users’ are perhaps too limited for the degree to which stakeholders are utilized in UAS research. The ‘linked actor’ concept, because of its dual role with ‘key users’, creates confusion. In addition to not always understanding what the terminology meant, participants in this case study did not feel they were represented by the terminology. Within each role, there were varying degrees to which participant felt they related. The role they played also had the potential to change over the course of the project. Other participants did not find their role included in the terminology at all. One specific role participants felt was important to specify was that of ‘translator’. One stakeholder who neither identified as a ‘link actor’ nor as ‘key user’ instead identified as a ‘translator’. Their role within the consortium was to translate the academic language of the researchers to the practical language of the other stakeholders. They recognized that they had a similar responsibility for translating the differing agendas, both research agendas and calendar agendas of all partners. It became clear that the needs of the stakeholder or research are not the same and the academic school year differs greatly from that of the stakeholder. Beckett et al. (2018) acknowledges that all of these components included in the term ‘translator’ are key for the success of co-production projects. Perhaps the roles defined within Contribution Mapping could be further refined in a similar way to Arnsteins’s (1969) ‘Ladder of Citizen Participation’. Though not directly reproducible for this situation, the ‘ladder’ presents a gradual increase in role and responsibility of the stakeholder that could be adapted to the evaluation framework, thereby, increasing the opportunities for stakeholders to identify with their position in the process (Voorberg, Bekkers and Tummers 2015).

The limitation in roles represented in Contribution Mapping is an example of a broader limitation. While the end product of this framework presents an overview of the ‘Key Users’, ‘Linked’, and ‘Unlinked’ actors, activities, and output throughout the three phases of the research process, the results remain almost flat and do not reflect the rich information captured in the interviews and focus groups. Researchers such as Cohen (2022) and Beckett et al. (2018) have presented possibilities for augmenting the results of Contribution Mapping through the collection of additional information. Cohen’s (2022) research into the institutionalization of public engagement presents a framework for analysing the barriers and enablers of public engagement so that impacts created with stakeholders can be better recognized, facilitated, and extended. The research done by Beckett et al. (2018) makes use of Davies, Powell and Nutley’s (2015), Knowledge Mobilization Archetypes, itself a multimethod mapping study, Pawson’s (2013) ideas of context, and micro, meso, macro levels to enrich the mapping exercise. This provides more specific information about impact created through the co-production process.

Perhaps these ‘limitations’ should not be seen as limitations but as ‘opportunities’ to create a more meaningful evaluation specific to the context of the research taken on by UASs. To do so, however, requires that the evaluator has a deeper understanding of the stakeholders and the context in which the research takes place in order to make these important components of the evaluation explicit. While some have suggested that Contribution Mapping leaves room for the multiple contextual factors of the research being evaluated (Riley et al. 2018) others have suggested that it is not explicit enough to do justice to the specific context of research conducted by UASs (van Beest et al. 2022). The work of van Beest et al. (2022), the Research Pathway Model (RPM), strives to create a more explicit means of identifying the context of research. Structured around the UAS research process, RPM presents a matrix of nine research activities and contexts meant to represent where reciprocal collaboration with practice takes place, and thus generates impact. A certain flexibility is required here as the contexts in which research is conducted changes throughout the project (Oliver and Parolin 2018). A deeper examination of both the options and changes could potentially aid in mapping the different contexts in play during a research project. It may also assist at the level of the individuals participating as each participant brings their own context and ‘actor scenarios’ that can potentially influence the process and the horizontal and vertical alignment efforts (Hegger et al. 2016).

The actual mapping of the interviews and the overlap made it difficult to separate the individual stories from each other and how they connect to each other and the context in which it occurred. It should be recognized that context is different for the stakeholders than for the researchers. While for a researcher this is research, for a stakeholder it is a different scenario and what they do, what they create, and their interactions are often from a different perspective than that of the researchers. In a true co-creation/co-production process both perspectives are equally acknowledged whether this be as co-implementer, co-designer, or co-initiator. It is important that in evaluating we do not lose sight of the process and interactions from both stakeholder and researcher perspectives. This is more than context alone and should stand centrally in the evaluation process.

Another ‘opportunity’ would be to follow the paths of those stakeholders who left the project. Herein lies a point of discussion around the responsibility of the individual participant or the institutional stakeholders and their commitment to the project. From a practical point of view in those instances where stakeholders exit the project prior to completion, the remaining participants are left scurrying to find appropriate replacements who then must become quickly familiarized with the project. It also results in changes in dynamics, time lag in the project, and potential outcomes and impacts (Beckett et al. 2018). However, those leaving the project leave with the potential for future impacts created by bringing the knowledge and experiences they have had in one project, to different contexts and experiences in the future. By indicating this in the mapping exercise, the potential for future impacts can be acknowledged.

These are but three ways in which others have experienced the limitations of Contribution Mapping and have tried to create a fuller picture of impact creation through modification. This would suggest that perhaps the performative assumption on which Contribution Mapping is based presents a starting point on which to build what is needed for a particular scenario. Changing the ‘limitations’ to modification would allow Contribution Mapping to tell a richer story and become a stronger framework for identifying contributions to impact and becoming a more formative tool that can be used in the evaluation of UAS research impact. This would then also allow for more co-creation and a demand driven research process and real-time evaluation. This could be in the area of context, stakeholders’ participation, types of interactions that take place with which type of institution, and the like. Including this type of information in an impact evaluation would aid researchers in being able to trace contributions, align efforts and in doing so, tell the story of their research impact in a more insightful way showing potential micro impacts that can lead to macro ones.

5.3 Modifications

In addition to the numerous ways in which Contribution Mapping can be enriched to strengthen the story the visualization tells, there is one modification that would need to be made for Contribution Mapping to fulfil all the recommendations for the evaluation of UAS research impact. As previously touched on, this changes the framework from ex-post to a framework that begins to evaluate the impact from the start of the project. This is similar to an exercise ASIRPA has recently conducted (Matt et al. 2023). There are several advantages to this timing.

A real-time impact assessment is ideally built in from the planning stage where the links between how research is done, who is engaged, and its potential impact are all considered at the beginning (Morton 2015). This could be time saving as the mapping becomes part of the routine; monitoring function built into the process, which allows the contributions and alignment efforts to be purposefully made. This would aid researchers in accounting for decisions and changes as well as attempting to ensure the desired impacts and outcomes are reached and opportunities for alignment are not missed (Hegger et al. 2016).

In addition, by including impact evaluation from the onset, it may be possible to avoid the miscommunication and adverse differences while encouraging the positives. In a nonlinear process, adaption is critical (Matt et al. 2023). However, through impact evaluation in real-time you run the risk of steering too much rather than allowing the impact to take its own course. As discussed by Oliver and Parolin (2018), conducting this in real-time can lead to pre-specifying the outcomes. By plotting details step-by-step beforehand, in much the same way as impact pathways function, the risk of losing that flexibility is created. Also, a further risk is that of the process becoming linear in a preconceived logic model that isolates specific parts of the process rather than the whole process (Kok 2021). By attempting to compress all opportunities into Contribution Mapping or any contribution analysis we run the risk of reducing it to a strict set of requirements that results in the messiness of UAS research spilling over and not being able to accomplish the evaluation of impact as desired (Oliver and Parolin 2018). Further research into how to balance a more informed evaluation without becoming overly constrained and linear, is required.

6. Conclusion

It is important to realize that frameworks are theoretical and thus open to change. A balance between the framework and its applicability needs to be found. This study into the use of Contribution Mapping in UAS research impact evaluations set out to do just that; to bring theory into practice in order to discover what we can learn from Contribution Mapping as an impact evaluation tool for a future UAS specific research impact evaluation framework.

This study explored the application of Contribution Mapping in evaluating the impact of UAS research. The framework, while providing a starting point for understanding potential impacts, faced challenges and limitations in real-life implementation. The structured and theoretical nature of Contribution Mapping posed execution and stakeholder participation challenges. These should be taken into consideration when developing a UAS specific impact evaluation framework.

Interviews and focus groups reveal diverse experiences and goals among participants, emphasizing the importance of nuanced information. Challenges included the structured interview format, the role of interviewers, time constraints, the complexity of terminology, and the changing of roles and priorities over time. The limitations in participant roles highlighted the need for a more comprehensive categorization, akin to Arnstein’s (1969) ‘Ladder of Citizen Participation’.

Perhaps viewing these challenges as opportunities would enrich the evaluation process. Additional frameworks, such as those addressing barriers and enablers of public engagement, were proposed to enhance Contribution Mapping's results and the role of stakeholders in the research and its evaluation. Acknowledging context variations and understanding stakeholder dynamics are required for a more meaningful evaluation.

Further modifications to Contribution Mapping would also enhance its useability. Results of this study advocate for a shift from an ex-post evaluation to a real-time impact assessment integrated from the project’s planning stage. This approach aims to be expeditious, facilitate purposeful contributions, and mitigate miscommunication. However, caution is advised against over-restriction and linear pre-specification of outcomes.

Based on this case study, and in answer to our main research question, we conclude that this framework of Contribution Mapping has potential as the underpinning structure of a performative, formative impact evaluation tool for UAS research, that requires methodological modifications, increased flexibility, and stakeholder orientation. Even without meeting all the requirements stipulated for evaluating the impact of research done by UASs, this has proven to be insightful. Lessons such as: having a framework (phases and questions) are beneficial, but should not overly complicate resulting in the potential trickle down effects getting lost in all kinds of roles and repressive structure; the use of exact definitions should not be more important than the outcomes; the focus should not be on the outputs but on the outcomes drawing our attention to the alignment efforts that take place; and finally, regardless of the framework, the actual focus on the trickle down effects created throughout the research process in interactions and outputs provides a wealth of information to follow up. It is not one big thing that creates impact but many small steps over time.

To meet the rest of the recommendations for evaluating UAS research impact, further pilots with this framework will need to start at the beginning of the project to allow for real time and balancing structure with adaptability. Future research will need to look at how to modify Contribution Mapping to create a taxonomy of evaluation that can assist in telling a more textured story without becoming entangled in s structured, linear framework that does the opposite of what is desired. In doing so we become steps closer towards evaluating the impact of UAS research in a robust and systematic way.

Acknowledgements

The authors would like to thank Paul Wouters of Leiden University for their critical comments and feedback.

Funding

No funding was used for this study.

References