Development and pilot study of a university research culture questionnaire

Abstract

A positive working environment and culture are essential for researchers as these enable them to conduct valuable, high-quality research. Yet, university staff frequently report their research culture as less than ideal. To understand researchers’ experiences of research culture to inform tangible change, several surveys have been conducted by research-related organizations and individual universities. However, despite a plethora of studies, there does not appear to be a widely adopted research culture questionnaire, with variation in content and length in those used to date. A 37-item research culture questionnaire was developed based on the extant literatures. It was piloted in one small-medium sized university with 177 academic staff across a range of disciplines engaged in research. Qualitative questions were included to provide a richer insight into current research perceptions. Exploratory factor analysis identified eight factors, providing an initial framework of research culture. This consisted of: School Research Value, University Research Value, Research Support, Research Knowledge, Collaboration, Wellbeing & Inclusivity, Open Research and Research Integrity. Whilst it will require further testing and refinement, a preliminary psychometric analysis provides initial indications of internal structure and internal reliability. The factor set provides insight into research culture drivers which can be used to target effective interventions. This type of research culture questionnaire would allow universities to not only assess their own culture but also benchmark their results against other universities. A standardized research culture measurement process (e.g. questionnaires, narratives), feeding into research evaluation activities, may have wider implications for those looking to facilitate research culture changes.

1. Introduction

Advances in research have transformed the world as we know it, offering solutions to the global challenges we face as a species. Having a supportive working environment and a positive research culture is vital for facilitating ongoing and future investigations. Definitions of research culture vary but it can be understood to reflect the values, norms and behaviours of our research communities (Royal Society 2017: 3). In essence, the ‘way that we do research’ within an organization, community or sector (e.g. UK higher education). A positive culture is typically collaborative, inclusive, supportive, and creative with leadership providing researchers time to focus on research priorities in a safe and secure environment (Wellcome Trust 2020). The importance of developing such a culture is beginning to be recognized at departmental, organizational and sector levels, with increasing value being placed on research culture and environment in research assessment activities (e.g. the UK Research Excellence Framework). However, evidence suggests that many researchers’ experiences of their research culture is less than ideal.

Competitive, pressured to publish, undervalued, and isolated are terms used to describe UK researchers’ perceptions of their working environment [e.g. Association of Research Managers and Administration (ARMA) 2020; Wellcome Trust 2020; Russell Group 2021]. In addition, early career researchers face problems regarding job insecurity, research integrity, and lack of support or mentorship from superiors, as well as reporting a poor work-life balance (Acton et al. 2019; Christian et al. 2021). Whilst there may be a lingering, erroneous belief that competitive research environments produce high-quality research outputs, evidence suggests that it is a positive research culture, underlying values and strong leadership that foster higher-performing research units (e.g. as measured by the UK Research Excellence Framework, Manville et al. 2015). These research culture issues are not specific to the UK (Acton et al. 2019) but are widespread, evident across the international research community (e.g. Australia, Christian et al. 2021; Germany, Austria and Switzerland, Johann, Raabe and Rauhut 2022; Spain, Ion and Castro Ceacero 2017; Finland, Aarnikoivu et al. 2019; USA, Morin et al. 2022; and in New Zealand, Spronken-Smith, Mirosa and Darrou 2014). Furthermore, there are concerns regarding how sustainable these negative research environments are in the long-term (Wellcome Trust 2020). Higher education is experiencing the ‘great resignation’ in which waves of researchers are leaving academia, as illustrated by Nature’s global 2021 careers survey (Gerwin 2022).

An organization’s research culture will also impact on post-graduate and post-doctoral researchers’ experiences, as well as undergraduate students’ perceptions of research (Spronken-Smith, Mirosa and Darrou 2014) via opportunities to contribute to the research environment, values and community. This enriches their research training, enhances research outputs and supports timely completions (Brew, Boud and Malfroy 2017). Research culture also feeds into the teaching culture (Healey 2005). However, evidence from the UK Postgraduate Research Experience Survey (PRES) indicates that research culture performs poorly compared to other elements of the research degree experience (Neves 2022), with a negative trend of decreasing post-graduate satisfaction with research culture (Pitkin 2021).

In short, research culture needs to be improved (Wellcome Trust 2020; Russell Group 2021; Science Europe 2021). Research organizations are under scrutiny to demonstrate that they value and promote a positive research culture, not only by governmental bodies and research funders, but by researchers themselves. Within the context of national research evaluation systems, there has been a growing shift towards recognizing the value of research culture and environment (e.g. UK Research Excellence Framework and the Excellence for Research in Australia), and consequently failure to address such issues can have a detrimental financial impact on an institution’s funding. In response to these concerns, organizations have begun to outline approaches for improving academic research culture (e.g. Royal Society 2017; Russell Group 2021). To be able to direct effective change, there needs to be a shared understanding of what research culture encompasses (e.g. a framework) and a set of methods for quantifying it. A valuable strategy has been to conduct research culture surveys to better understand these challenges, identify potential routes for development, and provide a benchmark for sustainable, long-term improvement (e.g. Wellcome Trust 2020; Russell Group 2021). Access to comparative data can be valuable for cross-institution benchmarking as well as for tracking sector trends. For example, Solans-Domènech et al. (2019) took a similar approach by developing a questionnaire for research impact, examining the concept and capturing the perceived impacts of research. However, there does not appear to be an empirically grounded and widely adopted research culture questionnaire available for research institutions/universities. Furthermore, there is variation in the content of the questionnaires, typically without an underlying framework of research culture, nor psychometric evaluation.

This paper briefly discusses research culture and how it is currently measured, and then outlines the development and pilot study of a research culture questionnaire at a Scottish University. Whilst this was developed within a single university, it is intended to offer a starting point from which other academic institutions can begin their own research culture journey.

2. Research culture

Given the breadth of experiences in research, research culture can be a fuzzy concept to define (Pratt, Margaritis and Coy 1999; Casci and Adams 2020). Research culture is broadly accepted as encompassing ‘the behaviours, values, expectations, attitudes and norms of our research communities’ (Hill and Haigh 2012; Royal Society 2017: 3). It determines the way the way that research is conducted and communicated, influencing researchers’ career paths and mental wellbeing. Research culture is commonly discussed in the context of a specific organizational team, department or university but can also refer to a broader culture existing within a geographic area (e.g. the UK, the US, or Europe). Consequently, in the organizational context, research culture can be understood as a sub-culture of the wider organizational culture of the university or institution (Tierney 2008; Schein 2016; Ion and Castro Ceacero 2017). Organizational culture typically relates to the organizational values that are communicated through norms, artefacts, and observed behavioural patterns (Schein 1992), that encapsulate the ‘way that we do things here’. However, there is variation in the way that research culture is defined and measured, impacting on the ability to direct effective change.

Culture varies from place to place, with different individuals having personal experiences as well as between and within disciplines (Wellcome Trust 2020). It may include disciplinary or interdisciplinary ideas and values, expert knowledge, cultural practices (e.g. how peer review is conducted), collaborative networks, and department sociability (Deem and Brehony 2000; Kashif et al. 2022), as well as tangible elements of the ecosystem (e.g. legal requirements, physical settings, facilities). In terms of capacity building perspectives, organizational research culture may be interpreted as a proxy for reaching a critical mass of research in a discipline, or expectations regarding publications and grant capture (Billot 2010; see Brew, Boud and Malfroy 2017). Therefore, research culture may also be interpreted as a proxy measure of research productivity.

From a theoretical perspective, this variation may be due to a lack of a clearly defined set of underlying research culture dimensions. For example, there are general university culture frameworks (e.g. Tierney 2008; Lacatus 2013) and literature reviews on the factors influencing research culture (Hazelkorn 2005), as well as identification of factors associated with research intensive universities (Billot and Codling 2013), but the literature appears to lack an empirically grounded, validated framework of research culture dimensions. Furthermore, the research culture dimensions vary across different surveys, and include a range of different elements such as research policy, leadership, resources and support, workload and inter-personal relationships (see Amin, Bashir and Ali 2020 for a review). Consequently, the definition of research culture may flex in relation to the context of usage, intent and level of analysis.

Within this paper, we are using an organizational culture lens, in which research culture can be viewed as a sub-culture of the wider organizational culture, in the university context. In essence, we view research culture as relating to the way that research is conducted and importantly how it feels to do research in that work environment. This will in turn shape research norms and behaviours. We recognize that research culture can be interpreted in different ways but believe that taking this approach is not only valuable for being theoretically grounded (e.g. Schein 2016), but places greater emphasis on meaningful and sustainable change at the institutional, and wider sector, level, rather than on individual researcher metrics (e.g. number of grants captured, papers published). It may also allow for variation in institutional capacity and size to support a positive research culture. In addition, it provides potential future opportunities to learn from other research areas that have examined aspects of organizational culture, such as culture questionnaires (e.g. safety culture methods or tools).

2.1 Research culture questionnaires

In recent years, significant efforts have been made to better understand the challenges associated with organizational research cultures. Organizational culture can be somewhat intangible. Schein (2016) theorizes that the less visible elements of organizational culture, such as values and norms, can be detected through examination of artefacts (e.g. stories, rituals, and language) and observable behavioural patterns. Organizational culture frameworks may provide initial indications of what research culture may consist of in terms of themes or components such as leadership vision, how research value is communicated and the ways that it is supported (e.g. resources, policies, or procedures) (Cheung, Wong and Wu 2011; Hogan and Coote 2014; Schein 2016). Examining these dimensions through questionnaires can be an effective route to making organizational research culture more visible. Furthermore, there is a need to develop an evidence base for research culture measurement, which in some instances is limited to proxy measures (e.g. signing of declarations or concordats) (Curry, Gadd and Wilsdon 2022).

Several sector organizations have conducted large scale questionnaire surveys to capture a wide range of researchers’ experiences (Association of Research Managers and Administration (ARMA) 2020; Vitae 2021). Qualitative approaches can provide insight into the complexities of research culture (Tynan and Garbett 2007) but such methods (e.g. interviews or focus groups) may be prohibitively time intensive for research organizations which may not have the resources to conduct such activities. Furthermore, where it is not feasible to run large numbers of workshops or focus groups, surveys can allow for a larger sample to share their experiences. Running a questionnaire survey can be valuable as this can provide quantitative data and a baseline from which interventions and change can be enacted (e.g. policy, funding and other resources), with their effectiveness measured in subsequent surveys. It is also possible that such activities may be valuable to feed into submissions for national research evaluation activities (e.g. UK REF). If multiple organizations adopt the same survey tool and share their results, a benchmarking process can be established providing within-sector comparisons.

Surveys have also been utilized to understand the difficulties that specific groups face, such as early career researchers in Australia (Christian et al. 2021), undergraduate students (Spronken-Smith, Mirosa and Darrou 2014), and principal investigators in the UK (Acton et al. 2019) as well as certain geographic areas (e.g. sub-Saharan Africa; Boshoff and de Jong 2020, Philippines, Salazar-Clemeña and Almonte-Acosta 2007). Whilst these questionnaires may be valuable, they are often too specific to one group or topic to be used to capture research culture across a research community. Furthermore, typically the question items are not based on a widely accepted framework of research culture. However, they are valuable for highlighting potential research culture components [e.g. research integrity (Martinson, Thrush and Lauren Crain 2013)]. Table 1 provides a summary of the research culture questionnaires available.

In some cases, these are theoretically grounded questionnaires that have been subjected to initial psychometric validation, such as the Research & Development (R&D) culture index piloted in UK primary care trusts (Whitford et al. 2005) or the faculty research culture scale piloted in USA universities (Borders, Wester and Gonzalez 2018). However, generally the purpose, length, content and underlying organizational research culture dimensions vary across surveys (see Amin, Bashir and Ali 2020 for a review) and are not based on a well-defined framework of research culture. Furthermore, the psychometric properties of organizational research culture instruments have received little attention. With the increasing recognition of research culture, and the associated evaluation activities (e.g. REF 2029 People, Culture & Environment element are anticipated to contribute 25% to the overall quality profile; UKRI 2023), there is a need for a validated research culture instrument as well as to better understand the underlying dimensions that drive research culture.

3. Study aims

The aims were to: (1) develop a relatively short research culture questionnaire based on the extant literature that would be suitable for individual universities to measure their research culture and (2) pilot test this questionnaire with research active, academic staff at one university, allowing for an initial examination of some of the psychometric properties.

4. Method

The research culture questionnaire was piloted with a modern (post 1992) Scottish University made up of 11 discipline specific schools which conduct teaching and research, and a specialist research institute. With support from senior management to improve research culture, it was an ideal situation in which to develop and pilot the research culture questionnaire.

4.1 Questionnaire development

The research culture questionnaire design criteria were: (1) easy to use, (2) to be completed within 15 min and (3) provide an initial baseline of key dimensions of research culture. It also offered an opportunity for open feedback on current challenges and methods for improvement. It was designed for research active academic staff (see sample).

An initial review of the available research culture questionnaires was conducted (see Table 1) as well as key sector surveys in detail to identify potential survey items [Royal Society 2017; Association of Research Managers and Administration (ARMA) 2020; Wellcome Trust 2020; Russell Group 2021; Vitae 2021; Neves 2022]. The extant literature was also reviewed to further identify potential questionnaire items and underlying research culture dimensions. For example, leadership and research policy (Puplampu 2015), workload (Bai et al. 2013), available resources and collaboration (Gasson and Bruce 2019) and motivation (Horodnic and Zaiţ 2015). The University of Glasgow’s survey had a realistic completion time of 10–15 min. Based on the available questionnaires and wider literature, a set of potential dimensions were identified as influencing research culture, driving the selection of the individual questionnaire items. The topics covered by the initial survey questions were open research and research integrity, research value, leadership, collaboration, workload, incentives, and career progression, as well as equality, diversity, and inclusion (EDI).

A preliminary questionnaire was developed by adopting and in some cases adapting items from the University of Glasgow’s research culture survey and the faculty research culture scale (Borders, Wester and Gonzalez 2018) as a guiding foundation. This included reference to Schein’s (2016) model of organizational culture in the item set by including examples of artefacts (e.g. celebration rituals) and observable behaviours. The initial set was then supplemented by items from the Wellcome Trust survey (2020), the research and development culture index (Whitford et al. 2005), Martinson, Thrush and Lauren Crain’s (2013) research integrity items, and the ARMA survey (2020). This work indicated that research culture dimensions typically include leadership vision, communication, collaboration, career development, and research value, as well as research knowledge and skills. Given that research culture, and the wider organizational culture, can vary across an institution, eight items were repeated with the wording changed from University to School. This school level research culture may be particularly important for teaching focused universities (Pifer and Gilio 2014). A technical check of the questionnaire resulted in three additional items which were written by the team to encompass time for research in workloads (Bai et al. 2013), value of research compared to other teaching and administrative tasks, and expertise for research. The team consisted of two industrial psychologists with experience of measuring other types of organizational culture, Vice Principal for Research & Community Engagement, the Head of Research Strategy & Policy, the Dean of the Graduate School and Research Information Systems & Data Manager.

The preliminary questionnaire consisted of 37 research culture questions, as well as three demographic items (gender, ethnicity, and school/research centre). A 1–5 Likert scale of agreement was selected for its prior application in surveys (e.g. University of Glasgow) and ease of use (strongly disagree, disagree, neither agree nor disagree, agree, strongly agree), with a Not Applicable option to accommodate the range of research roles. There were also two questions rating the university and school research culture (Very Poor, Poor, Neither Poor nor Good, Good, Very Good). Given the challenges associated with quantitative research evaluation measures, two qualitative open questions were included asking participants to write three terms to describe university and school research culture. This section was based on the prior use of word clouds to examine complex issues (e.g. Wellcome Trust 2020; Morin et al. 2022).

4.2 Pilot study survey method

Online Surveys (Jisc 2020) was used to develop a web-based survey. This software was selected for its GDPR compliance, development for research purposes, and ease of use. Ethical review and approval were provided by the University’s research office. The survey took ∼10–15 min to complete.

As the online survey was a pilot study, it was decided that it in this instance, only research active staff would be surveyed. This consisted of academic staff engaged in research activities, namely research assistants, research fellows, lecturers, readers, professors and heads of school. As data about research student perceptions of research culture were already available (internal PRES), this group were not included. Many other staff contribute to the research culture (e.g. research administrators), however, to include these roles would have required different survey question items. To support the response rate of the pilot survey by maintaining questionnaire brevity, these roles were not invited to take part. Following the pilot study, the research culture questionnaire was adapted and re-run specifically for professional support staff to provide a broader understanding of research culture at the university, including the different challenges that staff experience.

The survey was anonymous with only minimal demographic information requested with the option to not answer. Given the anticipated small sample sizes of research active staff per School, role type was not included in the survey to minimize the possibility of being able to identify an individual, and thus to support response rate. Links to the online survey were disseminated via research active staff email distribution lists and supplemented by emails to Heads of School and Research Leads to maximize survey distribution. The survey was carried out in November 2021.

4.3 Survey sample

The sample for analysis consisted of 177 research active staff from across 12 units. All 11 schools from across the university took part in the survey, (namely schools of business; art; social studies; computing; creative and cultural business; engineering; health sciences; law; nursing, midwifery and paramedic practice; pharmacy; architecture), as well as a specialist research centre. Whilst research capacity varies across the schools, ∼266 invites were distributed giving an overall 70% response rate (mean school response rate 64%). Of those who took part, 83 (46.9%) were female, 83 (46.9%) were male, and 11 (6.2%) preferring not to say. Only 10 respondents did not indicate which school they worked in, which might reflect a concern that there would be a loss of anonymity in schools where there are small numbers of research active staff.

4.4 Statistical analyses

Following the reverse coding of the two negatively worded items (Q9 and Q31), Quantile-Quantile plots were conducted with the 37 items, indicating that the data were normally distributed. No significant differences in gender or ethnicity across the research culture items were found, so the entire sample was used in the analysis. To examine the theoretical factor structure of the research culture questionnaire, factor analysis was conducted on the data from the 37 scale items survey data.

Exploratory factor analysis was conducted to identify a preliminary factor structure. Items that are grouped together in factors are presumed to be measuring the same underlying construct. Typically, between 5 and 10 participants are required for each item (Kass and Tinsley 1979) to provide a sufficient sample size for such analysis (i.e. above 185 participants for the current survey). Whilst Mundfrom, Shaw and Ke (2005) recommend that a sample size of between 150 and 180 is sufficient for a good agreement where there is a variables-to-factors ratio of 5 and high-wide communality, it is recognized that the current sample size is small for factor analysis. As this was intended as an initial examination and that there is not an existing factor structure within the extant literature, Exploratory Factor Analysis (EFA) was deemed a suitable preliminary approach. Analysis was conducted using IBM’s SPSS 26.

Due to the preliminary nature of the questionnaire, various criteria were evaluated to exclude unsuitable items stepwise from the item set. The Kaiser-Meyer-Olkin (KMO) coefficient (Kaiser 1970) (as a measure of sampling adequacy) of 0.885 indicates that there is a good level of common variance between the survey items. Together with a significant Bartlett (1951) test of sphericity (X² = 3,555.342, df = 666, P < 0.001), these show that the data are appropriate for conducting EFA. Reviewing the diagonal elements of the anti-image correlation matrix, all values were above the minimum 0.5 indicating that no variables needed to be initially excluded (Hogarty et al. 2005). As any identified factors will likely correlate based on their theoretical similarity and owing to the interrelationships between the items, as seen in the correlation matrix, an oblique rotation was selected for the exploratory factor analysis.

Factor loads after rotation were assessed together with the number of items that loaded onto a factor with the following conditions: a load of > 0.5 was generally aimed for (Watkins 2021) but loads of >0.3 were still accepted, if the factor had at least one other item loading with > 0.5. Typically, a factor requires a minimum of three items for statistical identification of a valid factor (Watkins 2021), however as this was an exploration of a preliminary question set with data from only one university, factors that loaded onto only two items were retained for future examination. Where there were multiple loads of an item, it was assigned the highest load, if the highest load exceeded the next smaller load by at least 0.050, unless there was a theoretical rationale for an alternative factor. With these criteria and having performed the factor analysis, items were identified for removal. Each item was removed individually, and the analysis repeated. Where items achieved improvement, these were permanently removed and the analysis re-run until the criteria were met.

Additionally, Cronbach’s alpha scores (Cronbach 1951) were calculated for each factor to assess internal reliability. Because of the variability of research capacity across the schools, an appropriate outcome measure was not identified, meaning that criterion validity could not be evaluated at this initial stage.

5. Results

Based on the Kaiser-Guttman criterion of retaining factors of an eigen value of >1 and the Cattell screen plot, an initial solution of eight factors was extracted. Both oblique rotations of Direct Oblimin and Promax (Kappa = 4) were used, with Promax producing a better model fit in terms of factor loadings and number of items per factor. One item was excluded (Q9) as it produced a single-item factor. The final eight-factor solution (see Table 2) showed the best model fit to the data and accounted for 69.17% of the variance. All factor loadings > 0.4, apart from four items loading below 0.4 but over 0.3 (Q15, Q18, Q21, and Q32). A full cross-loading matrix is given in Supplementary Table S1. Whilst a more stringent threshold could be applied, a lower threshold was used as it this was a preliminary evaluation of the factor structure using a small sample from one university. These four items were retained, pending stronger evidence for rejection. The internal reliability (Cronbach α) scores for the eight factors ranged from 0.71 to 0.89, which were at an acceptable level (Cortina 1993). Definitions of the eight factors are given in Table 3. The mean scores for the items are shown in Supplementary Table S2.

As can be seen from Table 2, the strongest factor was labelled School Research Value(F1) capturing perceptions of school leadership support for research. This was followed by University Research Value (F2). The emergence of two separate factors for this aspect of research culture may indicate a differentiation in how School and University leaders are perceived in this regard. For example, the mean factor scores are slightly higher for School Research Value than for University Research Value. Notably both Q15 and Q18 relating to the School and University valuing research quality over quantity had factor loading < 0.4, and cross-loaded with other factors (see Supplementary Files).

Research Support (F3) related to time for research and professional development, career prospects, and support for grant applications. Items relating to having time for career development and satisfaction with research career prospects were rated less favourably (Q22 and Q23), particularly regarding having a workload that gives sufficient time for research (Q24). Factor 4, labelled Research Knowledge (F4), covered self-reported research skill level and expertise.

Collaboration (F5) relates to researcher collegiality, support and mentorship. Wellbeing & Inclusivity (F6) had four items covering perceived support by Schools and the university for promoting diversity, equality and inclusion in research, as well as wellbeing. The item asking about school support for wellbeing (Q32) loaded > 0.4, cross-loading with School Research Value (F1) and Research Support (F3). As this was an initial examination, the item was included in F6 as it had a solid rationale, however further testing may provide evidence for alternative factors.

Open Research (F7) and Research Integrity (F8) captured the extent to which each is supported by the School and University. Whilst Open Research and Research Integrity have acceptable factor loadings and alpha scores, they only consist of two items each. Furthermore, it is interesting to note that items relating to open access and research integrity (Q1-Q8) loaded across three factors (Research Knowledge, Open Research and Research Integrity), suggesting that they were capturing different underlying elements relating to individual knowledge about these topics, and the perceived university and school support for them. However, as there is evidence to suggest that these elements are linked to research culture, it is suggested that they are retained for future testing.

The inclusion of the qualitative descriptive questions was valuable for providing additional insights into the perceived research culture. A wide range of terms were used to describe research culture: The most frequently used words for the University research culture were ‘undervalued’, ‘siloed’, ‘exclusive’ ‘unsupported’ and ‘mixed’. The most frequently used items for describing the School research culture were ‘supportive’, ‘developing’, ‘undervalued’ and ‘collaborative’. A total of 82 respondents provided comments. These typically concerned the need to improve research culture, feeling unmotivated, challenges with workload and career progression and feeling undervalued, as well as reports of positive, collaborative research project experiences.

6. Discussion

It is critical that the workforce and management within higher education institutions work together to create a positive and supportive research environment. To address this challenge, a relatively short (37 item) research culture questionnaire was developed and piloted with research active staff across one university. The response rate of ∼ 70% indicated the willingness of staff to share their views, signifying the value of such an exercise. Whilst the questionnaire will require further testing and refinement, the initial psychometric analysis provides a preliminary factor structure with an indication of internal reliability. Given the lack of a widely adopted framework of research culture, the study also outlines an empirically grounded preliminary framework of research culture, consisting of eight dimensions. It is intended that this approach places the focus on the people who are involved in research, and the organizations that they work in, at the forefront of improving research culture.

Research culture may be understood as the values, norms and behaviours associated with conducting research in a university, capturing how it feels to do research in that work environment. This is reflected in the preliminary eight research culture dimensions in the questionnaire. Research culture is driven by leadership vision which can be communicated in the way that research is valued at the School and University levels through support for research (e.g. time, resources, career development). Collegiality, mentorship and support from peers may also capture what it feels like to conduct research in a work environment, influencing collaboration. Research knowledge as well as support from open access and research integrity will also impact on the way that research is conducted at a university. It is likely that these factors are linked, interacting with each other (e.g. support for research may translate into time and space for research, facilitating collaboration).

Keeping in mind the often-limited resources available and the brief completion time, the research culture questionnaire offers a practical, time efficient means of providing individual universities with a baseline for improvement (e.g. compared to large interview studies). Furthermore, a standardized research culture measurement process feeding into research evaluation activities, may have wider implications for a range of organizations, funders, and policy makers, particularly with regards to national evaluation activities.

There is ongoing discussion as to how research culture may be measured at a national level, as part of the UK REF2029 People, Culture & Environment elements (UKRI 2023; Research Excellence Framework 2024; Vitae 2024). Given the variation in institution size and research capacity, it may be challenging to develop a suitable size-fits-all approach. However, identifying and tracking the underlying factors that support a positive research culture, such as those included in the questionnaire, when combined with a wider narrative may be valuable. For example, this may entail having an accepted measure of research culture, available to institutions to adapt to their different contexts and needs. How it is used and adapted could be captured to allow for refinement with a suite of supporting materials hosted on a central hub(s). Such an approach may allow for easier sharing of best practices, particularly for universities that do not have large budgets for research culture activities. Rather than instil a sense of competitiveness, it would be intended to facilitate sustainable research culture journeys.

6.1 Research culture factors

Given the variation across the literature regarding the underlying factors that influence research culture, the current study address this through the identification of a preliminary set of eight research culture dimensions. These were: School Research Value, University Research Value, Research Support, Research Knowledge, Collaboration, Wellbeing & Inclusivity, Open Research and Research Integrity. It is acknowledged that survey was conducted in one university with a relatively small sample size, thus the factor structure should be interpreted with caution. Nonetheless, it does provide a possible framework from which the questionnaire can be refined and tested. Furthermore, it contributes to the literature by offering a more comprehensive perspective of the underlying dimensions driving research culture. This can be used to develop a shared understanding of research culture, not only for scholars in this research area but also managers and sector stakeholders tasked with improving it. The preliminary questionnaire, consisting of the eight factors, can be used to capture a wider range of experiences of research culture across disciplines, institutions and locations, creating a more inclusive approach to research culture.

In comparison to the prior research culture questionnaire factor structures, the current scale has a wider range of factors, likely the result the larger number of items and topics included. For instance, Borders, Wester and Gonzalez’s (2018) research culture scale of 18 items loaded onto a single factor. However, there are similarities with Whitford et al.’s (2005) R&D culture index (18 items) with three factors comprising of personal skills and aptitude towards R&D, working environment facilitatory towards R&D, and organizational infrastructure encouraging R&D. Consequently, the preliminary questionnaire, and eight-dimension framework, when compared to existing questionnaires allows for a more comprehensive understanding of research culture (e.g. inclusion of additional topics such as leadership). Yet, as these questionnaires were used to inform the design of the current research culture questionnaire, it is unsurprising that there are similar factors emerging. As much of the available research culture questionnaires (see Table 1) have not been subjected to psychometric analysis, a comparison of factor structure is not feasible.

School Research Value and University Research Value were the strongest factors, representing perceptions of leadership support for research at the School and University levels respectively. Whilst it might be anticipated that school and university leadership could load into one Research Value factor, two separate dimensions were produced. Keeping in mind the variation of research culture across disciplines (Müller and de Rijcke 2017; Casci and Adams 2020), this may result in school level micro-cultures. Likewise, items relating to rewards and recognition loaded onto the School and University Research Value factors rather than a separate factor [e.g. Hogan and Coote’s (2014) employee recognition and appreciation in organizational innovation culture]. A similar structure can be found in the research integrity climate scale which also separates perceptions of organizational climate in the overall university setting and, in the individual’s own department or division (Wells et al. 2014). These two factors may relate to a greater sense of belonging to a school or department, impacting on motivation (Horodnic and Zaiţ 2015), such as is identified for students (Levisohn 2021; Pedler, Willis and Nieuwoudt 2022). Further testing of the questionnaire in other contexts (e.g. research focused universities) may provide insight into these two factors.

The grouping of survey items relating to Research Support reflects prior research and surveys. For example, it was found access to support training and career development was a key factor in positive research culture in R&I in healthcare (Whitford et al. 2005). Echoing earlier findings, the survey identified concerns regarding career development opportunities and workloads impacting on research time (Kenny and Fluck 2017; Acton et al. 2019). The importance of providing support for career development, sufficient time for research and workload as well as practical support are illustrated in this factor. Given the importance of mentoring (Christian et al. 2021), future iterations of the questionnaire may benefit from an additional item specifically examining access to mentoring for career progression for not only academic staff but also doctoral students.

Research Knowledge captures self-reported research skills and awareness of key research topics, such as publication ratings, authorship practices, and impact. These are not typical cultural elements but were included because of their use in prior surveys (e.g. University of Glasgow), allowing for the identification of areas for improvement. Items that related to open access and research integrity, loaded across Research Knowledge, Open Research and Research Integrity factors. The way that these items loaded suggests that they are capturing different underlying elements relating to individual knowledge about these topics, and the perceived university and school support for them. However, these are subjective measures of knowledge and consequently it is important to acknowledge overconfidence bias. Given the issues regarding reproducibility of published findings in several research areas (e.g. Sørensen et al. 2021), it may be beneficial to test these items further or develop alternative items that better capture these underlying aspects of research culture (e.g. Martinson, Thrush and Lauren Crain 2013; Wells et al. 2014).

6.2 Wider context

The research culture questionnaire data tell a familiar story, reflecting the challenges that many universities face across the UK and beyond. Given the increasing workloads, pressure to publish, reduced funding, and job insecurity, researchers can be forgiven for feeling undervalued or unmotivated (see Tynan and Garbett 2007; Ion and Castro Ceacero 2017; Wellcome Trust 2020). As with research impact, to effectively integrate research culture an organization needs to be convinced of the value of building a positive research culture (Blundo-Canto et al. 2019). The results regarding valuing research quality over quantity, reflect broader concerns that pressure to publish can lead to research quantity being rewarded over quality (Sahel 2011; Wellcome Trust 2020). The underlying factors driving the poorer scores for career development and career satisfaction in this context may not be clear, but it does echo the wider dissatisfaction with research careers in academia (Gerwin 2022). With many universities engaging in activities to support research culture, and ongoing wider debate regarding research evaluation (e.g. Smit and Hessels 2021; Feenstra and Delgado López-Cózar 2023), this study emphasizes an organizational approach that recognizes the different experiences that researchers have by providing a common language and framework. Given the limited resources available for such activities, this questionnaire offers a practical approach to direct interventions at an institutional level and school/department level (e.g. providing brief school specific research culture reports that can be used to direct local change). As University policies and procedures may be perceived to act as a barrier to a positive research culture, there is much that individual universities, funders and policy makers can do to support a positive research culture.

6.3 Limitations and future research

The preliminary research culture questionnaire provides a practical and sustainable means of tracking research culture within individual universities and research institutions. However, it is acknowledged that survey was conducted in one university with a limited sample size, thus the factor structure should be interpreted with caution. Furthermore, there are issues with loading levels and cross loading of items, as discussed above. Nonetheless, it does provide a framework from which the questionnaire can be refined and tested. Other universities are invited to test the suitability of the questionnaire items and factor structure and to share results. Given the difficulty of observing values and norms, future iterations of the survey would benefit from the items being re-phrased to be more observable.

Responsible research assessment and metrics have become a key issue over the last decade (Curry, Gadd and Wilsdon 2022; Peruginelli and Pölönen 2024), with the perils associated with quantitative metrics well documented (Lind 2019; Pinar and Unlu 2020). Qualitative metrics may not be able to fully capture complex issues such as research culture, with the potential for results to be skewed. However, given the limited research culture measurement tools available, the current survey is intended as a move towards understand how people experience research culture rather than using proxy measures (Curry, Gadd and Wilsdon 2022). These issues are likely to become increasingly pertinent with the potential introduction of the People, Environment and Culture section in the UK’s next REF cycle (Research Excellence Framework 2024; Vitae 2024). Whilst the survey data can provide a valuable snapshot into organizational research culture, it is recommended that it is supplemented with qualitative data to provide a richer understanding of staffs’ experiences (e.g. such as the qualitative questions included in the survey, workshops, or focus groups). Careful consideration should be given to how elements associated with research culture are captured and measured.

As this was a pilot survey, only research active academic staff were surveyed and treated as a single group. Given that a wide range of people contribute to research culture across an institution, this limits the generalizability of the results. Whilst it is likely that there will be overlap in terms of the research culture factors, it may be valuable to develop specific survey items for particular research community groups (e.g. professional support staff or post-graduate research students). For example, items relating to an ‘us versus them’ mindset may be suitable for professional support staff [Association of Research Managers and Administration (ARMA) 2020], or inclusion in the research community for research students (Brew, Boud and Malfroy 2017). Development of such items would benefit from input from these groups. It may also be beneficial to include items capturing individuals’ experiences of wellbeing and inclusion (Wellcome Trust 2020), as well as psychological safety (Kashif et al. 2022). Ongoing research has refined the research culture questionnaire, including developing group-specific items to for key groups (academics, professional support staff, and post-graduate research students, as well leadership) to provide a richer, more accurate snapshot of research culture across the university. Criterion related validity was not evaluated but it is worth considering suitable outcome measures for this purpose.

7. Conclusion

There are strong arguments for the need to improve research culture however, there does not appear to be an empirically grounded and widely accepted research culture questionnaire available. In response to this challenge, a relatively short questionnaire was developed and piloted in one university. Whilst the questionnaire will require further testing and refinement, the initial psychometric analysis provides a preliminary eight factor structure with an indication of internal reliability. Individual universities and institutional groups are invited test the questionnaire items and factor structure, adapting it to their own needs. Looking forward it may be valuable to emphasize the plurality of research cultures that exist between disciplines, within departments, and across institutions, as this will inform effective change. In the everyday work of universities, improving research culture can seem like another task or a ‘nice to have’. Yet, research culture, and all the people who work in it, are the foundation from which we move forward as a society, making improvement imperative.

Acknowledgements

We would like to thank all of those who gave their time to take part in the online survey.

Supplementary data

Supplementary data are available at Research Evaluation Journal online.

Funding

This research was supported by the Scottish Funding Council. The funding organization had no influence on the study design, and was not involved in the collection, analysis, or interpretation of data; in the writing of the report; and in the decision to submit the article for publication. There are no further conflicts of interest or competing interests to disclose from any of the authors of the present review paper.

References