Funders, governments and institutions across the globe are increasingly expected to demonstrate the societal impact of research. They need to answer questions such as: has investment in research and experimental development (R&D) yielded tangible benefits to individuals, communities, organizations and economies across diverse areas such as health, climate change and technological advancement? Despite this growing demand, currently there is no established way to measure this. To meet this challenge, the Institute for Scientific Information (ISI)™ has developed a framework to evaluate the societal impact of research in a responsible manner and introduces Societal Impact Profiles™.
Societal Impact Profiles will be included in our forthcoming AI-powered Web of Science Research Intelligence™ platform — empowering research institutions, funders and governments to better measure and showcase the impact of their research investments.
Our framework addresses the three primary challenges in measuring the societal impact of research:
Challenge 1. Acknowledging the diversity of societal needs
Societal needs are complex and the diversity of societal benefits does not conform to a simple or standardized scale. To reflect this heterogeneity, we have separated societal needs into eight facets: Political & Policy, Legal & Governance, Economic, Human Capital, Medical, Social & Cultural, Technological and Environmental.
Challenge 2. Beyond bibliometrics: balancing quantitative and qualitative approaches
Comprehensive evaluation of societal impact must go beyond traditional bibliometrics and incorporate additional indicators that account for a more diverse range of outputs and activities. Our framework leverages our extensive resources of enriched data which goes beyond scholarly output and extends to our data on patents, clinical trials, policy documents, media publications, data sets, educational outputs, funding data and much more, as well as external data sources where necessary.
Quantitative methods, such as traditional scientometric approaches, are valuable as they offer an easily scalable, objective view. However, they lack the depth required to measure specific facets of societal impact. To address this limitation qualitative methods, such as expert review, user case studies or narrative impact statements, are needed. However, these approaches are less scalable and risk being subjective.
We have taken a balanced approach that integrates both quantitative and qualitative methods as we believe this is critical for a responsible and comprehensive evaluation of the societal impact of research.
Challenge 3. Navigating delays: leveraging lagging and leading indicators for effective assessment
To mitigate the challenge of lengthy delays between research being conducted and any resultant societal benefits, our evaluation framework contains both lagging and leading indicators.
Lagging indicators offer insights into societal impact by retrospectively analyzing past outputs and activities that have had sufficient time to manifest their effects. They reveal how previous research efforts have contributed to societal outcomes, providing a picture of observed, tangible impact.
In contrast, leading indicators are forward-looking and analyze more recent signals that suggest the potential for research outputs and activities to create societal impact in the future, though without any guarantee.
Analysis of an institution’s performance by leading versus lagging indicators, can guide actions to either refine or adjust the research strategy to achieve better outcomes. They can help to determine whether the right teams are in place and working on the most promising activities and depending on the insights gained, this may lead to a course correction or a renewed focus – to enhance the likelihood of achieving a desired outcome.
Visualizing societal impact for informed decision making
To facilitate responsible evaluation, impact data must be presented in a format that is both meaningful and accessible to multiple, diverse users. These data must transparently reflect diverse types and levels of impactful activity, while supporting timely decision-making for optimal resource allocation. Let us look at an example of how societal impact will be displayed in the forthcoming Web of Science Research Intelligence platform, using a Societal Impact Profile.
Fig: Societal Impact Profile of an institution
The societal impact of an institution is visualized using a radar chart containing eight axes – one for each of the eight societal facets. To facilitate effective comparison, each point on the Societal Impact Profile is calculated by comparing a given institution’s performance against that of nearly 12,000 other organizations globally. Each facet has two points, one calculated using metrics from the lagging indicator groups and the other using metrics from the leading indicator groups. The underlying individual metrics will be available to support transparency and more granular analyses.
Societal Impact Profiles can be used to examine an institution’s performance as well as to benchmark their performance against other institutions. Exploration of competitive strengths and relative weaknesses compared to peer institutions helps to prioritize what to promote and also to identify more successful practices for possible implementation.
Our ongoing journey: future enhancements to our framework
We plan to share our progress regularly as it evolves, and we welcome feedback on both the framework itself and its applications. Community feedback will play an important role in shaping and refining this framework to deliver a solution that not only meets current expectations but is also adaptable to meet future needs.
Read our ISI report A responsible framework for evaluating the societal impact of research for a fuller explanation.
