In social science research, the same data often leads to different conclusions

Close to 500 independent analysts re-examined the data used in published studies and they often came to different conclusions. This is the key finding of a large-scale international joint study examining the robustness of social and behavioural sciences in terms of open science. The study was carried out by 457 independent analysts from all over the world, including Ariadna Angulo-Brunet, coordinator of the Measuring and Improving Student Success (MISS) research group at the Universitat Oberta de Catalunya (UOC), and the results have now been published in a series of articles in the journal Nature.

Specifically, the project in which Angulo-Brunet, member of the Faculty of Psychology and Education Sciences, took part re-examined the data from 100 published studies to see how the researchers' methodological decisions affected their final outcomes. It revealed that, despite starting with the same dataset and the same research question, there was analytical variability – significant freedom when it comes to cleaning data, defining variables or choosing statistical models – between researcher, which could clearly affect the final conclusions.

The findings of the study were striking: in 81% of cases, the analysts obtained different statistical results from the original study as a result of choosing different methodological approaches. As for the conclusions, if the strictest criteria are applied, only one in three studies (34% to be precise) can be considered fully robust on the basis that the five people who reviewed them agreed with the original conclusions. A different approach, however, yields a higher percentage, suggesting a more optimistic outlook: of the 504 reviews, 74% reached the same conclusion as the original study.

A paradox that supports the value of science

The study shows that the differences found are not due to a lack of expertise or to personal biases but to freedom of choice between equally valid statistical procedures. "Understanding this diversity helps us understand that the methods used aren't the end goal but the path that leads to it: they're the tools that enable us to move forward with confidence towards the true purpose, which is to advance knowledge," said Angulo-Brunet.

"These contradictory results are very encouraging, because they suggest that, when rigorous methods are used, objectivity prevails over professional etiquette," she said. Angulo-Brunet, who is a member of the UOC-FuturEd research centre, explained that the project clearly illustrates that there is no single right way to conduct research or analyse the world around us. "Using cooking as an analogy, the study shows that there's no single recipe for research. Even if each cook adds their own personal touch, if the dish is really good and no mistakes are made, the final outcome should always be an excellent meal. However, if a research outcome can only be achieved when a particular cook creates a dish with a strange mix of ingredients, it means that the discovery was less reliable than we thought."

Commitment to the multiverse analysis method

The findings of the study do not seek to challenge the validity of previous research but to highlight the risk of overconfidence from presenting a single analysis as absolute truth. The research team behind the study suggest moving towards a multiverse analysis model in which various sets of possible analyses are explored to establish whether the findings still stand or are weak. The idea is for either a single researcher to subject the available data to the full set of possible analyses or for a group of people working together to review whether the findings still stand regardless of the path chosen.

Amid the current growth of artificial intelligence and interdisciplinarity, Angulo-Brunet, who has been actively involved in the re-examination of the studies, believes in a pedagogical, rather than a punitive, approach to scientific methods. The aim is not to restrict researchers' freedom but to encourage full transparency through open science. "Methodology isn't a secondary concern: it lies at the heart of research."

Towards greater transparency in science

According to Angulo-Brunet, promoting greater transparency should not be seen as a bureaucratic burden but as the only way to ensure scientific integrity in a system that often rewards immediacy over rigour. Conscious that AI is starting to make its way into data analysis, experts stress that human judgement and methodological sensitivity are still essential when it comes to ensuring the reliability of scientific results. "Science will move forward when we care as much about how we reach the conclusion as we do about the conclusion itself. We must be able to recognize the limitations of each discipline to create synergies."

Commitment to open science

The UOC has taken part in this large international metascientific study as part of its commitment to the principles of open science. The project is part of the SCORE (Systematizing Confidence in Open Research and Evidence) programme, a collaborative initiative involving 865 researchers from all over the world that has led to the publication of a series of articles in the journal Nature. The project, coordinated by the Center for Open Science, has explored aspects of research such as credibility, reproducibility and robustness, and assessed the extent to which it is possible to predict whether results will be replicated. The researchers taking part in the SCORE project examined statements from a total of 3,900 articles published between 2009 and 2018 in 62 journals in a range of disciplines and established standard definitions for the key concepts used to ensure a consistent interpretation of results.

This study on research methods has an impact on all of the UOC's research missions and contributes to the UN Sustainable Development Goals (SDGs), and goals 4, Quality Education; 9, Industry, Innovation and Infrastructure; 16, Peace, Justice and Strong Institutions; and 17, Partnerships for the Goals, in particular.

Reference article

Aczel et al. (2026). Investigating the analytic robustness of the social and behavioural sciences. Nature. https://doi.org/10.1038/s41586-025-09844-9

Transformative, impactful research

At the UOC, we see research as a strategic tool to advance towards a future society that is more critical, responsible and nonconformist. With this vision, we conduct applied research that's interdisciplinary and linked to the most important social, technological and educational challenges.

The UOC’s over 500 researchers and more than 50 research groups are working in five research units focusing on five missions: lifelong learning; ethical and human-centred technology; digital transition and sustainability; culture for a critical society, and digital health and planetary well-being.

The university's Hubbik platform fosters knowledge transfer and entrepreneurship in the UOC community.

More information: www.uoc.edu/en/research