Science reproducibility in Brazil

In recent years a number of initiatives have been launched to test the reproducibility of published scientific data. Mostly they’re operating within specific scientific domains, like the cancer reproducibility project we reported on back in 2014. Now, speaking with Chris Smith, Brazilian researcher Olavo Amaral is going a step further and launching a reproducibility initiative that, for the first time, aims to assess performance at the level of his whole country…

Olavo - We're trying to get as much of a representative sample as we can of Brazilian biomedical science. We're from Brazil, and we're interested in how the country is doing, so we're taking a random sample of Brazilian articles from the last 20 years in the life sciences; constrained by method, because of course we have to do things that our lab network can perform, to actually repeat them in multiple labs to see how reproducible the original finding is.

Chris - How many laboratories are going to repeat each of the studies?

Olavo - We've put out an open call for laboratories within Brazil. We currently have more than 70 labs actually engaged in the initiative. The idea would be to have each results replicated in at least three laboratories, so we'll get a feeling of the intra-lab variation.

Chris - So how is this going to lead ultimately to an improvement in standards of science, both in Brazil and also internationally? In other words, what are you going to do with these results once you've got them?

Olavo - Yeah. Well we hope it will. We can promise. But I think I think first of all it's important to talk about the subject. Just actually having the project happening, I think, does make a difference in raising the debate about the problem. We've been having reasonable media exposure in Brazil, so just having a lot of people discussing this I think is already a plus. Once we have the results: one, we can get a better estimate, what is our responsibility? It is hard to compare, because it's not like we have a lot of statistics from other countries, or be like the first country to actually try to do that nationally. But I think we can also try to assess what predicts responsibility, what kind of findings are more reproducible, what findings are less. Are there features of the study of the method of the researcher that count, and can help predict what is reproducible, what is not? And I think that is actually very useful for agencies that fund science, for institutions who develop science, to try to actually have a better way to assess the literature that does not count only impact, how much a paper’s cited, but also what are the chances that this finding should be reproduced in other laboratories.

Chris - Because obviously funding is a sensitive subject in all countries, but it's especially a sensitive subject in Brazil, because recently, hasn't the government passed some kind of legislation suggesting that you're going to be on the same scientific spend annually for the next 20 years?

Olavo - It's only getting worse. We have a cap on public spending for the next 20 years, but we're actually getting cut. The funding for the Ministry of Science and Technology has just been cut around 40% a few weeks ago. Funding is about a third of what it was some years ago. So I mean, it's really unpredictable in terms of how it's going, and to us it is a very sensitive issue. On the other hand, I think it raises the question that we have to spend as wisely as possible. So I think it's important to study ways to better direct this funding.

Chris - And when's the study going to report? When are you aiming to be finished by?

Olavo - Probably in 2021. We're starting experiments in the second half of the year. There is a lot of labs, so they'll have different rhythms. Each lab is supposed to do between three and six experiments. I suppose that will happen over the course of a year, maybe, and it will probably be late, perhaps early 2021.

Chris - And given the parlous political and funding situation in Brazil at the moment, is this a securely funded initiative? As in, you're not going to get halfway through this, and then half the labs get their funding pulled, and the project never gets finished?.

Olavo - We actually have private funding for this. We're funded by the Serrapilheira Institute, which is a private funder which aims to fund science in Brazil. We do have the money already. We are not exactly sure how many experiments we can do with it, because I mean we still have to figure out some logistics and some costs, but we were pretty sure that we can do at least 60 experiments in three different labs Our aim is to get to 100. We'll have to see if our budget goes that far.