A toolkit to support transparency, reproducibility and quality in energy research

11 Feb 2020

A toolkit to support transparency, reproducibility and quality in energy research

Michael Fell and Gesche Huebner

We’re developing a toolkit to help energy researchers conduct transparent, reproducible (where appropriate) and quality research. This blog article explains why we think this is needed, introduces the toolkit, and asks you to get involved.

Energy use is key to global challenges such as climate change, while also playing an important role in the day-to-day lives of people and communities. But how sure can we be that the research done in these areas is providing reliable findings?

Unlike in fields such as medicine, energy research has not widely adopted many of the tools and practices (such as preregistration of studies/analysis, and use of reporting guidelines) needed to maximise the transparency, quality and reproducibility of studies.

This increases the risk of both inadvertent errors going unspotted (e.g. if data/code are not made available for checking), and of potentially misleading practices (such as HARKing -- hypothesising after results are known) being employed. The consequence of this is that evidence-based policy and practice may be built on shaky foundations.

Why is energy research lagging behind?

The potential reasons for this are manifold.

Energy research is very multidisciplinary; we find engineers, architects, mathematicians, economists, sociologists, anthropologists, statisticians, and others researching this field using a multitude of methods, such as interviews, focus groups, surveys, field and lab experiments, case studies, monitoring and modelling.

In addition, many of us work with commercially or personally sensitive data that are difficult to share.

Also, the energy sector is changing rapidly. Other disciplines are interested in establishing general principles whereas research in the energy area is often much more focused on the current situation, fully aware that ten years down the line, things will have changed. In fact, much of the research is aimed at helping deliver change to meet climate change targets.

Another factor is money and cost: Building an energy system model of the UK or running long field trials  easily takes several years and is very expensive.

Finally, there are a number of research areas where reproducibility, in a strict sense, might not even be an appropriate term to discuss. Many qualitative and participatory research projects focus on specific case studies where with different participants and different researchers you would expect to find different results. This is why we put the focus not just on reproducibility in a narrow sense but include transparency and quality as equally important.

In the vast majority of cases, the tools and practices that have been developed to support transparency, reproducibility and quality in other domains are applicable in the energy field. We believe they need to become more embedded in our practice as researchers, and in the teaching we deliver to future energy researchers. That is why we have started developing the TReQ Tools toolkit (TReQ = transparency, reproducibility and quality).

TReQ Tools

In TReQ Tools  we provide a practical introduction and guide to a number of simple approaches you can use: reporting guidelines, preregistration of studies and analysis plans; preprints; systematic evidence review; and open data/code. For each approach we explain why it is used, how and when in the research process, and what the benefits are to you – the researcher. We also provide links to templates and further resources.

As an example, one useful tool that appears not to be widely used in energy research is reporting guidelines. Usually accompanied by a simple checklist, they set out a best practice guide to what details you should include when you are reporting on a study (e.g. in a paper). The best known is perhaps the CONSORT checklist for randomized trials, but other include PRISMA for systematic reviews, TRIPOD for predictive modelling and COREQ for interview and focus group studies. 

Reporting guidelines may seem pretty mundane, but they present a number of benefits for researchers. They give you greater confidence that you are not forgetting to report important details of your study, and can make it easier to decide what to put in or leave out of your reports. As well as being more useful to readers, this can head off peer-reviewer criticisms (we’ve certainly reviewed enough papers where we wish the authors had used some guidelines!). They make it easier for future systematic reviews to capture useful information from your study.

It can also be useful to think about reporting guidelines while you are planning your study -- if you are going to have to report on why certain decisions were made, it helps to ensure you are giving this proper consideration at the time, rather rationalizing later on at the reporting stage.

Your field needs you

TReQ Tools is a work in progress, and we are keen to work with others across the community to develop it. Do you have ideas for other subjects or resources that should be included, or have you spotted problems that need fixing? Please get in touch with us if you would like to join as a collaborator or contributor (gesche.huebner@ucl.ac.uk and michael.fell@ucl.ac.uk). Alternatively, you can comment directly on the a Google Doc of the latest version of the toolkit.