Adriana Arellano, is the Research Director at GrupoFARO, an Ecuadorian, independent, plural, non-partisan, secular think tank. This post has been adapted from an original contribution to the TTI Exchange e-forum.

Research quality is a key concern and indicator for think tanks, one we keep enriching and trying to measure. The concept of research quality is made up of a mix of characteristics and recognised elements, including:

• Application of sound methods
• A comprehensive review of relevant literature
• Evidenced-based conclusions
• Consideration of limitations, making them explicit and avoiding bias
• Use of high quality data sets
• Relevance
• Timeliness
• Usefulness of the research products

Think tanks regularly measure or assess the quality of their research efforts through different mechanisms and quality markers:

1. Intra-organisation review processes and external peer review processes

2. Quality perception surveys

3. Number of indexed articles published

4. Research uptake by policy makers

1. Review processes: The first two mechanisms work not only as a means to measure but also to improve quality. Review processes can help a think tank gather feedback on flaws and observations on the quality of a publication.
2. Quality perception surveys: These are useful mechanisms used to assess the quality of products generated by a think tank. They provide an opportunity to understand stakeholders’ perceptions considering broader aspects of quality that include the relevance of research, timeliness and usefulness to the policy process.
3. Number of indexed articles published: This quality marker is often used by donors and think tanks to measure research quality. However, it is complicated to concentrate on this indicator, as it is more suited to the work of universities, who have a stronger focus on reaching an academic audience, rather than that of think tanks who work to reach a variety of audiences beyond academic journal readers[*].
4. Research uptake by policy makers: This indicator is difficult to assess as the policy making process is too complex. Policy development can take a lot of time, and depending on the context, in some cases think tanks will want to first raise awareness on an issue before building the grounds for a debate or the proposal of a specific policy. At other times, the process of research uptake can be indirect, through constituents or media representatives. One way to more closely evaluate this marker would be to measure the dissemination of research products with appropriate audiences and assess the levels of influence these have on the relevant stage of the policy-making cycle depending on the context.

When differentiating our organisations and our efforts from those of other research producers, like universities, the definition of research quality and its markers requires particular attention. The concepts of Modes of Knowledge Production (Gibbons 1998) may be useful for this purpose, with most universities falling under Mode 1 of knowledge production and most think tanks operating under Mode 2:

Many organisations may move from Mode 1 to Mode 2 depending on the type of work they are doing, their teams, the project they are working on and the issue they research. However, this framework can be used to better understand the realm of work for our organisations and help define more relevant quality markers and indicators for our research products.

[*] According to a recent study, half of academic papers are read only by their authors and journal editors,   http://www.smithsonianmag.com/smart-news/half-academic-studies-are-never-read-more-three-people-180950222/?no-ist