What is academics’ responsibility in dealing with indicators?

by Maximilian Fochler and Sarah De Rijcke

Source: "Piled Higher and Deeper" by Jorge Cham www.phdcomics.com
Source: “Piled Higher and Deeper” by Jorge Cham www.phdcomics.com

The metric tide is in. The use of quantitative indicators for the evaluation of the productivity and quality of the work of individual researchers, groups and institutions has become ubiquitous in contemporary research. Knowing which indicators are used and how they represent one’s work has become key for junior and senior academics alike.

Proponents of the use of metrics in research evaluation hope that it will provide a more objective and comparable way of assessing quality, one that is less vulnerable to the biases and problems often ascribed to qualitative peer-review based approaches.

However, critical research in science and technology studies and elsewhere increasingly points to considerable negative effects of the impeding dominance of quantitative assessment. In fact, indicator systems might serve as an infrastructure fueling hyper-competition, with all its problematic social and epistemic effects. They create incentives for researchers to orient their work to where high metric impact might be expected, thus potentially fostering mainstreaming at the expense of epistemic diversity, and prioritizing delivery over discovery.

Over recent years, many important initiatives have pushed for a more responsible use of metrics in research evaluation. The DORA declaration, the Leiden manifesto and the Metric Tide report are just the most prominent examples of discussions in academia and in the institutions that govern it. The recommendations of these initiatives have mostly focused on those actors which seem to have most bearing on the processes of concern: academic institutions, the professional communities providing the methods and data metrics build on, as well as evaluators.

But what about individual researchers? What is their responsibility in dealing with indicators in their everyday practices in research? Twenty years ago, when the metric tide was still but a trickle, the eminent anthropologist Marilyn Strathern (1997) wrote “Auditors are not aliens: they are a version of ourselves.” (p. 319). Still today, it would be simplistic and wrong to assume that researchers are merely victims to bureaucratic auditors imposing indicators on them.

Don’t we all strategically use those metric representations of our work we see as advantageous for whichever goals we are currently pursuing? Do metric logics structure the way we present ourselves in our profiles on academic social networks, and how we look at others’ portfolios? Isn’t there a secret joy in watching one’s citation scores and performance metrics grow? In how far do we individually play along the logics that we might criticize as a more collective phenomenon in our more reflexive moments? Is this a problem? If so, should finding ethical ways of dealing with indicators not be part and parcel of being a responsible researcher today?

These are the core questions of a recent debate “Implicated in the Indicator Game?” we edited for the journal Engaging Science Technology and Society. This debate gathers essays of a cast of junior and senior scholars in science and technology studies (STS). STS is an interesting context for discussing these wider questions, because scholars in this field have contributed particularly strongly to the critical discourse on indicators. Still, in their own careers and institutional practices, they often have to decide how to play the indicator game – for not playing it seldom seems a viable option.

In one essay in this collection, Ruth Müller asks, quoting an informant from her own fieldwork: “Do you think that the structure of a scientific career is such that it tends to make you forget why you’re doing the science?”. Diagnosing a loss of meaning in running to fulfill quantitative indicators, she points to aspects of work in science and technology studies which are indispensable for quality, but hardly to be expressed in indicators – interdisciplinary engagement with the sciences and engineering being the most important example for STS.

So, what can individual researchers and institutions do? Our collection contains many different answers to this question. All agree, however, that ignoring or boycotting indicators cannot be the solution. As Alan Irwin reminds us, the questions of accountability that indicators are supposed to answer will not go away. They need to be answered in different terms, by offering and celebrating new non-reductive concepts of the quality of research in different fields. For individual researchers, this calls for confidence to stand up for the quality also of those aspects of their work that cannot be well expressed in metrics, but also to recognize these qualities in others’ work.

As an outcome of our debate, we offer the concept of evaluative inquiry as a starting point for a more responsible dealing with indicators. In a nutshell, evaluative inquiries may present research work numerically, verbally, and/or visually – but aim to do so in ways which do justice to the complexity of actual practice and its engagements, rather than to reduce for the sake of standardization. They also do not jump to a reductive understanding of what counts in an assessment (such as publications), and aim to produce and represent the multiple meanings and purposes of researchers’ work. They are processual in the sense that the choice of criteria and of whether or not certain indicators make sense cannot be fully described in advance, but needs to be negotiated in the process of evaluating.

Of course this all sounds nice in theory. But it will require researchers to engage in these practices, rather than in hunting metric satisfaction. And it will require institutional actors to engage in more substantive discourses about the quality of research.

References

Strathern, M. (1997). ‘Improving ratings’: audit in the British University
system. European Review 5(03), 305—321.


Maximilian Fochler is assistant professor and head of the Department of Science and Technology Studies of the University of Vienna, Austria. His main current research interests are forms of knowledge production at the interface of science and other societal domains (such as the economy), as well as the impact of new forms of governing science on academic knowledge production. He has also published on the relations between technosciences and their publics as well as on publics’ engagement with science.

Sarah de Rijcke is associate professor and deputy director at the Centre for Science and Technology Studies (CWTS) of Leiden University, the Netherlands. She leads a research group that focuses on a) developing a theoretical framework on the politics of contemporary research governance; b) gaining a deep empirical understanding of how formal and informal evaluation practices are re-shaping academic knowledge production; c) contributing to shaping contemporary debates on responsible research evaluation and metrics uses (including policy implications).

Too bad to fail?

by Dorothea Born

"Piled Higher and Deeper" by Jorge Cham www.phdcomics.com
Source: “Piled Higher and Deeper” by Jorge Cham www.phdcomics.com

The other day I discussed the difficulties of living and working in academia with a very successful former professor of mine. When it came to his own career, he made an interesting confession: “It was pure chance that I ended up doing what I am doing now,” he said, “After I graduated from high-school, I tried out several jobs and studies, until I found my place. These early years, I always leave them out in my CV.” This made me wonder about the role of CVs in academic practice and careers.

Read More

Performing Moral Stories through Bodyweight

by Michael Penkler

In June, the United States Food and Drug Administration has approved a new weight-loss device: AspireAssist. The device is surgically inserted into the abdomen and allows patients who have failed at losing weight by other means to drain ingested food from the stomach. After eating, users go to the toilet, plug in a tubing set into a tube that leads to the stomach, and “aspirate” (or, less prosaically, pump) up to 30% of their meal into the toilet.

penkler-grafikSource: FDA

Read More

A reflection on engaging with public engagement with science

by Robin Rae

“And what can I do here?” people ask me curiously one after another, eyeballing a mountain bike standing upright in front of a computer screen. I am in the lecture room of the Department for Science and Technology Studies (STS), University of Vienna, which is filled with people, technological objects, and further installations about RFID chips, artificial intelligence, and visions of reproductive medicine and self-driving cars. It is a Friday night in April 2016, the so-called “Lange Nacht der Forschung” (i.e. Long Night of Research). This nation-wide biannual science communication event invites diverse publics to interactively explore current research at more than 250 institutions. With its interactive installations, the STS department aimed to spark discussions about how technologies affect and shape society, bodies, everyday lives, and futures. While that only partially explains the bike standing in the room, read on to learn how challenges in planning my installation contributed to its realization.

Read More

When mass communication turns into mass surveillance

by Pouya Sepehr, Maresa Barbara Wolkenstein, Helene Sorgner and Marilen Hennebach

New technologies have given governments an unprecedented means to access personal information. In order to ensure that all people can seek information and express themselves freely, there must be reasonable checks and balances on governments’ ability to access, collect, and store individuals’ data. Both security and freedom can be protected, but only through balanced laws and policies that uphold human rights. Surveillance happens at many levels: It can be eavesdropping programmes of foreign and local governments, it can be commercial corporations on a global scope, it can be more or less institutionalised and it has many different aspects, reaching from self-censorship to pleasure, from activism to fatalism. The question, though, is not so much if we mind but rather how and when we mind.

The revelation of NSA documents through Edward Snowden in 2013 had brought otherwise secret intelligence activities into the light of global attention. It has been shocking for many to realise that mass surveillance technologies are targeting civilian communication, including social media platforms. In fact, the era of mass communication has become the era of mass surveillance and hence, the question of personal freedom of expression has gained a technological dimension. The revelations have also shown that national security agencies have strong ties with giant tech companies which are willingly cooperating in giving access to information, proving that even civilians have “nowhere to hide” anymore.

mass surveillance seminar Read More

Society in the making: quantification and accountability

by Andreas Schadauer

Sites
© Schadauer 2016

“The top 10% of Austrian households own 61% of all real estate assets.” For a certain time, this statistical argument could be read in several newspapers, was taken for granted by some journalists and commentators, and was used as a strong argument for inheritance and wealth taxes. But how did this statistical argument get accepted, persistent and influential? Who or what was able and enabled to produce it? And who or what is accountable for this statistical argument?

For the last question, the answers provided by the textbooks of empirical research I read as student of Sociology at the University of Vienna are quite clear-cut. If produced methodologically correct, numbers and statistics represent reality objectively (e.g. Diekmann, 2007: 23f) and due to this have authority, superiority and are politically neutral (Kreutz, 2009: 3). This notion stands in stark contrast to approaches in STS which point out the social, political and institutional quality of scientific methods (e.g. Desrosières, 2002; Kenney, 2015; Law, 2010).

Read More

“Frugal Innovation” – an inquiry into a blind spot in STS

by Lisa Sigl

Frugal Innovation
source

Concepts and notions of innovation are societal and political battlegrounds. How strongly they imply imaginations about responsibilities between science and society becomes apparent when comparing notions of innovation in different socio-political contexts. In European policy contexts, innovation is mostly defined as technological innovation for the market, insinuating that its primary responsibility is to secure competitiveness, economic growth and jobs. With this market-orientation in mind, the vehemence with which the Indian government promotes “frugal innovation” as “sharply” contrasting with this “conventional approach” in its current Five Year Plan is striking. The absence of critical reflection in STS on this remarkable innovation concept is so striking that I want to open a discussion here.

Read More

Exploring new questions of “Gender, justice and the political economy of the cross-border fertility industry”. A wrap-up and outlook of an international workshop.

by Daniela Schuh

Gender, justice and the political economy of the cross-border fertility industry
The workshop was organized by Kathrin Braun (Univ. of Vienna), Gesine Fuchs (Hochschule Luzern) and Daniela Schuh (Univ. of Vienna).

While cross-border fertility travel has become an expanding industry, knowledge about its actual scope, structure, regulation and practices is still  sparse. A workshop in April organized by members of the University of Vienna and the Hochschule Luzern met this situation head on by bringing together a diverse program. Scholars from all across Europe and with diverse scientific and institutional backgrounds came together to collectively explore vital questions about the cross-border fertility industry:

How is this industry stratified in terms of gender, ethnicity, race, class, able-bodiedness, and further axes of inequality? How does the rise of cross-border fertility industry and/or corresponding state policies affect gender relations? How can we assess these policies and developments from a gender and social justice perspective? And how should we understand and engage with this industry in the first place?

Read More

Scratching the proverbial itch – with Alan Irwin

by Joanne Heng

Joanne and Alan Irwin
Joanne and Alan having a chat in the STS library.

In the semester just passed, our department had the pleasure of hosting Alan Irwin. While he needs little introduction to those in the field, for those less versed in STS, Alan’s illustrious career spans over thirty years in which he has written extensively about scientific governance, risk & decision-making, policy and public engagement of science. Currently a Professor at the Department of Organization at the Copenhagen Business School, he has received multiple awards for his work and is even a Knight of the Order of Dannebrog.

During his visit to our department, I took part in his seminar entitled “Governing Socio-Technical Futures. Science, democracy and innovation in the 21st century”. Touching on many of his pet topics such as public engagement with science and technology and the enactment of democracy with regards to present-future relations, I felt Alan had a remarkable ability to really open up these issues to the class by highlighting contentions, questioning the roles of key actors in shaping these issues whilst also sharing with us his past experiences and case studies – it made for a very enjoyable and thought-provoking few weeks. At the core of his course was the basic argument that, whilst current actions shape the possible futures that await us, our sense of the future(s) profoundly shapes the actions we take today. In line with some of his previous work, Alan was especially concerned with the implications of future-present relations for scientific governance – with examples such as climate change but also innovation policy very prominent here.

Read More

Can competition hurt science?

by Maximilian Fochler

Copyright: ScienceCartoonsPlus.com
Copyright: ScienceCartoonsPlus.com

Many of us privately or professionally interested in science think that healthy competition is what drives science forward. In current research policy and funding, this belief is so strong that competitiveness has become the central doctrine guiding the governance of research and of individual researchers.

But can too much competition also endanger the very aims we would like research to achieve, such as asking fundamental questions at the frontier of knowledge? This is what a group of high-profile life scientists, including a former editor of Science magazine, suggested in 2014 by warning that the current hyper-competition in the US life sciences may suppress “the creativity, cooperation, risk-taking, and original thinking required to make fundamental discoveries“ (Alberts, Kirschner, Tilghman, & Varmus, 2014, p. 5774).

But is science more competitive today than in the past, and if so, why and how? At least two main things have changed. First, in the late twentieth century, driven by rising societal expectations in the innovative capacities of research, many scientific fields have thrived and grown. The life sciences are a strong case here, but not the only example. However, this growth has happened much less in long-term institutional positions, but in a new form of the organisation of research work: project-based temporal employment. This has created a large generation of highly qualified young researchers competing for a basically stagnating or receding number of faculty positions. Second, the rise of new metric forms of keeping track and assessing scientific productivity has made competition seemingly more transparent, but also fuelled it. Impact factors, rankings and other indicator-based measurements do not only represent scientific work, they also shape it, as also Alex Rushforth and Sarah de Rijcke argue in a recent blogpost. Because they matter in competition they may also change the very nature of this work.

 

Read More