1 00:00:00,000 --> 00:00:14,595 *34C3 preroll music* 2 00:00:14,595 --> 00:00:17,870 Herald: Please give a warm welcome here. 3 00:00:17,870 --> 00:00:23,610 It’s Franziska, Teresa, and Judith. 4 00:00:23,610 --> 00:00:26,510 Judith, you have the stage, thank you. 5 00:00:26,510 --> 00:00:28,470 Judith Hartstein: Thank you, thanks! 6 00:00:28,470 --> 00:00:33,710 *applause* 7 00:00:33,710 --> 00:00:57,430 *inaudible* 8 00:00:57,430 --> 00:01:00,219 Judith: We believe that scientific performance indicators 9 00:01:00,219 --> 00:01:03,269 are widely applied to inform funding decisions and to 10 00:01:03,269 --> 00:01:08,270 determine the availability of career opportunities. So, those of you who are 11 00:01:08,270 --> 00:01:14,420 working in science or have had a look into the science system might agree to that. 12 00:01:14,420 --> 00:01:18,840 And we want to understand evaluative bibliometrics as algorithmic science 13 00:01:18,840 --> 00:01:27,640 evaluation instruments to highlight some things that do occur also with other 14 00:01:27,640 --> 00:01:38,020 algorithmic instruments of evaluation. And so we’re going to start with a quote from 15 00:01:38,020 --> 00:01:44,679 a publication in 2015 which reads “As the tyranny of bibliometrics tightens its 16 00:01:44,679 --> 00:01:49,130 grip, it is having a disastrous effect on the model of science presented to young 17 00:01:49,130 --> 00:01:58,999 researchers.” We have heard the talk of hanno already, and he’s basically also 18 00:01:58,999 --> 00:02:06,679 talking about problems in the science system and the reputation by the 19 00:02:06,679 --> 00:02:14,170 indicators. And the question is, is bibliometrics the bad guy here? If you 20 00:02:14,170 --> 00:02:18,530 speak of ‘tyranny of bibliometrics’, who is the actor doing this? Or are maybe 21 00:02:18,530 --> 00:02:24,950 bibliometricians the problem? We want to contextualize our talk into the growing 22 00:02:24,950 --> 00:02:30,440 movement of Reflexive Metrics. So those who are doing science studies, social 23 00:02:30,440 --> 00:02:34,940 studies of science, scientometrics and bibliometrics. The movement of Reflexive 24 00:02:34,940 --> 00:02:41,690 Metrics. So the basic idea is to say: “Okay, we have to accept accountability if 25 00:02:41,690 --> 00:02:45,540 we do bibliometrics and scientometrics.” We have to understand the effects of 26 00:02:45,540 --> 00:02:54,220 algorithmic evaluation on science, and we will try not to be the bad guy. And the 27 00:02:54,220 --> 00:03:03,640 main mediator of the science evaluation which is perceived by the researchers is 28 00:03:03,640 --> 00:03:09,630 the algorithm. I will hand over the microphone to… or I will not hand over the 29 00:03:09,630 --> 00:03:14,440 microphone but I will hand over the talk to Teresa. She’s going to talk about 30 00:03:14,440 --> 00:03:19,690 "Datafication of Scientific Evaluation". 31 00:03:19,690 --> 00:03:24,180 Teresa Isigkeit: Okay. I hope you can hear me. No? Yes? Okay. 32 00:03:24,180 --> 00:03:25,890 Judith: *mumbling* 33 00:03:25,890 --> 00:03:29,480 When we think about the science system what do we expect? 34 00:03:29,480 --> 00:03:33,700 What can society expect from a scientific system? 35 00:03:33,700 --> 00:03:38,200 In general, we would say reliable and truthful knowledge, 36 00:03:38,200 --> 00:03:41,810 that is scrutinized by the scientific community. 37 00:03:41,810 --> 00:03:44,320 So where can we find this knowledge? 38 00:03:44,320 --> 00:03:47,140 Normally in publications. 39 00:03:47,140 --> 00:03:51,920 So with these publications, can we actually say 40 00:03:51,920 --> 00:03:58,670 whether science is bad or good? Or is there better science than others? 41 00:03:58,670 --> 00:04:03,350 In the era of digital publication databases, 42 00:04:03,350 --> 00:04:07,280 there’s big datasets of publications. 43 00:04:07,280 --> 00:04:12,330 And these are used to evaluate and calculate 44 00:04:12,330 --> 00:04:16,840 the quality of scientific output. 45 00:04:16,840 --> 00:04:22,720 So in general, with this metadata we can tell you 46 00:04:22,720 --> 00:04:26,400 who is the author of a publication, 47 00:04:26,400 --> 00:04:30,020 where is the home institution of this author, 48 00:04:30,020 --> 00:04:37,960 or which types of citations are in the bibliographic information. 49 00:04:37,960 --> 00:04:44,890 This is used in the calculation of bibliometric indicators. 50 00:04:44,890 --> 00:04:51,580 For example if you take the journal impact factors, 51 00:04:51,580 --> 00:04:57,650 which is a citation based indicator, you can compare different journals. 52 00:04:57,650 --> 00:05:03,750 And maybe say which journals are performing better than others 53 00:05:03,750 --> 00:05:09,360 or if the journal factor has increased or decreased over the years. 54 00:05:09,360 --> 00:05:15,330 Another example would be the Hirsch-Index for individual scientists, 55 00:05:15,330 --> 00:05:22,670 which is also widely used when scientists apply for jobs. So they put 56 00:05:22,670 --> 00:05:27,710 these numbers in their CVs and supposedly this tells you something about the quality 57 00:05:27,710 --> 00:05:36,270 of research those scientists are conducting. With the availability of the 58 00:05:36,270 --> 00:05:45,590 data we can see an increase in its usage. And in a scientific environment in which 59 00:05:45,590 --> 00:05:52,260 data-driven science is established, scientific conduct decisions regarding 60 00:05:52,260 --> 00:06:04,419 hiring or funding heavily rely on these indicators. There’s maybe a naive belief 61 00:06:04,419 --> 00:06:12,020 that these indicators that are data-driven and rely on data that is collected in the 62 00:06:12,020 --> 00:06:26,610 database is a more objective metric that we can use. So here's a quote by Rieder 63 00:06:26,610 --> 00:06:32,430 and Simon: “In this brave new world trust no longer resides in the integrity of 64 00:06:32,430 --> 00:06:38,710 individual truth-tellers or the veracity of prestigious institutions, but is placed 65 00:06:38,710 --> 00:06:44,480 in highly formalized procedures enacted through disciplined self-restraint. 66 00:06:44,480 --> 00:06:53,150 Numbers cease to be supplements.” So we see a change of an evaluation system that 67 00:06:53,150 --> 00:07:00,449 is relying on expert knowledge to a system of algorithmic science evaluation. In this 68 00:07:00,449 --> 00:07:05,699 change there’s a belief in a depersonalization of the system and the 69 00:07:05,699 --> 00:07:15,090 perception of algorithms as the rule of law. So when looking at the interaction 70 00:07:15,090 --> 00:07:26,010 between the algorithm and scientists we can tell that this relationship is not as 71 00:07:26,010 --> 00:07:35,169 easy as it seems. Algorithms are not in fact objective. They carry social meaning 72 00:07:35,169 --> 00:07:43,020 and human agency. They are used to construct a reality and algorithms don’t 73 00:07:43,020 --> 00:07:48,290 come naturally. They don’t grow on trees and can be picked by scientists and people 74 00:07:48,290 --> 00:07:55,120 who evaluate the scientific system, so we have to be reflective and think about 75 00:07:55,120 --> 00:08:04,910 which social meanings the algorithm holds. So when there is a code that the algorithm 76 00:08:04,910 --> 00:08:11,169 uses, there is a subjective meaning in this code, and there is agency in this 77 00:08:11,169 --> 00:08:17,370 code, and you can’t just say, oh, this is a perfect construction of the reality of 78 00:08:17,370 --> 00:08:22,350 scientific system. So the belief that this tells you more about the quality of 79 00:08:22,350 --> 00:08:31,820 research is not a good indicator. So when you think about the example of citation 80 00:08:31,820 --> 00:08:37,049 counts the algorithm reads the bibliographic information of a publication 81 00:08:37,049 --> 00:08:47,230 from the database. So scientists, they cite papers that relate to their studies. 82 00:08:47,230 --> 00:08:55,740 But we don’t actually know which of these citations are more meaningful than others, 83 00:08:55,740 --> 00:09:01,319 so they’re not as easily comparable. But the algorithms give you the belief they 84 00:09:01,319 --> 00:09:11,230 are, so relevance is not as easily put into an algorithm and there is different 85 00:09:11,230 --> 00:09:19,199 types of citations. So the scientists perceive this use of the algorithms also 86 00:09:19,199 --> 00:09:24,790 as a powerful instrument. And so the algorithm has some sway above the 87 00:09:24,790 --> 00:09:30,019 scientists because they rely so much on those indicators to further their careers, 88 00:09:30,019 --> 00:09:38,069 to get a promotion, or get funding for their next research projects. So we have a 89 00:09:38,069 --> 00:09:42,970 reciprocal relationship between the algorithm and the scientists, and this 90 00:09:42,970 --> 00:09:52,399 creates a new construction of reality. So we can conclude that governance by 91 00:09:52,399 --> 00:09:59,019 algorithms leads to behavioral adaptation in scientists, and one of these examples 92 00:09:59,019 --> 00:10:08,069 that uses the Science Citation Index will be given from Franziska. 93 00:10:08,069 --> 00:10:12,610 Franziska Sörgel: Thanks for the handover! Yes, let me start. 94 00:10:12,610 --> 00:10:16,070 I’m focusing on reputation and authorship as you can see 95 00:10:16,070 --> 00:10:21,420 on the slide, and first let me start with a quote 96 00:10:21,420 --> 00:10:27,490 by Eugene Garfield, which says: “Is it reasonable to assume that if I cite a 97 00:10:27,490 --> 00:10:32,889 paper that I would probably be interested in those papers which subsequently cite it 98 00:10:32,889 --> 00:10:38,630 as well as my own paper. Indeed, I have observed on several occasions that people 99 00:10:38,630 --> 00:10:45,399 preferred to cite the articles I had cited rather than cite me! It would seem to me 100 00:10:45,399 --> 00:10:51,321 that this is the basis for the building up of the ‘logical network’ for the Citation 101 00:10:51,321 --> 00:11:01,709 Index service.” So, actually, this Science Citation Index which is described here was 102 00:11:01,709 --> 00:11:07,970 mainly developed in order to solve the problems of information retrieval. Eugene 103 00:11:07,970 --> 00:11:16,259 Garfield, also founder of this Science Citation Index – short: SCI – noted or 104 00:11:16,259 --> 00:11:22,170 began to note a huge interest in reciprocal publication behavior. He 105 00:11:22,170 --> 00:11:27,470 recognized the increasing interest as a strategic instrument to exploit 106 00:11:27,470 --> 00:11:33,410 intellectual property. And indeed, the interest in the SCI – and its data – 107 00:11:33,410 --> 00:11:39,189 successively became more relevant within the disciplines, and its usage extended. 108 00:11:39,189 --> 00:11:45,800 Later, [Derek J.] de Solla Price, another social scientist, asked or claimed for a 109 00:11:45,800 --> 00:11:53,029 better research on the topic, as it currently also meant a crisis in science, 110 00:11:53,029 --> 00:11:59,329 and stated: “If a paper was cited once, it would get cited again and 111 00:11:59,329 --> 00:12:05,089 again, so the main problem was that the rich would get richer”, which is also 112 00:12:05,089 --> 00:12:12,029 known as the “Matthew Effect”. Finally, the SCI and its use turned into a system 113 00:12:12,029 --> 00:12:17,860 which was and still is used as a reciprocal citation system, and became a 114 00:12:17,860 --> 00:12:25,119 central and global actor. Once a paper was cited, the probability it was cited again 115 00:12:25,119 --> 00:12:30,720 was higher, and it would even extend its own influence on a certain topic within 116 00:12:30,720 --> 00:12:38,369 the scientific field. So it was known that you would either read a certain article 117 00:12:38,369 --> 00:12:48,999 and people would do research on a certain topic or subject. So this phenomenon would 118 00:12:48,999 --> 00:12:58,569 rise to an instrument of disciplining science and created power structures. 119 00:12:58,569 --> 00:13:04,530 Let me show you one example which is closely connected to this phenomenon 120 00:13:04,530 --> 00:13:10,920 I just told you about – and I don’t know if here in this room there are any 121 00:13:10,920 --> 00:13:18,680 astronomers or physicists? Yeah, there are few, okay. 122 00:13:18,680 --> 00:13:24,580 That’s great, actually. So in the next slide, here, 123 00:13:24,580 --> 00:13:32,990 we have a table with a time window from 2010 to 2016, and social 124 00:13:32,990 --> 00:13:41,069 scientists from Berlin found out that the co-authorship within the field of physics 125 00:13:41,069 --> 00:13:50,519 extended by 58 on a yearly basis in this time window. So this is actually already 126 00:13:50,519 --> 00:13:55,920 very high, but they also found another very extreme case. They found one paper 127 00:13:55,920 --> 00:14:07,399 which had roundabout 7,000 words and the mentioned authorship of 5,000. So, in 128 00:14:07,399 --> 00:14:15,389 average, the contribution of each scientist or researcher of this paper who 129 00:14:15,389 --> 00:14:28,910 was mentioned was 1.1 word. Sounds strange, yeah. And so of course you have 130 00:14:28,910 --> 00:14:34,519 to see this in a certain context, and maybe we can talk about this later on, 131 00:14:34,519 --> 00:14:41,380 because it has to do with Atlas particle detector, which requires high maintenance 132 00:14:41,380 --> 00:14:46,019 and stuff. But still, the number of authorship, and you can see this 133 00:14:46,019 --> 00:14:52,519 regardless which scientific field we are talking about, generally increased the 134 00:14:52,519 --> 00:15:05,360 last years. It remains a problem especially for the reputation, obviously. 135 00:15:05,360 --> 00:15:12,399 It remains a problem that there is such high pressure on nowadays researchers. 136 00:15:12,399 --> 00:15:20,200 Still, of course, we have ethics and research requires standards of 137 00:15:20,200 --> 00:15:25,689 responsibility. And for example there’s one, there’s other ones, but there’s one 138 00:15:25,689 --> 00:15:30,850 here on the slide: the “Australian Code for the Responsible Conduct of Research” 139 00:15:30,850 --> 00:15:36,689 which says: “The right to authorship is not tied to position or profession and 140 00:15:36,689 --> 00:15:41,060 does not depend on whether the contribution was paid for or voluntary. 141 00:15:41,060 --> 00:15:46,409 It is not enough to have provided materials or routine technical support, 142 00:15:46,419 --> 00:15:50,969 or to have made the measurements on which the publication is based. 143 00:15:50,969 --> 00:15:55,129 Substantial intellectual involvement is required.” 144 00:15:55,129 --> 00:16:03,259 So yeah, this is, could be one rule to work with or to work by, to follow. 145 00:16:03,259 --> 00:16:08,079 And still we have this problem of reputation which remains, 146 00:16:08,079 --> 00:16:11,480 and where I hand over to Judith again. 147 00:16:11,480 --> 00:16:19,739 Judith: Thank you. So we’re going to speak about strategic citation now. So if you 148 00:16:19,739 --> 00:16:29,540 put this point of reputation like that, you may say: So the researcher does find 149 00:16:29,540 --> 00:16:36,239 something in his research, his or her research, and addresses the publication 150 00:16:36,239 --> 00:16:40,049 describing it to the community. And the community, the scientific community 151 00:16:40,049 --> 00:16:46,269 rewards the researcher with reputation. And now the algorithm, which is like 152 00:16:46,269 --> 00:16:54,730 perceived to be a new thing, is mediating the visibility of the researcher’s results 153 00:16:54,730 --> 00:17:00,720 to the community, and is also mediating the rewards – the career opportunities or 154 00:17:00,720 --> 00:17:04,919 the funding decisions etc. And what happens now and what is plausible to 155 00:17:04,919 --> 00:17:10,398 happen is that the researcher addresses his or her research also to the algorithm 156 00:17:10,398 --> 00:17:21,039 in terms of citing those who are evaluated by the algorithm, who he wants to support, 157 00:17:21,039 --> 00:17:29,230 and also in terms of strategic keywording etc. And that’s the only thing which 158 00:17:29,230 --> 00:17:33,809 happens new, might be a perspective on that. So the one thing new: the algorithm 159 00:17:33,809 --> 00:17:40,700 is addressed as a recipient of scientific publications. And it is like far-fetched 160 00:17:40,700 --> 00:17:46,260 to discriminate between so-called and ‘visible colleges’ and ‘citation cartels’. 161 00:17:46,260 --> 00:17:50,899 What do I mean by that? So ‘invisible colleges’ is a term to say: “Okay, people 162 00:17:50,899 --> 00:17:56,490 are citing each other. They do not work together in a co-working space, maybe, but 163 00:17:56,490 --> 00:18:00,820 they do research on the same topic.” And that’s only plausible that they cite each 164 00:18:00,820 --> 00:18:06,540 other. And if we look at citation networks and find people citing each other, that 165 00:18:06,540 --> 00:18:12,860 does not have necessarily to be something bad. And we also have people who are 166 00:18:12,860 --> 00:18:18,519 concerned that there might be like ‘citation cartels’. So researchers citing 167 00:18:18,519 --> 00:18:27,270 each other not for purposes like the research topics are closely connected, but 168 00:18:27,270 --> 00:18:35,720 to support each other in their career prospects. And people do try to 169 00:18:35,720 --> 00:18:40,730 discriminate those invisible colleges from citation cartels ex post from looking at 170 00:18:40,730 --> 00:18:45,990 metadata networks of publication and find that a problem. And we have a discourse on 171 00:18:45,990 --> 00:18:58,299 that in the bibliometrics community. I will show you some short quotes how people 172 00:18:58,299 --> 00:19:04,510 talk about those citation cartels. So e.g. Davis in 2012 said: “George Franck warned 173 00:19:04,510 --> 00:19:08,960 us on the possibility of citation cartels – groups of editors and journals working 174 00:19:08,960 --> 00:19:13,590 together for mutual benefit.” So we have heard about their journal impact factors, 175 00:19:13,590 --> 00:19:23,020 so they... it’s believed that editors talk to each other: “Hey you cite my journal, 176 00:19:23,020 --> 00:19:27,479 I cite your journal, and we both will boost our impact factors.” 177 00:19:27,479 --> 00:19:32,899 So we have people trying to detect those cartels, 178 00:19:32,899 --> 00:19:37,299 and Mongeon et al. wrote that: “We have little knowledge 179 00:19:37,299 --> 00:19:40,639 about the phenomenon itself and about where to draw the line between 180 00:19:40,639 --> 00:19:46,080 acceptable and unacceptable behavior.” So we are having like moral discussions, 181 00:19:46,080 --> 00:19:53,560 about research ethics. And also we find discussions about the fairness of the 182 00:19:53,560 --> 00:19:58,461 impact factors. So Yang et al. wrote: “Disingenuously manipulating impact factor 183 00:19:58,461 --> 00:20:02,700 is the significant way to harm the fairness of the impact factor.” And that’s 184 00:20:02,700 --> 00:20:09,780 a very interesting thing I think, because why should an indicator be fair? So the... 185 00:20:09,780 --> 00:20:15,580 To believe that we have a fair measurement of scientific quality relevance and rigor 186 00:20:15,580 --> 00:20:21,809 in one single like number, like their journal impact factor, is not a small 187 00:20:21,809 --> 00:20:29,669 thing to say. And also we have a call for detection and punishment. So Davis also 188 00:20:29,669 --> 00:20:33,780 wrote: “If disciplinary norms and decorum cannot keep this kind of behavior at bay, 189 00:20:33,780 --> 00:20:39,700 the threat of being delisted from the JCR may be necessary.” So we find the moral 190 00:20:39,700 --> 00:20:44,480 concerns on right and wrong. We find the evocation of the fairness of indicators 191 00:20:44,480 --> 00:20:51,070 and we find the call for detection and punishment. When I first heard about that 192 00:20:51,070 --> 00:20:56,640 phenomenon of citation cartels which is believed to exist, I had something in mind 193 00:20:56,640 --> 00:21:03,740 which sounded... or it sounded like familiar to me. Because we have a similar 194 00:21:03,740 --> 00:21:11,130 information retrieval discourse or a discourse about ranking and power in a 195 00:21:11,130 --> 00:21:19,159 different area of society: in search engine optimization. So I found a quote by 196 00:21:19,159 --> 00:21:26,830 Page et al., who developed the PageRank algorithm – Google’s ranking algorithm – 197 00:21:26,830 --> 00:21:32,990 in 1999, which has changed since that a lot. But they wrote also a paper about the 198 00:21:32,990 --> 00:21:42,500 social implications of the information retrieval by the PageRank as an indicator. 199 00:21:42,500 --> 00:21:45,909 And wrote that: “These types of personalized PageRanks are virtually 200 00:21:45,909 --> 00:21:50,150 immune to manipulation by commercial interests. ... For example fast updating 201 00:21:50,150 --> 00:21:54,350 of documents is a very desirable feature, but it is abused by people who want to 202 00:21:54,350 --> 00:22:00,529 manipulate the results of the search engine.” And that was important to me to 203 00:22:00,529 --> 00:22:09,350 read because we also have like a narration of abuse, of manipulation, the perception 204 00:22:09,350 --> 00:22:14,210 that that might be fair, so we have a fair indicator and people try to betray it. 205 00:22:14,210 --> 00:22:21,790 And then we had in the early 2000s, I recall having a private website 206 00:22:21,790 --> 00:22:24,560 with a public guest book and getting link spam from people 207 00:22:24,560 --> 00:22:26,719 who wanted to boost their Google PageRanks, 208 00:22:26,719 --> 00:22:32,779 and shortly afterwards Google decided to punish link spam in their 209 00:22:32,779 --> 00:22:37,860 ranking algorithm. And then I got lots of emails of people saying: “Please delete my 210 00:22:37,860 --> 00:22:43,890 post from your guestbook because Google’s going to punish me for that.” We may say 211 00:22:43,890 --> 00:22:51,509 that this search engine optimization discussion is now somehow settled and it’s 212 00:22:51,509 --> 00:22:57,940 accepted that Google's ranking is useful. They have a secret algorithm, but it works 213 00:22:57,940 --> 00:23:05,250 and that is why it’s widely used. Although that journal impact factor seems to be 214 00:23:05,250 --> 00:23:13,159 transparent it’s basically the same thing that it's accepted to be useful and thus 215 00:23:13,159 --> 00:23:17,410 it's widely used. So the journal impact factor, the SCI and the like. We have 216 00:23:17,410 --> 00:23:24,570 another analogy so that Google decides which SEO behavior is regarded acceptable 217 00:23:24,570 --> 00:23:28,480 and punishes those who act against the rules and thus holds an enormous amount of 218 00:23:28,480 --> 00:23:38,669 power, which has lots of implications and led to the spreading of content management 219 00:23:38,669 --> 00:23:44,910 systems, for example, with search engine optimization plugins etc. We also have 220 00:23:44,910 --> 00:23:52,809 this power concentration in the hands of Clarivate (formerly ThomsonReuters) who 221 00:23:52,809 --> 00:23:59,059 host the database for the general impact factor. And they decide on who’s going to 222 00:23:59,059 --> 00:24:05,140 be indexed in those journal citation records and how is the algorithm, in 223 00:24:05,140 --> 00:24:12,200 detail, implemented in their databases. So we have this power concentration there 224 00:24:12,200 --> 00:24:21,780 too, and I think if we think about this analogy we might come to interesting 225 00:24:21,780 --> 00:24:29,770 thoughts but our time is running out so we are going to give a take-home message. 226 00:24:29,770 --> 00:24:34,720 Tl;dr, we find that the scientific community reacts with codes of conduct to 227 00:24:34,720 --> 00:24:39,929 a problem which is believed to exist. The strategic citation – we have database 228 00:24:39,929 --> 00:24:44,830 providers which react with sanctions so people are delisted from the journal 229 00:24:44,830 --> 00:24:50,240 citation records or journals are delisted from the journal citation records to 230 00:24:50,240 --> 00:24:55,419 punish them for citation stacking. And we have researchers and publishers who adapt 231 00:24:55,419 --> 00:25:05,159 their publication strategies in reaction to this perceived algorithmic power. But 232 00:25:05,159 --> 00:25:12,470 if we want to understand this as a problem we don’t have to only react to the 233 00:25:12,470 --> 00:25:19,179 algorithm but we have to address the power structures. Who holds these instruments in 234 00:25:19,179 --> 00:25:24,210 in their hands? If we talk about bibliometrics as an instrument and we 235 00:25:24,210 --> 00:25:27,830 should not only blame the algorithm – so #dontblamethealgorithm. 236 00:25:27,830 --> 00:25:33,200 Thank you very much! *applause* 237 00:25:37,979 --> 00:25:43,630 Herald: Thank you to Franziska, Teresa and Judith, or in the reverse order. 238 00:25:44,510 --> 00:25:48,450 Thank you for shining a light on how science is actually seen 239 00:25:48,450 --> 00:25:50,800 in its publications. 240 00:25:50,800 --> 00:25:52,410 As I started off as well, 241 00:25:52,410 --> 00:25:55,650 it’s more about scratching each other a little bit. 242 00:25:55,650 --> 00:25:57,790 I have some questions here from the audience. 243 00:25:57,790 --> 00:25:59,740 This is Microphone 2, please! 244 00:25:59,740 --> 00:26:05,319 Mic2: Yes, thank you for this interesting talk. I have a question. You may be 245 00:26:05,319 --> 00:26:09,549 familiar with the term ‘measurement dysfunction’, that if you provide a worker 246 00:26:09,549 --> 00:26:14,220 with an incentive to do a good job based on some kind of metric then the worker 247 00:26:14,220 --> 00:26:19,960 will start optimizing for the metric instead of trying to do a good job, and 248 00:26:19,960 --> 00:26:26,200 this is kind of inevitable. So, don’t you see that maybe it could be treating the 249 00:26:26,200 --> 00:26:32,540 symptoms if we just react about code of conduct, tweaking algorithms or addressing 250 00:26:32,540 --> 00:26:37,200 power structures. But instead we need to remove the incentives that lead to this 251 00:26:37,200 --> 00:26:43,669 measurement dysfunction. Judith: I would refer to this phenomenon 252 00:26:43,669 --> 00:26:50,740 as “perverse learning” – learning for the grades you get but not for your intrinsic 253 00:26:50,740 --> 00:27:00,800 motivation to learn something. We observe that in the science system. But if we only 254 00:27:00,800 --> 00:27:10,440 adapt the algorithm, so take away the incentives, would be like you wouldn’t 255 00:27:10,440 --> 00:27:20,480 want to evaluate research at all which you can probably want to do. But to whom would 256 00:27:20,480 --> 00:27:32,740 you address this call or this demand, so “please do not have indicators” or… I give 257 00:27:32,740 --> 00:27:38,690 the question back to you. *laughs* Herald: Okay, questions from the audience 258 00:27:38,690 --> 00:27:46,499 out there on the Internet, please. Your mic is not working? Okay, then I go to 259 00:27:46,499 --> 00:27:52,130 Microphone 1, please Sir. Mic1: Yeah, I want to have a provocative 260 00:27:52,130 --> 00:27:56,690 thesis. I think the fundamental problem is not how these things are gamed but the 261 00:27:56,690 --> 00:28:01,200 fundamental problem is that if we think the impact factor is a useful measurement 262 00:28:01,200 --> 00:28:04,599 for the quality of science. Because I think it’s just not. 263 00:28:04,599 --> 00:28:06,950 *applause* 264 00:28:09,580 --> 00:28:12,110 Judith: Ahm.. I.. Mic 1: I guess that was obvious... 265 00:28:12,110 --> 00:28:13,490 Judith: Yeah, I would not say 266 00:28:13,490 --> 00:28:17,630 that the journal impact factor is a measurement of scientific quality 267 00:28:17,630 --> 00:28:23,519 because no one has like a definition of scientific quality. 268 00:28:23,519 --> 00:28:28,450 So what I can observe is only people believe this journal impact factor 269 00:28:28,450 --> 00:28:36,649 to reflect some quality. Maybe they are chasing a ghost but I… 270 00:28:36,649 --> 00:28:41,979 whether that’s a valid measure is not so important to me, 271 00:28:41,979 --> 00:28:45,239 even if it were a relevant or a valid measure, 272 00:28:45,239 --> 00:28:52,309 it would concern me how it affects science. 273 00:28:52,989 --> 00:28:56,499 Herald: Okay, question from Microphone 3 there. Please. 274 00:28:56,499 --> 00:28:59,499 Mic3: Thanks for the interesting talk. I have a question about 275 00:28:59,499 --> 00:29:04,400 the 5,000 authors paper. Was that same paper published 276 00:29:04,400 --> 00:29:09,449 five thousand times or was it one paper with ten page title page? 277 00:29:10,049 --> 00:29:15,259 Franziska: No, it was one paper ... 278 00:29:15,259 --> 00:29:19,599 ... counting more than 7,000 words. And the authorship, 279 00:29:19,599 --> 00:29:24,460 so authors and co-authors, were more than 5,000. 280 00:29:24,460 --> 00:29:31,010 Mic3: Isn’t it obvious that this is a fake? 281 00:29:31,010 --> 00:29:35,070 Franziska: Well that’s what I meant earlier 282 00:29:35,070 --> 00:29:44,049 when saying, you have to see this within its context. So physicists are working 283 00:29:44,049 --> 00:29:52,169 with this with Atlas, this detective system. As there were some physicists in 284 00:29:52,169 --> 00:30:01,780 the audience they probably do know how this works. I do not. But as they claim 285 00:30:01,780 --> 00:30:08,330 it’s so much work to work with this, and it, as I said, requires so high 286 00:30:08,330 --> 00:30:19,120 maintenance it’s... They obviously have yeah... 287 00:30:19,120 --> 00:30:21,719 Mic3: So everybody who contributed was listed? 288 00:30:21,719 --> 00:30:28,809 Judith: Exactly, that’s it. And if this is ethically correct or not, well, this is 289 00:30:28,809 --> 00:30:33,820 something which needs to be discussed, right? This is why we have this talk, as 290 00:30:33,820 --> 00:30:39,910 we want to make this transparent, and contribute it to an open discussion. 291 00:30:39,910 --> 00:30:44,610 Herald: Okay, I’m sorry guys. I have to cut off here because our emission out 292 00:30:44,610 --> 00:30:49,229 there in space is coming to an end. I suggest that you guys 293 00:30:49,229 --> 00:30:52,879 find each other somewhere, maybe in the tea house or... 294 00:30:52,879 --> 00:30:55,019 Judith: Sure. We are around, we are here. 295 00:30:55,019 --> 00:30:58,149 Herald: You are around. I would love to have lots of applause for these ladies, 296 00:30:58,149 --> 00:31:03,030 for it really lights on how these algorithms 297 00:31:03,030 --> 00:31:05,299 not or are working. Thank you very much! 298 00:31:05,299 --> 00:31:06,639 Judith: Thank you! 299 00:31:06,639 --> 00:31:21,629 *postroll music* 300 00:31:21,629 --> 00:31:26,741 *subtitles created by c3subtitles.de in the year 2018*