DCL Learning Series

Marianne Calilhanna

Welcome to the DCL Learning Series. We are very excited to share this story today with you on the conference that wasn't. We have a very special guest here and I'm excited to join, jump right into it. So today, we have our two panelists. If you could click the next slide please. Amanda Sulicz. Amanda is the Manager of Research Integrity at IEEE, and my colleague David Turner. David is DCL's Digital Transformation and Content Technologies Consultant. So there's so much to talk about today. This is quite a mystery, espionage and all the elements of a blockbuster movie. So David, Amanda, over to you. 

David Turner

Yeah. Thanks so much, Marianne. And Amanda, it is great to have you. I have been looking forward to this all week. All right, so just for everybody in terms of setting this up, there's been a lot of talk, a lot of things written in the industry along the lines of how today's speed and the pressure to publish has possibly, let's just say, encouraged some shady practices. And this sometimes leads to things like a failure of authors and editors to note articles or note things like retracted articles or other problematic research. And we hear tales of paper mills, their threat to research integrity.

But personally, I know for me, and I think probably for a lot of people, I've limited that to the realm of journal articles, and that's really not the only place. What we've been discovered is there's also this growing threat in the realm of conferences and conference proceedings, and in some fields, that's probably even more important than journals. So that's what our guest is here to talk about today, her adventures in conference land. So Amanda, let's dive in here. Tell me a little bit, I guess to start, just about how you're structured, how you fit in, the work that you're doing, and what you have to do with conferences that weren't, or wasn't or whatever.

Amanda Sulicz

That aren't? Yes. So just a quick little background about IEEE. So we are the Institute of Electrical and Electronics Engineers, which by being an institute, that means we're a society of societies. So we're made up of all different fields of electrical engineering. We have communications, antennas and propagations, we have power, we have electron devices, we have computers, we have everything. And as you already mentioned, David, some of these fields are actually conference proceedings papers are the journal currency. I have a PhD in chemistry, and back when I was getting my PhD, I was like "Conference papers, what's that? Nobody cares about that." But apparently, as I learned since working here, that's actually very important. So we've been seeing a transition in the last several years from journals having issues with their, what we call special issues here at IEEE. Some other publishers may call them a topical collection, but that misconduct is now coming into conferences. So one of the things about IEEE is we are a volunteer-led organization. We have over 500,000 members around the world,

4:00

and those members are responsible for helping us run our publishing, everything at IEEE. They're very dedicated and great, and one of the things that I've closely been working on since starting here for the last three years is an ad hoc committee that is working between our technical activities. Well, sorry, it started as technical activities, but now it's just with our conferences group, and then also with publications, which is where I work. And initially, this ad hoc committee came to be because of the growing concerns of misconduct in conference quality. And we've been focusing on a myriad of problems, but through this conference, this ad hoc was really instrumental in helping to detect and eventually retract this entire conference proceedings. They've also been really helpful in getting us to improve some of our policies and make updates there. One of the big policy updates we made with the help of this ad hoc, which in hindsight was a little silly that we didn't have this, but at least now we have it, but it's actually requiring the peer review information for conferences. That was never a requirement we had.

And as you'll later learn in this presentation, that actually was really instrumental in helping us identify this as a problematic conference. And they've also helped us to improve our educational modules for new conference organizers because sometimes you just might not know and it's important that you're taught this. And then also we've been working to work on our pre-screening procedures to make sure that we catch problematic content at submission rather than post-publication. And now we go to the 2024 International Conference on Optimization Computing and Wireless Communication, or ICOCWC, as you'll see it referred to throughout this presentation.

David Turner

All right. So I understand you've got this committee, and they're looking for things. Specifically with this conference, and sorry, I feel like I should call you Dr. Sulicz now that I know about this PhD. 

Amanda Sulicz

Oh, it's fine. 

David Turner

Specifically with this conference, what was it that drew your attention? What were the red flags that, how'd you pay attention to this? 

Amanda Sulicz

So the thing that first drew our attention, it actually came from a routine plagiarism complaint. So we have a publishing ethics alias, which I'll share that at the end of this presentation. IEEE also has a ethics reporting line where you can anonymously submit complaints. So this came in, and somebody had a complaint that an article published in this conference proceedings was plagiarized, or plagiarized other work. And as part of a plagiarism investigation, we have to look at, we get the Data that's in our, I guess one of our databases that's just who sponsored the conference, who was the conference chair, how was the conference held? Was it an in-person, a hybrid, online only? It has all of that kind of information there. And then I also have to go into Xplore, which is our publishing platform, and get the paper so that I can send it to the volunteers to review the complaint.

As I was looking at this, I go into our database to get the information so I can contact the organizers and the sponsors of the conference to let them know there's this investigation coming. And I noticed that it was a hybrid conference and I take note of where it is. And then as I was going into Xplore to download the paper so I could run a similarity check on it, I just was poking around Xplore, and I noticed that the conference was supposed to have taken place

8:00

at a University in Ethiopia. Every single author affiliation that was listed on Xplore, nobody was from that University, nobody was from Ethiopia, and nobody was from the continent of Africa. Everybody's affiliation was either in India or China, so that was a very like a "Wait, wait, what happened here?" kind of moment. And then I started to dig around more in it. Eventually, we realized that the conference, simultaneously, this ad hoc has been working to clean up our publication record.

So we've been using heavily the Problematic Paper Screener, which is a tool that was developed by Guillaume Cabanac, who is a sleuth out of France. He developed this tool and it actually screens for a variety of different things. You can look at papers that have cited retracted papers. It's called feet of clay, meaning that it's not built on a solid foundation. You can see if they've used Mathgen or SCIgen, which is just AI-created math essentially, or just completely fake papers. But this Problematic Paper Screener also was useful. I think that the way that that tool works is somebody either has to flag it to them or they have to find it on their own to upload it into the tool. It's not like an ambient screening type of thing.

So at the time when we got this plagiarism complaint, the ad hoc was aware of this conference as part of the tortured phrase detection we were using. But it was lower on our radar because I think at the time they had flagged maybe 20 articles, so we were going on priority of who had the highest amount. But then when we started to look at the papers, we realized that there was a lot more going on. And we looked at every single paper in that conference, which was 314 articles.

David Turner

Oh, my.

Amanda Sulicz

Yes. It was a long road of review. And we realized that the papers were just not – I don't know, I can't say for certain that they were all AI generated, but they looked to be AI generated. The math didn't fit in the paper when they would have equations. It had nothing to do with what they were talking about. Just poor quality overall speaking. So that led us to decide. The ad hoc said, "Let's retract the entire proceedings." And that's when we had more fun when we started to contact the authors and organizers.

David Turner

Oh, yeah, because well, so if I understand correctly, you started looking at this thing, and it was relatively routine at the beginning, but everything just started to come apart as you peeled the onion here. The conference location didn't match. None of the authors came from that University. The affiliations, things started to look really weird in the different papers, and it just all got – well, you started running it through these tools. What was the name of the tool again?

Amanda Sulicz

The Problematic Paper Screener? 

David Turner

Yeah. 

Amanda Sulicz

Okay. 

David Turner

All right, so then you have to obviously start contacting the authors and I'm guessing the organizers as well. So talk to me a little bit about that. Take us through the process of reaching out to an author and getting their take on this.

Amanda Sulicz

Right. So we follow the NISO CREC recommendations when we retract an article, which I believe COPE has also adopted, which is the Committee on Publication Ethics, which many people on this call are probably aware of. So as part of that process, anytime we're retracting an article, we contact the authors of the article, assuming we have their information available. Sometimes we try and it bounces back or something.

12:00

So we contacted the authors from every single one of these articles. We received responses on about 121 of them, so almost half of the authors responded, or half of the authors on papers responded. And as we started to go through these, I was noticing that I kept seeing the same thing over and over, which, these authors don't have any apparent connection with one each other. They're not all from the same University to where they might be sitting around having lunch together and say, "Hey, what did you write for this?" So 80% of the 121 responses we received were one of these two templates, and it was to a T, everything, the same bold that you see here. They would change their paper title, they would catch that. But it was very interesting as we were going through this. And it got even more interesting as I was starting to collect these because some of these authors actually left template text behind.

So I don't know who was sharing this template with them, but this person – so the subject line, we ask when we send the retraction, we say "Please respond to this email to state your agreement, disagreement, et cetera." And a lot of people were actually responding back with a different subject line, and they would say disagreement with retraction and paper reference, and then they put the DOI. So this person actually had left this, I guess, instruction set, I'll call it. And I think my favorite thing of all of this was we actually had an author respond to a retraction that they weren't actually an author on.

David Turner

Oh, great. 

Amanda Sulicz

I don't know how, and I triple checked. I even went back and I'm like "Did I accidentally contact this person?" No. Somehow, they found out about somebody else's paper being retracted, but they disagreed with it also. They didn't want the retraction. So it was interesting. And so initially, we don't usually contact the organizers of a conference. If it's more than a year out, we'll deal with the society that sponsored the conference instead. So in this particular case, it was over a year since the conference was held. So we didn't actually contact the organizers that were listed in that database I mentioned earlier, but they found out about all of these retractions and they contacted us.

David Turner

Oh.

Amanda Sulicz

And I was like "Okay." And this was actually the entire email that I had. I didn't truncate anything in this screenshot. They didn't tell us who they were. They just signed it, the organizers, that was all they said. And it comes from this anonymous, just, Gmail here. It's not coming from amandasulicz at Gmail or something. And we were like "What? Since you're contacting us, tell me who you are. And also, give me your peer review Data. Let's see what's going on here." So we asked for it in three business days. They ended up giving it to us in nine days. And I went through, and I looked at all of the peer review Data they gave. So there was 314 papers. And then there was – the thing that I noticed initially that was weird is every single paper had exactly three reviews. That is quite an anomaly in general to even find three reviewers to agree to review all of those papers.

IEEE's policy is that you have to have two, so they went above and beyond here and got three. And I hope that everyone in the audience here can hear the sarcasm when I say they went above and beyond

16:00

to get three reviewers because we're going to find out that there weren't three reviewers. But none of the articles suggested rejection or none of the reviewers suggested rejection. Reviewer responses also missed the start or ends of words as if somebody had copied and pasted them in. So if you were saying "This paper should be recommended," it would say lowercase H, his paper should be recommended. Or the end of the sentence would be slightly truncated. There was also, I don't even know how I managed to catch this, this was just dumb luck on my part, but there was at least one example where the content of the review had nothing to do with the topic of the paper.

So an example of that would be if I was talking about my favorite types of pie, and then I mentioned that you could serve pie a la mode, but then the reviewer report started talking about different flavors of ice cream, and had nothing to do with pie. So it had caught a piece of what the article was talking about, but it wasn't actually what the article was about. And then also, when the organizers responded back to say who they were, they weren't anybody that was listed on the organizing committee, and they were in fact authors of one of the papers that was being retracted. So again, don't know how they got that information. 

David Turner

Right, so you had very questionable responses from these authors, number one. And so then, not even asking for it, you got responses from the organizers, and their initial response was kind of questionable, so you asked for some additional Data and it just got worse. 

Amanda Sulicz

It got worse. 

David Turner

I think you mentioned earlier the conference website, there were some red flags there? 

Amanda Sulicz

Yes. 

David Turner

Can you talk a little bit about that? And then I'm also interested to see how this connects with the concept of a paper mill. I'll shut up and let you talk.

Amanda Sulicz

Yeah, I got all of – sit tight, David. I got all of this for you. 

David Turner

Love it. 

Amanda Sulicz

All right. So yeah, the conference website, simultaneously, I love to click buttons. I have so much fun just digging. I just love to dig, and I love doing all of this. And so first looping back to, I don't have a screenshot of this in the presentation, but first looping back to the fact that it was a hybrid conference, and being held at a University in theory. So the first thing that was weird about all of this, this conference website didn't have any information about travel, not even the room the conference was happening in. If you were like, "Oh, it's in the auditorium in the IEEE building," or something. There was no information, no hotel recommendations, nothing, absolutely nothing on the website. So I mean, if people are coming to this conference, they should know where they're going. 

David Turner

And it's not a small conference; you've got 300 papers, right? 

Amanda Sulicz

Right, yeah. So that was a little weird. But then I was just poking around to see if the information in our database for who was listed as an organizer was the same as what was on their website, because sometimes that can be an area where we can find discrepancies. But then as I'm clicking around, this person on the right was actually, I started to see his name everywhere, and that is what my little Spidey sense started to go off. So now I guess I'll say if there's any publishers on this call, flag this name because that's a public service announcement from IEEE, he is still active. So he does books publishing, he does conference publishing. I don't know what else he does,

20:00

but if you're seeing this name in a book or anything, proposal, make a mental note of that and check it out a little bit more thoroughly. So this gentleman here is a name I'm going to keep showing you. So as I'm clicking around on the website, I find this review process, and I see this reviewer form button. And I was like "What's that?" So I used to work as an in-house editor before I started IEEE, and I'm expecting this reviewer form to say something like "What areas do you want to review in? Tell me a little bit about yourself. What makes you qualified to be a reviewer? Are you a PhD candidate? Are you a professor?" Something along those lines. So I click this because I like to click buttons and I click this. And this is the reviewer form. Again, not truncated, this is the entire form. It asked me if I want to be a guest editor in special issues. For what? I don't know.

And then it asked me if I want to be on the editorial team of this Journal of Machine Learning and Signal Processing. And they were so kind to give me a website link, and I was like," Say less. Let's keep clicking. "So I keep clicking and I go to this website. Oh, sorry, I had animations there. So I go to this website, this is the art publisher's website. And again, look, look who's here. He's editor-in-chief of this journal.

David Turner

Surprise!

Amanda Sulicz

And if people in the audience aren't aware, this is clearly a predatory publishing website. There's no ISSN information. They're indexed by Google Scholar, that's not a thing. They're not indexed anywhere. This is not a reputable journal. So I start clicking around this website because I like to click my buttons, and I get to the contact information. I was like "I'm going to Google this web. I'm going to Google this address. See what comes up." So I Google, and I get the first journal here that I was just looking at that I just showed you. And then I come down here, and I see this Agasitiyar Research Technologies. So I started by IEEE beginning of 2023, and there was a case involving this when I was coming here, so they're kind of wrapping it up. And I was like, when I saw this on the Google results, I was like "Why do I know this name? Why do I know this name?" And I asked one of my coworkers who's been here for 20 years, and he was the one that handled this case, and he was like "Oh yeah, that's the Anna Abalkina paper mill." I'm like "Oh, sure enough. Sure enough."

So this is my connection to the paper mill now, David. So these were screenshots that Anna Abalkina, she's a paper mill sleuth. She's based in Germany. She has been so incredibly helpful. We have a great collaboration with her and some of her co-authors. I just want to do a quick little plug for her because I do really appreciate how helpful they've been in helping us identify conference papers specifically that have been posted online for selling authorship. We're actually working on another huge case with them right now. So these were screenshots she had sent us back, I think it was probably 2022, so it was before I started here. But you see the Agasitiyar website on this post, you see the logo up on this post, you see it over on this post. So it was everywhere. It's down here, got the same website that was on the Google screenshot I showed you previously. So it's like "Oh, this isn't good." And then I find – I was like "What's this DigiConfy thing? Let me check this out. Let me click more buttons."

24:00

So I click this, and this, when I searched for the name, this is actually what Google told me. It said that it is a branding for events and publication services. And the AI overview in Google knew that it is associated with a predatory publisher. Google knows this, so everyone should know it. So again, my public service announcement.

So these are just some quick little screenshots. It's the same information, but we have the same address here. This is a conference they're hosting. Look, it's our friend again. Our friend's popping up down here on conferences that are being hosted by this other platform. So I mean, it was – there was a lot that we connected back to this, and it's served a wealth of information to us in this process. We learned through this about what other policies we need to update and that we're working on updating. We've started to scrutinize all of our conferences more, so not just new conferences, because there's always the chance that just because a conference has been held three times before, it doesn't mean that the fourth time, it's not going to get hijacked in some way. So we're starting to look at who's actually organizing, doing some rough Googling before. Some of my colleagues have actually turned Amanda into a verb, so they have Amanda a conference now.

But just looking, do the conference logistics make sense? Is the local committee really local? So if you're, again, claiming you're hosting a conference at a University, you should have people from that University because you need somebody on the ground to help organize the catering. Just that simple stuff that you may not really think about. But if everybody is coming from a completely different country, they don't know maybe necessarily what's going on at that venue. But then the biggest flag here was where the author's coming from. It was listed as hybrid, so there's a chance that you might have people presenting online, but all in all, the entire author list shouldn't be from not even the continent where the conference is being held at. And then IEEE also has policies that your conference needs to align to our publication policy, so about peer review and the number of reviewers and the peer review Data you have to supply, and using certain screening tools you have to screen for plagiarism. Things like that need to be outlined on your website.

So that's an IEEE policy, but we also check now to making sure that they're saying that. And I mentioned this is the third year of the ad hoc, but we're still going and we're going to keep going. And for everyone in the room that wanted to know if you didn't do it on your own already, this is a street view from Google of the address where all of this was coming from. So yeah, I don't even really know much what to say about that, but this is the paper mill. Yeah. And here's my contact information and our publishing ethics contact line, just pub-ethics@IEEE.org. Also included David's email here. And I wanted to just go to this useful resources link. So I mentioned the Problematic Paper Screener. 

David Turner

Yeah.

Amanda Sulicz

This is a link to, or if you want to write down these archive, these pre-prints, but this pre-print will bring you to the Problematic Paper Screener.

28:00

And then Anna Abalkina and some co-authors recently published a pre-print called Opening Pandora's box: Paper mills in conference Proceedings. IEEE is quite heavily mentioned in this pre-print. But as I mentioned before, Anna and colleagues have been so helpful in bringing this to our attention and providing us with information to help us get this. And we're working on that project now. So thanks to both of these sleuths here, and all of the other sleuths that also help work with us and give us content to keep me busy.

David Turner

Yeah, definitely. Definitely. Well, and I know that it's really critical because as you talked about this example today, it just was piece after piece, and I think maybe some of the things were pretty extreme. I'm encouraged with how much you guys have learned and how much things have improved because not every conference that's questionable is as obvious as this one. And so I know you don't want to talk too much about tools unless people start figuring out ways around those tools. But I guess it's fair to say that every day you're investing in new tools, new processes, new ways, new communication. What are some of the most important pieces in terms of watching for these things to maybe find those examples that are not so obvious?

Amanda Sulicz

Oh, I mean, we have a suite of tools that we work with, a variety of vendors that we work with on their tools, DCL included. 

David Turner

Yeah. 

Amanda Sulicz

Yeah. IEEE is part of the STM Integrity Hub. So all of that, the suite of tools that's offered in there, but that is only in our journals, that we don't currently run those for conferences. We're hoping to one day make that happen, but there's all kind of screening in there. I know recently, well, it's actually not that recently, it's been about a year or two now, we've been working with DCL to help us with our plagiarism screening. And that's just reading the reports. IEEE gets over 200,000 submissions every year, and we have a mandate from our volunteers that every article that's published on Xplore needs to be screened for plagiarism, so similarity report. And we've been working with DCL to help streamline that process, and it's been a really great process.

And through working with DCL, do a little commercial plug here, even though that wasn't the intention of this, but we've actually identified more signals to help us detect paper mills and conferences that we didn't have before. And we're finding more and more every day, so I'm glad that this was just kind of serendipity that I also was doing this Learning Series, but it's been really helpful, just all of these different tools and everything's evolving too because the nefarious actors are evolving, so publishers need to, it's arms race, but it's been interesting and I know I have job security for the next eternity.

David Turner

Well, we appreciate the plug and all of that, and I think it's great. And I think one of the things that we've discovered in working together with you guys is that just getting the information can be really difficult because it can be overwhelming, right? So you get 300 papers or you start running things through similarity checks or what have you, you get volumes of information back.

32:00

And so then the problem is not so much finding the things, it's digging them out. What are the important ones? What are the ones that are likely not a problem? Where should we focus our efforts and really doing that? So hopefully we're able to. I know we've got a handful of projects that we're doing together, so hopefully we can help with that.

Amanda Sulicz

I think I have three more on deck for you. 

David Turner

Oh, well, come on. Let's do it. 

Amanda Sulicz

Yeah. 

David Turner

Anyway. All right. Well, Marianne, we actually finished here on the early side, but I'm sure that there are probably some questions out there. 

Marianne Calilhanna

Yeah.

David Turner

Most likely for Dr. Sulicz and not me, but.

Marianne Calilhanna

Yeah, Dr. Sulicz, we have some questions. Thank you everyone. Please, if you have more questions, let's continue this conversation, particularly while we have Amanda here. So Amanda, how big is the research integrity team at IEEE in terms of the total amount of time that you put into the investigation? Can you speak to that and give us an estimate?

Amanda Sulicz

Oh, well, so it's a weird answer for IEEE. So our publishing ethics team specifically, I think we have eight now, seven? We have seven right now. About to be eight. Oh my gosh. But we also –

Marianne Calilhanna

So under 10.

Amanda Sulicz

Yeah, so the team right now is split, and we do three different functions. So we have the IPR side, so getting copyrights, getting people registered to run that more housekeeping type of thing. We have me and another person who works exclusively on conferences. And then we also have people who work on journals specifically. And then we also collaborate very heavily with a different wing of the publications team that just helps give us Data all the time too, from ScholarOne, which is our manuscript submission system. And we work really closely with them. And I think there's seven people on that team. So they're not officially on the publishing ethics team, but we work extremely closely with them. We actually met with them this morning about cases. So we have an unofficial 15, I guess we'll call it. And then the time we spend on a case changes greatly depending on what the case is.

We have cases had this just been a routine plagiarism case. Those are handled relatively quickly. We still have a process where it needs to be reviewed by all of our volunteers, which does take some time because of just the cadence of meetings and things. But when it's a nice, you got that doc-to-doc comparison, you say "Oh, it's 60% plagiarism." We have to contact the authors, we give them time, but I mean those all in all, I think are handled within a couple months usually. But then I actually, the case isn't even done officially yet, but I have been working on an editorial misconduct case for three years now. And it's a lot of review. And I, again, work closely with the other side of the publications team to help do that kind of data analysis.

We've worked with multiple vendors on that case too. It just depends on the complexity of the case. And we've noticed, well, we've been noticing it since I started here, but our caseload has actually been getting more complex too. I remember one of my colleagues, Tony, when he started here 20 years ago, when they would get plagiarism complaints, he literally would sit there with a highlighter and do the overlap.

36:00

And then eventually we got authenticate, but then that was easy. And then it turned into people paraphrasing, so it took a little bit more time to find it, but you could still find it. And now we're getting these huge networks. So our caseload is increasing, but the time for each case is also significantly increasing.

 

Marianne Calilhanna

Right.

Amanda Sulicz

So it's a lot. 

Marianne Calilhanna

Yeah. How much involvement did the editorial office managers have in the whole process? 

Amanda Sulicz

I think we might be using editorial office manager, how are you using that definition?

 

Marianne Calilhanna

This was submitted from someone. So if the person who asked the question wanted to clarify that a little bit more, I think I interpret that as the response to you have that research integrity team, but then you said there was some crossover with other departments in IEEE. So thinking about production, editorial, research integrity, what's that overlap look like?

Amanda Sulicz

It depends on the case. So we work more heavily with the other team when we have a journal investigation, just because our conferences are outside of that submission system. But we do work really heavily. Oh, actually, well, I mean, what am I saying? We also work with the conferences team. They have a research integrity group also. Completely forgot about them, sorry about that. But there's a team of volunteers, and also staff members that work on our conferences, events, and experiences team. So it just depends what level of detail we need. It all varies, sometimes it goes from none to full involvement.

Marianne Calilhanna

Okay. And just to clarify, I think you answered the question, but by that, they mean people who run the day-to-day operations of the journal? 

Amanda Sulicz

Oh, yeah. Yes. 

Marianne Calilhanna

Yeah. 

Amanda Sulicz

When we need to, we work with them, yes. 

Marianne Calilhanna

And did you confirm that the authors of the papers in the proceedings were truly involved in the effort as opposed to simply having their identities used? 

Amanda Sulicz

Nobody claimed that they weren't an author so, when they got the retraction notice. That would be their opportunity to say I didn't author this paper and nobody said that. They just said that, "I disagree," and then a litany of reasons why they disagreed. 

Marianne Calilhanna

And then someone's just curious, how did you find your way into this field? PhD in chemistry to research ethics? 

Amanda Sulicz

So when I finished my PhD, I did a postdoc with Kirk Schanze, who is now retired. He retired last year. But he was an editor for an ACS journal, and so I started to learn about the publishing arm from him. And then when it came time for me to finish my postdoc, I got a job as an in-house editor for Applied Physics Reviews with AIP Publishing. And then I wore many hats at AIP Publishing, and then eventually found this research integrity. And when I was little, I wanted to be a lawyer really bad, so I think I always was interested in, I'll say, the investigative aspect of it. And this, I get the best of both worlds. Well, I mean, I'm in engineering now, not in chemistry, but I get to use my science, and I also get to scratch my CSI itch that I had as a child.

 

Marianne Calilhanna

I'm just curious, did you fill out the reviewer form, that Google form you shared? 

Amanda Sulicz

Oh no, goodness, no. Oh, no. 

Marianne Calilhanna

Okay. 

Amanda Sulicz

That's what that black box was. That was my personal email address on there. No. 

40:00

Marianne Calilhanna

I didn't know if you maybe made up your own anonymous persona. 

Amanda Sulicz

I've thought about it, but I have enough to keep me busy that I don't think I need to. 

David Turner

Yep.

Marianne Calilhanna

Yeah. 

Amanda Sulicz

Yeah. Too many rocks up. 

Marianne Calilhanna

I mean, another thing that I think is important to think about in this scholarly publishing ecosystem, we all want to share the tools that each other are using. We want to share names like the person's name who you identified. I did put that in the chat, so, Dr. Kannadhasan. And at we DCL like to talk about some of what we're doing, but there's a balance between sharing too much of what we do because we don't want these characters to get more ammunition and we're just constantly chasing each other around. Just wondered if you have any thoughts on that. 

Amanda Sulicz

So it's kind of funny. So yesterday I actually just watched Digital Sciences on Demand webinar about the FOCI report as well as Forensic Scientometrics. And there was three different sleuths who were presenting as part of it. And one of the sleuths on there was actually asked, one of the authors of the paper mill, Pandora box paper here was asked a similar question. And I think my answer is the same as hers. It's a fine line because you want to share this information so that publishers know to look out for these things or anyone who's publishing book chapters or just anything you want. But then we do need to gatekeep a little bit.

So as I mentioned earlier, the projects that I want to keep developing with DCL, we have signals we found, but I'm not going to share what that signal is. But I mean, I think that's also one of the benefits of when we have in-person publisher meetings. I think that's always an opportunity for us to talk, not online, so it's a little bit more of a closed door, which I think is great. And I've had the opportunity to go to SSP meetings and STM meetings, and I think that's always just a wonderful source of information to hear what other publishers are doing without having to make it so public.

Marianne Calilhanna

Right. Good pitch for attending these events. Do you have a sense for how the authors were recruited to submit the papers to the conference? Is this a network of paper people who know each other? 

Amanda Sulicz

Those are computer skills I wish I had to do network analysis better. Sometimes I feel like I want to go get a low-key computer science degree so I can learn how to do this. But I will say I can't speak for the motivation of these particular sets of authors. I know when we met with the Pandora's box authors a couple months ago now, one of the authors on that paper had mentioned how they've done really deep digging into various, I think it was India specifically, some of their incentive programs that their universities have. And it's actually cheaper. So some of the universities will offer an incentive, and I'm going to just make up numbers here because I don't remember them specifically, but they'll say "Oh, if you publish a paper, you get 500 US dollars."

But then if you look at the paper mill for these authorship for sale posts, you can maybe buy authorship on a paper for $50. So you can also use that as a moneymaking scheme individually, hoping that you don't get caught.

44:00

But then I mean, there's so many, just I guess the history of scientists love to assign numbers to things. And when the h-index was developed, that's being manipulated now by hyper-prolific authors. And now we're seeing citation mills that are happening. The impact factor was being gamed for a while, and that caused Clarivate to change how they were doing the impact factor. I think that's a blessing and a curse of scientists is that we want to be able to measure, but then also people learn how to game the measurement.

Marianne Calilhanna

Right. 

Amanda Sulicz

And then when you get these rankings, you get promotions at universities, you get grant money, you get more students, you bring notoriety to University. And I think there's just a lot that motivates people to want to do this. And we need to address the underlying issue to stop, I guess we would call it more of a cure than a prevention at this point, so cure the reason why people want to do this instead of just preventing the bad papers from being published.

Marianne Calilhanna

That's a big task. 

David Turner

Well, and Marianne, it sounds like we've got a whole host of new webinars' topics just laid out before. 

Marianne Calilhanna

Did the papers in this specific case go out for multiple rounds of revision? Did it look like they actually took any of the fake reviewer comments into account? 

Amanda Sulicz

Oh, I'm sorry if I wasn't clear for the person who asked that. Those were clearly all AI generated and not actually –

Marianne Calilhanna

They were AI. 

Amanda Sulicz

Yeah. 

Marianne Calilhanna

Yeah, okay. 

Amanda Sulicz

Okay. So no, I don't think any peer review happened at that conference. 

Marianne Calilhanna

Can you speak to a time period, the amount of time between your first, in this case, your first suspicions and then contacting the authors? 

Amanda Sulicz

Oh, let's see. I think I got the plagiarism complaint – let's see, everything was retracted in September of last year, and I think it took us about a year in total to go through everything. Contacting the authors is the last step, letting them know of the retraction. In this particular instance, we contacted them at the end. I mean, we give them two weeks to respond and then it just takes a little time to organize everything. So between the authors learning and it actually being retracted was maybe a month, if even. But we had been working on this case for almost a year.

Marianne Calilhanna

And were they using IEEE assets, your logo? Any other?

Amanda Sulicz

This was an IEEE-sponsored conference, so they were allowed to use –

Marianne Calilhanna

They were allowed to use that. 

Amanda Sulicz

Yes. 

Marianne Calilhanna

Okay. 

Amanda Sulicz

I did find one of the DigiConfy conferences that I don't think I put a screenshot of, I did find on there that they were claiming to be a Computer Society conference, and Computer Society was not, they had denied their application, so they got a nice take-down notice. So there are instances of our societies or IEEE as a whole being used without our permission. And when we do catch them, we tell them they need to stop with the help of our lawyers.

Marianne Calilhanna

Yeah, I was going to say that seems to be one area where legal can get involved. But with something like this, can legal get involved?

Amanda Sulicz

They didn't technically do anything illegal –

Marianne Calilhanna

Right. 

Amanda Sulicz

– from a criminality standpoint. I mean, I guess you could make a FBI finances investigation here, but that's not our problem. Not a publishing ethics problem. 

Marianne Calilhanna

Yeah. 

Amanda Sulicz

But our team works a lot with legal

48:00

for a variety of reasons. We have some people, IEEE is, we do ban people when they have violated our policies. So sometimes people get mad about that and then lawyers get involved. But we're truly a cross-functional team. 

Marianne Calilhanna

Right. 

Amanda Sulicz

We work with everyone. 

Marianne Calilhanna

Right. 

David Turner

All right. Any other questions? 

Marianne Calilhanna

We have answered all the submitted questions. I can't thank you enough for your time and for your work. What you're doing is really important. And yeah, talk about job security. It's blessing and a curse here, huh? Well, thank you to everyone who has joined today. I know everyone needs to get back to work, and certainly we need Amanda to get back on her cases. I want to remind all of you that the DCL Learning Series comprises things like this webinar. We also have a monthly newsletter and blog posts and other programs that we do put on throughout the year, so we hope to see you at our next one, but for today, this concludes our webinar.

Marianne Calilhanna

Thank you so much. 

David Turner

Thank you.

Marianne Calilhanna

Thanks, Amanda and David.