https://nicholas.carlini.com/writing/2022/a-case-of-plagarism-in-machine-learning.html [*] Main Papers Talks Code Writing Writing A Case of Plagarism in Machine Learning Research by Nicholas Carlini 2022-04-08 --------------------------------------------------------------------- I recently came to be aware of a case of plagiarism in the machine learning research space. The paper A Roadmap for Big Model plagiarized several paragraphs from one of my recent papers Deduplicating Training Data Makes Language Models Better. (There is some irony in the fact that the Big Models paper copies from a paper about data copying. This irony was not lost on us.) This is unfortunate, but to my dismay, our paper was not the only paper copied from: the Big Models paper copied from at least a dozen other papers. In the grand scheme of things, this particular form of copying isn't the worst thing ever. It's not like a paper has directly copied the method of a prior result and claimed it as its own. But even putting aside the fact that claiming someone else's writing as one's own is wrong, the value in survey papers is in how they re-frame the field. A survey paper that just copies directly from the prior paper hasn't contributed anything new to the field that couldn't be obtained from a list of references. (Please note the Big Models paper has a hundred authors. Likely only a few of the authors have participated in this copying. Misconduct by a small fraction of the authors should not be held against the majority of well-behaving authors.) See below for a few of the more egregious examples of this, with text from the Big Models paper on the left and the corresponding text from the original paper on the right. Copied text is highlighted in green. +-------------------------------------------------------------------+ |Text from the "Big Models" Paper |Text from the Original Paper | |---------------------------------+---------------------------------| | |[Original]: The privacy risks of | |The risks of data memorization, |data memorization, for example | |for example, the ability to |the ability to extract sensitive | |extract sensitive data such as |data such as valid phone numbers | |valid phone numbers and IRC |and IRC usernames, are | |usernames, are highlighted by |highlighted by Carlini et al. | |Carlini et al. [41]. While their |(2020). While their paper finds | |paper identifies 604 samples that|604 samples that GPT-2 emitted | |GPT-2 emitted from its training |from its training set, we show | |set, we show that over 1 of the |that over 1% of the data most | |data most models emit is |models emit is memorized training| |memorized training data. In |data. In computer vision, | |computer vision, memorization of |memorization of training data has| |training data has been studied |been studied from various angles | |from various angles for both |for both discriminative and | |discriminative and generative |generative models Deduplicating | |models Deduplicating training |training data does not hurt | |data does not hurt perplexity: |perplexity: models trained on | |models trained on deduplicated |deduplicated datasets have no | |datasets have no worse perplexity|worse perplexity compared to | |compared to baseline models |baseline models trained on the | |trained on the original datasets.|original datasets. In some cases | |In some cases, deduplication |deduplication reduces perplexity | |reduces perplexity by up to 10%. |by up to 10%. Further, because | |Further, because recent LMs are |recent LMs are typically limited | |typically limited to training for|to training for just a few epochs| |just a few epochs [47, 47], the |(Radford et al., 2019; Raffel et | |models can reach higher accuracy |al., 2020), by training on higher| |faster by training on higher |quality data the models can reach| |-quality data. The simplest |higher accuracy faster. The | |technique to find duplicate |simplest technique to find | |examples would be to perform |duplicate examples would be to | |exact string matching between all|perform exact string matching | |example pairs, but as we will |between all example pairs, but as| |show, this is insufficient. We |we will show, this is | |introduce two complementary |insufficient. We introduce two | |methods for performing |complementary methods for | |deduplication. First, using a |performing deduplication. First, | |suffix array [49], we remove |using a suffix array (Manber and | |duplicate substrings from the |Myers, 1993), we remove duplicate| |datasets if they occur verbatim |substrings from the dataset if | |in more than one example. Second,|they occur verbatim in more than | |we use MinHash [48] , an |one example. Second, we use | |efficient algorithm for |MinHash (Broder, 1997), an | |estimating the n-gram similarity |efficient algorithm for | |between all pairs of examples in |estimating the n-gram similarity | |a corpus, to remove entire |between all pairs of examples in | |examples from the dataset if they|a corpus, to remove entire | |have high n-gram overlap with any|examples from the dataset if they| |other example. causes researchers|have high n-gram overlap with any| |to over-estimate model accuracy a|other example. causes researchers| |nd biases model selection towards|to over-estimate model accuracy, | |models and hyperparameters that |but also biases model selection | |intentionally overfit |towards models and | | |hyperparameters that | | |intentionally overfit | |---------------------------------+---------------------------------| |propose the notion of a World |[Original]: propose the notion of| |Scope (WS) as a lens through |a World Scope (WS) as a lens | |which to audit progress in NLP. |through which to audit progress | |They define five WSs, and they |in NLP. We describe five WSs, and| |note that the most popular |note that most trending work in | |pre-training in NLP operates in |NLP operates in the second | |the WS2 (Internet) |(Internet-scale data). | |---------------------------------+---------------------------------| |In addition to BERT, where masked|[Original]: In addition to BERT | |words are predicted from the |where masked words are predicted | |non-masked words in the language |from the non-masked words in the | |modality, LXMERT proposes |language modality, LXMERT, with | |cross-modality model architecture|its cross-modality model | |that could predict masked words |architecture, could predict | |from the visual modality as well |masked words from the vision | |so as to resolve ambiguity. For |modality as well, so as to | |example, it is hard to determine |resolve ambiguity. For example, | |the masked word carrot from its |as shown in Fig. 2, it is hard to| |language Who is eating the carrot|determine the masked word | |?, but the word choice is clear |'carrot' from its language | |if the visual information is avai|context but the word choice is | |lable |clear if the visual information | | |is considered | |---------------------------------+---------------------------------| |a number of information-seeking |[Original]: a number of | |questions such as what is the |information seeking questions | |definition of ... as the prompts,|such as "what is the definition | |[...] show that this self-talk me|of ..." to discover [...] that | |thod substantially improves the |the self-talk procedure | |performance of zero-shot big |substantially improves the | |model baselines on four out of |performance of zero-shot language| |six commonsense benchmarks, and |model baselines on four out of | |competes with models that obtain |six commonsense benchmarks, and | |knowledge from external knowledge|competes with models that obtain | |bases. |knowledge from external KBs. | |---------------------------------+---------------------------------| |even if the social bias is |[Original]: even if the social | |eliminated at the word level, the|bias is eliminated at the word | |sentence-level bias can still |level,the sentence-level bias can| |exist due to the imbalanced |still be caused by the unbalanced| |combination of words [..] |combination of words [..] by | |replacing sensitive words in the |replacing sensitive words in the | |original sentence with words in a|original sentence with words in a| |similar semantic but different |similar semantic but different | |bias directions. |bias directions. | |---------------------------------+---------------------------------| | |[Original]: We propose two | |It proposes two methods to learn |methods to learn cross-lingual | |cross-lingual language models |lan- guage models (XLMs): one | |(XLMs): one unsupervised that |unsupervised that only relies on | |only relies on monolingual data, |monolingual data, and one | |and one supervised that leverages|supervised that leverages | |parallel data with a new |parallel data with a new | |cross-lingual language model |cross-lingual language model | |objective. [...] Both the CLM and|objective. [...] Both the CLM and| |MLM objectives are unsupervised |MLM objectives are unsupervised | |and only require monolingual |and only require monolingual | |data. For improving cross-lingual|data. We introduce a new | |pre-training, they introduce a |translation language modeling | |new translation language modeling|(TLM) objective for improving | |(TLM) objective. They consider |cross-lingual pretraining. In | |cross-lingual language model pre-|this work, we consider | |training with either CLM, MLM, or|cross-lingual language model | |MLM is used in combination with |pretraining with either CLM, MLM,| |TLM. |or MLM used in combination with | | |TLM. | |---------------------------------+---------------------------------| |to large-scale mono-lingual |[Original]: to large-scale | |corpora across many languages. |monolingual corpora across many | |The input texts are noised by |languages. The input texts are | |masking phrases and permuting |noised by masking phrases and | |sentences, and a single |permuting sentences, and a single| |Transformer model is learned to |Transformer model is learned to | |recover the texts. |recover the texts. | |---------------------------------+---------------------------------| |Such models not only have lower |[Original]: Such models not only | |inference latency, but they also |have lower inference latency, but| |do not suffer from the problem of|they also do not suffer from the | |errors that propagate from one |problem of errors that propagate | |component to the next |from one component to the next | |---------------------------------+---------------------------------| |has presented a study of adapters|[Original]: have presented a | |for multilingual ST and shown |study of adapters for | |that language-specific adapters |multilingual ST and shown that | |can enable a fully trained |language-specific adapters can | |multilingual ST model to be |enable a fully trained | |further specialized in each |multilingual ST model to be | |language pair. |further specialized in each | | |language pair | |---------------------------------+---------------------------------| | |[Original]: 147M | |147M conversation-like exchanges |conversation-like exchanges | |extracted from Reddit comment |extracted from Reddit comment | |chains over a period spanning |chains over a period spanning | |from 2005 through 2017. DialoGPT |from 2005 through 2017, DialoGPT | |[...] The GPT-2 transformer model|[...] The GPT-2 transformer model| |adopts the generic transformer |adopts the generic transformer | |language model [25] and leverages|language model (Vaswani et al., | |a stack of masked multi-head |2017) and leverages a stack of | |self-attention layers to train on|masked multi-head self-attention | |massive web-text data. [...] Dial|layers to train on massive | |oGPT inhered a 12-to-48 layer |web-text data. [...] Our model | |transformer with layer |inherits from GPT-2 (Radford et | |normalization, a initialization |al., 2018), a 12-to-48 layer | |scheme that accounts for model |transformer with layer | |depth that we modified, and byte |normalization, a initialization | |pair encodings [1236] for the |scheme that accounts for model | |tokenizer MMI employs a |depth that we modified, and byte | |pre-trained backward model to pre|pair encodings (Sennrich et al., | |dict source sentences from given |2016) for the tokenizer. MMI | |responses [...] a strong |employs a pre-trained backward | |preference can be observed for |model to pre- dict source | |DialoGPT over PersonalityChat |sentences from given responses. | |[1432] |[...] strong preference can be | | |observed for DialoGPT over | | |PersonalityChat. | +-------------------------------------------------------------------+ How did we find these examples? One of my coauthors was reading the Big Models paper and noticed that some of the text seemed oddly familiar, and after quickly looking things over we found that in fact a bunch of the text was directly copied from our paper. Given that this happened to us, we then set out to see if there were other examples too. As part of a prior project, I had collected a dataset of PDFs for (almost) every accepted paper at top machine learning venues (ICML/ICLR/NeurIPS/AAAI/ACL/etc). So all I did to find the above copied text was to take these PDFs, extract out all of the text and dump it into a single .txt file, and then run our dataset deduplication tools (that we developed for the paper that was copied from!) to find all repeated sequences that were contained both in the Big Models paper along with some other prior publication. To rule out false positives, I only considered sequences of 1. at least 10 words (after whitespace normalization), 2. that are contained sequentially in the Big Models paper, 3. and also present in a prior paper, 4. but are not present in more than one prior paper. This ensures that I won't flag any common phrases as copied (e.g., copyright blocks, citations to prior paper titles or author names, etc). And then from there, it was just a matter of quickly manually reviewing a few of the most egregious cases (shown above). There were other examples of self-plagiarism where the paper that was copied from shared an author with the new paper that I have omitted-while this isn't an ideal practice, it's less concerning. Because of this filtering process, and because my dataset of papers is not exhaustive over all prior publications (notably, it only contains accepted papers, not arXiv preprints), it is possible there is more copying going on here than I have identified. However even what we have found so far is already more than should happen, and I am saddened that this is happening at all. --------------------------------------------------------------------- If you want to be notified the next time I write something (maybe like this, maybe not, who knows) enter your email address here. [ ] [SUBMIT] There's also an RSS Feed if that's your thing.