Everyone interested in this topic is welcomed to join us. the following is the abstract for this week\u2019s paper.<\/p>\n\n\n\n

\n\t

\n\t\tImproving Language Understanding by Generative Pre-Training\n\t<\/p>\n\t

\n\t\tAlec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever\n\t<\/p>\n\t

\n\t\tAbstract\n\t<\/p>\n\t

\n\t\tNatural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering, semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant, labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon the state of the art in 9 out of the 12 tasks studied. For instance, we achieve absolute improvements of 8.9% on commonsense reasoning (Stories Cloze Test), 5.7% on question answering (RACE), and 1.5% on textual entailment (MultiNLI).\n <\/p>\n<\/div>\n\n\n\n

Improving language understanding by generative pre-training<\/a>\u4e0b\u8f7d<\/a><\/div>\n\n\n\n

NLPIR SEMINAR 32nd ISSUE COMPLETED<\/strong><\/h2>\n\n\n\n
Last Monday, Qinghong Jiang gave a presentation about the paper, Improving Language Understanding by Generative Pre-Training, and shared some opinion on it.<\/p>\n\n\n\n
$\"zero-shot-transfer@2x\"\/$ <\/figure>\n\n\n\n
This was a state-of-the-art word representation method before BERT. visit https:\/\/openai.com\/blog\/language-unsupervised\/<\/a> for more information.<\/p>\n","protected":false},"excerpt":{"rendered":"
NLPIR SEMINAR Y2019#19 INTRO In the new … \u7ee7\u7eed\u9605\u8bfb →<\/span><\/a><\/p>\n","protected":false},"author":862,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37,38],"tags":[],"_links":{"self":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts\/6970"}],"collection":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/users\/862"}],"replies":[{"embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/comments?post=6970"}],"version-history":[{"count":5,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts\/6970\/revisions"}],"predecessor-version":[{"id":6991,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts\/6970\/revisions\/6991"}],"wp:attachment":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/media?parent=6970"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/categories?post=6970"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/tags?post=6970"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}

INTRO <\/h3>\n\n\n\n In the new semester, our Lab, Web Search Mining and Security Lab, plans to hold an academic seminar every Monday, and each time a keynote speaker will share understanding of papers on his\/her related research with you.<\/p>\n\n\n\n

NLPIR SEMINAR 32nd ISSUE COMPLETED<\/strong><\/h2>\n\n\n\nLast Monday, Qinghong Jiang gave a presentation about the paper, Improving Language Understanding by Generative Pre-Training, and shared some opinion on it.<\/p>\n\n\n\n<\/figure>\n\n\n\n

INTRO <\/h3>\n\n\n\n
In the new semester, our Lab, Web Search Mining and Security Lab, plans to hold an academic seminar every Monday, and each time a keynote speaker will share understanding of papers on his\/her related research with you.<\/p>\n\n\n\n