Everyone interested in this topic is welcomed to join us. the following is the abstract for this week\u2019s paper.<\/p>\n\n\n\n

\n\t\t

\n\t\t\tPay Less Attention with Lightweight and Dynamic Convolutions<\/span>\n\t\t<\/p>\n\t\t

\n\t\t\tFelixWu, Angela Fan, Alexei Baevski, Yann N. Dauphin, Michael Auli<\/span>\n\t\t<\/p>\n\t\t

\n\t\t\tabstract<\/span>\n\t\t<\/p>\n\t\t

\n\t\t\tSelf-attention is a\nuseful mechanism to build generative models for language and images. It\ndetermines the importance of context elements by comparing each element to the\ncurrent time step. In this paper, we show that a very lightweight convolution\ncan perform competitively to the best reported self-attention results. Next, we\nintroduce dynamic convolutions which are simpler and more efficient than\nself-attention. We predict separate convolution kernels based solely on the\ncurrent time-step in order to determine the importance of context elements. The\nnumber of operations required by this approach scales linearly in the input\nlength, whereas self-attention is quadratic. Experiments on large-scale machine\ntranslation, language modeling and abstractive summarization show that dynamic\nconvolutions improve over strong self-attention models. On the WMT\u201914\nEnglish-German test set dynamic convolutions achieve a new state of the art of\n29.7 BLEU.<\/span>\n\t\t<\/p>\n\t<\/div>\n<\/p>\n\n\n\n

Pay Less Attention with Lightweight and Dynamic Convolutions<\/a>\u4e0b\u8f7d<\/a><\/div>\n\n\n\n\n\n\n\n

NLPIR SEMINAR 23rd ISSUE COMPLETED<\/strong><\/h2>\n\n\n\n
Last Monday, Zhaoyou\u00a0Liu<\/strong> gave a presentation about the paper, Pay Less Attention with Lightweight and Dynamic Convolutions<\/strong>, and shared some opinion on it.<\/p>\n\n\n\n
$\"\"$ <\/figure>\n\n\n\n
This paper was published as a conference paper at ICLR 2019.
Dynamic convolutions build on lightweight convolutions. The kernel is a function of the current time-step only as opposed to the entire context as in self-attention. This approach shares similarity to location-based attention which does not access the context to determine attention weights.
And the experiments show that dynamic convolutions perform as well as or better than self-attention with less time.<\/p>\n","protected":false},"excerpt":{"rendered":"
NLPIR SEMINAR Y2019#10 INTRO In the new … \u7ee7\u7eed\u9605\u8bfb →<\/span><\/a><\/p>\n","protected":false},"author":862,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37,38],"tags":[],"_links":{"self":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts\/6849"}],"collection":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/users\/862"}],"replies":[{"embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/comments?post=6849"}],"version-history":[{"count":2,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts\/6849\/revisions"}],"predecessor-version":[{"id":6871,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/posts\/6849\/revisions\/6871"}],"wp:attachment":[{"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/media?parent=6849"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/categories?post=6849"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.nlpir.org\/wordpress\/wp-json\/wp\/v2\/tags?post=6849"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}

INTRO <\/h3>\n\n\n\n In the new semester, our Lab, Web Search Mining and Security Lab, plans to hold an academic seminar every Monday, and each time a keynote speaker will share understanding of papers on his\/her related research with you.<\/p>\n\n\n\n

INTRO <\/h3>\n\n\n\n
In the new semester, our Lab, Web Search Mining and Security Lab, plans to hold an academic seminar every Monday, and each time a keynote speaker will share understanding of papers on his\/her related research with you.
<\/p>\n\n\n\n