Where does the knowledge graph of "Tsinghua University" come from: the status quo and future of entity relationship extraction

Author: the wave of artificial intelligence

recent years, the depth of learning caused by Han Xu, Sky High, Liu Zhiyuan (This switched Tsinghua University Liu Zhiyuan teacher zhihu personal column) sweeping the globe, the popularity of the Internet to bring vast amounts of data and resources rapidly under the domination of Moore's Law With the dual blessing of increased computing power, deep learning has deeply affected all directions of natural language processing and greatly promoted the development of natural language processing.

is now in 2019, and many limitations of deep learning are slowly being widely recognized. For natural language processing, in order to achieve a fine and in-depth semantic understanding, the essential problems cannot be solved by simply relying on data annotation and computing power input. Without the support of prior knowledge, "No one can beat Chinese table tennis" and "No one can beat Chinese football", in the computer view, there is no huge difference in semantics, but in fact the two sentences "cannot beat" "It means the opposite. Therefore, natural language processing that incorporates knowledge for knowledge guidance is the only way to fine and in-depth language understanding. However, where does this knowledge come from? This involves a key research problem of artificial intelligence-knowledge acquisition.

Knowledge Graph

Existing large-scale knowledge graphs, such as Wikidata, Yago, and DBpedia, are rich in massive world knowledge and stored in a structured form. As shown in the figure below, each node represents an entity in the real world, and their connecting edges mark the relationship between the entities. In this way, the relevant knowledge of American writer Mark Twain is recorded in a structured form.

Currently, this structured knowledge has been widely used in natural language processing applications such as search engines and question answering systems. However, compared with the rapidly growing amount of knowledge in the real world, the coverage of the knowledge graph is still insufficient. Because the scale of knowledge is huge and manual labeling is expensive, it is almost impossible to complete these new knowledge by manual labeling alone. In order to add richer world knowledge to the knowledge graph as quickly and accurately as possible, researchers strive to explore ways to efficiently and automatically acquire world knowledge, that is, entity relationship extraction technology.

Specifically, given a sentence and the entities appearing in it, the entity relationship extraction model needs to infer the relationship between entities based on the semantic information of the sentence. For example, given the sentence: "Tsinghua University is located in the suburbs of Beijing" and the entities "Tsinghua University" and "Beijing", the model can obtain the relationship of "location" through semantics, and finally extract the knowledge of (Tsinghua University, Beijing, Beijing) Triad.

entity relationship extraction is a classic task. It has been continuously researched and carried out in the past 20 years. Feature engineering, nuclear methods, and graph models have been widely used among them, and some stage results have been achieved. With the advent of the deep learning era, neural network models have brought new breakthroughs in entity relationship extraction.

neural network relation extraction model

has many neural network types for natural language text sequences, such as recurrent neural network (RNN, LSTM), convolutional neural network (CNN) and Transformer, etc. These models can be used for relations through appropriate transformation Extract. Initially, work [1, 2] proposed for the first time to use CNN to encode sentence semantics for relation classification, which was significantly better than non-neural network methods; work [3, 4] used RNN and LSTM for relation extraction; in addition, Work [5] proposed to use a recursive neural network to model the grammatical analysis tree of the sentence, trying to extract the semantic features while considering the lexical and syntactic features of the sentence. This idea has also been further explored by many subsequent works. Here, we list a table to summarize the effects of various typical neural networks on the benchmark test data set SemEval-2010 Task-8 [6].

As can be seen from the above table, these neural network models have achieved excellent experimental results, and there is no significant performance difference between each other. Does this mean that the problem of relation extraction is solved? In fact, this is not the case. The task of SemEval-2010 Task-8 is set to label a large number of training and test examples for pre-defined relationship categories. The examples are relatively simple short sentences, and the sample distribution of each relationship is relatively even. However, there are often many challenges in practical applications:

data scaleProblem: manually and accurately labeling sentence-level data is very expensive and requires a lot of time and manpower. In actual scenarios, for thousands of relationships, tens of millions of entity pairs, and hundreds of millions of sentences, it is almost impossible to rely on manual labeling of training data.

Learning ability problem: In actual situations, the frequency of inter-entity relationships and entity pairs often follows a long-tailed distribution, and there are a large number of relationships or entity pairs with fewer examples. The effect of the neural network model needs to rely on large-scale annotated data to ensure that there is a problem of "contradictory". How to improve the learning ability of the deep model and realize "inference from one another" is a problem to be solved in relation extraction.

Complex context problem: The existing model of mainly extracts the relationship between entities from a single sentence, and requires that the sentence must contain two entities at the same time. In fact, a large number of inter-entity relationships are often manifested in multiple sentences in a document, or even in multiple documents. How to conduct relation extraction in a more complex context is also a problem faced by relation extraction.

Open relationship problem: existing task settings generally assume that there is a set of pre-defined closed relationships, and the task is converted into a relationship classification problem. In this case, the new relationships between entities contained in the text cannot be effectively obtained. How to use deep learning models to automatically discover new relationships between entities and achieve open relationship extraction is still an "open" problem.

Therefore, there is a huge gap between the ideal setting of SemEval-2010 Task-8 and the actual scene. Only relying on neural networks to extract the semantic features of a single sentence is difficult to deal with the various complex requirements and challenges of relation extraction. We urgently need to explore more novel relation extraction frameworks, obtain larger-scale training data, have more efficient learning capabilities, be good at understanding complex document-level contextual information, and can be easily extended to open relation extraction.

We believe that these four aspects constitute the main directions for further exploration of entity relationship extraction. Next, we separately introduce the development status and challenges of these four aspects, as well as some of our thinking and efforts.

Larger scale training data

neural network relation extraction requires a lot of training data, but manual labeling of these training data is very time-consuming and expensive. In order to automatically obtain more training data to train the model, the work [16] proposed the idea of ​​Distant Supervision, which aligns the plain text with the existing knowledge graph, and can automatically label large-scale training data. The idea of ​​remote supervision of

is not complicated. Specifically: if two entities are marked as a certain relationship in the knowledge graph, then we think that all sentences containing these two entities are also expressing this relationship. Taking (Tsinghua University, Located, Beijing) as an example, we will treat all sentences that contain both entities "Tsinghua University" and "Beijing" as training examples for the relationship of "Locating". This heuristic labeling rule of

remote supervision is a double-edged sword. It is an effective strategy for automatic labeling of training data, but its excessively strong setting inevitably leads to incorrect labeling. For example, for the triple facts in the knowledge graph (Tsinghua University, President, Qiu Yong), the sentence "Qiu Yong is the president of Tsinghua University" can reflect the relationship between "President" between "Tsinghua University" and "Qiu Yong"; but the sentence "Qiu Yong was admitted to the Department of Chemistry and Chemical Engineering of Tsinghua University" and "Qiu Yong is a member of the Standing Committee of the Tsinghua University Party Committee" do not express the relationship of "principal", but will be incorrectly labeled as the relationship of "principal" by the heuristic rules of remote supervision Training examples.

Although the idea of ​​remote supervision is very simple and there are many problems, it opens a new era for collecting more training data. Inspired by this idea, many scholars actively consider how to eliminate as much as possible the interference of noise annotations in remote supervision data. Beginning in 2015, the neural relationship extraction model based on remote supervision and noise reduction mechanism has been greatly developed. Work [17] introduced a multi-instance learning method, using all instances containing the same entity pair to jointly predict the relationship between entities. Our research group Lin Yankai et al. [19] proposed a sentence-level attention mechanism, which assigns different weights to different instances to reduce the impact of noise instances. Work [20] introduced adversarial training to improve the model's resistance to noisy data. Work [21] builds aSet a reinforcement learning mechanism to filter out noise data and use the remaining data to train the model.

concluded that the existing noise reduction methods for remote supervision can take into account the robustness and effectiveness of relation extraction, and also have strong operability and practicality. However, using the existing knowledge map to align the text to obtain data to train the relation extraction model, and then using the model to extract knowledge and add it to the knowledge map, it has a flavor of chicken and egg. The text training data obtained from imperfect knowledge map alignment will also be imperfect. For those long-tailed knowledge, it is still difficult to obtain training examples through this remote supervision mechanism. How to propose a more effective mechanism to efficiently obtain high-quality, high-coverage, and high-balance training data is still a problem worthy of in-depth consideration.

More efficient learning ability

Even if high-quality training data can be automatically obtained through remote supervision and other methods, due to the long-tail distribution characteristics of relationships and entity pairs in real scenes, most of the available examples of relationships and entity pairs are still relatively small. less. Moreover, there are also very limited samples available for issues that are limited by the scale of data for specialized relationships in professional fields such as healthcare and finance. As a typical data-hungry technology, the neural network model will greatly affect its performance when there are too few training examples. Therefore, researchers hope to explore ways to effectively improve the learning ability of the model in order to make better use of limited training examples to achieve satisfactory extraction performance.

In fact, humans can learn knowledge quickly through a small number of samples, and have the ability to draw inferences from one another. In order to explore the ability of deep learning and machine learning to learn by analogy, a Few-shot learning task is proposed. By designing a small learning mechanism, the model can use the generalized knowledge learned from the past data, combined with a small number of training samples of new types of data, to achieve rapid migration learning, and has a certain inference ability.

In the past few learning researches were mainly concentrated in the field of computer vision, and there was little exploration in the field of natural language processing. The work of our research group Han Xu et al. [21] first introduced less-relational learning to relation extraction, and constructed a lesser-relational extraction data set FewRel, hoping to promote less-relational learning research that drives natural language processing, especially relational extraction tasks. As shown in the figure below, the relationship extraction problem only provides a very small number of examples (such as 3-5) for each relationship, and it is required to improve the relationship classification effect on the test examples as much as possible.

FewRel initially tried several representative learning methods including metric learning (Metric learning), meta learning (Meta learning), parameter prediction (Parameter prediction), etc. The evaluation showed that even the best prototype network (Prototypical Networks) ) Model, the performance of the few-time relation extraction is still far from human performance.

In order to better solve the problem of less learning in remote supervised relationship extraction, Gao Tianyu of our research group, etc. [22] proposed a prototype network based on a hybrid attention mechanism, considering both the instance-level and feature-level attention mechanisms , While reducing the impact of noise annotation, it can better pay attention to the useful features in the sentence, and achieve efficient and less learning. Work [23] proposed a multi-level matching and integration structure, fully learned the potential associations between training examples, and explored the potential information in the few examples as much as possible. Work [24] used the pre-trained language model BERT to deal with the problem of less learning in relation extraction. The BERT based on massive unsupervised data training can provide effective semantic features for the less learning model, which is obtained on FewRel data. Exceeds the level of human relations classification.

In the process of extracting and exploring the learning relationship of few times, Gao Tianyu of the research group further discovered two aspects that have been neglected for a long time [25]: To use the few times learning model in a production environment, it should have abundant resources. The ability to migrate domains to low-resource domains should also be able to detect whether the sentence really expresses a certain predefined relationship or does not express any relationship. For this reason, they proposed FewRel 2.0, which added the following two major challenges to the original data set FewRel: domain adaptation) and “none-of-the-above detection”.

For the challenge of domain migration, FewlRel 2.0 collects and annotates a large amount of data in the medical field. After the relationship extraction model is trained on the original corpus, it can also learn a few times on these new domain corpus. For the "none of the above" detection, FewRel 2.0 adds a "none of the above" option to the original N-way K-shot setting (given N new types, and K training samples for each type). Increase the difficulty of classification and detection. Initial experiments of

found that previous effective models (including BERT-based models) have significant performance degradation in these two challenging tasks. Although Gao Tianyu and others also tried some possible solutions in the FewRel 2.0 paper: For example, they tried classic adversarial learning methods for domain migration, and the performance of the model was improved to a certain extent; for the detection of “none of the above”, proposed based on The BERT-PAIR model of BERT next sentence prediction task can achieve a little effect in the "none of the above" challenge. However, these two major challenges still require more innovation and exploration.

In summary, it is still a very emerging research direction to explore the extraction of learning relations with less learning so that the relation extraction model has more powerful and efficient learning capabilities, especially the problem of less learning for relation extraction, and less learning in other fields. Compared with the problem, it has its own unique characteristics and challenges. Whether it is an improvement suitable for NLP and relation extraction based on the existing few-time learning technology, or a new low-time learning model suitable for relation extraction, it will maximize the use of a small amount of labeled data and promote the practical application of the relationship extraction technology. .

More complex text context

Existing relation extraction work mainly focuses on sentence-level relation extraction, that is, relation extraction based on sentence information. Various neural network models are also good at encoding sentence-level semantic information, which can be obtained in many public evaluation data best effect. In actual scenarios, a large number of inter-entity relationships are expressed through multiple sentences. As shown in the figure below, multiple entities are mentioned in the text and show complex interrelationships. According to statistics of manual annotation data sampled from Wikipedia, at least 40% of entity relationship facts can only be obtained jointly from multiple sentences. In order to achieve cross-sentence relationship extraction among multiple entities, multiple sentences in the document need to be read and reasoned, which is obviously beyond the capability of sentence-level relationship extraction methods. Therefore, it is imperative to extract document-level relationships.

Document-level relation extraction research requires large-scale manual annotation data sets for training and evaluation. Currently, there are few document-level relational extraction data sets. Work [26, 27] constructed two remotely supervised data sets, and the evaluation results are not reliable because they are not manually labeled. BC5CDR [28] is a document-level relationship extraction data set that is manually annotated. It is composed of 1,500 PubMed documents. It is a specific field of biomedicine and only considers the relationship of "chemically induced diseases". It is not necessarily suitable for exploring document-level relationship extraction. The universal method. Work [29] proposed the use of reading comprehension technology to answer questions to extract entity relationship facts from documents. These questions are transformed from "entity-relationship" pairs. Since this working data set is tailored for this method, it is not so suitable for exploring general methods of document-level relation extraction. These data sets either only have a small number of manually labeled relationships and entities, or have noisy labels from remote supervision, or serve a specific field or method, and have one or another limitation.

In order to promote the research of document-level relationship extraction, the research group Yao Yuan et al. [30] proposed the DocRED dataset, which is a large-scale manually annotated document-level relationship extraction dataset based on Wikipedia text and WikiData knowledge graph. , Contains 5,053 Wikipedia documents, 132,375 entities and 53,554 entity relationship facts. It is the largest manually annotated document-level relationship extraction data set. As shown in the figure below, the document-level relationship extraction task requires the model to have strong pattern recognition, logical reasoning, referential reasoning, and common sense reasoning capabilities. These aspects urgently need more long-term research and exploration.

More open relationship types

Existing relationship extraction work generally assumes that there is a predefined set of closed relationships, and the task is transformed into a relationship classification problem. However, in the real relationship extraction scenario in the open domain, the text contains a large number of open entity relationships, and there are many types of relationships, and the number of relationships will continue to grow, far exceeding the number of artificially defined relationship types. In this case, the traditional relationship classification model cannot effectively obtain the new relationships between entities contained in the text. How to use deep learning models to automatically discover new relationships between entities and achieve open relationship extraction is still an "open" problem.

In order to achieve open relation extraction for the open field, the task of Open Relation Extraction (Open RE) is researched and proposed, which is dedicated to extracting arbitrary relation facts between entities from open text. Open relation extraction involves three aspects of "opening": first, it is the openness of extracting relation types, which is different from traditional relation extraction, it hopes to extract all known and unknown relations; second is the opening of test corpus, such as news, medical and other fields , Its texts have different characteristics and need to explore cross-domain robust algorithms; the third is the openness of training corpus. In order to obtain the best possible open relation extraction model, it is necessary to make full use of various existing annotation data, including fine annotation , Remote supervision of labeled data, etc., and the relationship definition and distribution of different training data sets are also different, and it is necessary to use multiple sources of data at the same time.

In the pre-deep learning era, researchers also had the task of exploring Open Information Extraction (OpenIE). Open relation extraction can be regarded as a special case of OpenIE. At that time, OpenIE was mainly implemented through unsupervised statistical learning methods, such as the Snowball algorithm. Although these algorithms have good robustness to different data, their accuracy is often low, and they are still far from practical.

Recently, the work of Wu Ruidong and others in the research group [31] proposed a supervised open relationship extraction framework, which can realize the free switching between supervised and weak supervision modes through the "Relation Siamese Network" (RSN). Thus, the supervised data of the predefined relationship and the unsupervised data of the new relationship in the open text can be used at the same time to jointly learn the semantic similarity of different relationship facts. Specifically, the relational twin network RSN adopts the twin network structure, and learns the deep semantic features and mutual semantic similarity of relation samples from the annotation data of predefined relations, which can be used to calculate the semantic similarity of texts containing open relations. Moreover, RSN also uses two semi-supervised learning methods, conditional entropy minimization and virtual confrontation training, to further use unlabeled open relation text data to further improve the stability and generalization ability of open relation extraction. Based on the open relationship similarity calculation result calculated by RSN, the model can cluster the text relationship in the open domain, thereby inducing new relationships. The work of Gao Tianyu and others in the research group of

[32] from another perspective, for a specific new relationship in the open domain, only a small number of precise examples need to be provided as seeds, and the pre-trained relationship twin network can be used to snowball (Neural SnowBall), summarized more examples of this new relationship from a large amount of unlabeled text, and continuously iteratively trained a relationship extraction model suitable for the new relationship.

concludes that open-domain relation extraction has achieved some results in the pre-deep learning era, but how to combine the advantages of neural network models in the deep learning era to effectively expand the generalization capabilities of neural network relation extraction models is worth more in-depth exploration .

Summary

In order to expand the knowledge graph more timely, it has become the only way to automatically acquire new world knowledge from massive data. Knowledge acquisition technology represented by entity relationship extraction has achieved some results, especially in recent years, deep learning models have greatly promoted the development of relationship extraction. However, compared with the requirement of extracting complex challenges from the relationship of actual scenes, the existing technology still has greater limitations. We urgently need to start from the actual needs of the scene to solve the challenges of training data acquisition, less learning ability, complex text context, open relationship modeling, etc.Establishing an effective and robust relationship extraction system is also the direction that the entity relationship extraction task needs to continue to work hard on.

Our research team has been working on entity relationship extraction tasks since 2016. There are many students such as Lin Yankai, Han Xu, Yao Yuan, Zeng Wenyuan, Zhang Zhengyan, Zhu Hao, Yu Pengfei, Yu Zhijingcheng, Gao Tianyu, Wang Xiaozhi, Wu Ruidong, etc. Carried out research work. Last year, thanks to the efforts of Han Xu and Gao Tianyu, the OpenNRE toolkit [33] was released. After nearly two years of continuous improvement, it covers supervised relationship extraction, remote supervision relationship extraction, less learning relationship extraction, and document-level relationships. Rich scenes such as extraction. In addition, a large amount of research funds are also spent on annotating data sets such as FewRel (1.0 and 2.0) and DocRED, aiming to promote research in related directions.

This article summarizes our understanding of the current situation, challenges, and future development directions of entity relationship extraction, as well as our efforts in these areas. I hope that it will arouse everyone's interest and help everyone. We look forward to more scholars and students joining in this field of research. Of course, this article does not mention an important challenge, that is, the knowledge acquisition of complex structures represented by event extraction. We will have the opportunity to discuss this in the future.

is limited to personal level, it is inevitable that there will be biases and mistakes. Please feel free to point out in the comments and we will strive to improve. It should be noted that we did not intend to write this article as a rigorous academic paper, so we did not explain all the work in each direction clearly. If there are important omissions, please criticize and correct.

advertising time

, if you want to find internships or work in natural language processing and knowledge graphs, please contact Patrick Li (patrickpli@tencent.com), let us work together to develop interesting and meaningful research work.

About the author

Han Xu is a third-year PhD student in the Department of Computer Science and Technology, Tsinghua University. His main research directions are natural language processing, knowledge graphs, and information extraction. He has published many papers at internationally renowned conferences in the field of artificial intelligence, AAAI, ACL, EMNLP, COLING, NAACL, and is one of the developers of open source projects such as OpenKE and OpenNRE. Homepage: https://thucsthanxu13.github.io/

Gao Tianyu, a senior undergraduate in the Department of Computer Science, Tsinghua University, whose main research directions are natural language processing, knowledge graphs, and relation extraction. He has published many papers at internationally renowned conferences in the field of artificial intelligence AAAI and EMNLP, and is one of the main developers of open source projects such as OpenNRE. Homepage: https://gaotianyu.xyz/about/

Liu Zhiyuan, associate professor and doctoral supervisor of the Department of Computer Science, Tsinghua University. The main research directions are representation learning, knowledge graphs and social computing. Homepage: http://nlp.csai.tsinghua.edu.cn/~lzy/

References

[1] ChunYang Liu, WenBo Sun, WenHan Chao, Wanxiang Che. Convolution Neural Network for Relation Extraction. The 9th International Conference on Advanced Data Mining and Applications (ADMA 2013).

[2] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, Jun Zhao. Relation Classification via Convolutional Deep Neural Network. The 25th International Conference on Computational Linguistics (COLING 2014).

[3] DongxuZhang, Dong Wang. Relation Classification via Recurrent Neural Network. arXiv preprint arXiv: 1508.01006 (2015).

[4] Peng Zhou, Wei Shi, Jun Tian, ​​Zhenyu Qi, Bingchen Li, Hongwei Hao, Bo Xu. Attention-Based Bidirectional Long Short-Term Memory Networks for Relation Classification. The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).

[5] Richard Socher, Brody Huval, Christopher D. Manning, Andrew Y. Ng. Semantic Compositionality through Recursive Matrix -Vector Spaces. The 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL 2012).

[6] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid ó Séaghdha, Sebastian Padó , Marco Pennacchiotti, Lorenza Romano, Stan Szpakowicz. SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals. The 5th International Workshop on Semantic Evaluation (SEMEVAL 2010).

[7] Thien Huu Nguyen, Ralph Grishman. Relation Extraction: Perspective from Convolutional Neural Networks. The 1st Workshop on Vector Space Modeling for Natural Language Processing (LatentVar 2015).

[8] Cícero dos Santos, Bing Xiang, Bowen Zhou. Classifying Relations by Ranking with Convolutional Neural Networks. The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015).

[9] Shu Zhang, Dequan Zheng, Xinchen Hu, Ming Yang. Bidirectional Long Short-Term Memory Networks for Relation Classification. The 29th Pacific Asia Conference on Language, Information and Computation (PACLIC 2015).

[10] Minguang Xiao, Cong Liu. Semantic Relation Classification via Hierarchical Recurrent Neural Network with Attention. The 26th International Conference on Computational Linguistics (COLING 2016).

[11] Kun Xu, Yansong Feng, Songfang Huang, Dongyan Zhao. Semantic Relation Classification via Convolutional Neural Networks with Simple Negative Sampling. The 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).

[12] Yan Xu, Lili Mou, Ge Li, Yunchuan Chen, Hao Peng, Zhi Jin. Classifying Relations via Long Short Term Memory Networks along Shortest Dependency Paths. The 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).

[13] Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, Houfeng Wang. A Dependency-Based Neural Network for Relation Classification. The 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2015).

[14] Yan Xu, Ran Jia, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, Zhi Jin. Improved Relation Classification by Deep Recurrent Neural Networks with Data Augmentation. The 26th International Conference on Computational Linguistics (COLING 2016).

[15] Rui Cai, Xiaodong Zhang, Houfeng Wang. Bidirectional Recurrent Convolutional Neural Network for Relation Classification. The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).

[16] Mike Mintz, Steven Bills, Rion Snow, Daniel Jurafsky. Distant Supervision for Relation Extraction without Labeled Data. The 47th Annual Meeting of the Association for Computational Linguistics and the 4th International Joint Conference on Natural Language Processing (ACL-IJCNLP 2009).

[17] Daojian Zeng, Kang Liu, Yubo Chen, Jun Zhao. Distant Supervision for Relation Extraction via Piecewise Convolutional Neural Networks. The 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP 2015).

[18] Yankai Lin, Shiqi Shen, Zhiyuan Liu, Huanbo Luan, Maosong Sun. Neural Relation Extraction with Selective Attention over Instances. The 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016).

[19] Yi Wu, David Bamman, Stuart Russell. Adversarial Training for Relation Extractio n. The 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017).

[20] Jun Feng, Minlie Huang, Li Zhao, Yang Yang, Xiaoyan Zhu. Reinforcement Learning for Relation Classification from Noisy Data. The 32th AAAI Conference on Artificial Intelligence (AAAI 2018).

[21] Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, Maosong Sun. FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of- the-Art Evaluation. The 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018).

[22] Tianyu Gao, Xu Han, Zhiyuan Liu, Maosong Sun. Hybrid Attention-based Prototypical Networks for Noisy Few-Shot Relation Classification. The 33th AAAI Conference on Artificial Intelligence (AAAI 2019).

[23] Zhi-Xiu Ye, Zhen-Hua Ling. Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification. The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

[ 24] Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, Tom Kwiatkowski. Matching the Blanks: Distributional Similarity for Relation Learning. The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

[25] Tianyu Gao, Xu Han, Hao Zhu, Zhiyuan Liu, Peng Li, Maosong Sun, Jie Zhou. FewRel 2.0: Towards More Challenging Few-Shot Relation Classification. 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Langua ge Processing (EMNLP-IJCNLP 2019).

[26] Chris Quirk, Hoifung Poon. Distant Supervision for Relation Extraction beyond the Sentence Boundary. The 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2017).

[27 ] Nanyun Peng, Hoifung Poon, Chris Quirk, Kristina Toutanova, Wen-tau Yih. Cross-Sentence N-ary Relation Extraction with Graph LSTMs. Transactions of the Association for Computational Linguistics (TACL 2017).

[28] Chih-Hsuan Wei , YifanPeng, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Jiao Li, Thomas C. Wiegers, Zhiyong Lu. Overview of the BioCreative V Chemical Disease Relation (CDR) Task. The 5th BioCreative Challenge Evaluation Workshop (BioC 2015).

[29] Omer Levy, Minjoon Seo, Eunsol Choi, Luke Zettlemoyer. Zero-Shot Relation Extraction via Reading Comprehension. The 21st Conference on Computational Natural Language Learning (CoNLL 2017).

[30] Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, Maosong Sun. DocRED: A Large-Scale Document-Level Relation Extraction Dataset. The 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019).

[31] Ruidong Wu, Yuan Yao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun. Open Relation Extraction: Relational Knowledge Transfer from Supervised Data to Unsupervised Data. 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Con ference on Natural Language Processing (EMNLP-IJCNLP 2019).

[32] Tianyu Gao, Xu Han, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, Maosong Sun. Neural Snowball for Few-Shot Relation Learning. The 34th AAAI Conference on Artificial Intelligence (AAAI 2020).

[33] Xu Han, Tianyu Gao, Yuan Yao, Deming Ye, Zhiyuan Liu, Maosong Sun. OpenNRE: An Open and Extensible Toolkit for Neural Relation Extraction. The Conference on Empirical Methods in Natural Language Processing (EMNLP 2019).

largeEveryone is watching:

AAAI2020 is released on the list, and the reviewers are crazy! Look ahead to the eight selected papers!

ICCV2019 | Contempt proposes a lightweight target detection network ThunderNet