Natural Language Application Team

Principal Investigator: Professor Wen-Lian Hsu

—

Summary

Our goal of developing intelligent learning assistant (ILA) for primary school math word problem (MWP) is to build an example of explainable AI (XAI). Our ILA solves a MWP and produces step-by-step explanations. If ILA makes a mistake, we can easily unravel the error. When ILA solves a problem correctly, the explanation would allow students to understand the logic behind it. We adopt a statistical principle-based approach (SPBA) that preserves the advantages of both statistical machine learning (SML) and rule-based (RB) methods while avoiding their pitfalls. Our robot learns to solve MWP in a way similar to human learning. Although MWP involves limited domain knowledge and simpler sentential structure, major components of natural language (NL) understanding (NLU), such as anaphora resolution, ellipsis, fronting, entailment, parsing, ontology construction, and inference are all necessary to be incorporated. Hence, after solving MWP, we would have also built fundamental modules for many NLU applications, such as the dialogue system. Currently, we have achieved 80% of our goal, and are striving for semi-automatic ontology construction and natural language generation.

Keywords

Statistical Principle-based Approach (SPBA), Natural Language Processing, Chatbot, Virtual Assistant, Language Understanding, Intelligent Agents.

Innovations

Unlike most ML systems which exhibit an F-score around mid-80 percent, pattern recognition of MWP normally requires an F-score in the high-90’s in order to lay down logical and legible explanations. Furthermore, entailment and inference are crucial in constructing an all-in-one MWP system. To achieve this, a lot of common sense knowledge is required, and new techniques for extracting such knowledge and making use of it need to be developed. We have arranged such knowledge in verb and noun frames, including ontologies, and various equivalent descriptions in our knowledge-base and inference system, InfoMap. Below, we briefly describe our innovations.

1. Knowledge acquisition. The main goal is to learn ontologies semi-automatically. Words in an ontology of another word are usually collocated with that word. Via deep learning, we attempt to learn NN, NV, AN, and noun-classifier pairs. Currently, we have achieved 95% accuracy for noun-classifier (NC) pair, and 90% for the other pairs. NC pairs are crucial in the counting problem of MWP. Collocation pairs are the backbone for resolving many ambiguity problems in NL.
2. Anaphora and ellipsis resolution. We have resolved anaphora and ellipsis in MWP and reconstructed topic-fronting specific to Chinese using collocation pairs.
3. NL scripts. NL scripts are important for our NL agents. Instead of formal logic representations, we have adopted NL entailment and solution scripts to produce comprehensible answers for students.
4. MWP dialogue system. After solving an MWP, our system can take further questions, such as those regarding different aspects of the problem, as well as questions that alter parameters of the problem. In the future, we shall work on questions raised by students who encounter difficulties.
5. NL generation. Given a sentence, we can generate ontological instances. We attempt to generate NL sentences by inverting the above process through ontological instances. This step is crucial to question answering and chatbot response generation.

Early next year, we will complete the Chinese MWP solution system and extend it to English.

Benefits

1. Knowledge accumulation. Unlike the parameters learned in SML systems and their isolated modules, our system learns an ontological knowledge, which can be categorized, accumulated, and reused similar to text books. It can even be used according to the grade level of the student. Moreover, our core program can be easily extended to other chatbot systems to achieve deep understanding of domain knowledge provided an ontology is available.
2. Narration. Many existing MWP solution systems employ logical forms for the computer to perform theorem proving. Traditionally, the result produced by an automatic theorem proving system, e.g. in trigonometry, is very difficult for human to comprehend. Since we are designing a system for primary school students, it is essential to provide solutions similar to those written by humans with many alternative descriptions. Besides, the inference in MWP is not too complex, so simple entailment relations should suffice. Currently, we attempt to generate NL sentences as mentioned above. In the future, such a process will be further extended to generate paragraphs and stories.
3. NL agents. Our MWP solution agent is a NL agent built based on SPBA. In its core, the InfoMap kernel employs entailment script, semantic frames, and instance maps. By means of this kernel, we can match natural language sentences via semantic frames to concepts in MWPs and form instance maps. Then, we follow solution scripts to utilize question frames and concepts to simulate the solution process of humans. In a step-by-step fashion, we capture key concepts and calculate the correct answer. In the future, we will continue to provide detailed instructions to students in order to help them identify their blind spots, and supply hints in a chatbot-style. We also plan to incorporate such knowledge and build a general language education system. It will go beyond simple grammar checker and offer paraphrasing or writing suggestions, in order to substantially enhance the student’s writing ability.