Extracting and reasoning with structured information in natural language and beyond

Abstract

By discovering patterns and structure in vast unstructured data, we can organise and better leverage it to solve complex tasks. This thesis investigates the crucial role of structured information in natural language processing and artificial intelligence, with a focus on its extraction, utilisation, and extension to multimodal reasoning. We address three key research questions: (1) How can we develop more generalisable and comprehensive structure prediction models? (2) How can we leverage structured information with reasoning to effectively solve complex problems? (3) How should we evaluate the ability of multimodal models to reason over visual structures?

To answer these questions, we present a series of novel approaches and contributions. We first introduce a method for generalising relation triplet extraction to unseen relations. We then propose a cube-filling model for hyper-relational extraction, supporting more comprehensive structure prediction. To leverage structured information in complex reasoning tasks, we develop a framework that augments large language models with diverse knowledge sources, mitigating error propagation and hallucination. We further enhance language model reasoning by proposing a training framework that explores the branching structure of diverse reasoning paths. Extending beyond natural language, we construct a benchmark of multimodal puzzles based on abstract concepts and visual patterns to systematically evaluate the reasoning capabilities of multimodal models.

Our research contributes to the field by improving structure prediction, enhancing reasoning capabilities, and bridging the gap between structured information and complex problem-solving across multiple modalities. These advancements pave the way for more sophisticated systems capable of understanding and reasoning about the complex, multimodal world around us, while also addressing critical challenges such as generalisability, comprehensive structure representation, and robust reasoning processes.

Speaker’s Profile

Yew Ken Chia is under the EDB-Industrial Postgraduate Program (EDB-IPP) between Singapore University of Technology and Design (SUTD) and Alibaba DAMO Academy, under the supervision of Prof. Soujanya Poria and Dr. Lidong Bing. He received his B.Eng. degree from SUTD in 2020. His research focuses on information extraction, large language model reasoning and multimodal understanding.