Contradiction Detection

Discover how vector similarity expands data exploration beyond full-text search. Explore diversity sampling and more for enhanced data discovery!

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Requirements documents can contain several thousand individual requirements. They must be error-free to avoid unnecessary complications and costs in the later product development stages. An important part of this is to identify contradictions between two requirements. The first step is therefore to define what contradictions are and in what form they can occur in requirement documents. In this paper the scientific theories regarding contradictions are discussed, concerning to their usefulness for the topic. In doing so, the Aristotelian Logic proved to provide the best basis for an application in the Requirements Engineering context. Based on this theory, we have created specific subtypes of contradictions to match them to the requirements engineering field. The identification of these subtypes is done by a formalization of the requirement sentences and a subsequent analysis by means of simple questions. To validate the method, industrial requirement documents were searched for contradictions. For each detected type of contradiction, we present an example of the detection process. Thereby, we show that the method is easy to apply and may also be used by non-specialists. Thus, our method provides a taxonomy as a basis for further research on automated contradiction detection as well as on automated quality analysis of requirements documents.

In recent times, large language models (LLMs) have shown impressive performance on various document-level tasks such as document classification, summarization, and question-answering. However, research on understanding their capabilities on the task of self-contradictions in long documents has been very limited. In this work, we introduce ContraDoc, the first human-annotated dataset to study self-contradictions in long documents across multiple domains, varying document lengths, self-contradictions types, and scope. We then analyze the current capabilities of four state-of-the-art open-source and commercially available LLMs: GPT3.5, GPT4, PaLM2, and LLaMAv2 on this dataset. While GPT4 performs the best and can outperform humans on this task, we find that it is still unreliable and struggles with self-contradictions that require more nuance and context. We release the dataset and all the code associated with the experiments (https://github.com/ddhruvkr/CONTRADOC).

Already since the 1950s TRIZ shows that patents and the technical contradictions they solve are an important source of inspiration for the development of innovative products. However, TRIZ is a heuristic based on a historic patent analysis and does not make use of...

Maren Pielka, Svetlana Schmidt, and Rafet Sifa, “Generating Prototypes for Contradiction Detection Using Large Language Models and Linguistic Rules,” IEEE Xplore, Dec. 2023, doi: 10.1109/bigdata59044.2023.10386499.

We introduce a novel data generation method for contradiction detection, which leverages the generative power of large language models as well as linguistic rules. Our vision is to provide a condensed corpus of prototypical contradictions, allowing for in-depth linguistic analysis as well as efficient language model fine-tuning. To this end, we instruct the generative models to create contradicting statements with respect to descriptions of specific contradiction types. In addition, the model is also instructed to come up with completely new contradiction typologies. As an auxiliary approach, we use linguistic rules to construct simple contradictions such as those arising from negation, antonymy and numeric mismatch. We find that our methods yield promising results in terms of coherence and variety of the data. Further studies, as well as manual refinement are necessary to make use of this data in a machine learning setup.

A minimal example to demonstrate how LLM's can assist in detecting contradictions in documents. - datarootsio/knowledgebase_guardian

In the evolving landscape of intellectual property, extracting meaningful insights from patents is a challenging yet critical task. My recent analysis delved into how a large language model (LLM) can…

This paper introduces ALICE (Automated Logic for Identifying Contradictions in Engineering), a novel automated contradiction detection system tailored for formal requirements expressed in controlled natural language. By integrating formal logic with advanced large language models (LLMs), ALICE represents a significant leap forward in identifying and classifying contradictions within requirements documents. Our methodology, grounded on an expanded taxonomy of contradictions, employs a decision tree model addressing seven critical questions to ascertain the presence and type of contradictions. A pivotal achievement of our research is demonstrated through a comparative study, where ALICE’s performance markedly surpasses that of an LLM-only approach by detecting 60% of all contradictions. ALICE achieves a higher accuracy and recall rate, showcasing its efficacy in processing real-world, complex requirement datasets. Furthermore, the successful application of ALICE to real-world datasets validates its practical applicability and scalability. This work not only advances the automated detection of contradictions in formal requirements but also sets a precedent for the application of AI in enhancing reasoning systems within product development. We advocate for ALICE’s scalability and adaptability, presenting it as a cornerstone for future endeavors in model customization and dataset labeling, thereby contributing a substantial foundation to requirements engineering.