Automated requirement contradiction detection through formal logic and LLMs - Automated Software Engineering
This paper introduces ALICE (Automated Logic for Identifying Contradictions in Engineering), a novel automated contradiction detection system tailored for formal requirements expressed in controlled natural language. By integrating formal logic with advanced large language models (LLMs), ALICE represents a significant leap forward in identifying and classifying contradictions within requirements documents. Our methodology, grounded on an expanded taxonomy of contradictions, employs a decision tree model addressing seven critical questions to ascertain the presence and type of contradictions. A pivotal achievement of our research is demonstrated through a comparative study, where ALICE’s performance markedly surpasses that of an LLM-only approach by detecting 60% of all contradictions. ALICE achieves a higher accuracy and recall rate, showcasing its efficacy in processing real-world, complex requirement datasets. Furthermore, the successful application of ALICE to real-world datasets validates its practical applicability and scalability. This work not only advances the automated detection of contradictions in formal requirements but also sets a precedent for the application of AI in enhancing reasoning systems within product development. We advocate for ALICE’s scalability and adaptability, presenting it as a cornerstone for future endeavors in model customization and dataset labeling, thereby contributing a substantial foundation to requirements engineering.
Contradictory opposites, such as he is sick and he is not sick, are mutually exclusive; one must be true if the other is false, without overlap.
Contrary opposites, like it is black and it is white, cannot be true at the same time but can both be false, indicating they are not exhaustive.
Subaltern relation follows, where if everybody is sick is true, it implies some people are sick must also be true, demonstrating a logical step-down.
Simplex contradictions (from Latin ‘simple’) are characterized by direct opposition without conditional statements. These contradictions are straightforward but crucial for establishing our classification framework. For example, ‘The car must be red’ versus ‘The car must be blue’ showcases apparent, uncomplicated contradictions.
Idem contradictions (from Latin ‘same’) involve identical conditions leading to contradictory outcomes, presenting challenges due to their conditional nature. An example is ‘If the customer wishes, the car must be red’ and ‘If the customer wishes, the car must be blue,’ where the same condition yields conflicting requirements.
Alius contradictions (from Latin ‘different’), distinguished by differing conditions that result in incompatible conclusions, illustrate the complexity of engineering requirements. An instance of this is ‘If the customer wishes, the car must be red’ versus ‘If the car has four doors, the car must be blue,’ demonstrating how different conditions can lead to contradictory outcomes.