DSPy: Building a Logical Reasoner

Let’s build a logical reasoner using DSPy.

The principle behind the reasoner is simple. Given a text, it first extracts facts and categorizes them into premises and conclusions.

• Premises are statements or facts that an argument claims will support its conclusion. They are the reasons given for believing the conclusion to be true.
• Conclusions are statements that the premises are supposed to support or prove. It’s what the speaker or writer is trying to convince you to believe based on the premises provided.

Identifying premises and conclusions correctly is crucial. It involves understanding the structure of arguments, which can be explicit or implicit. Training in logic, critical thinking, or argumentation theory can enhance this skill.

After identifying premises and conclusions, the reasoner has to check the validity and soundness of the argument made.

• Validity concerns whether the logical structure of the argument is such that if the premises are true, then the conclusion must also be true. It’s a test of the argument’s form.
• Soundness is about the truth of the premises themselves. An argument is sound if it is both valid and its premises are true.

Code implementation

The first step, obviously, is to configure the language model we will use. Since we are going to do logical reasoning, it is wise to use GPT-4. It has better ‘reasoning’ capabilities than GPT-3.5-turbo.

import dspy
import os
from dotenv import load_dotenv

load_dotenv()

llm = dspy.OpenAI(
    model='gpt-4',
    api_key=os.environ['OPENAI_API_KEY'],
    max_tokens=100
)

dspy.settings.configure(lm=llm)

The second step is to define the check_logic signature that will help us verify that the conclusions extracted from the input text are coherent with the premises extracted.

class check_logic(dspy.Signature):
    """Given the premises and the conclusions of an argument, check if the conclusions are logical."""
    premises = dspy.InputField(desc="Premises of the argument")
    conclusions = dspy.InputField(desc="Conclusion of the argument")
    logical = dspy.OutputField(desc="Given the premises, the conclusion is logical or not")

Finally, we define the logical_reasoner module and test it on an example. As described previously, the logical_reasoner first extracts premises and conclusions. It then checks if conclusions are coherent with premises.

class logical_reasoner(dspy.Module):
    def __init__(self):
        super().__init__()
        self.logical_reasoner = dspy.Predict("text -> premises, conclusions")
        self.checker = dspy.ChainOfThought(check_logic)

    def forward(self,text):
        prediction = self.logical_reasoner(text=text)
        result = self.checker(premises=prediction.premises, conclusions=prediction.conclusions)
        return result

text="If it is raining, then the grass is wet. The grass is wet. Therefore, it is not raining."

lr = logical_reasoner()

print(lr(text=text))

As a result, our logical_reasoner correctly detects the logical issue in the text.

Prediction(
    rationale='produce the logical. We know from the first premise that if it is raining, then the grass is wet. However, the second premise only tells us that the grass is wet. It does not tell us why the grass is wet. There could be other reasons for the grass being wet, such as someone watering the grass or dew from the morning. Therefore, we cannot conclude that it is not raining just because the grass is wet.',
    logical='Given the premises, the conclusion is not logical.'
)

Second code implementation

In the first code implementation, we did not check whether or not each premise was sound in the first place. False premises can still logically lead to a conclusion. But that conclusion will be false since the premises are false.

Here is the code to also check the soundness of premises. If you understood what we did previously, it should be self-explanatory.

import dspy
import os
from dotenv import load_dotenv

load_dotenv()

llm = dspy.OpenAI(
    model='gpt-4',
    api_key=os.environ['OPENAI_API_KEY'],
    max_tokens=100
)

dspy.settings.configure(lm=llm)

class check_logic(dspy.Signature):
    """Given the premises and the conclusions of an argument, check if the conclusions are logical."""
    premises = dspy.InputField(desc="Premises of the argument")
    conclusions = dspy.InputField(desc="Conclusion of the argument")
    sound = dspy.InputField(desc="The premises are sound or not")
    logical = dspy.OutputField(desc="Given the premises, the conclusion is logical or not")


class check_premises(dspy.Signature):
    """Given the premises of an argument, check if the premises are sound."""
    premises = dspy.InputField(desc="Premises of the argument")
    sound = dspy.OutputField(desc="The premises are sound or not")

class logical_reasoner(dspy.Module):
    def __init__(self):
        super().__init__()
        self.logical_reasoner = dspy.Predict("text -> premises, conclusions")
        self.check_prems = dspy.Predict(check_premises)
        self.checker = dspy.ChainOfThought(check_logic)

    def forward(self,text):
        prediction = self.logical_reasoner(text=text)
        sound = self.check_prems(premises=prediction.premises)
        result = self.checker(premises=prediction.premises, conclusions=prediction.conclusions, sound=sound.sound)
        return result

text="If it is raining, then the grass is wet. The grass is wet. Therefore, it is raining."

lr = logical_reasoner()

print(lr(text=text))

If you want to learn more about DSPy, there are amazing resources on www.lycee.ai, the first LMS focused on AI content.

Here are two courses that can help you get started:

1. Start learning how to program language models using DSPy. Here's the perfect course for that.
2. After mastering the basics, explore the advanced use cases of DSPy. Here is the ideal course for that.

Happy coding !!