Skip to content

Instantly share code, notes, and snippets.

@alexandreteles
Created February 16, 2024 17:24
Show Gist options
  • Save alexandreteles/d80bd05608d1cc3fbcf1549600388bb6 to your computer and use it in GitHub Desktop.
Save alexandreteles/d80bd05608d1cc3fbcf1549600388bb6 to your computer and use it in GitHub Desktop.
Comparing SOV Extraction between models:

Original text:

The Yoga Sutras of Patanjali are in themselves exceedingly brief, less than ten pages of large type in the original. Yet they contain the essence of practical wisdom, set forth in admirable order and detail. The theme, if the present interpreter be right, is the great regeneration, the birth of the spiritual from the psychical man: the same theme which Paul so wisely and eloquently set forth in writing to his disciples in Corinth, the theme of all mystics in all lands.

Used prompt:

You are an expert entity extractor that always maintains as much semantic meaning as possible. You use inference or deduction whenever necessary to supply missing or omitted data. Examine the provided data, text, or information and generate a list of any entities or objects that match the requested format.
# Additional Instructions
1. Carefully read the provided data, text, or information to understand the context and the dynamics between entities.
2. Identify all subjects, verbs, and objects within the sentences, keeping an eye out for complex sentences that might contain multiple entities or actions.
3. Extract all Subject-Verb-Object (SVO) triples from each sentence or statement. Ensure to include all possible combinations in sentences with multiple subjects, verbs, or objects, creating separate triples for each variation.
4. When you encounter pronouns (e.g., he, she, they) or other words that do not clearly name an entity but refer to one mentioned in the text, substitute the pronoun or vague term with the correct, specific name of the entity it refers to. Use the context provided in the text to accurately determine to which entity the pronoun or term is referring. For example, if a sentence says "They offer great value," and "they" refers to "The new smartphones launched," then replace "they" with "The new smartphones launched" in your SVO triple.
5. If a sentence contains conjunctions like "and" or "or" that link subjects, verbs, or objects, split these into separate entities for your SVO triples.
6. Use your inference skills to complete the triples when the sentence structure is implicit, or some SVO parts are omitted but can be inferred from the context. This includes continuing to utilize specific names for entities rather than pronouns when the subject or object refers back to something previously mentioned.
7. In the case of compound verbs or objects where a series of actions or objects are listed for a single subject, create a distinct triple for each action or object linked to the subject. 
8. Consider indirect objects or prepositional phrases as part of the object in your SVO triples if they add significant meaning to the overall action described.
9. For verbs that are part of phrasal verbs or require prepositions to convey the full meaning (e.g., "look after"), include these as part of the verb in your SVO triple.
10. Double-check your list of triples to ensure comprehensiveness and accuracy, especially in dense or complex paragraphs that may contain multiple actions and entities interacting.
# Response format
{response_format}
- When providing integers, do not write out any decimals at all
- Use deduction where appropriate e.g. "3 dollars fifty cents" is a single value [3.5] not two values [3, 50] unless the user specifically asks for each part.
# Data to extract
\```
{data}
\```

Where {response_format} is a Pydantic model and data is the text mentioned above.

Results:

Phi-2, llama2 and Mistral-Tiny (7B) were unable to follow the output structure as requested. All models had Temperature set to 0.

GPT-4-Turbo-Preview

SVOTripleList(
    triples=[
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="in themselves exceedingly brief",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="contain",
            object="the essence of practical wisdom",
        ),
        SVOTriple(
            subject="The essence of practical wisdom",
            verb="is set forth",
            object="in admirable order and detail",
        ),
        SVOTriple(subject="The theme", verb="is", object="the great regeneration"),
        SVOTriple(
            subject="The great regeneration",
            verb="is",
            object="the birth of the spiritual from the psychical man",
        ),
        SVOTriple(
            subject="Paul",
            verb="set forth",
            object="the same theme wisely and eloquently in writing to his disciples in Corinth",
        ),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the theme of all mystics in all lands",
        ),
    ]
)

GPT-3.5-Turbo-0125

SVOTripleList(
    triples=[
        SVOTriple(
            subject="Yoga Sutras of Patanjali", verb="are", object="exceedingly brief"
        ),
        SVOTriple(
            subject="Yoga Sutras of Patanjali",
            verb="contain",
            object="essence of practical wisdom",
        ),
        SVOTriple(
            subject="Yoga Sutras of Patanjali",
            verb="set forth",
            object="in admirable order and detail",
        ),
        SVOTriple(subject="theme", verb="be", object="great regeneration"),
        SVOTriple(
            subject="theme",
            verb="be",
            object="birth of the spiritual from the psychical man",
        ),
        SVOTriple(subject="Paul", verb="set forth", object="theme"),
        SVOTriple(subject="Paul", verb="write", object="to his disciples in Corinth"),
        SVOTriple(
            subject="theme", verb="be", object="theme of all mystics in all lands"
        ),
    ]
)

Gemini Pro

SVOTripleList(
    triples=[
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="exceedingly brief",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="less than ten pages of large type in the original",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="contain",
            object="the essence of practical wisdom",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="set forth",
            object="in admirable order and detail",
        ),
        SVOTriple(subject="The present interpreter", verb="be", object="right"),
        SVOTriple(subject="The theme", verb="is", object="the great regeneration"),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the birth of the spiritual from the psychical man",
        ),
        SVOTriple(subject="Paul", verb="set forth", object="the theme"),
        SVOTriple(
            subject="Paul",
            verb="set forth",
            object="in writing to his disciples in Corinth",
        ),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the theme of all mystics in all lands",
        ),
    ]
)

Mistral Medium

SVOTripleList(
    triples=[
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="in themselves exceedingly brief",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="less than ten pages of large type in the original",
        ),
        SVOTriple(
            subject="They", verb="contain", object="the essence of practical wisdom"
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="set forth",
            object="the theme of the great regeneration",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="set forth",
            object="the birth of the spiritual from the psychical man",
        ),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the same theme which Paul so wisely and eloquently set forth in writing to his disciples in Corinth",
        ),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the theme of all mystics in all lands",
        ),
    ]
)

Mistral-small

SVOTripleList(
    triples=[
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="exceedingly brief",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="contain",
            object="the essence of practical wisdom",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="set forth",
            object="in admirable order and detail",
        ),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the great regeneration, the birth of the spiritual from the psychical man",
        ),
        SVOTriple(
            subject="The theme", verb="is", object="the same theme which Paul set forth"
        ),
        SVOTriple(subject="Paul", verb="set forth", object="the theme"),
        SVOTriple(
            subject="Paul",
            verb="set forth",
            object="the theme in writing to his disciples in Corinth",
        ),
        SVOTriple(
            subject="The theme",
            verb="is",
            object="the theme of all mystics in all lands",
        ),
    ]
)

Command (by Cohere)

SVOTripleList(
    triples=[
        SVOTriple(subject="Patanjali", verb="wrote", object="the Yoga Sutras"),
        SVOTriple(subject="the Yoga Sutras", verb="contain", object="practical wisdom"),
        SVOTriple(subject="the Yoga Sutras", verb="are", object="exceedingly brief"),
        SVOTriple(
            subject="the Yoga Sutras",
            verb="explain",
            object="the essence of spiritual rebirth",
        ),
    ]
)

Yi 34B

SVOTripleList(
    triples=[
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="are",
            object="in themselves exceedingly brief",
        ),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="contain",
            object="the essence of practical wisdom",
        ),
        SVOTriple(subject="Patanjali", verb="wrote", object="The Yoga Sutras"),
        SVOTriple(
            subject="The Yoga Sutras of Patanjali",
            verb="set forth",
            object="in admirable order and detail",
        ),
        SVOTriple(subject="Paul", verb="wrote", object="to his disciples in Corinth"),
        SVOTriple(
            subject="Paul",
            verb="set forth",
            object="the theme of all mystics in all lands",
        ),
    ]
)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment