Evaluation quickstart

Evaluations are a quantitative way to measure performance of LLM applications, which is important because LLMs don’t always behave predictably — small changes in prompts, models, or inputs can significantly impact results. Evaluations provide a structured way to identify failures, compare changes across different versions of your application, and build more reliable AI applications.

Evaluations are made up of three components:

1. A dataset with test inputs and optionally expected outputs.
2. A target function that defines what you’re evaluating. For example, this may be one LLM call that includes the new prompt you are testing, a part of your application or your end to end application.
3. Evaluators that score your target function’s outputs.

This quick start guides you through running a simple evaluation to test the correctness of LLM responses with the LangSmith SDK or UI.

This quickstart uses prebuilt LLM-as-judge evaluators from the open-source openevals package. OpenEvals includes a set of commonly used evaluators and is a great starting point if you’re new to evaluations. If you want greater flexibility in how you evaluate your apps, you can also define completely custom evaluators using your own code.

SDK

1. Install Dependencies

Python

pip install -U langsmith openevals openai

If you are using yarn as your package manager, you will also need to manually install @langchain/core as a peer dependency of openevals. This is not required for LangSmith evals in general - you may define evaluators using arbitrary custom code.

2. Create a LangSmith API key

To create an API key, head to the Settings page. Then click Create API Key.

3. Set up your environment

Because this quickstart uses OpenAI models, you’ll need to set the OPENAI_API_KEY environment variable as well as the required LangSmith ones:

export LANGSMITH_TRACING=trueexport LANGSMITH_API_KEY="<your-langchain-api-key>"# This example uses OpenAI, but you can use other LLM providers if desiredexport OPENAI_API_KEY="<your-openai-api-key>"

4. Create a dataset

Next, define example input and reference output pairs that you’ll use to evaluate your app:

Python

from langsmith import Client
client = Client()

# Programmatically create a dataset in LangSmith
# For other dataset creation methods, see:
# https://docs.smith.langchain.com/langsmith/manage-datasets-programmatically
# https://docs.smith.langchain.com/langsmith/manage-datasets-in-application
dataset = client.create_dataset(
    dataset_name="Sample dataset", description="A sample dataset in LangSmith."
)

# Create examples
examples = [
    {
        "inputs": {"question": "Which country is Mount Kilimanjaro located in?"},
        "outputs": {"answer": "Mount Kilimanjaro is located in Tanzania."},
    },
    {
        "inputs": {"question": "What is Earth's lowest point?"},
        "outputs": {"answer": "Earth's lowest point is The Dead Sea."},
    },
]

# Add examples to the dataset
client.create_examples(dataset_id=dataset.id, examples=examples)

5. Define what you’re evaluating

Now, define target function that contains what you’re evaluating. For example, this may be one LLM call that includes the new prompt you are testing, a part of your application or your end to end application.

Python

from langsmith import wrappers
from openai import OpenAI

# Wrap the OpenAI client for LangSmith tracing
openai_client = wrappers.wrap_openai(OpenAI())

# Define the application logic you want to evaluate inside a target function
# The SDK will automatically send the inputs from the dataset to your target function
def target(inputs: dict) -> dict:
    response = openai_client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[
            {"role": "system", "content": "Answer the following question accurately"},
            {"role": "user", "content": inputs["question"]},
        ],
    )
    return { "answer": response.choices[0].message.content.strip() }

6. Define evaluator

Import a prebuilt prompt from openevals and create an evaluator. outputs are the result of your target function. reference_outputs / referenceOutputs are from the example pairs you defined in step 4 above.

CORRECTNESS_PROMPT is just an f-string with variables for "inputs", "outputs", and "reference_outputs". See here for more information on customizing OpenEvals prompts.

Python

from openevals.llm import create_llm_as_judge
from openevals.prompts import CORRECTNESS_PROMPT

def correctness_evaluator(inputs: dict, outputs: dict, reference_outputs: dict):
    evaluator = create_llm_as_judge(
        prompt=CORRECTNESS_PROMPT,
        model="openai:o3-mini",
        feedback_key="correctness",
    )
    eval_result = evaluator(
        inputs=inputs,
        outputs=outputs,
        reference_outputs=reference_outputs
    )
    return eval_result

7. Run and view results

Finally, run the experiment!

Python

# After running the evaluation, a link will be provided to view the results in langsmith
experiment_results = client.evaluate(
    target,
    data="Sample dataset",
    evaluators=[
        correctness_evaluator,
        # can add multiple evaluators here
    ],
    experiment_prefix="first-eval-in-langsmith",
    max_concurrency=2,
)

Click the link printed out by your evaluation run to access the LangSmith Experiments UI, and explore the results of the experiment.

Next steps

To learn more about running experiments in LangSmith, read the evaluation conceptual guide.

• Check out the OpenEvals README to see all available prebuilt evaluators and how to customize them.
• Learn how to define custom evaluators that contain arbitrary code.
• For more details on evaluations, refer to the Evaluation documentation.
• For comprehensive descriptions of every class and function see the API reference.

Or, if you prefer video tutorials, check out the Datasets, Evaluators, and Experiments videos from the Introduction to LangSmith Course.