Systematic LLM Prompt Engineering Using DSPy Optimization

into the fascinating and quickly evolving science of LLM immediate iteration, which is a basic a part of Massive Language Mannequin Operations (LLMOPs). We’ll use the instance of producing customer support responses with a real-world dataset to indicate how each generator and LLM-judge prompts could be developed in a scientific vogue with DSPy. All of the code for this venture, together with a number of notebooks and a pleasant README written by Claude, could be discovered here.

The sphere of utilized AI, which generally includes constructing pipelines that join information to Massive Language Fashions (LLMs) in a approach that generates enterprise worth, is evolving quickly. There’s a giant and rising array of open and closed supply fashions for builders to select from, and most of the newest fashions match or surpass professional human degree efficiency in generative duties corresponding to coding and technical writing. A number of flagship fashions corresponding to Gemini 2.5 are additionally natively multimodal, with video and audio capabilities that facilitate dialog with customers in an uncannily human-like vogue. Certainly, AI help is already shortly creeping into our day by day lives — for instance I’ve been utilizing it an increasing number of over the previous few months for coding, brainstorming concepts and common recommendation. It’s a brand new world that we’re all studying to navigate and with such highly effective know-how at our fingertips, it’s straightforward to get began and construct a POC. However LLM-powered tasks nonetheless face vital hurdles on their approach from analysis to manufacturing. 

1.0 The problem of immediate iteration

Immediate iteration — central to constructing reliable generative AI merchandise — is difficult as a result of there are such a lot of methods to jot down a immediate, fashions could be very delicate to small modifications and with generative duties judgement of the outcomes is usually subjective. Over time, prompts can develop by way of iteration and edge case fixes into difficult textual content recordsdata which can be extremely optimized towards a specific mannequin model. This presents a problem when groups need to improve to the newest model or change mannequin supplier, doubtless requiring vital refactoring. If a regression testing course of is established, then iteration is feasible by incrementally adjusting the generator immediate, re-running the take a look at suite and evaluating the outcome. Nevertheless this course of can be tedious and analysis could be extremely subjective — even groups lucky sufficient to make use of subject material consultants will wrestle when these consultants disagree.

2.0 Who evaluates the output?

LLMs are non-deterministic, generative fashions. These attributes are core to their performance, however make them troublesome to guage. Conventional pure language processing metrics are hardly ever relevant, and sometimes there is no such thing as a floor reality towards which to match the LLM outputs. The idea of LLM judges might help right here however provides complexity. An LLM decide is usually a robust mannequin that tries to do the job of an professional annotator by figuring out the standard of the generator mannequin output. Cutting-edge basis fashions are usually superb at classifying whether or not or not a generated output meets sure pre-defined standards. Subsequently,  within the ideally suited scenario we’ve a decide whose immediate distills the thought strategy of a human SME and which produces outputs which can be reliably aligned with SME consensus. We are able to then apply this mechanically to a consultant growth set and evaluate outcomes throughout variations of the generator immediate to verify we’re iterating in the best path.

3.0 Including complexity

How do we all know that the LLM decide is dependable? Human effort is often nonetheless wanted to label a coaching set for the decide, whose immediate can then be aligned to generate outcomes that match the human labels so far as potential. All of the complexities and mannequin model dependencies that we mentioned above close to the generator immediate additionally apply to LLM judges and in a multi-stage venture with a number of LLM calls there can also must be a number of judges, every with their very own coaching set. 

Even for a single-prompt workflow, our setup has now change into fairly advanced. Now we have a generator immediate that we need to iterate on, and a growth dataset of inputs. We then have a separate dataset of inputs and outputs that’s used to develop the decide. That dataset ought to then be labelled by an SME and cut up right into a coaching portion — to construct the decide immediate — and a take a look at portion, to check the decide immediate and stop overfitting. As soon as the decide has been educated, it’s used to optimize the generator immediate towards the event dataset. We then ideally want one other holdout set which we are able to use to test whether or not or not the generator immediate has been overfit to the event set. Immediate variations and datasets ought to be logged in order that experiments are reproducible. The diagram beneath illustrates this example. 

With so many parts for the optimization of a single generator immediate, we ideally want an LLMOps framework with inbuilt logging and monitoring. Description of such frameworks is past the scope of this text, however I’d advocate the mlflow documentation or the later modules of this wonderful class for extra data.

4.0 Function of this text

On this article, we are going to construct the important parts of the system proven in determine 1 for a easy toy drawback that includes producing useful customer support responses. Particulars of the info era course of are shared within the subsequent part.

The generator mannequin’s intention can be to take a dialog and produce the subsequent assist agent message that’s as useful as potential. To maintain monitor of the prompts and carry out optimization, we’ll use the DSPy orchestration library. DSPy is exclusive in that it abstracts away uncooked, textual content based mostly prompts into modular Python code (utilizing what it calls Signatures and Modules, which we’ll talk about later), and gives instruments to outline success metrics and mechanically optimize prompts in direction of them. The promise of DSPy is that with metric and a few skill to calculate it (both floor reality information or an LLM decide), one can mechanically optimize the immediate without having to manually edit a textual content file. By this venture, we’ll see among the professionals and cons of this strategy. It ought to be famous that we solely actually scratch the floor of DSPy makes use of right here, and the library has wonderful documentation for additional studying. 

The sections are organized as follows:

• First we talk about the info and aims. We load the info, do some preprocessing, and pattern a number of hundred conversations to change into the premise for our immediate growth units
• We discover the baseline generator and use it to create outputs for the decide growth set. We see how this may be finished by way of direct API calls (i.e. no LLM orchestration framework) vs what it appears like with DSPy.
• Subsequent we talk about creating the gold commonplace dataset for the LLM decide. To save lots of time, we select to generate this with a robust closed-source LLM, however that is the step that in actual tasks will want human subject material professional enter
• With our gold commonplace decide dataset in place, we are able to use it to tune a immediate for the LLM decide that we’ll ultimately be utilizing to assist us iterate. Alongside the best way we’ll contact on how DSPy’s optimizers work
• We’ll then try to make use of DSPy with our optimized decide to tune our baseline generator immediate and get the very best scores potential on the event dataset

The aim of all that is to take steps in direction of a sturdy methodology for immediate growth that’s reproducible and pretty automated, which means that it might be run once more to re-optimize the generator immediate if we selected to modify mannequin supplier, for instance. The methodology is much from good and by way of outcome high quality I think it’s no substitute for cautious guide immediate engineering in collaboration with SMEs. Nevertheless, do I feel it encapsulates most of the ideas of analysis pushed growth and highlights the ability of DSPy to hurry up iteration in LLM tasks. 

5.0 Dataset and Goal

The Kaggle buyer assist dataset that we’re utilizing gives us with about 30k buyer questions associated to air journey. With a purpose to convert this right into a extra life like “buyer assist dialog” desk, I used gemini-2.5-flash to create a set of 5000 artificial conversations utilizing samples from the Kaggle dataset to seed the subjects. Every dialog incorporates each buyer and assist messages, and is related to a novel id and firm identify. This generation process is designed to create a fairly sized instance dataset that’s just like what might be constructed from actual buyer assist logs.

Right here’s an instance of an artificial pattern dialog associated to American Airways 

Buyer: Attempting to kind out a pal's return flight from Heathrow however no luck with the standard phone quantity. I believed any individual posted one other quantity a while in the past however I've searched and may't discover something.
Assist:  The principle quantity is 800-433-7300.  What data do you want concerning your pal's flight?  A affirmation quantity could be useful.
Buyer:  I haven't got a affirmation quantity.  It is for a pal; I solely have her identify and the approximate journey dates – someday in October.  Is there one other approach to monitor this down?
Assist:  Sadly, and not using a affirmation quantity or extra exact dates, monitoring her flight is unattainable.  Have her test her e mail inbox for a affirmation.  If she will be able to't discover it, she ought to contact us instantly.

The objective of our agent can be to imagine the function of buyer assist, responding to the consumer’s question in a approach that’s as useful as potential given the scenario. It is a difficult drawback since buyer questions could require extremely particular context with a view to reply effectively, or could also be utterly exterior the assist agent’s management. In such conditions the mannequin should do the very best it could actually, displaying empathy and understanding similar to a educated human agent would. It should additionally not hallucinate details, which is an important test that’s past the scope of this text however ought to definitely be made in manufacturing, ideally utilizing an LLM decide that has entry to appropriate data banks. 

To get the info prepared, we are able to apply some fundamental preprocessing, coated within the code here. We benefit from the parallelism supplied by Huggingface’s datasets library, which permits us to use fundamental fashions like a fasttext language detector on this dataset. We additionally must randomly truncate the conversations in order that the ultimate utterance is all the time a “Buyer” message, due to this fact establishing the mannequin to generate the subsequent “Assist” response. For this function we’ve this simple truncator class. Lastly, it appears useful to supply the mannequin with the corporate identify for added context (that is one thing whose accuracy raise we are able to take a look at with the framework proposed right here!, so we append that to the conversations too.  

from dspy_judge.data_loader.dataset_loader import CustomerSupportDatasetLoader
from dspy_judge.processor.conversation_truncator import ConversationTruncator
from dspy_judge.processor.utils import concat_company_and_conversation

data_loader = CustomerSupportDatasetLoader()
# load and preprocess 
dataset = data_loader.load_dataset(cut up="practice")
processed_dataset = data_loader.preprocess_dataset(dataset)

truncator = ConversationTruncator(seed=101)
# truncate the conversations
truncated_dataset = truncator.process_dataset(
    processed_dataset,
    min_turns=1,
    ensure_customer_last=True
)
# apply perform that concatenates firm identify to dialog
truncated_dataset = truncated_dataset.map(concat_company_and_conversation)

# pattern only a subset of the complete dataset
truncated_loaded_sampled = data_loader.get_sample(
truncated_dataset,n_samples=400,seed=10
)
# cut up that pattern into take a look at and practice segments
split_dataset = truncated_loaded_sampled.train_test_split(
test_size=0.4, seed=10
)

The event dataset that we use to tune our generator and decide must be sufficiently small that tuning is quick and environment friendly, however nonetheless consultant of what the mannequin will see in manufacturing. Right here we simply select a random pattern of 400 truncated conversations, which can be cut up additional into testing and coaching datasets within the coming sections. Smarter strategies for selecting a consultant pattern for immediate optimization are a subject of lively analysis, an fascinating instance of which is here.

6.0 Baseline generator and decide growth set

Let’s cut up our dataset of consultant inputs additional: 160 examples for decide growth and 240 for generator immediate growth. These numbers are arbitrary however mirror a sensible compromise between representativeness and time/price. 

To proceed with LLM decide growth we’d like some outputs. Let’s generate some by way of a baseline model of our generator, which we’ll refine later.

LLMs are usually superb on the customer support process we’re foreign money centered on so with a view to see significant efficiency positive aspects on this toy venture let’s use gpt-3.5-turbo as our generator mannequin. One of the highly effective options of DSPy is the convenience of switching between fashions with out the necessity to manually re-optimize prompts, so it will be straightforward and fascinating to swap this out for different fashions as soon as the system is constructed. 

6.1 A fundamental model with out DSPy

The primary model of a baseline generator that I constructed for this venture truly doesn’t use DSPy. It consists of the next fundamental manually-typed immediate

baseline_customer_response_support_system_prompt = "You're a customer support agent whose job is to supply a single, concise response to a buyer question. You'll obtain a transcript of the interplay thus far, and your job is to answer the newest buyer message. You will even be given the identify of the corporate you're employed for, which ought to assist you to perceive the context of the messages." 

To name an LLM, the code incorporates numerous modules inheriting from LLMCallerBase, which might additionally optionally implement structured output utilizing the trainer library (extra about how this works right here). There may be additionally a module known as ParallelProcessor, which permits us to make API calls in parallel and minimizes errors within the course of by utilizing backoff to mechanically throttle the calls. There are doubtless some ways to make these parallel calls — in a previous article I made use of Huggingface datasets .map() performance, whereas right here inside ParallelProcessor we instantly use python’s multiprocessing library after which re-form the dataset from the listing of outputs.

That is doubtless not an environment friendly strategy in case you’re coping with actually giant datasets, but it surely works effectively for the few thousand examples that I’ve examined it with. It’s additionally crucial to concentrate on API prices when testing and operating parallel LLM calls!

Placing these items collectively, we are able to generate the baseline outcomes from a pattern of the truncated dialog dataset like this

from dspy_judge.llm_caller import OpenAITextOutputCaller
from dspy_judge.processor.parallel_processor import ParallelProcessor

baseline_model_name = "gpt-3.5-turbo"
baseline_model = OpenAITextOutputCaller(api_key=secrets and techniques["OPENAI_API_KEY"])
baseline_processor = ParallelProcessor(baseline_model, max_workers=4)

#cut up dataset is split into the generator and decide growth segments
dev_dataset = split_dataset["train"]
judge_dataset = split_dataset["test"]

# company_and_transcript is the identify of the sphere generated by the concat_company_and_conversation
# perform  
baseline_results_for_judge = baseline_processor.process_dataset(
        judge_dataset,
        system_prompt=baseline_customer_response_support_system_prompt,
        model_name=baseline_model_name,
        input_field="company_and_transcript",
        temperature=1.0
    )

# create a full transcript by including the newest generated response to the tip
# of the enter truncated dialog
baseline_results = baseline_results.map(concat_latest_response)

6.2 How does this look with DSPy?

Utilizing dspy feels totally different from different LLM orchestration libraries as a result of the prompting element is abstracted away. Nevertheless our codebase permits us to comply with the same sample to the “direct API name” strategy above.

The core dspy objects are signatures and modules. Signatures are lessons that enable us to outline the inputs and outputs of every LLM name. So for instance our baseline generator signature appears like this 

import dspy 

class SupportTranscriptNextResponse(dspy.Signature):

    transcript: str = dspy.InputField(desc="Enter transcript to evaluate")
    llm_response: str = dspy.OutputField(desc="The assist agent's subsequent utterance")

Signatures also can have docstrings, that are basically the system directions which can be despatched to the mannequin. We are able to due to this fact make use of the baseline immediate we’ve already written just by including the road 

  SupportTranscriptNextResponse.__doc__ = baseline_customer_response_support_system_prompt.strip()

A DSPy module can be a core constructing block that implements a prompting technique for any given signature. Like signatures they’re absolutely customizable, although for this venture we are going to primarily use a signature known as ChainOfThought, which implements the chain of thought prompting technique and thus forces the mannequin to generate a reasoning area together with the response specified within the signature. 

import dspy 

support_transcript_generator_module = dspy.ChainOfThought(
    SupportTranscriptNextResponse
)

ParallelProcessor has been written to assist working with dspy too, so the complete code to generate our baseline outcomes with dspy appears like this

# Create DSPy configuration for multiprocessing
dspy_config = {
  "model_name": "openai/gpt-3.5-turbo",
  "api_key": secrets and techniques["OPENAI_API_KEY"],
  "temperature": 1
}

generation_processor = ParallelProcessor()

# word judge_dataset is the decide cut up from Part 5
baseline_results_for_judge = generation_processor.process_dataset_with_dspy(
  judge_dataset,
  input_field="company_and_transcript",
  dspy_module=support_transcript_generator_module,
  dspy_config=dspy_config,
)

You possibly can check out the process_dataset_with_dspy methodology to see the small print of the setup right here. To keep away from pickling errors, we extract the DSPy signature from the equipped module, deconstruct after which re-assemble it on every of the employees. Every employee then calls 

dspy_module.predict(transcript=input_text) 

on every row within the batch it receives after which the result’s reassabled. The ultimate outcome ought to be just like that generated with the “native API” methodology within the earlier sections, with the one variations arising from the excessive temperature setting. 

The large benefit of DSPy at this stage is that the module support_transcript_generator_module could be simply saved, reloaded and fed instantly into different DSPy instruments like Consider and Optimize, which we’ll see later.

7.0 The decide coaching dataset

With our baseline generator arrange, we are able to transfer on to LLM decide growth. In our drawback, we would like the LLM decide to behave like a human buyer assist professional who is ready to learn an interplay between one other agent and a buyer, decide the agent’s success at resolving the shopper’s situation and likewise give critiques to elucidate the reasoning . To do that, it is extremely useful to have some gold commonplace judgments and critiques. In an actual venture this may be finished by operating the decide growth dataset by way of the baseline generator after which having a topic professional assessment the inputs and outputs, producing a labeled dataset of outcomes. To maintain issues easy a binary sure/no judgement is usually preferable, enabling us to instantly calculate metrics like accuracy, precision and Cohen’s kappa. To hurry up this step on this toy venture, I used Claude Opus 4.0 together with a “gold commonplace” judge prompt rigorously designed with help of GPT5. That is highly effective, however isn’t any substitute for a human SME and solely used as a result of this can be a demo venture. 

As soon as once more we are able to use DSPy’s ChainOfThought module with a signature like this 

import dspy

class SupportTranscriptJudge(dspy.Signature):
    transcript: str = dspy.InputField(desc="Enter transcript to evaluate")
    glad: bool = dspy.OutputField(
        desc="Whether or not the agent glad the shopper question"
    )

As a result of we’re requesting chain of thought, the reasoning area will mechanically get generated and doesn’t must specified within the signature

Working our “gold commonplace decide” to simulate the SME labelling part appears like this 

import dspy
from dspy_judge.processor.utils import extract_llm_response_fields_dspy

dspy_config = {
      "model_name": "anthropic/claude-sonnet-4-20250514",
      "api_key": secrets and techniques["ANTHROPIC_API_KEY"],
      "temperature": 0
}

gold_standard_judge_generator_module = dspy.ChainOfThought(SupportTranscriptJudge)

gold_standard_dspy_judge_processor = ParallelProcessor()

dspy_judge_results_optimized = gold_standard_dspy_judge_processor.process_dataset_with_dspy(
  judge_dataset.select_columns(
    ["conversation_id","output_transcript"]
  ),
  input_field="output_transcript",
  dspy_module=gold_standard_judge_generator_module,
  dspy_config=dspy_config
)
gold_standard_dspy_judge_results = gold_standard_dspy_judge_results.map(
    extract_llm_response_fields_dspy
)

The decide coaching dataset incorporates a repair of “constructive” and “adverse” outcomes and their related explanations. That is fascinating as a result of we have to ensure that our LLM decide is tuned to know the way to distinguish the 2. This additionally has the benefit of giving us our first indication of the efficiency of the baseline generator. For our pattern dataset, the efficiency isn’t nice, with an virtually 50% failure price as seen in determine 2. In a extra critical venture, we’d need to pause at this SME labelling stage and conduct a cautious error evaluation to grasp the principle kinds of failure mode.

8.0 Optimizing the decide immediate

With our gold commonplace decide dataset in place, we are able to now proceed to develop and optimize a decide immediate. A technique to do that is begin with a baseline decide by attempting to encode among the SME reasoning right into a immediate, run it on the decide coaching set and make incremental edits till the alignment between the SME scores and judges scores begins to degree off. It’s helpful to log every model of the immediate and its related alignment rating in order that progress could be tracked and any leveling off detected. One other strategy is to start out by utilizing dspy’s immediate optimizers to do a few of this work for us. 

Dspy optimizers take our module, a metric perform and small coaching set and try and optimize the immediate to attain and maximize the metric. In our case, the metric would be the match accuracy between our decide’s binary classification and the bottom reality from the SME labelled dataset. There are a number of algorithms for this, and right here we give attention to MIPROv2 as a result of it could actually adapt each the system directions and create or edit few-shot examples. In abstract, MIPROv2 is an automated, iterative course of with the next steps

• Step 1: It runs the equipped module towards a subset of the coaching dataset and filters excessive scoring trajectories to generate few-shot examples. 
• Step 2: It makes use of LLM calls to generate a number of candidate system prompts based mostly on observations from step 1
• Step 3: It searches for combos of candidate system prompts and candidate few shot examples that maximize the metric worth when run towards mini-batches of the coaching information

The promise of algorithms like that is that they might help us design prompts in a data-driven vogue just like how conventional ML fashions are educated. The draw back is that they separate builders from the info in such a approach that it turns into tougher to elucidate why the chosen immediate is perfect, aside from “the mannequin mentioned so”. Additionally they have quite a few hyperparameters whose settings could be fairly influential on the outcome. 

In my expertise thus far, optimizers make sense to make use of in instances the place we have already got floor reality, and their output can then be used as a place to begin for guide iteration and additional collaboration with the subject material professional. 

Let’s see how MIPROv2 could be utilized in our venture to optimize the LLM decide. We’ll select our decide mannequin to be Gemini 1.5 flash, which is affordable and quick. 

import dspy
from dspy_judge.prompts.dspy_signatures import SupportTranscriptJudge

judge_model = dspy.LM(
    "gemini/gemini-1.5-flash",
    api_key=secrets and techniques["GEMINI_API_KEY"],
    cache=False,
    temperature=0
)
dspy.configure(lm=judge_model,track_usage=True,adapter=dspy.JSONAdapter())
baseline_judge = dspy.ChainOfThought(SupportTranscriptJudge)

Be aware that baseline decide right here represents our “finest try” at a decide based mostly on reviewing the SME labelled dataset.

Out of curiosity, let’s first see what the preliminary alignment between the baseline and gold commonplace decide is like. We are able to do this utilizing dspy.Consider operating on a set of examples, a metric perform and a module.

import dspy
from dspy_judge.processor.utils import convert_dataset_to_dspy_examples

#that is our easy metric perform to find out the place the decide rating matches the gold #commonplace decide label

def match_judge_metric(instance, pred, hint=None):
    example_str = str(instance.glad).decrease().strip()
    # that is going to be True or False
    pred_str = str(pred.glad).decrease().strip()

    if example_str == pred_str:
        return 1
    else:
        return 0

# Load the SME labelled dataset
dspy_gold_standard_judge_results = data_loader.load_local_dataset("datasets/gold_standard_judge_result")

# Convert HF dataset to listing of dspy examples
judge_dataset_examples = convert_dataset_to_dspy_examples(
    dspy_gold_standard_judge_results,
    field_mapping = {"transcript":"output_transcript","glad":"glad"},
    input_field="transcript"
)

evaluator = dspy.Consider(
    metric=match_judge_metric,
    devset=judge_dataset_examples,
    display_table=True,
    display_progress=True,
    num_threads=24,
)
original_score = evaluator(baseline_judge)

For me, operating this offers a baseline decide rating of ~60%. The query is, can we use MIPROv2 to enhance this?

Organising an optimization run is simple, although remember that a number of LLM calls are made throughout this course of and so operating a number of instances or on giant coaching datasets could be expensive. It’s additionally advisable to test the documentation for the hyperparameter explanations and be ready for the optimization to not work as anticipated.

import dspy

#cut up the SME labelled dataset into coaching and testing
training_set = judge_dataset_examples[:110]
validation_set = judge_dataset_examples[110:]

optimizer = dspy.MIPROv2(
    metric=match_judge_metric,
    auto="medium",
    init_temperature=1.0,
    seed=101
)

judge_optimized = optimizer.compile(
    baseline_judge,
    trainset=training_set,
    requires_permission_to_run=False,
)

At this stage, we’ve a brand new dspy module known as judge_optimized, and we are able to consider it with dspy. Evaluator towards the coaching and validation units. I get an accuracy rating of ~70% once I do that, suggesting that the optimization has certainly made the decide immediate extra aligned with the gold commonplace labels.

What particularly has modified? To seek out out, we are able to run the next

judge_optimized.inspect_history(n=1)

Which can present the newest model of the system immediate, any added few shot examples and the final name that was made. Working the optimization a number of instances will doubtless produce fairly totally different outcomes with system prompts that vary from minor modifications to the baseline to finish rewrites, all of which obtain considerably related last scores. Few-shot examples virtually all the time get added from the coaching set, which means that these are the principle drivers of any metric enhancements. Provided that the few-shot examples are sourced from the coaching set, it’s crucial to run the ultimate analysis towards each this and the decide validation set to guard towards overfitting, although I think that is much less of an issue than in conventional machine studying.

Lastly, we must always save the decide and baseline modules for future use and copy of any experiments. 

judge_optimized.save("dspy_modules/optimized_llm_judge",save_program=True)
baseline_judge.save("dspy_modules/baseline_llm_judge",save_program=True)

9.0 Utilizing the optimized decide to optimize the generator

Let’s say we’ve an optimized LLM decide that we’re assured is suitably aligned with the SME labelled dataset to be helpful and we now need to use it to enhance the baseline generator immediate. The method is just like that which we used to construct the optimized decide immediate, solely this time we’re utilizing the decide to supply us with floor reality labels. As talked about in part 6.0, we’ve a generator growth set of 240 examples. As a part of guide immediate iteration, we’d every generate a generator immediate model on these enter examples, then run the decide on the outcomes and calculate the accuracy. We’d then assessment the decide critiques, iterate on the immediate, save a brand new model after which attempt once more. Many iterations will doubtless be wanted, which explains why the event set ought to be pretty small. DSPy might help get us began down this path with automated optimization, and the code is similar to part 8.0, with one notable exception being the optimization metric. 

Throughout this part, we’ll be utilizing two LLMs within the optimization: The generator mannequin, which is gpt-3.5-turbo and the decide mode, which is gemini-1.5-flash. To facilitate this, we are able to make a easy customized module known as ModuleWithLM that makes use of a selected LLM, in any other case it should simply use no matter mannequin is outlined within the final dspy.configure() name

import dspy 

optimized_judge = dspy.load(
    "dspy_modules/optimized_llm_judge"
)

# our decide LLM can be gemini 1.5
judge_lm = dspy.LM(
    "gemini/gemini-1.5-flash",
    api_key=secrets and techniques["GEMINI_API_KEY"],
    cache=False,
    temperature=0
)


# A helper that runs a module with a selected LM context
class ModuleWithLM(dspy.Module):
    def __init__(self, lm, module):
        tremendous().__init__()
        self.lm = lm
        self.module = module

    def ahead(self, **kwargs):
        with dspy.context(lm=self.lm):
            return self.module(**kwargs)

# our new module, which permits us the bundle a separate llm with a beforehand
# loaded module
optimized_judge_program = ModuleWithLM(judge_lm, optimized_judge)

With this established, we are able to name optimized_judge_program inside a metric perform, which serves the identical function as match_judge_metric() did within the decide optimization course of.

def LLM_judge_metric(instance, prediction, hint=None):

    # the enter transcript
    transcript_text = str(instance.transcript)

    # the output llm response
    output_text = str(prediction.llm_response)

    transcript_text = f"{transcript_text}nSupport: {output_text}"
    
    if not transcript_text:
        # Fallback or increase; metric should be deterministic
        return False

    judged = optimized_judge_program(transcript=transcript_text)
    return bool(judged.glad)

The optimization itself appears just like the method we used for the LLM decide

import dspy
from dspy_judge.processor.utils import convert_dataset_to_dspy_examples

generate_response = dspy.load(
"dspy_modules/baseline_generation",save_program=True
)

#see part 1.3 for the outline of dev_dataset
 dev_dataset_examples = convert_dataset_to_dspy_examples(
    dev_dataset,
    field_mapping = {"transcript":"company_and_transcript"},
    input_field="transcript"
)

# A considerably arbitrary cut up into take a look at and practice
# The cut up permits us to test for overfitting on the coaching examples

optimize_training_data =  dev_dataset_examples[:200]
optimize_validation_data = dev_dataset_examples[200:]

optimizer = dspy.MIPROv2(
    metric=LLM_judge_metric,
    auto="medium",
    init_temperature=1.0,
    seed=101
)

generate_response_optimized = optimizer.compile(
    generate_response,
    trainset=optimize_training_data,
    requires_permission_to_run=False,
)

With the optimization full, we are able to use dspy.Consider to generate an general accuracy evaluation

evaluator = dspy.Consider(
    metric=LLM_judge_metric,
    devset=optimize_training_data,
    display_table=True,
    display_progress=True,
    num_threads=24,
)
overall_score_baseline = evaluator(generate_response)
latest_score_optimized = evaluator(generate_response_optimized)

Throughout testing, I used to be capable of get a good enchancment from ~58%  to ~75% accuracy utilizing this methodology. On our small dataset a few of this achieve is attributable to only a handful to evaluate outcomes switching from “unhappy” to “glad”, and for modifications like this it’s price checking if they’re inside the pure variability of the decide when run a number of instances on the identical dataset. In my opinion, the generator immediate optimization’s biggest power is that it gives us with a brand new baseline immediate which is grounded in finest follow and extra defensible than the unique baseline, which is usually simply the developer’s finest guess at immediate. The stage is then set for extra guide iteration, use of the optimized LLM decide for suggestions and additional session with SMEs for error evaluation and edge case dealing with.

At this stage it’s smart to look intimately on the information to study the place these modest high quality positive aspects are coming from. We are able to be a part of the output datasets from the baseline and optimized generator runs on dialog id, and search for cases the place the decide’s classification modified. In instances the place the classification switched from glad=False to glad=True my general sense is that the generator’s responses bought longer and extra well mannered, however didn’t actually convey any extra data. This isn’t stunning, since for many buyer assist questions the generator doesn’t have sufficient context so as to add significant element. Additionally, since LLM judges are identified to indicate bias in direction of longer outputs, the optimization course of seems to have pushed the generator in that path.

10.0 Essential learnings 

This text has explored immediate optimization with DSPy for each LLM decide and generator refinement. This course of is considerably long-winded, but it surely enforces good follow by way of growth dataset curation, immediate logging and checking for overfitting. For the reason that LLM decide alignment course of usually requires a human within the loop to generate labelled information, the DSPy optimization course of could be particularly highly effective right here as a result of it reduces costly iterations with subject material consultants. Though not likely mentioned on this article, it’s vital to notice that small efficiency positive aspects from the optimization won’t be statistically vital as a result of pure variability of LLMs.

With the decide arrange, it will also be utilized by DSPy to optimize the generator immediate (due to this fact bypassing the necessity for extra human labelling). My sense is that this will likely even be price pursuing in some tasks, if for no different cause however the automated curation of excellent few-shot examples. However it shouldn’t be an alternative choice to guide analysis and error evaluation. Optimization is will also be fairly expensive by way of tokens, so care ought to be taken to keep away from excessive API payments!

Thanks for making it to the tip. As all the time suggestions is appreciated and in case you have tried the same framework for immediate engineering I’d be very to study your expertise! I’m additionally curious in regards to the professionals and cons of extending this framework to extra advanced methods that use ReACT or prompt-chaining to perform bigger duties.