Evaluation-Driven Development for LLM-Powered Products: Lessons from Building in Healthcare

within the area of huge language fashions (LLM) and their functions is very speedy. Prices are coming down and basis fashions have gotten more and more succesful, capable of deal with communication in textual content, pictures, video. Open supply options have additionally exploded in range and functionality, with many fashions being light-weight sufficient to discover, fine-tune and iterate on with out big expense. Lastly, cloud AI coaching and inference suppliers reminiscent of Databricks and Nebius are making it more and more simple for organizations to scale up their utilized AI merchandise from POCs to manufacturing prepared programs. These advances go hand in hand with a diversification of the enterprise makes use of of LLMs and the rise of agentic functions, the place fashions plan and execute complicated multi-step workflows which will contain interplay with instruments or different brokers. These applied sciences are already making an affect in healthcare and that is projected to develop quickly [1]. 

All of this functionality makes it thrilling to get began, and constructing a baseline resolution for a selected use case might be very quick. Nevertheless, by their nature LLMs are non-deterministic and fewer predictable than conventional software program or ML fashions. The actual problem subsequently is available in iteration: How do we all know that our improvement course of is bettering the system? If we repair an issue, how do we all know if the change gained’t break one thing else? As soon as in manufacturing, how will we examine if efficiency is on par with what we noticed in improvement? Answering these questions with programs that make single LLM calls is difficult sufficient, however with agentic systems we additionally want to contemplate all the person steps and routing selections made between them. To handle these points — and subsequently achieve belief and confidence within the programs we construct — we’d like evaluation-driven development. It is a methodology that locations iterative, actionable analysis on the core of the product lifecycle from improvement and deployment to monitoring. 

As a knowledge scientist at Nuna, Inc., a healthcare AI firm, I’ve been spearheading our efforts to embed evaluation-driven improvement into our merchandise. With the help of our management, we’re sharing a number of the key classes we’ve discovered thus far. We hope these insights will likely be beneficial not solely to these constructing AI in healthcare but in addition to anybody growing AI merchandise, particularly these simply starting their journey.

This text is damaged into the next sections, which search to clarify our broad learnings from the literature along with methods and suggestions gained from expertise.

• In Part 1 we’ll briefly contact on Nuna’s merchandise and clarify why AI analysis is so important for us and for healthcare-focused AI typically. 
• In Part 2, we’ll discover how evaluation-driven improvement brings construction to the pre-deployment part of our merchandise. This entails growing metrics utilizing each LLM-as-Choose and programmatic approaches, that are closely impressed by this excellent article. As soon as dependable judges and expert-aligned metrics have been established, we describe the best way to use them to iterate in the correct course utilizing error evaluation. On this part, we’ll additionally contact on the distinctive challenges posed by chatbot functions. 
• In Part 3, we’ll talk about the usage of model-based classification and alerting to watch functions in manufacturing and use this suggestions for additional enhancements. 
• Part 4 summarizes all that we’ve discovered

Any group’s perspective on these topics is influenced by the instruments it makes use of — for instance we use MLflow and Databricks Mosaic Analysis to maintain monitor of our pre-production experiments, and AWS Agent Analysis to check our chatbot. Nevertheless, we imagine that the concepts introduced right here must be relevant no matter tech stack, and there are a lot of glorious choices out there from the likes of Arize (Phoenix analysis suite), LangChain (LangSmith) and Assured AI (DeepEval). Right here we’ll give attention to tasks the place iterative improvement primarily entails immediate engineering, however an analogous strategy could possibly be adopted for fine-tuned fashions too.

1.0 AI and analysis at Nuna

Nuna’s purpose is to cut back the full value of care and enhance the lives of individuals with continual situations reminiscent of hypertension (hypertension) and diabetes, which collectively have an effect on greater than 50% of the US grownup inhabitants [2,3]. That is performed via a patient-focused cell app that encourages wholesome habits reminiscent of treatment adherence and blood stress monitoring, along with a care-team-focused dashboard that organizes knowledge from the app to suppliers*. To ensure that the system to succeed, each sufferers and care groups should discover it simple to make use of, participating and insightful. It should additionally produce measurable advantages to well being. That is important as a result of it distinguishes healthcare know-how from most different know-how sectors, the place enterprise success is extra intently tied to engagement alone. 

AI performs a important, affected person and care-team-facing position within the product: On the affected person facet now we have an in-app care coach chatbot, in addition to options reminiscent of treatment containers and meal photo-scanning. On the care-team facet we’re growing summarization and knowledge sorting capabilities to cut back time to motion and tailor the expertise for various customers. The desk beneath exhibits the 4 AI-powered product parts whose improvement served as inspiration for this text, and which will likely be referred to within the following sections.

Respect for sufferers and the delicate knowledge they entrust us with is on the core of our enterprise. Along with safeguarding knowledge privateness, we should be sure that our AI merchandise function in methods which can be protected, dependable, and aligned with customers’ wants. We have to anticipate how individuals may use the merchandise and check each customary and edge-case makes use of. The place errors are potential — reminiscent of ingredient recognition from meal images — we have to know the place to spend money on constructing methods for customers to simply right them. We additionally must be looking out for extra delicate failures — for instance, recent research suggests that prolonged chatbot use can lead to increased feelings of loneliness — so we have to establish and monitor for regarding use circumstances to make sure that our AI is aligned with the purpose of bettering lives and decreasing value of care. This aligns with suggestions from the NIST AI Danger Administration Framework, which emphasizes preemptive identification of believable misuse eventualities, together with edge circumstances and unintended penalties, particularly in high-impact domains reminiscent of healthcare.

*This technique supplies wellness help solely and isn’t meant for medical prognosis, therapy, or to interchange skilled healthcare judgment.

2.0 Pre-deployment: Metrics, alignment and iteration 

Within the improvement stage of an LLM-powered product, it is very important set up analysis metrics which can be aligned with the enterprise/product targets, a testing dataset that’s consultant sufficient to simulate manufacturing conduct and a sturdy methodology to truly calculate the analysis metrics. With this stuff in place, we will enter the virtuous cycle of iteration and error evaluation (see this short book for particulars). The quicker we will iterate in the correct course, the upper our possibilities of success. It additionally goes with out saying that every time testing entails passing delicate knowledge via an LLM, it have to be performed from a safe surroundings with a trusted supplier in accordance with knowledge privateness laws. For instance, in the USA, the Well being Insurance coverage Portability and Accountability Act (HIPAA) units strict requirements for safeguarding sufferers’ well being data. Any dealing with of such knowledge should meet HIPAA’s necessities for safety and confidentiality.

2.1 Improvement dataset 

On the outset of a challenge, it is very important establish and have interaction with material specialists (SMEs) who may help generate instance enter knowledge and outline what success appears like. At Nuna our SMEs are guide healthcare professionals reminiscent of physicians and nutritionists. Relying on the issue context, we’ve discovered that opinions from healthcare specialists might be almost uniform — the place the reply might be sourced from core ideas of their coaching — or fairly diversified, drawing on their particular person experiences. To mitigate this, we’ve discovered it helpful to hunt recommendation from a small panel of specialists (sometimes 2-5) who’re engaged from the start of the challenge and whose consensus view acts as our final supply of fact. 

It’s advisable to work with the SMEs to construct a consultant dataset of inputs to the system. To do that, we must always take into account the broad classes of personas who could be utilizing it and the primary functionalities. The broader the use case, the extra of those there will likely be. For instance, the Nuna chatbot is accessible to all customers, helps reply any wellness-based query and likewise has entry to the person’s personal knowledge through instrument calls. A number of the functionalities are subsequently “emotional help”, “hypertension help”, “diet recommendation”, “app help”, and we would take into account splitting these additional into “new person” vs. “exiting person” or “skeptical person” vs. “energy person” personas. This segmentation is beneficial for the information technology course of and error evaluation in a while, after these inputs have run via the system.

It’s additionally necessary to contemplate particular eventualities — each typical and edge-case — that the system should deal with. For our chatbot these embody “person asks for a prognosis based mostly on signs” (we all the time refer them to a healthcare skilled in such conditions), “person ask is truncated or incomplete”, “person makes an attempt to jailbreak the system”. After all, it’s unlikely that each one important eventualities will likely be accounted for, which is why later iteration (part 2.5) and monitoring in manufacturing (part 3.0) is required.

With the classes in place, the information itself could be generated by filtering present proprietary or open supply datasets (e.g. Nutrition5k for meals pictures, OpenAI’s HealthBench for patient-clinician conversations). In some circumstances, each inputs and gold customary outputs could be out there, for instance within the ingredient labels on every picture in Nutition5k. This makes metric design (part 2.3) simpler. Extra generally although, knowledgeable labelling will likely be required for the gold customary outputs. Certainly, even when pre-existing enter examples should not out there, these might be generated synthetically with an LLM after which curated by the group — Databricks has some instruments for this, described here. 

How massive ought to this improvement set be? The extra examples now we have, the extra possible it’s to be consultant of what the mannequin will see in manufacturing however the costlier it will likely be to iterate. Our improvement units sometimes begin out on the order of some hundred examples. For chatbots, the place to be consultant the inputs may must be multi-step conversations with pattern affected person knowledge in context, we suggest utilizing a testing framework like AWS Agent Evaluation, the place the enter instance recordsdata might be generated manually or through LLM by prompting and curation. 

2.2 Baseline mannequin pipeline

If ranging from scratch, the method of pondering via the use circumstances and constructing the event set will possible give the group a way for the issue of this downside and therefore the structure of the baseline system to be constructed. Except made infeasible by safety or value considerations, it’s advisable to maintain the preliminary structure easy and use highly effective, API-based fashions for the baseline iteration. The principle function of the iteration course of described in subsequent sections is to enhance the prompts on this baseline model, so we sometimes preserve them easy whereas making an attempt to stick to basic immediate engineering finest practices reminiscent of these described on this guide by Anthropic.

As soon as the baseline system is up and operating, it must be run on the event set to generate the primary outputs. Operating the event dataset via the system is a batch course of which will must be repeated many occasions, so it’s value parallelizing. At Nuna we use PySpark on Databricks for this. Probably the most simple methodology for batch parallelism of this sort is the pandas user-defined function (UDF), which permits us to name the mannequin API in a loop over rows in Pandas dataframe, after which use Pyspark to interrupt up the enter dataset into chunks to be processed in parallel over the nodes of a cluster. An alternate methodology, described here, first requires us to log a script that calls the mannequin as an mlflow PythonModel object, load that as a pandas UDF after which run inference utilizing that. 

2.3 Metric design 

Designing analysis metrics which can be actionable and aligned with the function’s targets is a important a part of evaluation-driven improvement. Given the context of the function you might be growing, there could also be some metrics which can be minimal necessities for ship — e.g. a minimal charge of the numerical accuracy for a textual content abstract on a graph. Particularly in a healthcare context, now we have discovered that SMEs are once more important assets right here within the identification of further supplementary metrics that will likely be necessary for stakeholder buy-in and end-user interpretation. Asynchronously, SMEs ought to have the ability to securely overview the inputs and outputs from the event set and make feedback on them. Numerous purpose-built instruments help this sort of overview and might be tailored to the challenge’s sensitivity and maturity. For early-stage or low-volume work, light-weight strategies reminiscent of a safe spreadsheet might suffice. If potential, the suggestions ought to include a easy cross/fail determination for every enter/output pair, together with critique of the LLM-generated output explaining the choice. The concept is that we will then use these critiques to tell our alternative of analysis metrics and supply few-shot examples to any LLM-judges that we construct. Why cross/fail reasonably than a likert rating or another numerical metric? It is a developer alternative, however we discovered it’s a lot simpler to get alignment between SMEs and LLM judges within the binary case. It’s simple to combination outcomes right into a easy accuracy measure throughout the event set. For instance, if 30% of the “90 day blood stress time collection summaries” get a zero for groundedness however not one of the 30 day summaries do, then this factors to the mannequin scuffling with lengthy inputs.

On the preliminary overview stage, it’s typically additionally helpful to doc a transparent set of pointers round what constitutes success within the outputs, which permits all annotators to have a supply of fact. Disagreements between SME annotators can typically be resolved on the subject of these pointers, and if disagreements persist this can be an indication that the rules — and therefore the aim of the AI system — will not be outlined clearly sufficient. It’s additionally necessary to notice that relying in your firm’s resourcing, ship timelines, and danger degree of the function, it is probably not potential to get SME feedback on all the improvement set right here — so it’s necessary to decide on consultant examples. 

As a concrete instance, Nuna has developed a medicine logging historical past AI abstract, to be displayed within the care team-facing portal. Early within the improvement of this AI abstract, we curated a set of consultant affected person data, ran them via the summarization mannequin, plotted the information and shared a safe spreadsheet containing the enter graphs and output summaries with our SMEs for his or her feedback. From this train we recognized and documented the necessity for a variety of metrics together with readability, fashion (i.e. goal and never alarmist), formatting and groundedness (i.e. accuracy of insights in opposition to the enter timeseries). 

Some metrics might be calculated programmatically with easy checks on the output. This contains formatting and size constraints, and readability as quantified by scores just like the F-K grade level. Different metrics require an LLM-judge (see beneath for extra element) as a result of the definition of success is extra nuanced. That is the place we immediate an LLM to behave like a human knowledgeable, giving cross/fail selections and critiques of the outputs. The concept is that if we will align the LLM decide’s outcomes with these of the specialists, we will run it robotically on our improvement set and shortly compute our metrics when iterating. 

We discovered it helpful to decide on a single “optimizing metric” for every challenge, for instance the proportion of the event set that’s marked as precisely grounded within the enter knowledge, however again it up with a number of “satisficing metrics” reminiscent of % inside size constraints, % with appropriate fashion, % with readability rating > 60 and so on. Elements like latency percentile and imply token value per request additionally make perfect satisficing metrics. If an replace makes the optimizing metric worth go up with out decreasing any of the satisficing metric values beneath pre-defined thresholds, then we all know we’re stepping into the correct course. 

2.4 Constructing the LLM decide

The aim of LLM-judge improvement is to distill the recommendation of the SMEs right into a immediate that permits an LLM to attain the event set in a manner that’s aligned with their skilled judgement. The decide is often a bigger/extra highly effective mannequin than the one being judged, although this isn’t a strict requirement. We discovered that whereas it’s potential to have a single LLM decide immediate output the scores and critiques for a number of metrics, this may be complicated and incompatible with the monitoring instruments described in 2.4. We subsequently make a single decide immediate per metric, which has the additional benefit of forcing conservatism on the variety of LLM-generated metrics.

An preliminary decide immediate, to be run on the event set, may look one thing just like the block beneath. The directions will likely be iterated on through the alignment step, so at this stage they need to symbolize a finest effort to seize the SME’s thought course of when writing their criques. It’s necessary to make sure that the LLM supplies its reasoning, and that that is detailed sufficient to grasp why it made the dedication. We must also double examine the reasoning in opposition to its cross/fail judgement to make sure they’re logically constant. For extra element about LLM reasoning in circumstances like this, we suggest this excellent article. 

<job>
You might be an knowledgeable healthcare skilled who's requested to guage a abstract of a affected person's medical knowledge that was made by an automatic system. 

Please comply with these directions for evaluating the summaries

{detailed directions}

Now fastidiously examine the next enter knowledge and output response, giving your reasoning and a cross/fail judgement within the specified output format
</job>

<enter knowledge>
{knowledge to be summarized}
</enter knowledge>

<output_format>
{formatting directions}
</output_format>

To maintain the decide outputs as dependable as potential, its temperature setting must be as little as potential. To validate the decide, the SMEs must see consultant examples of enter, output, decide determination and decide critique. This could ideally be a distinct set of examples than those they checked out for the metric design, and given the human effort concerned on this step it may be small. 

The SMEs may first give their very own cross/fail assessments for every instance with out seeing the decide’s model. They need to then have the ability to see every little thing and have the chance to change the mannequin’s critique to turn out to be extra aligned with their very own ideas. The outcomes can be utilized to make modifications to the LLM decide immediate and the method repeated till the alignment between the SME assessments and mannequin assessments stops bettering, as time constraints enable. Alignment might be measured utilizing easy accuracy or statistical measures reminiscent of Cohen’s kappa. We’ve discovered that together with related few-shot examples within the decide immediate sometimes helps with alignment, and there’s additionally work suggesting that adding grading notes for every instance to be judged can also be helpful. 

We’ve sometimes used spreadsheets for such a iteration, however extra subtle instruments reminiscent of Databrick’s review apps additionally exist and could possibly be tailored for LLM decide immediate improvement. With knowledgeable time briefly provide, LLM judges are crucial in healthcare AI and as basis fashions turn out to be extra subtle, their means to face in for human specialists seems to be bettering. OpenAI’s HealthBench work, for instance, discovered that physicians had been typically unable to enhance the responses generated by April 2025’s fashions and that when GPT4.1 is used as a grader on healthcare-related issues, its scores are very properly aligned with these of human specialists [4]. 

2.5 Iteration and monitoring

With our LLM judges in place, we’re lastly in a superb place to start out iterating on our fundamental system. To take action, we’ll systematically replace the prompts, regenerate the event set outputs, run the judges, compute the metrics and do a comparability between the brand new and previous outcomes. That is an iterative course of with doubtlessly many cycles, which is why it advantages from tracing, immediate logging and experiment monitoring. The method of regenerating the event dataset outputs is described in part 2.1, and instruments like MLflow make it potential to trace and model the decide iterations too. We use Databricks Mosaic AI Agent Evaluation, which supplies a framework for including customized Judges (each LLM and programmatic), along with a number of built-in ones with pre-defined prompts (we sometimes flip these off). In code, the core analysis instructions seem like this

with mlflow.start_run(
    run_name=run_name,
    log_system_metrics=True,
    description=run_description,
) as run:

    # run the programmatic checks

    results_programmatic = mlflow.consider(
        predictions="response",
        knowledge=df,  # df comprises the inputs, outputs and any related context, as a pandas dataframe
        model_type="textual content",
        extra_metrics=programmatic_metrics,  # record of customized mlflow metrics, every with a operate describing how the metric is calculated
    )

    # run the llm decide with the extra metrics we configured
    # word that right here we additionally embody a dataframe of few-shot examples to
    # assist information the LLM decide.

    results_llm = mlflow.consider(
        knowledge=df,
        model_type="databricks-agent",
        extra_metrics=agent_metrics,  # agent metrics is an inventory of customized mlflow metrics, every with its personal immediate
        evaluator_config={
            "databricks-agent": {
                "metrics": ["safety"],  # solely preserve the “security” default decide
                "examples_df": pd.DataFrame(agent_eval_examples),
            }
        },
    )

    # Additionally log the prompts (decide and fundamental mannequin) and some other helpful artifacts reminiscent of plots the outcomes together with the run

Beneath the hood, MLflow will difficulty parallel calls to the decide fashions (packaged within the agent metrics record within the code above) and likewise name the programmatic metrics with related features (within the programmatic metrics record), saving the outcomes and related artifacts to Unity Catalog and likewise offering a pleasant person interface with which to match metrics throughout experiments, view traces and browse the LLM decide critiques. It must be famous MLflow 3.0, launched simply after this was written, and has some tooling which will simplify the code above. 

To id enhancements with highest ROI, we will revisit the event set segmentation into personas, functionalities and conditions described in part 2.1. We will evaluate the worth of the optimizing metric between segments and select to focus our immediate iterations on the one with the bottom scores, or with essentially the most regarding edge circumstances. With our analysis loop in place, we will catch any unintended penalties of mannequin updates. Moreover, with monitoring we will reproduce outcomes and revert to earlier immediate variations if wanted. 

2.6 When is it prepared for manufacturing?

In AI functions, and healthcare specifically, some failures are extra consequential than others. We by no means need our chatbot to assert that it’s a healthcare skilled, for instance. However it’s inevitable that our meal scanner will make errors figuring out components in uploaded pictures — people should not notably good at figuring out components from a photograph, and so even human-level accuracy can include frequent errors. It’s subsequently necessary to work with the SMEs and product stakeholders to develop practical thresholds for the optimizing metrics, above which the event work might be declared profitable to allow migration into manufacturing. Some tasks might fail at this stage as a result of it’s not potential to push the optimizing metrics above the agreed threshold with out compromising the satisificing metrics or due to useful resource constraints. 

If the thresholds are very excessive then lacking them barely could be acceptable due to unavoidable error or ambiguity within the LLM decide. For instance we initially set a ship requirement of 100% of our improvement set well being file summaries to be graded as “precisely grounded.” We then discovered that the LLM-judge often would quibble over statements like, “the affected person has recorded their blood stress on most days of the final week”, when the precise variety of days with recordings was 4. In our judgement, an affordable end-user wouldn’t discover this assertion troubling, regardless of the LLM-as-judge classifying it as a failure. Thorough guide overview of failure circumstances is necessary to establish whether or not the efficiency is definitely acceptable and/or whether or not additional iteration is required. 

These go/no-go selections additionally align with the NIST AI Risk Management Framework, which inspires context-driven danger thresholds and emphasizes traceability, validity, and stakeholder-aligned governance all through the AI lifecycle.

Even with a temperature of zero, LLM judges are non-deterministic. A dependable decide ought to give the identical dedication and roughly the identical critique each time it’s on a given instance. If this isn’t occurring, it means that the decide immediate must be improved. We discovered this difficulty to be notably problematic in chatbot testing with the AWS Evaluation Framework, the place every dialog to be graded has a customized rubric and the LLM producing the enter conversations has some leeway on the precise wording of the “person messages”. We subsequently wrote a easy script to run every check a number of occasions and file the common failure charge. Exams with failure at a charge that’s not 0 or 100% might be marked as unreliable and up to date till they turn out to be constant.This expertise highlights the restrictions of LLM judges and automatic analysis extra broadly. It reinforces the significance of incorporating human overview and suggestions earlier than declaring a system prepared for manufacturing. Clear documentation of efficiency thresholds, check outcomes, and overview selections helps transparency, accountability, and knowledgeable deployment. 

Along with efficiency thresholds, it’s necessary to evaluate the system in opposition to recognized safety vulnerabilities. The OWASP Top 10 for LLM Applications outlines widespread dangers reminiscent of immediate injection, insecure output dealing with, and over-reliance on LLMs in high-stakes selections, all of that are extremely related for healthcare use circumstances. Evaluating the system in opposition to this steering may help mitigate downstream dangers because the product strikes into manufacturing.

3.0 Put up-deployment: Monitoring and classification

Shifting an LLM software from improvement to deployment in a scalable, sustainable and reproducible manner is a fancy endeavor and the topic of wonderful “LLMOps” articles like this one. Having a course of like this, which operationalizes every stage of the information pipeline, may be very helpful for evaluation-driven improvement as a result of it permits for brand new iterations to be shortly deployed. Nevertheless, on this part we’ll focus primarily on the best way to really use the logs generated by an LLM software operating in manufacturing to grasp the way it’s performing and inform additional improvement. 

One main purpose of monitoring is to validate that the analysis metrics outlined within the improvement part behave equally with manufacturing knowledge, which is basically a check of the representativeness of the event dataset. This could first ideally be performed internally in a dogfooding or “bug bashing” train, with involvement from unrelated groups and SMEs. We will re-use the metric definitions and LLM judges inbuilt improvement right here, operating them on samples of manufacturing knowledge at periodic intervals and sustaining a breakdown of the outcomes. For knowledge safety at Nuna, all of that is performed inside Databricks, which permits us to reap the benefits of Unity Catalog for lineage monitoring and dashboarding instruments for simple visualization.

Monitoring on LLM-powered merchandise is a broad subject, and our focus right here is on how it may be used to finish the evaluation-driven improvement loop in order that the fashions might be improved and adjusted for drift. Monitoring must also be used to trace broader “product success” metrics reminiscent of user-provided suggestions, person engagement, token utilization, and chatbot query decision. This excellent article comprises extra particulars, and LLM judges may also be deployed on this capability — they might undergo the identical improvement course of described in part 2.4.

This strategy aligns with the NIST AI Danger Administration Framework (“AI RMF”), which emphasizes steady monitoring, measurement, and documentation to handle AI danger over time. In manufacturing, the place ambiguity and edge circumstances are extra widespread, automated analysis alone is commonly inadequate. Incorporating structured human suggestions, area experience, and clear decision-making is crucial for constructing reliable programs, particularly in high-stakes domains like healthcare. These practices help the AI RMF’s core ideas of governability, validity, reliability, and transparency.

Determine 4: Excessive degree workflow displaying parts of the post-deployment knowledge pipeline that permits for monitoring, alerting, tagging and analysis of the mannequin outputs in manufacturing. That is important for evaluation-driven improvement, since insights might be fed again into the event stage. Picture generated by the writer. 

3.1 Further LLM classification

The idea of the LLM decide might be prolonged to post-deployment classification, assigning tags to mannequin outputs and giving insights about how functions are getting used “within the wild”, highlighting surprising interactions and alerting about regarding behaviors. Tagging is the method of assigning easy labels to knowledge in order that they’re simpler to section and analyze. That is notably helpful for chatbot functions: If customers on a sure Nuna app model begin asking our chatbot questions on our blood stress cuff, for instance, this will likely level to a cuff setup downside. Equally, if sure types of treatment container are resulting in larger than common failure charges from our treatment scanning instrument, this implies the necessity to examine and possibly replace that instrument. 

In observe, LLM classification is itself a improvement challenge of the sort described in part 2. We have to construct a tag taxonomy (i.e. an outline of every tag that could possibly be assigned) and prompts with directions about the best way to use it, then we have to use a improvement set to validate tagging accuracy. Tagging typically entails producing constantly formatted output to be ingested by a downstream course of — for instance an inventory of subject ids for every chatbot dialog section — which is why imposing structured output on the LLM calls might be very useful right here, and Databricks has an example of how that is might be performed at scale.

For lengthy chatbot transcripts, LLM classification might be tailored for summarization to enhance readability and shield privateness. Dialog summaries can then be vectorized, clustered and visualized to realize an understanding of teams that naturally emerge from the information. That is typically step one in designing a subject classification taxonomy such because the one the Nuna makes use of to tag our chats. Anthropic has additionally constructed an inside instrument for related functions, which reveals fascinating insights into utilization patterns of Claude and is printed of their Clio research article.

Relying on the urgency of the knowledge, tagging can occur in actual time or as a batch course of. Tagging that appears for regarding conduct — for instance flagging chats for speedy overview in the event that they describe violence, unlawful actions or extreme well being points — could be finest suited to a real-time system the place notifications are despatched as quickly as conversations are tagged. Whereas extra basic summarization and classification can in all probability afford to occur as a batch course of that updates a dashboard, and possibly solely on a subset of the information to cut back prices. For chat classification, we discovered that together with an “different” tag for the LLM to assign to examples that don’t match neatly into the taxonomy may be very helpful. Information tagged as “different” can then be examined in additional element for brand new subjects so as to add to the taxonomy. 

3.2 Updating the event set 

Monitoring and tagging grant visibility into software efficiency, however they’re additionally a part of the suggestions loop that drives analysis pushed improvement. As new or surprising examples are available and are tagged, they are often added to the event dataset, reviewed by the SMEs and run via the LLM judges. It’s potential that the decide prompts or few-shot examples might must evolve to accommodate this new data, however the monitoring steps outlined in part 2.4 ought to allow progress with out the danger of complicated or unintended overwrites. This completes the suggestions loop of evaluation-driven improvement and permits confidence in LLM merchandise not simply after they ship, but in addition as they evolve over time. 

4.0 Abstract 

The speedy evolution of huge language fashions (LLMs) is remodeling industries and affords nice potential to learn healthcare. Nevertheless, the non-deterministic nature of AI presents distinctive challenges, notably in making certain reliability and security in healthcare functions.

At Nuna, Inc., we’re embracing evaluation-driven improvement to deal with these challenges and drive our strategy to AI merchandise. In abstract, the thought is to emphasise analysis and iteration all through the product lifecycle, from improvement to deployment and monitoring. 

Our methodology entails shut collaboration with material specialists to create consultant datasets and outline success standards. We give attention to iterative enchancment via immediate engineering, supported by instruments like MLflow and Databricks, to trace and refine our fashions. 

Put up-deployment, steady monitoring and LLM tagging present insights into real-world software efficiency, enabling us to adapt and enhance our programs over time. This suggestions loop is essential for sustaining excessive requirements and making certain AI merchandise proceed to align with our targets of bettering lives and lowering value of care.

In abstract, evaluation-driven improvement is crucial for constructing dependable, impactful AI options in healthcare and elsewhere. By sharing our insights and experiences, we hope to information others in navigating the complexities of LLM deployment and contribute to the broader purpose of bettering effectivity of AI challenge improvement in healthcare. 

References 

[1] Boston Consulting Group, Digital and AI Options to Reshape Well being Care (2025), https://www.bcg.com/publications/2025/digital-ai-solutions-reshape-health-care-2025

[2] Facilities for Illness Management and Prevention, Excessive Blood Stress Details (2022), https://www.cdc.gov/high-blood-pressure/data-research/facts-stats/index.html

[3] Facilities for Illness Management and Prevention, Diabetes Information and Analysis (2022), https://www.cdc.gov/diabetes/php/data-research/index.html

[4] R.Okay. Arora, et al. HealthBench: Evaluating Giant Language Fashions In direction of Improved Human Well being (2025), OpenAI

Authorship

This text was written by Robert Martin-Brief, with contributions from the Nuna group: Kate Niehaus, Michael Stephenson, Jacob Miller & Pat Alberts