Building Human-In-The-Loop Agentic Workflows | Towards Data Science

like OpenAI’s GPT-5.4 and Anthropic’s Opus 4.6 have demonstrated excellent capabilities in executing long-running agentic duties.

Consequently, we see an elevated use of LLM brokers throughout particular person and enterprise settings to perform complicated duties, comparable to working monetary analyses, constructing apps, and conducting intensive analysis.

These brokers, whether or not a part of a extremely autonomous setup or a pre-defined workflow, can execute multi-step duties utilizing instruments to realize objectives with minimal human oversight.

Nonetheless, ‘minimal’ doesn’t imply zero human oversight. 

Quite the opposite, human assessment stays essential due to LLMs’ inherent probabilistic nature and the potential for errors.

These errors can propagate and compound alongside the workflow, particularly after we string quite a few agentic parts collectively.

You’d have observed the spectacular progress brokers have made within the coding area. The reason being that code is comparatively simple to confirm (i.e., it both runs or fails, and suggestions is seen instantly).

However in areas like content material creation, analysis, or decision-making, correctness is commonly subjective and tougher to judge robotically.

That’s the reason human-in-the-loop (HITL) design stays important.

On this article, we’ll stroll by the right way to use LangGraph to arrange a human-in-the-loop agentic workflow for content material era and publication on Bluesky.

Contents

(1) Primer to LangGraph
(2) Example Workflow
(3) Key Concepts
(4) Code Walkthrough
(5) Best Practices of Interrupts

Yow will discover the accompanying GitHub repo here.

(1) Primer to LangGraph

LangGraph (a part of the LangChain ecosystem) is a low-level agent orchestration framework and runtime for constructing agentic workflows. 

It’s my go-to framework given its excessive diploma of management and customizability, which is significant for production-grade options.

Whereas LangChain presents a middleware object (HumanInTheLoopMiddleware) to simply get began with human oversight in agent calls, it’s completed at a excessive stage of abstraction that masks the underlying mechanics. 

LangGraph, against this, doesn’t summary the prompts or structure, thereby giving us the finer diploma of management that we’d like. It explicitly lets us outline:

• How information flows between steps
• The place choices and code executions occur
• The place human intervention is required

Subsequently, we’ll use LangGraph to reveal the HITL idea inside an agentic workflow.

Additionally it is useful to tell apart between agentic workflows and autonomous AI brokers.

Agentic workflows have predetermined paths and are designed to execute in an outlined order, with LLMs and/or brokers built-in into a number of parts. Then again, AI brokers autonomously plan, execute, and iterate in direction of a purpose.

On this article, we deal with agentic workflows, through which we intentionally insert human checkpoints right into a pre-defined stream.

(2) Instance Workflow

For our instance, we will construct a social media content material era workflow as follows:

1. Person enters a subject of curiosity (e.g., “newest information about Anthropic”).
2. The net search node makes use of the Tavily software to go looking on-line for articles matching the highest.
3. The highest search result’s chosen and fed into an LLM within the content-creation node to generate a social media put up.
4. Within the assessment node, there are two human assessment checkpoints:
   (i) Current generated content material for people to approve, reject, or edit;
   (ii) Upon approval, the workflow triggers the Bluesky API software and requests remaining affirmation earlier than posting it on-line.

Here’s what it seems like when run from the terminal:

And right here is the reside put up on my Bluesky profile:

Bluesky is a social platform just like Twitter (X), and it’s chosen on this demo as a result of its API is far simpler to entry and use.

(3) Key Ideas

The core mechanism behind the HITL setup in LangGraph is the idea of interrupts.

Interrupts (utilizing interrupt() and Command in LangGraph) allow us to pause graph execution at particular factors, show sure data to the human, and await their enter earlier than resuming the workflow.

Command is a flexible object that permits us to replace the graph state (replace), specify the following node to execute (goto), or seize the worth to renew graph execution with (resume).

Here’s what the stream seems like:

(1) Upon reaching the interrupt() perform, execution pauses, and the payload handed into it’s proven to the consumer. The payload handed in interrupt ought to usually be JSON or string format, e.g.,

determination = interrupt("Ought to we get KFC for lunch?") # String proven to consumer

(2) After the consumer responds, we cross the response values to the graph to renew execution. It entails utilizing Command and its resume parameter as a part of re-invoking the graph:

if human_response == "sure":
    return graph.invoke(Command(resume="KFC"))
else:
    return graph.invoke(Command(resume="McDonalds"))

(3) The response worth in resume is returned within the determination variable, which the node will use for the remainder of the node execution and subsequent graph stream:

if determination == "KFC":
    return Command(goto="kfc_order_node", replace={"lunch_choice": "KFC")
else:
    return Command(goto="mcd_order_node", replace={"lunch_choice": "McDonalds")

Interrupts are dynamic and will be positioned wherever within the code, in contrast to static breakpoints, that are mounted earlier than or after particular nodes.

That mentioned, we usually place interrupts both inside the nodes or inside the instruments known as throughout graph execution.

Lastly, let’s speak about checkpointers. When a workflow pauses at an interrupt, we’d like a option to save its present state so it could actually resume later.

We due to this fact want a checkpoint to persist the state in order that the state just isn’t misplaced through the interrupt pause. Consider a checkpoint as a snapshot of the graph state at a given time limit.

For improvement, it’s acceptable to save lots of the state in reminiscence with the InMemorySaver checkpointer.

For manufacturing, it’s higher to make use of shops like Postgres or Redis. With that in thoughts, we will use the SQLite checkpoint on this instance as a substitute of an in-memory retailer.

To make sure the graph resumes precisely on the level the place the interrupt occurred, we have to cross and use the identical thread ID.

Consider a thread as an single execution session (like a separate particular person dialog) the place every one has a novel ID, and maintains its personal state and historical past.

The thread ID is handed into config on every graph invocation in order that LangGraph is aware of which state to renew from after the interrupt.

Now that we’ve got lined the ideas of interrupts, Command, checkpoints, and threads, let’s get into the code walkthrough.

As the main target might be on the human-in-the-loop mechanics, we won’t be masking the excellent code setup. Go to the GitHub repo for the total implementation.

(4) Code Walkthrough

(4.1) Preliminary Setup

We begin by putting in the required dependencies and producing API keys for Bluesky, OpenAI, LangChain, LangGraph, and Tavily.

# necessities.txt
langchain-openai>=1.1.9
langgraph>=1.0.8
langgraph-checkpoint-sqlite>=3.0.3
openai>=2.20.0
tavily-python>=0.7.21

# env.instance
export OPENAI_API_KEY=your_openai_api_key
export TAVILY_API_KEY=your_tavily_api_key
export BLUESKY_HANDLE=yourname.bsky.social
export BLUESKY_APP_PASSWORD=your_bluesky_app_password

(4.2) Outline State

We arrange the State, which is the shared, structured information object serving because the graph’s central reminiscence. It contains fields that seize key data, like put up content material and approval standing.

The post_data key’s the place the generated put up content material might be saved.

(4.3) Interrupt at node stage

We talked about earlier that interrupts can happen on the node stage or inside software calls. Allow us to see how the previous works by establishing the human assessment node.

The aim of the assessment node is to pause execution and current the draft content material to the consumer for assessment.

Right here we see the interrupt() in motion (strains 8 to 13), the place the graph execution pauses on the first part of the node perform.

The particulars key handed into interrupt() comprises the generated content material, whereas the motion key triggers a handler perform (handle_content_interrupt()) to assist the assessment:

The generated content material is printed within the terminal for the consumer to view, they usually can approve it as-is, reject it outright, or edit it immediately within the terminal earlier than approving.

Primarily based on the choice, the handler perform returns one in every of three values:

• True (authorized),
• False (rejected), or
• String worth akin to the user-edited content material (edited). 

This return worth is handed again to the assessment node utilizing graph.invoke(Command=resume…), which resumes execution from the place interrupt() was known as (line 15) and determines which node to go subsequent: approve, reject, or edit content material and proceed to approve.

(4.4) Interrupt at Device stage

Interrupts can be outlined on the software name stage. That is demonstrated within the subsequent human assessment checkpoint within the approve node earlier than the content material is printed on-line on Bluesky.

As an alternative of putting interrupt() inside a node, we place it inside the publish_post software that creates posts through the Bluesky API:

Similar to what we noticed on the node stage, we name a handler perform (handle_publish_interrupt) to seize the human determination:

The return worth from this assessment step is both:

• {"motion": "verify"}, or
• {"motion": "cancel} , 

The latter a part of the code (i.e., from line 19) within the publish_post software makes use of this return worth to find out whether or not to proceed with put up publication on Bluesky or not.

(4.5) Setup Graph with Checkpointer

Subsequent, we join the nodes in a graph for compilation and introduce a SQLite checkpointer to seize snapshots of the state at every interrupt.

SQLite by default solely permits the thread that created the database connection to make use of it. Since LangGraph makes use of a thread pool for checkpoint writes, we have to set check_same_thread=False to permit these threads to entry the connection too.

(4.6) Setup Full Workflow with Config

With the graph prepared, we now place it right into a workflow that kickstarts the content material era pipeline.

This workflow contains configuring a thread ID, which is handed to everygraph.invoke(). This ID is the hyperlink that ties the invocations collectively, in order that the graph pauses at an interrupt and resumes from the place it left off.

You may need observed the __interrupt__ key within the code above. It’s merely a particular key that LangGraph provides to the end result at any time when an interrupt() is hit.

In different phrases, it’s the main sign indicating that graph execution has paused and is ready for human enter earlier than persevering with.

By putting __interrupt__ as a part of a whereas loop, it means the loop retains checking whether or not an interrupt remains to be ongoing. As soon as the interrupt is resolved, the important thing disappears, and the whereas loop exits. 

With the workflow full, we are able to run it like this:

run_hitl_workflow(question="newest information about Anthropic")

(5) Greatest Practices of Interrupts

Whereas interrupts are highly effective in enabling HITL workflows, they are often disruptive if used incorrectly.

As such, I like to recommend studying this LangGraph documentation. Listed here are some sensible guidelines to bear in mind:

• Don’t wrap interrupt calls in attempt/besides blocks, or they won’t pause execution correctly
• Preserve interrupt calls in the identical order each time and don’t skip or rearrange them
• Solely cross JSON-safe values into interrupts and keep away from complicated objects
• Be sure that any code earlier than an interrupt can safely run a number of occasions (i.e., idempotency) or transfer it after the interrupt

For instance, I confronted a difficulty within the net search node the place I positioned an interrupt proper after the Tavily search. The intention was to pause and permit customers to assessment the search outcomes for content material era.

However as a result of interrupts work by rerunning the nodes they have been known as from, the node simply reran the online search and handed alongside a unique set of search outcomes than those I authorized earlier.

Subsequently, interrupts work greatest as a gate earlier than an motion, but when we use them after a non-deterministic step (like search), we have to persist the end result or danger getting one thing completely different on resume.

Wrapping It Up

Human assessment can look like a bottleneck in agentic duties, nevertheless it stays important, particularly in domains the place outcomes are subjective or onerous to confirm.

LangGraph makes it easy to construct HITL workflows with interrupts and checkpointing. 

Subsequently, the problem is deciding the place to position these human determination factors to strike a superb steadiness between oversight and effectivity.