AM on a Tuesday (effectively, technically Wednesday, I suppose), when my cellphone buzzed with that acquainted, dreaded PagerDuty notification.
I didn’t even must open my laptop computer to know that the daily_ingest.py script had failed. Once more.
It retains failing as a result of our knowledge supplier all the time modifications their file format with out warning. I imply, they might randomly change from commas to pipes and even mess up the dates in a single day.
Often, the precise repair takes me nearly thirty seconds: I merely open the script, swap sep=',' for sep='|', and hit run.
I do know that was fast, however in all honesty, the true price isn’t the coding time, however relatively the interrupted sleep and the way arduous it’s to get your mind working at 2 AM.
This routine received me pondering: if the answer is so apparent that I can determine it out simply by glancing on the uncooked textual content, why couldn’t a mannequin do it?
We regularly hear hype about “Agentic AI” changing software program engineers, which, to me, actually feels considerably overblown.
However then, the thought of utilizing a small, cost-effective LLM to behave as an on-call junior developer dealing with boring pandas exceptions?
Now that gave the impression of a undertaking price making an attempt.
So, I constructed a “Self-Therapeutic” pipeline. Though it isn’t magic, it has efficiently shielded me from at the least three late-night wake-up calls this month.
And personally, something (regardless of how little) that may enhance my sleep well being is unquestionably a giant win for me.
Right here is the breakdown of how I did it so you may construct it your self.
The Structure: A “Strive-Heal-Retry” Loop
The core idea of that is comparatively easy. Most knowledge pipelines are fragile as a result of they assume the world is ideal, and when the enter knowledge modifications even barely, they fail.
As a substitute of accepting that crash, I designed my script to catch the exception, seize the “crime scene proof”, which is principally the traceback and the primary few strains of the file, after which cross it right down to an LLM.
Fairly neat, proper?
The LLM now acts as a diagnostic instrument, analyzing the proof to return the appropriate parameters, which the script then makes use of to mechanically retry the operation.
To make this method sturdy, I relied on three particular instruments:
- Pandas: For the precise knowledge loading (clearly).
- Pydantic: To make sure the LLM returns structured JSON relatively than conversational filler.
- Tenacity: A Python library that makes writing advanced retry logic extremely clear.
Step 1: Defining the “Repair”
The first problem with utilizing Giant Language Fashions for code era is their tendency to hallucinate. From my expertise, in the event you ask for a easy parameter, you typically obtain a paragraph of conversational textual content in return.
To cease that, I leveraged structured outputs by way of Pydantic and OpenAI’s API.
This forces the mannequin to finish a strict kind, performing as a filter between the messy AI reasoning and our clear Python code.
Right here is the schema I settled on, focusing strictly on the arguments that mostly trigger read_csv to fail:
from pydantic import BaseModel, Area
from typing import Elective, Literal
# We want a strict schema so the LLM does not simply yap at us.
# I am solely together with the params that truly trigger crashes.
class CsvParams(BaseModel):
sep: str = Area(description="The delimiter, e.g. ',' or '|' or ';'")
encoding: str = Area(default="utf-8", description="File encoding")
header: Elective[int | str] = Area(default="infer", description="Row for col names")
# Typically the C engine chokes on regex separators, so we let the AI change engines
engine: Literal["python", "c"] = "python"By defining this BaseModel, we’re successfully telling the LLM: “I don’t need a dialog or an evidence. I need these 4 variables stuffed out, and nothing else.”
Step 2: The Healer Operate
This perform is the center of the system, designed to run solely when issues have already gone improper.
Getting the immediate proper took some trial and error. And that’s as a result of initially, I solely offered the error message, which pressured the mannequin to guess blindly on the drawback.
I rapidly realized that to appropriately determine points like delimiter mismatches, the mannequin wanted to truly “see” a pattern of the uncooked knowledge.
Now right here is the large catch. You can’t truly learn the entire file.
Should you attempt to cross a 2GB CSV into the immediate, you’ll blow up your context window and apparently your pockets.
Happily, I came upon that simply pulling the primary few strains offers the mannequin simply sufficient data to repair the issue 99% of the time.
import openai
import json
shopper = openai.OpenAI()
def ask_the_doctor(fp, error_trace):
"""
The 'On-Name Agent'. It seems to be on the file snippet and error,
and suggests new parameters.
"""
print(f"🔥 Crash detected on {fp}. Calling LLM...")
# Hack: Simply seize the primary 4 strains. No must learn 1GB.
# We use errors='exchange' so we do not crash whereas making an attempt to repair a crash.
attempt:
with open(fp, "r", errors="exchange") as f:
head = "".be a part of([f.readline() for _ in range(4)])
besides Exception:
head = "<<FILE UNREADABLE>>"
# Preserve the immediate easy. No want for advanced "persona" injection.
immediate = f"""
I am making an attempt to learn a CSV with pandas and it failed.
Error Hint: {error_trace}
Knowledge Snippet (First 4 strains):
---
{head}
---
Return the right JSON params (sep, encoding, header, engine) to repair this.
"""
# We drive the mannequin to make use of our Pydantic schema
completion = shopper.chat.completions.create(
mannequin="gpt-4o", # gpt-4o-mini can be advantageous right here and cheaper
messages=[{"role": "user", "content": prompt}],
capabilities=[{
"name": "propose_fix",
"description": "Extracts valid pandas parameters",
"parameters": CsvParams.model_json_schema()
}],
function_call={"identify": "propose_fix"}
)
# Parse the end result again to a dict
args = json.masses(completion.selections[0].message.function_call.arguments)
print(f"💊 Prescribed repair: {args}")
return argsI’m type of glossing over the API setup right here, however you get the thought. It takes the “signs” and prescribes a “tablet” (the arguments).
Step 3: The Retry Loop (The place the Magic Occurs)
Now we have to wire this diagnostic instrument into our precise knowledge loader.
Prior to now, I wrote ugly whereas True loops with nested attempt/besides blocks that have been a nightmare to learn.
Then I discovered tenacity, which lets you adorn a perform with clear retry logic.
And the perfect half is that tenacity additionally means that you can outline a customized “callback” that runs between makes an attempt.
That is precisely the place we inject our Healer perform.
import pandas as pd
from tenacity import retry, stop_after_attempt, retry_if_exception_type
# A unclean world dict to retailer the "repair" between retries.
# In an actual class, this is able to be self.state, however for a script, this works.
fix_state = {}
def apply_fix(retry_state):
# This runs proper after the crash, earlier than the following try
e = retry_state.consequence.exception()
fp = retry_state.args[0]
# Ask the LLM for brand new params
suggestion = ask_the_doctor(fp, str(e))
# Replace the state so the following run makes use of the suggestion
fix_state[fp] = suggestion
@retry(
cease=stop_after_attempt(3), # Give it 3 strikes
retry_if_exception_type(Exception), # Catch every part (dangerous, however enjoyable)
before_sleep=apply_fix # <--- That is the hook
)
def tough_loader(fp):
# Examine if we've got a recommended repair for this file, in any other case default to comma
params = fix_state.get(fp, {"sep": ","})
print(f"🔄 Making an attempt to load with: {params}")
df = pd.read_csv(fp, **params)
return dfDoes it truly work?
To check this, I created a purposefully damaged file known as messy_data.csv. I made it pipe-delimited (|) however didn’t inform the script.
After I ran tough_loader('messy_data.csv'), the script crashed, paused for a second whereas it “thought,” after which fastened itself mechanically.
It feels surprisingly satisfying to look at the code fail, diagnose itself, and recuperate with none human intervention.
The “Gotchas” (As a result of Nothing is Good)
I don’t need to oversell this answer, as there are positively dangers concerned.
The Value
First, keep in mind that each time your pipeline breaks, you make an API name.
That may be advantageous for just a few errors, however in case you have an enormous job processing, let’s say about 100,000 information, and a nasty deployment causes all of them to interrupt directly, you can get up to a really nasty shock in your OpenAI invoice.
Should you’re working this at scale, I extremely suggest implementing a circuit breaker or switching to an area mannequin like Llama-3 by way of Ollama to maintain your prices down.
Knowledge Security
Whereas I’m solely sending the primary 4 strains of the file to the LLM, that you must be very cautious about what’s in these strains. In case your knowledge comprises Personally Identifiable Data (PII), you might be successfully sending that delicate knowledge to an exterior API.
Should you work in a regulated business like healthcare or finance, please use an area mannequin.
Significantly.
Don’t ship affected person knowledge to GPT-4 simply to repair a comma error.
The “Boy Who Cried Wolf”
Lastly, there are occasions when knowledge ought to fail.
If a file is empty or corrupt, you don’t need the AI to hallucinate a option to load it anyway, probably filling your DataFrame with rubbish.
Pydantic filters the dangerous knowledge, nevertheless it isn’t magic. You need to watch out to not conceal actual errors that you just really need to repair your self.
Conclusion and takeaway
You could possibly argue that utilizing an AI to repair CSVs is overkill, and technically, you may be proper.
However in a subject as fast-moving as knowledge science, the perfect engineers aren’t those clinging to the strategies they discovered 5 years in the past; they’re those always experimenting with new instruments to resolve previous issues.
Actually, this undertaking was only a reminder to remain versatile.
We are able to’t simply maintain guarding our previous pipelines; we’ve got to maintain discovering methods to enhance them. On this business, essentially the most worthwhile talent isn’t writing code sooner; relatively, it’s having the curiosity to attempt a complete new method of working.
