“Wait… are you sending journal entries to OpenAI?”
the very first thing my pal requested once I confirmed her Really feel-Write, an AI-powered journaling app I constructed throughout a hackathon in San Francisco.
I shrugged.
“It was an AI-themed hackathon, I needed to construct one thing quick.”
She didn’t miss a beat:
“Positive. However how do I belief what you constructed? Why not self-host your personal LLM?”
That stopped me chilly.
I used to be pleased with how rapidly the app got here collectively. However that single query, and those that adopted unraveled every little thing I assumed I knew about constructing responsibly with AI. The hackathon judges flagged it too.
That second made me understand how casually we deal with belief when constructing with AI, particularly with instruments that deal with delicate information.
I spotted one thing greater:
We don’t discuss sufficient about belief when constructing with AI.
Her reply caught with me. Georgia von Minden is an information scientist on the ACLU, the place she works carefully with points round personally identifiable info in authorized and civil rights contexts. I’ve all the time valued her perception, however this dialog hit completely different.
So I requested her to elaborate extra what does belief actually imply on this context? particularly when AI techniques deal with private information.
She advised me:
“Belief will be arduous to pin down, however information governance is an effective place to start. Who has the info, the way it’s saved, and what it’s used for all matter. Ten years in the past, I’d have answered this in a different way. However as we speak, with big computing energy and big information shops, large-scale inference is an actual concern. OpenAI has vital entry to each compute and information, and their lack of transparency makes it affordable to be cautious.
In relation to personally identifiable info, laws and customary sense each level to the necessity for robust information governance. Sending PII in API calls isn’t simply dangerous — it may additionally violate these guidelines and expose people to hurt.”
It jogged my memory that after we construct with AI, particularly techniques that contact delicate human information, we aren’t simply writing code.
We’re making selections about privateness, energy, and belief.
The second you accumulate person information, particularly one thing as private as journal entries, you’re moving into an area of duty. It’s not nearly what your mannequin can do. It’s about what occurs to that information, the place it goes, and who has entry to it.
The Phantasm of Simplicity
At the moment, it’s simpler than ever to spin up one thing that appears clever. With OpenAI or different LLMs, builders can construct AI instruments in hours. Startups can launch “AI-powered” options in a single day. And enterprises? They’re dashing to combine these brokers into their workflows.
However in all that pleasure, one factor typically will get neglected: belief.
When individuals discuss AI Agents, they’re typically referring to light-weight wrappers round LLMs. These brokers may reply questions, automate duties, and even make selections. However many are constructed unexpectedly, with little thought given to safety, compliance, or accountability.
Simply because a product makes use of OpenAI doesn’t imply it’s protected. What you’re actually trusting is the entire pipeline:
- Who constructed the wrapper?
- How is your information being dealt with?
- Is your info saved, logged — or worse, leaked?
I’ve been utilizing the OpenAI API for consumer use circumstances myself. Just lately, I used to be supplied free entry to the API — as much as 1 million tokens day by day till the tip of April — if I agreed to share my immediate information.
(Picture by Creator)
I virtually opted in for a private aspect challenge, however then it hit me: if an answer supplier accepted that very same deal to chop prices, their customers would do not know their information was being shared. On a private degree, which may appear innocent. However in an enterprise context? That’s a severe breach of privateness, and probably of contractual or regulatory obligations.
All it takes is one engineer saying “sure” to a deal like that, and your buyer information is in another person’s fingers.
(Picture by Creator)
Enterprise AI Raises the Stakes
I’m seeing extra SaaS corporations and devtool startups experiment with AI brokers. Some are getting it proper. Some AI Brokers allow you to carry their very own LLM, giving them management over the place the mannequin runs and the way information is dealt with.
That’s a considerate strategy: you outline the belief boundaries.
However not everyone seems to be so cautious.
Many corporations simply plug into OpenAI’s API, add a number of buttons, and name it “enterprise-ready.”
Spoiler: it’s not.
What Can Go Fallacious? A Lot.
If you happen to’re integrating AI brokers into your stack with out asking arduous questions, right here’s what’s in danger:
- Knowledge leakage: Your prompts may embrace delicate buyer information, API keys, or inside logic — and if that’s despatched to a third-party mannequin, it could possibly be uncovered.
In 2023, Samsung engineers unknowingly pasted inside supply code and notes into ChatGPT (Forbes). That information may now be a part of future coaching units — a serious threat for mental property.
- Compliance violations: Sending personally identifiable info (PII) by way of a mannequin like OpenAI with out correct controls can violate GDPR, HIPAA, or your personal contracts.
Elon Musk’s firm X realized that the arduous manner. They launched their AI chatbot “Grok” through the use of all person posts together with from EU customers to coach it, with out correct opt-in. Regulators stepped in rapidly. Beneath stress, they paused Grok’s coaching within the EU (Politico).
- Opaque habits: Non-deterministic brokers are arduous to debug or clarify. What occurs when a consumer asks why a chatbot gave a unsuitable advice or uncovered one thing confidential? You want transparency to reply that — and plenty of brokers as we speak don’t supply it.
- Knowledge possession confusion: Who owns the output? Who logs the info? Does your supplier retrain in your inputs?
Zoom was caught doing precisely that in 2023. They quietly modified their Phrases of Service to permit buyer assembly information for use for AI coaching (Fast Company). After public backlash, they reversed the coverage however it was a reminder that belief will be misplaced in a single day.
- Safety oversights in wrappers: In 2024, Flowise — a well-liked low-code LLM orchestration software — was discovered to have dozens of deployments left uncovered to the web, many with out authentication (Cybersecurity News). Researchers found API keys, database credentials, and person information sitting within the open. That’s not an OpenAI drawback — that’s a builder drawback. However finish customers nonetheless pay the worth.
- AI options that go too far: Microsoft’s “Recall” function — a part of their Copilot rollout — took computerized screenshots of customers’ exercise to assist the AI assistant reply questions (DoublePulsar). It sounded useful… till safety professionals flagged it as a privateness nightmare. Microsoft needed to rapidly backpedal and make the function opt-in solely.
Not All the things Must Be OpenAI
OpenAI is extremely highly effective. Nevertheless it’s not all the time the suitable reply.
Typically a smaller, native mannequin is greater than sufficient. Typically rule-based logic does the job higher. And infrequently, essentially the most safe choice is one which runs completely inside your infrastructure, beneath your guidelines.
We shouldn’t blindly join an LLM and label it a “sensible assistant.”
Within the enterprise, belief, transparency, and management aren’t non-compulsory — they’re important.
There’s a rising variety of platforms enabling that form of management. Salesforce’s Einstein 1 Studio now helps bring-your-own-model, letting you join your personal LLM from AWS or Azure. IBM’s Watson lets enterprises deploy fashions internally with full audit trails. Databricks, with MosaicML, enables you to prepare non-public LLMs inside your personal cloud, so your delicate information by no means leaves your infrastructure.
That’s what actual enterprise AI ought to appear like.
Backside Line
AI brokers are highly effective. They unlock workflows and automations we couldn’t do earlier than. However ease of growth doesn’t imply it’s protected, particularly when dealing with delicate information at scale.
Earlier than you roll out that shiny new agent, ask your self:
- Who controls the mannequin?
- The place is the info going?
- Are we compliant?
- Can we audit what it’s doing?
Within the age of AI, the largest threat isn’t unhealthy expertise.
It’s blind belief.
In regards to the Creator
I’m Ellen, a machine studying engineer with 6 years of expertise, at the moment working at a fintech startup in San Francisco. My background spans information science roles in oil & gasoline consulting, in addition to main AI and information coaching applications throughout APAC, the Center East, and Europe.
I’m at the moment finishing my Grasp’s in Knowledge Science (graduating Could 2025) and actively searching for my subsequent alternative as a machine studying engineer. If you happen to’re open to referring or connecting, I’d really admire it!
I really like creating real-world affect by way of AI and I’m all the time open to project-based collaborations as nicely.
Try my portfolio: liviaellen.com/portfolio
My Earlier AR Works: liviaellen.com/ar-profile
Help my work with a espresso: https://ko-fi.com/liviaellen
