Quick answer
Don't give an agent direct database access. Instead, expose a thin, purpose-built API layer in front of your legacy system, use retrieval-augmented generation (RAG) instead of fine-tuning for internal knowledge, require human approval for any action that writes data, and log every tool call the agent makes — not just the conversation.
Most AI agent demos are built against a clean sandbox — a small dataset, a mock API, a handful of well-formed documents. Real enterprise systems are not clean sandboxes: they're a decade-old database schema, three overlapping authentication systems, and a business process that lives partly in an application and partly in someone's inbox. Here's a practical way to bridge that gap without a costly rewrite.
Start with one narrow, well-defined task
The agents that actually ship in production are narrow: "answer questions about our return policy using our internal knowledge base" succeeds far more often than "be a general assistant for the whole company." Pick one workflow with a clear success criterion — ticket triage, invoice validation, internal search — before expanding scope.
Don't give the agent direct database access
A tempting shortcut is pointing an agent straight at your production database. It's also a fast way to create a security and data-integrity incident. Instead, expose a narrow, purpose-built API layer — a small set of endpoints the agent is allowed to call, each with its own validation and logging. This is usually a thin service sitting in front of the legacy system, not a change to the legacy system itself.
Use retrieval, not fine-tuning, for your internal knowledge
For most enterprise use cases, retrieval-augmented generation (RAG) — pulling relevant internal documents or records into the model's context at query time — is faster to build, easier to update, and easier to audit than fine-tuning a model on internal data. When your policy changes, you update the source document, not the model.
Put a human approval gate on anything that writes data
Read-only agents (answering questions, summarizing, searching) are low-risk. The moment an agent can create a record, send an email, or update a system, that action deserves a review step — at least initially. Multi-agent frameworks like Microsoft's Semantic Kernel make it straightforward to insert an approval step into a workflow without hand-rolling the logic yourself.
Log everything the agent does, not just what it says
A chat transcript alone isn't an audit trail. For any agent that touches business systems, log the tool calls it made, the data it retrieved, and the actions it took — separately from the conversational response. This becomes essential the first time someone asks "why did the agent do that," and it's also the foundation for AI governance reporting as regulation in this space matures.
Measure accuracy before you measure adoption
It's tempting to celebrate usage numbers early. But an agent that's used often and wrong often is worse than no agent at all — it erodes trust in the system faster than it builds it. Establish a small, realistic benchmark set of queries for your specific domain before rolling out broadly, and re-check it whenever you change the underlying model or prompts.
None of this requires touching your legacy system's core code — it requires a thin, well-governed integration layer around it. That's usually the fastest path from "AI demo" to something a real business team relies on daily. See our Vertical AI Agents and Agentic Process Automation services for how we scope this work.
Our free AI & Legacy Modernization Readiness Checklist walks through the exact data and integration checks to run first.