From AI Prototype to Production

What changes when an AI prototype goes from a few friendly users to production?

Everything around the model changes, even when the model does not. Friendly users forgive slow answers, odd outputs and lost data; production users need the tool to be secure, monitored, backed up and supported by someone with time to fix it.

• Users: more people, more edge cases and more ways to misuse the tool.
• Data: real customer or company data, often more sensitive than anything in the test.
• Load: more requests at once, and model costs that grow with every call.
• Expectations: someone has to answer when it breaks, not whoever built it in their spare time.

The same holds for an AI agent prototype to production move, with one addition: an agent that acts inside your systems needs narrow permissions and a log of every action before more people rely on it.

What should we check before more people use a tool we built with ChatGPT?

Check the gaps that AI-written code most often leaves: secrets in the code, missing sign-in and permission checks, unchecked inputs, outdated dependencies and no backups. Fix them before you widen access, because each one gets harder to fix once real users and data are in the system.

• Secrets: API keys and passwords moved out of the code into a secrets store, then rotated.
• Access: every page and API checks who the user is and what they may see.
• Inputs: user text treated as untrusted, including text that is sent on to the model.
• Dependencies: unused packages removed, the rest updated and scanned.
• Data: backups running, and one restore tested.
• Logging: errors and model calls logged, without sensitive data in the logs.

Prompt injection deserves its own line on that vibe coded app security checklist. The OWASP Top 10 for LLM Applications (2025) ranks prompt injection first: user prompts that alter a model's behavior or output in unintended ways.

Many teams arrive with a simple request, "fix my vibe coded app", and this list is usually where that work starts, before any new feature.

How do we test an AI tool's answers before launch?

Build a small test set from real questions your users ask, write down what a good answer looks like for each, and run the tool against it before every release. That turns "it seems fine" into a pass or fail you can repeat.

• Real tasks, not invented ones, including the awkward cases friendly users never tried.
• A pass rule for each answer: correct, grounded in your data, safe and in the right format.
• Attack cases: attempts to make the tool ignore its instructions or reveal data.
• A rerun after every prompt edit, model change or data change.

Should we keep the prototype or rebuild it as an MVP?

Keep the parts that work and can be secured, and rebuild the parts that cannot be tested or made safe. The decision is usually made part by part, not all or nothing.

Keep the prototype's core when the code is readable, the data model fits real use and the review found fixable gaps. Rebuild as an MVP when the code cannot be tested, when security fixes would touch every file, or when the tool must connect to systems the prototype was never designed for.

Who maintains the tool once it is live?

Someone with the time, access and skills to keep it running, which is rarely the person who built it on the side. A maintenance owner fixes bugs, updates dependencies, watches the model's answers and handles user questions.

For many teams the answer is an outside maintenance partner with a written agreement: what is covered, how requests are prioritized and how the tool is handed back if the arrangement ends. If the build stalled before launch, the first job is a handover review and a rescue plan, not maintenance.

How much does it cost to maintain an AI tool?

It depends on how much the tool changes, how many people use it and how often it calls the model. There is no fixed price, but the cost drivers are predictable, so you can plan a yearly budget before launch.

• Model usage: every request to a hosted model has a usage cost that grows with users and prompt length.
• Change rate: frequent prompt, data or feature changes need more testing and release work.
• Integrations: each connected system adds updates and failure points to watch.
• Data and compliance: sensitive data adds monitoring, access reviews and audit work.
• Support: user questions and bug reports grow with the user base.

What changes when the AI tool handles healthcare data?

Risk management becomes a written process, not a judgment call. Healthcare AI risk management means knowing which data the tool touches, what harm a wrong answer could cause and who reviews outputs before they affect a patient or a record.

The NIST AI Risk Management Framework is a voluntary framework that organizes AI risk work into four functions: Govern, Map, Measure and Manage. It gives a small team a practical structure: name an owner, list the risks, test against them and act on what the tests show.

If the tool stores or processes protected health information, it also needs HIPAA-ready hosting, access controls and audit logging before wider use, and a person reviewing any output that could reach a patient.

Sources

• OWASP: OWASP GenAI Security Project, LLM01:2025
• NIST: NIST AI 100-1, AI Risk Management Framework