10 questions to Ask an AI Development Company before Signing

TL;DR

• Most AI projects fail, so picking the vendor is the decision that matters most. By some estimates, more than 80% of AI projects fail. That’s twice the rate of ordinary IT projects (RAND).
• The market makes vetting hard on purpose. After ChatGPT, plenty of agencies relabeled web development as “AI development.” A polished demo tells you nothing about production skill.
• Ten questions expose the difference. Each comes with what a good answer sounds like and what a red flag sounds like. So you score answers, not impressions.
• Two questions matter more than buyers expect: true total cost, and the EU AI Act. The wrong architecture can quietly change your own regulatory role.
• Use the weighted scorecard. Grade every vendor on the same ten answers, and the choice usually makes itself.

---

Choosing an AI development company comes down to asking questions a rebranded dev shop can’t answer well. That means production systems, data, architecture trade-offs, security, regulation, ongoing costs, and ownership. The ten below, each with a good-answer script and a red-flag script, are that filter.

The stakes justify the homework. By some estimates, more than 80% of AI projects fail (RAND, 2024). That’s twice the failure rate of IT projects without AI. MIT’s Project NANDA put it more bluntly in 2025: just 5% of integrated AI pilots were extracting real value. The rest showed no measurable P&L impact. And S&P Global found the share of companies abandoning most AI initiatives jumped from 17% to 42% in one year.

Those failures rarely trace to the model. They trace to the problem being misunderstood, the data being unready, and the system never being engineered for production. In other words, they trace to the vendor and the setup. So the interview below is the cheapest insurance you can buy.

Why AI vendors are harder to vet than software vendors

Ordinary software either works or it doesn’t, and a portfolio shows you which. But AI is different in two ways.

First, the output is probabilistic. A system can demo beautifully and still fail in production. The demo never met messy data, adversarial users, or a model update. So past production behavior, not past demos, is the only evidence that counts.

Second, the market is muddy. After ChatGPT launched, a wave of agencies repackaged existing services as AI development. That’s an industry-observed pattern rather than a measured statistic, but every buyer runs into it. Meanwhile, a GPT API call wired to a web form looks, in a pitch, the same as an engineered system. The questions below are designed to make the difference audible.

The 10 questions

So ask all ten, in a live conversation, and take notes against the scorecard at the end. The pattern to listen for throughout: good answers are specific, quantified, and honest about failure. Red-flag answers are confident and vague.

1. Name an AI system you took to production. How long has it been live, and what broke after launch?

• Why it matters: most AI projects die at the pilot stage, so a demo reel proves nothing. Production tenure and post-launch repairs are the real credential.
• A good answer: names the system, explains it technically, and volunteers what broke. For example: “users asked questions outside the indexed corpus, so we added a retrieval-confidence threshold and a fallback.” Then look for a pilot-to-production rate near 70% or better, with honest reasons for the rest.
• A red flag: a vague portfolio that collapses under one follow-up, “it performed well on the test set,” or no live system they can name.

2. How do you assess our data before committing to an approach?

• Why it matters: data problems are the dominant failure driver. In our own survey of 72 executives across 30+ industries, data quality was the top blocker to production AI, cited by 58%.
• A good answer: they ask about your data before your budget: where it lives, how clean it is, who owns it, what’s sensitive. They treat data readiness as a gate, not an afterthought.
• A red flag: “data isn’t a concern at this stage” or “we’ll handle that as it comes up.” It always comes up.

3. Would you use RAG, fine-tuning, agents, or classical ML here — and why?

• Why it matters: this choice drives cost, accuracy, and lock-in, and the honest answer is sometimes “you don’t need AI for this at all.”
• A good answer: trade-offs in plain language: RAG to answer from your documents, fine-tuning for tone, agents only where autonomy earns its complexity. Expect a question back, too, about what in your data would change the recommendation.
• A red flag: one default pattern for everything, “we use ChatGPT,” or an immediate push toward custom fine-tuning. That push can mean billable R&D you don’t need.

4. How will this integrate with our existing systems?

• Why it matters: production AI lives inside CRMs, ERPs, warehouses, and ticketing queues. Without a path into your workflow, the model is a demo with a monthly bill.
• A good answer: a discovery pass over your APIs, data flows, access model, and latency needs. Then a clear statement of what must be built or fixed before the AI layer goes in.
• A red flag: no curiosity about your stack, or an architecture that quietly assumes a greenfield.

5. What are your security certifications, and how do you handle GDPR and the EU AI Act — including whether your design changes our regulatory role?

• Why it matters: the EU AI Act carries penalties up to €35M or 7% of global turnover. Its high-risk obligations were scheduled for August 2, 2026. The Digital Omnibus defers the Annex III set to December 2, 2027. Here’s the part buyers miss: if a vendor substantially modifies a model, heavy fine-tuning for instance, you can be reclassified from “deployer” to “provider.” That role carries far heavier obligations.
• A good answer: named certifications (ISO 27001, ISO 9001, or equivalents), VPC-isolated deployment, and a written guarantee your data never trains public models. Add adversarial testing before release, and a clear view on whether their architecture keeps you a deployer.
• A red flag: “we take security very seriously” with zero specifics, or a blank look at the AI Act.

6. How do you monitor drift and support the system after launch — and who owns maintenance?

• Why it matters: model providers ship updates and your data shifts. So an AI system degrades quietly if nobody watches it. Ship-and-forget delivery is where AI investments go to die.
• A good answer: live monitoring of accuracy, drift, and latency; retraining triggers with human approval; model versioning with rollback. Plus an SLA, and a named owner for maintenance as distinct from the build.
• A red flag: “we’d catch that in the next quarterly review,” or no structured post-launch process at all.

7. What’s your evaluation methodology — how do you know the system is right, before and after release?

• Why it matters: probabilistic systems need evaluation harnesses, not just functional QA. Bugs you can’t reproduce on demand still cost you customers.
• A good answer: evaluation sets tied to every release (“no merge without an eval”), tracing and observability tooling, and hallucination-rate alerts. Versioned prompts round it out, so a bad one rolls back like code.
• A red flag: “we log everything to CloudWatch,” or problems that were first noticed when a user complained.

8. What’s the total cost — including inference, tokens, and monthly infrastructure — not just development?

• Why it matters: the development quote is the visible fraction. Token and infrastructure economics decide whether the system makes or loses money at scale. A design that ignores them turns success into a penalty.
• A good answer: a breakdown across build, monthly infrastructure, model/API spend, and maintenance. Ideally you get a modeled TCO under different loads before committing to rollout, as a scoped pilot should produce.
• A red flag: only the development cost quoted, or “we’ll figure out infrastructure later.”

9. Who owns the IP, the code, the model, and the data?

• Why it matters: some contracts assign IP to the vendor or retain licensing rights over the model. That converts your project into their product, and you into a captive customer.
• A good answer: a clear, written assignment: you own the code once the work is paid for. Your data stays inside your cloud boundary and never trains public models. No vendor licensing tail on the delivered system, and all of it confirmed in the contract before work begins.
• A red flag: resistance to written IP terms, retained model rights, or “our legal team handles that later.”

10. Who will actually do the work — and will senior people be in discovery?

• Why it matters: the people who win the pitch often aren’t the people who build. It’s the most common complaint in outsourced AI work. Discovery run by juniors produces junior architecture.
• A good answer: named senior engineers with backgrounds you can check, and clarity on who holds architectural sign-off. Senior people sit in the discovery room from day one.
• A red flag: founder credentials in the pitch, unnamed “delivery teams” in the contract.

Scoring the answers

Impressions lie; scores don’t. So grade every vendor 1–5 on each category, multiply by the weight, and compare totals.

Category	Weight	What you’re grading
Production experience	20%	Q1 — live systems, tenure, honest failure stories
Technical & architecture depth	20%	Q3, Q4, Q7 — trade-offs, integration, evaluation
Data strategy	15%	Q2 — readiness assessed before approach
Security & EU AI Act	15%	Q5 — certifications, isolation, regulatory role
MLOps & post-launch	10%	Q6 — monitoring, drift, named ownership
Cost transparency	10%	Q8 — TCO including tokens and infrastructure
IP & ownership	10%	Q9 — written assignment, no lock-in

The remaining signal — Q10, who does the work — acts as a veto rather than a weight: if the senior team isn’t in discovery, the other scores don’t survive contact with delivery. Regulatory and IP terms carry a full quarter of the weight here, and that’s deliberate, because they’re the two categories most vendor checklists skip and the two hardest to fix after signing.

How we’d answer these ourselves

We at SumatoSoft publish our answers, because a vetting guide from a vendor should survive its own test. AI work here runs under the Agentic Development Lifecycle (ADLC) — our framework for building probabilistic systems with the governance built in: readiness assessment as a gate, token-economics modeling before build, red-teaming before release, evaluation on every change, and drift monitoring after launch. In practice, data runs in VPC-isolated environments and never trains public models. You own the code on payment. That’s 350+ projects over 14+ years across 25+ countries, ISO 27001- and ISO 9001-certified, with a 98% satisfaction rate.

So bring this exact list to a call with us and ask all ten — talk to us. If our answers don’t beat the scorecard, you’ll have lost thirty minutes and gained a calibration.

Frequently asked questions

What questions should you ask an AI development company?

Ten cover the ground: a production system they can name, how they assess your data, which architecture they’d choose and why, integration with your stack, security and EU AI Act handling, post-launch monitoring, evaluation methodology, total cost including tokens, IP and data ownership, and who actually does the work. Then grade the answers against a written scorecard rather than by impression.

How can you tell a real AI development company from a rebranded agency?

Ask about production, not demos. A real one names live systems, volunteers what broke after launch, explains architecture trade-offs, and models token costs before building. A rebranded one gives confident, vague answers, defaults to a single pattern for every problem, and quotes only the development cost.

Why do so many AI projects fail?

By some estimates, more than 80% fail — roughly twice the rate of ordinary IT projects (RAND, 2024). The recurring causes are misunderstood problems, unready data, and systems never engineered for production. Vendor selection sits upstream of all three, which is why the vetting interview matters.

What is the EU AI Act, and why does it matter when hiring a vendor?

It’s the EU’s AI regulation, with penalties up to €35M or 7% of global turnover. High-risk obligations were scheduled from August 2, 2026, with the Annex III set deferred to December 2, 2027. It matters at hiring time because a vendor’s design choices — for example, heavy fine-tuning versus retrieval — can shift your own role from “deployer” to “provider,” which changes your obligations substantially.

Who should own the IP in an AI development contract?

You should — in writing, before work begins. That means full code ownership on payment, your data staying inside your own cloud boundary and never training public models, and no vendor licensing rights over the delivered system. Resistance on any of these is a reason to walk.

How much does it cost to hire an AI development company?

It varies too widely for one number, which is exactly why question eight matters: insist on total cost of ownership — development plus monthly infrastructure, model and token spend, and maintenance — modeled before rollout, not just a build quote. A scoped pilot with a TCO projection is the cleanest way to get a real figure.

Summary

AI vendor selection is a different sport from software vendor selection, because demos prove nothing and the market is full of confident generalities. The ten questions above turn the interview into evidence: production history, data discipline, architecture honesty, integration, security and the EU AI Act, post-launch ownership, evaluation, true cost, IP, and the actual team. Score the answers, and weight regulation and ownership as heavily as the technology. Finally, treat vague confidence as the loudest red flag in the room. The vendors worth hiring will enjoy the interrogation.