AI Solution Architecture: 8 Questions to Ask Before You Choose a Model

Andrey Koleda on November 17, 2026
AI solution architecture questions to ask

On one enterprise project, we selected five models with care. None of them made it to delivery. Approvals, procurement, and regional availability moved slower than the model market, and every swap meant redoing prompts, tests, and evaluations.

That project changed how we approach AI solution architecture. The model is the least stable part of an enterprise AI design. The parts that last are the problem definition, the data, the constraints, and the type of system you build. So when a request for proposal (RFP) asks "which model should we use?", our answer is "not yet." These are the eight questions we ask first.

Key takeaways

  • Choose the model last. Models are superseded, retired, or unavailable in a region faster than enterprise procurement moves.
  • Before any design work, prove the problem needs AI. Then define a measurable outcome, confirm the data, and list the constraints.
  • Decide the system type (workflow, retrieval, agent, classical machine learning, or a hybrid) before you compare models.
  • Design for model replacement and for failure from the start: stable interfaces, reusable evaluation sets, tracing, and human oversight.
  • A proof of concept should test a specific assumption. Impressing the room is not the goal.

What an RFP asks for and what the business needs

Business teams know their bottlenecks well. They rarely know the AI problem behind them. Three requests show how far apart those can be.

  • Technician training. A company held a knowledge base of video repair instructions. Training technicians of different skill levels was slow and costly. The fit was a retrieval-based system: an engineer asks how to fix a device and gets generated instructions linked to the exact video timestamp.
  • Capital management advisory. Users submit sensitive company data. The system validates the documents, applies an advisory framework, and uses several AI agents to produce recommendations. Confidential financial data made security the main architectural driver.
  • Tender processing. A company received large volumes of tender documents in several languages. One department sorted and analyzed them by hand.

The three projects differ in data, users, risks, and success criteria. That is why the first question is never about the model.

8 questions that shape an AI solution architecture

Each question settles one part of the design before a model is named.

Question What it decides
1. Why do you need AI? Whether AI belongs in the solution at all
2. What will get better, and by how much? The outcome, the baseline, and the business owner
3. Does the data exist, and can we use it? Data readiness, access, and legal use
4. Is it a workflow, a library, or an agent? The system type
5. Which constraints will decide for us? Security, residency, hosting, language, and budget
6. What happens when a model changes? How replaceable each model is
7. How will the system fail? Reliability, recovery, and human oversight
8. What must the proof of concept prove? Scope, assumptions, and the recommendation

1. Why do you need AI?

Use AI only where ordinary software cannot solve the problem with less uncertainty. Deterministic software is usually cheaper and more predictable. AI earns its place when repetitive work depends on unstructured inputs, such as documents, video, or free text, that are costly to turn into structured data.

We also separate two kinds of projects early: adding AI features to an existing application, and building a product where AI is the core experience. They differ in architecture, risk, evaluation, and estimation.

2. What will get better, and by how much?

"We will use AI" is not a business outcome. A usable goal names a measurable improvement: time per task, throughput, or correction rate. Many organizations have no baseline for these numbers, and without one, they cannot show improvement.

We also confirm three things before estimating:

  • A business owner who can define acceptable errors.
  • Domain experts the delivery team can reach.
  • An agreed error tolerance.

When the answers are weak, we propose a discovery phase and write down the assumptions.

3. Does the data exist, and can we use it?

Data work often takes longer than AI integration. Access, cleaning, permissions, and evaluation datasets all sit on the critical path. Before designing anything, we check:

  • Where the data lives and who owns it.
  • Whether it can legally be used for this purpose.
  • How role-based access applies to AI answers.
  • How mature analytics is, and whether baseline metrics exist.
  • Whether processing must run in real time or in batches.

An ambitious AI scope with no data readiness plan is the strongest red flag we see in an RFP.

4. Is it a workflow, a library, or an agent?

Choose the system type before the model. Most enterprise requests fit one of a few patterns, and each suits a different kind of problem.

System type When it fits
Deterministic workflow The process is fully known in advance
Pipeline with one or two model calls The process is fixed, with a step that ordinary code cannot handle
Retrieval-augmented generation (RAG) People need conversational access to a large knowledge base, like a library with a chat interface
AI agent The next step depends on context or on earlier results
Constrained generation The output must follow a fixed schema
Classical machine learning Routine predictions, for example, a random forest or a small vision model
Fine-tuning Rarely the default. It is a separate research and development effort

In the technician training case, RAG results pointed to video timestamps instead of document pages. Agents add complexity, so we use them only when the task needs them.

Hybrid designs are common. Small predictive models handle the routine cases and pass the ambiguous ones to generative models or agents.

5. Which constraints will decide for us?

In enterprise AI, constraints often outweigh raw model capability. We map five of them before comparing any models.

  • Security. Usually the first constraint. Where do prompts, responses, logs, and telemetry go, and which certifications apply? In the capital management case, this shaped the architecture more than anything else.
  • Data residency. The specific model, deployment type, and capacity must exist in the required region, and the full data path must comply.
  • Hosting. Self-hosting is a lifecycle, not a download. It needs graphics processing units (GPUs), specialized operations, monitoring, patching, scaling, and a replacement plan. Model size, precision, GPU memory, context length, concurrency, latency, and throughput all drive the cost. Several of our clients chose managed services after a total cost analysis.
  • Language. Language changes token counts, and token counts change cost. One estimate put Arabic content at about twice the tokens of comparable English content. The ratio varies by model and tokenizer, so measure it on real client documents.
  • Budget and guardrails. Rate limits, quotas, spending thresholds, and alerts are architectural requirements. So is protection against prompt injection, jailbreaks, personal data leaks, and disallowed content.

6. What happens when a model changes?

Treat every model choice as temporary. A model can be superseded, retired, or unavailable in your region. On one project, we had to contact the cloud provider directly to request capacity.

The durable assets are the ones around the model: tests, evaluation sets, and stable interfaces that keep a swap local. With those in place, replacing a model becomes a planned task rather than a rebuild.

That does not mean wrapping every provider in an abstraction layer by default. Add portability where the risk is real: regional availability, model retirement, capacity, pricing, or contractual requirements.

7. How will the system fail?

Agent failures cost more in production than in a demo. A timeout or dropped connection after an external action, followed by a blind retry, can duplicate an order, message, or financial operation.

Production agent workflows need:

  • Step-level tracing.
  • Timeouts and controlled retries.
  • Durable checkpoints and a record of completed actions.
  • Idempotent tools or idempotency keys, so a repeated call has no second effect.
  • Compensation for partially completed work.
  • Alerts and a way for an operator to step in.

Generative systems are not deterministic, and no design makes them so. If the business needs strict outputs, use deterministic workflows, constrained generation, and validation. For medical, legal, financial, and other high-impact decisions, keep a person accountable for the final call.

8. What must the proof of concept prove?

A proof of concept (PoC) should test a specific assumption. Demos and prototypes are useful discovery tools because clients react more to a working system than to a document.

Demos also trigger feature requests on the spot, and those requests can expand scope without anyone deciding to. We record each one in the proposal as an explicit option or assumption.

A strong proposal then does three things:

  1. It separates facts, assumptions, open questions, and risks.
  2. It compares a small number of viable architectures against clear criteria: security, data readiness, quality, cost, regional availability, delivery time, and operational complexity.
  3. It gives one main recommendation, with the alternatives as explicit branches.

AI tools help us explore alternatives. We still verify every conclusion against official documentation, real availability, cost calculations, and expert input.

AI model selection comes last

When an RFP asks which model to use, the answer is still "not yet." First establish that the problem needs AI, which outcome will improve, what data is available, and which constraints apply. Then choose the models. You usually use several, mixed with ordinary software, and you can replace any of them.

None of the five models we started with survived that project. The questions did, and we bring them to every AI engagement.

Planning an AI initiative or preparing an RFP? Talk to our AI team.

Frequently asked questions

What is AI solution architecture?

AI solution architecture is the set of design decisions that define how an AI system solves a business problem. It covers the system type, the data it uses, the models, security and compliance, hosting, and how the system is monitored and recovered when it fails.

When should you choose the model in an enterprise AI project?

Choose the model last. First confirm that the problem needs AI, define a measurable outcome, check that the data exists and can be used, and list the constraints. Models change faster than enterprise approvals and procurement, so treat every model choice as temporary.

RAG vs. AI agents: when do you need an agent?

Use an agent when the next step depends on context or on earlier results. Use a deterministic workflow when the process is fully known. Use retrieval-augmented generation (RAG) when people need conversational access to a large knowledge base. Agents add complexity, so use them only when necessary.

Is self-hosting a large language model cheaper than a managed service?

Not necessarily. Self-hosting requires GPUs, specialized operations, monitoring, patching, scaling, and a plan to replace the model. Compare the total cost of ownership, not the license price. Several of our clients chose managed services after that analysis.

What should an AI proof of concept prove?

An AI proof of concept should test a specific assumption that the architecture depends on. Record feature requests that come up during a demo as explicit options or assumptions, so the scope does not grow unnoticed.

What is the biggest red flag in an AI RFP?

An ambitious AI scope with no plan for data readiness. Access, ownership, legal use, permissions, and evaluation data often take longer than the AI work itself.

More on Data and AI

All articles