Skip to content
AI.info

Generative AI

Prompt, RAG, Tool, or Fine-Tune?

Choose among prompting, context engineering, retrieval, deterministic tools, and weight adaptation using diagnosis and total system cost.

By the end you can

The wrong intervention can improve the demo and worsen the system

New York City's MyCity chatbot told employers they could take a cut of workers' tips. It told landlords they could refuse tenants with housing vouchers. Both answers broke city and state law. The Adams administration had launched the bot in October 2023 to advise small businesses, and The Markup reported the failures on 29 March 2024.

It was not an unlucky sample. “when ten separate Markup staffers asked the same question, the bot told all of them no, buildings did not have to accept housing vouchers”. On 3 April 2024 the Associated Press reported that the city had left the chatbot online, with Mayor Eric Adams conceding that its answers were “wrong in some areas”.

A system that states the opposite of the governing statute, identically, ten times out of ten, is not short of fluency. It is short of an authority it can be held to. Fine-tuning on newer examples is the tempting reply. It may well change the common answers, but it buries the regulation inside weights: a hidden, slowly updated knowledge store with no version and no citation.

That choice has been measured. A 2024 paper asked the question in its title — Fine-Tuning or Retrieval? — and ran both on the same current-events questions. Mistral 7B scored 0.481 as a base model, 0.504 after regular fine-tuning and 0.588 after fine-tuning on paraphrase-augmented data. Answering the same questions with retrieval scored 0.875. Orca2 7B moved the same way: 0.456 as a base model, 0.511 fine-tuned, 0.876 with retrieval. The authors' own summary is blunt: “Our findings reveal that while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge.”

0.504 against 0.875 on the same questions is the whole argument. The actual need is current, inspectable authority. Retrieval or a structured policy service addresses that need directly, and it supports versioned evidence.

Fix the source of the failure rather than adding the most fashionable mechanism.

Visual

A diagnosis-first intervention map

Move from the cheapest reversible change toward deeper adaptation, and only when evidence justifies it. Two of these branches exist because someone measured them.

Retrieval-augmented generation was introduced at NeurIPS 2020. It paired a pre-trained seq2seq parametric memory with a non-parametric dense vector index of Wikipedia. It set the state of the art on three open-domain QA tasks and produced more specific, diverse and factual language than a parametric-only baseline. The scores were not the stated motivation. The abstract names the two things weights cannot do: “Additionally, providing provenance for their decisions and updating their world knowledge remain open research problems.” That sentence is the evidence branch of this map, written by the people who built it.

The exact-computation branch has its own number. PAL, published in 2023, had the model write a program and hand the arithmetic to an interpreter: “In this paper, we present Program-Aided Language models (PAL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter.” On GSM8K, PAL scored 72.0%, against 65.6% for chain-of-thought with the same Codex model and 56.9% for chain-of-thought with PaLM-540B. Then the authors made the numbers bigger. On GSM-HARD, where one number per question is replaced by a random integer of up to 7 digits, chain-of-thought collapsed to 23.1%. PAL held 61.2%. Prompting lost two thirds of its accuracy to bigger numbers. The interpreter barely noticed them.

FigureProcess · 6 steps
  1. 1

    Is the task unclear?

    Improve instructions, schema, examples, or decomposition.

  2. 2

    Is evidence missing or changing?

    Use retrieval, databases, APIs, or context selection.

  3. 3

    Is exact computation required?

    Call deterministic tools and validate arguments and results.

  4. 4

    Is behavior stable but repeatedly wrong?

    Consider supervised or preference-based fine-tuning.

  5. 5

    Is the base model unsuitable?

    Choose another model family, size, modality, or deployment route.

  6. 6

    Is the task still unjustified?

    Redesign the workflow or do not deploy generation.

Comparison

Interventions differ in what they change and how quickly they update

A complete decision includes who will operate the system, not only how it scores offline. The last row — swap the model, or route between two — is the one most often asserted rather than measured. RouteLLM measured it. Its routers send each query to either a strong or a weak model, and the paper states the result as a headline: “The results show that our routers achieve cost savings of up to 3.66x, demonstrating that routing can significantly reduce cost while maintaining response quality.”

Then read the table under it. At estimated costs of $24.7 per million tokens for GPT-4 and $0.24 per million tokens for Mixtral 8x7B, that 3.66x saving is on MT Bench, while retaining 95% of GPT-4 quality. On MMLU the saving is 1.41x at 92% quality. On GSM8K it is 1.49x at 87%. The same routers, three workloads, and the saving moves by more than a factor of two. Routing gains are a property of the traffic, not of the technique.

FigureComparison · 4 columns

Prompt and context

Change runtime instructions and supplied information.

  • Fast to iterate and roll back
  • Visible in traces
  • Consumes context and can be brittle
  • Does not alter base capabilities

Retrieval and tools

Connect the model to external evidence or deterministic operations.

  • Supports current and inspectable state
  • Adds integration and security work
  • Stage failures are observable
  • Can reduce unsupported generation

Fine-tuning

Modify model weights using curated training signals.

  • Useful for stable repeated behavior
  • Can reduce prompt burden
  • Requires data, compute, and regressions
  • Knowledge updates remain opaque and slow

Model replacement or routing

Select a different model for part or all of the workload.

  • Can improve capability or economics quickly
  • Introduces provider and compatibility changes
  • Needs retesting of every component
  • May be simpler than tuning

Key idea

Several mechanisms can be necessary without becoming one opaque stack

A production system may use a tuned model, retrieval, tools, and structured outputs together. Each component should have a declared job and an evaluation that can fail on its own. Adding every mechanism at once means nobody can tell which one worked. Build a baseline, introduce one change for a measured failure, and keep the ability to remove it later.

The division of labor is visible in the literature. Shuster and colleagues put neural retrieval inside the loop of knowledge-grounded dialogue and counted what it fixed: “We show through automatic and human evaluations that standard (non-retrieval augmented) large language models indeed suffer from hallucination, whereas our best models substantially curtail the issue, reducing hallucinated responses by over 60%.” Their knowledgeability gains over the baseline were 70% on in-distribution data and 85% on out-of-distribution data. The improvement is largest exactly where the weights are emptiest. That is the signature of retrieval doing a job tuning was not doing.

LIMA supplied the complementary result in 2023. A 65B-parameter LLaMa was fine-tuned on only 1,000 curated prompt-response pairs, with no reinforcement learning and no preference modelling. In a controlled human study its responses were equivalent or strictly preferred to GPT-4's in 43% of cases, 58% against Bard and 65% against DaVinci003. The authors' reading of that: “Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.” A thousand examples bought the shape of a good answer. They did not buy its content. That is precisely why fine-tuning is the wrong lever for a missing housing-voucher statute, and retrieval is the wrong lever for an answer that is well sourced and badly formed.

Complex systems remain understandable when every mechanism has a specific contract, metric, owner, and rollback path.

Analogy

Choose a repair after locating the broken part

Repaint the display, replace the sensor, recalibrate the controller, reassign the operator. Four repairs for one machine reporting one wrong measurement, and they fix four different faults. Prompting, retrieval, tools, and fine-tuning are distinct in the same way. The MyCity answer that ten Markup staffers received identically was a wrong measurement, and the broken part was the missing authority, not the wording of the display.

Sensors are simpler than this. A faulty sensor fails identically every time, whereas several causes of a bad answer can interact and mask one another. Locating the broken part is an experiment, not an inspection. That is why the numbers matter more than the intuitions. 0.504 against 0.875. 23.1% against 61.2%. 1.41x against 3.66x. Each one is a diagnosis nobody could have reached by reasoning about the machine from outside.

Diagnosis determines the intervention; architecture fashion should not.

Steps

Run an intervention ladder

Escalate to a more complex step only after the simpler one has been measured fairly. Every anchored comparison in this lesson has the same shape: a fixed task, a base model, one changed mechanism. That shape is the reason its numbers mean anything.

FigureProcess · 6 steps

  1. 1. Freeze a representative test set

    Include the failure, nearby contrasts, and broad regressions.

  2. 2. Establish the base model

    Record prompt, context, decoding, cost, and latency.

  3. 3. Improve the interface

    Test clearer instructions, examples, schemas, and context.

  4. 4. Add external capability

    Evaluate retrieval or deterministic tools for missing evidence and exact work.

  5. 5. Test adaptation

    Fine-tune only when repeated stable behavior remains deficient.

  6. 6. Compare total cost

    Include data, operations, security, evaluation, serving, and change management.

The simplest adequate system is easier to trust and change

No mechanism is inherently more sophisticated in any useful sense. The best choice meets the product contract with the least hidden state and the least operational burden. A chatbot that told ten out of ten testers the wrong thing about housing vouchers, and stayed online while its answers were conceded to be “wrong in some areas”, needed an inspectable source of law. Not a larger training run.

The next lessons examine fine-tuning data, parameter-efficient adaptation, and synthetic supervision. Those tools should enter the architecture only after this decision process supports them, and after someone has produced the equivalent of 0.504 against 0.875 for the specific failure being fixed.

Key takeaways