Stop Fine-Tuning Llama (Unless You Have To)

Fine-tuning an open-source LLM has become a status signal in enterprise AI teams. There's a certain prestige to having trained your own model — it sounds more serious, more proprietary, more defensible than 'we're calling the OpenAI API.' The market for GPU clusters, LoRA tutorials, and Hugging Face consultants has exploded accordingly. And yet, most of the fine-tuning projects we encounter in enterprise settings were the wrong solution to the actual problem.

The decision between prompt engineering, retrieval-augmented generation, and fine-tuning is one of the most consequential architectural choices in a GenAI project. Get it wrong and you spend months on infrastructure that delivers worse results than a well-designed prompt would have achieved in a week. This post is the decision framework we use with clients before they commit to a fine-tuning path.

What Fine-Tuning Actually Does

Fine-tuning changes the weights of a pre-trained model on a task-specific dataset. The model learns, during training, to produce outputs that match the examples in your dataset. It changes what the model knows and how it behaves — its default tone, its output format preferences, its likelihood of using certain patterns over others.

Fine-tuning does not reliably inject knowledge. A model fine-tuned on your internal documents does not reliably recall specific facts from those documents at inference time. It might learn the style and terminology of your domain, but if you ask it about a specific policy change from last month's internal memo, it will confabulate. This is the most common misconception that leads teams down the fine-tuning path when they should be building a RAG system.

The Three Reasons to Fine-Tune (and How Rare They Are)

1. You Need a Specific Output Format the Model Consistently Refuses

If your use case requires the model to always output structured JSON, always respond in a specific dialect, always follow a particular template — and prompt engineering alone cannot achieve reliable consistency — fine-tuning on format-compliant examples can help. This is legitimate, though often solvable with structured output APIs or constrained decoding before you reach for fine-tuning.

2. You're Building a Task-Specific Model with No Retrieval

If your application is genuinely a specialised task — text classification, entity extraction, document parsing to a fixed schema — and you need to run it at high volume with low latency and cost, a fine-tuned smaller model often outperforms a larger general-purpose model on that narrow task. This is fine-tuning working as intended: teaching a smaller, faster model to do one thing well.

3. You Have a Specialised Domain with Unique Terminology

If your domain uses highly specialised terminology that doesn't appear in general pre-training data — specific medical procedure codes, proprietary engineering specifications, domain-specific legal language — fine-tuning can help the model understand and produce that terminology fluently. Even here, start by testing whether a strong prompt with terminology examples achieves acceptable results before committing to a training run.

When to Use Prompt Engineering Instead

Prompt engineering — system prompts, few-shot examples, chain-of-thought instructions — solves a wide range of problems that teams reflexively reach for fine-tuning to address. A well-designed system prompt with five examples of correct output format is faster to build, easier to update, and often equivalent in quality to a fine-tuned model for the same task. The advantage of prompts is iteration speed: you can update a prompt in minutes; retraining a model takes days and significant compute.

When to Use RAG Instead

Retrieval-augmented generation is the right architecture when the problem is access to knowledge — your model needs to answer questions about documents, policies, products, or events that either aren't in its training data or change frequently. RAG gives the model real-time access to your knowledge base at inference time, rather than trying to bake that knowledge into weights that will become stale the moment you update a policy document.

• Use RAG when the model needs to recall specific facts from your documents
• Use RAG when your knowledge base changes faster than your training cadence
• Use fine-tuning when the task requires a specific output behaviour that prompt engineering can't reliably achieve
• Use fine-tuning when you're serving high-volume narrow tasks at low latency and cost
• Use both when you need domain-adapted behaviour AND access to a dynamic knowledge base — RAG handles the retrieval; fine-tuning handles the output format and domain terminology

The Hidden Costs of Fine-Tuning

Fine-tuning is not a one-time cost. Every time your requirements change, every time you want to update the model's behaviour, every time the base model releases a new version — you face a choice between retraining, falling behind on base model improvements, or maintaining two diverging model branches. The operational complexity of a fine-tuned model in production is significantly higher than a RAG system over a versioned base model.

The GPU compute, the data preparation, the evaluation pipeline, the serving infrastructure, the retraining cadence — these costs add up to something that is only justified if the problem genuinely cannot be solved another way. Most can.

GYSP's AI & ML Development practice works through this decision framework with every client considering a fine-tuning project. The majority of the time, we redirect them to a better-matched solution that delivers faster results with lower ongoing cost. When fine-tuning is the right call, we design a training and evaluation pipeline that keeps the operational overhead manageable.

“The appeal of fine-tuning is that it sounds like ownership. But a fine-tuned model you can't efficiently update is not ownership — it's debt.”

— Rahul, AI/ML Delivery Head — GYSP.tech