Fine-tuning an open-source language model has moved from a specialised deep-research task to a practical, budget-friendly workflow any technical team can implement—provided you know which levers to pull. Off-the-shelf models like Llama 3, Mistral, or Qwen deliver impressive general knowledge, but they lack the domain-specific nuance, internal terminology, and consistent tone that make a business tool truly useful. With Parameter-Efficient Fine-Tuning (PEFT) and Low-Rank Adaptation (LoRA), you can adapt a 7B-parameter model on a single consumer GPU with 24 GB of VRAM—no cluster, no cloud credits, no university budget. This guide walks you end-to-end: curating a clean dataset, configuring LoRA hyperparameters, training on an RTX 3090 or 4090, evaluating with both automated metrics and human review, and finally deploying your custom model via Ollama for low-latency inference. By the end, you will have a production-ready pipeline that converts a generic base model into a domain specialist—without sacrificing control or incurring per-token API costs.
Why Fine-Tuning Matters for Your Business Model
Generic models answer broad questions well, but they fail at tasks requiring consistent formatting, internal vocabulary, or nuanced decision logic. A legal assistant that hallucinates case citations or a customer-support chatbot that uses a flippant tone directly harms credibility. Fine-tuning solves this by adjusting the model's weights to align with your specific data distribution. With LoRA, you do not retrain the entire model—instead, you inject trainable low-rank matrices into key layers, reducing the parameter count to train from billions to a few million. This slashes VRAM requirements to roughly 8–12 GB for a 7B model, making it feasible on consumer hardware.
Beyond cost and privacy—your data never leaves your machine—fine-tuning delivers measurable performance gains. In a recent benchmark, a LoRA-tuned 7B model on legal contracts outperformed GPT-4 on domain-specific extraction tasks by 8 percent while running at a fraction of the latency. For businesses processing internal documents, support tickets, or codebases, this delta translates into hours of saved manual review each week. The decision to fine-tune is no longer a question of capability but of workflow efficiency. If your team repeatedly prompts a base model with lengthy context and still receives inconsistent results, you are ready to move from prompting to training.
Preparing a High-Quality Dataset for Instruction Tuning
Your model is only as good as your data. Start by collecting real-world examples of the exact input-output pairs you expect the model to handle in production. For a customer-support assistant, that means actual resolved tickets with the customer query and the agent's final answer—not synthetic variations. For a code generator, it means paired function descriptions with clean, idiomatic implementations from your own codebase. Aim for at least 500 high-quality examples; 2,000–10,000 is ideal. Quality wins over quantity every time—a handful of inconsistent or incorrectly formatted examples can degrade performance for the entire task.
Format your data using a consistent chat template that matches your base model's tokenizer. For Llama and Mistral-based models, use the following structure:
{"instruction": "Classify this email as spam or not-spam.", "input": "Claim your free prize now!", "output": "Spam."}Convert these into prompt-completion pairs your trainer understands. Use libraries like datasets (Hugging Face) or jsonl files with one example per line. Validate every example for encoding errors, missing fields, and label leakage—do not include the answer in the input field. A simple Python script that checks for empty strings, duplicate hashes, and token-length limits will save you hours of debugging later. Finally, split your dataset into training (80 %), validation (10 %), and test (10 %) sets, stratifying by task category if your data covers multiple domains.
Setting Up LoRA Training on Consumer Hardware
With your dataset ready, the next step is configuring a training environment that fits within 24 GB of VRAM. Start with one of the widely supported base models: NousResearch/Llama-3.2-7B-Instruct or mistralai/Mistral-7B-Instruct-v0.3 are solid choices with strong Hugging Face ecosystem support. Use the transformers, peft, and trl libraries from Hugging Face, all installable via pip. The core script loads the model in 4-bit quantisation using BitsAndBytes, which brings memory consumption down to approximately 6–8 GB for a 7B model, leaving room for activations and optimiser states.
Below is the essential training configuration for a single RTX 4090 (24 GB VRAM):
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype="bfloat16"
)
model = AutoModelForCausalLM.from_pretrained(
"NousResearch/Llama-3.2-7B-Instruct",
quantization_config=bnb_config,
device_map="auto"
)
lora_config = LoraConfig(
r=16,
lora_alpha=32,
target_modules=["q_proj", "v_proj", "k_proj", "o_proj"],
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM"
)For a 7B model, a rank of 16 with alpha at 32 is a reliable starting point. Target the query, key, value, and output projection modules for the best balance of adaptation speed and quality. Use trl.SFTTrainer with a batch size of 4 and gradient accumulation steps of 4 to simulate a batch size of 16. This setup will complete one epoch on 2,000 examples in roughly 30 to 45 minutes on a 4090. Monitor VRAM usage with nvidia-smi and expect about 18–20 GB during training—well within the safety margin for a 24 GB card.
Configuring Hyperparameters for Stable and Efficient Training
Fine-tuning is notoriously sensitive to learning rate and batch size. For LoRA with 4-bit base models, a learning rate of 2e-4 with a cosine schedule and 10 % warm-up steps is a proven conservative starting point. If your loss curve oscillates or spikes above 2.0 in the first 100 steps, reduce the learning rate to 1e-4. If the loss decreases monotonically and you see no improvement on the validation set after 500 steps, increase it to 3e-4. Use the AdamW optimiser with a weight decay of 0.01 to prevent overfitting, especially when your dataset is under 1,000 examples.
Set your maximum sequence length to 2048 tokens—this covers most business documents without wasting VRAM on padding. If your examples consistently exceed this, consider chunking or summarising the input before training. Train for 3 epochs is typically enough for instruction tuning; training beyond 5 epochs often leads to catastrophic forgetting of the base model's general knowledge. Log your loss to Weights & Biases or a local CSV and watch for divergence between train and validation loss as a signal to stop early. A smooth, decreasing validation loss that stabilizes after epoch 2 or 3 indicates your model has absorbed the task-specific patterns without overfitting.
Evaluating Your Fine-Tuned Model Before Deployment
Automated metrics like perplexity or BLEU score give a quick sanity check but cannot replace task-specific evaluation. Build a small golden test set of 50–100 examples—separate from your training and validation splits—with ground-truth outputs you consider perfect. Run your fine-tuned model against this set and measure exact-match accuracy, answer completeness, and adherence to your formatting rules. For example, a model trained to write SQL queries should produce syntactically valid code; a support agent should never output markdown in a live chat channel.
Additionally, perform a qualitative review with three to five stakeholders from your business team. Provide them with ten model outputs from your test set and a simple rubric: accuracy, tone, completeness. Your engineers might be thrilled by a 0.02 drop in perplexity, but if the business team finds the tone robotic or the answers overly verbose, you need to adjust your dataset or training configuration. Log feedback as structured annotations—good, acceptable, reject—and use this to prioritise additional data collection or a second training run with revised example formatting. Only when your golden test set shows at least 85 % good or acceptable ratings should you consider moving to deployment.
Deploying Your Custom Model with Ollama
Ollama has become the de facto local inference server because of its simplicity and broad model support. After fine-tuning, you need to convert your Hugging Face checkpoint into the GGUF format that Ollama requires. Use the llama.cpp conversion script from the Ollama ecosystem: run python convert.py your-checkpoint-folder --outfile model.gguf --ctx 4096. If your checkpoint includes LoRA adapters, merge them first using the peft library's merge_and_unload() method; Ollama does not load adapters separately. The conversion takes about five minutes for a 7B model on a modern CPU.
Create an Ollama Modelfile in the same directory as your GGUF file:
FROM ./model.gguf
TEMPLATE """{{ .Prompt }}"""
SYSTEM "You are a contract analyst at a law firm. Answer based on the provided document clause and keep responses under three sentences."
PARAMETER temperature 0.3
PARAMETER top_p 0.9
PARAMETER stop "</s>"Run ollama create your-model-name -f Modelfile and then ollama run your-model-name. You now have a fully local inference endpoint that supports OpenAI-compatible API calls via http://localhost:11434/v1. The model will load in approximately 8–10 seconds on a 24 GB GPU and respond with latencies under 200 ms per token. Your entire fine-tuned system—from training to Ollama API—runs entirely on your own hardware, with no monthly API bills or data leaving your network.
Maintaining and Iterating on Your Fine-Tuned Model
Deployment is not the end of the pipeline—it is the beginning of a feedback loop. Log every prompt and response from your Ollama instance using a simple middleware script that writes to a JSONL file. Label these interactions with a thumbs-up or thumbs-down from your users. When you accumulate 200 to 500 new positive examples (understood as responses that received a thumbs-up), merge them back into your training dataset and schedule a retraining run. This incremental approach keeps your model aligned with evolving business language, new products, or shifting customer expectations.
Version your models using Ollama's tagging system: ollama tag your-model-name v1.2 before each retraining. Keep at least the last two major versions in your local registry so you can roll back if a new iteration underperforms. Track evaluation metrics from your golden test set across versions—if a retraining run drops your acceptable-response rate below 80 %, revert and investigate your new data sources for inconsistencies or label errors. A disciplined iteration cadence (every two to four weeks for moderate-volume use cases) ensures your model grows smarter without becoming brittle.
Your business runs on specialised knowledge; your LLM should too. Start small, measure diligently, and let real-world feedback guide your training decisions.
What is the minimum GPU required for fine-tuning a 7B model with LoRA?
A consumer GPU with at least 12 GB of VRAM can handle a 7B parameter model using 4-bit quantisation and a batch size of 2. For a smoother experience with larger batch sizes and gradient accumulation, an RTX 3090 or 4090 with 24 GB is recommended. If you are using an 8B or 9B model, opt for a 24 GB card to maintain headroom for the optimiser state and activations.
How much training data do I realistically need for a useful fine-tune?
For a single task (e.g., email classification or summarisation), 500 to 1,000 high-quality, manually reviewed examples are sufficient to see a clear improvement over the base model. For multi-task instruction tuning, aim for 2,000 to 10,000 examples. Data quality is more important than quantity—fifty perfectly
