in this blog post “What is Supervised Fine-Tuning (SFT)” we will unpack what supervised fine-tuning is, when it’s the right tool, how it works under the hood, and how to run a robust SFT project end-to-end—from data to deployment.
What is Supervised Fine-Tuning (SFT)?
Supervised fine-tuning adapts a pretrained language model to perform better on a target behavior by training on paired inputs and outputs. The model learns via next-token prediction with teacher forcing, typically optimizing cross-entropy loss on the target response tokens. In practice, SFT is used to align models with desired formats (e.g., helpful answers, safe completions, tool-use schemas) and domains (e.g., support, legal, medical, coding) without retraining from scratch.
Key characteristics:
- Data: input-output pairs (e.g., instruction → answer). Often called instruction tuning or task-specific SFT.
- Loss: next-token cross-entropy; commonly mask loss on prompt tokens and compute loss only on the response.
- Goal: improve adherence to instructions, factuality in a domain, stylistic consistency, and output structure.
- Scope: from small task adapters (parameter-efficient finetuning) to full-model updates.
When (and When Not) to Use SFT
Good use cases
- Consistent formatting: APIs requiring JSON, function-call arguments, or specific templates.
- Domain adaptation: customer support, documentation, financial or legal drafting, coding conventions.
- Instruction following: more reliable step-by-step answers vs. a base model.
- Latent knowledge activation: making better use of pretrained knowledge with domain exemplars.
Consider other approaches if
- You need preference optimization across multiple acceptable outputs: consider RLHF/DPO after SFT.
- You need tool integration without training: try prompting or structured output constraints first.
- You only need light behavior change: try prompt engineering or system prompts before SFT.
- Your data is scarce or noisy: risk of overfitting or regressions; invest in data quality first.
How SFT Works Under the Hood
The model is fed a concatenation of the prompt and the target response. During training, the labels are the next tokens of the sequence. To prevent the model from “learning” to reproduce the prompt, a loss mask is applied so only response tokens contribute to the loss. This preserves instruction-following while reinforcing the desired answers, style, and structure.
Modern chat models also use conversation templates (system, user, assistant roles). It’s critical to format SFT data with the exact chat template expected by the tokenizer and model, including special tokens. Misalignment here often causes degraded performance.
The Most Important Variable
Data types
- Human-authored instruction–response pairs: highest quality, costly to scale.
- Human-edited synthetic data: model-generated drafts reviewed/edited by experts; good cost-quality balance.
- Pure synthetic data: useful for coverage; requires heavy filtering and held-out evaluation to avoid bias.
- Logs and transcripts: mine real-world prompts and outcomes, with careful anonymization and curation.
Quality checklist (data)
- Coverage: reflect the distribution of prompts you expect in production.
- Diversity: vary phrasing, difficulty, length, and edge cases.
- Correctness: verify factuality and adherence to policies.
- Consistency: stable formats, with explicit acceptance criteria.
- Safety: remove harmful content or annotate with policy-compliant alternatives.
- Deduplication: avoid near-duplicate prompts/answers; reduces overfitting and memorization.
- Licensing and privacy: ensure rights to use; redact sensitive data.
Common formats
Many teams use JSONL with fields like instruction, input, and output, or chat-style role messages.
{"instruction": "Summarize the text.", "input": "<article>...</article>", "output": "<summary>...</summary>"}
{"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Explain transformers in 3 bullets."},
{"role": "assistant", "content": "- ...\n- ...\n- ..."}
]}
Model Choice and Parameter-Efficient Fine-Tuning
Pick a base model that already performs reasonably on your domain and supports your context length and deployment constraints. If latency or memory is tight, smaller models or quantization-aware methods help.
Parameter-efficient fine-tuning (PEFT) like LoRA/QLoRA updates a small number of adapter parameters while freezing the base model. Benefits: lower memory, faster training, easier rollback, and composable adapters for multiple behaviors. Full fine-tuning may yield slightly higher ceilings but at higher cost and risk.
Training Recipe (Illustrative)
The following example sketches a typical SFT setup using common open-source tooling. Adjust to your stack as needed.
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, DataCollatorForCompletionOnlyLM
model_name = "your-base-model"
dataset = load_dataset("json", data_files="data.jsonl")
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
# If chat model, ensure correct chat template
# tokenizer.apply_chat_template(...) during preprocessing
# Only compute loss on assistant responses
response_template = "\nassistant:" # match your prompt template exactly
collator = DataCollatorForCompletionOnlyLM(
response_template=tokenizer.encode(response_template, add_special_tokens=False),
tokenizer=tokenizer,
)
lora_config = LoraConfig(
r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM"
)
model = get_peft_model(model, lora_config)
trainer = SFTTrainer(
model=model,
tokenizer=tokenizer,
train_dataset=dataset["train"],
eval_dataset=dataset.get("validation"),
max_seq_length=2048,
packing=True, # pack multiple examples per sequence to improve throughput
data_collator=collator,
args=dict(
per_device_train_batch_size=2,
gradient_accumulation_steps=8,
learning_rate=2e-4, # LoRA often uses higher LR than full ft
lr_scheduler_type="cosine",
warmup_ratio=0.05,
num_train_epochs=3,
logging_steps=50,
save_steps=1000,
bf16=True,
gradient_checkpointing=True,
),
)
trainer.train()
model.save_pretrained("./sft-model")
Notes:
- Match your prompt/response templates exactly when constructing inputs and masks.
- Turn on gradient checkpointing to fit larger context windows.
- Use bfloat16 if supported; it tends to be stable and fast on modern GPUs.
- Pack shorter examples to reduce padding waste.
Hyperparameter Hints
- Sequence length: set to your operational context size; train near the max you plan to serve.
- Batch size: increase effective batch size with gradient accumulation when VRAM-limited.
- Learning rate: 1e-5 to 5e-5 for full fine-tuning; 1e-4 to 3e-4 for LoRA are common starting points.
- Warmup: 3–10% of steps; cosine or linear schedulers both work.
- Early stopping: monitor validation loss and task metrics; avoid overfitting to stylistic quirks.
- Regularization: mix in general instruction data (e.g., 10–30%) to preserve breadth.
Evaluation: Know What “Good” Means
Automatic metrics
- Exact match and F1 for QA with canonical answers.
- BLEU/ROUGE for summarization, but beware they can miss factuality.
- Multiple-choice accuracy (MC1/MC2) for knowledge checks.
- Pass@k or unit-test pass rate for code generation.
- Schema adherence: JSON parse rate, field presence, JSON schema validation.
Preference and human evaluation
- Pairwise win-rate vs. baseline on representative prompts.
- Rubric-based scoring: helpfulness, harmlessness, faithfulness, formatting.
- Red-teaming: prompt families targeting safety and robustness.
Test design
- Holdout split: no leakage from train to eval. Deduplicate at the prompt and n-gram levels.
- Stratification: include lengths, difficulty, and edge cases proportional to production.
- Statistical confidence: use multiple seeds; report confidence intervals where feasible.
Safety, Policy, and Compliance
- Policy conditioning: include system messages describing rules; reinforce with examples.
- Refusals and deflections: include exemplars of safe alternatives for disallowed content.
- PII handling: redact training data; test for unintended memorization with targeted prompts.
- Licensing: confirm rights to use data; track provenance and opt-out lists.
- Guardrails: combine SFT with runtime filters or classifiers for high-risk domains.
From Lab to Production
Packaging and deployment
- Adapters: with PEFT, ship only adapter weights; keep base model immutable for reuse.
- Quantization: 4/8-bit inference for cost; validate accuracy and latency impacts.
- Prompt contracts: version your system prompts and templates alongside the model.
Monitoring
- Quality KPIs: win-rate vs. baseline, schema-parse rate, task success, latency, cost.
- Safety KPIs: flagged content rate, false-positive/negative rates in moderation.
- Drift: track prompt distribution changes and performance by segment over time.
- Feedback loops: collect user ratings and flagged cases to fuel continuous improvement.
Iteration
- Data engine: prioritize new training examples from observed failure modes.
- Curriculum: stage training with general instructions, then domain, then format-heavy data.
- Preference optimization: consider DPO or RLHF after SFT for finer control of trade-offs.
Common Pitfalls and How to Avoid Them
- Mismatched templates: ensure the exact same chat/prompt template is used at train and inference.
- Loss on prompts: mask non-response tokens; otherwise the model learns to echo inputs.
- Over-narrow data: mixing a small % of broad instructions preserves general capabilities.
- Data leakage: deduplicate across train/dev/test; watch for copy-paste contamination.
- Unstable training: too high LR, too long sequences without checkpointing, or no warmup.
- Hallucinations from synthetic data: add human verification for high-stakes tasks.
- Evaluation mismatch: do not rely on a single metric; triangulate with human eval.
A Minimal End-to-End Checklist
- Define target behaviors and acceptance criteria.
- Assemble and clean instruction–response data; deduplicate and redact.
- Choose base model and context window; decide PEFT vs. full FT.
- Implement exact prompt/response templates and loss masking.
- Train with conservative hyperparameters; log and checkpoint frequently.
- Evaluate on varied, held-out tests; include human preference checks.
- Harden safety with policy examples and runtime guardrails.
- Package adapters, version prompts, and deploy with monitoring.
- Collect feedback; iterate data and, if needed, add preference optimization.
FAQ
How much data do I need?
It depends on the gap between the base model and your target. Hundreds to a few thousand high-quality examples can materially improve formatting and instruction-following. Domain depth and style often benefit from tens of thousands of curated pairs. Quality beats quantity.
What hardware is required?
For PEFT on 7B–13B models with 2k–4k context, a single modern GPU (e.g., 24–80 GB VRAM) can suffice with gradient accumulation and checkpointing. Larger models or longer contexts require multi-GPU setups. Validate throughput and memory early with a small sample.
Will SFT reduce general capabilities?
It can, if training is narrow. Mix in general instruction data and monitor broad benchmarks to mitigate regressions.
How does SFT differ from RLHF/DPO?
SFT learns from labeled targets; RLHF/DPO learn from preferences between outputs. Many production systems use SFT first for instruction adherence and format, then apply preference optimization to fine-tune trade-offs like verbosity and tone.
Takeaways
- SFT is the most accessible, high-leverage method to steer LLMs toward your domain and formats.
- Data quality and correct templating matter more than clever hyperparameters.
- Evaluate with both automatic metrics and human judgment; monitor in production and iterate.
With disciplined data curation, careful training, and rigorous evaluation, supervised fine-tuning can turn a capable base model into a reliable system tailored to your organization’s needs.
Discover more from CPI Consulting
Subscribe to get the latest posts sent to your email.