Fine-Tune Qwen2.5-7B with QLoRA on Your Own Data

What you'll build

Take a JSONL dataset of instruction/response pairs, fine-tune Qwen2.5-7B-Instruct with QLoRA (4-bit base, LoRA adapters), confirm the trainer masks loss on the right tokens, and end up with a merged model you can run locally or serve. Includes real numbers on time and cost, and a clear answer to "should I even do this."

Prerequisites

• Linux, NVIDIA GPU with 24GB+ VRAM (RTX 4090, L4, A10G, A100). QLoRA training at the batch size/sequence length used below runs roughly 14-18GB; leave headroom either way.
• NVIDIA driver supporting CUDA 12.1+ (nvidia-smi should show CUDA Version: 12.1 or higher).
• Python 3.10 or 3.11 in a fresh virtualenv.
• A Hugging Face account (Qwen2.5-7B-Instruct is Apache-2.0, ungated, but you'll want an account for caching and rate limits).
• 500-5,000 labeled examples in your domain. Fewer than a few hundred and LoRA usually underperforms good prompting; you're wasting a GPU.

Pin your stack exactly. TRL's SFTConfig has moved fields around across releases, and version drift is the single biggest source of "it silently trains wrong" bug reports:

python -m venv .venv && source .venv/bin/activate
pip install torch==2.4.1 --index-url https://download.pytorch.org/whl/cu121
pip install transformers==4.46.3 trl==0.12.2 peft==0.13.2 \
 bitsandbytes==0.44.1 accelerate==1.0.1 datasets==3.1.0

Step 1: Decide if you actually need this

Fine-tuning fixes: consistent output format across thousands of calls, domain jargon the base model garbles, shorter prompts because instructions are baked into weights instead of repeated every call. It doesn't fix: knowledge that changes weekly (use RAG), reasoning the base model fundamentally lacks (more data won't add it), or a one-off task (few-shot prompting is cheaper and faster to iterate on).

Cost reality: QLoRA on a rented L4 or A10G runs roughly $0.50-1.20/hour on spot instances. A 2,000-example dataset for 3 epochs takes 1-3 hours depending on sequence length, so budget under $10 in compute. Data prep and eval cost more than the GPU time does. If you're reaching for full fine-tuning (all params, no LoRA) on anything above 7B, you're now talking multi-GPU and real money, don't default to it.

Step 2: Prepare the dataset

Use the chat message format TRL expects. One JSON object per line:

{"messages": [{"role": "system", "content": "You are a support agent for Acme Cloud."}, {"role": "user", "content": "My deploy is stuck at pending."}, {"role": "assistant", "content": "Check `kubectl describe pod` for the pending pod..."}]}

Split into train.jsonl and val.jsonl (90/10 is fine for a few thousand examples). Load and format:

from datasets import load_dataset
from transformers import AutoTokenizer

base_model = "Qwen/Qwen2.5-7B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(base_model)

raw = load_dataset("json", data_files={"train": "train.jsonl", "validation": "val.jsonl"})

def format_example(example):
 return {"text": tokenizer.apply_chat_template(example["messages"], tokenize=False)}

dataset = raw.map(format_example, remove_columns=raw["train"].column_names)

apply_chat_template wraps turns in Qwen's <|im_start|>role\n...<|im_end|> format, which you'll need for masking in Step 4.

Step 3: Load the base model in 4-bit

import torch
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
from peft import prepare_model_for_kbit_training, LoraConfig, get_peft_model

bnb_config = BitsAndBytesConfig(
 load_in_4bit=True,
 bnb_4bit_quant_type="nf4",
 bnb_4bit_compute_dtype=torch.bfloat16,
 bnb_4bit_use_double_quant=True,
)

model = AutoModelForCausalLM.from_pretrained(
 base_model,
 quantization_config=bnb_config,
 device_map="auto",
 torch_dtype=torch.bfloat16,
)
model.config.use_cache = False # incompatible with gradient checkpointing; flip back for inference

model = prepare_model_for_kbit_training(
 model,
 use_gradient_checkpointing=True,
 gradient_checkpointing_kwargs={"use_reentrant": False},
)

# Belt-and-suspenders: this is what lets gradients flow into the frozen,
# quantized embeddings under checkpointing. PEFT wires it automatically,
# but transformers/peft version drift has broken it silently before.
model.enable_input_require_grads()

lora_config = LoraConfig(
 r=16,
 lora_alpha=32,
 lora_dropout=0.05,
 bias="none",
 task_type="CAUSAL_LM",
 target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
)
model = get_peft_model(model, lora_config)

use_reentrant=False is the non-reentrant checkpointing path; current PyTorch and Hugging Face guidance both recommend it over the reentrant default, which throws "tensor does not require grad" errors under PEFT more often. Since prepare_model_for_kbit_training already turned checkpointing on, you'll set gradient_checkpointing=False in the trainer config below so it isn't fighting PEFT for the same mechanism.

Step 4: Configure the trainer

Version note: dataset_text_field, max_seq_length, and packing are SFTConfig fields as of TRL 0.12.2. Later TRL releases reorganized this config (fields renamed or moved). If you didn't pin trl==0.12.2, these may error out or get silently ignored.

from trl import SFTConfig, SFTTrainer, DataCollatorForCompletionOnlyLM

response_template = "<|im_start|>assistant\n"
collator = DataCollatorForCompletionOnlyLM(response_template=response_template, tokenizer=tokenizer)

sft_config = SFTConfig(
 output_dir="./qwen2.5-7b-support-lora",
 dataset_text_field="text", # TRL 0.12.x only
 max_seq_length=2048, # TRL 0.12.x only, reorganized in later versions
 packing=False,
 gradient_checkpointing=False, # already handled in Step 3
 per_device_train_batch_size=4,
 gradient_accumulation_steps=4,
 num_train_epochs=3,
 learning_rate=2e-4,
 lr_scheduler_type="cosine",
 warmup_ratio=0.03,
 logging_steps=10,
 eval_strategy="steps",
 eval_steps=50,
 save_strategy="steps",
 save_steps=50,
 save_total_limit=2,
 bf16=True,
 report_to="none",
)

trainer = SFTTrainer(
 model=model,
 args=sft_config,
 train_dataset=dataset["train"],
 eval_dataset=dataset["validation"],
 data_collator=collator,
 tokenizer=tokenizer,
)

This setup masks correctly for single-turn examples, one assistant turn per example, like Step 2. For multi-turn data, also pass instruction_template="<|im_start|>user\n" so the collator re-masks each later user turn. Without it, only tokens before the first assistant turn get masked, everything after (including later user turns) leaks into the loss.

One more subtlety: response_template matching works reliably here because <|im_start|> and <|im_end|> are real special tokens in Qwen's vocab, not regular BPE-merged text, so the template tokenizes identically whether it appears standalone or mid-sequence. That's not guaranteed for every chat format, which is exactly why Step 5 exists instead of trusting this by inspection.

Also: with transformers ≥4.46, passing tokenizer= to SFTTrainer throws a deprecation warning in favor of processing_class=. It still works on these pinned versions, ignore it.

Step 5: Verify the loss mask before you burn GPU hours

Don't trust masking by eyeballing the template string. Pull an actual batch through the trainer's own dataloader, since that's the exact path used during training:

batch = next(iter(trainer.get_train_dataloader()))
labels = batch["labels"][0]
input_ids = batch["input_ids"][0]

masked = (labels == -100).sum().item()
total = labels.numel()
print(f"masked tokens: {masked}/{total}")

response_ids = input_ids[labels != -100]
print(tokenizer.decode(response_ids))

The printed text should be only the assistant's response, no system prompt or user turn. If your data is multi-turn and you skipped the instruction_template fix, this is exactly where you'll catch it, a later user turn's text sitting inside the unmasked region.

Why this method and not a manual re-tokenize: hand-tokenizing a sample string with tokenizer(text, add_special_tokens=False) only matches training exactly for tokenizers that add no BOS token, which Qwen's happens to be. For a tokenizer that does prepend BOS (Llama, Mistral), a hand-tokenized check would be off by one token from what the trainer actually sees. Going through get_train_dataloader() is ground truth regardless of tokenizer quirks.

Step 6: Train and evaluate

trainer.train()
trainer.save_model("./qwen2.5-7b-support-lora/final")

Saving explicitly to a final directory means you're not hunting through checkpoint-50, checkpoint-100, etc. afterward to figure out which one to merge.

Watch train loss and eval loss together. Eval loss climbing while train loss keeps dropping past epoch 1-2 means overfitting, either cut epochs or add data. For a genuine read, don't trust loss alone: hold out 20-30 prompts, generate from both base and fine-tuned checkpoints, and read them side by side. Loss can look fine while the model still ignores your formatting requirements.

Step 7: Merge and use the model

You can't merge LoRA weights into a 4-bit base directly, you need the base in full precision:

from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(base_model, torch_dtype=torch.bfloat16, device_map="auto")
merged = PeftModel.from_pretrained(base, "./qwen2.5-7b-support-lora/final").merge_and_unload()
merged.save_pretrained("./qwen2.5-7b-support-merged")
tokenizer.save_pretrained("./qwen2.5-7b-support-merged")

If you don't have headroom for the full-precision load plus merge on the same GPU, set device_map="cpu" instead, slower but no VRAM ceiling. Serve the result however you'd serve any local model: vllm serve ./qwen2.5-7b-support-merged or plain transformers.generate.

Verify it works

• trainer.state.log_history should show eval loss trending down and stabilizing, not diverging.
• The masking check in Step 5 should print clean assistant-only text with no system/user leakage.
• Generate on 5-10 held-out prompts with the merged model and compare against base. You should see the fine-tuned model consistently matching your format/tone; if it looks identical to base, your learning rate or epoch count was too low, or your dataset is too small/noisy to move the model.

Troubleshooting

• CUDA out of memory: drop per_device_train_batch_size to 2, raise gradient_accumulation_steps to compensate, or reduce max_seq_length. QLoRA is memory-friendly, but 2048-token sequences at batch size 8 will still blow past 24GB.
• element 0 of tensors does not require grad: gradient checkpointing wasn't wired to the input embeddings, or you're on the reentrant path. Confirm model.enable_input_require_grads() ran after model prep, use_reentrant: False is set, and gradient_checkpointing=False in SFTConfig so the trainer isn't fighting PEFT over the same mechanism.
• SFTConfig throws unexpected keyword argument 'max_seq_length': you're on a newer TRL than 0.12.2. Either pin back to trl==0.12.2 or rewrite the config using that version's field names.
• Loss is nan after a few steps: usually the learning rate is too high for LoRA (try 1e-4), or you're on fp16 instead of bf16 on hardware that supports bf16, causing overflow.

Next steps

Once the adapter proves out, quantize the merged model (AWQ or GGUF) for cheaper serving, and build a real eval harness instead of eyeballing outputs, either lm-evaluation-harness for standard benchmarks or a small rubric-based judge model for your specific task. If you need the model to prefer certain responses over others rather than just imitate a fixed target, look at DPO on top of this SFT checkpoint using the same TRL install.