Train Your Own SDXL LoRA on a Single GPU with kohya-ss

What you'll build

You'll train a ~45 MB SDXL LoRA from 15–30 photos of a product using kohya-ss/sd-scripts (the trainer behind the kohya_ss GUI), then load it into a diffusers pipeline and generate on-brand images with a trigger word.

Prerequisites

• Linux (Ubuntu 22.04/24.04) or WSL2, NVIDIA GPU with 12 GB+ VRAM. The docs say 8 GB works with --network_dim 4–8; this guide assumes 12 GB.
• Python 3.10.x and git. sd-scripts lists 3.11/3.12 as "will work but not tested."
• Verified against: sd-scripts main (v0.11.1, July 2026), PyTorch 2.6.0+cu124 (RTX 50-series needs 2.8.0+cu128), diffusers 0.32.1 and accelerate 1.6.0 (both pinned by sd-scripts' requirements.txt), huggingface_hub 0.34.3.
• About 15 GB free disk: 7 GB for the SDXL base checkpoint plus latent caches.
• 15–30 JPG/PNG photos of one subject, ideally 1024 px or larger on the short side, with varied angles, backgrounds and lighting.

1. Install sd-scripts

git clone https://github.com/kohya-ss/sd-scripts.git
cd sd-scripts
python3.10 -m venv venv
source venv/bin/activate
pip install torch==2.6.0 torchvision==0.21.0 --index-url https://download.pytorch.org/whl/cu124
pip install --upgrade -r requirements.txt
accelerate config default --mixed_precision bf16

On an RTX 50-series card replace the torch line with pip install torch==2.8.0 torchvision==0.23.0 --index-url https://download.pytorch.org/whl/cu128. accelerate config default writes a single-GPU config without the interactive questionnaire.

Download the single-file SDXL base checkpoint (6.94 GB) rather than the diffusers repo, which pulls fp32 weights twice that size:

mkdir -p ~/lora/models ~/lora/data/product ~/lora/output
hf download stabilityai/stable-diffusion-xl-base-1.0 sd_xl_base_1.0.safetensors --local-dir ~/lora/models

2. Prepare the dataset

Copy your photos into ~/lora/data/product, then create one caption .txt per image with the same basename. Start every caption with a rare trigger token (here ohwx) followed by the class, then describe what varies between shots — background, angle, lighting — so the LoRA learns the product and not the kitchen counter:

cd ~/lora/data/product
for f in *.jpg; do echo "ohwx bottle, product photo" > "${f%.jpg}.txt"; done

Now edit each file, e.g. IMG_0412.txt:

ohwx bottle, product photo, on a wooden desk, window light from the left, slight top-down angle

Images without a caption file fall back to class_tokens from the config below, so a missing file won't crash training — it just trains on a weaker caption.

3. Write the dataset config

Save as ~/lora/dataset.toml (replace /home/you; TOML doesn't expand ~):

[general]
caption_extension = ".txt"
keep_tokens = 1

[[datasets]]
resolution = 1024
batch_size = 1
enable_bucket = true
min_bucket_reso = 640
max_bucket_reso = 1536
bucket_reso_steps = 64

  [[datasets.subsets]]
  image_dir = "/home/you/lora/data/product"
  class_tokens = "ohwx bottle"
  num_repeats = 10

enable_bucket groups images by aspect ratio so nothing gets center-cropped; the min/max values must be divisible by bucket_reso_steps. With 20 images × 10 repeats you get 200 steps per epoch.

Add a sample prompt file so you can watch the LoRA converge. Save as ~/lora/sample_prompts.txt:

ohwx bottle on a marble kitchen counter, soft morning light --n blurry, lowres, watermark --w 1024 --h 1024 --d 1 --s 28 --l 7

--n is the negative prompt, --d the seed, --s steps, --l CFG scale.

4. Train

cd ~/sd-scripts && source venv/bin/activate
accelerate launch --num_cpu_threads_per_process 1 sdxl_train_network.py \
  --pretrained_model_name_or_path ~/lora/models/sd_xl_base_1.0.safetensors \
  --dataset_config ~/lora/dataset.toml \
  --output_dir ~/lora/output --output_name product-lora \
  --save_model_as safetensors --save_precision bf16 \
  --network_module networks.lora --network_dim 16 --network_alpha 8 \
  --network_train_unet_only \
  --learning_rate 1e-4 --optimizer_type AdamW8bit \
  --lr_scheduler cosine --lr_warmup_steps 80 \
  --max_train_epochs 8 --save_every_n_epochs 2 \
  --mixed_precision bf16 --gradient_checkpointing --sdpa \
  --cache_latents --cache_latents_to_disk \
  --cache_text_encoder_outputs --cache_text_encoder_outputs_to_disk \
  --no_half_vae --noise_offset 0.0357 --min_snr_gamma 5 \
  --sample_prompts ~/lora/sample_prompts.txt --sample_every_n_epochs 2 --sample_sampler euler_a \
  --seed 42 --logging_dir ~/lora/logs

Why these flags:

• --network_train_unet_only is mandatory once you cache text-encoder outputs (the script asserts it), and the SDXL docs recommend it anyway because training both text encoders gives unpredictable results.
• --cache_latents + --cache_text_encoder_outputs skip the VAE and CLIP passes every step; combined with --gradient_checkpointing, --sdpa and AdamW8bit this is what fits 1024 px training in 12 GB.
• --no_half_vae keeps the VAE in fp32 during latent caching; it's insurance against the SDXL fp16 VAE producing NaNs.
• --noise_offset 0.0357 matches what SDXL base was trained with; --min_snr_gamma 5 stabilizes early loss.
• 1600 total steps at 1e-4 with cosine decay is a reasonable starting point for a single object; a 4090 finishes in roughly 20–25 minutes, a 3060 in about an hour.

The first run encodes latents and text embeddings to disk (.npz files next to your images) before the step counter starts.

5. Load the LoRA in diffusers

diffusers needs PEFT to load LoRA weights, which sd-scripts doesn't install:

pip install peft

Save as ~/lora/generate.py:

import os
import torch
from diffusers import StableDiffusionXLPipeline

home = os.path.expanduser("~/lora")

pipe = StableDiffusionXLPipeline.from_single_file(
    f"{home}/models/sd_xl_base_1.0.safetensors", torch_dtype=torch.float16
).to("cuda")

pipe.load_lora_weights(f"{home}/output", weight_name="product-lora.safetensors")

image = pipe(
    "ohwx bottle on a matte black coffee table, studio lighting, editorial product shot",
    negative_prompt="blurry, lowres, watermark, text",
    num_inference_steps=30,
    guidance_scale=7.0,
    cross_attention_kwargs={"scale": 0.8},
    generator=torch.Generator("cuda").manual_seed(1),
).images[0]

image.save(f"{home}/product-table.png")
print("saved", f"{home}/product-table.png")

from_single_file reuses the checkpoint you already downloaded. cross_attention_kwargs={"scale": 0.8} dials LoRA strength between 0 (base model) and 1 (full LoRA); 0.7–0.9 usually keeps the subject faithful without cooking the composition.

Verify it works

Training should end with the step bar at 100% and two log lines:

steps: 100%|██████████| 1600/1600 [23:41<00:00,  1.13it/s, avr_loss=0.0862]

saving checkpoint: /home/you/lora/output/product-lora.safetensors
model saved.

avr_loss for SDXL LoRA typically settles somewhere around 0.08–0.12; a value that keeps climbing or hits nan means the learning rate is too high.

Check the outputs:

ls -la ~/lora/output ~/lora/output/sample

You should see product-lora.safetensors (about 45 MB for dim 16, U-Net only), epoch checkpoints named product-lora-000002.safetensors, -000004, -000006, and PNGs in sample/ for epochs 2, 4, 6 and 8. Flip through the samples: by epoch 4 the bottle's shape and label should be recognizable, and by epoch 8 it should be consistent across seeds. If epoch 6 looks better than 8 (over-baked, blown-out contrast), use that checkpoint instead.

Then generate:

python ~/lora/generate.py

saved /home/you/lora/product-table.png

Run it again with cross_attention_kwargs={"scale": 0.0} — the bottle should disappear into a generic one. That difference is your LoRA working.

Troubleshooting

AssertionError: network for Text Encoder cannot be trained with caching Text Encoder outputs — you passed --cache_text_encoder_outputs without --network_train_unet_only. Add the flag, or drop the caching flags and add --text_encoder_lr1 1e-5 --text_encoder_lr2 1e-5 if you actually want to train the text encoders.

No data found. Please verify arguments (train_data_dir must be the parent of folders with images) then the script exits — image_dir in dataset.toml is wrong or contains no images. For --dataset_config the path must point directly at the folder holding the images (no 10_bottle subfolder convention), and it must be absolute.

RuntimeError: NaN detected in latents: /home/you/lora/data/product/IMG_0412.jpg — the VAE overflowed in half precision while caching. Make sure --no_half_vae is on the command line; if you're on a GPU without bf16 (GTX 16xx/RTX 20xx) and used --mixed_precision fp16, that flag is the fix. If a single file is still named, the image itself is corrupt — re-export it.

torch.OutOfMemoryError: CUDA out of memory. Tried to allocate ... — first drop --sample_prompts (sampling loads a full inference pipeline mid-training), then set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True in front of the launch command to reduce fragmentation, then lower --network_dim to 8 and resolution to 768 in the TOML.

ValueError: PEFT backend is required for this method. from generate.py — peft isn't installed in the venv you're running from. pip install peft and rerun.

Next steps

• Train the text encoders too for a stronger trigger word: remove both --cache_text_encoder_outputs* flags and --network_train_unet_only, add --text_encoder_lr1 1e-5 --text_encoder_lr2 1e-5, and expect ~2 GB more VRAM.
• Auto-caption larger sets with finetune/tag_images_by_wd14_tagger.py --onnx --repo_id SmilingWolf/wd-eva02-large-tagger-v3 --remove_underscore (needs pip install onnx onnxruntime-gpu).
• Try --optimizer_type Prodigy --learning_rate 1.0 to skip learning-rate tuning, or LoHa/LoKr via docs/loha_lokr.md for style LoRAs.
• Prefer clicking? git clone --recursive https://github.com/bmaltais/kohya_ss && ./setup.sh && ./gui.sh gives you the same trainer behind a Gradio UI, and it prints the equivalent CLI command it runs.
• The .safetensors file drops straight into ComfyUI's models/loras or A1111's models/Lora with <lora:product-lora:0.8> in the prompt.