Skip to content
← All projects

AI/ML · Full Stack

TuneLab

LoRA fine-tuning with honest evaluation

Status
Complete
Year
2026
Live application
Runs locally on Apple Silicon. Synthetic task, single seed per configuration; not publicly deployed.
Training run with live loss curve and weight verification
all-fields exact match, test split
8.3% → 100%
all-fields exact match, test split
on held-out phrasings
5.0% → 63.3%
on held-out phrasings
of parameters trained (2.93M of 494M)
0.59%
of parameters trained (2.93M of 494M)

01 Problem

Fine-tuning demos usually show a loss curve and a cherry-picked example. TuneLab trains LoRA adapters with MLX-LM and evaluates each one against the frozen base model on the same data, prompt and decoding, including a held-out split of phrasings the model never saw during training.

02 Product overview

A FastAPI backend dispatches training and evaluation jobs to an isolated worker subprocess, streams live loss curves over SSE and records everything in SQLite. Each run verifies that weights actually changed (lora_b norm from 0 to above 0) and stores the adapter's SHA-256. A Next.js UI compares runs, shows base-vs-LoRA outputs side by side and offers a playground.

Implemented

  • Deterministic synthetic ticket-triage dataset, plus chat JSONL upload with validation and hash-based splits
  • LoRA training in a worker subprocess with live SSE loss curves, cancel and crash recovery
  • Weight-update verification and adapter SHA-256
  • Automatic baseline and tuned evaluation: JSON validity, per-field accuracy, teacher-forced loss
  • Run comparison, base-vs-LoRA playground and adapter download

Planned, not yet built

  • More paraphrase diversity for urgency phrasing
  • Multiple seeds per configuration

03 System architecture

/apieventsNext.jsruns · compare · playgroundFastAPIjobs · SSESQLitedatasets · jobs · eventsDispatcherone job at a timeWorker processtrain · evaluateAdapterssafetensors · SHA-256
Each job runs in its own subprocess for memory isolation and safe cancellation; SSE replays stored job events.clientservicestoremodelqueue

04 Core technologies

Training

  • MLX 0.32
  • mlx-lm
  • Qwen2.5-0.5B-Instruct (4-bit)
  • LoRA

Backend

  • FastAPI
  • SQLAlchemy
  • SQLite (WAL)
  • SSE

Frontend

  • Next.js 16
  • React 19
  • TanStack Query
  • Tailwind 4
  • Plotly

Quality

  • pytest
  • Vitest
  • Playwright
  • Docker
  • GitHub Actions

05 Engineering challenges

Proving the adapter learned

Loss alone can mislead, so every run checks the LoRA weight norm moved from zero and evaluates on held-out phrasings the model never trained on.

Real training in CI

Instead of mocking training, Linux CI runs genuine LoRA steps on MLX's CPU backend with a tiny random model.

Fitting on a laptop

A frozen 4-bit base with LoRA (QLoRA-style) kept peak memory under 3 GB on a 16 GB M4.

06 Technical decisions

  1. 01MLX on Apple Silicon rather than cloud GPUs
  2. 02Pinned base-model revision for reproducibility
  3. 03Identical prompt and decoding for base and tuned models
  4. 04One worker subprocess per job

07 Verified test results

pytest (incl. real LoRA training)
27
Vitest
8
Playwright (incl. a 10-step LoRA run)
7

Latest GitHub Actions run on main: passing. Counts as reported in the repository's README and testing docs.

Explore the results interactively in the Lab

08 Screenshots

Training run with live loss curve and weight verification
Live loss curve with weight-update verification
Base model versus LoRA adapter comparison
Base vs. LoRA on identical prompts
Comparison of multiple fine-tuning runs
Run comparison
Playground for trying the base and tuned model
Playground

Screenshots captured from the running application and published in the repository's docs.

09 Links

No public deployment. Runs locally on Apple Silicon. Synthetic task, single seed per configuration; not publicly deployed.