AI/ML · Full Stack
TuneLab
LoRA fine-tuning with honest evaluation
- Status
- Complete
- Year
- 2026
- Live application
- Runs locally on Apple Silicon. Synthetic task, single seed per configuration; not publicly deployed.

- all-fields exact match, test split
- 8.3% → 100%
- all-fields exact match, test split
- on held-out phrasings
- 5.0% → 63.3%
- on held-out phrasings
- of parameters trained (2.93M of 494M)
- 0.59%
- of parameters trained (2.93M of 494M)
01 Problem
Fine-tuning demos usually show a loss curve and a cherry-picked example. TuneLab trains LoRA adapters with MLX-LM and evaluates each one against the frozen base model on the same data, prompt and decoding, including a held-out split of phrasings the model never saw during training.
02 Product overview
A FastAPI backend dispatches training and evaluation jobs to an isolated worker subprocess, streams live loss curves over SSE and records everything in SQLite. Each run verifies that weights actually changed (lora_b norm from 0 to above 0) and stores the adapter's SHA-256. A Next.js UI compares runs, shows base-vs-LoRA outputs side by side and offers a playground.
Implemented
- Deterministic synthetic ticket-triage dataset, plus chat JSONL upload with validation and hash-based splits
- LoRA training in a worker subprocess with live SSE loss curves, cancel and crash recovery
- Weight-update verification and adapter SHA-256
- Automatic baseline and tuned evaluation: JSON validity, per-field accuracy, teacher-forced loss
- Run comparison, base-vs-LoRA playground and adapter download
Planned, not yet built
- More paraphrase diversity for urgency phrasing
- Multiple seeds per configuration
03 System architecture
04 Core technologies
Training
- MLX 0.32
- mlx-lm
- Qwen2.5-0.5B-Instruct (4-bit)
- LoRA
Backend
- FastAPI
- SQLAlchemy
- SQLite (WAL)
- SSE
Frontend
- Next.js 16
- React 19
- TanStack Query
- Tailwind 4
- Plotly
Quality
- pytest
- Vitest
- Playwright
- Docker
- GitHub Actions
05 Engineering challenges
Proving the adapter learned
Loss alone can mislead, so every run checks the LoRA weight norm moved from zero and evaluates on held-out phrasings the model never trained on.
Real training in CI
Instead of mocking training, Linux CI runs genuine LoRA steps on MLX's CPU backend with a tiny random model.
Fitting on a laptop
A frozen 4-bit base with LoRA (QLoRA-style) kept peak memory under 3 GB on a 16 GB M4.
06 Technical decisions
- 01MLX on Apple Silicon rather than cloud GPUs
- 02Pinned base-model revision for reproducibility
- 03Identical prompt and decoding for base and tuned models
- 04One worker subprocess per job
07 Verified test results
- pytest (incl. real LoRA training)
- 27
- Vitest
- 8
- Playwright (incl. a 10-step LoRA run)
- 7
Latest GitHub Actions run on main: passing. Counts as reported in the repository's README and testing docs.
Explore the results interactively in the Lab08 Screenshots
09 Links
No public deployment. Runs locally on Apple Silicon. Synthetic task, single seed per configuration; not publicly deployed.


