Skip to content

L.1 · TuneLab

Did fine-tuning actually generalize?

Every LoRA run is scored against the frozen base model on the same prompts. The gap between the test split and the held-out phrasings shows how much was memorized versus learned.

Data provenance. Synthetic ticket-triage task, n = 120 per split, one seed per configuration. Base model: Qwen2.5-0.5B-Instruct (4-bit). Source: tunelab-llm-finetuning · README.md, docs/RESULTS.md ↗

View data table
Did fine-tuning actually generalize?
RunTest exact matchHeld-out exact matchHeld-out priorityTrain minutesTrainable params (M)
Base model8.3%5.0%46.7%--
LoRA r8 · 300 steps100.0%63.3%69.2%9.72.93
LoRA r8 · 100 steps100.0%50.0%50.0%4.42.93
LoRA r16 · dropout 0.199.2%40.0%40.0%10.15.87

L.2 · AEGIS AI

Catching fraud without flooding analysts

A good fraud model alerts on almost every attack pattern while staying quiet on legitimate edge cases like travel or new devices. Compare alert rates by scenario, the two candidate models and the final confusion matrix.

Data provenance. Synthetic transactions. Test split: 25,877 transactions, 564 fraudulent (2.18%). Threshold 0.432. Source: aegis-ai-real-time-fraud-platform · README.md, docs/ML_MODEL.md ↗

View data table
Catching fraud without flooding analysts
ScenarioTypeAlert rate
Impossible travelFraud100%
Card testingFraud98.7%
Account takeoverFraud83.6%
Collusive merchantFraud81.8%
High-value anomalyFraud60%
Legit new deviceLegitimate8.8%
Legit large purchaseLegitimate0.95%
Legit travelLegitimate0.57%
NormalLegitimate0.3%
Logistic regressionModelP 0.564 · R 0.601 · F1 0.582 · PR-AUC 0.614 · ROC-AUC 0.976
Hist. gradient boostingModelP 0.842 · R 0.885 · F1 0.863 · PR-AUC 0.926 · ROC-AUC 0.997
Confusion matrixHGB @ 0.432TN 25219 · FP 94 · FN 65 · TP 499

L.3 · WorkMesh

Where does permission-aware latency go?

Every WorkMesh read applies authorization in SQL under row-level security. This benchmark shows the spread from median to tail latency per endpoint. The dashboards that aggregate across teams have the longest tails.

Data provenance. 300 requests per endpoint at concurrency 10 on seeded data (51 people, 12 projects, 98 work items). Local machine, zero errors. Source: workmesh-company-visibility-platform · README.md ↗

View data table
Where does permission-aware latency go?
Endpointp50 (ms)p95 (ms)p99 (ms)req/s
Health15.859.5108.9478
Employee dashboard80.2101.5174.2118
Team Pulse68.9121.7138.3129
Leadership dashboard83.3164.2252.3105
Project list32.73840.5298
People graph5463.768.5182
Work graph39.853.757.6242
Update feed3137.539.4314
Search54.962.688.2177

L.4 · SourceAtlas

Does verified RAG hold up with a real model?

SourceAtlas is evaluated twice: with deterministic fixtures and with a live local model. Live answer correctness climbed across three evaluation runs as the pipeline was fixed.

Data provenance. 16-question synthetic evaluation set. “Live” uses qwen3:4b + nomic-embed-text; “Fixture” uses deterministic stand-ins. Source: sourceatlas-agentic-rag · README.md, eval/results/ ↗

View data table
Does verified RAG hold up with a real model?
MetricLiveFixture
Answer correct92.3%84.6%
Citation correct95.8%69.2%
Abstention87.5%93.8%
Conflict detection100.0%100.0%
Injection handling100.0%100.0%
Live run 1: answer correct53.8%-
Live run 2: answer correct69.2%-
Live run 3: answer correct92.3%-