LucilleAI

Why LucilleAI exists

LucilleAI answers a simple question: can you train a real LLM from scratch — no PyTorch, no Python — on a laptop GPU with only 4 GB of VRAM?

The answer is the repository itself. It is a from-scratch deep-learning stack written in Rust that drives an NVIDIA GTX 1650 Ti (Turing, SM75) through its own CUDA bridge. Pre-training runs on a 14 GB Persian text corpus and supervised fine-tuning (SFT) on 55k instruction examples, and the whole pipeline — tokenizer, dataset reader, trainer, mixed precision, inference — lives in the codebase.

What makes it special

• No wrappers. There is no Python, no PyTorch and no CUDA framework dependency: cuBLAS is called directly through FFI and the compute graph is hand-rolled.
• Runs where “everyone” says it can’t. Training a transformer on 4 GB of VRAM required solving real memory problems: arena allocation, zero-copy dataset streaming, and careful activation budgeting.
• A complete language story. A Unigram tokenizer for Persian, sequence packing, and an SFT pipeline for instruction data are all implemented here — not imported.

Architecture (from the source)

The crate lucille_ai exposes several binaries, and its library is split into focused modules:

Area	Source modules	What lives there
Core tensors	core/tensor, core/shape, core/memory	4D tensor views, zero-copy memory, canonical shape4
Math backend	compute/simd, compute	AVX2 + FMA scalar kernels; SIMD dispatch
GPU stack	gpu/*	cuda_bridge (FFI), cublas, matmul, arena, pinned_memory, sampler, kernel sources
Model	ml/model, llm	Transformer blocks/stacks, attention, FFN, LayerNorm
Training	training/*	Trainer, AdamW, grad clip, GradScaler (AMP), readiness inspector
Tokenizer	tokenizer	Unigram + character tokenizers for Persian
Runtime	runtime/*	Device/execution-mode dispatch (cpu/cuda/fusion), AMP autocast, hardening & tracing
Inference & brain	inference, core/brain, core/tools	Decoding, an agent “brain” loop with tools (web_search, reader, …), policy/router/session
Assistants	code_assistant, workspace	Auto-fix, debugger orchestrator, suggestions, security checks
Metrics & eval	evaluation, metrics, observability	Eval suites, comparisons, reports
Export	export, pdf	Layout engine, PDF output, contract tests

Key binaries (src/bin/): train (pre-training), train_sft (instruction fine-tune), chat (agentic chat), eval, scientific_eval, tensor_bench, phase1_smoke_bench, workspace_server.

There is also a lucille-ui/ (Next.js) companion and a patent/ folder with finalized patent-filing documents (claims + description in Persian).

Real numbers measured on-device

• Pre-train (14 GB corpus, 5.3M lines): several configs validated; e.g. the “balanced” 3.2M-parameter run reached loss 9.5 → 5.9 within 200 steps, and the “performance” 5.5M-parameter run ran without OOM at 127 MB VRAM.
• SFT (55k Persian instructions): 1.9M-parameter model, seq 512 + packing removed ~70% padding; 11,647 steps/epoch; 11.6 step/s at 87 ms/step on the GTX 1650 Ti; best_loss 2.36.
• Speed-ups: AVX2+FMA ~3.2× vs scalar; Tensor-Core AMP FP16 ~3.8× vs pure FP32.
• Tests: 347+ tests across modules (cargo test --features cuda,amp runs warning-free).

Dependency map (from Cargo.toml)

• Async/runtime: tokio, tokio-tungstenite, futures-util, axum, tower
• Serialization: serde, serde_json
• Parallel & SIMD: rayon, wide, crossbeam, num_cpus
• Memory & I/O: memmap2, memchr, tempfile
• Random: rand, fastrand
• Text/Unicode: unicode-normalization, unicode-segmentation, regex, urlencoding
• CLI/errors/logs: clap, anyhow, thiserror, tracing, tracing-subscriber
• Numeric: ndarray, ndarray-stats, nalgebra, half (optional, for FP16)
• HTTP client: reqwest (rustls)
• Doc/PDF: printpdf, image, sha2
• Misc: chrono, ahash, dashmap, once_cell, lazy_static
• CUDA (feature cuda/amp): direct FFI to libcuda/cublas; kernel build via cc in build.rs
• Optional runtime feature amp: FP16 autocast + GradScaler

Getting started

# CPU-only build
cargo build --release

# GPU with cuBLAS (no AMP)
cargo build --release --features cuda

# GPU + Tensor Cores (recommended on Turing)
cargo build --release --features cuda,amp

# quality gates
cargo fmt --check && cargo clippy -- -D warnings && cargo test

Train a small pre-train run (Eco config):

./target/release/train \
  --corpus <path>/pretrain.txt --device cuda --amp --attn-backend flash \
  --batch-size 4 --seq-len 32 \
  --embed-dim 128 --hidden-dim 256 --layers 2 --num-heads 4 \
  --learning-rate 0.001 --lr-scheduler none --epochs 1 --dataset-mode stream \
  --tokenizer-vocab 2000 --train-tokenizer --threads 8 \
  --save-every 100 --output-dir artifacts/pretrain_eco

Then fine-tune with SFT and chat:

./target/release/train_sft --dataset <path>/persian_sft_data.jsonl \
  --init-from artifacts/pretrain_balanced/latest.ckpt \
  --device cuda --amp --attn-backend flash \
  --batch-size 16 --seq-len 512 --packing \
  --dataset-mode stream --epochs 3 \
  --learning-rate 0.0003 --lr-scheduler cosine --warmup-steps 100 --min-lr 0.00003 \
  --output-dir artifacts/sft_balanced

./target/release/chat   # read checkpoint from config.toml

Why it matters on this résumé

Training runs with 11.6 steps/s on a 1650 Ti are not about the hardware — they are the result of engineering around it: thresholded GPU dispatch, FP16 accumulation, sequence packing and memory pooling. LucilleAI shows that I do not treat deep learning as a black box: I can build the box, open it, and make it fit in 4 GB.