Loading
Research · Thread 04
Domain LLM fine-tune
Teaching a general-purpose model the vocabulary of one market, on one GPU.
QLoRA is a way of adapting a large language model by training a small set of extra weights instead of retraining the whole thing, which is what brings the job down from a server rack to a single graphics card. The question that made affordable to ask was whether a domain-tuned model actually beats a base model with retrieval for a narrow, terminology-heavy vertical where the entities are rare. Barnes Dubai LLM is the production answer for Dubai luxury real estate, trained on national registry and brokerage records rather than on scraped listings.
Findings
Domain fine-tune vs base + RAG
QLoRA made a 7-13B fine-tune tractable on one GPU. Barnes Dubai LLM v8 is the production test: does a domain tune actually beat base + RAG for Dubai real estate?
Yes, on terminology and named-entity recall.
A multi-source property corpus
15,369 pairs: DLD transactions, buildings, brokers, developers, plus UK/FR/SG comparables. Tool-calling agents wrap it with inventory and valuation lookups.
The corpus taught the model the team's judgment, not just the registries' wording.
Paper to shipped
Paper
QLoRA: Efficient Finetuning of Quantized LLMs (Dettmers, Pagnoni, Holtzman, Zettlemoyer, NeurIPS 2023)4-bit NormalFloat + double quantization + paged optimisers, the toolkit that brought 33B+ fine-tunes onto a single workstation GPU.
View abstract
QLoRA is an efficient finetuning approach that reduces memory usage enough to finetune a 65B-parameter model on a single 48GB GPU while preserving full 16-bit-finetune task performance. QLoRA backpropagates gradients through a frozen 4-bit quantized pretrained language model into Low-Rank Adapters (LoRA), and introduces three contributions: a new 4-bit NormalFloat data type, double quantization, and paged optimisers to manage memory spikes.
Implementation
15,369 instruction pairs, multi-source Dubai corpusBuilt from national registry transactions and brokerage inventory rather than scraped listings, then wrapped in tool calls so the model looks a figure up rather than recalling it.
Shipped
Barnes Dubai LLM v8 in productionFastAPI service the brokerage team dogfoods in a playground before WhatsApp distribution to brokers. The customer-facing answer to 'can a domain LLM beat base + RAG for our vertical' is currently 'yes, on terminology and named-entity recall'.