How Dutch universities train open models
Strategic dependence on closed, foreign cloud providers carries significant risks in terms of digital sovereignty, data leaks, intellectual property, and cultural representation. Where commercial hyperscalers pump trillions of tokens through gigantic infrastructures, resulting in closed black-box models, Dutch knowledge institutions such as the University of Amsterdam, TU Delft, Utrecht University, Radboud University, TNO, and NWO operate within a fundamentally different framework. They pool public compute power, academic datasets, and open science principles to train verifiable and transparent models that anyone can freely inspect and deploy.
In this article, we look at the concrete methodologies, compute infrastructures, data pipelines, architecture choices, and evaluation suites that Dutch researchers use when building open-source language models. These initiatives form the technical backbone for an independent open-source ecosystem in the Low Countries, a development that closely aligns with the observations in the Dutch AI scene August 2026 in which technological sovereignty and local inference take center stage.
Compute capacity: National supercomputers and EuroHPC clusters
Training a modern LLM from scratch (pre-training) or deeply adapting one via continuous pre-training requires thousands of GPU hours that an individual university faculty cannot deliver with its own departmental servers. The linchpin of Dutch research compute is SURF, the cooperative association for ICT in Dutch education and research. Via the national supercomputer system Snellius, physically located at Amsterdam Science Park, research teams get access to hundreds of GPU nodes equipped with NVIDIA A100 and H100 accelerators.
When projects scale up to model sizes of 30 to 70 billion parameters, even a national cluster falls short in terms of continuous capacity. Dutch consortia therefore structurally turn to European EuroHPC supercomputers, such as the LUMI facility in Finland (based on AMD Instinct MI250X accelerators) or the Leonardo cluster in Italy. Effectively distributing a training run across hundreds of nodes requires refined parallelization strategies. Researchers combine Tensor Parallelism (TP), Pipeline Parallelism (PP), and Data Parallelism via frameworks such as Megatron-LM and DeepSpeed ZeRO-3.
In multi-node setups over InfiniBand networks (often 400 Gbps to 800 Gbps non-blocking fabrics), communication overhead is the biggest risk of wasted compute. When a node stalls or data traffic gets stuck on inter-node synchronizations (such as all-reduce operations for gradients), expensive accelerators immediately fall back to idle cycles. To address this, engineering teams implement asynchronous distributed checkpointing on parallel NVMe storage systems (Lustre or GPFS), so that training can resume seamlessly within minutes after a hardware failure.
Data collection and curation: Clean Dutch versus web noise
The biggest bottleneck for training a language-specific model isn't compute power, but data quality. The Dutch language area represents only a fraction of the volume available for English on the public internet. In unfiltered web dumps such as Common Crawl, a large percentage of Dutch text consists of automated translations, casino spam, SEO filler, machine-generated e-commerce catalogs, and fragmentary web pages. When such data ends up uncorrected in a pre-training corpus, it leads to syntactic pollution and hallucinations in the final model.
Academic teams therefore use multi-layered curation pipelines. Primary sources include the Delpher archive of the Koninklijke Bibliotheek (historical and modern newspapers and books), public parliamentary records of the Tweede Kamer der Staten-Generaal, open-access publications from Dutch academic repositories (such as NARCIS), curated news archives, and checked Wikipedia dumps.
Processing goes through strict filtering steps:
- Language detection and quality heuristics: FastText classification models filter out non-Dutch passages and dialect mixtures. Documents with too low a ratio between unique words and total text length (low lexical diversity) or an excess of punctuation are rejected outright.
- Deduplication at document scale: MinHash Locality Sensitive Hashing (LSH) identifies and removes duplicates and near-duplicates across trillions of tokens, preventing overfitting on repeated boilerplate text.
- Perplexity filtering: A small, robust language model (often trained exclusively on high-quality books and legal texts) computes the cross-entropy loss of candidate texts. Texts with extremely high perplexity (illogical sentence structure) or extremely low perplexity (repetitive text sequences) are excluded.
Tokenization and vocabulary optimization for Dutch
Standard tokenizers of models such as Llama, Mistral, or GPT are heavily optimized for English. Such tokenizers lack many common Dutch stems and compound words, causing words to fall apart into inefficient sub-tokens or individual bytes. A compound noun such as "aansprakelijkheidsverzekering" (liability insurance) can be split into 7 to 9 tokens by a generic English-language tokenizer, whereas a language-specific vocabulary represents it in 2 or 3 tokens.
This sub-token fragmentation has direct technical consequences: the model's effective context window shrinks drastically, inference costs per word shoot up, and the model has to expend extra compute to build internal associations between separate syllables. Dutch research groups therefore choose between two methodological solutions:
- Training entirely new tokenizers: Using SentencePiece (BPE or Unigram), a vocabulary of 64,000 to 128,000 tokens is built that is specifically tuned to the morphological structure of Dutch, including common prefixes and compound words.
- Tokenizer surgery and vocabulary expansion: The existing vocabulary of an international base model is extended with 10,000 to 30,000 Dutch subwords. The corresponding new embeddings in the input and output layers are initialized via cross-lingual correlation matrices and then embedded during an initial training phase.
The Python snippet below shows how an optimized SentencePiece Byte-Pair Encoding tokenizer is trained on a curated Dutch corpus, including explicit context tokens for structured data processing:
import sentencepiece as spm
spm.SentencePieceTrainer.train(
input='corpora/gecureerd_nederlands_corpus.txt',
model_prefix='nl_bpe_tokenizer_64k',
vocab_size=64000,
character_coverage=0.9998,
model_type='bpe',
max_sentence_length=16384,
split_digits=True,
byte_fallback=True,
pad_id=0,
unk_id=1,
bos_id=2,
eos_id=3,
user_defined_symbols=[
'[SYSTEM]', '[USER]', '[ASSISTANT]',
'[BRON_START]', '[BRON_EIND]', '[FEIT]'
]
)Pre-training from scratch versus continuous pre-training
Training a foundation model from scratch offers maximum freedom over the data architecture and licensing status, but requires astronomical budgets. For that reason, universities often opt for a strategic balance between building entirely from scratch and continuous pre-training on existing open architectures. Projects such as GEITje and GPT-NL show how both approaches have their own technical pros and cons.
In continuous pre-training, catastrophic forgetting is the biggest risk: when an existing model (for example Mistral or Llama) is fed exclusively with Dutch texts, the new gradients overwrite earlier neural connections. The result is that the model does learn more fluent Dutch, but seriously loses complex reasoning ability, formal logic, and coding functionality. To prevent this degradation effect, researchers use a replay buffer: the Dutch training corpus is mixed with 30% to 50% high-quality English-language code and math data (such as StarCoder and OpenWebMath subsets).
Anyone wanting to follow the broader international movement around weights and open-source licenses will find an extensive framework in the overview on open-weight models from international sources, which analyzes the distribution mechanisms of open weights worldwide.
| Strategy | Compute investment | Dutch language proficiency | Reasoning capacity | Risk of degradation |
|---|---|---|---|---|
| Scratch pre-training | Extremely high (>100k GPU hours) | Deep, authentic, no bias | Fully dependent on compute | None (no prior knowledge) |
| Continuous pre-training | Medium (5k - 25k GPU hours) | Excellent at the idiomatic level | Retains base ability given replay mix | Moderate to high (catastrophic forgetting) |
| Supervised fine-tuning (SFT) | Low (100 - 1.5k GPU hours) | Superficial / stylistic | Equal to underlying base model | Low (limited weight adjustment) |
Instruction tuning and alignment: Capturing the Dutch context
A base model that has processed billions of tokens is a powerful pattern predictor, but not yet a helpful assistant. It generates text completions but doesn't understand instructions or dialogue. The step from raw base model to an instruction-following model happens via Supervised Fine-Tuning (SFT) and Preference Alignment. In the past, researchers machine-translated American instruction sets (such as Stanford Alpaca or ShareGPT) using translation models, but this introduced so-called 'translationese': stilted Dutch, American cultural assumptions, and incorrect legal concepts.
Dutch academics today focus on locally curated instruction datasets. These sets contain concrete tasks from Dutch practice: summarizing municipal policy documents, extracting legal entities from court records, and answering questions about the Dutch tax and healthcare systems. This explicitly steers on the difference between formal forms of address (the 'u' form) and informal interactions ('je'), which isn't directly anchored causally in English-language source models.
For the alignment phase, Direct Preference Optimization (DPO) or KTO (Kahneman-Tversky Optimization) is preferred over traditional RLHF with reward models. DPO optimizes the policy parameters directly on pairs of preferred and rejected answers. This allows research teams to specifically suppress unwanted biases, hallucinating patterns, and unsafe statements without having to train an unstable secondary neural network.
Legal and ethical frameworks: GDPR, AI Act, and PII scrubbing
Where commercial entities sometimes scrape data under the guise of fair use and settle afterward, public universities operate within strict ethical and legal frameworks. Research projects must fully comply with the General Data Protection Regulation (GDPR) and the European AI Act. This means training data must be traceable and personal data must be thoroughly removed before weights are optimized.
The anonymization process includes automated pipelines with Named Entity Recognition (NER) models, supplemented with deterministic regex filters for citizen service numbers (BSN), IBAN account numbers, phone numbers, email addresses, and postal code-house number combinations. In addition, the AI Act requires training corpora to be documented via standardized Data Sheets for Datasets and Model Cards, which explicitly report on the origin of sources, copyright agreements, and potential demographic imbalances in the data.
Evaluation methodology: Why English-language benchmarks fall short
Measuring model quality is one of the most persistent challenges in Dutch LLM research. Standard benchmarks such as MMLU (Massive Multitask Language Understanding), GSM8K (mathematical reasoning), and HumanEval (code) were developed for the English language and the American context. Automatically translated versions of these benchmarks introduce structural errors: math problems lose their semantic consistency due to translation errors, and questions about legislation or societal organization miss the mark because they refer to American state institutions.
To truly validate performance reliably, Dutch universities have set up specific evaluation tracks. These measure not only grammatical correctness and vocabulary but also test models on standardized reading comprehension tasks, Dutch laws and regulations, and cultural-historical knowledge. To understand how these test setups are structured and which methodological pitfalls occur in automated assessment, the overview on benchmarks for Dutch-language model output offers in-depth technical background on objective evaluation suites.
Practical implementation: Running models in a local environment
The academic goal of open AI research is only achieved once the trained weights can actually be deployed in society. The trained models are published under open licenses (such as Apache 2.0 or MIT) on platforms such as Hugging Face. This allows government agencies, software developers, and healthcare institutions to download the weights and run them on their own servers, without dependence on external API infrastructure.
Because university models range from compact 7B/8B models to heavier 70B variants, quantization plays a crucial role in adoption. By reducing the original FP16 or BF16 weights to 4-bit or 8-bit precision using techniques such as GGUF (for CPU/Metal inference) or AWQ/EXL2 (for GPU inference), these models run smoothly on common workstations and local servers. This shift toward local autonomy was already visible early on in the signals from the Dutch AI scene in July 2026.
Anyone wanting to get straight to hosting academic weights on their own hardware will find a concrete guide in the article on Running Dutch and European open models locally, which works out runtime optimizations and memory requirements in detail.
Conclusion: The balance between autonomy, costs, and effectiveness
Training open language models at Dutch universities shows that technological independence is achievable, provided sharp choices are made. Knowledge institutions cannot compete with commercial giants in terms of raw compute power and budget, as those invest hundreds of millions in individual training runs. However, by focusing on superior data quality, language-specific tokenization, methodological transparency, and strict ethical standards, Dutch researchers deliver models that are invaluable to the local language area.
The combination of national compute infrastructure at SURF, European collaboration via EuroHPC, and an open-source distribution strategy ensures that the Dutch language area doesn't degrade into a purely consuming market for foreign black-box systems, but retains an independent, research-driven, and sovereign position within the global AI landscape.
