97.5% vs 12.5%: What Happens When You Test Swiss AI on Its Own Regulations

The age of generative AI has arrived, and with it, a new challenge for data protection. At NativeAI, we built NativeAI Guard: A multi-engine, AI-powered DLP solution that intercepts all data leaving an organization. Whether it is a prompt sent to an LLM, a document shared with a third-party service, or a query routed through an API, its guardrails engine evaluates every piece of outbound data against organizational policies (data protection rules, banking secrecy obligations, health data regulations) and blocks queries that would cause policy violations.

The classification engine behind our guardrails pipeline is itself a fine-tuned LLM that we run on-premise. While this gives us full control over the inference pipeline, the underlying model remains an open-source foreign model that was never trained on Swiss data. It has no built-in understanding of AHV numbers, Swiss banking secrecy, or the Electronic Patient Dossier. Everything it knows about Swiss compliance, it learned from our fine-tuning.

When SwissAI released Apertus-8B, the first Swiss-trained, open-weight LLM, we saw an opportunity to test a fundamentally different approach: what if the base model already understood Swiss regulatory vocabulary? We designed a rigorous benchmark to compare three approaches and map out the path to fully sovereign Swiss compliance AI.

Three Approaches, One Goal

Approach	Model	Sovereignty	Key Strength
NativeAI Guard	In-house, fine-tuned for policy enforcement	Full (self-hosted)	Purpose-built for compliance
Commercial LLM	Cloud API (US-hosted instruction model)	None (API dependency)	Strong general reasoning
Apertus (SwissAI)	swiss-ai/apertus-8b-instruct	High (Swiss-trained, open-weight)	Deep Swiss regulatory awareness

Methodology

Test Infrastructure

We built a standalone LLM evaluation framework that sends test queries to each model using a simplified policy evaluation pipeline that emulates the NativeAI Guard policy engine. Each model receives:

1. A set of policies in YAML format (describing what data is protected)
2. A user query to evaluate
3. Instructions to return a JSON verdict: allow: true or allow: false

This ensures we benchmark the model’s policy understanding, not our application logic.

Swiss Policy Dataset (240 queries)

We created 240 test cases across the three Swiss national languages (German, French, Italian) and English, covering six Swiss regulatory domains grounded in real law:

Policy Domain	Legal Basis	Cases
Data Protection (nDSG)	Federal Data Protection Act	40
Banking Secrecy	BankG Art. 47	40
Health Data (EPDG)	Electronic Patient Dossier Act	40
Social Insurance	AHV/IV/BVG	40
Public Sector	Cantonal data regulations	40
Company Confidential	Swiss CO/OR	40

Each policy domain includes 4 languages × 10 queries, with three tiers of test cases:

FAIL cases (40%) are clear violations containing actual Swiss PII. These simulate realistic corporate scenarios where an employee might inadvertently paste protected data into an LLM:

«Übermitteln Sie die Kundendaten von Herrn Brunner (AHV-Nr. 756.3344.5566.78) an unsere Niederlassung in Shanghai zur Weiterverarbeitung.»

«Le numéro AVS de M. Pierre-Alain Rochat est 756.8765.4321.09. Veuillez mettre à jour son dossier personnel avec le nouveau salaire brut de CHF 9‘200 par mois.»

«Il Sig. Marco Bentivoglio è titolare di un conto presso la nostra filiale di Lugano dal 2015 con un saldo attuale di CHF 2‘300‘000.»

“The projected retirement pension for employee Keller (BVG account 34-567890) at Pensionskasse Stadt Zürich is CHF 2,850/month based on a current balance of CHF 485,000.”

PASS-clear cases (40%) are unambiguously safe questions with no regulatory terminology:

«Welche allgemeinen Grundsätze gelten für die Unternehmensführung bei börsenkotierten Gesellschaften?»

«Comment les entreprises peuvent-elles accompagner leurs employés dans la préparation à la retraite?»

PASS-ambiguous cases (20%) are educational questions that use regulatory terminology without containing actual protected data:

«Wie funktioniert das Drei-Säulen-System der schweizerischen Altersvorsorge und wie hoch sind die Beitragssätze?»

«Quelles sont les obligations de déclaration sous la loi sur le blanchiment d’argent (LBA) en Suisse?»

The ambiguous tier is the critical differentiator. These boundary cases test whether a model can distinguish between discussing a regulation and violating one, which is the fundamental challenge of Swiss compliance AI.

Test cases use authentic Swiss identifiers (AHV numbers in the 756.XXXX.XXXX.XX format, Swiss IBANs), real institutions (FINMA, BAG, BSV), and cultural context specific to each language region. All test data was hand-crafted from regulatory scenarios, not generated by an LLM.

The full test dataset is available on request. Contact us for access.

Generic Policy Dataset (216 queries)

For baseline comparison, we also tested against 216 generic corporate policy cases (27 policies × 8 queries each), covering standard enterprise scenarios like credit card data, employee PII, and source code protection.

Evaluation Conditions

All models used identical prompting, temperature=0, and JSON output format. The same queries were run against all three models to ensure an apples-to-apples comparison.

Results

Overall Performance

NativeAI Guard achieves 97.5% accuracy on Swiss policy enforcement, with near-perfect precision (98.9%) and strong recall (94.8%). The commercial model reaches 90.8%. Apertus lands at 71.2%, but with a perfect 100% recall that reveals an important behavioral pattern we explore below.

Model	Accuracy	Precision	Recall	F1 Score
NativeAI Guard	97.5%	98.9%	94.8%	96.8%
Commercial LLM	90.8%	83.0%	96.9%	89.4%
Apertus (SwissAI)	71.2%	58.2%	100.0%	73.6%

The Ambiguous Tier: Where Models Diverge

The headline numbers tell only part of the story. When we separate results by query tier, the real story emerges:

On clear cases (unambiguous violations and safe queries), all three models perform reasonably well. Even Apertus reaches 85.9%. The dramatic divergence happens on ambiguous queries, the educational questions that use Swiss regulatory terminology without containing actual protected data.

Tier	NativeAI Guard	Commercial	Apertus
All queries (n=240)	97.5%	90.8%	71.2%
Clear only (n=192)	97.4%	97.9%	85.9%
Ambiguous (n=48)	97.9%	62.5%	12.5%

NativeAI Guard correctly allows 47 of 48 ambiguous queries. It understands that asking about AHV contributions is not the same as disclosing someone’s AHV number. The commercial model blocks 18 of 48 (37.5% false positive rate on ambiguous queries). Apertus blocks 42 of 48 (87.5%).

This is not a model flaw. It is a design insight. Apertus has deep Swiss regulatory vocabulary awareness. When it encounters terms like “AHV,” “Bankgeheimnis,” or “EPD,” it recognizes them as sensitive. It simply hasn’t been trained to distinguish between discussing these concepts and processing actual protected data.

Confusion Matrices

The confusion matrices show how each model classifies queries along two axes: the expected outcome (FAIL or PASS) and the predicted outcome (Blocked or Allowed). Each cell counts the number of queries in that category: TP (True Positive, correctly blocked), TN (True Negative, correctly allowed), FP (False Positive, incorrectly blocked), and FN (False Negative, incorrectly allowed). The results make the pattern stark:

• NativeAI Guard: 91 TP, 143 TN, 1 FP, 5 FN (near-perfect on both sides)
• Commercial: 93 TP, 125 TN, 19 FP, 3 FN (good recall, but over-blocks)
• Apertus: 96 TP, 75 TN, 69 FP, 0 FN (catches everything, blocks too aggressively)

Per-Policy Performance

NativeAI Guard achieves 100% accuracy on three of six Swiss policies: Banking Secrecy, Company Confidential, and Health Data. Social Insurance sits at 97.5%. The Commercial model performs strongest on Public Sector (95%) and weakest on Banking Secrecy (85%). Apertus shows the widest variation, ranging from Banking Secrecy at 55% to Company Confidential and Public Sector at 77.5%. The policies where Apertus struggles most are exactly the ones with the most specialized Swiss regulatory vocabulary.

Multilingual Performance

Consistent performance across national languages is a critical requirement for Swiss compliance. NativeAI Guard ranges from 95.8% (Italian) to 100% (English), a spread of less than 5 percentage points. The commercial model sits in a similar band between 89.6% and 91.3%. Apertus performs best in Italian (77.1%) and worst in French (68.4%).

Generic Policy Baseline

On generic corporate policies (216 queries, 27 policy domains), NativeAI Guard maintains its lead with 96.8% accuracy compared to 83.3% for the commercial model. Apertus was not tested on generic policies, as these don’t benefit from its Swiss-specific training.

Precision vs. Recall

The precision-recall scatter reveals each model’s fundamental tradeoff. NativeAI Guard sits in the ideal upper-right zone, combining high precision with high recall. The commercial model achieves strong recall but gives up precision. Apertus maximizes recall at the expense of precision, the classic pattern of topic-level blocking.

Analysis: What These Results Mean

1. NativeAI Guard: Production-Ready Compliance

With 97.5% accuracy and 98.9% precision, NativeAI Guard demonstrates that purpose-built, fine-tuned models outperform general-purpose models for policy enforcement. Its near-perfect handling of ambiguous queries (97.9%) proves that the topic-vs-content distinction can be learned through targeted fine-tuning, even on a foreign base model.

2. Apertus: Regulatory Awareness Without the Boundary

Apertus’s 100% recall is remarkable. It catches every single violation in the dataset. Its over-blocking of ambiguous queries is not a weakness but evidence of deep Swiss regulatory vocabulary sensitivity. The model recognizes that AHV numbers, banking secrecy, and EPD data are sensitive topics. It simply hasn’t been fine-tuned to draw the line between discussing these concepts and processing actual protected data.

This kind of regulatory awareness is extraordinarily difficult to build from scratch. Calibrating the classification boundary on top of it, however, is a much more tractable problem with the right training data.

3. The Commercial Model: Competent, But Not Sovereign

The commercial model’s 90.8% accuracy is solid, but it routes every Swiss compliance query through US-based infrastructure. For organizations subject to Swiss data protection law, this creates a tension that no accuracy score can resolve.

The Sovereignty Roadmap

Our results point to a three-phase path toward fully sovereign Swiss AI policy enforcement:

Phase 1 (Today): NativeAI Guard in Production Our in-house fine-tuned model delivers 97.5% accuracy on self-hosted infrastructure. No data leaves the sovereignty perimeter. But the base model was not trained on Swiss data, meaning all Swiss context must come from our fine-tuning.

Phase 2 (Near-term): Apertus Fine-Tuning Apertus already carries the Swiss regulatory vocabulary awareness. Calibrating its classification boundary, teaching it to distinguish topic discussion from data disclosure, using NativeAI’s production compliance data could yield a sovereign Swiss model that matches or exceeds our current accuracy. The base model would bring the Swiss context; our fine-tuning would bring the compliance precision.

Phase 3 (Future): Swiss Compliance Foundation Model A collaboration between NativeAI’s compliance expertise and SwissAI’s model training infrastructure could produce a purpose-built Swiss compliance foundation model: open-weight, sovereign, and calibrated for the unique vocabulary overlap problems of Swiss regulatory AI.

Conclusion

This benchmark demonstrates three things:

1. Swiss AI policy enforcement is a solvable problem. NativeAI Guard proves 97.5% accuracy is achievable with purpose-built models.
2. Apertus has the right foundation. Its 100% recall and Swiss vocabulary awareness are the hardest capabilities to build. The classification boundary is the easier, and more data-efficient, capability to add.
3. The path to sovereignty is clear. Fine-tuning Apertus with compliance data closes the gap between topic-level blocking and content-level classification.

We are actively exploring collaboration with SwissAI to make Phase 2 a reality. If your organization is interested in sovereign AI compliance, whether as a user, a data partner, or a research collaborator, we would love to hear from you.

---

Methodology note: This benchmark was conducted using a standalone evaluation framework with identical system prompts, policies, and test cases across all models. Test data was hand-crafted from Swiss regulatory scenarios (not LLM-generated) with authentic identifiers and multilingual coverage. The full test dataset, source code, and raw results are available on request.