RAG Knowledge Base for Public Authorities: A Guide for the Public Sector

Planning a RAG knowledge base for a public authority? WZ-IT builds knowledge bases with source citations and permissions, operated in your own network or on a managed GPU server, see Internal assistant and RAG Proof of Value in the AI hub. Schedule a meeting

Public authorities manage large collections that have grown over years: decrees and circulars, procedural instructions, technical reports, expert opinions, statements, minutes. People looking for a rule often know its content but not the file reference or title. A RAG knowledge base (Retrieval Augmented Generation) searches the authority's own collection for matching passages and has a language model formulate an answer with the source.

Several German states and the federal government now run their own AI assistants. Most guidance on the topic stays at the level of principles. This article describes how an authority actually builds a RAG knowledge base: which documents cause which effort, where permissions and VS-NfD set limits, what data protection supervisors and the AI Act require, and which operating models exist. As of September 2026, not legal advice.

Table of contents

1. What a RAG knowledge base does in public administration
2. Government AI assistants in Germany at a glance
3. Preparing the document collection
4. Permissions, protection needs and VS-NfD
5. Data protection under the DSK guidance on RAG
6. Classification under the AI Act
7. Operating models for public authorities
8. Architecture from open-source components
9. Guidelines and procurement
10. Our approach at WZ-IT
11. Further guides

What a RAG knowledge base does in public administration

A general AI assistant drafts texts, summarises and translates. It knows neither the current decree of the ministry nor the internal procedural instruction. A RAG knowledge base closes this gap without training the model. The principle is explained in the knowledge article What is RAG?; in short:

1. Documents are split into passages, enriched with metadata and stored in a search index.
2. A question triggers a search in the index, and the relevant passages are selected.
3. The language model formulates the answer only from these passages and cites the sources.

Three properties matter for public administration: every answer is tied to a source and can therefore be verified. The collection can change without touching the model. And the search can enforce permissions before a passage ever reaches the model.

Typical use cases are research in collections of decrees and regulations, specialist knowledge for caseworkers and citizen services, onboarding new staff, drafting statements based on earlier statements, and searching expert opinions and technical reports. What a knowledge base should not do: decide individual cases or generate administrative decisions without human review.

Government AI assistants in Germany at a glance

Before an authority plans its own system, it is worth looking at what already runs in its state or at federal level. The following platforms are publicly documented, as of September 2026:

Sources: Baden-Württemberg State Ministry on F13, F13 on openCode, Hamburg Senate Chancellery on LLMoin, BMDS on AI in public administration, Bavarian Ministry of Finance on BayernKI, CityLAB Berlin on BärGPT and on Parla.

Some points from the primary sources that matter for planning:

• LLMoin is used by around 40,000 employees in Hamburg according to the Senate Chancellery. Lower Saxony, Bremen, Saxony-Anhalt and Rhineland-Palatinate have decided to reuse it. According to Dataport, authority-specific data sources can be integrated.
• KIPITZ runs in the ITZBund data centre. According to the BMDS, the closed system ensures that no administrative data flows back into model training.
• BayernKI runs in the Bavarian government network. Since June 2025, 50 pilot municipalities also use it (Bayerische Staatszeitung, 13 Aug 2026).
• F13 is model-agnostic according to openCode and designed for local operation in government infrastructure. This makes the software reusable for municipalities and other states as well.

Where a dedicated knowledge base fits. The state platforms are broad: many users, general text work, research in uploaded or centrally predefined collections. A dedicated knowledge base becomes relevant when a specialist collection must be maintained permanently, versioned and searchable with permissions per organisational unit, when document repositories or specialist applications have to be connected, or when the state platform is not available at the authority's level of government. It is often not a replacement but an additional data source for the existing platform.

Preparing the document collection

The quality of a knowledge base is decided during preparation, not when choosing the model. Administrative collections follow their own patterns:

Scanned PDFs. Without a text layer, search finds nothing. OCRmyPDF (MPL-2.0) adds a searchable text layer to PDFs using the OCR engine Tesseract (Apache-2.0). For complex layouts with tables, a structure-preserving parser such as Docling (MIT) is suitable. OCR does not replace checking figures that matter for the subject, so samples belong in the acceptance test.

Versions. The most frequent error in regulatory collections is not the wrong answer but the outdated one. Repealed decrees must be removed from the search space. If they are only ranked lower, they resurface with the next question.

Metadata. File reference, date, validity, issuing body, organisational unit and protection level should accompany every passage. They enable filters and permission checks and make the source citation readable for users.

Exact identifiers. File references, sections of law and decree numbers are found poorly by pure vector search. A combination of keyword and semantic search followed by reranking solves this, as described in the knowledge article Hybrid search and reranking. How to split long administrative and legal texts is covered in Chunking strategies for RAG.

Permissions, protection needs and VS-NfD

In an authority, not every person may see every document. A knowledge base must respect these boundaries, and it must do so before the search: passages a person has no access to must never reach the model's context. An instruction in the prompt is not enough. The technical implementation is described in the knowledge article RAG and permissions.

Three patterns have proven themselves:

The DSK recommends implementing purpose limitation through roles and through client or functional separation in the vector database, and checking this already while preparing the documents (DSK guidance on RAG, section 3.4).

VS-NfD. VS-NfD ("classified - for official use only") is the lowest German classification level. At federal level, classified information is governed by the Classified Information Directive (VSA) of 13 March 2023, in force since 1 April 2023; the states have their own rules. Responsibility lies with the authority's classified information security officers. IT security products that implement security functions for protecting classified information must be approved by the BSI under section 51 VSA (BSI, approval).

For a knowledge base, this leads to a clear sequence:

1. Limit the first index to unclassified collections.
2. Add VS-NfD documents only after approval by the security officers, in a separate index on a system cleared for this purpose.
3. Higher classification levels do not belong in a general RAG system.

Whether a specific system may process VS-NfD is decided by the authority, not by the software vendor.

Data protection under the DSK guidance on RAG

In October 2025, the German Data Protection Conference (DSK) published dedicated guidance on RAG systems (version 1.0, press release by the Berlin data protection commissioner). It explicitly addresses companies and public authorities. The key statements:

The DSK also points out that RAG does not change the data protection assessment of model training. A model trained on unlawfully processed data remains so within a RAG system. For model selection this means: document origin and licence, and do not pass inputs to providers that use them for training. Further background is in the knowledge article GDPR-compliant AI.

Classification under the AI Act

An internal knowledge base for decrees and technical documents is generally not a high-risk system. For public administration, Annex III of the AI Act mainly lists two areas where this can differ:

If a system is deployed in one of these areas, public bodies as deployers must additionally carry out a fundamental rights impact assessment under Article 27 before use. Under the Digital Omnibus, Regulation (EU) 2026/1744, obligations for stand-alone high-risk systems under Annex III apply from 2 December 2027.

Independently of this, the AI literacy obligation for staff under Article 4 and the transparency obligations under Article 50 already apply. The detailed assessment is in the article RAG and the AI Act: is a RAG system high-risk AI?, the deadlines in the article on the Digital Omnibus.

Operating models for public authorities

The knowledge base consists of the index, search services and the language model. In almost all variants, index and documents stay in the authority's own environment; what matters is where the language model runs:

US hyperscalers are an option for many authorities only to a limited extent because of the US CLOUD Act, even with data centres in the EU. A comparison of European providers is in the article European LLM APIs compared.

GPU memory depends on the language model and the number of concurrent requests, hardly on the size of the collection. Embedding model and reranker need little memory. A sizing aid is the article GPU VRAM sizing for LLMs. WZ-IT's managed GPU servers are available with NVIDIA RTX PRO 4000 Blackwell (24 GB) or RTX PRO 6000 Blackwell (96 GB). The AI Cube is a local AI appliance with 128 GB unified memory and 1, 2 or 4 TB NVMe storage for operation in your own network.

Architecture from open-source components

A knowledge base can be built entirely from open-source components. This supports reuse and avoids lock-in to one vendor:

Which vector database fits is shown in the comparison Qdrant vs. pgvector. Which embedding model handles German administrative language well is covered in Best embedding models for German. Logging for traceability and audit is described in the article on Langfuse.

Guidelines and procurement

For federal and state authorities, several guidance documents now support design and tendering:

Contract templates. For cloud services, the Federal Government Commissioner for Information Technology has published the EVB-IT Cloud terms. Which EVB-IT template fits hardware, implementation and operation of a knowledge base depends on how the services are split and should be clarified with the procurement office.

Accessibility. For federal public bodies, the BITV 2.0 also covers electronically supported administrative processes and graphical user interfaces. The interface of a knowledge base must therefore be usable by keyboard and screen reader. The states have their own rules.

Pilot before tender. A defined pilot with one collection and agreed reference questions shows whether the collection works before a large tender is drafted. The results provide solid requirements for the specification. How answer quality is measured and accepted is described in RAG evaluation.

Our approach at WZ-IT

We build knowledge bases as a project with a fixed sequence and hand over a system whose components are documented.

1. Collection analysis. Formats, share of scanned documents, versioning, metadata and permissions. The result is a written assessment of which parts of the collection can be processed and what should be cleaned up first.
2. Pilot with reference questions. In the RAG Proof of Value, a defined collection is assessed against 40 to 60 agreed reference questions, including source and "unknown" tests. It ends with a quality report and a recommendation for production.
3. Permissions and protection needs. Separate indexes or filters per organisational unit; classified documents only after approval by the authority.
4. Operation. On your own hardware in the government network with the AI Cube, in your existing environment, or on a managed GPU server from WZ-IT. Ongoing operation with updates and monitoring via Managed AI.
5. Integration. The knowledge base runs with its own interface or, where the existing platform supports it, as a connected data source.

For publicly accessible specialist collections such as funding guidelines or information portals, there is the Knowledge Navigator, which guides visitors to the authoritative passage with source and version. Support, consulting and implementation by WZ-IT.

Further guides

• RAG and the AI Act: is a RAG system high-risk AI?, Annex III, provider and deployer roles.
• European LLM APIs compared, providers, locations and contracts for models from the EU.
• Best embedding models for German, which models represent German specialist language well.
• Chunking strategies for RAG, splitting administrative and legal texts.
• RAG data sources: Nextcloud, SharePoint and DMS, connecting existing repositories.
• On-premise AI solutions compared, operating models for AI in-house.
• AI hub, all AI services from WZ-IT.

Planning a knowledge base for your authority? We review your collection, define permissions and operating model and assess answer quality against your own questions. Schedule a meeting

Sources

• Baden-Württemberg State Ministry, F13 becomes open-source software (23 Jul 2025)
• openCode, F13 KI Assistenz
• Hamburg Senate Chancellery, LLMoin as a nationwide model (15 Dec 2025)
• Dataport, LLMoin available for reuse
• BMDS, AI in public administration
• Bavarian Ministry of Finance, AI in the Bavarian administration
• Bayerische Staatszeitung, BayernKI (13 Aug 2026)
• CityLAB Berlin, BärGPT
• BärGPT on GitHub
• CityLAB Berlin, Parla
• German Data Protection Conference, guidance on RAG systems, version 1.0, October 2025
• Berlin data protection commissioner, press release on the DSK RAG guidance
• Classified Information Directive (VSA) of 13 March 2023
• BSI, approval of IT security products
• AI Act, Annex III
• AI Act, Article 27
• Regulation (EU) 2026/1744 (Digital Omnibus)
• Guidelines for the use of AI in the federal administration
• BSI, criteria catalogue for generative AI in the federal administration (24 Jun 2025)
• BSI, Generative AI models: opportunities and risks for industry and authorities
• Bavarian Ministry of Finance, AI guideline for authorities (April 2025)
• Digitale Verwaltung, current EVB-IT templates
• BITV 2.0, section 2 scope
• OCRmyPDF, Tesseract, Docling