Certifying LLM-Based Applications: A Multi-Dimensional Approach
Large Language Models (LLMs) are reshaping how applications are designed and deployed. They power chatbots, generate code and configuration files, and increasingly take decisions inside complex systems. Yet their probabilistic nature and the opacity of their training data introduce new security, privacy, and ethical risks. Our research shows why traditional assurance falls short for LLM-based applications and proposes a multi-dimensional certification scheme to evaluate their non-functional behavior.
The Core Challenge
Traditional assurance techniques assume deterministic behavior and full inspectability of the system under evaluation. LLM-based applications break both assumptions because of:
- Probabilistic, non-deterministic outputs
- Opaque pre-training data and models
- Complex life cycles spanning pre-training, adaptation (e.g., fine-tuning), and inference
- Tight integration with legacy systems, microservices, and nanoservices
As a result, evaluating only what an LLM outputs is not enough to gain justifiable confidence in properties such as security, fairness, or transparency.
Not All LLM-Based Applications Are the Same
We define a taxonomy of LLM-based applications, each requiring a different certification approach:
- Standalone: a foundation LLM whose output is consumed directly, such as a chatbot.
- Integrated, artifact generation: an adapted LLM that produces artifacts, such as code or deployment files, used by the rest of the application.
- Integrated, decision-making: an LLM that selects actions within a larger application.
Key Gaps in Current LLM Assessment
Our analysis identified five gaps that limit the reliability of current approaches:
- Limited transparency into pre-training data and models.
- Focus on functional metrics rather than non-functional properties such as security.
- Lack of precise definitions of non-functional properties for LLMs.
- Over-reliance on reference datasets and LLM-as-a-Judge evaluations.
- Output-only evaluation that ignores the pre-training and adaptation life cycle.
A Multi-Dimensional Certification Scheme
Our scheme addresses these gaps by making the behavior of the application and the supporting evidence explicit:
- Four certification dimensions: pre-training, adaptation, inference, and generated artifacts.
- Hypergraph-based certification model: a property is paired with a hypergraph describing the compatible behaviors that support it and guiding evidence collection.
- Verification steps: each step combines a target mechanism with an activity such as testing, monitoring, or inspection.
- Structured certification process: evidence is collected through the hypergraph, insufficient paths are pruned, and a signed certificate is awarded listing the verified behaviors.
- Flexibility: dimensions whose processes are not accessible can be excluded or assessed through documentation.
Putting It to the Test
We applied the scheme to an LLM-based application for cloud security assessment, built on self-hosted open-weight LLMs and composed of three modules:
- Recommendation engine (decision-making): certified for resilience to order bias. All tested models showed significant bias, and larger models were not necessarily less biased.
- Explanation engine (standalone): certified for readability for non-expert users. Explanations were syntactically readable but relied on unfamiliar vocabulary, and more detailed prompts performed worse.
- Deployment engine (artifact generation): certified for the security of generated Kubernetes deployment files. Results varied widely across models, from configurations that rarely met the requirements to models that met them consistently.
Certifying LLM-Based Applications with Moon Cloud
The Moon Cloud platform can be used to put the certification scheme described above into practice. In particular, Moon Cloud supports:
- Evidence collection: gathering the evidence required by the certification model across the dimensions of the LLM-based application, from pre-training and adaptation to inference and generated artifacts.
- Verification: executing the verification steps, such as testing, monitoring, and inspection, that assess whether the application exhibits the behaviors supporting the target property.
This makes it possible to assess non-functional properties of LLM-based applications, such as security, fairness, and transparency, in a systematic and repeatable way, providing the evidence needed to award a certificate.
Why It Matters
Regulatory frameworks such as the EU AI Act increasingly require evidence that AI systems are trustworthy. Certificates that make behaviors and evidence explicit can be inspected, compared, and reused, supporting both compliance and informed adoption of LLM-based applications. Future work includes behavior templates, semi-automated human-in-the-loop generation of certification models, and integration with model cards and a chain-of-trust model.
References
N. Bena, M. Anisetti, E. Damiani, A. Della Bruna, C. Y. Yeun and C. A. Ardagna, “A Certification Scheme for Large Language Models-Based Applications,” in ACM Transactions on Intelligent Systems and Technology, 2026