This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Build and Deploy

Securing AI-generated code, validating AI models, testing AI applications, and enforcing secure build and deployment controls.

Validating AI Artifacts and Securing AI-Enabled Software Delivery

Artificial intelligence introduces new security considerations into the build and deployment process. AI-generated code, machine learning models, prompts, agent configurations, and supporting dependencies must be validated before they are incorporated into production software.

Phase 2: Build and Deploy focuses on ensuring that AI-generated code and AI application components are securely built, tested, verified, and authorized for deployment.

This phase extends traditional CI/CD security practices to address AI-specific vulnerabilities, model integrity, adversarial testing, agent permissions, and the traceability of AI-related software artifacts.

The objective is to establish automated security controls that verify the integrity and security of AI-enabled software before it reaches production.

Phase 2 Outcomes

By implementing these practices, DevOps teams establish automated security controls for validating and deploying AI-enabled software.

Expected outcomes include:

  • AI-generated code is validated using established secure development and testing practices.
  • AI models and supporting artifacts are verified for integrity and provenance.
  • AI applications are tested for prompt injection, unsafe behavior, and unauthorized tool execution.
  • Autonomous agents and AI services operate within defined deployment permissions.
  • Software releases include traceable AI artifacts, dependencies, and security evaluation results.
  • Deployment authorization and security policies are consistently enforced.
  • Security evidence is available for ongoing vulnerability management and compliance assessments.

These outcomes help ensure that AI-enabled applications enter production with verified artifacts, tested security controls, and traceable release information.

The following frameworks provide guidance relevant to Phase 2 AI security activities.

Framework Relevant Guidance
NIST SP 800-218 – Secure Software Development Framework Secure software production, security testing, artifact protection, and vulnerability identification.
NIST SP 800-218A – AI SSDF Profile AI-specific secure development practices, including AI model security and lifecycle considerations.
NIST AI Risk Management Framework AI risk measurement, evaluation, validation, and risk management.
OWASP Top 10 for LLM Applications Prompt injection, sensitive information disclosure, supply chain vulnerabilities, excessive agency, and other AI application risks.
OpenSSF Model Signing Signing and verification of AI model artifacts to support integrity and provenance.
SLSA Software supply chain integrity, build provenance, and protection against artifact tampering.

These frameworks and projects provide guidance and technical capabilities that can support AI security controls. They do not all establish mandatory regulatory compliance requirements.

Integrating AI Security into the CI/CD Pipeline

AI security controls should be incorporated into existing CI/CD automation rather than implemented as disconnected manual processes.

This section will cover a typical AI-enabled software delivery workflow to include:

1 - AI-Generated Code Validation

Validating AI-generated code through automated security testing, dependency analysis, regression testing, and controlled CI/CD release gates.

AI-Generated Code Validation - Phase 2: Build and Deploy

Artificial intelligence is increasingly used to generate source code, update dependencies, modify configurations, and create pull requests. These changes can accelerate software delivery but may introduce vulnerabilities, insecure coding patterns, compatibility problems, and unintended functionality.

During Build and Deploy, organizations should independently validate AI-generated changes using established security testing and software quality controls.

AI-generated code should not be trusted solely because an AI assistant or autonomous agent reports that its changes are secure or functional.

This activity focuses on incorporating automated scanning, testing, and release authorization into CI/CD workflows to ensure AI-generated code meets established security requirements before deployment.

AI-Generated Code Validation

Objective: Ensure AI-generated code and configuration changes are independently tested, reviewed, and validated for security vulnerabilities, vulnerable dependencies, exposed secrets, and unintended functionality before software is released.

The NIST Secure Software Development Framework (SSDF) establishes practices for reviewing, analyzing, and testing software to identify vulnerabilities before release.

The NIST AI SSDF Profile (SP 800-218A) extends secure development guidance to address AI-related development considerations.

For DevOps teams, automated CI/CD validation provides an independent security control between AI-generated changes and production deployment.

Security gates should prevent code from progressing when required validation fails. Human review or authorized policy decisions should determine whether changes are accepted, rejected, or returned for remediation.

AI Security Requirements and Open-Source Tooling

Framework Reference Security Activity CI/CD Implementation Suggested Open-Source Tools
NIST SSDF PW.7 Static code security analysis Run SAST against AI-generated code to identify insecure coding patterns and potential vulnerabilities. Semgrep, Bandit
NIST SSDF PW.4, PW.7 Dependency vulnerability scanning Evaluate dependencies introduced or modified by AI-generated code against known vulnerability databases. OSV-Scanner, Trivy
NIST SSDF PW.5, PW.7 Secrets detection Scan AI-generated source code and configuration files for exposed credentials, tokens, and API keys. Gitleaks, detect-secrets
NIST SSDF PW.8 Automated security and regression testing Execute unit, integration, regression, and applicable security tests against AI-generated changes. pytest, OWASP ZAP
NIST SSDF PW.8 Dependency compatibility validation Build and test applications after AI-generated dependency upgrades to detect compatibility failures and regressions. Jenkins, Tekton
NIST SSDF PO.1, PW.7 Security policy enforcement Apply automated acceptance criteria and prevent changes from proceeding when required security checks fail. Open Policy Agent, Conftest
NIST SSDF PS.2 Artifact integrity Ensure release artifacts are produced from approved changes and verify applicable artifact signatures. Sigstore Cosign, in-toto
NIST SSDF PW.7, RV.2 Failed validation and remediation tracking Record failed security checks, rejected changes, and remediation activities for review and resolution. DefectDojo, Jenkins

These mappings apply established secure software development practices to AI-generated changes. AI-generated code remains subject to the same security requirements as human-written code.

Expected Security Evidence

  • Automated static application security testing reports.
  • Dependency vulnerability scanning results.
  • Secrets detection and remediation records.
  • Unit, integration, and regression test results.
  • Build and dependency compatibility validation logs.
  • Records of rejected or failed AI-generated changes.
  • Code review and release approval records.
  • Security policy enforcement results.
  • Artifact verification and provenance records.
  • Traceable pull requests linking AI-generated changes to validation results.

These artifacts demonstrate that AI-generated changes have been independently validated before entering production.

Security findings should be resolved, rejected, or formally accepted through an authorized risk management process before deployment.

Additional Resources

2 - AI Model and Artifact Integrity

Verifying AI model provenance, signatures, artifact integrity, and trusted model distribution throughout the CI/CD pipeline.

AI Model and Artifact Integrity - Phase 2: Build and Deploy

AI-enabled applications increasingly incorporate pretrained models, fine-tuned models, embeddings, model adapters, and supporting software artifacts obtained from external repositories.

These artifacts introduce supply chain security risks, including compromised models, unauthorized modifications, unsafe serialization formats, and untrusted dependencies.

During Build and Deploy, organizations should verify that AI models and supporting artifacts originate from approved sources and have not been modified unexpectedly.

This activity extends established software supply chain security practices to AI models, associated dependencies, and release artifacts.

AI Model and Artifact Integrity

Objective: Ensure AI models and supporting software artifacts are verified for provenance, integrity, authenticity, and known security risks before packaging or deployment.

The NIST Secure Software Development Framework (SSDF) establishes practices for protecting software artifacts and verifying software integrity.

The NIST AI SSDF Profile (SP 800-218A) addresses additional security considerations associated with AI model development and integration.

The OpenSSF Model Signing project provides open-source capabilities for signing and verifying AI model artifacts.

For DevOps teams, model verification should become an automated CI/CD activity similar to container signature verification, dependency scanning, and software artifact validation.

Model signatures and checksums help establish artifact integrity, but do not independently prove that a model is free of malicious behavior or vulnerabilities. Additional security assessments remain necessary.

AI Security Requirements and Open-Source Tooling

Framework Reference Security Activity CI/CD Implementation Suggested Open-Source Tools
NIST SSDF PS.2; OWASP LLM03 Model provenance verification Verify model origin, approved distribution source, version, and associated provenance records before use. OpenSSF Model Signing, in-toto
NIST SSDF PS.2 Cryptographic model integrity Validate cryptographic hashes or signatures to detect unauthorized modifications to model artifacts. OpenSSF Model Signing, Sigstore Cosign
NIST SSDF PW.4; OWASP LLM03 Model file security Scan supported AI model formats for potentially malicious serialized content and unsafe loading mechanisms. ModelScan
NIST SSDF PW.4, PW.7 Model dependency vulnerability analysis Scan frameworks, libraries, containers, and supporting software components for known vulnerabilities. Trivy, OSV-Scanner
NIST SSDF PS.3 Model and dependency inventory Record model identifiers, versions, approved sources, and supporting software dependencies. Syft, CycloneDX
NIST SSDF PS.2, PS.3 Model version traceability Associate model versions and approved artifacts with application release records. DVC, Git LFS
NIST SSDF PS.2 Build provenance and attestations Generate and preserve provenance records identifying how artifacts were built and verified. SLSA, in-toto
NIST SSDF PO.1, PS.2 Artifact admission policies Prevent unapproved or unverified model and software artifacts from proceeding to deployment. Kyverno, Open Policy Agent

Artifact integrity and model security are related but distinct. Signature verification establishes properties of the artifact and its origin, while behavioral evaluations and vulnerability scanning assess different security risks.

Expected Security Evidence

  • Inventory of approved AI models and artifact versions.
  • Model provenance and source verification records.
  • Model checksum and signature verification results.
  • Model file security scanning reports.
  • Software dependency vulnerability reports.
  • Model and software component inventories.
  • Build provenance and release attestations.
  • Records of approved and rejected model artifacts.
  • Deployment policy enforcement results.
  • Traceable relationships between model artifacts and application releases.

These artifacts provide evidence that AI model integrity and provenance are evaluated before deployment.

Verified model records should be retained for post-deployment inventory management, vulnerability analysis, and incident investigation.

Additional Resources

3 - AI Security Testing and Red Teaming

Evaluating AI applications for prompt injection, sensitive data disclosure, excessive agency, unsafe outputs, and adversarial security risks.

AI Security Testing and Red Teaming - Phase 2: Build and Deploy

AI-enabled applications introduce security risks that traditional application security testing may not fully identify.

Large language models (LLMs), retrieval-augmented generation (RAG) systems, and autonomous AI agents process dynamic inputs, interact with external information, and may invoke tools or execute actions.

These capabilities create attack surfaces involving prompt injection, sensitive information disclosure, unauthorized tool execution, insecure output handling, and excessive agency.

During Build and Deploy, organizations should incorporate AI-specific security evaluations and adversarial testing into CI/CD workflows.

This activity focuses on identifying unsafe AI application behavior before production deployment and establishing repeatable security testing practices.

AI Security Testing and Red Teaming

Objective: Identify AI-specific vulnerabilities and unsafe behavior through automated security evaluations, adversarial testing, and independently enforced release acceptance criteria.

The NIST Secure Software Development Framework (SSDF) establishes security testing and vulnerability identification practices for software releases.

The NIST AI Risk Management Framework emphasizes evaluating and managing risks throughout the AI system lifecycle.

The OWASP Top 10 for LLM Applications identifies common AI application security risks, including prompt injection, sensitive information disclosure, improper output handling, and excessive agency.

For DevOps teams, AI security evaluations should be automated wherever practical and incorporated into existing build and release validation processes.

Security testing should evaluate not only the model’s responses but also whether the application enforces access controls, protects sensitive information, and prevents unauthorized tool actions.

Because AI behavior may be nondeterministic, teams should define repeatable test scenarios, execution parameters, evaluation criteria, and acceptable failure thresholds.

AI Security Requirements and Open-Source Tooling

Framework Reference Security Activity CI/CD Implementation Suggested Open-Source Tools
OWASP LLM01 Prompt injection testing Test whether malicious direct or indirect prompts can override instructions or cause unauthorized application behavior. Promptfoo, Garak
OWASP LLM02 Sensitive information disclosure testing Evaluate whether AI responses expose confidential data, credentials, personal information, or restricted context. Promptfoo, Microsoft Presidio
OWASP LLM05 Improper output handling Test whether AI-generated content can trigger unsafe downstream operations, including injection or command execution. OWASP ZAP, Semgrep
OWASP LLM06 Excessive agency testing Evaluate whether AI agents can invoke unauthorized tools, exceed assigned permissions, or perform unapproved actions. Inspect AI, Promptfoo
OWASP LLM08 RAG and retrieval security testing Test retrieval workflows for untrusted content injection, unauthorized access, and manipulation of retrieved information. Garak, Promptfoo
OWASP LLM10 Resource consumption testing Assess whether requests can cause excessive model execution, resource exhaustion, or uncontrolled service usage. k6, Locust
NIST SSDF PW.8; NIST AI RMF MEASURE Automated security evaluation Execute predefined adversarial evaluations and retain measurable security testing results. Inspect AI, Promptfoo
NIST SSDF PW.8, RV.2 Security regression testing Reexecute security evaluations after changes to models, prompts, tools, dependencies, or application logic. Jenkins, Tekton
NIST SSDF PO.1, PW.8 Security release gates Apply measurable security acceptance criteria and prevent releases when required AI security evaluations fail. Open Policy Agent, Conftest

AI security testing tools have different capabilities and coverage. No single testing tool can establish that an AI application is free of vulnerabilities or unsafe behaviors.

Expected Security Evidence

  • AI application security test plans and acceptance criteria.
  • Prompt injection testing results.
  • Sensitive information disclosure evaluation reports.
  • Agent permission and tool execution test results.
  • RAG and retrieval security testing records.
  • Resource consumption and abuse testing results.
  • Security regression test results.
  • Records of identified vulnerabilities and remediation activities.
  • Automated security gate results.
  • Approved exceptions and documented residual risks.
  • Traceable AI evaluation results associated with application releases.

These artifacts demonstrate that AI applications have undergone defined security evaluations before deployment.

Security testing should be repeated when significant changes occur to models, prompts, agent permissions, retrieval sources, or connected tools.

Additional Resources

4 - Secure AI Deployment

Deploying AI applications and agents with least-privilege access, secure configurations, secrets protection, workload isolation, and release authorization.

Secure AI Deployment - Phase 2: Build and Deploy

AI-enabled applications often depend on models, external APIs, vector databases, retrieval services, and autonomous agents that interact with other applications and infrastructure.

These integrations introduce security risks when deployed applications or agents receive excessive permissions, unrestricted network access, sensitive credentials, or unauthorized tool capabilities.

During Build and Deploy, organizations should ensure AI applications and autonomous agents are deployed within controlled environments that enforce access policies, workload isolation, secure configuration, and release authorization.

This activity extends established CI/CD deployment security practices to address the operational capabilities and security requirements of AI-enabled applications.

Secure AI Deployment

Objective: Ensure AI applications, models, and autonomous agents are deployed using verified artifacts, controlled identities, least-privilege permissions, protected credentials, and independently enforced security policies.

The NIST Secure Software Development Framework (SSDF) establishes practices for protecting software development environments and ensuring the integrity of software releases.

The OWASP Top 10 for LLM Applications identifies excessive agency, prompt injection, and improper output handling as risks that can affect deployed AI applications.

For DevOps teams, secure deployment should ensure that AI services and agents operate only within approved access and execution boundaries.

AI-generated instructions must not independently grant permissions or authorize privileged operations. Access decisions should be enforced by trusted application, identity, and infrastructure controls.

AI Security Requirements and Open-Source Tooling

Framework Reference Security Activity CI/CD Implementation Suggested Open-Source Tools
NIST SSDF PO.5; OWASP LLM06 Workload identity and authentication Assign controlled workload identities to AI applications and agents and authenticate access to protected services. SPIRE, Keycloak
NIST SSDF PO.5; OWASP LLM06 Least-privilege authorization Restrict access to repositories, databases, APIs, tools, and infrastructure based on approved policies. Open Policy Agent, Casbin
NIST SSDF PO.5 Secrets management Protect credentials and API keys through secure configuration and controlled secrets distribution. SOPS, External Secrets Operator
NIST SSDF PO.5 Workload isolation Deploy AI workloads in controlled environments with restricted execution privileges and resource boundaries. Kubernetes, gVisor
OWASP LLM06 Agent tool authorization Enforce restrictions on the tools, commands, and external operations that agents can invoke. Open Policy Agent, Kyverno
NIST SSDF PS.2 Deployment artifact verification Verify approved container images and applicable artifact signatures before deployment. Sigstore Cosign, Kyverno
NIST SSDF PO.5 Network access restrictions Restrict connections between AI workloads, internal resources, and external services. Cilium, Kubernetes NetworkPolicy
NIST SSDF PO.1, PS.2 Deployment authorization Apply release approval policies and prevent deployment of unauthorized or unverified AI components. Open Policy Agent, Argo CD
NIST SSDF PO.3, PS.1 Deployment traceability Record deployed versions, configuration changes, approvals, and release history. Argo CD, Flux

These controls apply established software security and infrastructure practices to AI-enabled applications. Deployment authorization and infrastructure security controls should operate independently of model-generated decisions.

Expected Security Evidence

  • Approved AI deployment configurations.
  • Workload identity and authentication records.
  • Agent permission and authorization policies.
  • Secrets management and credential access configurations.
  • Workload isolation and network security policies.
  • Verified container and artifact signatures.
  • Deployment policy enforcement records.
  • Release approval and change authorization evidence.
  • Deployment configuration and version history.
  • Records of rejected or unauthorized deployment attempts.

These artifacts provide evidence that AI applications and agents are deployed within controlled environments and operate under defined security restrictions.

Deployment evidence should be retained for continuous security monitoring, vulnerability management, and incident response during Phase 3: Post Deploy.

Additional Resources

5 - AI Release Evidence and Inventory

Maintaining traceable inventories of AI models, prompts, dependencies, configurations, security evaluations, and deployment evidence.

AI Release Evidence and Inventory - Phase 2: Build and Deploy

AI-enabled software releases may include more than traditional source code, libraries, and container images.

Applications can depend on AI models, prompts, agent configurations, retrieval components, external services, and supporting datasets that influence their behavior.

Without an accurate inventory of these components, organizations may struggle to identify which AI assets are deployed, determine which systems are affected by security findings, or reproduce the configuration of a previous release.

During Build and Deploy, organizations should create and retain inventories and security evidence that associate AI components with their application releases and deployment records.

This activity focuses on establishing traceability across the AI software supply chain and ensuring security evidence remains available after deployment.

AI Release Evidence and Inventory

Objective: Maintain accurate, traceable records of AI models, prompts, agent configurations, software dependencies, security evaluations, and deployment artifacts associated with each software release.

The NIST Secure Software Development Framework (SSDF) establishes practices for preserving software integrity, tracking software components, and generating evidence associated with secure development.

The NIST AI SSDF Profile (SP 800-218A) extends secure development guidance to address AI model development and lifecycle considerations.

For DevOps teams, release inventories should extend beyond traditional Software Bills of Materials (SBOMs) to include relevant AI assets and their relationships to deployed applications.

An SBOM identifies software components and dependencies, but may not fully capture models, datasets, prompts, and agent configurations. Additional AI-specific inventories or metadata may therefore be necessary.

The resulting release evidence should enable teams to determine what was built, what was tested, what was approved, and what was deployed.

AI Security Requirements and Open-Source Tooling

Framework Reference Security Activity CI/CD Implementation Suggested Open-Source Tools
NIST SSDF PS.3 Software component inventory Generate SBOMs identifying application packages, libraries, containers, and supporting software dependencies. Syft, CycloneDX
NIST SP 800-218A; NIST SSDF PS.3 AI model inventory Record model identifiers, versions, approved sources, artifact hashes, and relationships to application releases. DVC, MLflow
NIST SSDF PS.1, PS.3 Prompt and agent configuration tracking Associate approved prompt versions, agent instructions, tool definitions, and configuration revisions with releases. Git, DVC
NIST SSDF PS.3 AI dependency relationships Document relationships among AI models, application components, software packages, and supporting artifacts. CycloneDX, GUAC
NIST SSDF PS.2 Artifact integrity evidence Record cryptographic verification results and signatures for approved release artifacts. Sigstore Cosign, OpenSSF Model Signing
NIST SSDF PS.2, PS.3 Build provenance Generate attestations identifying trusted build processes, source revisions, and resulting release artifacts. SLSA, in-toto
NIST SSDF PW.8 AI security evaluation evidence Preserve test cases, AI security evaluation results, identified risks, and release acceptance decisions. Promptfoo, Inspect AI
NIST SSDF PO.3, PS.3 Deployment evidence Record approved application versions, deployment configurations, release identifiers, and target environments. Argo CD, Flux
NIST SSDF RV.1, RV.2 Vulnerability traceability Associate identified vulnerabilities and remediation records with relevant software components and releases. OSV, GUAC, Ortelius

Release inventories may require multiple complementary formats and tools. SBOMs, model metadata, provenance attestations, security evaluation results, and deployment records serve different purposes and should be linked using stable identifiers wherever practical.

Expected Security Evidence

  • Software Bills of Materials for applicable software artifacts.
  • AI model inventories identifying model names, versions, and sources.
  • Model integrity verification records.
  • Approved prompt and agent configuration versions.
  • AI dependency and component relationship records.
  • Build provenance and artifact attestations.
  • AI security testing and evaluation reports.
  • Vulnerability scanning and remediation records.
  • Release approvals and authorization decisions.
  • Deployment records identifying target environments and application versions.
  • Traceable links between AI artifacts, application releases, and deployed systems.
  • Retained security evidence supporting investigation, audits, and vulnerability management.

These artifacts provide a record of the software and AI components included in each release and the security controls applied before deployment.

Release inventories should remain available after deployment so newly discovered vulnerabilities can be correlated with affected software components, AI models, applications, and environments.

This information provides the foundation for continuous vulnerability management, affected-system identification, and incident response during Phase 3: Post Deploy.

Additional Resources