Code and Prebuild

• 1: AI-Assisted Code Development
• 2: AI Agent Identity and Permissions
• 3: AI Application Threat Modeling
• 4: AI Model and Data Supply Chain
• 5: Prompt and Agent Configuration Security

1 - AI-Assisted Code Development

AI-Assisted Code Development - Code and Prebuild

Securing AI-Generated Code During Software Development

Artificial intelligence is increasingly used to generate application code, recommend dependencies, create configuration files, and automate software development tasks. While AI-assisted development can accelerate delivery, generated code may introduce security vulnerabilities, insecure coding patterns, outdated dependencies, exposed credentials, or licensing concerns.

During Code and Prebuild, organizations should apply established secure development practices to all AI-generated code and changes. AI coding assistants and autonomous development agents should operate within defined security boundaries, and their output should be subject to the same security requirements as human-written code.

This activity focuses on identifying and preventing security vulnerabilities before AI-generated changes enter trusted source code repositories or proceed through the CI/CD build pipeline.

The following activities map NIST Secure Software Development Framework (SSDF), the AI SSDF Profile (NIST SP 800-218A), and OWASP security guidance to actionable CI/CD practices and suggested open-source tools. The tools are illustrative and are not mandated by these frameworks.

AI-Assisted Code Development

Objective: Ensure AI-generated code is reviewed, tested, and validated for security vulnerabilities, unsafe dependencies, exposed secrets, and licensing risks before being incorporated into the software delivery pipeline.

The NIST Secure Software Development Framework (SSDF) establishes secure coding, code review, vulnerability analysis, and software verification practices that apply to both human-written and AI-generated software.

NIST SP 800-218A, the AI SSDF Profile, extends these practices to address additional security considerations associated with AI development.

AI-generated code should not be treated as inherently secure or trusted simply because it was produced by an automated system. Development teams remain responsible for verifying that generated code meets their security requirements.

For DevOps teams, AI-generated changes should be incorporated into existing source code management, peer review, static analysis, dependency scanning, and vulnerability management workflows. Automated security checks should identify potential weaknesses before code is merged, while policies should prevent AI agents from bypassing required reviews or security gates.

Particular attention should be given to generated dependencies, insecure API usage, authentication and authorization logic, secrets handling, and changes to security-sensitive configurations.

AI-assisted development should accelerate coding activities without weakening established secure software development controls.

AI Security Requirements and Open-Source Tooling

Framework references identify relevant security practices. Tool selection and implementation depend on the organization’s development environment and security requirements.

Expected Security Evidence

• Documented secure coding requirements applicable to AI-generated code.
• Records of code reviews and approvals for AI-generated changes.
• Static application security testing results.
• Dependency vulnerability scanning reports.
• Secrets detection results and remediation records.
• Automated test execution and validation results.
• Software component inventories identifying introduced dependencies.
• License analysis reports for applicable third-party software.
• Records of security policy enforcement and rejected changes.
• Traceable pull requests linking generated changes to review and validation outcomes.

These artifacts provide evidence that AI-assisted development follows established secure software development practices and that generated code is evaluated before entering the build and deployment process.

Security findings identified during Phase 1 should be resolved or appropriately documented before the affected changes proceed to Phase 2: Build and Deploy.

Additional Resources

• NIST Secure Software Development Framework (SP 800-218)
• NIST AI SSDF Profile (SP 800-218A)
• OWASP Top 10 for LLM Applications
• OpenSSF Best Practices
• OpenSSF Security Tooling

2 - AI Agent Identity and Permissions

AI Agent Identity and Permissions - Phase 1: Code and Prebuild

Artificial intelligence agents are increasingly used to generate code, modify repositories, update dependencies, execute commands, and automate software development activities. Unlike traditional coding assistants, autonomous agents may perform actions directly within development environments and interact with external tools and services.

These capabilities introduce security risks when agents are granted excessive permissions, unrestricted tool access, or credentials that provide access to sensitive resources.

During Code and Prebuild, organizations should establish identity, authentication, authorization, and access control requirements for AI agents before they are integrated into development workflows.

This phase focuses on preventing unauthorized agent activity, protecting credentials, and ensuring that AI agents operate within explicitly defined security boundaries.

The following activities map NIST Secure Software Development Framework (SSDF) practices and OWASP AI security guidance to actionable CI/CD practices and suggested open-source tools. The tools are illustrative and are not mandated by these frameworks.

AI Agent Identity and Permissions

Objective: Ensure autonomous AI agents operate with verified identities, least-privilege permissions, controlled access to development resources, and appropriate authorization for security-sensitive actions.

The NIST Secure Software Development Framework (SSDF) emphasizes protecting development environments, securing software development processes, and preventing unauthorized changes to software artifacts.

The OWASP Top 10 for LLM Applications identifies excessive agency as a risk when AI systems are granted unnecessary functionality, permissions, or autonomy.

For DevOps teams, AI agent access should be treated similarly to other automated service identities. Agents should use dedicated identities where supported, receive only the permissions required for their assigned tasks, and operate within controlled execution environments.

Agent-generated instructions should never replace independent access control decisions. Permissions must be enforced by repository platforms, identity services, execution environments, and other trusted systems.

AI Security Requirements and Open-Source Tooling

Framework references identify relevant security practices and risk categories. Specific agent access controls are implementation recommendations rather than requirements explicitly prescribed for agents by every referenced framework.

Expected Security Evidence

• Documented AI agent identities and assigned responsibilities.
• Defined access control policies for repositories, tools, and services.
• Records demonstrating least-privilege permissions.
• Credential management and secrets protection configurations.
• Approved agent tool and command allowlists.
• Security policies for agent execution environments.
• Records of required reviews and approvals for agent-generated changes.
• Agent activity logs and security-relevant execution records.
• Evidence of authentication and authorization policy enforcement.

These artifacts provide evidence that autonomous AI agents are operating within controlled security boundaries and that development teams retain accountability for agent-initiated actions.

Agent security requirements established during Phase 1 should be validated during Phase 2: Build and Deploy and monitored throughout Phase 3: Post Deploy.

Additional Resources

• NIST Secure Software Development Framework
• NIST AI SSDF Profile
• OWASP Top 10 for LLM Applications
• OWASP Agentic AI Security
• SPIFFE and SPIRE

3 - AI Application Threat Modeling

AI Application Threat Modeling - Phase 1: Code and Prebuild

Artificial intelligence applications introduce security risks that extend beyond those found in traditional software architectures. Applications incorporating large language models (LLMs), retrieval-augmented generation (RAG), autonomous agents, external data sources, and tool integrations create new attack surfaces and trust relationships.

These systems may process untrusted prompts, retrieve external content, invoke privileged tools, and generate outputs that influence application behavior.

During Code and Prebuild, organizations should perform AI-specific threat modeling to identify potential vulnerabilities before application components are implemented or integrated into the CI/CD pipeline.

This phase focuses on identifying attack paths, evaluating security boundaries, and establishing controls to prevent unauthorized model behavior, sensitive information disclosure, and unsafe agent actions.

The following activities map NIST security guidance and OWASP AI risk categories to actionable threat modeling practices and suggested open-source tools.

AI Application Threat Modeling

Objective: Identify and document AI-specific attack surfaces, trust boundaries, potential threats, and appropriate security controls before developing or deploying AI-enabled applications.

The NIST Secure Software Development Framework (SSDF) emphasizes integrating security requirements and risk analysis into software design and development activities.

The NIST AI Risk Management Framework provides guidance for identifying, measuring, and managing risks associated with AI systems.

The OWASP Top 10 for LLM Applications identifies risks including prompt injection, sensitive information disclosure, supply chain vulnerabilities, improper output handling, and excessive agency.

For DevOps teams, threat modeling should be integrated into application architecture reviews, development planning, and security requirements definition.

AI components should be evaluated as part of the complete application architecture rather than as isolated models.

Threat models should identify how users, models, agents, retrieved information, and external services interact and where trust boundaries must be enforced.

AI Security Requirements and Open-Source Tooling

Threat modeling tools support risk identification and documentation. Testing tools such as Promptfoo and Garak can help validate selected threats but do not replace architectural threat modeling.

Expected Security Evidence

• Documented AI application threat models.
• Architecture diagrams identifying AI components and external services.
• Data flow diagrams showing model inputs, outputs, and tool interactions.
• Identified trust boundaries and security-sensitive integration points.
• Prompt injection and agent misuse risk assessments.
• Sensitive information exposure assessments.
• Documented AI security risks and assigned risk owners.
• Security requirements and mitigation plans derived from identified threats.
• Records of architecture reviews and threat model updates.

These artifacts provide evidence that AI-specific security risks were considered during application planning and design.

Threat models developed during Phase 1 should inform the automated AI security testing and release validation activities performed during Phase 2: Build and Deploy.

Additional Resources

• NIST Secure Software Development Framework
• NIST AI SSDF Profile
• NIST AI Risk Management Framework
• OWASP Top 10 for LLM Applications
• OWASP Threat Dragon
• MITRE ATLAS

4 - AI Model and Data Supply Chain

AI Model and Data Supply Chain - Phase 1: Code and Prebuild

AI-enabled applications increasingly depend on externally sourced models, training datasets, machine learning frameworks, model repositories, and third-party software components.

These dependencies introduce security risks beyond traditional software package management. AI models may originate from untrusted sources, datasets may contain malicious or manipulated content, and model files may include unsafe serialization formats or dependencies.

During Code and Prebuild, organizations should establish processes for evaluating the provenance, integrity, licensing, and security of AI models and data before incorporating them into applications.

This phase focuses on identifying supply chain risks early, maintaining visibility into AI dependencies, and establishing trusted sources for AI components.

The following activities map NIST Secure Software Development Framework (SSDF), the AI SSDF Profile, and OWASP AI security guidance to actionable CI/CD practices and suggested open-source tools.

AI Model and Data Supply Chain

Objective: Ensure AI models, datasets, frameworks, and third-party dependencies are evaluated for provenance, integrity, vulnerabilities, licensing, and supply chain risks before entering the software delivery pipeline.

The NIST Secure Software Development Framework (SSDF) establishes practices for identifying, evaluating, and securing third-party software components.

The NIST AI SSDF Profile (SP 800-218A) extends secure development guidance to address additional security considerations associated with AI model development and integration.

The OWASP Top 10 for LLM Applications identifies AI supply chain vulnerabilities and data and model poisoning as significant risks.

For DevOps teams, AI model and data dependencies should be evaluated before being introduced into trusted development environments.

This includes verifying component sources, reviewing applicable licenses, identifying known vulnerabilities, and documenting dependencies between models, datasets, frameworks, and application components.

Organizations should also establish requirements for maintaining accurate inventories of AI assets so they can be tracked throughout the build, deployment, and operational lifecycle.

AI Security Requirements and Open-Source Tooling

These tools address different aspects of AI supply chain security. Traditional SBOM tools do not necessarily capture complete model, dataset, or prompt inventories. Model provenance, dataset security, and licensing may require additional records and specialized validation.

Expected Security Evidence

• Inventory of approved AI models and associated versions.
• Records identifying model sources, repositories, and provenance.
• Model integrity verification results and available signatures.
• Dataset source records and integrity assessments.
• Documented assessments of data and model poisoning risks.
• Software dependency vulnerability scanning results.
• Model file security scan results.
• Third-party component license and usage reviews.
• AI component inventories and applicable SBOMs.
• Records of rejected or unapproved AI components.
• Traceable relationships between models, datasets, dependencies, and application development artifacts.

These artifacts provide evidence that AI models and supporting dependencies have been evaluated before entering the software delivery process.

Supply chain records created during Phase 1 should support model verification and artifact integrity controls in Phase 2: Build and Deploy, as well as continuous vulnerability management during Phase 3: Post Deploy.

Additional Resources

• NIST Secure Software Development Framework
• NIST AI SSDF Profile
• OWASP Top 10 for LLM Applications
• OpenSSF Model Signing
• CycloneDX
• OpenSSF

5 - Prompt and Agent Configuration Security

Prompt and Agent Configuration Security - Phase 1: Code and Prebuild

AI-enabled applications rely on system prompts, agent instructions, tool definitions, and configuration files to determine how models interact with users, applications, and external services.

These artifacts can directly influence application behavior, data access, tool execution, and security boundaries.

Unlike traditional source code, prompts and agent configurations may be modified independently of application binaries. Unauthorized or poorly reviewed changes can introduce security risks even when the underlying application code remains unchanged.

During Code and Prebuild, organizations should treat security-sensitive prompts, agent instructions, and configurations as managed development artifacts.

This phase focuses on protecting AI configurations against unauthorized modification, establishing secure review processes, and ensuring that agent capabilities remain aligned with approved security policies.

The following activities map NIST security guidance and OWASP AI risk categories to actionable CI/CD practices and suggested open-source tools.

Prompt and Agent Configuration Security

Objective: Ensure system prompts, agent instructions, tool configurations, and AI application settings are version-controlled, reviewed, protected against unauthorized changes, and aligned with established security requirements.

The NIST Secure Software Development Framework (SSDF) establishes practices for protecting software artifacts, controlling changes, and integrating security requirements into development workflows.

The OWASP Top 10 for LLM Applications identifies prompt injection, system prompt leakage, and excessive agency as security concerns affecting AI-enabled applications.

For DevOps teams, prompts and agent configurations should be managed using established source control and change management practices.

Security-sensitive changes should be reviewed before acceptance, and automated policies should prevent unauthorized modifications to agent permissions, tool access, and execution settings.

System prompts should not be treated as a security boundary. Authorization must be enforced by trusted application and infrastructure controls independently of model instructions.

AI Security Requirements and Open-Source Tooling

The referenced frameworks provide relevant software security practices and AI risk categories. The implementation recommendations translate those practices into CI/CD controls for prompts, agent instructions, and configuration artifacts.

Expected Security Evidence

• Version-controlled system prompts and agent instructions.
• Approved tool definitions and agent configuration records.
• Security review and approval records for configuration changes.
• Documented policies restricting agent permissions and tool access.
• Configuration validation and policy enforcement results.
• Secrets scanning results for prompts and configuration files.
• Prompt injection testing results and identified security findings.
• Records of unauthorized or rejected configuration modifications.
• Traceable configuration versions associated with application changes.
• Documented procedures for updating and restoring approved AI configurations.

These artifacts provide evidence that AI instructions and configurations are managed through controlled development processes.

Configuration controls established during Phase 1 should support security testing and release verification during Phase 2: Build and Deploy.

After deployment, changes to AI instructions, tools, and permissions should remain subject to authorization, monitoring, and audit requirements.

Additional Resources

• NIST Secure Software Development Framework
• NIST AI SSDF Profile
• OWASP Top 10 for LLM Applications
• OWASP Generative AI Security Project
• Open Policy Agent
• Promptfoo