Key64 All articles
Security Engineering

Convenience Over Correctness: Why Cryptographic Checks Disappear Inside Modern CI/CD Pipelines

Key64
Convenience Over Correctness: Why Cryptographic Checks Disappear Inside Modern CI/CD Pipelines

Every security policy document written in the last decade has included some variant of the phrase "verify before you trust." And yet, inside the build pipelines that ship production software to millions of users, cryptographic verification is frequently the first control to be commented out, toggled off, or simply never configured in the first place. The gap between policy and practice here is not subtle — it is structural.

Understanding why this happens, and what it costs, is essential for any engineering organization that takes supply chain integrity seriously.

The Path of Least Resistance

Modern CI/CD tooling has made deployment faster than at any prior point in software history. GitHub Actions, GitLab CI, CircleCI, Jenkins, and their peers have abstracted enormous complexity away from developers. That abstraction, however, cuts both ways. When a pipeline stage fails — particularly during a release window — the immediate organizational pressure is to restore flow, not to diagnose root cause.

Signature verification failures are a common source of that friction. A GPG key expires. A checksum file is missing from an artifact repository. A container image pulled from a registry lacks a valid Notary or Sigstore signature. Each of these conditions can halt a pipeline, and each of them has a quick fix: disable the check, mark the step as non-blocking, or simply delete the verification stage entirely.

The troubling part is not that engineers make this call once under pressure. The troubling part is that the change rarely gets reverted. It becomes the new baseline.

Three Patterns That Expose Production Systems

Field observations across enterprise and mid-market environments reveal several recurring failure modes.

The orphaned verification stage. A pipeline is initially built with a checksum validation step that references a specific artifact server. When the artifact server is migrated or renamed, the verification step breaks. Rather than updating the reference, the team disables the step to meet a deadline. Months later, no one remembers it was ever there.

The inherited no-verify flag. Package managers like npm, pip, and Gradle all support flags that bypass signature or integrity checks — --no-verify, --ignore-scripts, --insecure, and their equivalents. These flags often appear in Dockerfiles or pipeline scripts inherited from earlier engineers who used them during rapid prototyping. They persist into production configurations because no one audits them systematically.

The trusted-by-default container pull. Kubernetes and container orchestration platforms will happily pull and run unsigned images if image signing policy is not explicitly enforced at the admission controller level. Many organizations assume their registry is implicitly trustworthy, particularly when using a managed service from a major cloud provider. That assumption does not account for registry compromise, credential theft, or dependency confusion attacks that substitute a malicious image for a legitimate one.

Each pattern represents a different organizational failure — deadline pressure, insufficient onboarding documentation, and misplaced platform trust, respectively. None of them require malicious intent. All of them create exploitable conditions.

What Attackers Do With the Gap

The SolarWinds compromise, the Codecov breach, and the broader pattern of software supply chain attacks documented by CISA and the NSA over the past several years share a common structural feature: adversaries inserted themselves into a trusted delivery path that was not being cryptographically verified at consumption time.

When a build pipeline pulls a dependency, an artifact, or a base image without verifying its provenance, it is extending implicit trust to whatever happens to be at that location. In a benign environment, that is the artifact the team intended. In a compromised environment — whether through a poisoned registry, a hijacked package namespace, or a man-in-the-middle attack on an unencrypted artifact feed — that is whatever the attacker placed there.

The absence of verification does not just create a theoretical risk. It eliminates one of the only technical controls capable of detecting substitution attacks before execution.

A Framework for Restoring Validation Without Breaking Velocity

The most common objection to mandatory cryptographic validation in pipelines is that it slows things down. That objection deserves to be taken seriously rather than dismissed — and then addressed directly.

Audit before you enforce. Before adding new controls, map every pipeline stage that touches external resources: package registries, container registries, artifact stores, and third-party APIs. Document which of those stages currently perform any form of integrity verification. This baseline is frequently alarming, but it is necessary.

Implement verification as a non-blocking warning first. Introducing hard failures immediately creates the same pressure that disabled verification in the first place. Start by logging verification failures without blocking the pipeline. This surfaces the scope of the problem without triggering emergency rollbacks, and it builds the organizational case for enforcement.

Adopt Sigstore for container and artifact signing. The Sigstore project — now a graduated CNCF project with backing from Google, Red Hat, and Chainguard — provides keyless signing infrastructure that significantly reduces the operational burden of managing long-lived signing keys. Cosign, Rekor, and Fulcio together enable pipeline-native artifact signing that integrates cleanly with GitHub Actions and most major CI platforms.

Enforce at the policy layer, not the pipeline layer. Admission controllers in Kubernetes — OPA Gatekeeper, Kyverno, and similar tools — can enforce image signature requirements at the cluster level, independent of individual pipeline configurations. This means a pipeline that skips signing cannot silently deploy to production, because the cluster itself will reject the unsigned workload.

Pin dependencies with lock files and verify them. For language-level dependencies, lock files (package-lock.json, Pipfile.lock, go.sum) provide a cryptographic record of expected content. Treat deviation from these files as a pipeline failure, not a warning. Renovate and Dependabot can automate lock file updates while preserving the integrity guarantee.

Rotate and audit signing keys on a defined schedule. If your organization uses long-lived signing keys, they need the same lifecycle management as TLS certificates — expiration dates, rotation procedures, and revocation paths. An expired signing key that blocks a production deployment is a process failure, not a reason to disable verification.

The Accountability Problem

One underappreciated dimension of this issue is organizational. In most engineering teams, no single person owns the security posture of the CI/CD pipeline. Developers own their application code. DevOps or platform engineers own the pipeline infrastructure. Security teams write policies. The intersection of all three — the actual verification behavior at deployment time — frequently belongs to no one in particular.

Formalizing pipeline security ownership, including cryptographic validation, as a named responsibility within either the platform engineering or security engineering function is a precondition for sustained improvement. Without accountability, even well-designed controls degrade over time.

Closing the Gap

Cryptographic validation in build pipelines is not a novel concept. The tooling is mature, the threat model is well-documented, and the regulatory pressure — from Executive Order 14028 on software supply chain security to the NIST SSDF framework — is only increasing. What remains is the organizational will to treat verification as a non-negotiable property of the delivery process rather than an optional enhancement.

The pipelines shipping your software today are either verifying what they consume or they are not. For most organizations, the honest answer to that question is more uncomfortable than it should be.

All Articles

Related Articles

Certificate Sprawl and the Outage Clock: Rebuilding Enterprise TLS Lifecycle Programs That Actually Survive

Certificate Sprawl and the Outage Clock: Rebuilding Enterprise TLS Lifecycle Programs That Actually Survive

Poisoned at the Source: The Growing Threat of Malicious Cryptographic Dependencies in Production Systems

Poisoned at the Source: The Growing Threat of Malicious Cryptographic Dependencies in Production Systems

Audit Logs You Cannot Trust: The Case for Cryptographic Integrity in Logging Pipelines

Audit Logs You Cannot Trust: The Case for Cryptographic Integrity in Logging Pipelines