Where the Bytes Come From: A Field Audit for Randomness in Cloud, Container, and On-Prem Systems
Photo: Jemimus, CC BY 2.0, via Wikimedia Commons
Entropy is the one input in a cryptographic system that no amount of code review can inspect. A key derivation function fed by a weak source still produces output that passes test vectors, satisfies lint rules, and clears every compliance checklist — until two systems generate the same value. That is why randomness failures surface as incidents rather than findings: duplicate session tokens across tenants, colliding key identifiers, certificates that share a factor with a stranger's.
Auditing entropy means auditing provenance. The question is not whether the bytes look random. The question is where they were born, what produced them, and whether anything between the source and the consumer could have duplicated, stalled, or depleted them.
Why statistical testing proves so little
The instinct on most teams is to run a randomness test suite and call it an audit. This is a category error. Suites such as NIST SP 800-22 or Dieharder test a stream of bytes, not a system. They will happily certify the output of a deterministic PRNG seeded with a timestamp, because the output of a well-constructed deterministic generator is, by design, statistically indistinguishable from noise.
The useful signal is architectural. You want to know which interface the application calls, which kernel or hypervisor facility sits behind it, what seeds that facility at boot, and how many distinct consumers share it. Statistical testing belongs at the end of the audit, as a sanity check on a source you already understand — not at the beginning, as a substitute for understanding it.
Step one: build an entropy inventory
Start with a census. For every system that generates keys, tokens, nonces, salts, identifiers, or initialization vectors, record four facts:
- The API actually called by the code (for example, the
getrandomsyscall,/dev/urandom, a language runtime default, a hardware instruction, or a cloud key service). - The layer that satisfies that call (kernel CSPRNG, hypervisor entropy device, hardware RNG, HSM, or an external API).
- The seeding path at boot, including how early in the boot sequence the pool receives its first trusted contribution.
- The consumers downstream, especially shared libraries and frameworks that quietly call the same source on behalf of many services.
Most organizations discover at this stage that they have far fewer entropy sources than they assumed, and that languages and frameworks make the choice for them. Two services written by different teams may already share a single source with a single failure mode.
Step two: the virtualization problem
Virtual machines introduce failure modes that do not exist on physical hardware, and the most serious is state duplication. When an image is snapshotted or cloned, the guest's kernel entropy pool is captured along with everything else. Restore that snapshot on twenty hosts and you have twenty identically seeded generators. If the guest reseeds quickly from a paravirtual entropy device, the exposure is short. If it does not, the window is as long as your provisioning pipeline.
The audit question is concrete: after a clone, restore, or live migration, what forces a reseed, and how is that verified? Look for paravirtual entropy devices configured and actually used, guest agents that credit entropy correctly, and — critically — automated post-clone validation that generates a value on two hosts and confirms they differ. Some hypervisors offer snapshot-time entropy handling; confirm it is enabled rather than assumed.
Step three: fork safety and the container illusion
On Unix-like systems, a forked child inherits the parent's memory, including any userspace generator state. A userspace PRNG that does not reseed on fork will hand parent and child the same stream. Modern runtimes generally handle this, but third-party libraries, older language versions, and embedded components frequently do not. Enumerate every userspace generator in your dependency tree and check its fork behavior against the version you actually deploy.
Containers add a subtler issue. A container shares the host kernel, so it usually inherits the host's CSPRNG — which is good — but it also inherits the host's boot-time seeding history. A host that booted with a thin pool, then spent its first minutes under heavy load, will present the same weak history to every container it launches. Container isolation does not create entropy independence.
Step four: starvation under load
Entropy starvation is largely a historical concern for blocking interfaces on modern kernels, but it remains real in three places: embedded and appliance hardware with no persistent seed file, freshly provisioned fleets that boot in parallel from a common image, and systems that consume enormous numbers of random values during startup. A fleet of two hundred instances booting simultaneously from the same AMI is a collective seeding event, not two hundred independent ones.
The test is to reproduce the worst case deliberately. Provision a full fleet at once, immediately generate keys on every host, and compare. Then measure how long until each host reaches a state you would trust. If the answer depends on network availability or a blocking read, you have found a boot-order dependency worth documenting.
Step five: hardware sources and trust boundaries
Hardware random number generators are useful and are also the most over-trusted component in the chain. On-chip instruction-based generators should be treated as one input to a mixed pool, not as the sole root of trust, and firmware settings that disable the mixing deserve scrutiny. Note in your inventory which platforms rely on which vendor facilities and whether your threat model includes a compromised CPU or hypervisor. For high-assurance keys, the HSM or cloud key service is the correct boundary; the audit should confirm that generation happens inside it rather than being imported after the fact.
Step six: make it continuous
An entropy audit is a snapshot of a system that changes with every image build and base-image update. Convert the findings into controls: a check in the image pipeline that verifies the entropy configuration, a post-provisioning test that compares generated values across hosts, and an alert on any service that generates keys before its documented seeding milestone.
Randomness failures are quiet by construction. The only reliable defense is to know, in writing, where every byte comes from — and to test that knowledge before an incident tests it for you.