Operators at a regional colocation facility kept seeing the same thing in their logs. Every night between 1 and 4 a.m., temperature sensors at the top of one rack row climbed past 35°C. Daytime readings were fine. Cooling units were running normally. It took two weeks to trace the cause: a tenant’s nightly data job was reprocessing three years of transaction history from scratch, pinning a full row of servers at maximum load, and the hot exhaust was spilling over the tops of the racks into the cold side.
The fix ended up having two parts. One was sheet metal. The other was code.
Where the Heat Actually Goes
Servers pull cool air in the front and push hot air out the back. Lined up in rows facing alternating directions, they create cold aisles and hot aisles. The trouble is that air doesn’t respect the plan. Hot exhaust curls over the top of racks and around the ends of rows, gets pulled back into server intakes, and the equipment in the worst spots runs hot while the cooling system works harder to compensate.
Hot aisle containment panels close off that escape route. Ceiling panels span the hot aisle, doors seal the ends, and the trapped exhaust is ducted back to the cooling units or into a return plenum above the ceiling. The rest of the room stays at a comfortable supply temperature.
The alternative, cold aisle containment, encloses the intake side instead. It’s usually cheaper to retrofit into an existing room, since it doesn’t need a return path above the ceiling. But everything outside the enclosure, including where people walk, ends up hot. For new builds and higher-density rooms, I’d choose hot aisle containment almost every time.
Before any of that, check for missing blanking panels. Empty rack slots let hot air pass straight through to the front, and a few dollars of plastic often fixes problems people blame on the cooling plant.
Density Changed the Math
A traditional enterprise rack drew 5 to 10 kilowatts. Racks full of AI training hardware can draw 40, 80, or more than 100. Air containment does well up to roughly 30 or 40 kilowatts per rack. Beyond that, operators move to rear-door heat exchangers or liquid cooling piped directly to the chips.
Containment doesn’t disappear in those rooms. Storage, networking, and older servers still run on air, and they share the floor with the liquid-cooled racks.
What the Brochures Leave Out
Fire codes come first. Containment ceilings can block sprinkler coverage, so panels are typically designed to drop away or shrink out of the way when temperatures reach a set threshold. Check this with the local fire authority before ordering anything.
Inside a contained hot aisle, temperatures can exceed 40°C, which is unpleasant and, over a long cabling job, a genuine safety concern for technicians. Plan work rotations accordingly. And every cable passing through a panel needs a brush grommet or seal, or the containment slowly leaks.
The Software Side of the Heat Problem
Back to the tenant’s nightly job. Containment fixed the recirculation, but the servers were still running flat out for four hours a night doing work that didn’t need doing. Only a few thousand records changed each day. The pipeline reloaded millions.
The tenant’s data team rebuilt it to process only new and changed records. They used Prophecy, which let them lay out the pipeline visually while generating standard code stored in version control, so engineers could review each step and spot where full-table reloads were hiding. The nightly run dropped from about four hours to under half an hour.
Rebuilding pipelines takes effort, and teams with hundreds of hand-written legacy jobs rarely convert them all at once. Starting with the longest-running jobs usually gives the biggest return, both on compute bills and on heat.
There’s a loop here too. The temperature sensors that flagged the spike feed their own data pipelines, streaming readings every few seconds into monitoring dashboards and capacity planning models.
Facilities staff and data engineers rarely sit in the same meetings. In this case, both groups could see the 2 a.m. spike from their own side for months, and each assumed it was the other team’s problem to solve.
