In Part 1, we talked about all the hidden complexity inside AI systems: the pipelines, GPUs, embeddings, vector databases, orchestration layers, and everything else that quietly determines how reliable an AI-first product really is. But all of that software still rests on something far less glamorous: the physical infrastructure underneath it. And the more time I spend around reading up on AI-first companies, the more obvious it becomes that the datacenter is often the part people forget to appreciate… until something goes wrong.
Because here’s the unpolished truth: even the most elegant AI stack in the world is completely at the mercy of real-world things like cooling, power, networking, and rack health. It’s not fun to talk about, it’s not shiny, and it’s definitely not the kind of thing you see in launch demos. But it has a bigger impact on AI reliability than most people realize. When physical infrastructure stumbles, AI workloads don’t just slow down gently. They stumble instantly, and very visibly.
AI Doesn’t Run in the Cloud. It Runs in Buildings Filled With Heat and Electricity.
AI discussions often make it sound like large models float around magically in “the cloud,” responding to prompts from some futuristic compute fabric. But in reality, AI runs inside rooms full of humming machines, densely packed GPU racks, miles of networking cables, and mechanical systems doing very unglamorous work to keep everything online. These are the things your users never see, but they feel every time something falters.
If a cooling unit underperforms for thirty minutes, GPUs begin to throttle, and suddenly your inference latency doubles, even though nothing “changed” in your code. If the power delivery to a rack becomes uneven, a few nodes might reboot, quietly throwing off your load balancing. And if the networking fabric has a minor hiccup, your distributed inference jobs may start timing out or drifting out of sync. None of these physical issues announce themselves cleanly. Instead, your AI product just feels “slow,” “weird,” or “inconsistent.” And if you ask your engineers why, you’ll probably hear, “We’re still investigating.”
The point is: physical infrastructure doesn’t just support AI. It shapes AI performance in ways that are immediate and unforgiving.
Why AI Places Stress on Datacenters Like Nothing Before
Traditional software can tolerate a surprising amount of fluctuation in heat, power, and networking. Most web apps don’t collapse because one switch gets warm or one rack loses a bit of airflow. But AI workloads are different, they’re compute-heavy, sensitive to environmental conditions, and extremely unforgiving when anything interrupts the rhythm of inference.
GPUs, for example, are notorious for their sensitivity to temperature. A few degrees of heat can cause them to throttle, which cascades directly into slower reasoning, slower streaming, and slower responses for end users. Similarly, networking issues that would barely impact a typical microservice can wreak havoc on distributed inference jobs or multi-node training runs. Even storage bandwidth matters more than people expect; when embeddings or model checkpoints can’t be fetched quickly enough, entire AI pipelines begin to wobble.
None of these problems start in the application layer. They start in the building, and show up in front of users.
Why Physical Incidents Are So Hard to Detect (and Even Harder to Interpret)
One of the challenges with datacenter-level problems is that they rarely show up as clear, actionable alerts. A cooling system doesn’t send a polite message saying, “Hi, I’m losing efficiency; please wake someone up.” Instead, your AI model starts responding slower than usual, and everyone thinks it’s a software issue. A power circuit doesn’t announce it’s about to destabilize, you just see intermittent GPU resets and wonder why.
By the time a datacenter operator or hardware engineer confirms the root cause, your application team has already spent an hour checking pipelines, gateways, vector databases, and everything else upstream. It’s not that physical systems are mysterious. It’s that their symptoms look like application-level problems long before they look like hardware failures.
With AI workloads, you see the smoke before you even realize there’s a fire.
Datacenter Failures Demand Coordinated, Real-Time Human Response
The other challenge with physical infrastructure issues is how many people they involve. A single incident might pull in infrastructure engineers, SRE, NetOps, on-site datacenter staff, hardware vendors, cloud partners, and AI platform engineers, all of whom own different parts of the stack. And because physical issues escalate quickly, there’s almost no margin for slow notifications, missed pings, or assumptions that “someone else probably saw it.”
Cooling failures don’t wait for morning standup. Power fluctuations don’t pause until Slack is checked. GPU racks don’t schedule their malfunctions around on-call rotations.
When the datacenter falters, the AI workload reacts immediately. And when that reaction hits users, which it always does, you need a reliable way to mobilize people right away. Not “whenever someone checks their messages.” Not “once the right team is tagged.” Immediately.
This Is Where OnPage Helps AI-First Teams Bridge Physical and Application Layers
The interesting thing about datacenter incidents is that the detection part is often solved. Datacenters already have environmental sensors, DCIM platforms, power monitoring, networking logs, and hardware health alerts. The problem isn’t the signal. The problem is the response…the human part.
OnPage steps in by ensuring that when any part of the physical environment begins impacting AI workloads, the right person actually gets notified. Not through an email at 2 AM. Not through a Slack message that gets buried. Not through a dashboard that someone happens to be watching. Through a high-priority mobile alert that persists until someone acknowledges it, and escalates automatically if needed. Every action along the way is logged, which makes it easier to piece together what happened once the incident is resolved.
For AI-first companies, this matters because physical failures often look like software failures until someone digs deeper. And by the time the root cause is understood, user experience has already taken a hit, and engineers have already spent valuable time chasing issues in upstream services, when the real problem lives further down the stack. A proactive, persistent, human-centered incident alerting and on-call management system makes the difference between catching a cooling issue early and discovering it only after GPUs begin throttling en masse.
If Part 1 Was About the AI Stack… Part 2 Is About Protecting the Ground Beneath It
AI reliability doesn’t end at pipelines and models. It stretches all the way down to power grids, cooling systems, rack designs, and the everyday realities of datacenter operations. AI-first teams need to treat physical infrastructure as part of their reliability strategy, not an afterthought someone else “probably has under control.”
OnPage helps close that gap by connecting the physical world to real-time human response. Because no matter how advanced your ML stack is, no matter how optimized your inference pipeline might be, and no matter how many automated safeguards you have, AI still relies on actual humans responding to the right alerts at the right moment.
And that’s what makes AI operations work in the real world — not just in the demo videos.
About The Author
