Root Cause Analysis: Cache System Outage Impacting Custom POS Order Processing

Impact Window: 20:00 IST – 20:28 IST (2026-07-16)

Impact: During this window, custom POS order processing slowed down significantly due to a shared cache system outage, requiring manual routing of orders and, in some cases, order cancellation due to timeouts. This was limited to only Custom POS Integrations. No other workflow was impacted.

What Happened
An issue originating from a menu-related workflow caused a shared cache system to restart. The cache did not recover automatically afterward. This cache is used across multiple workflows and was not intended to be a single point of failure for the custom POS order flow.

Why Custom POS Order Processing Was Impacted
The cache system is not a strict dependency for the custom POS order flow. However, a misconfiguration caused the custom POS order flow to slow down significantly while the cache was unavailable, which is why the outage affected it despite the lack of a direct dependency.

Resolution
Monitoring alerted the team immediately, and investigation began within minutes. Once the root cause was identified, the team manually recovered the cache system, fully restoring service.
Preventative Actions

• Fixed the underlying query that triggered the cache system failure.

• Resolved the issue that prevented the cache from recovering automatically.

• Making the custom POS order flow more resilient to the misconfiguration identified, so future cache outages have minimal impact on order processing.

We sincerely regret the disruption caused and are committed to preventing similar incidents going forward.