Logo
UrbanPiper – Status
Operational
Updated: IST
COMPONENTS
MAINTENANCE
ISSUE HISTORY
Components
Atlas Portal
Operational
icon
Hub
icon
Prime
icon
Meraki
icon
POS Integration
icon
Auth Service
Operational
icon
Maintenance
No upcoming or ongoing maintenances reported
Issue History
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
Prime - Increased latency
Investigating

Several users have reported an increase in the latency, the team is checking the issue.

IST

Several users on a specific network have been affected by this issue. We are receiving multiple reports concerning users on Jio network. Please see the following link for more information: https://www.timesnownews.com/technology-science/jio-is-down-thousands-of-users-facing-issues-with-the-network-article-111081514

IST
Identified

We have received multiple reports of issues with the Jio network, affecting only users on this network. If you are experiencing this problem, please switch to a different network temporarily.

IST
Monitoring

We are noticing improved latency for users on the Jio network. If you still encounter issues, we recommend switching to another network. Our systems are performing optimally, and we are actively monitoring the situation.

IST
Resolved

The issue has been resolved. Thank you for your patience.

IST
No issues reported.
No issues reported.
No issues reported.
Taxes and PC not updating to Menu ingestion service
Investigating

We are currently experiencing an issue with one of our databases, which has affected the menu ingestion service. Our team is actively investigating the matter. We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve the issue.

IST
Identified

Due to a technical issue, menu ingestion has been temporarily disabled.

IST
Monitoring

The menu ingestion service has been recovered and is now re-enabled. Thank you for your patience during this time. Our team will continue to monitor the service to ensure its stability.

IST
Resolved

The issue has been resolved, and the team will continue to monitor the situation.
..........................................................
RCA

At 1200 hrs on 14th June, our teams were alerted about failures in processing of taxes & charges during menu ingestion. The resolution team was engaged immediately and identified that failures were related to a DB table used for storing menu data related to taxes & charges. This table had reached the maximum integer range of its primary index. Due to this, the taxes & charges data in the new ingestion request payloads were not being processed.

The team was able to identify the issue immediately, and made the required change to the concerned DB Table, however this change took close to 3 hours to apply due to the size of the table. During this time, Team put out banners in Atlas & Prime proactively informing users that menu ingestion was temporarily suspended. Once the operation had finished, Team enabled the menu ingestions, and closely monitored the processing of taxes & charges to ensure correct functionality.

To prevent similar incidents in the future, Team has already initiated the changes required for other at risk tables. Team will also conduct regular audits of DB tables & setup alerting to ensure timely identification and mitigation of potential issues.

IST
No issues reported.
No issues reported.
Performance degradation
Investigating

We noticed some of our services running on kubernetes are not performing as intended, the team is investigating the issue.

IST

We are currently experiencing issues with the following workflows: Menu Publish, Meraki Websites, and new logins to the Atlas and Prime portals. We apologize for the inconvenience caused. Our team is actively working on resolving the issue.

IST
Monitoring

The services have recovered and are now performing optimally. Our team will continue to monitor the situation.

IST
Resolved

The issue has been resolved, and the team will continue to monitor the situation. A detailed RCA will be shared with the stakeholders.
............................................................

RCA

Our infrastructure team was alerted about a surge in costs associated with a cluster of API services a few days back. During the investigation, the team noticed an unexpected increase in the number of nodes in our cluster. To manage costs and resource allocation, the team decided to scale down the nodes. Unfortunately, one of the nodes that were scaled down hosted a critical service responsible for managing network traffic orchestration.

The termination of this critical service caused a disruption in how traffic is routed to a subset of our services. This led to immediate alerts for the services running on the cluster. While none of the critical Hub workflows such as Order Ingestion, Order Updates, Item/Modifier availability, Order tracking, etc, were affected, several other workflows experienced degradation. These included:

 - Menu Publish for merchants using MenuV2
 - Authentication for new logins on Atlas (existing logged-in users/sessions were not affected)
 - POS Billing service

At 1401 hrs IST, the team identified the root cause and reverted the changes, rolling back to the previous configuration. This action successfully mitigated the issue, and services were restored to normal.

IST
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
No issues reported.
Powered by
logo