Between 19:15 and 19:29 UTC, new orders were not being processed. Order status updates were affected from 19:22 to 19:29 UTC.
We deployed a change to our deployment pipeline which caused an unintended side effect of not restarting some queue workers correctly. Orders placed during this window were accepted and queued, but were not processed until those workers were restarted.
At 19:29 UTC order and order status processing was restored, and the queued orders were processed shortly afterwards.
Orders were delayed rather than lost, although some orders were cancelled during the delay.
What we are doing about it: we have removed the change responsible so it cannot happen again, and we are adding a safeguard that prevents a queue worker from being stopped unless the same deployment restarts it.
We are sorry for the disruption this caused you and your customers.
Resolved
Between 19:15 and 19:29 UTC, new orders were not being processed. Order status updates were affected from 19:22 to 19:29 UTC.
We deployed a change to our deployment pipeline which caused an unintended side effect of not restarting some queue workers correctly. Orders placed during this window were accepted and queued, but were not processed until those workers were restarted.
At 19:29 UTC order and order status processing was restored, and the queued orders were processed shortly afterwards.
Orders were delayed rather than lost, although some orders were cancelled during the delay.
What we are doing about it: we have removed the change responsible so it cannot happen again, and we are adding a safeguard that prevents a queue worker from being stopped unless the same deployment restarts it.
We are sorry for the disruption this caused you and your customers.
Investigating
Between approximately 19:15 and 19:45 UTC, some orders, status updates and webhooks were processed with delay. Individual queues were affected for roughly 15 minutes each.
The cause was an internal deployment configuration issue and has been corrected. Processing capacity is fully restored and the backlog is clearing.
All queued work is being processed.
Next update within 30 minutes.
