Back to overview
Degraded

Upscaling not working

Aug 07 at 08:00am EDT
Affected services
Heroku autoscaling
Render autoscaling
Amazon ECS autoscaling
Fly.io autoscaling
Railway autoscaling

Resolved
Aug 07 at 09:52am EDT

Upscaling is working as expected. We'll be writing up a full report of what happened and what we're doing about it, but here's a summary:

At 2025-08-07 00:25:00 UTC we ingested some unexpected queue metrics for a far-future date. It looks like a customer triggered this from their laptop. Our upscale logic looks at the most recent metrics bucket as a boundary, so this data broke our upscaling.

We have many automated checks in place, but none caught this, and unfortunately it happened overnight for us, so we weren't aware of the issue until this morning around 2025-08-07 12:00:00 UTC.

We identified the bad data and deleted it at 2025-08-07 12:36:00 UTC. Upscaling immediately started working as expected.

We're implementing a permanent fix now, as well as automated checks so that we're alerted immediately in the future. Check our blog later today for a more thorough write-up.

Updated
Aug 07 at 08:43am EDT

We've identified the potential cause of the issue and applied a fix, upscaling should be working now, we'll continue to monitor.

Created
Aug 07 at 08:00am EDT

We've received some reports of upscaling not triggering, and are investigating.