Upscaling not working
Resolved
Aug 07 at 09:52am EDT
Upscaling is working as expected. We'll be writing up a full report of what happened and what we're doing about it, but here's a summary:
At 2025-08-07 00:25:00 UTC we ingested some unexpected queue metrics for a far-future date. It looks like a customer triggered this from their laptop. Our upscale logic looks at the most recent metrics bucket as a boundary, so this data broke our upscaling.
We have many automated checks in place, but none caught this, and unfortunately it happened overnight for us, so we weren't aware of the issue until this morning around 2025-08-07 12:00:00 UTC.
We identified the bad data and deleted it at 2025-08-07 12:36:00 UTC. Upscaling immediately started working as expected.
We're implementing a permanent fix now, as well as automated checks so that we're alerted immediately in the future. Check our blog later today for a more thorough write-up.
Affected services
Heroku autoscaling
Render autoscaling
Amazon ECS autoscaling
Fly.io autoscaling
Railway autoscaling
Updated
Aug 07 at 08:43am EDT
We've identified the potential cause of the issue and applied a fix, upscaling should be working now, we'll continue to monitor.
Affected services
Heroku autoscaling
Render autoscaling
Amazon ECS autoscaling
Fly.io autoscaling
Railway autoscaling
Created
Aug 07 at 08:00am EDT
We've received some reports of upscaling not triggering, and are investigating.
Affected services
Heroku autoscaling
Render autoscaling
Amazon ECS autoscaling
Fly.io autoscaling
Railway autoscaling