Elevated wait time

Incident Report for TestingBot

Postmortem

Today we experienced long waiting times for Windows + macOS machines.
The issue was caused by a server issue, where memory issues were causing restarts of our hub (hub.testingbot.com)

This in turn caused the machines running the tests to fail to communicate with the hub, causing tests from clients to wait up to 5 minutes before a test could start.

If your test runner is configured with a http connect timeout < 5 minutes, this may have caused issues where tests would fail to start.

The memory issue was resolved and we are looking into a solution to prevent this from happening again.

Posted Sep 05, 2025 - 07:47 CEST

Resolved

Since 1:00 UTC time we have been experiencing elevated wait times, causing some tests to wait for up to 5 minutes before they could start.

We deployed a fix and are monitoring the waiting times. They are back to < 10 seconds since 04:30 UTC.
Posted Sep 05, 2025 - 07:43 CEST