EU - Incident with Actions
Incident
Report for GitHub Enterprise Cloud - EU
Resolved
Between January 31, 2025 19:08 UTC and Feb 1, 2025 16:33 UTC, GitHub-hosted runners for Actions experienced a disruption on GitHub Enterprise Cloud in EU and Australia regions. 100% of Actions jobs in these regions using hosted runners failed with an error indicating an internal failure when running the job.
The issue was caused by an unexpected removal of critical DNS records caused by the interaction of two unrelated configuration changes. Each change was validated during deployment, but the hosted runner impact started after deployment validation was completed. A gap in alerting led to a significant delay in detecting the impact and initiating mitigation.
We have completed monitoring improvements for earlier detection, updated the guidance on DNS changes, and are working on improvements to harden the deployment and validation of similar changes for the future.
The issue was caused by an unexpected removal of critical DNS records caused by the interaction of two unrelated configuration changes. Each change was validated during deployment, but the hosted runner impact started after deployment validation was completed. A gap in alerting led to a significant delay in detecting the impact and initiating mitigation.
We have completed monitoring improvements for earlier detection, updated the guidance on DNS changes, and are working on improvements to harden the deployment and validation of similar changes for the future.
Update
We have found and mitigated the issue, actions should now be working again.
Update
We're continuing to investigate the cause of the issue with Actions
Update
We have detected an issue where users are unable to run actions using GitHub hosted runners.
Investigating
We are investigating reports of degraded performance for Actions
This incident affected: Actions.
Subscribe to our developer newsletter
Get tips, technical guides, and best practices. Twice a month. Right in your inbox.
Subscribe