GitLab on Tor Project status

[Resolved] GitLab unresponsive

Mon, 27 Apr 2026 09:39:00 -0400

GitLab is seeing an unusual load that depletes all CPU. All services are unresponsive because of this. We’re working to reduce the impact on normal services.

For details, see: https://gitlab.torproject.org/tpo/tpa/team/-/issues/42604

A partial block was imposed, which resolved the issues.

[Resolved] GitLab CI containers failures

Mon, 20 Apr 2026 11:58:39 -0400

The GitLab Container Registry is refusing requests to GitLab CI pipelines between projects.

This started on Friday, when the GitLab 18.11 upgrade was deployed. We are hoping for a quick fix from upstream, but in the meantime, we need your help!

Normal GitLab operation should be unaffected. GitLab CI still works, it just fails with images from our own registry when accessed from personal projects.

We’ve found a workaround that requires you to report the failures to us. In tpo/tpa/team#42595, we ask that you write a new comment with the following information:

source project: failing project (e.g. group/failing_project)
registry project: project it is trying to pull from (e.g. tpo/tpa/base-images)
example failed job: URL of the failed job

For example, this was our first report:

source project: Diziet/arti

registry project: tpo/tpa/base-images

example failed job: https://gitlab.torproject.org/Diziet/arti/-/jobs/1485144

You might be able to fix this yourself. If you have access to the “registry project” above, you can manually add the source project to the CI/CD job token allowlist of the registry project", by following:

Settings
CI/CD
Job token permissions
add the “source project” to the allowlist

You MUST keep track of that exception and remove it once this incident is resolved!

[Resolved] GitLab maintenance

Wed, 25 Jun 2025 10:52:41 -0400

GitLab is having various issues: mail not going out, merge requests failing. On top of that, it’s running out of disk space.

We’re doing an emergency maintenance on the server, we’re hoping for this to resolve within one or two hours.

Details of this are available in tpo/tpa/team#42218 which, of course, might not be available right now archive.org copy.

[Resolved] Router maintenance at Hetzner

Tue, 03 Dec 2024 03:30:00 +0000

Hetzner has planned an emergency maintenance window on all of their routers, which will cause a network outage on all of our hosts in their datacenters. The maintenance window is planned on December 3rd 2024 from 3:30 UTC to 5:30 UTC, during which the network may experience spurious outages.

Services should come back online automatically as soon as network connectivity is restored by Hetzner.

[Resolved] GitLab migration to another machine cluster

Thu, 28 Nov 2024 18:00:00 +0000

Starting on November 28th at 18:00 UTC, gitlab.torproject.org will be taken offline in order to migrate it to our other machine cluster.

The transfer time is currently estimated at 18h, so gitlab will be coming online on the next day, friday 29th. If we’re lucky the transfer might finish sooner.

If you have any issues during that time, please reach out to us on IRC or via email.

[Resolved] Gitlab and CollecTor outage

Wed, 06 Dec 2023 14:00:00 +0000

We’ve experienced heavy load and unresponsiveness on some of our services (e.g. Gitlab and CollecTor) leading to outages and disruptions.

The issue seems to have resolved itself, investigation seemed to show this was a routing issue upstream.

Update: issue have crept up again, root cause was elevated temperature with the hard drives on the affected server. Upstream has replaced fans in the server and situation has returned to normal.

See issue tpo/tpa/team#41429 for detailed analysis and updates.

[Resolved] outage at main provider

Sat, 04 Feb 2023 01:57:31 +0000

We are witnessing a large outage at our main service provider, Hetzner. According to the information we have gathered so far, four switches (4!) have failed and that affects four (yes, again, 4!) of the servers in the 8-node Ganeti cluster.

Affected servers:

check-01
chives
colchicifolium
cupani
fsn-node-01
fsn-node-02
fsn-node-04
fsn-node-07
gitlab-02
gnt-fsn
henryi
loghost01
majus
materculae
media-01
onionoo-backend-01
onionoo-backend-02
onionoo-frontend-02
perdulce
polyanthum
relay-01
static-master-fsn
staticiforme
submit-01
tbb-nightlies-master
weather-01

Upstream resolved the situation after a few hours of downtime. According to Hetzner it was “a short disruption of a line card in one of our access routers”. Details of the incident are available in tpo/tpa/team#41057.

[Resolved] routing issues at main provider

Tue, 25 Jan 2022 06:00:00 +0000

We are experiencing intermittent network outages that typically resolve themselves within a few hours. Preliminary investigations seem to point at routing issues at Hetzner, but we have yet to get a solid diagnostic. We’re following that issue in issue 40601.