{
  "is": "issue",
  "title": "outage at main provider",
  "body": "\u003cp\u003eWe are witnessing a large outage at our main service provider,\nHetzner. According to the information we have gathered so far, four\nswitches (4!) have failed and that affects four (yes, again, 4!) of the\nservers in the 8-node Ganeti cluster.\u003c/p\u003e\n\u003cp\u003eAffected servers:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003echeck-01\u003c/li\u003e\n\u003cli\u003echives\u003c/li\u003e\n\u003cli\u003ecolchicifolium\u003c/li\u003e\n\u003cli\u003ecupani\u003c/li\u003e\n\u003cli\u003efsn-node-01\u003c/li\u003e\n\u003cli\u003efsn-node-02\u003c/li\u003e\n\u003cli\u003efsn-node-04\u003c/li\u003e\n\u003cli\u003efsn-node-07\u003c/li\u003e\n\u003cli\u003egitlab-02\u003c/li\u003e\n\u003cli\u003egnt-fsn\u003c/li\u003e\n\u003cli\u003ehenryi\u003c/li\u003e\n\u003cli\u003eloghost01\u003c/li\u003e\n\u003cli\u003emajus\u003c/li\u003e\n\u003cli\u003ematerculae\u003c/li\u003e\n\u003cli\u003emedia-01\u003c/li\u003e\n\u003cli\u003eonionoo-backend-01\u003c/li\u003e\n\u003cli\u003eonionoo-backend-02\u003c/li\u003e\n\u003cli\u003eonionoo-frontend-02\u003c/li\u003e\n\u003cli\u003eperdulce\u003c/li\u003e\n\u003cli\u003epolyanthum\u003c/li\u003e\n\u003cli\u003erelay-01\u003c/li\u003e\n\u003cli\u003estatic-master-fsn\u003c/li\u003e\n\u003cli\u003estaticiforme\u003c/li\u003e\n\u003cli\u003esubmit-01\u003c/li\u003e\n\u003cli\u003etbb-nightlies-master\u003c/li\u003e\n\u003cli\u003eweather-01\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eUpstream resolved the situation after a few hours of\ndowntime. According to Hetzner it was \u0026ldquo;a short disruption of a line\ncard in one of our access routers\u0026rdquo;. Details of the incident are\navailable in \u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/issues/41057\"\u003etpo/tpa/team#41057\u003c/a\u003e.\u003c/p\u003e\n",
  "createdAt": "2023-02-04 01:57:31 +0000 UTC",
  "lastMod": "2026-06-02 16:38:52 -0400 -0400",
  "permalink": "https://torprojectstatus.org/issues/2023-02-04-hetzner-outage/",
  "severity": "down",
  "resolved": true,
  "informational": false,
  "resolvedAt": "2023-02-04 03:33:46 +0000",
  "affected": ["metrics.torproject.org", "Tor Check", "torproject.org", "GitLab", "Gitolite"],
  "filename": "2023-02-04-hetzner-outage.md"
}