{
  "is": "issue",
  "title": "Tor Weather data loss",
  "body": "\u003cp\u003eThe \u003ca href=\"https://weather.torproject.org/\"\u003eTor Weather service\u003c/a\u003e broke during an \u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252\"\u003eupgrade to Debian\nBookworm\u003c/a\u003e. The database was destroyed (but eventually recovered from\nbackups) and the service was down for about a week.\u003c/p\u003e\n\u003cp\u003eSpecifically, all changes made after 2023-10-27 06:23:11 UTC have been\nlost. The service itself went offline some time before 2023-10-31\n20:33UTC, therefore about 4 days of data was lost. The data loss was\ndue to a flaw in the \u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm#postgresql-upgrades\"\u003ebookworm upgrade procedure\u003c/a\u003e, since then\n\u003ca href=\"https://gitlab.torproject.org/tpo/tpa/wiki-replica/-/commit/7f50f2989d7f98ff716844f416aa487fe74fd77c\"\u003ecorrected\u003c/a\u003e, combined with expiration of the continuous backups\nwhich would normally have allowed full recovery.\u003c/p\u003e\n\u003cp\u003e\u003cstrong\u003eIf you have registered or made any changes on Tor Weather after\nOctober 27th, you will need to redo those changes.\u003c/strong\u003e\u003c/p\u003e\n\u003cp\u003eThe service was restored on November 8th (2023-11-08 18:29UTC) from\nthe October 27th backup. So the service was offline for a little over\n8 days, from October 31th to November 8th.\u003c/p\u003e\n\u003cp\u003eA more detailed discussion and post-mortem can be found in \u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388\"\u003eissue\n41388\u003c/a\u003e, including a \u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388/timeline\"\u003efull timeline of the incident\u003c/a\u003e, heroic\n\u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/backup#recovering-deleted-files\"\u003edeleted file recovery procedures\u003c/a\u003e, and \u003ca href=\"https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388/#future-improvements\"\u003efuture improvements\u003c/a\u003e to\nour systems that should keep this specific problem from happening\nagain.\u003c/p\u003e\n\u003cp\u003eIt is not in our habit to lose data, and we apologize profusely for\nthis mishap.\u003c/p\u003e\n",
  "createdAt": "2023-10-27 06:23:11 +0000 UTC",
  "lastMod": "2026-06-02 16:38:52 -0400 -0400",
  "permalink": "https://torprojectstatus.org/issues/2023-11-08-tor-weather-outage/",
  "severity": "down",
  "resolved": true,
  "informational": false,
  "resolvedAt": "2023-11-08 18:29:00 +0000",
  "affected": ["weather.torproject.org"],
  "filename": "2023-11-08-tor-weather-outage.md"
}