<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <link rel="alternate" type="text/html" href="https://torprojectstatus.org/"/>
    <title>Weather.torproject.org on Tor Project status</title>
    <link>https://torprojectstatus.org/affected/weather.torproject.org/</link>
    <description>Incident history</description>
    <generator>github.com/cstate</generator>
    <language>en</language>
    
    <lastBuildDate>2024-03-19T15:00:00+00:00</lastBuildDate>
    <updated>2024-03-19T15:00:00+00:00</updated>
    
    
    
      <atom:link href="https://torprojectstatus.org/affected/weather.torproject.org/index.xml" rel="self" type="application/rss+xml" />
    
    
      <item>
        <title>[Resolved] Tor Weather outage</title>
        <link>https://torprojectstatus.org/issues/2024-03-19-tor-weather-outage/</link>
        <pubDate>Tue, 19 Mar 2024 15:00:00 +0000</pubDate>
        <guid>https://torprojectstatus.org/issues/2024-03-19-tor-weather-outage/</guid>
        <category>2024-03-25 08:58:00 &#43;0000</category>
        <description>We are in the process of upgrading Tor Weather to version 2.0. For more information about the details and expected downtime, see the tracking ticket. We are sorry for the inconvenience.</description>
        <content type="html">&lt;p&gt;We are in the process of upgrading Tor Weather to version 2.0. For more information about the details and expected downtime, see the &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/issues/41556&#34;&gt;tracking ticket&lt;/a&gt;. We are sorry for the inconvenience.&lt;/p&gt;
</content>
      </item>
    
      <item>
        <title>[Resolved] Tor Weather data loss</title>
        <link>https://torprojectstatus.org/issues/2023-11-08-tor-weather-outage/</link>
        <pubDate>Fri, 27 Oct 2023 06:23:11 +0000</pubDate>
        <guid>https://torprojectstatus.org/issues/2023-11-08-tor-weather-outage/</guid>
        <category>2023-11-08 18:29:00 &#43;0000</category>
        <description>The Tor Weather service broke during an upgrade to Debian Bookworm. The database was destroyed (but eventually recovered from backups) and the service was down for about a week.
Specifically, all changes made after 2023-10-27 06:23:11 UTC have been lost. The service itself went offline some time before 2023-10-31 20:33UTC, therefore about 4 days of data was lost. The data loss was due to a flaw in the bookworm upgrade procedure, since then corrected, combined with expiration of the continuous backups which would normally have allowed full recovery.</description>
        <content type="html">&lt;p&gt;The &lt;a href=&#34;https://weather.torproject.org/&#34;&gt;Tor Weather service&lt;/a&gt; broke during an &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/issues/41252&#34;&gt;upgrade to Debian
Bookworm&lt;/a&gt;. The database was destroyed (but eventually recovered from
backups) and the service was down for about a week.&lt;/p&gt;
&lt;p&gt;Specifically, all changes made after 2023-10-27 06:23:11 UTC have been
lost. The service itself went offline some time before 2023-10-31
20:33UTC, therefore about 4 days of data was lost. The data loss was
due to a flaw in the &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/upgrades/bookworm#postgresql-upgrades&#34;&gt;bookworm upgrade procedure&lt;/a&gt;, since then
&lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/wiki-replica/-/commit/7f50f2989d7f98ff716844f416aa487fe74fd77c&#34;&gt;corrected&lt;/a&gt;, combined with expiration of the continuous backups
which would normally have allowed full recovery.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;If you have registered or made any changes on Tor Weather after
October 27th, you will need to redo those changes.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The service was restored on November 8th (2023-11-08 18:29UTC) from
the October 27th backup. So the service was offline for a little over
8 days, from October 31th to November 8th.&lt;/p&gt;
&lt;p&gt;A more detailed discussion and post-mortem can be found in &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388&#34;&gt;issue
41388&lt;/a&gt;, including a &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388/timeline&#34;&gt;full timeline of the incident&lt;/a&gt;, heroic
&lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/wikis/howto/backup#recovering-deleted-files&#34;&gt;deleted file recovery procedures&lt;/a&gt;, and &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/issues/41388/#future-improvements&#34;&gt;future improvements&lt;/a&gt; to
our systems that should keep this specific problem from happening
again.&lt;/p&gt;
&lt;p&gt;It is not in our habit to lose data, and we apologize profusely for
this mishap.&lt;/p&gt;
</content>
      </item>
    
      <item>
        <title>[Resolved] Tor Weather outage</title>
        <link>https://torprojectstatus.org/issues/2023-04-04-tor-weather-outage/</link>
        <pubDate>Tue, 04 Apr 2023 21:00:00 +0000</pubDate>
        <guid>https://torprojectstatus.org/issues/2023-04-04-tor-weather-outage/</guid>
        <category>2023-04-12 15:46:00 &#43;0000</category>
        <description>We needed to take down Tor Weather temporarily as it was spamming the tor-commits mailing list. Once we have an acceptable solution to that problem it will be re-enabled. Sorry for the inconvenience.</description>
        <content type="html">&lt;p&gt;We needed to take down Tor Weather temporarily as it was &lt;a href=&#34;https://gitlab.torproject.org/tpo/tpa/team/-/issues/41118&#34;&gt;spamming the
tor-commits mailing list&lt;/a&gt;. Once we have an acceptable solution to that problem it will be re-enabled. Sorry for the inconvenience.&lt;/p&gt;
</content>
      </item>
    
      <item>
        <title>[Resolved] Tor Weather outage</title>
        <link>https://torprojectstatus.org/issues/2023-03-30-tor-weather-outage/</link>
        <pubDate>Thu, 30 Mar 2023 12:41:00 +0000</pubDate>
        <guid>https://torprojectstatus.org/issues/2023-03-30-tor-weather-outage/</guid>
        <category>2023-03-31 08:46:35 &#43;0000</category>
        <description>Shortly after launching our new Tor Weather service the relay operator tornth found a serious violation of privacy expectations that resulted in suspending the Tor Weather service until that issue gets fixed. We are sorry for that inconvenience.</description>
        <content type="html">&lt;p&gt;Shortly after &lt;a href=&#34;https://lists.torproject.org/pipermail/tor-relays/2023-March/021110.html&#34;&gt;launching&lt;/a&gt; our new Tor Weather service the relay operator tornth &lt;a href=&#34;https://gitlab.torproject.org/tpo/network-health/tor-weather/-/issues/57&#34;&gt;found a serious violation of privacy expectations&lt;/a&gt; that resulted in suspending the Tor Weather service until that issue gets fixed. We are sorry for that inconvenience.&lt;/p&gt;
</content>
      </item>
    
  </channel>
</rss>
