tag:status.authzed.com,2005:/historyAuthzed Status - Incident History2024-03-29T07:52:48ZAuthzedtag:status.authzed.com,2005:Incident/152637472022-12-15T18:54:37Z2022-12-15T18:59:54ZWe're investigating API failures<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>18:54</var> UTC</small><br><strong>Resolved</strong> - We've monitored the service and are confident that this incident has concluded.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>18:09</var> UTC</small><br><strong>Monitoring</strong> - We've identified the issue and rolled out a fix.<br />We'll continue to the monitor the situation for some minutes before we close the incident and release a post-mortem.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>17:32</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.authzed.com,2005:Incident/136766182022-11-25T21:15:03Z2022-11-25T21:15:03ZAPI availability and response times are degraded.<p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>21:15</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>21:12</var> UTC</small><br><strong>Update</strong> - the issue has been identified and resolved</p><p><small>Nov <var data-var='date'>25</var>, <var data-var='time'>18:55</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.authzed.com,2005:Incident/136359682022-11-22T15:35:33Z2022-11-22T15:35:33ZAPI availability and response times are degraded.<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:35</var> UTC</small><br><strong>Resolved</strong> - API latencies and availability have returned to normal levels.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>15:18</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.authzed.com,2005:Incident/111141702022-09-13T01:10:10Z2022-09-13T01:44:48ZServerless Platform Ingress outage<p><small>Sep <var data-var='date'>13</var>, <var data-var='time'>01:10</var> UTC</small><br><strong>Resolved</strong> - A configuration change was applied to the SpiceDB Serverless platform's primary network ingress.<br />The configuration was immediately noticed and rolled back as soon as possible.<br /><br />There was approximately 7 minutes of outage between 9:10PM EDT and 9:17PM EDT.<br />The root cause was a slight version difference between Contour in our production and staging environments.<br />Resolution was delayed by a few minor but vital workflows that collectively consumed time:<br />- External ingress health-checking that runs on a larger interval than our internal health-checks<br />- Lack of log retention of crash-looped pods Kubernetes making `kubectl logs` workflow less ideal<br /><br />We will be improving our process for vetting configuration to make sure that this cannot possibly happen again.</p>tag:status.authzed.com,2005:Incident/109140022022-08-25T10:35:20Z2022-08-25T10:35:20ZInvestigating a spike in query time<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>10:35</var> UTC</small><br><strong>Resolved</strong> - Database operations have returned to normal, and all systems appear to be operating normally at this time.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>09:22</var> UTC</small><br><strong>Monitoring</strong> - Our upstream database provider appears to be in service again and we're currently in contact about how to restore full service. At this time, the API appears to be working again, with a slight performance overhead</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>09:03</var> UTC</small><br><strong>Update</strong> - We are currently investigating an apparent outage with our upstream database provider</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>08:59</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.authzed.com,2005:Incident/99622332022-05-13T13:30:36Z2022-05-13T13:30:37ZScheduled Upgrades<p><small>May <var data-var='date'>13</var>, <var data-var='time'>13:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>13:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>May <var data-var='date'>11</var>, <var data-var='time'>00:05</var> UTC</small><br><strong>Scheduled</strong> - We will be performing a scheduled upgrade which may result in intermittent request errors.</p>tag:status.authzed.com,2005:Incident/93368072022-02-19T22:00:58Z2022-02-22T21:21:47ZIncreased API Latency<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>22:00</var> UTC</small><br><strong>Resolved</strong> - A fix has been deployed and the service has returned to normal.<br /><br />A detailed post-mortem is in progress and will be posted on the following business day.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>01:38</var> UTC</small><br><strong>Monitoring</strong> - Perceived latency has returned to regular performance. We'll be monitoring over the next day to determine a root cause.</p><p><small>Feb <var data-var='date'>16</var>, <var data-var='time'>22:25</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating spikes in API latency.</p>tag:status.authzed.com,2005:Incident/74868982021-07-15T19:38:38Z2021-07-15T19:38:39ZAuthzed Read-Only Maintenance<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>18:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jul <var data-var='date'>12</var>, <var data-var='time'>20:33</var> UTC</small><br><strong>Scheduled</strong> - We will be undergoing a backend data migration and Authzed will be in read-only mode temporarily.</p>