Back to overview
Downtime

App Server, Processing Server (Full + Snippet 0), and 24 other services are down

Dec 11, 2023 at 7:38pm UTC
Affected services
App Server
Processing Server (Full + Snippet 0)
Processing Server (Full + Snippet 1)
Processing Server (Full + Snippet 2)
Processing Server (Full + Snippet 3)
Processing Server (Snippets + Full 4)
Processing Server (Snippets + Full 5)
Processing Server (Snippets + Full 6)
Processing Server (Snippets + Full 7)
Processing Server (Scheduler)
Processing Server (TikTok 0)
Processing Server (TikTok 1)
Processing Server (TikTok 2)
Processing Server (TikTok 3)
Processing Server (TikTok 4)
Processing Server (TikTok 5)
Processing Server (TikTok 6)
Processing Server (TikTok 7)
PROD Full /Snippet Server 0 - JobPriority Not 0
PROD Full /Snippet Server 1 - JobPriority Not 0
PROD Scheduler Server - JobPriority Not 0
TikTok 0 - JobPriority Not 0
TikTok 1 - JobPriority Not 0
TikTok 2 - JobPriority Not 0
TikTok 3 - JobPriority Not 0
Manual Update Cron (FB)

Resolved
Dec 12, 2023 at 5:23pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 recovered.

Updated
Dec 12, 2023 at 5:23pm UTC

TikTok 2 - JobPriority Not 0 recovered.

Updated
Dec 12, 2023 at 5:23pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0, PROD Scheduler Server - JobPriority Not 0, TikTok 0 - JobPriority Not 0, and 1 other resource recovered.

Updated
Dec 12, 2023 at 5:10pm UTC

PROD Scheduler Server - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 5:10pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 5:10pm UTC

TikTok 2 - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 5:09pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0, TikTok 0 - JobPriority Not 0, and TikTok 1 - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 5:07pm UTC

Processing Server (Full + Snippet 0) and Processing Server (Scheduler) recovered.

Updated
Dec 12, 2023 at 5:04pm UTC

Processing Server (TikTok 4) and PROD Scheduler Server - JobPriority Not 0 recovered.

Updated
Dec 12, 2023 at 5:04pm UTC

Processing Server (Full + Snippet 1), Processing Server (TikTok 5), PROD Full /Snippet Server 0 - JobPriority Not 0, and 1 other resource recovered.

Updated
Dec 12, 2023 at 5:03pm UTC

Processing Server (Full + Snippet 2), Processing Server (Full + Snippet 3), Processing Server (TikTok 2), and 1 other resource recovered.

Updated
Dec 12, 2023 at 5:02pm UTC

TikTok 1 - JobPriority Not 0 recovered.

Updated
Dec 12, 2023 at 5:02pm UTC

TikTok 0 - JobPriority Not 0 recovered.

Updated
Dec 12, 2023 at 5:02pm UTC

Processing Server (Snippets + Full 7), Processing Server (TikTok 0), Processing Server (TikTok 1), and 2 other resources recovered.

Updated
Dec 12, 2023 at 5:01pm UTC

Processing Server (Snippets + Full 4), Processing Server (Snippets + Full 5), Processing Server (Snippets + Full 6), and 2 other resources recovered.

Updated
Dec 12, 2023 at 4:38pm UTC

Processing Server (Full + Snippet 2) and Processing Server (Full + Snippet 3) went down.

Updated
Dec 12, 2023 at 4:37pm UTC

Processing Server (Full + Snippet 1) went down.

Updated
Dec 12, 2023 at 4:37pm UTC

Processing Server (Full + Snippet 0) went down.

Updated
Dec 12, 2023 at 4:37pm UTC

Processing Server (TikTok 3) went down.

Updated
Dec 12, 2023 at 4:37pm UTC

Processing Server (TikTok 7) went down.

Updated
Dec 12, 2023 at 4:37pm UTC

Processing Server (Snippets + Full 4), Processing Server (TikTok 0), Processing Server (TikTok 1), and 3 other resources went down.

Updated
Dec 12, 2023 at 4:36pm UTC

Processing Server (Snippets + Full 7) went down.

Updated
Dec 12, 2023 at 4:36pm UTC

Processing Server (Snippets + Full 6) and Processing Server (TikTok 4) went down.

Updated
Dec 12, 2023 at 4:36pm UTC

Processing Server (Snippets + Full 5) and Processing Server (Scheduler) went down.

Updated
Dec 12, 2023 at 4:21pm UTC

TikTok 1 - JobPriority Not 0 and TikTok 2 - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 4:21pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0, PROD Scheduler Server - JobPriority Not 0, and TikTok 0 - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 4:20pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 went down.

Updated
Dec 12, 2023 at 12:40pm UTC

Manual Update Cron (FB) recovered.

Updated
Dec 12, 2023 at 10:05am UTC

App Server recovered.

Updated
Dec 12, 2023 at 9:59am UTC

App Server went down.

Updated
Dec 12, 2023 at 7:22am UTC

Manual Update Cron (FB) went down.

Updated
Dec 12, 2023 at 12:20am UTC

Manual Update Cron (FB) recovered.

Updated
Dec 11, 2023 at 10:48pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and PROD Scheduler Server - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:48pm UTC

TikTok 2 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:47pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0, TikTok 0 - JobPriority Not 0, and TikTok 1 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:25pm UTC

TikTok 2 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 10:25pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0 and TikTok 1 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 10:25pm UTC

PROD Scheduler Server - JobPriority Not 0 and TikTok 0 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 10:25pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 10:19pm UTC

TikTok 2 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:19pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:19pm UTC

TikTok 1 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:19pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:19pm UTC

TikTok 0 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:19pm UTC

PROD Scheduler Server - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:18pm UTC

TikTok 3 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 10:11pm UTC

PROD Scheduler Server - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 10:11pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 10:10pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0, TikTok 0 - JobPriority Not 0, TikTok 1 - JobPriority Not 0, and 1 other resource went down.

Updated
Dec 11, 2023 at 8:57pm UTC

TikTok 2 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 8:57pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0 and TikTok 1 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 8:57pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0, PROD Scheduler Server - JobPriority Not 0, TikTok 0 - JobPriority Not 0, and 1 other resource recovered.

Updated
Dec 11, 2023 at 8:24pm UTC

TikTok 1 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 8:24pm UTC

PROD Scheduler Server - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 8:23pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and TikTok 3 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 8:23pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0 and TikTok 0 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 8:20pm UTC

Processing Server (Full + Snippet 0) recovered.

Updated
Dec 11, 2023 at 8:19pm UTC

Processing Server (Full + Snippet 0) went down.

Updated
Dec 11, 2023 at 8:18pm UTC

TikTok 1 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 8:18pm UTC

TikTok 3 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 8:17pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 8:17pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0, PROD Scheduler Server - JobPriority Not 0, and TikTok 0 - JobPriority Not 0 recovered.

Updated
Dec 11, 2023 at 7:57pm UTC

PROD Full /Snippet Server 1 - JobPriority Not 0 and PROD Scheduler Server - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 7:56pm UTC

TikTok 3 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 7:56pm UTC

PROD Full /Snippet Server 0 - JobPriority Not 0, TikTok 1 - JobPriority Not 0, and TikTok 2 - JobPriority Not 0 went down.

Updated
Dec 11, 2023 at 7:56pm UTC

TikTok 0 - JobPriority Not 0 went down.

Created
Dec 11, 2023 at 7:38pm UTC

Manual Update Cron (FB) went down.