Some of the team have been working this weekend to try and track down the chat issues we’ve been seeing - first on console, and then on PC/Mobile more recently.
We have a theory what’s going on…
TL;DR: Something outside our system broke - we think we might have found it - and we’ll be rebooting chat for 30-60 min’s sometime on our Monday to try replacing it.
More Technical Description: Our chat server logs are showing occasional “errors” in the form of delays when attempting to write data to storage. These delays completely block input from players, leaving incoming chat messages either delayed or lost. We use a number of cloud services for chat, and it seems likely that one of the storage-related ones has changed recently, meaning that synchronous writes to EFS/NFS drives can fail, and block other processes.
Our attempted fix for this will be to take chat briefly offline, switch over to an EBS drive, and then restart chat.
Hopefully we’ll be fixed soon!