Some of the team have been working this weekend to try and track down the chat issues we’ve been seeing - first on console, and then on PC/Mobile more recently.
We have a theory what’s going on…
TL;DR: Something outside our system broke - we think we might have found it - and we’ll be rebooting chat for 30-60 min’s sometime on our Monday to try replacing it.
More Technical Description: Our chat server logs are showing occasional “errors” in the form of delays when attempting to write data to storage. These delays completely block input from players, leaving incoming chat messages either delayed or lost. We use a number of cloud services for chat, and it seems likely that one of the storage-related ones has changed recently, meaning that synchronous writes to EFS/NFS drives can fail, and block other processes.
Our attempted fix for this will be to take chat briefly offline, switch over to an EBS drive, and then restart chat.
Thanks for the update Sirrian! Glad to hear you guys tracked down what you think is the issue, we appreciate all the hard work the team puts into this great game
@Sirrian What about just simply using a server to delegate connections between those actively in chat and just rely more on P2P via the linked consoles. And since it would be delegated from a server, any malicious attempts would have to go through the server first in order to get to the P2P network. Just an idea.
The servers were just restarted, so you may have noticed an disconnection (if you were not already experiencing an issue). Please restart your game if you did. The team are continuing to investigate chat issues this week.
