An Update on Chat Server Issues


#1

Hi everyone,

Some of the team have been working this weekend to try and track down the chat issues we’ve been seeing - first on console, and then on PC/Mobile more recently.

We have a theory what’s going on…

TL;DR: Something outside our system broke - we think we might have found it - and we’ll be rebooting chat for 30-60 min’s sometime on our Monday to try replacing it.

More Technical Description: Our chat server logs are showing occasional “errors” in the form of delays when attempting to write data to storage. These delays completely block input from players, leaving incoming chat messages either delayed or lost. We use a number of cloud services for chat, and it seems likely that one of the storage-related ones has changed recently, meaning that synchronous writes to EFS/NFS drives can fail, and block other processes.
Our attempted fix for this will be to take chat briefly offline, switch over to an EBS drive, and then restart chat.

Hopefully we’ll be fixed soon!


Global Chat still broken! How long will this continue?
Ps4 and xbox one connection problems
#2

Awesome sirrian im glad you guys finally spotted the problem, thank you for the hard work you guys doing every day, this is really appreciate.

Also i can’t wait for 3.3 :slight_smile:


#3

Thanks for the update Sirrian! Glad to hear you guys tracked down what you think is the issue, we appreciate all the hard work the team puts into this great game :grinning:


#4

yay for finding possible fix for it.

GL with it working


#5

Awesome! i love you


#6

Dang, thank you for sacrificing your weekend on this. I know you’d rather be elsewhere lol a lot of players will appreciate this.


#7

This will be most welcomed! Thanks @Sirrian


#8

@Sirrian What about just simply using a server to delegate connections between those actively in chat and just rely more on P2P via the linked consoles. And since it would be delegated from a server, any malicious attempts would have to go through the server first in order to get to the P2P network. Just an idea.


#9

The servers were just restarted, so you may have noticed an disconnection (if you were not already experiencing an issue). Please restart your game if you did. The team are continuing to investigate chat issues this week.


#10

Ty @Cyrup i hope it worked we will need it for raid boss next week? :slight_smile:


#11

Unable to get back on at all, 2 restarts and 20min later and its unable to reconnect on PC.


#12

Same here cannot connect to chat on xbox


#13

We only expected it to be about 10 minutes, but the team is still working on this and the chat will be down for a little while.


#14

Please keep it down as long as it takes to resolve the issue. A few hours of down time is nothing in order to have working Guild chat.


#15

The chat issues should now be resolved. Please restart your game and let us know how it is!


#16

Logging in and crossing fingers


#17

Can log in on xbox but look like problem isn’t fixed i just writted something and waiting to see it appear on the screen

Edit: also it’s feeezed i can’t switch between global and guild chat

Restarting just in case

Edit 2: nope not fixed sorry still can’t connect to chat


#18

We needed to do another reboot, I’ll let you know how it goes.


#20

Working ten chrs


#21

Ps4 - restarted, tried to connect to chat. Error. Game crashed, self-closed.
Tried again, seems ok :crossed_fingers: