Server Migration Progress

Official News
1

I’ve created and stickied this thread so we have somewhere to keep you all updated about the progress of Server Migration. I’m also closing the thread, so it’s restricted to the dev team… mainly this is here to keep everyone updated.

I’ll keep a rolling list of events going through here with the latest on the top. All dates/times in GMT.

  • Wednesday Jan 18 -3:30pm GMT - Everybody on PC/iOS/Android should now be on the new servers. We’re goign to stick around as we approach peak period in about 1 hour though and make sure everything behaves itself!
  • Wednesday Jan 18 -11:45am GMT - And 100% of Steam is now on the new servers (when the game is updated… we just pushed out a VERY VERY tiny update for you). Now we sit and watch the servers for 6 hours and see what happens!
  • Wednesday Jan 18 -10:30am GMT - Latency should be back to normal for mobile players. We’ve put out a Steam update that moves 30% of Steam players onto the new servers.
  • Wednesday Jan 18 -7:45am GMT - Some players on mobile will be experiencing higher than normal latency & timeouts (retry connection messages) as we remove some servers and create new more powerful ones. Normally we would just add new servers, but Amazon won’t let us add any more new ones right now
  • Wednesday Jan 18 - 6:30am GMT - Fixed an issue on thew new servers where the Guild Seal cap was increasing.
  • Wednesday Jan 18 - 6:15am GMT - Update on Guilds - we’re going to see about getting Steam players over onto the new servers asap, HOWEVER we’ve encountered a small (temporary) problem with Amazon (details in a post below)
  • Wednesday Jan 18 - 5:45am GMT - Investigating guild roster issues & no-pvp-opponent issues. No data has been lost or corrupted - it’s all safe and sound. Still investigating, but it DOES seem to be a problem with the old servers.
  • Wednesday Jan 18 - 5:30am GMT - We’re seeing a weird issue related to the OLD servers where Guild Rosters and PvP opponents are appearing blank/empty. Currently investigating…
  • Wednesday Jan 18 - 5:00am GMT - And we now have 100% of iOS/Android players on the new servers! w00t (though some will still be on the old servers until they restart their apps). Steam next, but perhaps tomorrow.
  • Wednesday Jan 18 - 4:00am GMT - Performance tweaks looking good - we’ve halved CPU usage & server latency. Now adding up to 75% of iOS/Android players.
  • Wednesday Jan 18 - 3:30am GMT - We’re making a small experimental tweak to the new server environment to try and improve performance.
  • Wednesday Jan 18 - 3:00am GMT - Now added 50% of all iOS/Android players to the new servers… no issues so far
  • Tuesday Jan 17 - 3:00pm GMT - Just letting you know that we approach peak in about 1 hr… so we’re in the office monitoring everything.
    8 Tuesday Jan 17 - 7:00am GMT - The move to the C4 servers is complete, and seems to be running okay for now… we’ll be keeping an eye on things during our evening & night. Moving more folks over on iOS/Android tomorrow!
  • Tuesday Jan 17 - 5:30am GMT - Moving to a higher-performance server type (M4 → C4) in anticipation of adding more players over the next few days
  • Tuesday Jan 17 - 4:00am GMT - We’re seeing some issues with network latency… CPU usage is low, but latency is high… we’re adding more RAM to compensate.
  • Tuesday Jan 17 - 3:00am GMT - Increased new AWS server capacity by 250% based on current usage
  • Tuesday Jan 17 - 2:30am GMT - ERROR 13 on iOS/Android. Fix deployed
  • Tuesday Jan 17 - 2am GMT - 20% of iOS/Android players moved to new servers
  • Monday Jan 16 - 11pm GMT - Extra database resources deployed to address outage
  • Monday Jan 16 - 5pm GMT - Database issue - serious intermittent outage for players on all platforms
  • Monday Jan 16 - 9 am GMT - 8% of iOS/Android players moved to new servers
  • Monday Jan 16 - 7am GMT - weekly rollover successful on both new and old servers
  • Monday Jan 16 - midnight GMT - 2% of iOS/Android players are on the new servers
30 Likes
4

Quick note… 7:00am GMT is the end of the day for us here in Australia, so no major changes for the next 15 hours.

Both Andrew and myself will be keeping an eye on the servers late into tonight for any signs of instability, particularly around 3pm GMT (4am here!!! :cry:) when we’re at peak load.

13 Likes
5

Okay we’re all back in the office today.
We spent the morning reviewing any issues/errors from last night, and the only ‘mongo database’ errors that occurred seem to have come from players on the OLD servers.
That’s good, so we can continue moving people over today… first thing will be cranking the iOS/Android migration up to 50% of the playerbase and scaling the servers accordingly!

13 Likes
6

Quick note about Guild Roster issues (where it’s appearing empty)… no data has been lost on the server, so no need to panic… it’s just a display issue!

Currently, it seems that players on the OLD servers are experiencing this. No idea of the cause yet, but killing off and restarting the mobile app should force you onto the new servers and fix the issue.

On Steam - sorry guys - no news for you yet… though we’re investigating fast-tracking your migration so you don’t experience the problem any longer than you have to!

10 Likes
7

Ha! We were going to send all the Steam players onto the new servers asap, but we’ve been temporarily foiled by Amazon Web Services!

Amazon Web Services have a temporary limit to how many server instances you can spin up and put in place. This is obviously to stop folks creating some evil botnet to perform DDoS attacks, or to stop companies spinning up instances quickly then not paying their bills!

We can get the cap lifted pretty quickly I think, but not immediately. Until then we’re going to experiment with using the same number of more powerful servers to see if we can bring Steam players over. Otherwise, timeframe is back to “tomorrow” for Steam players, I’m afraid!

13 Likes
8

If anyone makes it down this far…

Everybody is now on the new servers

STEAM PLAYERS
Make sure you get the update! If you still have no PvP opponents or still see an empty Guild Roster, your update hasn’t applied yet… Steam can be a little weird like that!

iOS/ANDROID PLAYERS
You’ve been on the new servers most of the day.
Note that some Android Players have seen an issue where they’re still managing to play (somehow) on version 2.1.5
If you’re on android, and the game is behaving REALLY weird… that might be your problem… check the store for an update

CONSOLE PLAYERS
You’re next!
But we’re likely going to leave you until next week, while we observe the current servers
The good news for you is that although you had to wait a little longer, the PC/Mobile guys will have found most of the bugs for you!

19 Likes