We ran our first experiment with our new servers last week. About 1% of iOS/Android players were moved to the new servers (Mobile only… not Steam or Console).
[Note that SERVERS are separate to the DATABASE… we migrated to the new database quite a few months ago, and it’s been running very well]
It showed up a few problems…
some minor things that you would hardly notice (such as a couple of very rare error messages)
some moderate issues (such as a higher guild seals cap & a brief issue with XP/Soul multipliers which affected only the 1% who moved),
and some fairly major issues (a new database query which we think may have been messing with the server stability for the last few days at peak time - 2am GMT… which has affected everyone on at those times on ALL platforms)
I believe we’ve now addressed all the known issues from last week, so we’re going to start moving people over the next 5-7 days.
There’s nothing you need to do for this… we’ll likely move another 1% today to test the weekly reset (the busiest server time of the week) and then we’ll start moving people over in larger numbers tomorrow.
Consoles will begin their move in a couple of days’ time.
Well, that may explain the hiccups we encountered over the past week. Considering how many player accounts you intend to migrate in the next few days, would it not be best to just shut down access to the game for a day to make this go more smoothly? A simple notification at launch would suffice. Something to the effect: We apologized for the inconvenience. Game is unavailable until the server migration is complete. Go to www.gemsofwar.com for more info. Thank you.
Hi Smuggler (I just typo’ed and almost called you “snuggler”… oops - lol)
Actually, no.
While it might seem like a good idea at first, the issues we encounter are mainly due to the scale of things.
For example, when we test with 10 accounts on the new servers simultaneously, it all works beautifully!
At 100 Accounts… a few minor issues might become apparent…
If we step that up to 1,000, things start to interact in interesting ways… and resource-related issues occur, which need extra server power to solve… we need to ensure that scales up and down automatically and reliably.
When we step it up to 10,000 people, all smashing away simultaneously, then we start getting really weird interactions as database records lock & queue for access…
And so on and so forth…
So, yeah, we need to bring people over gradually so we have a chance to address each set of issues as they begin to happen.
Oh, regarding Amazon Servers… nothing to do with actually being on the Amazon store (though we’re planning that for later this year, I hope).
Amazon Web Services (AWS) are one of the premium server providers out there… their service is ultra-reliable, and scales up really nicely. Even if our current server provider, Parse, were not shutting down at the end of this month, we’d likely be needing to move to AWS to handle the rate at which we’ve been growing.
Thanks for the announcement Sirrian. Though it would be better if you guys make official announcement (like this one, here on forum, not just in global 001 where some people tend to avoid) before migrating players to the new server. It would prevent or reduce the panic and or anger that caused by the server issues. Communication is the key.
While using Global chat is convenient, on the console end, some won’t read any of it due to their own privacy settings. Next best thing is the News portion in-game. That’s certainly one way to notify that some hiccups are likely to cause a boo-boo here and a boo-boo there, here a boo-boo, there a boo-boo, everywhere a boo-boo…
Many thanks for notifying us here officially, Sirrian.
Apparently “very rare” means “10+ times” over a 24 hour period. Oh, wait, consoles haven’t even started? So what the heck is causing the mass amount of errors then, I wonder?
You know you really make me miss my old career. I remember going through that pain to reach multi-million transactions. I remember when I would sit and read query reports and figure out where we were getting deadlocks or rowchaining issues.
I miss my old team.
Damn…
Now I am going to go sit back here and find some epic music to go have a decent cry.
Btw, I totally totally appreciate the heck you guys are going through.
OK, that explains how it was that my guild leader saw his weekly seal limit climb to over 15K seals, and no one else in the guild was affected. Man, if it was me, I could have had a mythic Courage by now!
I still haven’t seen anything posted about the issue where Treasure Map turns were starting at 14 and not 8 this past week. I was/am waiting to see if it goes back to normal when the events change.
Mine jumped to 4900 then 5350 in the matter of one match. I had the xp and soul issue too last reset… that was a fun 30 mins or so… thousands of seals and 15 levels before reset (I leveled every match…)
@sirrian you might have spared your loving community some spleen by getting some official news like this out before the migration steps begin causing issues rather than after…
Fair call… but it was a double-edged sword really… we’ve been running small tests since Dec 19th, without anyone noticing, and we figured if we announced TOO early then we’d just be inviting ourselves to the “taking too long” criticism
We’d always planned to announce THIS week, but we never anticipated the issues that occurred at the 1% mark, since we thought these early tests were just about learning how to scale the server resources, not addressing a new issue based around database index access from the new server code.
Has this been what is affecting chat for console? Chat won’t load 80% of the time for me. When I press the back button or click on guild chat I just get an infinite loading ring and have to restart the game. it also just happened so whatever issue you detected and fixed doesn’t seem to apply to chat.
Also, in the rare times when I get chat to load, I regularly can’t send messages. i.e., I type out a message, press RT, and nothing is sent. Tried multiple times today and couldn’t send anything in chat when it was actually working.
