Hey everyone,
Thanks for your patience over the holidays.
Straight into it with the details:
Over the weekend of Christmas, incorrect bans were placed on roughly 0.37% of the active user base.
The development team noticed the issue at just after midnight on Christmas morning and we had 3 team members dropping in as they could to assist with solving the problem from that time right up until almost 11 pm the same day.
By the end of the 26th the issue was solved and all the affected players were unbanned.
When bans are placed, the game automatically removes all the banned player’s event progress and half their guild trophies to ensure they don’t keep their leaderboard positions or rewards they may have earned by cheating.
Obviously, in this case it added another layer of complexity to the issue, but by the end of the 28th everyone’s Trophies, event and Campaign progress were restored.
So that’s what happened, WHY did it happen though?
Gems of War has automatic cheat detection.
Due to a bug, players who shouldn’t have been, were detected as cheating and banned.
This happened because the tool can both detect and ban.
We’ve been developing the automatic cheat detection tool for years now.
The first step was just detection so we could ensure accuracy over the entire player base and over an extended period of time.
With that being the case, we have been slowly rolling out automatic bans in small steps to continue ensuring the tool was working appropriately.
This roll out has gone perfectly smoothly throughout all stages up until Christmas, when due to a number of unexpected interactions, the tool banned these players incorrectly.
Originally it did look like this issue was caused by Kris Kringle’s map giving ability and the way those rewards are logged on the server, however, a key factor ended up being Maps earned over not just one battle but over a number of battles - which made the problem grow exponentially as time went on regardless of whether you were using Kris Kringle or not. Kris Kringle basically exacerbated the issue but didn’t cause the issue. This is also why the issue wasn’t picked up when we were testing Kris Kringle.
We owe our community an apology
First and foremost we apologize to the affected players who were prevented from playing over the holiday weekend. We also acknowledge the stress this put on their Guilds as we know how horrible it was seeing your Guild’s progress and Leaderboard ranks drop because of this.
We also want to acknowledge that we can and should have been more transparent about the rollout of the cheat detection tool. We’re usually quite reserved in discussing anything to do with cheat detection as we’re always trying to stay ahead of people creating cheat tools and bots. While we do feel that the tool is a very positive addition to our toolkit in assisting us with managing such a large community, in retrospect we should have let everyone know that we were taking the next steps from detection to testing actual automatic bans.
With that said, for full transparency, we will be using automatic bans moving forward.
In light of what happened over Christmas we are obviously keeping an extra close eye on all bans.
However, due to what happened we’re currently discussing taking a step back from full automation to still requiring a staff member to approve ban batches, so that if the number of players in the ban queue spikes like it did over Christmas, a staff member will see that and be able to reject the batch before the bans are actually placed. This is an ongoing discussion after having this issue so that we can prevent anything like this happening for any reason again.
With that said, anyone who writes a ban appeal for their account still has their ban checked manually by staff.
Now that the bugs have been fixed and everyone’s progress corrected and we have more staff back from holidays, we will be organizing compensation for the banned players. If you’d like to be notified once compensation has been sent out please follow the help center article here
Safekeeping in case of edits
