Server Roll Back & Weekend Issues (March 21st)

Bug Reports
1

Making another new thread, cos there are now numerous from over the weekend about the incident that happened on the weekend.
[Link to Server Rollback thread]

From what I am gathering, as I am also still trying to get information myself as to the specifics, there was a deployment out to Production (live servers) on Friday.
As most people may be aware, we try to avoid any released on Fridays, because, well, we aren’t in on weekends.

This deployment was unintentional, and contained untested, broken code. Which cause players to receive a mix of rewards from the Library Pass - which we had intended release today, after we had finished testing, as I had already mentioned

Now, due to the above, the way to resolve it was to roll back the servers to a point before this deployment was released. Which in turn, obviously has caused a lot of other issues, such as progress loss and resources etc.

That all being said, the team will be working on this as a priority, to make sure everyones data is correct. Once the situation is resolved, only then will I be able to get them onto compensation.

I am also waiting to find out for players who made real money purchases, if you need to submit a ticket or if we will be able to grab account data from outside to resubmit purchases.
For my sake, if anything, I am hoping it will be something the team can automate.. or I will have a lot of tickets to dig through

Please do not submit any tickets just reporting general resource loss, so we can grab any real money purchases much faster and easier, if that does become what we need to do.

I am extremely sorry that a simple mistake became something so huge, especially on a weekend and please be patient as 1, I catch up on the situation and get information where I can - which I will share when I can, and 2, with the team, as we fix this mess.

I understand the frustrations already shared, I have my own big feelings about this situation, but I will be deleting comments and suspending people who break our community guidelines - further negativity in that way does not help get anything fixed faster, or make my day any easier.
Not saying at all you can’t be angry or disappointed, please be mindful of how you are sharing those feelings.

There will likely be delays in support ticket responses for general issues, as well as some possibly delays in replies in the forum as a whole from me.

image (1)
image (1)465×466 44 KB

Again… the team will be working on this as a priority, to make sure everyones data is correct. Only once the situation is resolved, will we be able to start on compensation

UPDATE: 2:03 PM AEDT 26/03

Failed purchases are now being restored on player accounts.
Players who purchased the Campaign Pass on the weekend, will not see the bonus 200 Stars added to their account, as they are not a resource.

The team will be working on getting them added to your account. There is no need to submit a ticket.

UPDATE: 12:58 PM AEDT 26/03

Failed purchases are looking to be restored within the two hours or so.

Players who purchased the Campaign Pass on the weekend, will not see the bonus 200 Stars added to their account, as they are not a resource.

The team will be working on getting them added to your account. There is no need to submit a ticket.

UPDATE: 11:36 AEDT, 26/03

Real money purchases made that were affected by the server roll back, are still expected be restored today. Currently we are awaiting a more specific ETA.

There is no need to submit any tickets to our Support team about this weekends issues.

As mentioned yesterday, once they are restored, they will go to your Rewards tab of your Inbox. Please ensure you have space in your mailbox as it has a capacity of 100 items.

There is a chance as well that global compensation for the weekends incident will go out tomorrow - this is currently unconfirmed as the compensation itself is still being worked on.

We are also still working on a fix for the Library Pass Purchase missing rewards, and expect that to go out with the missing level 50+ Library Rewards - at this stage we expect this to go out after the weekend.

[UPDATE] 10:04 AM AEDT, 23/03 - There is no need to submit any tickets about this issue. We will be able to grab players accounts from the database, to restore any lost real money purchases without a ticket being submitted.

26 Likes
(!) Spoiler Alert (!) - Post Taransworld, links to info
List of losses due to rollback
3

Very small PSA here in regards to the real money purchases– I would expect anywhere from 3-5 business days from the time the refund is processed until you see it in your (insert payment method here) if refunds are the path here. Some may see refunds sooner. Some may be 5 business days. This is normal. Also, chargebacks are an entirely different story and would easily need a thread of its own.

That’s not something @Jeto can control nor is it anything that the devs can control or the like. Payment processors are always much faster on payments than refunds.

I have no affiliation with IP2 or anything, Rather I used to sell those credit card machines you see at every merchant and now sit on the data side.

tl;dr: this situation sucked, yes. But once refunds are issued, there’s literally nothing else you can do but wait.

5 Likes
4

To that, you can only request a refund from the platform (Steam, SONY, Microsoft) - we do not have access to your funds or any way to revert them.

In every instance where there is a purchasing issue in the past, we send your missing item or send you a different pack of the same value - this will be the same. We will get players their lost purchase

But yes, the platform can take a couple of days to process it.

4 Likes
5

Are you serious @Jeto ???

First of all we don’t care if your having a bad day or if you have to sort through a heap of tickets, that is not our problem and that is something that needs to be dealt with in house not with your customers.

Clearly your customer service skills need a lot of work. The fact that this is your companies fault yet you feel the need to threaten your customers shows you have no customer service skills and you and the company you work for have no idea what you are you doing. Your threats have deaply offended me.

This whole situation is scary because people have given you their hard earned money and the only thing you can come back with is threats. What a joke!!!

May I suggest that the only way out of this mess is to come up with a compensation amount that will make everyone happy. If that’s 10,000 gems for everyone then so be it, you will just have to suck it up and accept the loss as remember this was your companies fault. This will also save you a lot of work.

Just keep in mind that this will cause a lot of players to quit the game so what you do and say from here can make a big difference to players that are deciding to leave or stay. Threaten your customers is a terrible look and something you should NEVER do.

Good luck in sorting through all this as you are going to need it, let’s hope the right thing is done for your customers as without us there is no Gems of War.

6 Likes
6

Should point out, none of this was a threat and pointing out that in the community forum players do need to still adhere to the community guidelines.

10 Likes
7

In case it gets lost in the post above.

I will be attempting to bulk reply to all the existing tickets, so apologies if you submitted a ticket about a different and it gets caught in this reply. I will be addressing that within in the response I send out to everyone.

4 Likes
9.1 Known Issues
8

Jeto,
Thank you for the update and the apology.
Personally, I like that you vibe with us like an actual human being and not like you’re reading off some corporate CS script.
I also think it’s good for all of us that community guidelines are enforced here. Also fewer distractions when you have a massive situation to clean up.
Hope the resolution goes as well as possible.
~ Goblin

12 Likes
9

My big annoyances of this situation:

  1. Sokun not following up on the servers being down and when it’d come back on. Heck, Sokun never came back.

  2. The rollback wiping out the full day and not sometime closer to the mail. That stung.

When coming up with the compensation for the rollback:

  1. Player time is worth more than 0. Especially when played on game modes that aren’t fun (bounty/tower of doom).
  2. Underspire torches were lost with no way to buy them back (honestly, just give everyone 5 lanterns/$2)
  3. PvP, 2 Citadels went to no one. Players lost the Citadel battle rewards, valravens and the bonus rewards from the day before and voting for next week.
  4. IP2 has a history of underdelivering and adjusting upwards after for game balance (and then not adjusting upwards ever).. now’s not the best time to underdeliver…

Also other dailies like dungeon, astral spire stars, xp booster etc.

24 Likes
10

:index_pointing_up:This.. so much this.

Neither Sokun nor anyone from the Mod/Dev forum groups even logged into forums after the rollback/servers will be down was announced.

It made a bad situation Exponentially worse.. like it used to happen before Jeto took over the forums..

A huge avoidable misstep.

Glad Jeto is back online, because Communication Matters.

:vulcan_salute:

17 Likes
11

For transparency and so everyone is aware in case they still try to send a ticket, this is what everyone is receieving

On March 21st, an unintentional server deployment was released to our live servers. This caused many players to receive the Library and Library Pass reward mail before we had finished testing the fix for this existing known issue, that we were intending to release this week.

Because this broken update went live, to resolve it, we needed to shut down our game servers and roll them back to a point before this update went live.

Unfortunately, due to the timing of this occurrence, we were unable to make clear announcements to the community that it had occurred and that we would need to shut down the game servers and roll back players’ data to before it occurred.

Our team is now currently working to fix any remaining issues that were caused by this incident. This involves restoring any lost real money purchases to players who bought anything in the period of time that we had to roll back. We will be automatically getting these accounts from our server data, so Support does not need to be contacted to have your purchase restored.

If you have already requested a refund from your platform, there is also no need to contact our Support, if you still get this purchase added to your account.

If you purchased anything with resources during this period, because your account data was rolled back, the spent resources would have been reverted. Anything that was purchased with resources would also no longer exist on your account, as with the data roll back it is as though the purchase had not happened in the first place.

This also means that any rewards collected from your mailbox for the Library and Library Pass issue, were removed.
We will still be sending compensation for the Library issues once the fix is completed.

We will also be sending out global compensation for this server incident, only after we have completed resolving any remaining issues. This compensation will be sent out globally, Support will not be sending individual compensation out to players manually.

As servers are live once more, the team is actively working on fixing any remaining issues and all compensation or missing real money purchases will be sent out automatically. I will be closing this ticket.

For live updates on this issue, please see this article: Server Rollback

Then directed to submit a new ticket or reply if it’s a different issue.

2 Likes
Two Real Money Issues and no Response
13

Hi Jeto - this summary seems to be excluding or ignoring the ‘RNG based rewards people got from their correctly earned resources’ during this period.

The most highly valued of these being a new egg dragon, a new vault troop, a new sentinel in the underspire or a perfect run in the dungeon.

With refunded resources, a player/customer has a much reduced chance of recovering that new item.

It’s like running a lottery, if you rolled it back, and then re-ran it…..from your side the awarded prizes would still amount to the same (as the prize pool is still won). But for the customers/players it is very unlikely that the people who won the jackpot the first time will win it the second time.

Please read the thread called ‘list of lost rewards’ to get a better read on what compensation the player base is requesting so that you can feed it into the wider team discussions on what is fair and reasonable.

20 Likes
14

It is not intended to come off as ignoring them, but it is not possible to account for every single player who earned resources/rewards/troops from things such as chests, guild task rewards, Soulforge eggs, etc.
We cannot search each single player account for instances where they got a specific item that they specifically were targeting and then also remove the resources spent for it.

Also, I have read the other threads.

We will not be able to provide individual compensation, when it does come time to compensate players, these are things I will be raising with the team.
But as it is not possible to target something as specific as a player getting one specific troop they needed, the compensation is only able to be sent out globally.

Also, at no point has anyone insinuated that rolling the data back and then allowing players to return to playing would it yeild the same results. The development team is aware of how it works.

3 Likes
15

Once the bulk of the compensation is sorted, this might be worth considering to be tackled through individual tickets, since having a lucky pull Mythic/Dragon Egg/etc removed can result in lost players.

:crossed_fingers::vulcan_salute:

7 Likes
16

I believe it is something that will come up in discussion, but it then also becomes a balance of.. how much time does it take to process 1 ticket.

Not even to confirm what someone is saying is true (cos there will be players who would exploit that for sure) but just to go through the manually process of sending those rewards to each player and responding to each single ticket.

Cos again, that is a manual process which is why we cannot automate it with the global compensation.

Additionally, then confirming what was rewarded was from resources already owned and not the resources unintentionally rewarded from the broken Library mail in the first place. If that is even possible.

Also, not saying that the time players have lost isn’t valuable or the chance to get back a valuable troop doesn’t mean time shouldn’t be spend to get players back what was lost where possible. Not at all. If I had a way to have 50 clones of myself it wouldn’t become such a growing list of individual variables and things we (I) would have to confirm on each single account, for each single player, who submits a ticket.

If I had a whole force of support team members, who knew how to go through that process, it would be way less of an issue to deal with manually compensation - just generally, not just this incident.

9 Likes
17

Jeto - Thanks very much for cleaning up the communication, regarding the paid purchases. I had hoped that you folks would be able to see them on your end and re-apply them.

Luckily, I’m just missing lanterns for Underspire but I know a lot of other players are missing way more expensive purchases.

Very disappointing but I’m also a programmer so I know how these things go. Stuff just goes sideways. Couldn’t have happened on the worst day though, it sounds like a lot of people spent a lot of time doing things (I think I saw someone said they finished a class from lvl 91… that’s 400+ pvp battles!).

Hope you’re all able to get this fixed up and it’s not too taxing. G’day.

4 Likes
18

yep, i lost my hard earned vizinium

5 Likes
19

i’m quitting due to this,

6 Likes
20

i just want my original gems count back and really dont have a clue how you going do that b/c we bought into bounty, and made and hourly income that we lost plus traiting and leveling a troop and lost orbs doing all that, the missing hrs of play just made me try another game that i am liking thanks to this mistake. i know this will take some doing but these mistakes happen every update except this was the worst and i feel your community is going to get lower b/c we all are frustrated on all these update bugs, i also lost a perfect run in dungeon and lost 100 gold marks because i bought a book of deeds , the book of fire deed i got with the purchased but at rollback i lost the book of fire deed and did not get the 100 gold marks added back on. i been hearing complaints like mine in forums and in our guilds anyway i do hope you can fix this b/c most of us like the game and we been loyal players for many years

3 Likes
22

If resource is as limited as you say, then I think effort should be spent on people who did gain a new egg dragon or sentinel (not me).

There should not be too many people claiming they found a new sentinel or new egg dragon in this time, but I suspect you will lose those customers and potentially others if you don’t make this minimum effort.

Dragonite from a perfect run and new troops from vault are ‘lesser losses’.

6 Likes