Knowing the exact nature of the bug, it should be fairly easy to calculate the rates we are getting served now and see if they line up with sampling.
The internal drop tables would look something like this (for, say, Amanithrax in this example):
"Rewards": [{
"Type": "Troop",
"Chance": 0.7,
"RarityChance": [0, 0.59, 0.26, 0.11, 0.04, 0],
"Included": [6756, 6757, 6758, 6759],
}, {
"Type": "Treasure",
"Chance": 0.3,
"RarityChance": [0.25, 0.25, 0.25, 0.14, 0.08, 0.03],
"Included": [6114, 6115, 6116, 6117, 6118, 6119],
}]
When full drop rate data used to be sent to the client, the raw data appeared in a similar format. What essentially happened here is that the server does not make a distinction “troop” and “treasure” as different “types”… they are both “troops”. And the troop drop pools were simply combined. Thus, essentially, we got data that was rolled from tables like this:
"Rewards": [{
"Type": "Troop",
"Chance": 0.7,
"RarityChance": [0, 0.59, 0.26, 0.11, 0.04, 0],
"Included": [6756, 6757, 6758, 6759, 6114, 6115, 6116, 6117, 6118, 6119],
}, {
"Type": "Treasure (Also Troop)",
"Chance": 0.3,
"RarityChance": [0.25, 0.25, 0.25, 0.14, 0.08, 0.03],
"Included": [6756, 6757, 6758, 6759, 6114, 6115, 6116, 6117, 6118, 6119],
}]
Note that some of these now include IDs on their drop tables for which they have zero chance. These simply won’t drop on those tables - if you roll a “troop”, you can’t get a common or mythic because the respective rarity chances are zero. However, since both these are already displayed as chance of the total drop pool, the overall chances of mythic and common treasures should not change at all. Every other rarity should work out to be the average of their overall chances (with each rarity having an equal chance of two drops except common/mythic, which only have one), giving us an overall rarity distribution of:
| C | R | U | E | L | M |
|---|---|---|---|---|---|
| 7.50% | 48.80% | 25.70% | 11.90% | 5.20% | 0.90% |
When the displays are updated, we should see:
| Overall | Common | Rare | Ultra-Rare | Epic | Legendary | Mythic | |
|---|---|---|---|---|---|---|---|
| Troops | 45.80% | - | 24.40% | 12.85% | 5.95% | 2.60% | - |
| Treasures | 54.20% | 7.50% | 24.40% | 12.85% | 5.95% | 2.60% | 0.90% |
As far as at least treasures, this is very close to the midpoint of what was sampled in most instances and well within the confidence intervals for the 95% confidence level estimate over thousands of samples in all instances. Translation: Very likely that these are the “true rates” we have been experiencing up to this point, and this is what we should when the patch hits the server.
So I feel that the “how” was covered pretty well here, but the “why” wasn’t.
I didn’t mean to imply “no mistakes ever happen again”. Mistakes happen, and I understand that. The frustration comes from mistakes being repeated. I’m not talking about this or any specific particular game error. I’m talking about a pattern related to the cycle a game issue, particularly a drop rate issue, goes through on the forums that keeps happening.
Truthfully, this was a completely avoidable issue stemming from a process that was repeated many times - a misunderstanding with how something needed to be configured (or simple error) happened and then the chance to correct it was passed because everything was assumed to be correct without sampling the output. The second part is the big one, because in this instance only hundreds of samples needed to be checked to show that the treasure rate was far, far off what was being reported. Long before you even publicized the numbers, you’d have known it was far, far off what you intended to be configured. Before delves even released on live, even. Because by your own admission the drop rates shown up to this point are what was intended to be configured, it just never worked correctly (the optimization that apparently caused the error is older than delves themselves).
I feel like automated relatively small scale sampling is something that could be done a lot more. You don’t need to do a million-sample-proof every week, but every time a drop table changes, some thousands automatic sampling to verify things are in line with expectation even in the face of unknown variables would help insulate against issues like this in the future. You won’t be guaranteed to catch everything with sampling like this, but it would give another layer of protection/verification to an absolutely critical system. As noted, a sample as small as hundreds and very simple comparisons would have shown the portal output to be too different from what was configured as to not possible to be correct.
These things, in particular, could use work:
- Drops tables aren’t being scrutinized until after an issue is brought up. To this point, even, five years into the game. This has lead to repeated mistakes in the past. Troops missing from certain chests after they are supposed to be there, key types drop rates shifting suddenly, an entire rarity chance category being set to zero for weeks (and mythic rarity at that, and requiring direct support intervention to refund keys chasing it, leaving people out of the loop to just assume they were unlucky)… all these are things that have happened, some of them more than once. All had to be caught by the players and brought up to receive correction. Most of them with the backing of data that is no longer available because it is no longer sent to the client.
- The use of “nothing has changed” or “working as intended” before specifically checking with the people that might have changed something or seeing if it is actually working as intended, but simply relying on no specific knowledge anything that has changed and no strong indication things are not working as intended to make the assertion. While this is usually due to a breakdown in communication, speaking honestly, it makes it really hard to take these responses at face value.
- The perceived unwillingness to look at anything drop-rate or RNG related without a mountain of evidence and widespread discussion. (more on this)
To start off with, perceived discrepancies just kind of get written off as “RNG is RNG” and nothing more heavily than when drop rates is involved. I realize there are so many claims with so little to back up anything that I can see it being irritating that things that have been tested being brought up over and over again. Which is fine, most of the players would step in and back you up there on a lot of them… even more of us when we can prove things are. But then we move into issues where if even if some evidence is presented that might not strictly pass as “proof” that something wrong, but rather an “indication something might not be correct”, it is almost always still summarily written off.
Examples
As an example, see discussion about Adventure Board traitstone rates (here)in the 6 weeks or so following the patch where everyone got the same rewards. While recent samples have brought us more in line with what we were seeing prepatch in terms of traitstones (and thus, no strong indication we currently have a significantly higher traitstone appearance rate than we did in the previous version, which was the spirit of the report), this response was still incredibly concerning because of the “we didn’t change anything” (when things had to have changed to some degree even to keep them relatively the same, because deeds were added) the tone is more “you don’t have the samples to prove it” than “I checked with the team and they said that the traitstone rates before and after should be similar”. It was framed and reported as a bug, as in, something that might be unintended and therefore not known by whoever implemented, evidence was presented that things had changed significantly (not the strongest evidence, granted, but enough that players were having some pretty valid concerns), and it was dismissed as “not a bug” without any checks. Not “we didn’t intentionally change traitstone rates to be higher, they should actually be slightly lower now because we added more stuff” or “we will keep an eye on it as things develop” but “not a bug”.
On top of this you don’t disclose the rates and some the items in question either come so far apart or are grouped in such a way that makes it effectively impossible to even use sampling to submit as evidence that something is wrong. Orbs? Delve Chests? Adventure Boards? Well, since we don’t know what the rates are supposed to be, we can’t possibly be correct when we bring up something might be wrong. Makes it really easy to speak from a position of authority because it can’t be challenged, but it also has the habit of backfiring when the cracks are revealed. So when I see something like this, I’d love to be able to jump in and back you up. To be fair, this thread made an easily refutable and ridiculous assertion that was clearly hyperbole (obviously ascension orbs didn’t completely stop dropping, anyone getting a single one could say they didn’t) and some of the responses I read more in the tone of venting and I was going to step in myself and show the numbers involved and why streaking for even a month under the rates we think Ascension orbs drop at wouldn’t that uncommon compared to other events of similar probability. But when the dev response is “The amount of data players have access to is very small in comparison to us.”, with this issue still ongoing, my immediate thought was “Well, did you check it?”. And then I realize I just can’t in good faith defend orb drops being correct I can’t be sure that somehow adding tokens in explore didn’t mess with the rate of Ascension orbs based on the last selected difficulty (I’m not saying this happened, I’m pulling a ridiculous example of an interaction that I have no way to even begin to test even with months of evidence from multiple people and would be completely unintentional on your end and therefore you would have no way to know if it happened for sure either without actually checking the output side, which was readily apparent was not happening).
Even when the playerbase “wins” because of it, the entire situation ends up being a black mark. Like the whole fiasco where tributes were bugged for two years to compound the statue bonus for every pull. Even this is indicative of the same pattern - discussion happened, it was revealed something might be bugged, there was no way for us to tell 100% whether or not it was bugged and devs were pinged just to be sure, and everything stayed as it was with everyone thinking “must be working as intended or they would have fixed it by now” for two years before this (bug report when the payouts suddenly changed without warning) followed by this (“we didn’t change anything”) and then finally this (oops bugfix). But again, the entire issue would have been avoided by either sampling the output ahead of time and/or giving scrutiny to the issue when it was originally reported.
The one time I attempted to raise a drop rate issue DMs (no, I won’t show it here, but those involved know what I’m talking about) I was met with a strikingly similar pattern of outright denial followed by having to push too hard followed by eventual correction.
I’m not trying to scapegoat any one person here - the original thread where tribute payouts were being discussed and tried to be brought up goes all the way back before Salty or Kafka’s forum accounts even existed. Nor am I just dredging up past mistakes to hold them over you. I’m trying to point out the pattern that keeps repeating over and over and over again with little sign of improvement, and why it needs to change.
I suppose this post in and of itself honestly discussing the mistake that was made is indicative of some willingness to change for the better and I am thankful for that, but the rest of it still needs to change. We shouldn’t go back to a status quo where we deal with drop issues after they arise and reach critical mass, they need to be prevented proactively. We shouldn’t have players wanting to raise an issue needed to do so much work that it often isn’t worth it to pursue. We need you to have better tools so it isn’t a giant pain for you to even test if things are working correctly. This is what “improving your processes” should mean.
Right now this process isn’t just inelegant, it is downright disheartening. What I’m asking here is what can I do to make any evidence backed report about a discrepancy, especially a discrepancy with drop rate or RNG, stand out and not immediately get shot down. Preferably without needing to sample rewards from what represents hundreds or thousands of hours of player-time to gather the resources or weeks to months of real time to overcome time gates necessary to obtain the rewards to conduct the study in the first place. Again, I’m not asking for dozens of dev-hours to be diverted every time someone spent 100 gem keys and didn’t get their mythic. But feeling like we do have a line of communication for this sort of thing is very important because randomness and specifically random drops are central to this game. Sacrosanct, even. Because even a temporarily bugged drop rate can have permanent impact on a player’s experience and drop rates are tied directly to your monetization.
I’m sorry for the rant. I thought about trimming it down, but felt it was best to leave this unabridged version to best convey the utter frustration that is trying to convince someone to check a potential mistake when they think they have more or better information than you but are unwilling to check it because a bunch of other people are also trying to convince them they made a different but similar mistake twenty times a day. It is exhausting for all parties involved. Thats why the process needs to improve - starting with being able to recognize which reports are credible and having the tools to quickly investigate them.
So, onto the last part…
Keeping the current drop rates and fixing the bug is an acceptable outcome, IMO. I’m glad the underlying issue is getting fixed now, because leaving it in very likely could have led to issues when the next drop system was implemented and it was “forgotten” that it never worked properly. As a casual observation, a lot of issues seem to be caused by misunderstanding how a piece of code will use a particular piece of data, and I’m not really sure to solve that as an outside observer looking in… I could say more rigor and documentation, but really I have no idea how you distribute your workload that leads to these misunderstandings in the first place. I would have liked Guild Guardian style drops, but understand there might be upcoming features that make this less necessary as we move forward. If/when those are revealed, we can revisit the issue if need be. That is fine. It is the other stuff I feel can still be done better.
To sum up, I’m hoping for:
- A way for credible reports to be quickly supported or refuted with a reasonable degree of confidence through sampling with dev tools (Thus arming you with evidence to back your “working as intended” or “nothing has changed” without wasting a lot of your time and not making it feel like every drop/RNG issue is ignored and that we are wasting our time)
- Automatic sampling when drop tables change and on any new drop system ahead of time so that we are highly unlikely to even reach the point where a report needs to be made in the first place. (a way for you to catch mistakes before they make it this far)
- Disclosure of remaining undisclosed drop rates (Adventure Boards, Chaos Orbs, Tokens, Gnome Rates, Gnome Drops, Vault Rewards, Ingots from PvP, Daily Ingots, Troops in Event Shops, Delve Chests, Legendary Tasks, full troop pools and not just rarity distributions, and anything else I may have missed) as a good faith gesture that everything is working as intended. (a way for us, at least potentially, even if it is probably way too much work, to catch mistakes if they do make it this far)
The mistake isn’t that a bug happened, it was not correcting the circumstances that allowed it to happen, nor improving the process that needs to happen when attempting to get stuff like this looked at. Please recognize this.