The Bug
Before you change this to [Not a Bug] without reading because it is about drop rates, this is math backed and the rates displayed aren’t even close to what is being sampled. So bear with me. And please actually read some of it. At the very least, if you are going to skim the math, read the proposal at the end/in the linked thread: [Poll] How do you feel about the current Treasure drop rate? - #11 by Mithran.
Platform, device version and operating system:
All of them.
Screenshot or image:
What you were expecting to happen, and what actually happened:
Expected portal drop rates to display something reasonably close to what they are dropping.
How often does this happen? When did it begin happening?
Always. Well, the drop rates were always what they are now, for at least as far back as I could find data. The rate displays have at least been wrong since this screen has existed.
Steps to make it happen again
Go to open portal on any kingdom and click the question mark. Open any large number of portals to get a good sample size (500 is more than enough, even 100 shows things are off). Observe that the stated rates and what you got aren’t even close.
Now that that out of the way, lets look at how I know that the rates being displayed cannot possibly be the rates being used to push actual drops to players.
The Math
I have to start off by assuming each drop is independent and random. If they aren’t, things are already more screwed up than I thought, because the drop rate display alone doesn’t really mean anything if the rates aren’t independent and random.
For now, we will be focusing on just treasure drops. This is enough to show that the stated drop rate display as a whole is not in line with what is dropping.
We will start with some community gathered samples, mostly taken from here:
I compiled a table with all drops (including two sets from guild members). Results here:
Table
| Submitter | C | R | U | E | L | M | Total Treasures | Shards Spent | Portals Opened |
|---|---|---|---|---|---|---|---|---|---|
| TheIdleOne (Thieves) | 28 | 84 | 56 | 19 | 15 | 1 | 203 | 7600 | 380 |
| TheIdleOne (Amanithrax) | 27 | 81 | 52 | 32 | 7 | 3 | 202 | 7400 | 370 |
| TheIdleOne (Silver Nec.) | 32 | 106 | 52 | 30 | 10 | 5 | 235 | 8200 | 410 |
| TheIdleOne (Dark Pits) | 31 | 90 | 47 | 26 | 7 | 3 | 204 | 7600 | 380 |
| Fleg (Dark Pits) | 25 | 98 | 60 | 22 | 11 | 4 | 220 | 7700 | 385 |
| Fleg (Wild Court) | 42 | 161 | 90 | 32 | 19 | 7 | 351 | 13,000 | 650 |
| Dibbs (Wild Court) | 23 | 105 | 63 | 11 | 4 | 13 | 219 | 7800 | 390 |
| CrzBoarder (Wild Court) | 37 | 133 | 65 | 25 | 9 | 8 | 277 | 10,000 | 500 |
| TOTALS | 245 | 858 | 485 | 197 | 82 | 44 | 1911 | 69300 | 3465 |
| % of Treasures | 12.82% | 44.90% | 25.38% | 10.31% | 4.29% | 2.30% | 100.00% | ||
| % of Total | 7.07% | 24.76% | 14.00% | 5.69% | 2.37% | 1.27% | 55.15% | ||
| Stated % of Total | 7.50% | 7.50% | 7.50% | 4.20% | 2.40% | 0.90% | 30.00% | ||
| Confidence Interval (95% Confidence) | 6.22%~7.92% | 23.32%~26.20% | 12.84%~15.15% | 4.91%~6.46% | 1.86%~2.87% | 0.90%~1.64% | 53.50%~56.81% | ||
| Confidence Interval (99% Confidence) | 5.95%~8.19% | 22.87%~26.65% | 12.48%~15.52% | 4.67%~6.70% | 1.70%~3.03% | 0.78%~1.76% | 52.98%~57.33% | ||
| Confidence Interval (99.9999% Confidence) | 4.94%~9.20% | 21.17%~28.35% | 11.11%~16.88% | 3.76%~7.61% | 1.10%~3.63% | 0.34%~2.20% | 51.02%~59.28% |
A bit on the calculations. Each subset is considered to be a binomial normal distribution (all random, all independent, chance of given drop versus total chance of not). Confidence intervals are calculated using a normal approximation interval (Wald interval). Note that while this may skew results with low samples, our smallest number of “successes” in this is 44 on the mythic treasures column out of 3465 total trials. This can be estimated in other ways, but that is getting away from the main issue - we don’t need to know that, for example, mythic treasures rates with a high degree of confidence to know that the overall display table is wrong, we just need to show that some of the things is wrong, and we can actually show more than one.
Our biggest indicator from this data set is total amount of treasures. Stated in-game treasure rate is overall 30%, sampled rate is over 55%. Our 99.9999% confidence level estimate for the samples we have here has the “true rate” being 30% far outside of possibility. Similarly, the cumulative probability to have 1911 successes out of a sample size of 3965, which is what this set shows, is infinitesimally small. Can’t even find a good way to measure it due to the scale of the numbers involved. But if you break this into subsets based on submitter, each of these sets are also so far off the scale as to be basically impossible to describe as “luck”. For example, our first subset shows 203 successes out of 380 trials (total treasures out of portal pulls), and with the stated individual probability of 0.3, the cumulative probability to pull any amount of treasures greater than or equal to 203 is 1.80E-21. As in, the sum of the binomial probabilities to pull 203, 204, 205, 206, etc combined is 1 in 555,555,555,555,555,555,555 (1 in 555 quintillion). This test can be repeated for every sub-set to show that if 0.3 were the correct rate, every recorded individual sub-set would be describing an outlier of similar scale, give or take a few orders of magnitude (and considering “a few order of magnitude” wouldn’t affect our conclusion alone should be a tip off).
Probably the most noticeable indicator, the one that nearly anyone playing doing a lot of portal pulls the game could tell you anecdotally, is the relative drop rates of coin purses versus gold rings versus priests chalices. The in-game drop rate display has all of them at an equal 7.5% drop rate. Our table here has them at consistently more rings than chalices and more chalices than purses, at almost a 4 to 2 to 1 ratio. Also neither Rings nor Chalices overall rates their respective 99.9999% confidence level estimate even close to their stated percentages independently.
From just our treasure subset, we can take also take any number of sub sets and see that things just don’t line up. For example, we can check only the drops of rings and chalices, which are supposed to be equal, so our expected success rate is 0.5 and our total set size is 1343. Doesn’t matter which one we think of as a “success”, 485/1343 and 858/1383 are such outliers for both success and failure for a 50% assumed success rate as to be normally described as “statistically impossible”. If you compare the ratio of gold rings or chalices to any of the drops on the table you see that they are far off even the stated ratios within the treasure set. Nearly everything else is “off” generally in a predictable manner, but these two are the most obvious and measurable.
When compiling this, I considered the following possibilities:
- The real drops are actually not configured from the same data that does the drop rate display, despite constant claims to the contrary, and thus the display is incorrect.
- The real drop calculations and displayed rates are configured from the same data, but there is an error in how the drops are calculated by the server. Unlikely since none of the other chests seem to be off with relation to their midline drops (but maybe they are? I have no proof one way or the other for these)
- The real drops cannot be normally distributed for whatever reason (eg., having prior results skew future ones) If this is the case, then the drop rate displays are useless anyways because the conditions to get said rates are not described. And yes, I know, we are dealing with pseudo RNG, which might not be perfect, but this isn’t even ballpark over a massive sample size
- Something influences the real drops in a way that is not represented by the displayed drop rate table (which would make the sample set not describable by a binomial normal distribution but would also make the displayed drop rate table, again, incorrect)
- Sampling error. It is possible these samples contain errors, but I have one screenshot backed set of 500 that alone would describe a statistical impossibility (see below).
- Sampling Bias. Most samples were taken from the forums, but treasure rates aren’t supposed to be based on anything related to the individual player, except with relation to how many chests they opened (how many samples in a given sub-set) - everybody gets that same rate display, everybody should be pulling things at the same rate. If sampling bias were even possible this would imply different odds for different people or different sets of circumstances, then, again, the display is wrong because the display never changes.
- Mathematical error. Feel free to check all of my math. I’m fairly confident it is correct barring any initial assumptions being incorrect, which I already covered here (which, again, would make the in-game droprate display pretty much useless). You can use less stringent tests and smaller samples to show that it is highly highly unlikely the sample can be used to describe the stated drop rates
Heres one last one. Even if we assume every other set had some kind of sampling error, this one has screenshots to back it.
These are from a guildmate of mine and are before and after screenshots of treasures, and the screenshot for Wild Court troops, showing a 10,000 shard pull (500 total portals).
Screenshots and Table
Treasures Before
Treasures After
Troops After (none before)
| Submitter | C | R | U | E | L | M | Total Treasures |
|---|---|---|---|---|---|---|---|
| (Wild Court) | 37 | 133 | 65 | 25 | 9 | 8 | 277 |
| TOTALS | 37 | 133 | 65 | 25 | 9 | 8 | 277 |
| % of Treasures | 13.36% | 48.01% | 23.47% | 9.03% | 3.25% | 2.89% | 100.00% |
| % of Total | 7.40% | 26.60% | 13.00% | 5.00% | 1.80% | 1.60% | 55.40% |
| Stated % of Total | 7.50% | 7.50% | 7.50% | 4.20% | 2.40% | 0.90% | 30.00% |
| Confidence Interval (95% Confidence) | 5.11%~9.69% | 22.73%~30.47% | 10.05%~15.95% | 3.09%~6.91% | 0.63%~2.97% | 0.50%~2.70% | 51.04%~59.76% |
| Confidence Interval (99% Confidence) | 4.38%~10.42% | 21.51%~31.69% | 9.13%~16.87% | 2.49%~7.51% | 0.27%~3.33% | 0.15%~3.05% | 49.67%~61.13% |
| Confidence Interval (99.9999% Confidence) | 1.67%~13.13% | 16.93%~36.27% | 5.64%~20.36% | 0.23%~9.77% | -1.11%~4.71% | -1.15%~4.35% | 44.53%~66.27% |
Even with just this sample set, with a single calculation, we can see just how far off things are. A total of 277 treasures were pulled out of 500 draws. With a supposed 30% drop rate, if we look at the cumulative probability of getting 277 or more drops our total chances are somewhere in the order of 4.00E-32 (the best resolution I could get for the value, not that it matters much with that order of magnitude). Represented in decimal notation and as a percent, thats:
0.000000000000000000000000000004%
Or a 1 in 25 nonillion(1/2.5e31) chance. Hard to even put that into context. There are estimated to be about 1e22 stars in the known universe. This number is 25 billion times that. So… one in that many. That would be the chance of getting at least that many treasures with 500 pulls assuming the 30% rate is correct.
Note that this subset also follows the predictable pattern of drop imbalance on the first three rarities of treasures and can show even these individual ratios are off with a fair degree of confidence.
Just in case it was fixed in the last month or so (like last patch), I did a 100 sample draw (about 2 days ago now by the time this post goes up) and still got these results:
Screenshots
Still 48 treasures out of 100.
The smaller set skews some of the smaller numbers involved, but for total treasures alone this outcome is still several order of magnitude more likely using the sampled estimate droprate of 55% than the stated rate of 30%.
There is absolutely no way to reconcile the display rates coinciding with what is actually being pulled.
Of course, without any complex computations at all, simply granting a dev account, say, 20k shards and opening portals will highlight that the current drop rates and the displayed drop rates describing the same thing is pretty much impossible by looking at them, much in the same way could determine that a coin that hits 70 heads out of 100 is probably not a “fair coin”. In significantly less time. Less, even, if you had a tool to do this automatically. Especially when the trial can be repeated, and larger samples don’t move you any closer to your expected result.
TL;DR (for this section) Displayed portal rates are wrong. And theres the math.
About Treasure Drop rates
The rate of treasures in portals is a big deal, as the entire underworld and several systems extended out of underworld are balanced around it. Please don’t just knee jerk “fix” this by hastily changing whatever table you have on the back end to whatever one is used for display and call it a day because that was the one that was “initially intended”. This system has been live for over a year now at these rates, and feedback has been amassing based on these rates. Changing the current in-game displayed rates to the ones displayed would be tantamount to a massive nerf for those that consistently attempt to push delve progress, specifically, faction teams. It has already been (repeatedly) expressed that certain factions with the faction team were too difficult, and while I don’t believe pushing stats much higher in tiny increments would help them all that much overall, taking away the ability to get these stats by taking away treasures certainly wouldn’t help either. The average xp per portal pull is estimated currently about 31.845, while the stated rates would yield about 21.075 xp per portal pull on average - about a one third reduction. An overall reduction on treasures to this degree is an overall regress on the ability to obtain stats to basically before deeds or medals were added, invalidating any sort of impact those systems have with the intent of making level 500 faction team runs less painful.
Overall sentiment also seems to be split on the issue of getting more troops if it means less treasures will drop. Some would prefer more troops to drop because they don’t push delve progress and just want the troops, and an adjustment to the displayed rates would facilitate that and lower the average shard cost threshold to mythic (1 copy of each) a set of troops by an average of about 20%. Some would prefer the amount of treasures we have now, and some would prefer more treasures.
Proposed win-win-win: Guardian Style faction troop drops.
The numbers and proposal on this are outlined here:
The basic gist is that we can reach a similar level of progress speed for most subsets of players with the displayed rates being the actual rates. We can do this by implementing the Guild Guardian-style faction troop drops - basically, any faction where you have every troop at mythic and four copies, no more troops will drop from portals there. Any potential troop drop will just become a treasure drop instead.
Since the displayed troop rarity ratios are also a bit more punishing that what is currently going on in game (we currently generally mythic ascend our high rarity troops before our low rarity ones, this happen significantly less frequently with a 70% troop drop rate under the stated rates), implementing the stated rates with guardian style drops does have its downsides for players. Any reduction in treasure rates during the beginning of the scale delves harder to “break into” (get your first hoard quality 10 for decent rewards, with each hoard quality milestone meaning more shards per day and each one taking a bit longer to get than now). However, I believe this outcome satisfies the most player subsets, and also the dev mindset of keeping progress to within a specific intended “schedule”. Also, since the average amount of shards needed to get the mythic + 4 threshold (estimated at ~7200 under the best conditions) is still greater than the average amount of shards you’d get from even a full month of daily delves (estimated at 6k~6600 even under best conditions), people on the threshold would be greater incentive to play the delve events if they want treasures. Unless treasure distribution ratio is rescaled for when troops leave chests, this does mean times bump to 3 1/3 times as many treasures after all troops are mythic +4 and you keep pulling, but keep in mind that we suffer a 30% initial reduction in overall treasure xp before mythic, so the total difference from what we are seeing now means we wouldn’t even catch up in total hoard xp until about 8k shards are spent, and our difference in xp at 12k shards spent is roughly hoard level 211 after versus hoard level 190 before. Beyond even this many shards, keep in mind that since hoard levels are not a linear function of hoard xp, you need more xp (and gold) every level, and the amount of chaos shards obtainable per month has a very clearly defined softcaps both before premium currency is converted, and we need much more effort and currency to get more shards, and therefore treasures, which then runs into another softcap converting xp to hoard levels, this ends up being a lot more effort for a little more hoard level progress. Which seems right in line with the overall progress paradigm at endgame.
Since we have similar (only slightly bigger, since current expectations are to mythic the troops in rarity order instead of reverse rarity order like the drop rate display implies) estimated mythic +4 thresholds dealing with the current actual drop rates, Guild Guardian style drops could also be a benefit if the drop rate display gets changed to what the rates are now rather than vice versa. With the treasure rates being so far off stated, assuming they wouldn’t be rescaled for the “treasures only” table, this is significantly less of a bump after all troops are gone, about 1.8 times treasures dropped. We’d get a very similar bump from hoard level 195 to 210 at about 12k shards spent, and fewer than the “guardians style troop drops, 70% initial troop rate” model afterward. I personally think “guardians style drops, 45% initial troop rate” keeps the progress curves a bit smoother for all players, but its also a straight buff, which is less likely to cover the dev “progress schedule” mindset. I will point out though, again, chaos shard gains are fairly clamped to a certain range that requires premium currency to break through, and the level of gains we are talking here is less of a buff as it is making the system feel less “wasteful” - you still need massive amounts of premium currency and effort for any given stat point granted by hoard level at this point.
Closing
I didn’t want to have to force the issue, but we seem to hit a pretty hard wall whenever anybody try to bring up in-game rates being wrong. About anything. And this is a very egregious case. It should be a big deal that these rates aren’t right, and in this case doesn’t even pass an “eyeball test” that everything is as it should be.
Please, take things like this seriously when they reported. I know you can’t chase down every “drop rate is wrong” claim, but there was lots of data floating around already that suggested that things weren’t correct for anybody paying attention. Like anybody pulling 100 portals or more. Its been obvious things are off for a while and have been gathering evidence, but I didn’t want to make a bug report like this until I was absolutely certain by just how much and had an entire free day to go over the calculations back and forward, because it seems like I get stonewalled if I try to mention things like this with anything less. Being told, again, that the displayed rates were configured from the same data used to push actual drops (without any actual checks being done, because this one doesn’t even pass muster with the most basic of tests) in the face of mounting evidence told me I had to force the issue, lest it come back to bite us in another year with “whoops, it was a bug the whole time! heres 50 gems, we cool right?”. I also wanted enough to make a counter proposal to just “nerf everyone” that I expect as the outcome after this does get taken seriously (or get stumbled upon by a coder and “fixed” summarily without any thought on the impact of the “fix” on the game economy as a whole). I hope by taking the time to examine the numbers involved and feedback already given that some thought will be put into which direction to take on correcting the drop rate display. We saw the ripple impact a “fix of a long term bug to originally intended amounts” brought with PvP points scaling, don’t make the same mistake again.
Please, please, please, for everyone’s sake, have some tools where you can quickly sample drops from the live drop pool. This is the umpteenth time drops have been wrong, and its really bad when there is now an in-game official source that reports things differently.
TL/DR - Actionable feedback items:
- Make an official news post about how this issue will be fixed, how these were wrong in the first place, and why this will never happen again
- Then fix portal drop rate displays
- Whether or not this fix involves changing treasure rates from what they are now, implement faction wide “guardian style” troop drops for each faction (once you have all troops from there mythic +4, you won’t see anymore from that faction)
- If the problem is potentially applicable to any other system in the game, check those too
- Implement a tool to test live server rates through sampling so you can quickly check when there is a credible report that rates are off.
- Please let us know how we, as players, are supposed to report things like this and not have them be ignored. Preferably with significantly less effort than I had to put in here.
Sorry in advance everyone. It had to be done. Hopefully some good will come out of it.
