Executive Summary at the bottom.
The anecdotal base rate for vault keys is probably at least closer to ~1% than 3.18%, which also aligns better with the data we got from the first weekend. An assumed 3.18% (making the vault event “triple drop rate” for vault keys) would still have everybody posting here firmly on the “poor” luck side to a degree that is still improbable to the degree of being basically impossible to describe the assumed rate with the sample set with a binomial hypothesis test (x3 vault key in event means an expected 3.33% of VKs are EVK during event instead of 10%, where 0.7% is still way too much of an outlier over 1000 samples, whereas a x10 during the event would mean an expected 1% of VK during the event are EVK instead of 10%, which is within the realm of plausibility for the first data set).
At this point, there have been ample opportunities to clarify that the 10% only applies outside of the vault event and that epic vault keys are not proportionally increased inside the vault event, (nor are they intended to? which is the real question) which would probably irritate a lot people but would clear up all confusion. If this was the case, it should have been clarified one of the many times it was brought up, including when the question was asked directly. That we had to into another vault weekend without having an easily verifiable plausible explanation is extremely disheartening.
A second set of samples was mostly to help to eliminate any perceived bias and error more than anything.
But lets look at the recent weekends set, pre commited only. We now have 5 for 218. However, it would be disingenuous to just directly compare this do the original claimed ratio of 10% (10:1, or 9.1%) because of this information:
There are multiple quotes from devs on the forums that what was experienced on the first weekend was not a bug. They they looked into this, and despite it not being a bug in their eyes, slapped a flat increase on this. So an intended 10% going into this wasn’t even accurate anyway, because it should be higher than 10%.
But lets do it anyway. The best possible outcome for the initial claim being “correct”, even if we take liberties giving the benefit of the doubt with what correct means (10:1 ratio rather than 10%), means that we want to perform a binomial test on 5/218 versus a rate of 9.1%. Even if we do this with the precommited samples, at 99.99% confidence that the rate of 9.1%, we still have to reject this.
If you want to repeat the test yourself, heres a short guide:
Or just use an online calculator to get the exact binomial calculation:
http://vassarstats.net/binomialX.html
Based on what we are observing here, the following is now the prevailing theory, and the only one that makes sense at this point.:
- Epic Vault Keys were set to drop about 10% as much as regular vault keys (or, alternatively, about 10% of regular vault keys were meant to be epic)
- For whatever reason, while Vault Key rates are increased dramatically during vault events, EVKs were not
- WE DONT KNOW IF THIS RATE WAS INTENDED OR NOT FROM THE START, and there isn’t enough information to assume one way or the other from context This is what I would really like to get clarified more than anything.
- This was complained about, and ceremoniously dismissed. In response, the EVK drop rate was increased from last time (second hand information, but it tracks with the data). This could be as much as double what it was
- EVK absolute rate is still not increased proportionally to VK rate during the vault event, so we still fall far far short of the initial claim of 10% rate
- If the EVK rate was increased for all gnomes, it is no longer “about 10%”, which now makes even the initial claim incorrect for that test case in the other direction
So what about other theories?
- Direct error in the code (other than what was stated above) unlikely due to a direct code check.
- Rate of EVK set incorrectly. Ie., rate set to 0.1% of VK rather than 10% of EVK. We have enough data to fairly definitively say that much more than 0.1% of VK are EVK, just like we have enough to say that <10% of VK are EVK. Although, the original unmodified EVK rate per gnome would have likely been roughly 0.1%, and were this number also not increased during vault events, we would get numbers similar to what we are experiencing. This might have been where the original 10% claim came from, but the fact still remains that EVK have to also not increase proportionally during vault events for this to make any sense. I’m fairly certain EVK rate is set to exactly what it is intended to.
- “Not enough samples” Nope, we have plenty to perform a hypothesis test. All data available, including a set from pre-commited sources, points to “it is not 10% as stated”. The amount of data is absolutely statistically significant for the degree at which the numbers differ.
- Negativity Bias/Confirmation Bias - Latest data comes from pre-committed sources
- The rate communicated is just the wrong rate - See above. Ample opportunities have been given, including direct asks, to clarify whether or not EVK rates proportionally increased during vault events.
Why does any of this matter?
The decision making process and how this affects the larger game metrics. EVKs were released with a claim that they would be “about 10%” as rare as vault keys. Vault Keys are nearly non-existent outside the vault event already, nobody “hunts” for keys outside of the event weekend. The claim heavily implies that, “on average, about every 10th vault key I earn from battles will be an epic vault key”. With nearly all vault keys from battles for an average coming from a vault weekend already, one would measure this in number of weekends, ie., number of sessions before it is likely for them to get an EVK, and push accordingly.
So lets throw out a very rough estimate of 120 battles on a vault weekend to get a vault key on average. Doable in roughly 40-45 minutes grinding explore 2s… on average. So your expectation of EVK would come on average after 6.6-7.5 hours. Due the the rarity, this expectation would mean that you could grind explore2s for 8 hours a day during an entire vault weekend and still have a tangible chance to not get an EVK. So already pretty rare if EVK rates are proportionally increased during a vault event and 10% of vault keys are always epic.
But if they arent, we have to assume ten times that long as a baseline. So our expectation of a single EVK comes at roughly 66 to 75 hours of mind-numbing grinding on vault weekend time. So your expectation becomes… I’m not getting one in one sitting, or even one weekend. We can triple that expectation outside of vault weekend due to lower gnome rates overall, but if they aren’t increased during vault events then you have a proportionally massively increase chance to see it outside the event given your “inside event” time would only be about three times as efficient at finding EVKs rather than thirty, and there are (usually) 25 days of non-vault-event time per 3 days of vault event time. This changes the EVK achievement from a “go chase this on vault weekend” to a “you’ll get it when you get it” kind of deal after thousands of battles. This would fundamentally affect how a huge number of players interact with the game, as this claim is artificially inflating interaction numbers on vault weekends (at least the first time, I’m already seeing signs of burnout here after just two vault weekends). And while not directly correlated 1:1, interactions drive monetization. Hard to say whether the fallacious (or at best misleading) claim is going to help or hurt here beyond very short term (especially given the potential scandal), but the fact remains that this number being out there the way it is is causing money to change hands.
And while this is probably less severe of an issue than rates being published wrong on directly monetized stuff. As I have said before, drop rates in this type of game are sacrosanct. There is a responsibility to get it right, so we can trust that it is right, even if “getting it right” results in no buff to EVKs and just a clarification on how they are intended to work based on an off-the-cuff claim given in the patch notes. Once it is correct, then we can move on to whether or not they are “too rare”. But we need to start by getting on the same page. Like, if you are intended to get it randomly as a fluke and barely have any influence over when other than being three times as efficient during vault weekend, it might shift perception a bit on the role these items are intended to play.
@Saltypatra -
tl;dr: Epic Vault Keys are not about 10% as rare as Vault Keys, at least during vault weekends, they weren’t last weekend, and they still aren’t now. The simplest explanation (and the only one that even makes sense anymore given claims on both sides and data gathered) is that they do not proportionally increase (or more likely don’t increase at all) during event weekends, and we (the playerbase) still don’t know whether or not they are even intended to. If EVK drops are not intended to increase proportionally to Vault Key drops during vault event weekend, please clarify this so we can move on from there about discussing how rare they are even at intended rates. If they are intended to increase proportionally to Vault Key drops during vault event weekend, well, there is ample evidence to suggest that they currently aren’t, and need to be fixed.