# Vault Weekend data collection - for w/e 13 Sept

**URL:** <https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449>\
**Category:** Gameplay Chat (General)\
**Created:** [August 14, 2020, 7:24am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449 "2020-08-14T07:24:21Z")\
**Posts on this page:** 1\
**Showing post:** 167

<div class="post-metadata">

**Author:** ![Mithran](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/mithran/32/18064_2.png) [@Mithran](https://community.gemsofwar.com/u/Mithran)\
**Post date:** [September 14, 2020, 7:15pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/167 "2020-09-14T19:15:57Z")

</div>

Executive Summary at the bottom.

The anecdotal base rate for vault keys is probably at least closer to ~1% than 3.18%, which also aligns better with the data we got from the first weekend. An assumed 3.18% (making the vault event “triple drop rate” for vault keys) would still have everybody posting here firmly on the “poor” luck side to a degree that is still improbable to the degree of being basically impossible to describe the assumed rate with the sample set with a binomial hypothesis test (x3 vault key in event means an expected 3.33% of VKs are EVK during event instead of 10%, where 0.7% is still way too much of an outlier over 1000 samples, whereas a x10 during the event would mean an expected 1% of VK during the event are EVK instead of 10%, which is within the realm of plausibility for the first data set).

At this point, there have been _ample_ opportunities to clarify that the 10% only applies _outside of the vault event_ and that epic vault keys are **not** proportionally increased inside the vault event, (nor are they intended to? which is the real question) which would probably irritate a lot people but _would_ clear up all confusion. If this was the case, it should have been clarified one of the many times it was brought up, including when the question was asked directly. That we had to into another vault weekend without having an easily verifiable plausible explanation is extremely disheartening.

A second set of samples was mostly to help to eliminate any **perceived** _bias_ and _error_ more than anything.

But lets look at the recent weekends set, _pre commited_ only. We now have 5 for 218. However, it would be disingenuous to just directly compare this do the _original_ claimed ratio of 10% (10:1, or 9.1%) because of this information:

> [@Grundulum](#):
>
> Supposedly they also increased the rate of epic vault keys compared to the first vault weekend. (The beta testers mentioned the issue in the beta discord and got a response.)

There are **multiple** quotes from devs on the forums that what was experienced on the first weekend was not a bug. They they looked into this, and _despite it not being a bug in their eyes_, slapped a flat increase on this. So an _intended_ 10% going into this wasn’t even accurate anyway, because it **should** be **higher** than 10%.

But lets do it anyway. The _best_ possible outcome for the initial claim being “correct”, even if we take liberties giving the benefit of the doubt with what _correct_ means (10:1 ratio rather than 10%), means that we want to perform a binomial test on 5/218 versus a rate of 9.1%. Even if we do this with the **precommited** samples, at 99.99% confidence that the rate of 9.1%, we **still** have to reject this.

If you want to repeat the test yourself, heres a short guide:

> **[How to Perform a Binomial Test in Excel - Statology](https://www.statology.org/binomial-test-excel/)**
>
> A simple explanation of how to perform a binomial test in Excel, including a step-by-step example.

Or just use an online calculator to get the exact binomial calculation:  
[http://vassarstats.net/binomialX.html](http://vassarstats.net/binomialX.html)

Based on what we are observing here, the following is now the prevailing theory, and the only one that makes sense at this point.:

- Epic Vault Keys were set to drop about 10% as much as regular vault keys (or, alternatively, about 10% of regular vault keys were meant to be epic)
- For whatever reason, while Vault Key rates are increased _dramatically_ during vault events, EVKs were **not**
- **WE DONT KNOW IF THIS RATE WAS INTENDED OR NOT FROM THE START, and there isn’t enough information to assume one way or the other from context** This is what I would really like to get clarified more than anything.
- This was complained about, and ceremoniously dismissed. In response, the EVK drop rate was increased from last time (second hand information, but it tracks with the data). This could be as much as _double_ what it was
- EVK absolute rate is _still_ not increased proportionally to VK rate during the vault event, so we still fall far **far** short of the initial claim of 10% rate
- If the EVK rate was increased for _all_ gnomes, it is no longer “about 10%”, which now makes even the initial claim incorrect for that test case _in the other direction_

So what about other theories?

- Direct error in the code (other than what was stated above) unlikely due to a direct code check.
- Rate of EVK set _incorrectly_. Ie., rate set to 0.1% of VK rather than 10% of EVK. We have enough data to fairly definitively say that _much_ more than 0.1% of VK are EVK, just like we have enough to say that \<10% of VK are EVK. _Although_, the original unmodified EVK rate _per gnome_ would have likely been roughly 0.1%, and were this number **also** not increased during vault events, we would get numbers similar to what we are experiencing. This _might_ have been where the original 10% claim came from, but the fact still remains that EVK have to also _not_ increase proportionally during vault events for this to make any sense. I’m fairly certain EVK rate is set to exactly what it is intended to.
- “Not enough samples” Nope, we have plenty to perform a hypothesis test. All data available, including a set from pre-commited sources, points to “it is not 10% as stated”. The amount of data is absolutely statistically significant for the degree at which the numbers differ.
- Negativity Bias/Confirmation Bias - Latest data comes from pre-committed sources
- The rate communicated is just the wrong rate - See above. Ample opportunities have been given, including direct asks, to clarify whether or not EVK rates proportionally increased during vault events.

Why does any of this matter?

The decision making process and how this affects the larger game metrics. EVKs were released with a claim that they would be “about 10%” as rare as vault keys. Vault Keys are nearly non-existent outside the vault event already, nobody “hunts” for keys outside of the event weekend. The claim heavily implies that, “on average, about every 10th vault key I earn from battles will be an epic vault key”. With nearly _all_ vault keys from battles for an average coming from a vault weekend already, one would measure this in number of weekends, ie., number of sessions before it is likely for them to get an EVK, and push accordingly.

So lets throw out a very rough estimate of 120 battles on a vault weekend to get a vault key on average. Doable in roughly 40-45 minutes grinding explore 2s… _on average_. So your expectation of EVK would come _on average_ after 6.6-7.5 hours. Due the the rarity, this _expectation_ would mean that you could grind explore2s for 8 hours a day during an entire vault weekend and still have a tangible chance to _not_ get an EVK. So already pretty rare if EVK rates are proportionally increased during a vault event and 10% of vault keys are **always** epic.

But if they _arent_, we have to assume **ten times** that long as a baseline. So our expectation of a single EVK comes at roughly 66 to 75 hours of mind-numbing grinding **on vault weekend time**. So your expectation becomes… I’m not getting one in one sitting, or even one weekend. We can triple that expectation _outside_ of vault weekend due to lower gnome rates overall, but if they _aren’t_ increased during vault events then you have a proportionally massively increase chance to see it outside the event given your “inside event” time would only be about three times as efficient at finding EVKs rather than **thirty** , and there are (usually) 25 days of non-vault-event time per 3 days of vault event time. This changes the EVK achievement from a “go chase this on vault weekend” to a “you’ll get it when you get it” kind of deal after thousands of battles. This would fundamentally affect how a huge number of players interact with the game, as this claim is artificially inflating interaction numbers on vault weekends (at least the first time, I’m already seeing signs of burnout here after just _two_ vault weekends). And while not _directly_ correlated 1:1, interactions drive monetization. Hard to say whether the fallacious (or at _best_ misleading) claim is going to help or hurt here beyond very short term (especially given the potential scandal), but the fact remains that this number being out there the way it is is causing money to change hands.

And while this is probably less severe of an issue than rates being published wrong on _directly_ monetized stuff. As I have said before, drop rates in this type of game are _sacrosanct_. There is a responsibility to get it right, so we can **trust** that it is right, even if “getting it right” results in _no_ buff to EVKs and just a clarification on how they are intended to work based on an off-the-cuff claim given in the patch notes. Once it is _correct_, then we can move on to whether or not they are “too rare”. But we need to **start** by getting on the same page. Like, if you are intended to get it randomly as a fluke and barely have any influence over when other than being three times as efficient during vault weekend, it might shift perception a bit on the role these items are intended to play.

@Saltypatra -  
tl;dr: Epic Vault Keys are not about 10% as rare as Vault Keys, at least during vault weekends, they weren’t last weekend, and they still aren’t now. The simplest explanation (and the only one that even makes sense anymore given claims on both sides and data gathered) is that they do not proportionally increase (or more likely don’t increase at all) during event weekends, and we (the playerbase) still don’t know whether or not they are even intended to. If EVK drops are _not intended_ to increase proportionally to Vault Key drops during vault event weekend, please clarify this so we can move on from there about discussing how rare they are even at intended rates. If they **are** intended to increase proportionally to Vault Key drops during vault event weekend, well, there is **ample** evidence to suggest that they currently _aren’t_, and need to be fixed.

---

_[View the full topic](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449)._
