# Vault Weekend data collection - for w/e 13 Sept **URL:** \ **Category:** Gameplay Chat (General)\ **Created:** [August 14, 2020, 7:24am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449 "2020-08-14T07:24:21Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![Mithran](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/mithran/32/18064_2.png) [@Mithran](https://community.gemsofwar.com/u/Mithran)\ **Post date:** [August 17, 2020, 12:44pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/21 "2020-08-17T12:44:07Z")
> [@cyberkiwi](#): > > \*\* _It should be noted that data was collected from 11 people only. 5 of them got epic vault keys. The sample size of epic vault keys is too small to draw statistically significant conclusions._ Sort of… If we assume the given rate is correct, the cumulative probability of getting the sample set as the one collected is worse than 179 million to 1. I’ve seen samples for other random events that described outliers or anomalies where nothing was actually wrong, and willing to give the benefit of the doubt in a lot of situations for way longer than I should given past events (though these days anything that doesn’t pass even a 99% confidence level test makes me _nervous_, especially if I see a trend), but we are way _way_ beyond that. We have a \>99.9999% confidence from the samples taken that the true rate falls outside what was stated, and 300 samples is absolutely a significant amount given a stated rate of 10% and our sampled rate of 1.6%. It is much harder to zero in one the _actual_ rate through sampling, for sure, we’d have way less confidence saying “the actual drop rate is 1%, so yours is wrong”, for example, but we can be pretty confident in saying “based on this, I’m pretty sure it isn’t actually 10%”. Worth noting that also the _best_ “luck” recorded data sets from person to person posted here would still fall on the side of “bad luck” for a true rate of 10%. For a similar situation, consider a situation where you are running a coin flip trial and getting heads 24 out of 25 times, and then declaring there is no way to tell whether or not this is a “fair” coin. Yes, you’d want more data to _prove_ it with “statistical significance”, but it is super easy to just flip a coin more times whereas, on our end, trying to collect Epic Vault Key data is extremely tedious, and going forward without even event weekend, borderline impossible until it rolls around again. By the way, hitting this hypothetical coin situation with a “fair” coin is a roughly 1 in 1.3 million chance, more than _100 times_ more likely than hitting 5 (or fewer) successes in a sample of 303 with an _actual_ rate of 10%. At some point, we have to request that, even if it potentially “wasting the devs time”, that this is at least “enough” for some checks be performed on their side both to see if the “correct” rate is in game (both during the vault event and not, and/or if we were quoted the “correct” rate in the 5.1 update notes). I’d imagine (or hope) they have a tool to show them _all_ vault key drops over a weekend from Live data (rather than simulated) as well as _all_ Epic Vault keys, at which point it wouldn’t take more than a glance to see how close it was to 10%. For example, if 10,000 total vault keys were dropped and anything less than 800 were epic, that is a big enough sample to declare with similar confidence that the rates are “not 10%”, even though it is much less “off” than our number, percentage wise. The point being, any scenario where **we** have “enough data” to “prove” the rates are a given number with any degree of certainty is a scenario where a potential bug has been in the game way too long and way too much legwork had to be done to the playerbase if it is ever revealed as an actual bug. I’d be lying if I said that past mistakes have happened with very similar things factored in to how much benefit of the doubt I’d be willing to extend here, but I can picture about a dozen ways this could have gone wrong, even a bunch where even if they pulled simulated data it would line up with “expected” while still being discrepant from the data given by the players. At this point, it just seems far more likely someone screwed up in setting/communicating the rates (or both) than _all_ the people that happened to be recording results and willing to post them (and/or just happened to be recording and spreadsheeting all their results but only decided to post _after_ they were “unlucky”) had some degree of “bad luck” ranging from “minor” to “extreme outlier”. Its not _impossible_ everything is working correctly, but… ---
**Author:** ![cyberkiwi](https://avatars.discourse-cdn.com/v4/letter/c/8dc957/32.png) [@cyberkiwi](https://community.gemsofwar.com/u/cyberkiwi)\ **Post date:** [August 17, 2020, 12:58pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/22 "2020-08-17T12:58:57Z")
You have to also consider out of 11 people, 5 got an epic key each, and the 5 ALL used fewer tries. Statistics lie all the time when you have no control over the sample. If the samples were chosen _before_ data collection and everyone followed through with submitting data, I agree the data is compelling. But when the data is heavily skewed by 2-3 sets that did not get the epic vault key, it becomes questionable. This is like taking a huge stream of die rolls that print onto a ticker tape, and then selectively choosing the most interesting portions - the 11 sixes in a row multiple times, 20 sixes in a row, then a few normal runs, and using that very tiny sample of the population for chi-squared tests and projecting confidence intervals. A recipe for disaster. I am not vouching for the developers at all, I’m just not willing to make any conclusion from an extremely small and limited data set that is not robust. For all we know, there is probably a bug somewhere in the application of said 10%. I can agree with the _possibility_. Let me give you a hypothetical interpretation of the data: - the population of players is way more than 3000 (plucked from thin air) - this volume provides for 0/51 as a possibility 0.5% of the time, or 15 players - 2 of those 15 players self-elect to report their outcomes - you have your very biased 11-person dataset ---
**Author:** ![actreal](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/actreal/32/1945_2.png) [@actreal](https://community.gemsofwar.com/u/actreal)\ **Post date:** [August 17, 2020, 1:15pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/23 "2020-08-17T13:15:58Z")
I’m with @Mithran. Bayesian probability says that: since **every other time** players have had data that looks as bad as this, there’s been a bug or a miscommunication inside the dev team or both, it is extraordinarily likely that this is a dev error. I didn’t collect data sufficiently accurately for inclusion in a data set, but no one in my guild got a single Epic Vault Key during a busy weekend of playing, and I’m not seeing any player gleefully post about getting more than one Epic key. Whatever went wrong, I hope the devs fix it before next vault weekend so we can all have a real chance of getting one. ---
**Author:** ![Mithran](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/mithran/32/18064_2.png) [@Mithran](https://community.gemsofwar.com/u/Mithran)\ **Post date:** [August 17, 2020, 2:33pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/24 "2020-08-17T14:33:19Z")
> [@cyberkiwi](#): > > I am not vouching for the developers at all, I’m just not willing to make any conclusion from an extremely small and limited data set that is not robust. For all we know, there is probably a bug somewhere in the application of said 10%. I can agree with the _possibility_ . > > Let me give you a hypothetical interpretation of the data: > > - the population of players is way more than 3000 (plucked from thin air) > - this volume provides for 0/51 as a possibility 0.5% of the time, or 15 players > - 2 of those 15 players self-elect to report their outcomes > - you have your very biased 11-person dataset I will concede that this is a _possible_ scenario, but I do not believe this is the most _probably_. Based on _all_ the information we have now (including past data gathering endeavors by at least some of the people posting here and past mistakes made by the dev team), it is far more likely that this is a mistake on their end rather than some extreme confluence of a bunch of people recording and then withholding data. This isn’t the first data gathering endeavor on this forum, and skewing to the degree that would need to “corrupt” this data set to the point where it didn’t at least point to something being “off” would imply a bias far far greater than any I have experienced in any other data set I collected here. For example, if we removed _all_ the people that got 0 vault keys from the larger set (leaving 5 out of 148), we still have a situation where our sample set at even 99.999% confidence level does not include the stated rate of 10% within in its confidence interval. We’d have to assume nearly every sample is skewed or biased, and while we can’t _prove_ the rigor of this data, it is odd that there is no evidence to the contrary to be found. Unless I’m completely misinterpreting the data? > [@cyberkiwi](#): > > You have to also consider out of 11 people, 5 got an epic key each, and the 5 ALL used fewer tries. The sheet shows every one of the five who got a key having at best 1 out of 11 successes, and at worst 1 out of 56. That would make every single data set “bad luck” if the true rate were 1 in 10 (neutral luck if it were 1 in 11, depending on how you interpreted the 10% in the original post, the 10% could be describing 1 EVK per 10 vault keys, which is closer to 9.1% successes) Now, the absence of evidence does not necessarily mean the evidence of absence, but given even _inconclusive_ data, no matter how you arrange it that trends heavily _away_ from the conclusion “the rate of 10% is correct as stated”, based on the history of the multiple reliable reporting sources versus the proven fact that _very similar game issues have happened more than once in the past_, if forced to draw a conclusion, it is far more likely that the mistake occurred on their end. Yes, absolutely, the “correct” response for an **objective statistical analysis as _proof_** is to gather more data and ensure it is more “pure”, but also probably underestimating what a _monumental_ task this can be, even over multiple participants and that fact that it should not be on us, the playerbase, to _ever_ need to collect enough data on something being wrong to rise to the level of “statistical proof” (both in terms of real time that needs to elapse, months or sometimes even _years_, in addition to the time spent actually gathering the data) before it will be examined from the developer side, where they have much better tools to find out what may have gone wrong in a fraction of the time and effort. Which was the larger point I was trying to make in my previous post. But, at this point, given the inability to just “gather more data” in any _reasonable_ way, even with an organized effort, until a point in which we’ve already resigned to leaving a potential drop error unaddressed in the game for _at least another month_, which is more likely?: - extreme reporting bias due to unhappy people, including several known data collectors, with no evidence to the contrary, where we have to assume we _only_ got bad data in addition to some amount of “bad luck” - any one of a myriad of human errors that could have caused something that could explain the discrepancy, from logic/programming error to a data input error to simple miscommunication combined with a rushed development schedule and lack of testing compounded by sub-optimal working conditions to cause a similar errors that have already happened _outside_ of these conditions on more that one occasion If you weren’t around for the Chaos Portal drop rate saga, you are probably have a higher internal burden of proof before you are willing to even raise it as “probably an issue”. Having gone through that once already, I really _really_ hope it never has to make it that far, because it just drags _everything_ down. To the point where even if it turns out to be “a waste of time”, the _smart_ move would be for the devs to at least give it a check. Thus, yes, they should also lower their burden of proof as well. tl;dr: I can’t call the data “proof”, but I can definitely say there is enough of a trend that it needs to be double checked on the dev side, and if something _is_ wrong and _get_ to the level of “proof” before it is fixed, that is a slew of negative attention that they don’t need piled onto it just _being wrong_ in the first place. If it were me sitting on the other side of the fence, I’d be pushing to check this out asap, on my own time if had to, because of the implications of continued repeated problems and what that means for the game’s image and how it affects overall discourse. ---
**Author:** ![cyberkiwi](https://avatars.discourse-cdn.com/v4/letter/c/8dc957/32.png) [@cyberkiwi](https://community.gemsofwar.com/u/cyberkiwi)\ **Post date:** [August 17, 2020, 2:42pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/25 "2020-08-17T14:42:51Z")
> [@Mithran](#): > > Unless I’m completely misinterpreting the data? You read it perfectly right. 5/148 is pretty abysmal 🙂 I am captive to logic, and your reasoning is perfectly sound - I acquiesce. I am a fan of data, and I agree the burden should not be on players to find proof. But on a side-note, my data gathering exercise wasn’t aimed at proving anything - in fact I started it before the 10% drop rate was disputed. I just want data to look at and play with, to reverse engineer what the devs won’t publish. 😆 ---
**Author:** ![actreal](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/actreal/32/1945_2.png) [@actreal](https://community.gemsofwar.com/u/actreal)\ **Post date:** [August 17, 2020, 2:46pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/26 "2020-08-17T14:46:30Z")
> [@cyberkiwi](#): > > I just want data to look at and play with, to reverse engineer what the devs won’t publish. That’s what pretty much everyone else who collects data on GoW is doing too. 🙂 It’s just that sometimes we reverse engineer stuff that the devs don’t yet know was not going to plan. ---
**Author:** ![Magnusimus](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/magnusimus/32/50652_2.png) [@Magnusimus](https://community.gemsofwar.com/u/Magnusimus)\ **Post date:** [August 17, 2020, 5:44pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/27 "2020-08-17T17:44:46Z")
To add to the pile, though I don’t have data regarding _when_ gnomes did/did not drop: Personally killed enough gnomes to get 5 free Vault Keys from the tracker. Got 6 or 7 dropped from the gnomes I killed. Even less robust stats (but I still think worth mentioning): The Unforgiven Family **in total** (6 guilds) got _hundreds-of-thousands worth of trophies_ played last week; even if we assume only a fifth of these were gathered during the weekend for eligible gnome-generating gameplay, it’s pretty crazy that only 1 or 2 people (to my knowledge) pulled an Epic Vault Key across our guild family. Could be we have a lot of unreported keys, but we’re pretty communicative, so I’m thinking that’s unlikely 🤷‍♂️ ---
**Author:** ![Jefferson](https://avatars.discourse-cdn.com/v4/letter/j/8491ac/32.png) [@Jefferson](https://community.gemsofwar.com/u/Jefferson)\ **Post date:** [August 17, 2020, 10:09pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/28 "2020-08-17T22:09:37Z")
I can post some other numbers from members in SoA. 57 vault keys 0 epic 51 vault keys 0 epic (mine previously posted) 44 vault keys 0 epic 32 vault keys 0 epic As a guild only 2 people got an Epic Vault key. not everyone reported their exact vault key totals, but from the one’s that did report all their keys we farmed 497 keys and 2 of them were Epic Vault Keys. ---
**Author:** ![Graeme](https://avatars.discourse-cdn.com/v4/letter/g/50afbb/32.png) [@Graeme](https://community.gemsofwar.com/u/Graeme)\ **Post date:** [August 17, 2020, 10:53pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/29 "2020-08-17T22:53:20Z")
30 vault keys 0 epic for me. Only 1 person in our guild reported finding an EVK at the weekend. Don’t have the data from any other members but we are a very active and communicative guild so would have expected more if the ‘10%’ was correct. ---
**Author:** ![cyberkiwi](https://avatars.discourse-cdn.com/v4/letter/c/8dc957/32.png) [@cyberkiwi](https://community.gemsofwar.com/u/cyberkiwi)\ **Post date:** [August 17, 2020, 11:02pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/30 "2020-08-17T23:02:10Z")
Many thanks @Graeme, @Jefferson, @Magnusimus for your feedback. Sorry that I haven’t added them to my stats since the data’s quite light on details. I might revisit that later. but it looks like 11 or 15 samples won’t make a difference with this one. ---
**Author:** ![EraserParticles](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/eraserparticles/32/62373_2.png) [@EraserParticles](https://community.gemsofwar.com/u/EraserParticles)\ **Post date:** [August 17, 2020, 11:03pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/31 "2020-08-17T23:03:57Z")
Made my post on the wrong thread! Serves me right for browsing on mobile in multiple tabs. I deleted the last one to move it here, apologies for the confusion. Mine (I didn’t post mine to the guild discord) looked like: 10.5k trophies earned during vault weekend (explore d7, so over 5k battles at 2 trophies per) Nearly 600 gnomes from tracker (fell asleep before getting the exact number) 57 vault keys 14 tracker keys 0 epic vault keys Pandemic + vault weekend ---
**Author:** ![Jonathan](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/jonathan/32/66724_2.png) [@Jonathan](https://community.gemsofwar.com/u/Jonathan)\ **Post date:** [August 18, 2020, 4:58am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/32 "2020-08-18T04:58:20Z")
> [@cyberkiwi](#): > > If the samples were chosen _before_ data collection and everyone followed through with submitting data On this note, I think a few of the players that submitted data _always_ collect it, e.g. Dust Angel, and had probably been collecting it before seeing the thread, so could realistically be treated as pre-selected samples 😝🤷‍♂️. ---
**Author:** ![Fleg](https://avatars.discourse-cdn.com/v4/letter/f/5e9695/32.png) [@Fleg](https://community.gemsofwar.com/u/Fleg)\ **Post date:** [August 18, 2020, 5:57am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/33 "2020-08-18T05:57:48Z")
410 gnomes 31 vk (not including pity keys) 0 evk ---
**Author:** ![Neritar](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/neritar/32/39263_2.png) [@Neritar](https://community.gemsofwar.com/u/Neritar)\ **Post date:** [August 18, 2020, 6:03am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/34 "2020-08-18T06:03:30Z")
182 gnomes 18 vault keys (not including 8 keys from tracker) 0 epic vault keys Our guild was quite active on weekend (85k trophies in the previous week), but I’m aware of only 1 player which got epic vault key, many others got 10-20 vault keys and 0 epic vault keys. ---
**Author:** ![HounganAtLarge](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/hounganatlarge/32/28697_2.png) [@HounganAtLarge](https://community.gemsofwar.com/u/HounganAtLarge)\ **Post date:** [August 19, 2020, 3:07am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/35 "2020-08-19T03:07:50Z")
78 Gnomes. 5 Vault keys. 0 Epic keys. ---
**Author:** ![Sytro](https://avatars.discourse-cdn.com/v4/letter/s/df788c/32.png) [@Sytro](https://community.gemsofwar.com/u/Sytro)\ **Post date:** [August 20, 2020, 7:26am UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/36 "2020-08-20T07:26:43Z")
Just one suggestion on the topic. > Vault week probably increased the drop rate of vault keys, but **did not** incease drop rate of epic vault keys. Don’t remember the multipliers (for getting a vault key during vault event), but it might be that we’re looking for a much lower ratio of EVK vs VK than 1:10 on Vault weeks. Maybe 1:30 or even lower ? ---
**Author:** ![Planet13](https://avatars.discourse-cdn.com/v4/letter/p/aeb1de/32.png) [@Planet13](https://community.gemsofwar.com/u/Planet13)\ **Post date:** [August 20, 2020, 1:13pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/37 "2020-08-20T13:13:37Z")
Thank you Dust\_Angel ---
**Author:** ![Mithran](https://sea1.discourse-cdn.com/flex019/user_avatar/community.gemsofwar.com/mithran/32/18064_2.png) [@Mithran](https://community.gemsofwar.com/u/Mithran)\ **Post date:** [September 8, 2020, 9:32pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/38 "2020-09-08T21:32:51Z")
With another Vault event fast approaching, it would be great if we could get some people stating _ahead of time_ their willingness to record and post data so we can a) eliminate as much sampling bias as possible and b) see if the trend from last month holds. ---
**Author:** ![covertmuffin123](https://avatars.discourse-cdn.com/v4/letter/c/5daacb/32.png) [@covertmuffin123](https://community.gemsofwar.com/u/covertmuffin123)\ **Post date:** [September 8, 2020, 9:38pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/39 "2020-09-08T21:38:33Z")
I plan to do this, and was going to make a thread asking people to precommit to recording data. I guess we could just do it here. Since gnomes are automatically tracked by the game, we really just need people to record keys/epic keys. Total battles doesn’t seem relevant because the gnome rate itself isn’t in question. ---
**Author:** ![Fleg](https://avatars.discourse-cdn.com/v4/letter/f/5e9695/32.png) [@Fleg](https://community.gemsofwar.com/u/Fleg)\ **Post date:** [September 8, 2020, 11:40pm UTC](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449/41 "2020-09-08T23:40:47Z")
Imho it’s already confirmed that it’s not 10% from the above data collections. If the coming weekend shows droprate more in line to what we expect then for sure it means a stealth fix from the devs with no admittance of error in 1st vault event of course. [Previous page](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449.md?page=1) [Next page](https://community.gemsofwar.com/t/vault-weekend-data-collection-for-w-e-13-sept/65449.md?page=3)