For anyone that is interested here is a developers detailed writings and brief outline of the certification and approval process from Hinterland Studio creators of The Long Dark. This only includes an outline for Steam, Xbox, and PS. In addition it also includes an apology for a rough launch, everyone makes mistakes.
Direct link
Hello community,
First of all – I want to say how sorry I am for the state of the Xbox and PS4 versions at launch, and the struggles we’ve had since then, to fix issues and get you patches in a timely manner. It is totally unacceptable that there are issues with crashing, and progress blockers in the game at launch.
I thought it might be useful to provide a bit more insight into how the patching process works across the different platforms, to help explain why patches may come out more quickly on some platforms than others.
General Fix & Build Process: We make fixes in Unity and submit content and code changes through a version control system called Perforce. These changes can be checked in Unity (using the editor), but occasionally what you see in the editor is not the same as what is seen in the released game, so we have to “make a build”, which is a process where we (using a “build machine”) assemble all the relevant assets and code into a complete, playable, and standalone version of the game. We have various types of builds, but the one we base our releases on is a “Release Candidate” build, which means it has things like the debug menu turned off, etc.
Building the game for all the platforms we launch on takes about 4-5 hours (about 1 hour for Windows, an hour for Mac, etc.). We check each platform and then test internally, as well as with the support of an external testing lab in Montreal. If things look good, we release to the platforms, where (apart from on Steam) each will go through some kind of review process before it is released to the public. We go through this process for every patch we make.
STEAM: There is no certification process on Steam. Once we are confident about the builds, we upload them to Steam and then release them when we are ready. The whole process takes about an hour. Since there is no certification delay and the upload/release process is 100% controlled by us, we can update the game on our own schedule and release as many patches – big or small – as we like. This is how we updated The Long Dark throughout Early Access as well. This convenience and speed is why the Steam version of the game tends to be the most up to date version.
PLAYSTATION 4: We have two separate PlayStation builds – one for SCEA (Sony America), and one for SIEE (Sony Europe). Each is essentially a unique company so all the submissions, certifications, ratings, pricing, and patching are all managed through different channels depending on which PlayStation we’re talking about.
The initial certification process for the game takes 5 days. There is also about a 5-day waiting period for the Sony Store to set up your game. Once that’s all done, you can release your game to the PS Store. You have go through this process for both Sonys (and a third one if/when you launch in Japan).
After you are certified, your next update is a Day 1 patch. This also takes about 5 days to approve. If it passes, you can then use the Rapid Patching system which allows us to update the game within a few hours, provided we don’t do anything that would require a new certification pass.
For The Long Dark, the PS4 SCEA version is far enough along that we are able to use the Rapid Patching system to get updates to you, which is why our PS4 SCEA version is able to be as up to date as Steam.
Unfortunately, our PS4 Europe version is still getting through the Day 1 patch part of the process. Last night we received word that our Day 1 for PS4 Europe has been rejected due to a well-known crash that will be fixed as soon as we can Rapid Patch. Unfortunately, the way the process is set up, we may have to re-submit a new Day 1 patch that does not have the crash in it, before we can unlock Rapid Patching. We are waiting to hear back from Sony Europe on this, and should know by tomorrow. This has delayed our updating of the PS4 Europe version. We’re working with Sony Europe to get the Day 1 patch out ASAP so that we can rapid patch it to the most recent version of the game.
I should be clear that we are not blaming Sony for this – they have a strict process that we have to work through, and it’s designed to protect consumers from issues that could brick their PS4s. But the process takes a while and until Rapid Patching is unlocked, we have to go through these steps. So, we beg for your patience as we navigate this with Sony and work through it as best we can.
Our focus right now is to get through the PS4 Europe Day 1 patch so that we can unlock Rapid Patching, at which point we will be able to ensure both PS4 versions are as up to date as the Steam build.
XBOX ONE: For The Long Dark on Xbox One, every patch goes through a 5-day review by the certification team. If the patch fails the review for some reason, we have to restart the process over again. We try to uncover issues as the process unfolds so that we can cancel and restart early if needed, but sometimes we only find out about issues after the 5-day process is complete. We have released one CU (Content Update, i.e. Patch) for Xbox so far, and we have a second one in with the Cert team right now. We submitted that one on Tuesday so it’s making its way through internal review, and hopefully we’ll hear something about it tomorrow (at the earliest).
Since Xbox patches take so much longer than Steam or PS4, we’re bundling up fixes in them so that while you get less frequent patches, they should have more fixes in them. That said, this version of the game will be a week behind the others in terms of patches. Microsoft has been a great partner in this and they are doing everything they can to help us get these updates out to you, without compromising their internal verification process.
New Territories
Making the game available for purchase in new territories means different things on Xbox and PS4. In both cases, there is a certification process involved. Unfortunately, you cannot be simultaneously patching the game AND in certification – the internal processes don’t allow for it. So, until the worst stability and save loss issues are resolved on PS4 and Xbox, we won’t be releasing in new territories such a Australia or New Zealand (which we now have a rating for). We’ll release an updated schedule for AUS/NZ and any other territories that are currently in the pipeline (we are working on ratings approval for Japan, Brazil, etc.) as soon as we hear more from those ratings boards.
State of the Game
I’m not at all happy with how our launch has gone.
We didn’t anticipate some of the issues that would arise at launch and we’ve been, frankly, scrambling to fix things as quickly as we can for you. Although it may not seem like it, we did extensive testing in-house, and also had 8 testers from a professional testing lab working on the game for the last 8 weeks of development, to help us identify issues. We caught and fixed a lot of issues before launch. Some things that have come up, were impossible to anticipate up front. Other things slipped through due to carelessness on our part. In hindsight, the amount of content we released in Episode one and Episode Two was too much for our team to test properly, even with external help. We took for granted that 3 yrs of Early Access testing and stability would translate to a very stable Story Mode launch, and we did not fully understand until too late, the scope of some of the technical issues that would be uncovered by our launch.
Our goal over the coming days and weeks will be to continue fixing all the issues we find and you report (thank you!), and get the game to the place it should have been when we launched it. After that, we will be spending some time in deeper investigation of what went wrong, to see how we can improve our processes, pipelines, testing tools, and the technology we’re using, to ensure we never put you through this again.
I know reading this message doesn’t give you much comfort if you are waiting for an Xbox patch because you can’t get through Episode Two, or you are playing the European PS4 version that is missing fixes from the last 4-5 days, but I hope at least that you will understand that we are working very hard – along with our partners at Microsoft, Sony America, and Sony Europe – to remedy the issues, which we take very seriously.
We’ll continue to be available to support you and answer questions, and in the mean time we’re working to fix issues and get those fixes out to you as quickly as we’re able, more quickly on some platforms than others.
Thanks for reading, and I’m very sorry for the instability, bugs, and patch delays.
- Raphael