A Few Thoughts On.. (Experimental Branch, Server Availability..)

[Hicks_206 (DayZ) 4297]

This morning after having a coffee and catching up on e-mail I started going over tweets people had sent to myself, and the official DayZ Dev Team twitter account and I noticed some statements, albeit from a albeit small portion of the community, but ones I thought needed addressing none the less.
First off however, let me take a moment to thank the overwhelming support the DayZ community has given us thus far in our efforts to iterate on 0.60 experimental branch. You guys have helped us isolate a large amount of issues that didn't come up under the first stage of testing with our internal QA teams and that is outstanding. 
Hats off, applause all around.

You guys *rock*.

Now, while going through the tweets this morning I did notice a good deal of frustration from this vocal minority in the community surrounding a lot of issues (Requests to push the current 0.60 build to stable right away, Frustration that experimental servers were not online at all times, Anger over experimental branch servers not being increased to ensure all those interested could log in, Misunderstanding over allowing content creators to show 0.60 to their audiences, Requests to take stable branch official servers offline and switch them to experimental, etc) - but the core of the issues all pointed back to one shared root: Availability of player slots on Experimental Branch.
While I sympathize with those who are frustrated that they cannot get on to an experimental server due to demand, as it is regrettable that the nature of experimental branch does not allow all of you to be guaranteed access I think it is important we revisit this stickied forum post from 2014:

"This branch is by its own definition "Unstable". In simplest terms, the purpose of experimental branch is for the development team to experiment with builds in a larger userbase than our internal QA.
Sometimes that results in us finding issues we cannot catch internally, sometimes that means we can verify fixes on issues with very low repro rates.
Sometimes the builds are very unstable. (As mentioned prior)

The branch exists for load/volume/stress testing. Those who go through the process of manually opting into this branch (its not super visible - by design) and dealing with whatever issues the current build on it may have, get to sometimes see content and systems not quite ready for prime time. However, that does not mean that is the purpose of the branch.
As the nature of the experimental branch is for the above mentioned testing methods, neither uptime, character data, nor stability is guaranteed."  

Experimental Branch usage is (like the rest of development) an iterative process. We (the DayZ Dev Team - and specifically Production, QA, and Build Management) start small and scale up as we identify critical issues and gain more confidence in the build's performance in stress scenarios. You might have seen this visible in the next day update to experimental branch, and the increase of player slots per instance to 60 players. To those who don't follow experimental testing closely this might not be super obvious, but this has provided us with *VERY* valuable data.

I'm going to try and answer/explain a few of the topics tweeted at us over the span of last night below:

Why won't you just push 0.60 to stable now? / It has to be better than 0.59:

Well, simply put we just can't do that. There are quite a few issues that would flat out prevent a large amount of the userbase from playing DayZ at all. From the ever present "Application Error" to black screens on launch, server crashes, and many more - The build *has* to stay on experimental so we can get it to a state that allows the vast majority of the DayZ playerbase to play and test it on stable branch. 
Lets not forget that all branches of DayZ Early Access are testing branches! The testing base and goals scale up from our Internal QA, to Experimental Branch, and finally to the Stable branch as the default build you play when you install DayZ on Steam. I know it is not the popular thing to say, and the less fun part of my job involves having to take the brunt of passing on bad news - but your patience, participation, and feedback is and will help us get this build to stable as soon as possible.

Why are the experimental servers down in my region? / Why aren't the devs saying anything about the server being down in my area?:

Keep in mind the DayZ Dev Team operates out of Central Europe, and depending on where you are in the world there is a high chance the developers are in bed / haven't had time yet to analyze the issues causing the outage. Time Zone differences can be confusing, trust us we know! Pair that with the fact that server availability, distance from the Prague offices, and latency can fluctuate greatly depending where you are in the world - and it can and will lead to server downtimes in your region. A good example of this is the Oceanic and Asia regions - of which we've been fighting server issues and connection speeds throughout experimental branch usage. So please, be patient with us - we haven't forgotten anyone and will get the servers up as soon as time, and process allows. As well, sending tweets out every time a server crashes or goes offline due to a bug or server hardware issue isn't practical given how frequent this can happen on the experimental/unstable branch. 

Why aren't there more player slots? / Lots of people want to play, let us all play:

The experimental branch testing phase is *not* set up to ensure availability and accessibility to all. Besides it not being practical on an infrastructure point, the nature of how we use experimental branch means we need to be able to create false stress situations for the game server instances, the central hive, and in some cases even the server hardware. Now, as experimental branch testing moves forward on any given build we slowly increase the player slots for that branch whenever possible. Be it actual game server instances slots, or allocating additional hardware when possible. Simply put, especially at the initial stages of a build being on experimental branch, we *need* to see how the build operates under extreme demand, we need to see it fail, and we need to analyze that data and make changes where necessary. That said, we are looking into increasing the branch player capacity within reason. Don't expect several thousands more player count capacity, as that is what stable branch is for - but we recognize the demand and are doing everything we can to help alleviate the issue.

Why are streamers given special treatment? / I should be able to play there too:

Early on after the first build of 0.60 was pushed to experimental I realized we just didn't have the infrastructure or hardware available to let everyone who wanted to see 0.60 on experimental branch that wanted to, and we wouldn't be able to flex up to accept all of the demand. Realizing this, at the end of the work day a development box that I use to capture footage for our Youtube Channel, and bug bash internal development builds was taken offline and converted to point to the experimental branch in the hopes that we could at least through allowing content creators to showcase the build let those who couldn't get onto an experimental branch server *see* how 0.60 was operating, performing, and moving along. The hardware used was not powerful enough to actually operate an actual experimental branch server, and would for certain buckle under the demand, but through reallocating this one small instance we could at least provide 30 or so content creators from across the globe, and across multiple languages give thousands more people a look at how things were looking on the march towards 0.60 stable branch. Creators of all audience sizes, and regions across the globe were given access to the box and asked to use said access to help alleviate the stressed demand on experimental branch by sharing their experiences on their distribution methods (Youtube, Twitch, Twitter). Through this method we were able to allow thousands of additional people to see and experience 0.60 experimental branch that we would not have been able to otherwise. 
Some people seem to think that experimental server resources were diverted to allow content creators access to 0.60, or player slots that could have been used for general experimental branch usage were given to these users. Simply put that is not the case, as mentioned before the server utilized was my personal work box and would not be able to handle such a demand. This was easily demonstrated earlier this morning when some enterprising folks acquired and distributed the IP address to this PC and the game server rapidly failed and stopped operating for several hours.
No one was being denied playable server slots on experimental, and while we did ask content creators to be discrete so as to not compound the issue with experimental branch being at capacity that plan upset a small portion of the community, and for that I am sorry. It was never my intention to make anyone feel less important than anyone else in the community, the intent was explicitly to allow as many people as possible to at least *see* 0.60 experimental.

Why don't you just take stable servers offline and convert them? / No one is using stable servers, switch them over:
 
While it may seem that way to some users, DayZ has an active player base of several hundred thousand that cycle in and out (call it turnstile if you will) and it is important that we not just switch off game servers that users call home. We already fight issues with this in the case of some stable branch official servers, you may have seen it visible in forum posts asking where a specific official branch server that went offline was, and how users could get their camps, vehicles, etc back. As we move forward with development and push changes such as the final redesign of our server browser out, increasing visibility and thus reliability in these stable branch servers becomes even more paramount. For a game that is effectively a service in some definitions, like DayZ can be, we need to provide a certain amount of stability and reliability with the official DayZ servers. Switching them off and converting them to experimental branch due to a short term increase in demand is - in short, as I like to call it - "A bandaid on a broken leg". We all recognize that as we continue to move forward with major engine updates to Enfusion/DayZ demand will likely increase on this branch, and while we still have to keep the experimental branch within a certain limited capacity compared to stable branch, we *are* working on being able to increase (flex) experimental branch player slots to try and alleviate the demand as best we can with resources that do not compromise the integrity of stable branch official servers.
 
 
 
I hope the answers above help you understand why/how things are done the way they are. I know some of the answers might not be what some of you want to hear at the time, but I felt it was important to address the questions and concerns raised via twitter over last night. 
Thank you all for being so awesome, and joining us in this incredible and sometimes stressful journey that is the development of what I think is one of the best gaming experiences of all time.