Experimental Update 1.01

[ImpulZ 2491]

New Server Admin Logs

As part of the Experimental Update 1.01, we're introducing some long-awaited updates to our server logs. 

The following log messages remain the same:

Connect/Disconnect

Chat

#toadmin message

We've created some new logs to allow for a better control over how, and when your players are dying, or dealing damage to others.

Examples of new/updated messages

Hits:

• Player "Survivor (8)" (id=DAYZGUID pos=<6332.3, 7823.4, 304.9>) hit by Player "Survivor" (id=DAYZGUID pos=<6319.5, 7873.2, 312.1>) into Torso with Bullet_45ACP from 51.9096 meters

Consciousness/unconsciousness:

• Player "Survivor (3)" (id=DAYZGUID pos=<13333.2, 6235.4, 6.7>) is unconscious

Kills:

• Player "Survivor (17)" (id=DAYZGUID  pos=<13469.2, 4357.8, 5.5>) killed by Player "Survivor (39)" (id=DAYZGUID  pos=<13447.8, 4298.4, 5.5>) with M4-A1

Environmental deaths:

• Player "Survivor (56)" (id=DAYZGUID pos=<12902.9, 10153.8, 6.0>) died. Stats> Water: 229.539 Energy: 464.77 Bleed sources: 1

Placement

• Player "Survivor (17)" (id=DAYZGUID  pos=<13469.2, 4357.8, 5.5>) placed Bear Trap

 

Hopefully, these should provide some more options now! Let us know what you think, what you'd like to know from the logs and report any issues you may find with the logging options via our Feedback Tracker or send your other suggestions via servers@dayz.com. 

[MirekManena 27]

Server Optimisations

For most of you, especially server owners, server performance is a huge topic. For this reason, I've compiled this report on our effort to optimize the server performance during BETA/1.0 release, and the improvements we achieved in January.

After the Christmas holidays, we've made several server FPS measurements. In a nutshell, these were our results:

After profiling the servers, we've identified several problems.

The first cause of this decline in performance was tied to the despawning system. Its task is to despawn entities on clients when they are outside of the network bubble. This despawn routine was called once every 3 seconds. During the BETA stage, we've found a pretty ugly bug within this system, so we decided to call the despawn routine with every network simulation tick. This naturally puts more load on the server.

Another cause of performance drops is tied to the animation LOD system and AI counts. In one of the previous Status Reports, I've mentioned that we were working on an animation LOD system, which should optimise the evaluation of animations on the server.

This evaluation phase was accelerated 4 times with the LOD system, so we decided to double the AI counts as a followup. Although the performance of the animation evaluation was correct, the increased AI count had an impact on the AI (brain) simulation and on its replication processing (the construction of gamestate update data for every player) . This, unfortunately, had a severe impact on performance that was made even worse with higher numbers of connected players.

And finally, the new input system we've introduced during the BETA development stage made the input processing on the servers worse, too. We can say that what we've gained by animation LOD system on player's animation evaluation, we've lost by the new input system.

Because of that, on January 17th, we've released a Stable Update with these improvements:

Despawn system - despawn routine is called once per 3 seconds, as it was before

Attached lights - we were still using an Arma legacy system for updating light positions, which we don't really need on the server - we removed it completely

AI step noises - when moving, every AI entity was making noise that players and other AI entities could hear  - we removed the AI to AI footstep hearing completely as this has minimum impact on gameplay, it was basically just creating interactions between animals and infected, and for creating those, we think the AI to AI vision triggers are enough. Players can still hear AI footstep noises of course.

 

The followings are the results from the above mentioned Stable Branch update (measurement was done on a machine running 3 instances of the game server):

Although the performance has improved a lot or has significantly improved, you can see how it goes down with more connected players. That's why for today's 1.01 Experimental, we've introduced more optimisations:

AI targeting system - we optimised the routine for getting a target list - now it should be much more independent from the connected players count

AI obstacles avoiding - we optimised the avoidance system by removing the unnecessary calls of the expensive function for getting positions on the navmesh - this function is called only when it's needed

Replication system - the simulation often marked an entity often marked an entity for synchronization even if it wasn't needed - we've improved it, which will ignore these marks so that the replication system doesn't have to do as much work

Central Economy respawn system -We've optimised the loot/ai respawners to limit its impact on the performance. the loot/AI respawners will affect the performance less.

 

Of course, the results of our optimisation efforts now need to be properly tested in the Experimental environment under some authentic server load. Please, let us know if you notice any improvements or performance issues (or on the contrary: any specific performance issues)!

[MirekManena 27]

Server Optimisations Part 2

Hello survivors,

last week I published my first Report about how we improved the server performance over the course of January. Today, I'd like to show you some results from the most recent 1.01 Experimental phase. I will also include some other optimisations, which made it to the Experimental build, and discuss future optimisation possibilities.

I will begin with the improvements which we've made since the last performance report.

Optimised Input System

In my last report, I mentioned that the new input system further impaired the server performance - to be more precise, the update of the character controller, which is reading input states, was worse. We improved this controller update and now we are receiving better numbers than with the old input system.

Safe system creates a continuous performance degradation

Because the server has to simulate a lot of players and AIs, there's a high chance that the server will get into a so called "spiral of death" - this means that if one frame is long enough, another frame can take even more time and this repeats over and over. We decided to add a server config parameter to limit the number of simulated players per frame, which can prevent this behavior. This mechanism can split one long frame into multiple shorter frames, thus the server frame-rate should be much more stable. You can set the number of simulated players per frame using the simulatedPlayersBatch config parameter. This value shouldn't be too low or high, so I recommend to set it around 20.

So here are the promised results from the current experimental version. In every row there are average values in one minute intervals (player counts/fps/memory used/active AI):

After we saw these nice numbers, we decided to increase maximum players count to 80. We received these statistics:

While these results are nice as well, we have had several reports that remote players are sometimes moving in a jittery way. This could be caused by several reasons (one of the reasons could be that the simulatedPlayersBatch parameter was too low - 12 in that case), so we have to investigate the cause and then we will decide, how we can improve it.

Future Optimisation Possibilities

We would like to focus on optimising the character simulation, especially when it comes to physics and scripts, as these parts are taking the most CPU cycles. In the case of physics, we'd like to reduce the number of the shape casts used for the collision detection. In the case of scripts, we have to make some investigation first and see, what could be optimized, because the scripts are taking around 40 percent of the player's simulation.

Another option is to make some logic parallel and move it to other CPU cores. This can be problematic when multiple server instances are running on one machine (which is the case for our official servers), because there won't be free hardware threads, but community servers can benefit from parallel computation a lot, if they're not running multiple server instances. Or we can increase the slots per server instance and reduce the number of instances. So this is a topic which we are still thinking about, meaning this investigation will be probably become a thing in next few weeks.

That's all for now, next Server Performance Report will be published before the 1.02 experimental, where I'd like to share the decisions made on these issues.