Hello everybody.
I've been having some discussions for a few days regarding the performance of my private DayZ server for PC.
It turns out that I'm using some mods and one of them in particular (ExpansionAI) creates AI patrols that appear in specific locations when a player approaches and start to fight zombies, players and other AI.
Initially this server was hosted by Nitrado (if it is prohibited to mention the name here, please, let me know and I will remove it from the post).
It turns out that after I configured several of these patrols, my server started to experience huge FPS drops: it started with an acceptable FPS and, at certain times, the FPS dropped to 15 or less. This caused several inconveniences for players, such as desynchronization of cars that went flying, medium and long-range shots that missed targets, zombies teleporting and a sort of other unexpected behaviors.
I thought the problem could be with the hosting, so I tested using a virtual machine on AWS, where I created an EC2 with Windows 2023 Server, 16GB RAM and 4x 2.4Ghz EPYC vCores. What happened? The problem persisted.
In contact with other server administrators, my response to this was that "the Core's clock is too low, the server does not run multithreaded and you are using a VM that comes with shared vCores, and therefore you will not be able to use the full power of the cores".
I thought "Okay, this doesn't make much sense considering the technical information I have about how AWS works, but let's try another approach".
It was then that I unearthed an old 2010 MacPro with a Xeon W3520 2.67Ghz and 32GB of memory to use as a prototype server. I ran all possible tests to check if the processor, memories and network were healthy and, as expected, everything works perfectly well with Windows 10 Pro installed. I installed what was necessary to run the server (the only unrelated applications installed were Notepad++ and Google Chrome) and started testing. Result? The same. After the AIs started to appear, the server's FPS dropped from 4k to close to 10FPS, making even tasks such as opening doors impossible.
It was then that I reached my limit: I decided to install the server on my personal computer ( Ryzen 5600GT 3.60 GHz [4.6GHz when boosted], 32GB RAM, 1TB M.2 2k I/O ), leaving everything closed while I tested the server, playing from another computer inside the same network. And to my indignation the result was the same: as soon as the AI appeared, the server's FPS went to waste.
After that I decided to use my technical knowledge as an IT worker since 2002 to delve deeper into the situation and discovered the following: if I set DayZServer_x64.exe's processing core affinity to a single core, even my boosted Ryzen to it's 4.6Ghz, it stays at 100% usage on the core I select and the IDLE FPS drops ~15%. If I select multiple cores it distributes the processing load and the FPS become normal once more. This means that it is capable of using more than one processing core simultaneously, so it is not just the processor clock that makes a difference in overall performance.
However, regardless of the hardware I use the application on, DayZServer_x64.exe seems to be unable to use more CPU if any internal MOD process needs it, or else the MOD has some internal problem that creates a bottleneck that causes a queuing of tasks that the application cannot send to the processor cores simultaneously. This can be easily tested when the server is configured to use all processor cores, the FPS drops due to AI but the processor, memory and disk usage remains stable.
Another point is that there is a "secondary mod" (from TrueDolphin) that uses the same base AI structure as ExpansionAI to create AI spawns that does not cause a drop in FPS or excess resource usage when the AI appears. This made me even more suspicious that there is something wrong internally in the MOD, that it uses some inherited function that causes this bottleneck or, alternatively, that some unexpected interaction in the settings causes this type of problem.
And now, when asked about the AI performance problems, the module developer responds that "the game main loop is single thread" as if that answered the question: why when the MOD makes the AI appear the FPS drops so drastically and why Doesn't another MOD that uses the same AI cause the same problem?
For that reason I'm here to do what he tell me to do: to ask the devs from BI if that behavior can be something inherit from the server's core structure or if something on the MOD itself can be messed up.
