Server crashes and lock-ups

[sinnaman 90]

Hey guys,

 

as a server owner administering a popular private shard server, we tend to get crashes every night, pretty much every hour if the server is proper packed. Most of you know these crashes as the inability to open doors, change weapons or do anything in game, along with your fellow travel companions seemingly running in place, or the infamous "No message received".

 

What I've been trying to figure out is the exact reason for these crashes, and the theory that is backed up the most is the following;

 

When the server is full and people keep spam-joining the server waiting for slots (basically attempts at joining repeatedly when there are no slots available), we think there may be instances where a slot will open up, the server will accidentally let in two players at the same time (because they clicked join at the exact same second), and the server will freeze up or crash.

 

We've been considering setting BEC to kick any player beyond 48/50 to see if this helps solve the problem, but before we go this far we'd like to get other server admins thoughts on the cause of these crashes. Are we completely off track here, or is this possibly a viable solution until BI fix the server infrastructure and implement queuing and the like?

 

All opinions welcome on this, thanks for your help guys.

Sinnaman

Edited January 27, 2015 by Sinnaman

[sniperdoc 251]

I don't think it's related to joining max pop servers... it could be... but I've seen servers hanging/freezing like that on low pop as well. I was on a server last night with 8/30 and ended up getting stuck in a truck because the server crashed without any indicator it had done so (i.e. no message received for n seconds). I'm guessing there's a memory leak somewhere as it causes the client to crash as well. I think the client and server crashes are related.

[The Riddler (DayZ) 410]

Sinnaman makes a good point though that I have observed on my server as well when it seems to freeze up. Every time I have seen our server freeze, we are near capacity (was 40, now 50) and one thing I have witnessed when its frozen is in the player list showing somebody trying to join. I have even gotten in the habit of checking the player list when we suspect the server is frozen and every time I see a player trying to join. Its certainly plausible that the last player trying to connect could be in conflict with another player who tried to join at the exact same second. 

 

Sinnaman - since we're both on Fragnet, send in a ticket right after a crash; the kind support folks with Fragnet replied back to one of my tickets indicating there is a "bug" in .52 causing crashes and they sent off our logs for BI to review. Might be worth it if indeed the BI people are able to review said logs. 

 

I don't think it's related to joining max pop servers... it could be... but I've seen servers hanging/freezing like that on low pop as well. I was on a server last night with 8/30 and ended up getting stuck in a truck because the server crashed without any indicator it had done so (i.e. no message received for n seconds). I'm guessing there's a memory leak somewhere as it causes the client to crash as well. I think the client and server crashes are related.

 

You make a good point as well in regards to a memory leak. I have sat inside a house logged in as Admin and just observed the memory usage that the server is using (can't recall the command you can type in to see the performance). It fluctuates up and down, but over longer periods of time, i.e. the longer the server is running before restart,  the memory count seems to continue to rise and rise and I wonder at some point if it starts to leak off or possibly run out thus causing poor performance and eventually a lockup.

[The Riddler (DayZ) 410]

Ok so here's something interesting to follow what Sinnaman was discussing above. 

 

Last night as the server was full we had a player in TS spamming connect to try and get in. As he finally started to connect we noticed the server hang up (unable to open doors, do anything with inventory etc) as his name had an hourglass next to it as well as another player right below him in the list with an hourglass next to his name as well. They were the very last two spots in the list (bottom). 

 

Player on TS mentioned he was stuck on "Please Wait" so with the server as it was I asked him to disconnect, and almost immediately we noticed the server started responding again. The player in the list that was below him connected as well. 

 

Its only one example so I'll keep an eye on it tonight as we play and fill up after a restart to see what happens. I have a feeling it does have something to do with the server allowing 2 players to connect at once into possibly the same slot. 

 

I'll keep you updated Sinnaman

[gtefox 9]

We have the same Problems with our Server.

 

The Server crashes the most Time if their are 40/40 Players or at least 38/40 Players.

Than it is impossible to change Weapons, open Doors, pick up Loot, drink, eat and Players who want to join only see a endless "Waiting for Host".

 

Most of the Time, we have to restart the Server manually (if its possible and we aren't at Work or anything else) otherwise the Server keeps running with 40/40 Players without them beeing on the Server for real and no Players can join (One time we had Players on our Server Activitylist who played for 9 Hours without disconnecting what is impossbile since Servers are force to Restart every 4 Hours) . Eventually the Script from our Hoster kills the Server if he crashes, but most of the Time it won't because the Script dosn't detects a Crash.

 

 

We already contacted our Hoster with this Problem.

They told us to re-install the Server (what we did) and delete the Storage File (what we also did) but even this Things didn't solve this Problem. Even they said that they can't see any Problem or "Crash" in the Servers Logfiles..

Our Server still keeps crashing alot and keeps 40/40 Players until one of us restart the Server manually.

Edited January 25, 2015 by SniperMexican

[sinnaman 90]

As he finally started to connect we noticed the server hang up (unable to open doors, do anything with inventory etc) as his name had an hourglass next to it as well as another player right below him in the list with an hourglass next to his name as well. They were the very last two spots in the list (bottom). 

 

Exactly this. This can also happen on a server that isn't full, which happened to us the other night. Think we had about 30/50 and the last two players had hourglasses when the server locked up.

 

I think we're getting somewhere!  :thumbsup:  ^_^

[baron334 17]

Hey guys, we run two popular servers that stay 50/50 in the US.  One is public and one is on a private shard but both lock up from a lot recently.  We have only noticed this since .52.  I play regularly and I have been able to narrow our lockups down to right before restarts 90% of the time.  Our server is constantly spammed with people trying to join or spamming refresh and it appears to always lock up to the point where I can even call the lockups.

 

What happens is we get to a restart or right before it and the server locks up and we have to force a restart.  About 10-15 minutes after we restart the server the scheduled server restart happens.  This isn't something that happens occasionally this is happening quiet often but not every restart.  I can't tell you if it happens with under 50 people because we always have 50 but it is annoying and requires our admins to be around a lot to restart the servers.  I am hoping it is just an issue with this build and will be cleared up in the next stable.

[The Riddler (DayZ) 410]

Good info here guys. Has anybody sent a feedback tracker into the Dev's? If not, I'll volunteer to do so with information based in this thread. I'll also make sure and give credit where credit is due by mentioning forum names in that feedback tracker.  

 

Just lemme know as I'll be happy to submit this info.  :thumbsup:

Edited January 26, 2015 by The Riddler

[sinnaman 90]

 

Good info here guys. Has anybody sent a feedback tracker into the Dev's? If not, I'll volunteer to do so with information based in this thread. I'll also make sure and give credit where credit is due by mentioning forum names in that feedback tracker.  

 

Just lemme know as I'll be happy to submit this info.   :thumbsup:

 

Sure, dude, go right ahead. I don't care who tells them as long as they get to fixing it right quick.

 

But on a side-note, I think you are the right one to do it. :P

[sneakydudes 278]

I am finding weird reboots on the server 1 hour before the time is scheduled. It maybe something on the server end of it. The freezing is forcing me to shut the server off and turn it back on. Seems to be happening a crap load this .52 The processes are very low on the server and its a low very low pop private server locked down.

 

Server ran just fine until we turned persistence on.

 

Now we are finding the pvp server, and the master pve server both slower then snails with persistence on. We have low pops right now, but I doubt the servers can handle anymore then 3 people with persistence on.

 

I also found some public servers doing the same with persistence on 7/50 and I can run around the map then get restarted 15 minutes prior - resaved location.

 

Server didn't kick us, but the admins where forced to reboot it.

 

So its all of them, I can best guess with persistence on. Possible loot explosions doing it?

 

I dunno

[sneakydudes 278]

Hey guys, we run two popular servers that stay 50/50 in the US.  One is public and one is on a private shard but both lock up from a lot recently.  We have only noticed this since .52.  I play regularly and I have been able to narrow our lockups down to right before restarts 90% of the time. 

This is exactly the same thing with 3/30 or 1/50 doesn't matter but its happening right before the normal restart even between the first weird reboot/restart 1 hr prior then it goes funny until the normal restart. I am forced to manually go into the server from the webpage to shut it down.

 

The recon tool is also timed out showing players in the server. When I try to reconnect it says Disconnected.... I know its not allowing the rcon tool to work. So I am forced to go into vilayers web/admin site and force the hard shutdown.

 

Persistence on is the only thing I can say that is causing this to happen. Doesn't go that bad with persistence off. I will ask the guys to turn it off again, and retest the persistence off.

 

Best I can do to determine if its fubar with persistence off or on.

[baron334 17]

We don't run persistence on either of ours and I can confirm it happens to us often.  Last night both servers were locked up when I got home from work.

[sneakydudes 278]

We don't run persistence on either of ours and I can confirm it happens to us often.  Last night both servers were locked up when I got home from work.

I wonder then if its the server timed out to the dayz hive?

 

I thought at first it had to be the database, but maybe its networking related.

[The Riddler (DayZ) 410]

I am finding weird reboots on the server 1 hour before the time is scheduled. It maybe something on the server end of it. The freezing is forcing me to shut the server off and turn it back on. Seems to be happening a crap load this .52 The processes are very low on the server and its a low very low pop private server locked down.

 

 

 

From my experience, this is a restart forced by your GSP. I've seen Fragnet restart our server shortly after our scheduler performed the restart. Fragnet has mentioned before they run a script to look for servers running over 4 hours and force restart it. How often do you have your server set to restart? 

 

I'm having trouble with my login again with the feedback tracker. I have Acco (admin on here) looking into but this is what I get when I view my account :( (see below). I will log this info as soon as I have a fix!!!! 

 

APPLICATION ERROR #12

This account is protected. You are not allowed to access this until the account protection is lifted.

[saintofthesystem 5]

Talking about these issues a month later, when the teddy bear effect has worn off, is pointless.  Those fucking things actually could have contributed.

It's obviously wasn't top priority for a hotfix, and the silence means: either deal with it or pull your servers.

 

    http://forums.dayzgame.com/index.php?/topic/217766-server-resource-usage-and-crashes-pre-and-post-052/#entry2188280

 

    http://feedback.dayzgame.com/view.php?id=20237

[The Riddler (DayZ) 410]

Now that .53 has been released to stable and in the known issues of the Change Log it states "Server Freezes" , lets keep this thread going on reporting issues. 

 

Let's see if we can find any correlation to the lockup/freezes we saw in .52 and help report data through the feedback tracker for the Dev's to see.

 

Only played .53 for a few hours so far but I saw our server perform restarts as it should with no freeze ups or temporary hangs like we saw quite a bit in .52. I'll keep reporting based on what we see or get reported to us on our TS. 

[The Riddler (DayZ) 410]

Thought i'd bring this thread up to page 1. Anybody else seeing lockups/freezes in .53? We're seeing the server freeze about 10 - 15 mins. before restarting when we're at capacity which is pretty much all the time now. 

 

We have no choice but to restart the server manually. We've sent a ticket to our GSP and they sent our logs onto BI for review. 

 

Anybody else? 

[sinnaman 90]

The server maintenance last Wednesday seemed to bring increased stability to the servers, and I believe there was a small hotfix a couple of days back as well. Server crashes are a lot less frequent, but we did just have a freeze, so they're not completely eliminated.

[The Riddler (DayZ) 410]

We're still seeing them daily. Its always about 15 mins or so before a scheduled restart. It simply hangs up and unable to do anything in invetory nor do ingame commands work. I also noticed RCON doesn't work when this happens thus making us goto our GSP CP and restart the server :( 

[The Riddler (DayZ) 410]

Another crash reported on our server :(

 

Consistently we're crashing when we're at full capacity (50/50). An admin played for a majority of the morning when the population on the server was below half and the server restarted itself on schedule with no problems. I'm wondering if this is still the same issue we're seeing when 2 players trying and connect at the same time like we discussed in .52? 

 

Just a thought......

[Apollo'sFire 31]

I wonder if maybe the scheduling put forth by our GSP is conflicting with BEC? perhaps syncing fragnet schedule with BEC might fix the issue?

[sinnaman 90]

I also noticed RCON doesn't work when this happens thus making us goto our GSP CP and restart the server :(

 

We never restart using RCON as it tends to create more problems than it fixes. Always have CP open and the logs from there in addition to DaRT.

 

Another crash reported on our server :(

 

Consistently we're crashing when we're at full capacity (50/50). An admin played for a majority of the morning when the population on the server was below half and the server restarted itself on schedule with no problems. I'm wondering if this is still the same issue we're seeing when 2 players trying and connect at the same time like we discussed in .52? 

 

Just a thought......

 

We had a full afternoon of 50 players, no crash, but then later in the night, at around 23:00 we froze with 25-30 players on. No two players with hourglasses, so literally have no idea what's causing it now.

[The Riddler (DayZ) 410]

Anybody notice it takes a while for your server to come back up after you manually restart it? Ours usually only took no more than a minute or two to restart and be the correct time, but now its taking around 4 or 5 mins to fully restart. 

 

Seeing hangups more and more now :( Its so random

[sinnaman 90]

Hmm, we don't seem to be seeing the same issues then. Our server is usually back up within a minute of our manual restarting the server (longer if the scheduled one kicks in), and we are seeing less and less hangups. We are seeing more temporal hangups though, where the server will appear frozen for a few moments (up to half a minute) before returning to its normal state.

 

Could be the difference between public/private hives?

[The Riddler (DayZ) 410]

Possibly but I wouldn't think that would make a difference? 

 

Need to check with our other admins to see how yesterday (Saturday) went as I didn't have a chance to play. Being weekends are always solid with players at all hours I'll be curious to see how many times we manually restarted.