Dayz Server Monitor...

[cm. (DayZ) 28]

Anyway we can get a "lite" version of this that doesn't control how things start/stop.

we have firedeamon setup to control all the starting and stopping of servers. we also have a web interface that admins can login to so they can restart servers if anything goes wrong without handing out RDP access. I noticed it does a lot of "checking" which is fine, but I would prefer it if I could control when to start/stop my servers and not some random program.

[Mojo (DayZ) 66]

Dear God, this right here.

I don't mind an auto-updater -- in fact Owl offered to code a deployment system with delta patching two weeks ago and received no reply -- but I want to control the server we pay for out of our pockets the way I want to control it. I want to set core and thread masks how I want to set them. I want to load balance how I want to load balance. I want to monitor how I want to monitor. I want trusted moderators to have the ability to restart the server without needing RDP access.

I'm not going to lie, hearing rumours of this server monitor honestly makes me consider pulling the plug.

[cm. (DayZ) 28]

I've disabled it for now anyway as it was just too annoying to use, back to using firedaemon.

[Mojo (DayZ) 66]

I've disabled it for now anyway as it was just too annoying to use' date=' back to using firedaemon.

[/quote']

Yup. Got reports of the server "crashing", saw somebody had logged into RDP. Stayed logged in for two hours, and came back to an RDP desktop that looked like this:

Needless to say, that is not how I leave things. FireDaemon services disabled completely, things spitting out errors about needing to run cfg files again (what the hell? how do you even run a config?), what looks like somebody trying to elevate the server executable process because one of the errors is bitching about process elevation and leaving the properties window open. Just wow. Two hours. And then left like this.

As a smoker, my reaction looked a little something like this:

Needless to say, the Chicago 1 Series will be down for a few hours for, uh, "maintenance". I'll likely change to environment variables just to add a difficulty setting for things, too.

[cm. (DayZ) 28]

hahaha I have the same sort of thing, all of our difficulty settings were wiped for whatever reason, the app also wasn't set up properly and was launching our second server into a non-existant profile... people could see markers and other players on the map ffs.

Needless to say I wrote a message/instruction guide in paint and set it as our wallpaper haah.

[Mojo (DayZ) 66]

hahaha I have the same sort of thing' date=' all of our difficulty settings were wiped for whatever reason, the app also wasn't set up properly and was launching our second server into a non-existant profile... people could see markers and other players on the map ffs.

Needless to say I wrote a message/instruction guide in paint and set it as our wallpaper haah.

[/quote']

Oh I've done that before, left big font size = 32 notepad messages open and focused in the RDP. That way even if they have wallpaper turned off...

I looked into it more, found multiple cfg directories, as if somebody thinks you actually need different paths to use two different files when there's no default file name for the configs that can be specified.

All our server-specific tweaks like MTU, bandwidth, all that that was carefully measured on day zero before any deployment of DayZ -- wiped out. Just gone. Has to all be redone now (thankfully kept notes). The servers are renamed, too.

I'm seriously wondering if it's worth fixing this shit or just refunding our donors out of pocket and pulling the plug. This is fucking nuts.

Top it off with the total lack of respect for people who've already paid for licences for FireDaemon, and this is just unbelievable.

The best part is my other admin (and co-leaser) is working on a bunch of physics stuff right now (I've been trying to convince him to let me shoot his laser at the moon, maybe now he'll let me do it just to cheer me up) and doesn't even know about this yet, as he's checking in sporadically for the last couple of days. He is going to hit the roof.

I had finally found a 3D modeller too, one with Maya Enterprise, and was going to work on making a bunch of mod-suitable clothing/models/skins (my last bandit skin was a fiasco since I was totally unfamiliar with Arma unit configs after five years away from BI game modding and the person I got to help me with it did the one thing -- hex edit -- I told him not to do). I wanted to do it gratis, just for the pleasure of working on a game again, help with the load where I could. Given this total lack of professionalism and respect, you can damn well be certain that's out the window.

And given this notion of carte blanche over expensive computer hardware I am financially and legally responsible for, and the idea of installing executables that nowhere fall under the purview of our previously agreed-upon arrangement, me and Owl are going to have a long talk about how we're going to proceed from here tomorrow morning.

[rocket 16567]

Guys I understand the problems, to say we get pretty desperate is an understatement. We have over 200 servers that we have to try and monitor, and a handful of servers somewhere are causing catastrophic data loss (corrupting everyones position data).

Often we end up in an emergency situation and we have to react very fast. The mod has exploded out of all proportion and it's just been insane to try and manage everything. This has put us into difficult circumstances.

What I would recommend, is that if you host several servers and you can provide a key point of contact, that you can deactivate the RDP access for DayZ staff and then you take over the full active management of the server. This now our preferred method as we simply cannot cope with managing the servers.

This is how we manage Multiplay and the other large groups, they simply keep them ticking over and check for updates etc...

As I said in another thread, there was no intention to steamroll server owners or treat them like crap. To say things are, at times, completely desperate is an absolute understatement. We have had to scale up from 0 to over 75000 active users in under three weeks. We went from having a trusted server host environment to having to lock down everything we could server side as we had a couple of server admins doing very dodgy things.

The risk of the database being wiped or corrupted has at times been quite high, and this necessitated some pretty extreme reactions from the DayZ team. We're completely overwhelmed and just trying to stay afloat with it. All I can really ask is that if people can highlight this stuff like you have, we'll try and work through it.

My recommendation is that if you're comfortable managing it yourselves, you can take full responsibility for it. Note however that would mean that you need to apply the patches and updates manually, and that DayZ staff won't be able to resolve difficulties at your end. Also unmanaged servers that aren't working because they aren't being restarted or etc... can end up blacklisted to prevent data corruption (this data corruption has happened twice from badly managed servers).

Hope that provides some background to the situation.

[anthony_henson2003@yahoo.com 33]

Guys I understand the problems' date=' to say we get pretty desperate is an understatement. We have over 200 servers that we have to try and monitor, and a handful of servers somewhere are causing catastrophic data loss (corrupting everyones position data).

Often we end up in an emergency situation and we have to react very fast. The mod has exploded out of all proportion and it's just been insane to try and manage everything. This has put us into difficult circumstances.

What I would recommend, is that if you host several servers and you can provide a key point of contact, that you can deactivate the RDP access for DayZ staff and then you take over the full active management of the server. This now our preferred method as we simply cannot cope with managing the servers.

This is how we manage Multiplay and the other large groups, they simply keep them ticking over and check for updates etc...

As I said in another thread, there was no intention to steamroll server owners or treat them like crap. To say things are, at times, completely desperate is an absolute understatement. We have had to scale up from 0 to over 75000 active users in under three weeks. We went from having a trusted server host environment to having to lock down everything we could server side as we had a couple of server admins doing very dodgy things.

The risk of the database being wiped or corrupted has at times been quite high, and this necessitated some pretty extreme reactions from the DayZ team. We're completely overwhelmed and just trying to stay afloat with it. All I can really ask is that if people can highlight this stuff like you have, we'll try and work through it.

My recommendation is that if you're comfortable managing it yourselves, you can take full responsibility for it. Note however that would mean that you need to apply the patches and updates manually, and that DayZ staff won't be able to resolve difficulties at your end. Also unmanaged servers that aren't working because they aren't being restarted or etc... can end up blacklisted to prevent data corruption (this data corruption has happened twice from badly managed servers).

Hope that provides some background to the situation.

[/quote']

will 1.5.8 have any additional security measures to protect against some of the hacking/duping/crashing of servers that has been going on?

[Difecti 11]

I have Monitor running on my server, and while i though at the start that it was causing issues (I was wrong btw), I actually find it easier to manage the server using it. it will lighten the load on the Dev's by giving a single point of update, without them having to go in and update manually. I'll be leaving my RDP up for them, incase I do need help with something DayZ related, but I see the Monitor as a godsend for the Dev Team.

Yes, it may change your settings at first, but they're not too hard to reconfigure.

[rocket 16567]

will 1.5.8 have any additional security measures to protect against some of the hacking/duping/crashing of servers that has been going on?

I'm trying, that's been the delay. I spent about 20 hours straight coding attempts to fix all the duping bugs before I eventually crashed out and decided I had to sleep or I would break something.

Given everything I do is done at the script level, and not at the source level - it is very hard for me to effect decent changes in this regard. The only real solution is for the game mode to be compiled into the executable itself and then prevent it from running SQF scripts. This alone would stop hacking and such problems.

If/when this becomes it's own standalone game, this kind of thing is very easy to fix.

PS. Server hosters, please feel free to add me on Skype and you can raise any issues directly with me. If you email dayzdevteam@gmail.com with SERVER HOSTER: ROCKET'S SKYPE in the subject I will reply with my skype address. Then if you have any issues, I can get them directly and immediately.

There are a number of hosters whom manage their systems entirely for themselves, either just giving us access to their RDP "just in case" or (as I prefer) they don't give us RDP at all but manage entirely themselves and apply the updates. This is the model we prefer, because we simply don't have the resources to setup/manage more than about 20 servers ourselves.

[anthony_henson2003@yahoo.com 33]

will 1.5.8 have any additional security measures to protect against some of the hacking/duping/crashing of servers that has been going on?

I'm trying' date=' that's been the delay. I spent about 20 hours straight coding attempts to fix all the duping bugs before I eventually crashed out and decided I had to sleep or I would break something.

Given everything I do is done at the script level, and not at the source level - it is very hard for me to effect decent changes in this regard. The only real solution is for the game mode to be compiled into the executable itself and then prevent it from running SQF scripts. This alone would stop hacking and such problems.

If/when this becomes it's own standalone game, this kind of thing is very easy to fix.

PS. Server hosters, please feel free to add me on Skype and you can raise any issues directly with me. If you email dayzdevteam@gmail.com with SERVER HOSTER: ROCKET'S SKYPE in the subject I will reply with my skype address. Then if you have any issues, I can get them directly and immediately.

There are a number of hosters whom manage their systems entirely for themselves, either just giving us access to their RDP "just in case" or (as I prefer) they don't give us RDP at all but manage entirely themselves and apply the updates. This is the model we prefer, because we simply don't have the resources to setup/manage more than about 20 servers ourselves.

[/quote']

Thanks for the reply makes me feel a little better knowing that you are trying your ass off to fix things within the limits of what you can do. Been up for about the same amount of time waiting for 1.5.8 lol.

[Difecti 11]

Every now and then on the IRC channel, someone will come to myself or Greg[uK4] and ask us for help with server issues and we help as much as we can, with either of us being able to get the issue resolved (Unless its IP whitelisting). If needed, I'm willing to help in some kind of (official) capacity, as I'm sure Greg[uK4] would be willing to also, that's if you need additional hands on that side of things.

[rocket 16567]

Definitely additional hands are helpful. We've exclusively taken our team from server hosters, as it is much easier to trust someone who just trusted us with root access to their box. Even from those helping on the code side, they are nearly all from our major server hosters. Best place to start is to email Matt at dayzmatt@gmail.com. He processes a number of our servers and if you keep in touch with him, he can see about getting some help (I think his email inbox is now getting overwhelmed too).

[blergh (DayZ) 14]

I've sent matt an email explaning that i have the proper knowledge to give a helping hand, hopefully i can be of some use to you guys, seeing as i work with this stuff (kinda).

[shwat 3]

It's nice to know "Jim" won't be fiddling around with things he clearly doesn't know how to fiddle with in the future, at least on the Chi1 servers.

Database corruption from poorly-managed servers, eh? That would explain why the server monitor is supposedly rebooting servers every 6 hours, yes? Even if they just automatically rebooted from a crash recovery shortly before, or so I've heard. What happened to the admins for these servers? Players notice when a server needs to be rebooted, because loot/zombie spawns become abnormal, desync occurs more often, etc. Do these admins just not give a shit, are they inactive, or are they busy duping stuff off in a corner? Is it just innocent inaction?

[borstal 4]

I had a weird issue with my monitor pogram that would create a second instance of the server as if it thought the first one crashed when it indeed had not.

This of course caused both servers to crash as my box isn't built to handle two servers.

Seems to (i hope) have been fixed in 0.7.4 of the monitor.

[Mojo (DayZ) 66]

Database corruption from poorly-managed servers' date=' eh?[/quote']

Yes. I know how to do it. I've known how to do it for several weeks. I will not repeat it here (or anywhere) for the same reason I sent in the duplication glitch in email and not a forum post.

Do these admins just not give a shit' date=' are they inactive, or are they busy duping stuff off in a corner? Is it just innocent inaction?

[/quote']

The admins are, in most cases, ignorant. They've never administered a server before, are enamored with the mod, and are utterly clueless. Probably being in the demographic with the most liquid cash (late teens), the cost of server administration doesn't bite into their monthly overhead, whereas for my old ass (31 and grey at the temples and after seeing the mess on my Rdesktop, greyer at the temples) supports a wife, a dog, vehicles, bills, etc.

Not to say that the ones doing this are bad or whatnot. This is what will get them their admin stripes. But ignorant? Oh most definitely. In a few weeks or months, these newbie admins will discover that being an atheist is wrong, because they can prove God exists. They will know this because they will have a direct conduit to God's wrath, and discover power beyond their reckoning when their cup overfloweth. Perhaps this God is Korne. BLOOD FOR THE BLOOD GOD!

The admins who know administrating like to do things with batch scripting, environment variables, symbolic links, junctions, Session 0, thread masking, core masking, address assignment. We are very picky, and no two admins are the same. The new admins, well, they want things done for them because they haven't the foggiest just what they're supposed to do with their new-fangled boxes.

Admins come in many flavours. You have admins like me, who like to rip open the guts of the server and pare it down as neatly and efficiently as possible, and admins like Owl, who favour using the server's power towards automation. If you ever hear us arguing on TeamSpeak over server administrating, Owl is trying to automate something I would rather do with symlinks.

If we wanted to get to space, Owl would engineer a space shuttle and then engineer robots to build that space shuttle. I would just redefine gravity. In the end, both of us will accomplish the same thing through very different means.

These admins who require these hands-on server monitors are not yet there. Some of them will get there, some of them will fade. But right now they're learning. They have a lot of disposable income and a blank mind, and they've paid their tuition to Break Fucking Everything U.

[rocket 16567]

*shrug*

All I can say is, my philosophy is to walk a mile in a man's shoes before you criticize him. There are only three people who understand the full architecture of the system. There are only five team members of our server team, none of them are teenagers, all of them very qualified, together they are the ones who provide around half off all the provided servers.

We chose those people because they had trusted so much to us with gifting their servers, and they were the most familiar with the operation of the servers.

Just because someone is knowledgeable with Windows hosting, or running an ArmA2 server, does not mean they will be any good at dealing with the DayZ servers. A significant amount of hackery is involved and much of the interactions is complex and fraught with difficulty. The greatest difficulty we have is with server administrators who take matters into their own hands, and that has been the source of the last two major data corruption issues.

As I said, if you want to administrate the servers without providing RDP access then that is fine, and it is our preferred management method. However, this does mean that instead of us coming in to fix the server when there is a problem, the server will be blacklisted and unable to connect to the central system.

Those that we manage directly, we try to have a standardized layout with everything the same - regardless of how retarded that configuration is. We lost several admins today because of the abuse they received, which means that I have had to delayed the release of the update yet again.

I think, given how I can see you feel from your posts, my recommendation would be that you take a break from hosting DayZ until it is more stable. This would save yourself, and the team here, from much unneeded frustration and difficulty.

[Mojo (DayZ) 66]

*shrug*

All I can say is' date=' my philosophy is to walk a mile in a man's shoes before you criticize him. There are only three people who understand the full architecture of the system. There are only five team members of our server team, none of them are teenagers, all of them very qualified, together they are the ones who provide around half off all the provided servers.

[/quote']

I'm talking about the flood of new servers from people with liquid cash to meet the sysreqs and no experience admin'ing a server, not the DayZ staff. Or the other admins who do have experience administrating a server, for that matter.

I know this is what they are, because who gets the PMs and guys hopping on TeamSpeak asking for help?

^-- This guy.

[howichrgelazer 69]

I wonder, rocket...

Working with the process you guys have in place currently, maybe it may help to have somebody up some ticket system for server requests? Something like osTicket. That way you guys have a central source of documentation and every request is logged and clearly displayed. It might relieve stress on all the parties involved.

The reason why I bring this up, is that during my stint of getting a server operational, it passed through three different server team members. All of which didn't entirely know where the last one ended up. I ended up having to re-install ArmA to help one of them out, so they could start fresh. Eventually the files were ripped out of my install directory and put on my user's desktop. I don't mind, but now I have to either re-install again or edit my registry to allow for beta patching if necessary in the future.

As a server owner, I'd be willing to help contribute to a project like that, especially if it leads to easier methods of server setup/installation.

I also have help desk experience, not with osTicket, but with Spiceworks at my current job. I'm sure configuration knowledge is transferable in some degree between systems.

[rocket 16567]

Right, I'm going to level with you guys here:

I have received reports of server administrators, such as UK11, modifying files and changing scripts to change daylight hours and every other such thing. I have spent the last 36 hours (with four hours of sleep) coding fixes and changes and security closures all because of these server administrators. I have now wasted the entire weekend and will not be able to sleep tonight if I want to get the update out.

Yes, we are very grateful to those who donate their servers. I have the utmost respect for that. But we have a number of server owners who, once they have the files, and have proceeded to do the most insane things. Some of the team here were so fed up with the difficulties they wanted to start blacklisting the IP's for these rogue servers straight away. I talked them out of it, because of the discussions on the forums and the feelings and sentiments and because I thought we should try to make more of an effort to reach out to the server administrators. I now regret that decision.

So I am PLEADING with the server owners here. Please do not keep doing whatever the hell you want. You are now disrupting the entire project. Think about the bigger picture. This is an Alpha. I understand many have gone out and rented servers with high hopes of their clan sitting around the fireplace, holding teddybears, and singing kumbaya.

My weekend of updates was entirely ruined because of three server owners who decided to take matters into their own hands and make changes. Then, they criticize the server team here who have been working their whole weekend to try and help me.

[Mojo (DayZ) 66]

I wonder' date=' rocket...

Working with the process you guys have in place currently, maybe it may help to have somebody up some ticket system for server requests? Something like osTicket. That way you guys have a central source of documentation and every request is logged and clearly displayed. It might relieve stress on all the parties involved.

The reason why I bring this up, is that during my stint of getting a server operational, it passed through three different server team members. All of which didn't entirely know where the last one ended up. I ended up having to re-install ArmA to help one of them out, so they could start fresh. Eventually the files were ripped out of my install directory and put on my user's desktop. I don't mind, but now I have to either re-install again or edit my registry to allow for beta patching if necessary in the future.

As a server owner, I'd be willing to help contribute to a project like that, especially if it leads to easier methods of server setup/installation.

I also have help desk experience, not with osTicket, but with Spiceworks at my current job. I'm sure configuration knowledge is transferable in some degree between systems.

[/quote']

I would love you forever if you implemented this. My solution would have been a dynamically-generated proctored, timed online exam with a minimum pass mark of 80% to ensure that not only are the server system requirements met, so is admin knowledge.

[colonel_black 0]

Rocket, is there any chance we can have a "Server Administrator" section and then we can sticky the vital information that they need to follow. Then they have NO excuses to do things out of regulations.

[howichrgelazer 69]

I would love you forever if you implemented this. My solution would have been a dynamically-generated proctored' date=' timed online exam with a minimum pass mark of 80% to ensure that not only are the server system requirements met, so is admin knowledge.

[/quote']

Hahahahah, maybe that can be the captcha for registering an account ;)

But yeah, I think it'd help the server team immensely if they had a help desk. There's too many different lines of communication as of right now. Two different e-mail addresses, an IRC channel, and the forums.

We still have people asking if they can password servers or kick for reserves. If they were all in the help desk when they sign up to host a server, there's no arguing or debate. It's written clear as crystal.

Plus you'd be able to keep track of RDP information and who is running monitor and who isn't. Makes it a lot easier to head off problems.

I'd love to help make something like that happen.

[Mojo (DayZ) 66]

Rocket' date=' is there any chance we can have a "Server Administrator" section and then we can sticky the vital information that they need to follow. Then they have NO excuses to do things out of regulations.

[/quote']

Also something I would love to see. Due diligence: Did you as a server admin just corrupt the SQL server altering a certain file that shall go unnamed because God knows people will just alter it more?

You knew, or reasonably ought to have known, this was contrary to the server host whitelist agreement. Now you are blacklisted.

Edited for clarity: This is the editorial usage of the word 'you', a general statement to the audience. Any person interpreting this as a direct accusation against Black is a horrible monster.