They should definitely be kept, but the system that governs them needs to be changed. Shooting a survivor in self defence shouldn't result in someone turning into a bandit. I also think perhaps a system where the less murders you have committed within a given time (or perhaps over your collective lives) the shorter you stay as a bandit might be better. Or maybe you are only revealed as a bandit to a player if the player has followed you for some time. It's not easy to say what is the best way of governing it without trying, but as someone who doesn't shoot survivors I strongly feel bandits skins have a purpose. Yes in the real life it wouldn't be easy to tell who was a bandit or not, but in the real world life has more value, you don't just respawn at a beach when you're time is up.
