Server Lag | Tried Almost Everything

I come in request of some help. I've tried almost everything. I've spent weeks going through Google, messing with flags and switches and everything I could possibly think of. My server ALWAYS lags at some point, and very bad. Sometimes it's fine, it can take 15 players on and not lag at all. But other times, it just starts lagging horribly - we're talking 20+ seconds chat/block lag at 50-80% CPU usage. And it doesn't go away either unless I restart the server (usually) or just let it go on for a few hours. And this is just 15 people - shouldn't I be able to have more than that on?

Server Specs
CentOS 5.5
Intel Core i7-2600 @ 3.40GHz
16GB DDR3 RAM
300GB 15K RPM SAS
1Gbps Private Line

java -version
> java version "1.6.0_17"
> OpenJDK Runtime Environment (IcedTea6 1.7.5) (rhel-1.16.b17.el5-x86_64)
> OpenJDK 64-Bit Server VM (build 14.0-b16, mixed mode

Run line: java -d64 -Xms8G -Xmx15G craftbukkit.jar nogui

bukkit version
CraftBukkit RB#670, MC 1.4
Plugins: WorldEdit, WorldGuard, ClothColor, BorderGuard, Colorme, PvPReward, Permissions, Elevators, HeroicDeath, Cleaner, mcbans, ecoCreature, Parties, HeroBounty, Silence, Command iConomy, iConomy, Essentials, Blacksmith, iConomyChestShop, MCStats2
Disabled: AutoSave, BackupPlugin

Essentials has a release memory on player log out which is set to false.

I don't run the Nether anymore, even pre-generated because that only makes it worse. Just a single map, limited to 5000x5000. Mobs are on. PvP is on. Animals are on. Warping is disabled aside from /home and /spawn.

My console only spits out "Server cannot keep up..." if I do something big with WorldEdit (like add 10000 blocks of water). Other than that, never see it. I never get any errors, actually - everything seems to run smoothly. Just very laggy at times.

The only thing running on this box is the game server.

I've gone through all the large 'optimize' threads and played with the java flags randomly, hoping one would work - but none did. Maybe I just never got the right combination, it's a confusing subject, even after reading all those threads.

Only two things I haven't attempted yet, because they seems like daunting tasks, is run the server on a RAMDISK or tmpfs and update to Java 7. Java 7 seems ridiculously hard to install without the .rpm (and it's beta) even after searching Google, however I am a Linux newb. RAMDISK I've only looked into a little, because I've read differing opinions on that - helps, doesn't help. I figure I'll try at least that out and report back, not sure about Java 7.


Could anyone maybe give me some idea of what is wrong? Is there anything I could do to get rid of the lag? Thanks in advance!

 

2 things i would try right off the bat, replace your openjava with oracle java and update it to 1.6 24, then disable mcstats. You should be able to update/change the java over with an rpm but i use ubuntu so cant help with the particulars on that.

with a 15000rpm disk and very few cant keep up notices its probably not access so ram disk probably wont do much for you, it just puts the world in ram so its faster to access.

The other thing you could try would be using the following as your start script:

Code:

java -server -Xmx2G -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:ParallelGCThreads=2 -XX:+CMSParallelRemarkEnabled -XX:+DisableExplicitGC -XX:MaxGCPauseMillis=500 -XX:SurvivorRatio=16 -XX:TargetSurvivorRatio=90 -jar $folder/$files/$mc nogui

replace the $folder/$files/$mc with your information. Also make sure you are running the 64 bit versions if you can.

Hope some of this helps.

 

I'll give that all a shot and let you know how it goes, thank you.

EDIT: Could I ask why I would be setting -Xmx at 2G? Isn't that a bit low for 16G total RAM?

 

I also just noticed this other thread down the list, http://forums.bukkit.org/threads/40-slot-laggfree-vanilla-server-but-we-got-plugins.12394/

you both have a few of the same plugins, remove all the common plugins one at a time and see what happens, I also know borderguard just completely rewrote their code, that can lead to unknown bugs, they touted it as running on a separate cpu thread so these are just some more thoughts.

 

Yeah, I've been messing with swapping plugins constantly. I actually just added BorderGuard today and I've been experiencing this for a long time, so I'm not sure if that'd do it - but I'll certainly check it out. Maybe it's just an unhealthy combination of plugins...

I'll definitely try that if I'm still experiencing the issue after changing to Oracle/Sun JRE, and running your suggested command line.

 

Thanks for the info on borderguard and the fact you just added it so its not really the culprit, good to know.

 

Code:

bukkit version
 CraftBukkit RB#670, MC 1.4
 Plugins: WorldEdit, WorldGuard, ClothColor, BorderGuard, Colorme,  PvPReward, Permissions, Elevators, HeroicDeath, Cleaner, mcbans,  ecoCreature, Parties, HeroBounty, Silence, Command iConomy, iConomy,  Essentials, Blacksmith, iConomyChestShop, MCStats2
 Disabled: AutoSave, BackupPlugin

Essentials has a release memory on player log out which is set to false.

I used to run Essentials a long time ago. It's bloatware plugin and caused plenty of issues, lagging being one of them. Disable it and download General or DefaultCommands. See how that goes.

Is MCStats SQL?

iConomy is resource-heavy, but from your list it shouldn't really be effecting you. Try getting rid of mcbans and replacing it with KiwiAdmin, or if you use DefaultCommands you won't need to.

Get the plugin RemoteTookKit. Best thing ever, it will auto-save in set incrementals as well as restart the server in set incrementals. This helps release usage.

Honestly though, your server specs are amazing. I'm very shocked you're having issues. Try re-doing all the config's aside from Permissions. Also, as the above user noted, try out Java. I definately recommend learning how to install Java 7 as it is an increase in performance. My server uses Java 7-b-136 and noticed an increase in performance.


Good Luck!

 

I'll look into a new basic commands plugin to replace Essentials, starting with the 2 you recommended.

MCStats is not MySQL as far as I can tell - though I haven't found another stats program that tracks what I really need (pvp kills and deaths).

I'll look into KiwiAdmin if I don't like the feature-set of DefaultCommands!

And I'll definitely look into RemoteToolKit. Hadn't heard of that, but it sounds like a much better alternative to the others I am using.

Yeah, I am quite surprised myself. I'll see if I can get through a Java 7 installation alive, haha. Hopefully with that (or at the very least, Java 6) and swapping in some more light-weight plugins from the recommendations, it will start running like it should.

Thank you!

 

try using world edit or world guard command /butcher cant remember witch it will kill mobs arround u check if that will remove the lag

 

that was probably unintentional.. crank it back up..
(unless he suspects bad RAM, which i doubt)

 

Yeah, I have it at 15G max, still. I managed to manually install Java 7 on CentOS - it was a lot of trouble, but I think it was worth it. I have noticed an increase in performance right off the bat with that. Still struggling to find decent replacements for Essentials/Spawn. General seems like it will be the one, even though it won't fulfill all my needs.

I've left MCBans in for now. Removing it made no noticeable increase or decrease in latency or server stability.

I'm getting tiny spurts of lag at random times, particularly on player log ins, but so far nothing as big as before. I'll be watching my CPU/latency closely to see how it goes from here on out.

Just need to get RTK running properly!

EDIT: Butcher always clears out around 800-1200 monsters, which may be a lot. I remember a while ago, I would only kill around 400 per butcher. I may look for a plugin that limits that, since it could be part of the issue.

 

I'm so glad it's not just me...

I've recently set up a server of my own and I am experiencing the same issues and feel like I have tried everything to resolve it but nothing works. The server itself is more than powerful and is backed by a fiber optic connection to the internet (with plenty of bandwidth).

Full specs:
3.3GHz Quad Core i5 2500
8GB DDR3 RAM
6GB/s SATA III HDD
1Gbps Upload

Ubuntu Server 10.10 (Maverick) x64
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

I've tried a new network card, using the TCP Buffer Proxy, checking my DNS and network connection and increasing the RAM available to the game, nothing makes a difference.

One major clue I have had so far is that it seems to be absolutely fine until a particular person joins the session. It then immediately lags out and as soon as he leaves, the lag disappears.

I'm wondering if my DIR-615 can't handle the network throughput so that's going to be my next test, buying a decent router.

My plugin list is here if anyone is interested.

Edit:
Just out of curiosity, what are your hardware brands and models (just to compare chipsets etc)

 

Please don't forget to write here if you find: easy wayt to install java7 and find replacement to Essential

I have same problem than you when i have more than 10 Players : heavy lag, rollback blockcreate.

I don't have a weak homemade server, but datacenter server
Intel Xeon 8 core (X3450)
8 Gb RAM
1 Gbit/Sec (host in datacenter)

That is just not possible to have these lag with that.
I begin to test and see tant NPCX make me log of trouble with 10 NPC inside the main town. Achievement/stats seem make trouble also.

That strange because if i went far of the town : lag gone !

 

@Guenael: What plugins are you using?

 

pm me plase on how u got java 7! i cant get it

 

To install java 7:
cd /opt
wget http://www.java.net/download/jdk7/a...re-7-ea-bin-b137-linux-x64-07_apr_2011.tar.gz
tar -zxvf jre-7-ea-bin-b137-linux-x64-07_apr_2011.tar.gz
alternatives --install /usr/bin/java java /opt/jre1.7.0/bin/java 2
alternatives --config java
2 (enter)

 

Something no one ever suspects. Shit Internet. Your home internet might be under heavy load, it might be crap. I have a server in Germany on a 100mbps line and I am in Australia and I get zero lag. For ages i got really bad lag and it turned out to be my home internet. I made some patient and not so patient phone calls to my ISP and it's all better now.

I receive complaints of others lagging, I tell them it's their net because 9 times out of 10 it is their internet being crap. For what it's worth it could be your ethernet cables

My server has 8GB with a 4 core AMD crap thing. Has 2 crappy 2tb hdd's and a not so good external connection. Runs perfectly. PS: the server runs more then just SMP and java is run at 2048M.

You can see a list of plugins I use at my website. I tend to use minimal plugins, check ALL of the features of plugins you use and try to minimise plugins as much as possible and it'll help too. Make having as little plugins as possible your goal.

 

Upload bandwidth isn't the problem. Most people here have a 1Gbps line or above for the upload.

The player's bandwidth seems to be a factor though. If the player has a throttled bandwidth, high ping or unreliable connection, it has the potential to cause all sorts of horrible lag. From what I've heard, this game seems to be using TCP instead of UDP for all its packets and them spamming them to players at a high rate with no buffering, which is pretty advanced idiocy when writing a networked online game that needs to handle many players (I used to be a professional games network programmer).

Ethernet wasn't the problem either as I replaced them completely. I am trying a new router today though to see if it makes a difference.

 

Also try a new Ethernet card if you go the whole 10 miles.

 

Already did unfortunately, didn't work. I even went as far as to purchase a 10/100 card instead of a 10/100/1000 card just to be sure it wasn't a problem with gigabit networks. Both NICs have Realtek (ugh) chips in though so I ordered a new Intel NIC with the Cisco router. Will let you know tonight if it makes a difference.

 

I have a cisco router with a realtek onboard card. Mind you I am limited to 512KiB upload but I still have fine connection to most servers (I have my 20MiB connection to thank for that).

 

Are you basically saying that a player with a high ping can cause all players to experience a latency increase?

 

From my personal experience, that has been the case. We had 8 people on, with no lag whatsoever. The laggy player joins and pretty soon, all mobs stop moving, CPU goes to 100%, server lags out for everyone. He leaves, everything goes back to normal. It may or may not be his internet connection. Could just be where he is in the map - haven't had a chance to investigate it much.

 

Yes it is possible. Especially if everyone else is relying on their packets. If their packets are delayed then everyone elses will be.

 

Hm, I'll have to look into that.

However, today I did get rid of a lot of lag by using MobLimiter plugin. I normally have 1200~ mobs spawned at once, so I blacklisted slimes/squid and set the max limit to 250 and so far I haven't had a single spurt of lag for anyone. It might be because I restarted the server after installing it, but it's been up for several hours and things are going very, very smooth.

 

I can also say that the laggy git who usually crashes my server was able to play fine earlier after using MobLimiter to disable squids and slimes.

 

I had bad lag issues with 670 but tried out the new 677 version last night and it seems less laggy than 670 did but still not as good as 617 used to run. SirHedgeHog have you played around much with values on moblimiter? What size map do you have and does 250 seem enough? I am currently try out 400 but I was unsure what value to start with. I pretty much dont notice any difference with the amount of NPC's around.

 

250 on my map is very small; but I'm going to mess with it. My server hasn't lagged at all, ALL day. It's a miracle. MobLimiter has saved me now, as far as I'm concerned. Why mobs cause lag, I do not know - but apparently they do. I like this because now I can still have mobs, but they are just rarer. PvP server anyway.

My map is limited to 5000x5000, with an average of 20 users spreading out constantly. I may up it to 400 as well and see what happens.