Constant Server Crashes, Server Can't Keep Up Since R4: Will PAY for help!

Hello Everyone,

We recently updated to the R4 version of bukkit, and ever since we've had laggout-style crashes somewhat regularly (ranging from every 5 to 30 minutes), and an endless stream of "Can't Keep Up" messages.

We've been pulling our hair out trying to fix them (we're not noobs, and very aware of how our entire system works), and we've at this point tried just about every trouble shooting method we could think of, leaving us kind of desperate, and hence we're posting here with hopes that someone can help.

Unfortunately, the only thing the logs are showing us are the "Can't Keep Up" messages, as well as this one:

2012-02-18 15:39:11 [SEVERE] Block at -149,1,150 is FURNACE but has null. Bukkit will attempt to fix this, but there may be additional damage that we cannot recover.

There are also occasionally some inconsistent Java errors, but my hunch is they are more of a side effect of whatever the root problem is, as they aren't the same or frequent.

Here are some of the steps we have taken so far to no avail:

- Trimmed the whole world using WorldBorder and set a worldborder to limit the size of the map.
- Checked for Redstone clocks using RedstoneClock Detector and removed any suspicious clocks (a few switches, etc)
- Checked all plugins and versions to make sure they were updated (no specific plugins are throwing error messages in the logs)
- Downgraded Bukkit back to R3 (didnt fix it) then re-upgraded.
- Uninstalled some of our most performance intensive plugins and rebooted.

Our server is a Proliant ML570 with four xeon (single core) 2.2 ghz processors and 7 gb ram, with 15k SCSI disks. As we only have single core processors, we can only run 32bit stuff, and as such only have 2 gb of ram dedicated to Minecraft. I know this isn't a ton, but previous to this update we had several weeks with 50-75 people on our server on average without any problems. Instead, this seems to be a sudden, new problem unrelated to our server hardware.

We are home-hosted, but on a Verizon Fios business class line with 25up/25down, so I do not think that's the problem either. Nothing has changed on our router.

We do also host a web site and a Ventrilo server on the same box, along with a MySQL database for bukkit, but, as I said, none of this was a problem at all prior to the update.

Our best guess at this point is that there is something wrong with the database, as when using certain plugins that interact with the database (for example, our shops plugin), the lagouts seem to happen in tandem. Yet, I can't figure out what it is I could do to fix that problem, or even if it is the problem at all.

I've attached a good chunk of the unedited log file, but as you'll see it's not revealing much of usefulness (just a bunch of chat messages, "Can't Keep Up" errors and a few non-specific odd errors that I think are just from people DCing from the lag).

We're really at a loss, as we want very badly to eliminate this lag problem! I would actually be willing to PAY REAL MONEY to whoever can successfully eliminate this problem (name your price, within reason!)

 

You might want to run Chunkster see if it can fix the Furnace bug.

 

I had this, all I did was teleport to the offending furnace and replaced the furnace by hand with another furnace and it fixed my crashes.

But it still happens pretty frequently. Got it twice within the last three days.

 

Ok, so we ran Chunkster on the map and reuploaded it. At first it seemed to fix the furnace bug, but now it's back again. It's random furnaces all the time, so I don't really see a pattern or rhyme or reason there. Plus, I believe I read on another topic that that particular bug isn't anything to worry about per se.

Anyways, even after running Chunkster, we still seem to be getting intermittent lagouts.

Any other ideas as to what's going on?

And, as I see the Bukkit.org forum "Upload a File" feature doesn't seem to add the file to the post, here's a Pastebin copy of our logs:

http://pastebin.com/242pdMin

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Try downgrading to 1.17PEX

 

OK here's the latest.

We downgraded to permissionsex 1.7 (not sure why that would have helped anyways, but we did). Problem's still there.

We're considering trying Craftbukkit++, although we're nervous about venturing into that kind of territory. And, again, we had no problem with normal Craftbukkit prior to the R4 release.

Any other ideas?

 

That's what left my server at a drop. Just thought I'd throw it out there, it breaks a lot of plugins.
It just appears that
A) A MySQL server is timing out and your server isn't reaching the MySQL server in time
B) Connection is being entirely reset when the server is on a hang.

 

Yeah that's more or less what my hunch was, too, but I can't figure out why it's timing out. Any ideas? As for hints along those lines, I can tell you that our web site also uses the same MySQL server (it's hosted on the same box), and that we're running a stats plugin (Stats 2.0) that logs a lot of data in the database. The web site pulls some of that info for a stats page on our site (http://www.mumausserver.com/mc_stats).

Among our current bugfixing efforts, I have tried uninstalling the stats plugin, although, I have NOT tried shutting down our web site (for obvious reasons). Plus, all of this was fine before the update, so I'm not sure why Craftbukkit is now having a hard time accessing the database.

 

Maybe a new storage method of a plugin? Clear your databases?
Maybe disable all plugins using MySQL one by one to see which one is giving the error(if one is?)

 

Well, in desperation we just uninstalled the stats plugin we were using, and I dropped the related tables from our server entirely for now (people lose their stats, but oh well, we'll try anything at this point).

So, I've narrowed the MySQL aspect down to this: There are now only three plugins we have installed that access the database at all: iConomy, Jail, and PermissionsEx.

I deleted all entries from our iConomy database that had the default amount of money or less in them. That trimmed it to about 1000 entries.

The Jail tables are minimal (about 100 total entries).

And the permissions database is what it is --not anything too crazy, but an entry for all players who have come on, of course.

Same problem. Tons of lag when selling things, and Can't Keep Up messages

Any ideas where to go from there?

 

I would just try CraftBukkit++ anyway. It can't hurt to use it.

 

Would it be showcase? There was a few showcase errors. I use ChestShop, so I can't really say. Try disabling shops temporarily.

 

We more or less isolated Showcase as NOT being the problem, because if we spam /money pay x 0.1 the same problems happen (i.e., back to the database problem).

 

I've just always encountered problems with that showcase plugin. Maybe copy all of your current files over, take out the ones that use MySQL & run it without any MySQL locally. See if it crashes.

 

Don't mean to be snotty, but this topic should be in the bukkit help sub-forum.

 

Yeah I apologize -- I accidentally posted it here. Is there a way to move it to the proper forum?

We are seriously dying here from this. Let me stress again that I will actually pay someone with the expertise to help us if they can fix this problem!

Just for the record, we did also install Craftbukkit++ and although it does seem to help overall performance, we're still getting crashes every 5-30 minutes.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Posting a full server log would be helpful!

 

It's up at the top -- scroll up.

 

I would try running the server vanilla for a while if its a possibility.

Judging from your server log, it looks like the issue could have to do with not being able to host that many people with your internet connection, although im not to sure. Trying some internet tests for upload and download and dropped packets may be a good thing to try, possibly your ISP could be having issues.

 

I thought that the ISP was possibly part of the problem, too, but from what I've googled over the past few days Verizon Fios doesn't throttle (especially for business class -- i got that specifically because it ALLOWS for hosting servers). I'm pretty sure if we ran on vanilla we'd be fine, but then, that's solving the problem by dropping a nuclear bomb on everything, lol. Which is not solving the problem -- it's getting rid of the universe in which the problem exists, lol.

As far as having that many people on, we had two weekends in row with that many people on without problems. It was only AFTER we installed R4 (and on a slow day, traffic-wise), that the problems happened.

So, although I would normally agree with you, we actually did fine with 50-75 people on...until we took an arrow to the knee (sorry for the bad joke, it's been a LONG several days of epic troubleshooting).

 

Well if the server would run fine without plugins then it looks like its time for the wonderful task of removing plugins one by one to see whats causing the issues.

Also, what is your CPU and ram usage when it starts to get laggy?

 

Yeah, I mean that's basically all we're left with (process of elimination for plugins). Is there any way of profiling plugins or something to see what is causing the problem without doing the guessing game method? [*EDIT* FYI -- we've even already started and tried at least some of the process of elimination method with the plugins we've most suspected].

CPU usage has been hard to pin down. I use Webmin to monitor. Under "system info" it rarely says we're using more than 25%, but if i check the individual Minecraft process, it always says we're like 150%+ (I don't know why there's a discrepency -- if you know a better way of checking that inside of Ubuntu, it would help unto itself -- I tend to lean on Webmin). Ram usage is all of the 2gb we're giving it (I can't give it more than 2GB cause we're on four single-cores). But again, this is out of the blue, and not related to a jump in traffic -- those resources were more than enough before the arrow to the knee.

 

Isn't register outdated?

 

Try to run with no plugins,lag usually comes from poorly coded plugins

 

The CPU under system processes should be the right one.

150% CPU usage and 2GB ram for that many players doesnt seem like it would work that well. I suspect that is your issue to be honest, but to be sure I would go with trying to eliminate plugins

 

Other than it being some particular plugin, the CPU problem is my other hunch. But, as I said, I really did witness it with my own eyes -- we had 50-75 people on for like 2-3 days straight, two weekends in a row, without a problem in-between. I'm not lying, haha. And even now, our traffic has dropped substantially because of all the crashes, and even when there's just a few people on we still get the crashes.

Inevitably, our plans are to rent/buy another server, but given that I can't for at least another week for financial/logistical reasons, I'm trying to figure out why our perfectly well-running server all of a sudden started having problems.

 

Start the server temporary with a new map.
We had a bad chunk on our server that caused similar problems.

 

Another possibility is to downgrade to RB-3 for the time being. (You kept a back up.....riiiiiight?)


I always back up the whole server, Bukkit build, plugins, world files, and all. Just so I can take a step back if something like this happens.

 

Screw chunkster! It always screws up the terrain generator if you're running Multiverse! The furnace message is caused by the Craftbukkit.jar. You might wanna upgrade to 1940 or higher.

 

Did you recently upgrade CommandBook? A full plugin list would be nice as well.