Server claims 100% CPU, hangs, and crashes daily. No crash report or errors in log.

Code:

INFO Plugins (29): World Portal, LagMeter, SkyChest, WorldEdit, DwDNPC,
Storm, mcjobs, PermissionsBukkit, BOSEconomy, Vault, WorldGuard, CityGates,
PayButtons, Shopkeepers, GoldRush, ChestBank, OrebfuscatorLite, OpenInv,
DeathChests, ItemSwitcher, mcMMO, ButtonWarp, Residence, ShowCaseStandalone,
Sellchest, CommandSigns, Sign Bank, ResidenceSigns, AntiCheat

I'm aware SellChest is out of date, however this issue was happening before we added it.
We also got this issue before we added OrebfuscatorLite, Sign Bank, CityGates, DwDNPC, CommandSigns, and possibly a few others, not 100% sure on them. Though these I'm fairly positive about.

Our host is Minecrafted.net (which I'm not a huge fan of, but I don't own the server. I'm merely working to get it fixed at the moment as our owner is having surgery.)

1,2,3:

Code:

Relevant Details:
- Minecraft Version: 1.3.2
- Operating System: Linux (amd64) version 2.6.18-308.4.1.el5
- Java Version: 1.7.0_04, Oracle Corporation
- Java VM Version: Java HotSpot(TM) 64-Bit Server VM (mixed mode), Oracle Corporation
- Memory: 1818129440 bytes (1733 MB) / 2095185920 bytes (1998 MB) up to 2095185920 bytes (1998 MB)
- JVM Flags: 8 total; -Xmx2048M -Xms2048M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=6 -XX:+AggressiveOpts -Xincgc

4: We're using Multicraft, and this is where it claims 100% CPU before the server hangs. Up until then LagMater claims fine TPS and normal RAM usage.

5: CraftBukkit version git-Bukkit-1.3.2-R2.0-b2396jnks (MC: 1.3.2) (Implementing API version 1.3.2-R2.0). Also no custom jars. So I have no control over what jar is uploaded.
6: Whatever Multicraft uses, we have no custom scripts apart from auto-save and backup on command.
7: Listed at the top of OP
8: Getting NO ERRORS apart from the auto-update issue at the moment. We have gotten errors in the past and I removed those plugins, and it still hangs. We occasionally get

Code:

"2012-10-17 19:48:51 [WARNING] Wrong location for EntityFireball['entity.Fireball.name'/287, l='world_nether', x=-136.00, y=242.00, z=-181.00] in world 'world_nether'!
2012-10-17 19:48:51 [WARNING] Entity is at -136.0,-181.0 (chunk -9,-12) but was stored in chunk 0,0"

which I found out is caused by Storm, but even removing Storm doesn't fix the hang/crash
9: I have tried removing any plugin to throw out errors or plugin we didn't have on our last server. I have not tried removing all the plugins because this issue isn't immediate. Sometimes it's 6 hours later, sometimes it's an entire day later. I have however set up a test server on my PC (will post it's credentials below) with the same plugins we have, plus a few more and it's been going strong since 2 AM on Oct 17.

Test Server Information
1: Windows 7 64 bit
2: x64
3: JAva SE Rutime (1.7.0_05-b06), Java Hotspot 64-bit server (23.1-b03, mixed)
4: Running straight from CMD
5: Craftbukkit Recommended 1.3.2-R1.0 from http://bukkit.org/threads/craftbukkit-1-3-2-r1-0-is-now-available.102934/
6:

Code:

@ECHO OFF
SET BINDIR=%~dp0
CD /D "%BINDIR%"
java -Xmx1024M -Xms1024M -jar craftbukkit-1.3.2-R1.0.jar
PAUSE

7:

Code:

WorldPortal,LagMeter,SkyChest,Minequery,WorldEdit,DwDNPC,Storm,
Reforestation,mcjobs,PermissionsBukkit,Quests,BOSEconomy,Vault,TerrainControl,
LWC,WorldGuard,UberForest,CityGates,TelePads,Shopkeepers,PayButtons,GoldRush,
ChestBank,OrebfuscatorLite,OpenInv,DeathChests,ItemSwitcher,mcMMO,ButtonWarp,
ManySmallTweaks,Residence,ShowCaseStandalone,Citizens,Sellchest,CommandSigns,
SignBank,Alchemist,ResidenceSigns,va_postal,Denizen,Sentry,CitiTrader,Blacksmith,
Builder,AntiCheat

8: No errors. Seems to be running strong
9: same as above
10: same as above

I'm not real server savvy so I don't know if we should be expecting an error in the logs or not. The host is telling us that it's not their machines and that it's out plugins causing the issue, claiming that it's throwing errors out...but they wouldn't give us any more infomation. We can't see any errors so I don't know what they're referring to, and why wouldn't they post to us the errors they're receiving?

My test server has an 11 hour uptime as of now. I'd prefer not to leave the test running for two days with my PC up, but I may have to to see if it will eventually crash as well.

I'm hoping for an efficient fix however if we end up having to remove all the plugins one by one that would be a nightmare considering how long it can take for the server to crash again.

I can backup the world and plugins, wipe the FTP, and re-install them but would that fix the issue? I'd hate to have to re-do the configs to all of the plugins we have. Any time I edit yml's I check it with the parser.

On our previous server, hosted with Beastnode, we also used Multicraft and never had a hang issue, so I'm unsure of what to think about this honestly.

Any help would be appreciated. If any more information is needed please let me know as I'll be checking this throughout the day/week/month or whatever is needed to find the fix.

All in all I guess my question is if anybody has every had an issue similar to this? When the server hangs I'm forced to go into the Multicraft panel and click stop...then it waits for a bit and has to forcefully stop the server. No errors. No crash report. Nothing useful from the server

Thanks.

Post updated with more host OS info.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Change this:

Code:

-Xmx2048M -Xms2048M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSIncrementalPacing -XX:ParallelGCThreads=6 -XX:+AggressiveOpts -Xincgc

To this:

Code:

-Xincgc -Xmx2048M

You won't need the rest, and you actually negate some of it anyway, since you try to combine two different Java GC methods, and Java will use the last one mentioned.

As for the crashes, it will help to see your full server.log to help understand what exact problems you are running into.

 

I can't change that I don't think. The host has control over that unless there's a way to change it that I'm unaware of? And the server.log lists nothing out of the normal as far as start-up goes that I can tell and when it hangs and has to be forcefully stopped it lists only that, though I will post the most recent log.

http://pastebin.com/uM7unyqV

 

Here is the steps i need you to follow, please do not skip any..

1. Download a plugin called NoLagg
2. Disable all of it's features BESIDES examine!
3. While the server is running at 100%, attempt to run /lag examine
4. When done, it'll create a exam file *in /plugins/*
5. Move the file to your computer
6. Double click NoLagg.jar
7. Open the file and.... examine! (Large MS = bad)
8. (optional) upload the file on here for me to examine
9. Remove all high MS using plugins
10. See if the problem continues...
11. If the problem continues, attempt to install and download Spignot from spout (By mikeprime/MD_5)
12. If the problem IS STILL THERE! Delete your existing map/maps
13. If you still have the problem, flip off your host, and migrate to Avantehosting, or Beastnode!




Hope i helped!!

 

When the server runs at 100% nothing can be done. There is a "dead-zone" so to speak that makes you think you're still playing but nothing you do gets registered and nothing you type gets sent to the server. It's as if you're lagging out. So I get disconnected from the game, go into the Multicraft Panel and click stop, and it claims it's stopping...but then after a while it notices that the game isn't stopping and has to forcefully stop the server.

However I have removed some plugins recently and it's been running for a few hours and no hangs as of yet. If I get the hang again I'll update everything. Our server also doesn't have a lot of traffic, currently 3 or 4 players max but the last two days have been around 2 players, on 2GB of RAM.

Strangely enough my test server is still running almost 24 hours later and doing just fine. I'm not sure why it would do this on my host, but not my PC if it's a plugin issue. Especially since I copied everything straight over from the FTP.