Constant Crash

Since upgrading to 670 I've been getting some frequent errors. Of course I'm getting the well known Read Timeout, but I'm also getting a nice slew of spam when players dc, and at least once - five times a day I'm getting a complete server crash due to the following error:

Code:

>#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000000006db85ab5, pid=4784, t
id=404
#
# JRE version: 6.0_24-b07
# Java VM: Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode windows-amd64
compressed oops)
# Problematic frame:
# V  [jvm.dll+0x395ab5]
#
# An error report file with more information is saved as:
# C:\Users\Sanctus Server\Desktop\Acryid - Beta 5\hs_err_pid4784.log
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

I will post the other errors when I get them again. Anyone have any ideas whats going on here?

 

Wow something is seriously wrong on that computer, java crashing is pretty bad. Uninstall and freshly install it, check the memory for corruption (memtest86+).

 

It's running out of ram one of the mods I installed has a memory leak I'm trying to narrow down which one.

This is an additional error that I am getting that never occurred until I moved to 670.

Code:

11:02:16 [INFO] /173.234.147.81:54207 lost connection
11:05:06 [SEVERE] java.net.SocketException: Connection reset by peer: socket wri
te error
11:05:06 [SEVERE]       at java.net.SocketOutputStream.socketWrite0(Native Metho
d)
11:05:06 [SEVERE]       at java.net.SocketOutputStream.socketWrite(Unknown Sourc
e)
11:05:06 [SEVERE]       at java.net.SocketOutputStream.write(Unknown Source)
11:05:06 [SEVERE]       at java.io.DataOutputStream.write(Unknown Source)
11:05:06 [SEVERE]       at net.minecraft.server.Packet.a(Packet.java:121)
11:05:06 [SEVERE]       at net.minecraft.server.NetworkManager.e(NetworkManager.
java:101)
11:05:06 [SEVERE]       at net.minecraft.server.NetworkManager.d(NetworkManager.
java:233)
11:05:06 [SEVERE]       at net.minecraft.server.NetworkWriterThread.run(SourceFi
le:89)

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Isn't that just a user disconnecting badly or do I mistake it for some similar error message?

 

This error is also happening to me. I upgraded from 617 to 670 and about 20-30 minutes after I start the server it freezes up and says READ TIMEOUT and CONNECTION RESET over and over until it gives a fatal error from the java runtime. Sometimes it doesn't even make it to the error and just becomes totally unresponsive, to the point that I can't even kill or terminate the process! Atm I am writing this post and I literally tried to bring the server down 20 minutes ago and even though the console for the server is down now the process is still running and java won't close no matter what!

System Specs: 2.4ghz Pentium 4, 1GB RAM, Ubuntu 10.04.1, Java SDK with Java Runtime 1.6.0_24b07
Plugins (all updated to latest versions as of about 12 hours ago of this post): Authme, Blast Control, Cuboid Plugin, DeathTP Plus, Essentials, FlyRidgeFly, Heroic Rebuke, iConomy, LogBlock, LWC, MCBans, mcMMO, Minecart Mania, QuickStrasse, Residence, Spells, Vanish, WorldEdit

 

You might be right the problem is that it's happening anytime any user disconnects.

Perhaps you can provide some insight to me Plague. I'm running the following setup.
Server Specs:
CPU: AMD Phenom 9850 Quad 2.5ghz
Ram: 6gb DDR3
Connection: Fiber 100down 5up
OS: Windows 7 64bit Home
Nothing else is running on the box but the server itself and I've been watching it all morning. It's steadily increased it's ram usage starting at around 900 megs at 10am to 2.2gigs now at 2:30pm Is this normal? Max players that have been online have been 7 today so far has been around 10 fluctuating mostly at around 7.

I run the following Mods all of which were up-to-date as of 9 a.m. CST this morning:

Code:

AssignPermissions

Backup

BlockCantKeepUpSpam

ClearInventory

CommandBook

CraftBukkitUpToDate

DelayedStop

dynmap

HelpPages

HeroBounty

iAuction

iChat

iConomy

Lottery

LWC

MagicCarpet

mcMMO

MinecraftRkitPlugin

MobSpawn

MyHome

Permissions

PlugMan

PvPReward

SimpleSave

SimpleShop

SpawnX

StealthLogin

Tombstone

Vanish

Whitelist

WorldEdit

WorldGuard

xWarp

I had also been running MonsterHunt but thought it might be the culprit to some major memory issues and while removing it did cut my ram usage in half I am still seeing a steady climb in the ram usage until the server crashes atm. To remedy this I've set the server to restart at earlier intervals but this is more of a workaround than a solution. Keep in mind right now I'm only holding up to roughly 15-20 players and my machine specs (From what I've read and currently understand) should be more than sufficient for two to three times that? Any ideas as to what the problem might be with the memory growth? Do you know of anyways to monitor plugin memory usage? Finally if you don't mind what kind of performance should I be expecting out of a machine with my specs? Thanks in advance to Plague or anyone who decides to contribute answers to these questions.

JD

 

Having the same problem.. wtf is going on? =/

 

I'm not so sure it's a plugin issue. We both get the same error and we only have a few plugins in common. Also I have tried moving down a version on all my plugins and it still has the same issue. As long as the craftbukkit build is 670 I get that error. Also it usually starts happening about 20 minutes after I start the server up and sometimes just lags everyone horribly or crashes or hangs and the process isn't responsive at all not even to be killed!

I have also recently tried disabling monsters, still no go.
Tried fixing all my map chunks, no errors as far as chunkster shows.
I also seem to get my server console spammed with warnings about the server not keeping up or the time changing.

 

So at 2:30pm I made the post above and I restarted my server at that time. It's now 6:40pm Roughly 4 hours and my server has already climbed back up to using 3 gigs of ram again with never more than 12 players on....

** Edit** 2 Mins after this post Server Crash again with EXCEPTION_ACCESS_VIOLATION again...

 

Since I set spawn-animals to false in server.properties it has crashed a LOT less frequently. The lag has also stopped being so frequent! So it looks like to me the problem is related to animals. Maybe it's the new animal: wolf? I'm not sure as I don't know how to disable animal types one by one, but if someone could confirm which animal it would probably help the dev team fix it faster.

Also it doesn't stop crashing just less frequent. There is still a horrible crashing issue.

 

I'm also getting severe lag and crashes since 670 - no idea what the cause is as of yet, though.

 

7:23 Server Start up - 350,000k Memory Usage
7:48 Server check 1 - 1,090,464k Memory Usage
8:58 Server check 2 - 3,618,700k Memory Usage and rising....10 people online peak.

 

I just tried flying around maps and it seems the server doesn't free up memory when the player leaves a chunk. I was flying around maps and when I would leave an area the chunks would unload from my client and when I returned they would have to load again client-side. Whenever the server loaded a chunk the ram usage would go up, and when the player left that chunk the ram would stay the same (if the player went to another place that was already loaded) or go up (if the player went to an area that chunks were not loaded yet server-side). Maybe if the server was set to unload chunks when they weren't in use it would not eat up so much ram?

 

I don't think your far off the mark here. I was just on my server alone and took off running while i was rocking my son to sleep (auto-run is pro ) I ran in a straight line for an hour. When I came back my servers ram usage had gone from 2.7gigs to 4.8gigs. I just restarted and am about to go on a running marathon again and test this theory. I'll let you know the results. Furthermore as I think about this more every time the server has crashed hard enough for a hard boot I've come back to chunk corruption and have had to roll back to the latest back up saved. Getting errors like chunks out of place. I think this might be the path to our problem commencing testing.

​

So after testing here are the results.
This is with me being the only person on the server.
Flying at a height of 105 on a magic carpet and running in a straight line.
OS: Windows 7 Home Premium 64 Bit
Java Version: jre-6u24-windows-x64 (64 bit)
Wrapper: [ADMN] RemoteToolkit [R10 a8.3]
CraftBukkit Version: 670
Plugins:

Code:

AssignPermissions
Backup
BlockCantKeepUpSpam
CommandBook
CraftBukkitUpToDate
DelayedStop
dynmap
HeroBounty
iChat
iConomy
LWC
MagicCarpet
MinecraftRkitPlugin
MyHome
Permissions
SimpleSave
SimpleShop
SpawnX
Vanish
Whitelist
WorldEdit
WorldGuard
xWarp

Test 1 - 30~ Tiles going NorthWest
Start

Ram Usage @ (350MB)​

After 30 Tiles going NW

Ram Usage @ 1,063k (1 gig)​

Test 2 - Return Trip
Start

Ram Usage @1,063k (1 gig)​

After 30~ Tiles going SE

Ram Usage @ 1,090k (1gig negligible change)​

​

Test 1 I ran in a straight line for 30 tiles. Ram usage more than doubled and was getting close to tripling. Obviously chunks are being loaded into memory and not being released. But they are not being reloaded once you pass over them as you can see from Test 2. Unfortunately the first problem is bad enough. This increase in ram usage was caused by one player, put 15 on the server. Furthermore certain plugins are suffering horrendous effects from this lack of memory release as well. Add MonsterHunt to your server and watch how fast your ram usage will sky rocket exponentially based on players.

Update before bedtime. I have been sitting in the same spot since ending Test 2 Roughly 1 Hour ago, and am Still the only person on server. Ram usage has not changed at all. I fully believe now that the problem is chunks being loaded into memory and not being released from memory.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

Agree'd. The only solution besides a new build with a fix is getting all your players to stay in one area. This of course is nearly impossible as most players like to build away from other players but recently I have gotten my players to work with me on a city so we've been sticking to the same area mostly and that way no new chunks are loaded after the city chunks are loaded. We need a fix for this
If chunks would unload when not in active use this problem would cease. I mean animals don't spawn on chunks that don't have any players near it so shouldn't they unload too?

 

Nothing wrong with that if it is within the -Xmx limits...

 

@Plague there is a major problem with that when on build 617 with up to 20 players I never rose above 2gigs of memory usage, and that growth continued until 9:15ish when the server hit 5gigs of ram usage and crashed.

I'm going to see if the chunk memory issue replicates with no plugins now.

Regardless this issue didn't exist in any of the RB I used for 556 to 617 it only appeared upon migrating to the 670 build but I'm going to give the benefit of the doubt until I finish testing just craftbukkit with no plugins.

 

Indeed, this problem is new in 670 and wasn't in the previous recommended builds. I mean I had the same exact setup on 617 and it was fine, but in 670 I have this problem with the server. I have the max RAM usage set to 900MB which worked fine in the last build, but in 670 it uses all of my ram and even starts to eat at my swap space before it crashes. Even worse is most of the time it crashes and hangs, I mean I press control+c to kill it while in the console after it crashes and stops accepting any input, then the console closes but when I open up the process manager (I use "top") java is still shown taking up all the ram and cpu and when I try to kill or terminate the java process it will not even respond to being killed it just stays running for a while after the console has been killed and randomly stops the process between 5 and 30 minutes after trying to kill it.

I have been trying to use Essentials option to free memory on player exit but it has NO EFFECT AT ALL! When a player exits it says in the console that it freed some number of MB however, the amount of ram the server process is using does not go down AT ALL.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

I've given up. Going to try installing Debian to run the server....

 

We're having the same problem. Just sat, teamviewered into the server, while usage climbed. It didnt climb when I got all my players to sit on the same spot near spawn for 30 minutes though. Tried both with and without plugins - it happens either way, so it is either a bukkit issue or a minecraft issue.

 

Climbing is not a problem, the problem is that it crashes afterwards instead of freeing some memory.

 

So any idea if there's plans to fix this issue with not freeing up chunks?

 

Exactly. A rise in ram as exploration occurs is expected as every new chunk allows more animals/mobs to spawn so you should see a rise but the fact that they are never being released is the problem. Even monitoring garbage collection earlier the file just climbed and climbed until it hard faulted.

As of now from what I've "seen" and "heard" this is not something that's really being recognized as the people I've trouble shot it with are looking at every possible avenue besides craftbukkit being the problem. I don't blame them for that. This issue is not a single occurrence but at the same time it's not widespread hence why I'm installing linux to see if this os follows suit.

 

The OS shouldn't matter as it's using a runtime that is OS independant. Also I am using Ubuntu Server which is a distro based off of debian and in many ways very similar. However let us know if you get any changed results. I might try the same thing and run my server on Windows instead of linux just to see if it works any better, however I still believe it won't matter as the code the server is using to run is java and the java code doesn't change because of OS.

BTW @JDD How did running without plugins go? Still the same issue?

 

After further testing:
fresh CB 670 install no plugins: same result
fresh linux install fresh cb 670 install no plugins: same result

After speaking with a few different people over on irc most people are growing at a rate of 5mb per chunk. For me the growth is between 15mb-40mb per chunk. Memory is never being freed up from these chunks either.

 

So then there is no doubt the troubling code is in the bukkit build. Can the developers please look into this?

 

Maybe re-test with 617 to make sure it's the 670 build doing it

 

I might try that in a bit plague.

**Edit Testing now**

Get ready to revert to 617 test 1 complete doing a few more to be thorough then will post my results.

EDIT by Moderator: merged posts, please use the edit button instead of double posting.

 

We reverted to 699 for a bit, and that seems to have a slower (but still noticable) leak.
My hardware guy is going to be real unhappy if I tell him to revert back to 617, as that's what we were on before I told him "hey look, 670 is out" but I'm still interested in your results.

 

If it really is a problem with 670, go to leaky.bukkit.org and look it up/report it.