Is there a way to show java thread memory?

I posted this to bukkit help, but 2 days and no replies, so I am guessing that's the wrong forum, that is probably more for "how do I use bukkit" kind of stuff, and my question is more something developers know, not sysadmins like myself. I hope this doesn't annoy anyone by me cross posting it.

I've had this problem for months now, and I just can't figure it out. Randomly I'll get an error that looks like this:

Code:

2012-01-14 11:09:18 [SEVERE] Exception in thread "Thread-5"
2012-01-14 11:09:18 [SEVERE] java.lang.OutOfMemoryError: unable to create new native thread
2012-01-14 11:09:18 [SEVERE]    at java.lang.Thread.start0(Native Method)
2012-01-14 11:09:18 [SEVERE]    at java.lang.Thread.start(Thread.java:657)
2012-01-14 11:09:18 [SEVERE]    at org.bukkit.craftbukkit.scheduler.CraftWorker.<init>(CraftWorker.java:28)
2012-01-14 11:09:18 [SEVERE]    at org.bukkit.craftbukkit.scheduler.CraftThreadManager.executeTask(CraftThreadManager.java:14)
2012-01-14 11:09:18 [SEVERE]    at org.bukkit.craftbukkit.scheduler.CraftScheduler.processTask(CraftScheduler.java:107)
2012-01-14 11:09:18 [SEVERE]    at org.bukkit.craftbukkit.scheduler.CraftScheduler.run(CraftScheduler.java:64)
2012-01-14 11:09:18 [SEVERE]    at java.lang.Thread.run(Thread.java:679)

And it will be a different thread everytime, sometimes it stops logblock from logging, or if its the listen thread, nobody new can log in.

Code:

2012-01-12 22:39:22 [SEVERE] Exception in thread "Listen thread"
2012-01-12 22:39:22 [SEVERE] java.lang.OutOfMemoryError: unable to create new native thread
2012-01-12 22:39:22 [SEVERE]    at java.lang.Thread.start0(Native Method)
2012-01-12 22:39:22 [SEVERE]    at java.lang.Thread.start(Thread.java:657)
2012-01-12 22:39:22 [SEVERE]    at net.minecraft.server.NetworkManager.<init>(NetworkManager.java:68)
2012-01-12 22:39:22 [SEVERE]    at net.minecraft.server.NetLoginHandler.<init>(NetLoginHandler.java:27)
2012-01-12 22:39:22 [SEVERE]    at net.minecraft.server.NetworkAcceptThread.run(SourceFile:49)

What I'm asking for help with is, how do I debug this? I tried using jdb and attaching to it, and I can list the threads, but I can't see how much memory each is using. There is a "memory" command but it says "command is no longer supported". The heap space isnt being used up, I give it -Xmx8192M and when this occurs its less than half full so its some other kind of memory. Any way to debug this directly?
I suspect that something is using up some resource, and the fact that random threads die is just a symptom, but how to find the cause?
I already tried removing all plugins, but some I can't remove some like worldguard or the world will burn down. It doesnt run out of memory until a random amount of time passes, sometimes an hour after starting, sometimes almost a full day can pass without it occurring. Its hard to test without putting it on one of my production servers.
I tried lots of debugging options, like kill -3 on the server, and an alternate debugger, I just can't figure out what I should be looking at to figure this out.

 

This might be helpful: http://visualvm.java.net/

(Credit to wwsean08 for posting that link originally)

 

Had this problem myself. Are you on Ubuntu 11.10?

 

Actually yes.

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 11.10
Release: 11.10
Codename: oneiric

I still haven't traced this down precisely, but it looks like its not related to bukkit or java. Its if a background process allocates too much memory, even for an instant. Some process running as root went out of control, and I was able to get it to do those "java.lang.OutOfMemoryError: unable to create new native thread" errors even when bukkit started up.

I was able to sort of throw resources at the problem to mitigate it, by putting "ulimit -s 16384" in the start script, and putting this in limits.conf

minecraft hard stack stack 16384
minecraft soft stack stack 16384

Then "ulimit -a" to verify it is being raised.
Then finally I added -Xss2048k to the start options for minecraft.

So after making all the changes, I havent gotten the OutOfMemoryError, and that was Jan 22.
I still don't know what the heck I'm doing, so I'd love to find a cleaner fix. Like is there a way to just ensure java allocates whatever memory it needs, without some background process able to allocate it. This is different than normal RAM usage, its not like the server was even swapping, swap was at zero, so this is some other resource that its running out of.
I dont understand "stack size" to be honest, but it looked to be related.

 

Had this problem all through Ubuntu 11.10. Finally switched to Debian. Worked fine.

 

MonsieurApple has found the solution to this problem of Ubuntu 11.10 and posted details in another thread.

 

He is completely right about it forking uncontrollably on Ubuntu 11, but I found that if you have anything else running on the server that eats up all the memory, bukkit will not have any memory for threads anymore and stuff will start dying. It can be blamed on java I'm sure (I mean why the hell have all of that -Xmx8192M stuff to reserve ram but still run out?!?), but all I know is to make damn sure nothing else on the server eats up the ram!