[EXPERIENCE/GUIDELINES] 6 months of bukkit on 7y old laptop.

Discussion in 'Bukkit Help' started by uroskn, Dec 7, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    uroskn

    Hey. It's been six months, since i've started up a small Minecraft server, running from my home. This server is (was) running on (by community standards) ancient harware. I would like to share my experience in optimizing bukkit for really old computers.

    THIS IS NOT A TUTORIAL. I AM SHARING MY *PERSONAL* EXPERIENCE I HAD IN RUNNING BUKKIT ON CRAPPY HARDWARE IN HOPES THAT SOMEONE WILL TAKE SOMETHING USEFULL OUT OF IT.

    THINK OF THIS AS MORE OF MY PERSONAL GUIDELINES THAT I'VE DECIDED TO PUBLISH. I DO NOT TAKE ANY RESPONSIBILTY FOR ANYTHING YOU DO WITH IT.​
    This could probably be also applied to budget VPS accounts. Haven't got chance to test it, tough.​
    Any comments (and even critique) are more than welcome.​
    This assumes that you're running bukkit on Linux machine with root access.​

    Server specifications (HP Compaq nx6125):
    AMD Turion mobile ML-32 (1.8GHz)*
    625 MB RAM (DDR),
    60GB 5400 RPM IDE disk.

    In essence, i'm running this on an really old laptop.

    * For comparison: 3 years old Atom processor (N270) is 300% (factor 3) faster than this! And even this is a shitty processor!

    OS'n'stuff:
    Arch Linux, 32 bit, custom kernel

    ::kbox /opt/craftbukkit# java -version
    java version "1.7.0_09"
    OpenJDK Runtime Environment (IcedTea7 2.3.3) (ArchLinux build 7.u9_2.3.3-1-i686)
    OpenJDK Client VM (build 23.2-b09, mixed mode)

    Bukkit?
    Been running every (mostly recommended builds) bukkit release from 1.2.5-R1.0 up to 1.4.5-R0.2 (current).

    Plugins (30): PortableCrafting, WorldBorder, Orebfuscator, BlockLog, WorldEdit, NoCheatPlus, NoPl, EntityCleaner, ChatCo, PermissionsBukkit, VanishNoPacket, MCTelnet, Vault, PluginReloader, motd, qSpawner, ModReq, RAM, iConomy, CoreProtect, xAuth, Duties, UltraLogger, eTeleport, Timber, OpenInv, KarmicJail, Watcher, HyperConomy, Residence

    It could handle ~10-15 players online at the same time, afther that processor started to crumble (not ram!) - what a suprise, eh?

    This server was also running MySQL database, had regular backup every 15 minutes, also running apache and had a bunch of custom scripts written that ran alongside of bukkit. All on the same box.

    Optimizing for processor & memory, lessons learned working with bukkit'n'stuff:
    - WATCH WHAT PLUGINS ARE YOU INSTALLING! Before installing, setup a test server, set plugin-profiling to true and just mess around a bit. Watch what your plugin is doing, and if it's CPU hungry, think twice before installing. Also think twice when you're duplicating content, you're probably duplicating your computing resources aswell. For instance, if you already have Residence, why also install WorldGuard? Profile, profile, profile and profile again. Then roll in production. Especially any plugin that hooks in OnPlayerMove() and similar. These things can eat your CPU cycles like crazy!

    - Tune MySQL! This only applies if you hold larger databases, like me. Make sure that (especially) INSERT queries get executed as fast as possible. Some plugins halt bukkit main thread untill MySQL says OK. This can even cause some lag!

    InnoDB is great for many-inserts, few selects (usual operation). However it is also memory hungry, thus it is a good thing to optimize it in my.ini (setting it's values to lowest acceptable values). If possible, use it, and don't forget to enable Direct I/O. Bukkit will be gratefull. ;-)

    If takes a lot of time to get data out of MySQL, take a look what queries are hitting your database and set indexes. You will never need to purge data ever again (unless you run out of disk space)! :-D

    - Stay the fuck out of the GUI-land. And don't even think about windows here!

    - Get some server monitoring software. Right now! Munin, for example. Or anything, just so you have 24/7 view over what is going on with your box. This will help you notice a lot of troubles, before they're noticed by your players.

    - max-viewing-distance in server.properties can save more memory and cycles than any other optimization you will ever do! I've observed that setting it down to 6 is still unnoticable for normal gameplay (it's not so appealing to the eyes anymore, but it's still completly playable). I would recommend with starting minimum acceptable and then gradually increasing it. Keep in mind however that rise in memory usage with every increased viewing distance is exponential, not linear.

    - Linux gave you a lot of managment tools. Use them. Prioritize stuff that runs on your box! Java first (especially when it comes to I/O), then MySQL, only then everything else. Look at cgroups - they are good to ensure that some tasks however do get some slice of processor (backup scripts for example). When possible, for anything non-bukkit related, enable direct I/O. Using this and your machine can do a lot of stuff that previously just couldn't do! I for example run minecraft overviewer, buch of statistics on MySQL database, some graphing jobs on the same machine and even backups, all while server is running, with minimal performance impact.

    - KEEP IT SIMPLE, STUPID! Do not optimise something that will give you only marginal performance advantage! It will do more harm than good in the long run. Profile your application, see where are your biggsest bottlenecks and optimise there. So, no actually you DONT need to pass over 9000 arguments to java!

    - EDUCATE YOURSELF. Read online about java, about java's garbage collectors, about what OS does with ram, what java does with RAM, etc... It may take some time, but it will be worthwille in the end (or you can just throw better hardware at it and leave this thread right now - that works too).


    Garbage collection, garbage collection, garbage collection, java tuning!
    Beacause I was running on such slow disk, and such low ammount of RAM it was essential to keep java's memory footprint as low as possbile. I also could not do some heavy GC, beacause I was also heavily limited by my processing power.

    Note, that a lot of time bukkit spends running is actually running the garbage collection. So, buch of -XX:CompileThisShit -XX:YetAnotherFlagThatSeemsCool are actually useless. What does it help you when you get your java bytecode compiled into native faster (and thus less optimized), if you spend most of your CPU cycles doing GC? ;-)

    UNDER NO CONDITION LET JAVA TO BE SWAPPED OUT!

    - Java memory and it's relation to OS.
    Resident set (i think it has similar name in windows - not sure tough) or RSS is amount of ram that any given procecess occupy (let's call it that for simplicity reasons). So, if RSS for java is 400MB this means that java is using 400MB of your physical ram (more can be swapped out!). But, does that mean that bukkit itself is using 400MB ? Hell no. When java GC releases some memory, this memory is not returned to OS. It is kept by java for any subsequent allocations (to improve performance - requesting and releasing memory from/to OS is expensive task). So, out of those 400MB, bukkit may acutually be occupying anywhere from 50-70% of that RAM. Rest is just empty or garbage that is used by java if more memory is needed (meaning that if java occupies 2GB of memory, there is anywhere between 400-1000MB of RAM that is seen as "free" by java). Note, that "free" ram that java has is seen by OS as a used ram!

    This can sometimes be undesired effect, we would rather keep ammout of "free" ram that java has to a minimum, to let OS use remaining RAM for something else (like page cache - portition of unused RAM used by OS to speed-up disk access). This can lead to increased I/O performance and can be in extreme cases equally fast as running things from ramdisk (and more reliable)! Ammount of free ram that java will keep for allocations, and how much can be returned can be set using -XX:MinHeapFreeRatio and -XX:MaxHeapFreeRatio. If your goal is to keep memory consuption of java as low as possible, beacause you're struggling with slow I/O, try setting these two flags to really small values (defaults are 50 and 70 respectivly). Requesting and releasing memory to OS is still million times faster than one disk seek that bukkit has to wait for!

    - What about -XX:+DisableExplicitGC?
    Erm, no. Even altough it does save huge chunk of processor usage, it causes much more "spikes" in java's memory usage (especially in conjuction with Min/MaxHeapFreeRatio). This in turn can easily rape you page cache and do more harm than good.

    - Stock garbage collector? Incremental? CMS?
    They sucks. Period.

    - Garbage-First collector (G1).
    Now, we're talking! I've had best experience with this garbage collector. It has the smallest pauses in java, plus it can actually handle memory under extreme constraints (like my configuration). This collector will split most of the java's memory into regions and clean and compact those regions that has most garbage in it first. This means that it does not clean up all garbage at once (thus reducing pauses that bukkit has to suffer) - perfect for our workload. It can also do mark phases while application is running.

    To, show how well it behaves, here are two graphs:
    [​IMG]
    [​IMG]
    (i've used munin to generate graphs, RAM and MCTelnet plugins for extracting data from bukkit and a bunch of PHP scripts).

    As you can see, memory usage is (almost) flat line hovering between 200-300MB of used RAM. Regardless of players. With almost no impact on processor. The similar results are seen even on peaks when all 15 slots are occupied. These results i've observed only with G1 collector, mixed wih Min/MaxHeapFreeRatio. All other collectors or different configuration had either raped my RAM, processor or both.

    So, final configuration?
    Startup script (in 6 months, i havent yet seen OutOfMemoryException being thrown!):
    Code:
    nice -n -5 ionice -c 1 java -Xmx400M -Xms32M -XX:MinHeapFreeRatio=1 -XX:MaxHeapFreeRatio=2 -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=0 -XX:MaxGCPauseMillis=100 -jar craftbukkit.jar nogui
    -XX:MinHeapFreeRatio=1, -XX:MaxHeapFreeRatio=2
    This instructs java to return as much memory as possbile back to OS. This memory is in turn used for page-cache, speeding up disk access. If you have shitton of memory to spare, you can skip this.

    -XX:+UseG1GC
    Instruct JVM to use G1 Collector.

    -XX:InitiatingHeapOccupancyPercent=0
    How soon should G1 start marking memory? I've set this to do this all the time, so when time comes to do collection it already has pretty good picture, which regions should be cleaned. I've set this to 0 due to my crappy processor, you can probably ommit it tough.

    -XX:MaxGCPauseMillis=100
    I think this one is self-explanatory.

    Of course, i've also set this in server.propreties:
    Code:
    view-distance=6
    Oh, i've disabled warn-on-overload. This just means that you've reached your server's (recommended) limit. This doesn't mean you can't sucsefully push it even futher. ;-)

    Conclusions
    You do not need state-of-the-art machine to run bukkit server for smaller community. Many people, who has some old computers at home or even throws them away, can actually run quite sucsessfull mid-sized community servers on it. Just there is a little bit of work with tuning everything. If I had better processor, i could probably with these 600MB of ram ran a 15-30 (+/- 5) slot server. :)

    I've just got Pentium G630 and 2GB of RAM. Let's see how many players we can get it on. :)

    Next time: Running sucessfull bukkit server on your Android phone. Now, off to bed, its 3:30 AM here!
     
  2. Offline

    UltraMC

    My hero.
     
    MrBluebear3 likes this.
  3. Offline

    computerxpds

    Omg.. this is what I do everyday even on my prod that has really awesome specs, the less you slam the cpu the less lag you are going to have. You sir are awesome.
     
    UltraMC likes this.
  4. Offline

    UltraMC

    Now we are talking server administration :)
     
  5. Offline

    Ugpzt

    Wow..thanks for sharing.:) I will try this on my son's AWS' free server.

    I am new to linux and server-related things, please bear with me for stupid questions. How do I handle "nice"? I presume that I had to chmod first?
     
  6. Doesn't apply to me directly as I do not know anything about Linux, but just wanted to say it was cool to look over. I'm in the same mind set as you, a write up like that will be very useful to many I'm sure.

    DD
     
  7. Offline

    falkensmaze

    Don't chmod anything! It's a bad idea to have setuid on shell scripts or java itself. You can use "renice" as root (`renice -5 -p <pid>') to set the nice value of a process after it has been started. nice values go from -20 to 20, where -20 is a higher-priority process and 20 is a lower priority process. Setting all the way to -20 is probably a bad idea, since there are system processes more critical than CraftBukkit, ones that CraftBukkit may be depending on. As a rule, only root can decrease (or make more negative) the nice value of a forked process, so I'm wondering whether OP is starting java as root, which I wouldn't recommend for security reasons.

    Automating this on startup in a secure way is beyond the scope of this comment. Ambitious readers are encouraged to pester me, however.

    The notes about garbage collectors and Java command-line options are likely to be applicable on any operating system.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 30, 2016
  8. Offline

    uroskn

    I am starting server as root, yes, however bukkit itself does not run with full privileges.

    I have installed bukkit under special limited user (i have mine called "bukkit" - wow!), and stating it up using simple shell script that you run as root. That script first sets up all priorities (as a rule of thumb: if you increase priority beyond normal root is required), then switches to limited user and runs java.

    This is snippet from my startup script:
    Code:
    #!/bin/bash
    nice -n -5 ionice -c 1 su -s /bin/bash -c "cd /srv/mcserv/craftbukkit; java \
      -Xmx768M -Xms32M -XX:MinHeapFreeRatio=1 -XX:MaxHeapFreeRatio=2 \
      -XX:+UseG1GC -XX:InitiatingHeapOccupancyPercent=0 -XX:MaxGCPauseMillis=100 \
      -jar craftbukkit.jar nogui" bukkit
    Problem is also that renice -n -5 -p pid will renice only it's main thread (which spends most of it's time idle), i prefer to set it up at start and then leave it alone. :)

    Falkensmaze wrote a good reply about nice values. Simplest is to start your server first (i'm hoping you're not running it as root), then change it's nice value using:
    Code:
    sudo renice -n <desirednicevalue> -g $(ps -Aj | grep java | awk '{ print $2}');
    I wouldn't recommend setting more than -10. Even that is a bit too much imo.
     
    falkensmaze likes this.
  9. Offline

    Ugpzt

    Thank you, I figured as much that I had to use sudo. I will test it out on my Ubuntu first.
     
  10. Offline

    hackal

    I am very new to how java works. I run my own server on a linux. Only thing I have problem with is that my ram will get full very fast. I start server with
    Code:
    java -Xincgc -Xmx3G -jar craftbukkit.jar
    When I use top command java is using around 87% of memory and only goes up until server crash. I prevent crashing by restarting server at 0:30 every night. Is there any way how can I "unload" or something from ram?
     
  11. Offline

    falkensmaze

    This is a good topic for a new thread, hackal.
     
  12. Offline

    hackal

    Ok, I did new thread
     
  13. Offline

    Hoolean

    jacklin213 likes this.
Thread Status:
Not open for further replies.

Share This Page