[MISC] CraftProxy 0.2.0 - Reduce bandwidth use by caching chunk data [1.6]

Discussion in 'Archived: Plugin Releases' started by Raphfrk, May 1, 2011.

  1. Offline

    Raphfrk

    CraftProxy - Reduce bandwidth use by caching chunk data
    Version: 0.2.0

    Description

    This plugin and local client allows chunk data to be cached locally by players. This reduces the bandwidth required for hosting the server. It also helps users who are on slower connections.

    I am not sure what the status of this system is. It doesn't count as a plugin, since it has 2 parts.

    Even when running the plugin, players who don't use the client proxy can still connect. However, they will use the full bandwidth.

    The system can reduce bandwidth by 70-90% (after the 2nd login).

    Setup

    Server

    Add the plugin file to the plugins folder

    Client

    Start minecraft client and login
    Double click on the client jar file
    Enter login details
    Enter the server location/port in the GUI
    Press start on the GUI
    Connect to localhost on the minecraft client

    Stable Builds

    None yet

    Dev Builds

    Warning: These may not be stable

    Client
    Plugin

    Had it get the compress/decompression gain backwards.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
    Jushy, Martin1704, Nathan C and 24 others like this.
  2. Offline

    codename_B

    Whenever you /reload you are reloading the plugins. This includes the server-side proxy plugin.
     
  3. Offline

    Cid

    I got it finally.

    A full on reload of the plugin and cfg did it. I must have really botched something up.

    Thanks for this one.. oh, and btw... is 24% reduction a good number?
     
  4. Offline

    Raphfrk

    Instead of starting it as a plugin, you can start it with

    You can run this on the same computer.

    If you want it on a different computer, then you need to add the server ip.

    It should be around 70%. You should try to disconnect and reconnect immediately.

    However, if it is -24%, then it means that you aren't caching at all.

    There was a bug fix in v45, so you might not have picked that up.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
    MineralMC likes this.
  5. Offline

    NotYetRated

    That data was from my first start up. I will see in a bit how we do with more usage. I will say my users have been using it most of the day now. A few of them anyways. I am awaiting results from them on their usage. Though my useage meter is holding a steady 1 Mbit/s with 14 users on, which is a huge improvement.... I will post some data over time for some trending. So far, the plugin looks amazing!

    As for CPU usage, I have not seen a significant spike or anything out of the ordinary on my server. Should I be seeing some? Does the server also compress and decompress? Or just clients? .... Obviously both if the clients are doing both....
     
  6. Offline

    Raphfrk

    Great, I would be really interested in performance reports.

    Also, if you set the monitor parameter to something it will give a per-user bandwidth report.

    monitor=60

    would report every 60 seconds, which is probably to much.

    Maybe monitor=600

    I would be interested in those too, as it would show what packets are using up the bandwidth.

    It depends on what was causing the lag. If you were flooding your uplink, then the proxy will help with that.

    From looking at task manager, it looks like the proxy roughly doubles CPU usage during login. However, it is multi-threaded, so if you have more than 1 core it should be pretty efficient. Minecraft mostly uses just 1 thread.

    Hopefully, this means that the extra CPU load is not noticeable, the proxy should mostly help people with servers using home broadband who are mostly limited by unlink.

    If you are collecting states, a CPU usage over time graph would be good too.

    The server takes the chunk and compresses it.

    The proxy has to decompress the data, so that it can scan for matches/hashes. Once it has done that it recompresses it.

    So it is chunk data -> compress -> decompress -> proxy -> compress ->

    The sizes are roughly

    Uncompressed: 80kB per chunk
    Compressed: 5kB per chunk
    Cached: 0.8kB per chunk

    On the client side, it has to do the reverse

    -> decompress -> replace from cache -> compress -> decompress ->

    The steps in bold are a waste of time, there is no point in the server compressing if I immediately de-compress. It just uses CPU for no benefit.
     
  7. Raph I need ur help , My friends are not able to log in ! My Server is running on Port 25565 ( i think ) Me , I'm on localhost:25564 ( the server is on the same computer with the client ) So my friends need to set up the CraftProxy , but how . and ( this can be used even if my friends and I are not playing on LAN? )
     
  8. Offline

    Raphfrk

    Well, the server has to be online. You don't need to have the GUI running.

    Other player should just connect with the GUI and all defaults. The only thing they need to change is to add your server's IP into the Remote server box.
     
  9. the GUI is the your .jap application ? The CraftProxyLite ? My friend tried to put minecraftquebec.no-ip.org in the Remote server Box and with the ( default port ) 20000 and he gets an error ( I can't remember :S ) But my Server has to be Online ( Open ) it doesnt matter if my server is in Offline-mod right ?
     
  10. Offline

    dirkson

    What happens when a proxy-less user tries to connect?
     
  11. Offline

    silentdojo

    Thanks raphfrk it took a while to get working but I read every post and found the answer.
    Kept getting cannot connect to backend.
    Ended up changing the plugin settings (on server) to theip:25565 instead of just the port and it worked perfect.
    Gonna test it out more.

    Also Saw a new bukkit build thanking you! Just wondering what it was for and if that build may conflict with this plugin?
     
  12. Offline

    Cid

    Occasionally myself and another tester get an error:

    Code:
    Internal exception: java.lang.NegativeArraySizeException
    The server.log just shows it as a disconnect.EndofStream [INFO] entry.

    The GUI actually says Connection closed by proxy.

    Any ideas?
     
  13. Offline

    XORDYH

    A proxy-less user can just connect using the regular server IP and port number. You can have CraftProxy uers and non-CraftProxy users on the server at the same time, but only the CraftProxy users will get the benefits from the plug-in.

    @Raphfrk:
    I enabled bandwidth usage monitoring by setting monitor=300 (every 5 minutes), but instead CraftProxy is spamming the server log with updates as fast as it can. I'm seeing full reports (meaning the stats for the last 0 seconds and the stats since last login) about 3 times a second. I have quiet=true and debug=false.

    It would also be nice if when you set a log file (as I have) and have quiet=true that the bandwidth reports only went to the CraftProxy log file and not the general server log.
     
  14. Offline

    dirkson

    Ah! Users can still connect normally, excellent.

    Did you mean port 20,000 UDP or TCP? (I opened both, but no reason to leave both open if it only uses one or the other)

    Installed it as a test, but any attempted usage of the system results in extreme lag. No errors by the server, client, or proxy.
     
  15. So Only PLayers connected to your own connection ( I mean LAN ) can you this ? Well Local = LAN?
     
  16. Offline

    Brain

    Discovered something strange:
    I'm running CraftProxy headless with the following command line:
    Code:
     java -jar CraftProxyLite.jar 20000 myserver.tld:25566 quiet staticlocalhost log log_filename.txt
    The directory containing the jar is correctly privileged, user running the program can create directories, files, read, write etc. OS is Debian Squeeze, Java version is 1.6.0_18.
    I have the logfile, the CraftProxyLite.jar and a directory named CPL_cache. However this cache directory remains empty and I don't notice any difference in performance between using CraftProxy and connecting directly. I even tried running it as root with the same result.
    What am I doing wrong?
     
  17. Offline

    MineralMC

    Works a treat, even shaves a couple of seconds off of local connections when loading chunks en massé.
     
  18. Offline

    Sich

    Hello,

    Thanks again for this mod.
    Is there any way to run the client proxy on a headless linux station ?
    Then I could connect with multiple MC client on the LAN.
     
  19. Offline

    MineralMC

    A thought arises - IP bans would be impossible if using this, wouldn't they?
     
  20. Offline

    Raphfrk

    It depends on how you have the ports set up

    By default

    plugin listens on port 20000
    Bukkit server listens on port 25565

    If a client directly connects it will connect to 25565, so will connect as normal.

    If the player uses the GUI, then the GUI/proxy connects to 20000 and caching is used.

    Also, if a player connects directly to 20000, the proxy will just handle it as a normal connection. However, that just puts load on the plugin for no reason, but it should be a small load.


    Sometimes Bukkit seems to bind to 127.0.0.1:25565 and sometimes it binds to <your-ip>:25565. I am not sure why.

    Also, the latest version of the plugin should automatically try all local interfaces.

    You should probably use the recommended builds :), however, unless the protocol changes (which should only happen when Notch updates), the plugin should be unaffected.

    I submitted 3 changes, but only 2 have been accepted so far. They said they liked the third one, but needs testing, so they will probably add that too :).

    The disappearing chunk on teleport bug was really annoying and made teleport plugins less useful.

    This is instead of the plugin right?

    If so, then you would expect the CPL_cache directory to be blank, since the server doesn't need to cache the data. (A previous version cached hashes, but it wasn't a good way of doing it)

    I have updated the OP for "advanced" setup.

    You need to set it as bridge_connection, otherwise it won't be able to authenticate.

    Your GUI will auth with the server-proxy and then the server proxy won't be able to auth with the Bukkit server, since it doesn't know the player's password.

    Bridging means for authentication, the proxy just acts as a pass-through. The Bukkit server does authentication, as normal.

    Yes, or at least they would need to be handled by the proxy.

    There is a banned file that you can set. However, that uses usernames and also requires a restart to update.

    What kind of function would you like? What I could do is check if the username matches or if the IP-address string matches.

    Hmm, in theory, I could detect the ban-ip command (it is just a chat command), and then add the to the file.

    Yeah, see the OP.

    You would need to add an IP to the second parameter.

    Code:
    java -jar CraftProxyLite.jar 20000 <minecraft-server-ip>:25565 bridge_connection quiet static_localhost
    
    You are running it on a separate computer, or just running it as a separate process speeds things up?

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  21. Hmm came here to report an issue, then I saw you already updated a few times, and it seemed you found this issue. However, trying latest download this still occurs:

    Do a /reload, a java thread is spawned that sticks at 100% load.
    Reload again, you get another 100% thread.

    Very problematic right now!
     
  22. Offline

    Raphfrk

    Blast it, I thought I had it fixed. However, a 100% thread is at least an easy test to fix.

    Btw, are you seeing much of a decrease in bandwidth? I guess it is just an a test server now.
     
  23. the GUI is the your .jap application ? The CraftProxyLite ? My friend tried to put minecraftquebec.no-ip.org in the Remote server Box and with the ( default port ) 20000 and he gets an error ( I can't remember :S ) But my Server has to be Online ( Open ) it doesnt matter if my server is in Offline-mod right ?Before that pluggin my friends were loggin with minecraftquebec.no-ip.org ( no port needed ) and now it says Port Not Free or something like that if they try to log ONLY with the Minecraft Client , not the GUI .


    So Only PLayers connected to your own connection ( I mean LAN ) can you this ? Well Local = LAN?
     
  24. Offline

    Brain

    Hmmm... my setup looks like this:

    Dedicated server somewhere running Minecraft/Bukkit <--> CraftProxy plugin <----> Evil Internets <----> Home gateway running CraftProxy standalone and headless <--> My gaming PC running the Minecraft client

    I would expect the CraftProxy running on the home gateway to store data somehow. But it doesn't. The plugin on the server doesn't even create a CPL_Cache directory.
    "Headless" means: no GUI. Not even a graphics card there.
     
  25. Offline

    Raphfrk

    You need to forward port 20000 to the computer which is running the server, in addition to the forwarding port 25565.

    Your Bukkit server handles authentication, so it decides if a connection is allowed.

    You can also run the plugin standalone, if you want. See the OP.

    It should, but you need to add local_cache to make it work.

    The default command lines are given in the OP.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  26. Offline

    MatCat

    MAybe I am blind but where do you set the cache size client side?
     
  27. Offline

    Cid

    Update Details/RePost:
    This is a regular occurance when ever we get into a certain area of the map. We can't be around the town (the part of the world file that has existed since alpha) or it will give us this area and crash. We still get this error occasionally otherwise, but always around the original 'town' chunk.
     
  28. Offline

    Raphfrk

    It can't be with the GUI, I need to add it as an option.

    Thanks for the info. The client says that the exception occurred in the kick message?

    What happens if you go there when connecting to directly to the server without the GUI?
     
  29. Offline

    MineralMC

    Separate computer - the actual server sat downstairs! That couple of seconds is the difference between a direct connection, and a proxy connection. Doesn't make a great deal of sense to me, and I know networking fairly well. Still, happy it works! :)
     
  30. Offline

    codename_B

    Any update on my request? :D

    Is it going to happen?
     
  31. Offline

    Raphfrk

    Well, I guess the second server is handling all the decompress/recompress.

    Maybe. I am still thinking about how to do it efficiently.

    In theory, this was supposed to be a "lite" version of the proxy, but it has more features than the original :).

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016

Share This Page