[MISC] CraftProxy 0.2.0 - Reduce bandwidth use by caching chunk data [1.6]

Discussion in 'Archived: Plugin Releases' started by Raphfrk, May 1, 2011.

  1. Offline

    Raphfrk

    CraftProxy - Reduce bandwidth use by caching chunk data
    Version: 0.2.0

    Description

    This plugin and local client allows chunk data to be cached locally by players. This reduces the bandwidth required for hosting the server. It also helps users who are on slower connections.

    I am not sure what the status of this system is. It doesn't count as a plugin, since it has 2 parts.

    Even when running the plugin, players who don't use the client proxy can still connect. However, they will use the full bandwidth.

    The system can reduce bandwidth by 70-90% (after the 2nd login).

    Setup

    Server

    Add the plugin file to the plugins folder

    Client

    Start minecraft client and login
    Double click on the client jar file
    Enter login details
    Enter the server location/port in the GUI
    Press start on the GUI
    Connect to localhost on the minecraft client

    Stable Builds

    None yet

    Dev Builds

    Warning: These may not be stable

    Client
    Plugin

    Had it get the compress/decompression gain backwards.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
    Jushy, Martin1704, Nathan C and 24 others like this.
  2. Offline

    Raphfrk

    Are you sure you are connecting to the right ports.

    The assumed ports are

    Server Side
    Bukkit Server: 25565 (set in server.properties)
    Plugin proxy: 20001 (set in /plugins/Craft Proxy Lite/craftproxy.txt

    Client side
    Local proxy port: 20000
    Default server port: 20001 (connects to plugin (not the server))

    Are you connecting to port 20000 (or whatever you set the plugin to listen to)?

    When the local proxy connects to the plugin, it uses a much higher client version than normal.

    If you use it to connect to the server directly, the server will think that it is being connected to by a client with version 111 of the protocol and will kick the player.
     
  3. Well on page 1 you say to TCP forward port 20000 on the server to 25565. It didn't make sense to me in the first place...the local proxy connects to 20000 which is the plugin, so why would iptables redirect be needed...

    So I removed the redirect, now I get the socket error someone had above ^

    I already applied your suggestion, then I got:

    11:56:09 [INFO] [11:56:09 AM] 81.207.132.187/59591 (): Local cache detected for client
    11:56:09 [INFO] [11:56:09 AM] 81.207.132.187/59591 (): Connecting to: 127.0.0.1:25565
    11:56:09 [INFO] [11:56:09 AM] 81.207.132.187/59591 (): Unable to open socket to 127.0.0.1:25565
    11:56:09 [INFO] [11:56:09 AM] 81.207.132.187/59591 (): Unable to open connection to backend server

    Then I thought, it likely means the MC server only allows connecting on the outside IP by default, so I changed that, and that fixed that issue.

    Which leads to the current issues:

    2011-05-03 11:58:19 [INFO] Disconnecting Joymo [/:34599]: Failed to verify username!
     
  4. Offline

    Zenithas

    ... Very nice, though I'm also interested in seeing an offline mode for this.

    My connection isn't all that great, so sometimes us in Oz (local, but all on a shared server) can't connect to minecraft.net, giving us no minecraft connection when in online mode. I love the idea of the cache, you should really talk to Notch about letting the basic client do this.

    +1 diamonds, though honestly wish I could give more.
     
  5. Offline

    MatCat

    Ok I put the proper IP in the setting, now I get the cannot verify username error like everyone else.
     
  6. Offline

    Raphfrk

    What did you change to fix that? Is it a Linux/OS thing?

    I must have messed up the authentication code somehow. I assume if you put the Bukkit server in online-mode, it would connect?

    It is supposed to use a "bridging login" which allows the login to bypass the proxy.

    Probably, when I tested it, I had my server in online mode (since it needs to be in online mode otherwise).

    Btw, on a priority thing. Do most people have an excess of download bandwidth speed? I could probably improve things a bit by sending more info from the local proxy to the server.
     
  7. The localhost should just be replaced by hostname or external IP.

    I have online-mode=true already, so, that can't be the problem...

    Not sure about priority? Download speed is generally quite a bit higher than upload, do you mean excess upload speed?
     
  8. Offline

    GmK

    Fantastic! Will give it a spin later as an option for our players.

    I am excited to see how it handles our high-traffic servers with many people in one region at once, doing many changes.

    My guess would be that the caching doesnt really have an effect in such a case?
     
  9. Offline

    Raphfrk

    I mean why you can't connect to localhost, is it an OS setting?

    I mean as a quick test, see if it works with online-mode=false.

    I mean should I try to keep both low. There is a trade off between memory/CPU/bandwidth.

    In theory, I could eliminate the need for the server to use disk space by having the player send all hashes it knows about at startup. However, that would mean the server would end up using more download bandwidth.

    I think people who run their server on a home machine would prefer if the plugin keeps upload bandwidth low, even if it means a little more download bandwidth usage.

    I think it should still help a lot. It splits each chunk into 8 block high sections and handles each separately.

    If a player changes 1 block in a chunk, only one section would need to be resent.
     
  10. Offline

    MatCat

    I am a bit confused as to how to get this to work... I too get the cannot verify username problem...
     
  11. Offline

    Raphfrk

    They are saved as "sections", but yeah. The local client saves the files locally.

    Right, the proxy would just have to send "This chunk is made up of the following 16 sections" instead of the entire chunk.

    Right, that is how I tested it in fact. I teleported a few times.

    It will be an improvement, but the LAN should send the info pretty quick anyway.

    Also, minecraft always sends chunk data at a certain rate, so distant chunks would still take time to be sent.

    As for the username thing, I must have tested it in offline mode by accident.

    Yeah looks like I messed up initial testing.

    What happens if you run the Bukkit server with online-mode=false ?
     
    MCbyWay likes this.
  12. I can connect to localhost fine, OS wise, it's the MC server that does not accept connections on it. It seems to bind to external address only, by default.

    I can't test offline mode false right now, but I'll believe that works. Will test again when it works on true, as I can't enable that false on live server.

    For my case, I'd want to use as much server b/w as possible, drowning in bandwidth server-side :p Perhaps should be configurable ;)
     
  13. Offline

    thepackett

    so one question before i download, this makes it so you need less upload and download speeds per player, right? (forgive me if i'm wrong, my knowledge of bandwidth and the internet is very limited, please correct me if i'm wrong)
     
  14. Offline

    MatCat

    I for one am not going to turn auth off on my live server to test... Too many people, not a good idea :).
     
  15. Offline

    Faust

    I still haven't been able to succesfully login with online mode set to true.. because of the failed to verify.. but for now i run it in offline mode and it works great
     
  16. Offline

    Raphfrk

    Ahh, OK, so the cache is just to help clients?

    I was getting about 80% bandwidth reduction (the first connect normally is +25% bandwidth due to setup). Is that what you are seeing too?

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  17. Offline

    Faust

    I don't know about the actual bandwidth reduction, but the cache does help a lot.. Less strain on the server bandwidth and much faster chunk loading for clients..
     
  18. Offline

    MatCat

    When will you get this fixed to work with auth on?
     
  19. Offline

    Raphfrk

    So, users tried to log in a few times and noticed a speed up on the second connect :)?

    It says how much of a reduction on the client proxy GUI.

    Do you have a CPU usage over time monitor running for the server?

    Probably this evening, it is likely just a small setup error by me.

    Also, I can change it so that local connections don't need to be authed when connecting to the local proxy.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  20. Offline

    Faust

    Cpu usage over time.. I don't have but I watched it in general and it wasn't high at all.. max peak was at 6% on a quad core.. Second connection was a bit faster than the first, but not that much, the connection time is generally fast.. I believe the gui showed around 80%+ reduction

    On another note the Client GUI could the local server default port be the standard minecraft server port? That way you won't have to write the localhost port and just do with localhost
     
  21. Yes, that would be the only purpose for us :p

    Also, I presume it doesn't add load to MC's main thread? As that shouldn't be stressed more than it is ofc :p
     
  22. Offline

    Raphfrk

    That is what I see too.

    Yeah, obviously, that is what I should have done :). To many ports and I wasn't thinking clearly :)
     
  23. Offline

    ZachBora


    You misunderstood me, this is my CLIENT startup command. This is what I have to use to not crash, ever. It's a memory issue with nvidia cards when the client reaches 1gb ram usage. It's all detailed in the mojang ticket but you don't need to worry about that.

    All I want to know is how to NOT get my memory issue with your client proxy. Do the client proxy implement minecraft or does it open a new java process? I would assume it opens a new java process, or else there will be major memory issue for me. Don't misunderstand, I have 4gb of ram but 32bit java with nvidia card crashes at 1gb, at least for me and a thousand other users.

    If I executed this command would it do the same as my line above + the proxy or would that change nothing?

    "C:\Program Files\Java\jre6\bin\java.exe" -Xmx512M -Xms512M -cp "D:\Minecraft\CraftProxyLite.jar"
     
  24. Offline

    Raphfrk

    It is a completely separate thread. In fact, if you want you can actually run the proxy as a stand alone process.

    I only made it a plugin for ease of installation.

    You start it with

    java -jar CraftProxyLite.jar <listen-port> <Bukkit-server port> quiet staticlocalhost log log_filename.txt

    You can also add IP/hostname instead of just the port.

    It must be run on the same server as the Bukkit server. The standard server has flood protection. However, it doesn't apply to connections from localhost.

    I think you need:
    "C:\Program Files\Java\jre6\bin\java.exe" -Xmx512M -Xms512M -jar "D:\Minecraft\CraftProxyLite.jar"

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  25. Tested quickly with online set to false, works!

    I haven't been able to test long enough to know if there's a performance improvement though, I guess the main improvement comes if most of the chunks are cached.
     
  26. Offline

    Ithato

    Is the server-side port number configurable? That would be a must-have feature for me, as I use a paid host and would need to run a much higher port number on the server-side proxy. Other than that, it sounds like a fan-damn-tastic idea, especially for very long-distance minecart travel.
     
  27. Offline

    Raphfrk

    It will increase CPU usage as it has to decompress/examine/recompress the data stream.

    The performance increase would be that there should be fewer bandwidth spikes.

    Minecraft limits the rate at which chunk updates are sent out. Chunk updates should only be sent once every half a second.

    Yeah, there is a config file which lets you set both port numbers.

    /plugins/craft proxy lite/

    However, you would still need 2 actual port numbers internally. Do you have a virtual server?

    What you want to do is move the Bukkit server away from 25565. This is the server-port field in server.properties.

    server-port=23456

    You can then edit the craft proxy lite config file:

    Make the plugin listen on 25565

    listen_hostname_port=25565

    Make the plugin connect to 23456

    default_server=23456

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  28. Offline

    wassilij

    Will this avoid lag on Players PC too? Because they have cached the Chunks and dont need to ask the Server, or?
     
  29. Offline

    thepackett

    This would make it so that you could have more players even with less upload speed, right?
    (I went to http://canihostaminecraftserver.com/ and entered in my computers specs, and after experimenting i found out that my problem was that i didn't have enough upload speed even though i have 2.5 MBps it said i could only have about 9 people on it at once. (I have way more than enough download speed for up to about 64 people))
    If that's how it works then how much would it reduce the usage of upload speed?
     
  30. Offline

    Raphfrk

    If the player's internet link speed is limiting the download speed, then it should help.

    I would say that it reduces it by 3 on average. If players teleport a lot, then it reduces it by around 5.

    The next biggest bandwidth usage is entity update data.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  31. Offline

    Cid

    OK, forgive my ignorance. I have read these posts over and over and I feel like the plugin is configured properly but I have no way to verify that. I get no connection error and can get to the server. I was looking for a way to verify that it was doing something. I have no bandwidth meters avail right now as it is a hosted server. I did notice a folder in the directory I store the client side.jar file called CPL_cache. Problem is there is nothing in that folder. Should there be?

    I am running the stock config right now with no changes as we use default ports.

    If the port wasn't open on the providers end would I even be able to connect? Also, how do I know this is hooking in on the client side, is there a special directory the .jar file needs to reside in on the client side?

    Thanks so much, this plugin will be fantastic once its up and running.
     

Share This Page