[MISC] CraftProxy 0.2.0 - Reduce bandwidth use by caching chunk data [1.6]

Discussion in 'Archived: Plugin Releases' started by Raphfrk, May 1, 2011.

  1. Offline

    Raphfrk

    CraftProxy - Reduce bandwidth use by caching chunk data
    Version: 0.2.0

    Description

    This plugin and local client allows chunk data to be cached locally by players. This reduces the bandwidth required for hosting the server. It also helps users who are on slower connections.

    I am not sure what the status of this system is. It doesn't count as a plugin, since it has 2 parts.

    Even when running the plugin, players who don't use the client proxy can still connect. However, they will use the full bandwidth.

    The system can reduce bandwidth by 70-90% (after the 2nd login).

    Setup

    Server

    Add the plugin file to the plugins folder

    Client

    Start minecraft client and login
    Double click on the client jar file
    Enter login details
    Enter the server location/port in the GUI
    Press start on the GUI
    Connect to localhost on the minecraft client

    Stable Builds

    None yet

    Dev Builds

    Warning: These may not be stable

    Client
    Plugin

    Had it get the compress/decompression gain backwards.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
    Jushy, Martin1704, Nathan C and 24 others like this.
  2. Offline

    RandomMiner

    I guess this is for the serverside proxy not the GUI right? (I tried on the GUI side by setting bufferlatency=0 in the config file for the GUI and I still have a slight delay).

    I also have some friends who connect to the server by using the internet, wouldn't a bufferlatency of 0 have a negative impact?

    I test the lag by jumping down slopes by the way, if I connect without any port given the "ouch" is coming at the same time as the sound you make when you step on cobblestone (I do that a few times to make sure it's not some freak accident).

    When connection with a port given (keep in mind I've not tried setting bufferlatency to 0 for the "server" version of craftproxy) the ouch comes notably after the step sound.

    I'll try the buffer 0 setting and report the result

    Edit: The question of Kizzard is pretty much what I'm interested in, too!
     
  3. Offline

    Raphfrk

    It would slightly increase network bandwidth since every small packet needs a header. It might also increase CPU usage since the proxy has to track more packet fragments. I doubt it is a massive deal either way.

    The proxy inherently combines (minecraft) packets into (internet) packets, so even with zero latency, it will always send each packet as a single unit. This may get you most of the benefit anyway.

    The issue is that the server can potentially send each little piece of information as a packet.
     
  4. Offline

    RandomMiner

    The lag seems to persist even with bufferlatency set to zero for both the server craftproxy and the gui. I guess it comes down to the fact that craftproxy has to handle the packets before they actually get passed to the server/ minecraft.

    It's not bad anyway, having my friends use this and me simply connecting to my own system without using craftproxy will still be more than sufficient to increase networking performance. If it can be fixed, great, if not, such is life.
     
  5. Offline

    Raphfrk

    I added the option to set the latency on the GUI. Were you running the local proxy from the command line?
     
  6. Offline

    Simanova

    1) Current size in client GUI shows "unknown" - is this a known bug?

    2) I tried to add "compression_level=9" into command line startup - but it wont work ( Craftproxy is not starting )- any idea? perhaps without "="?
     
  7. Offline

    RandomMiner

    Yes, java -jar CraftProxyLite.jar 20000 25565 auth_off bufferlatency 0 (tried with and without the setting). I didn't really notice it while playing so it's not really significant amount of lag (I guess around 50-100 ms).

    However the fact that I saved about 9 MB (well theoretically at least) of Upload over the course of 14 Minutes is astonishing ( especially if one takes into account my upload speed - it would take approx 7 Minutes to upload these 9 MB for me). And I was the only one on the server (well it isn't a public 24/7 server) so I guess people with several dozens of players might find this very useful.

    I do have to excuse myself thought, its late night over here so I can do further tests later tomorrow after I come home from work.
     
  8. Offline

    rtcabooservb

    Any idea how to stop the spam of "Delay Through fairness manager"?
     
  9. Offline

    Kizzard

    Oh dear, so in my first test with 2 players in the server, not going well.

    'Delay through fairness manager: 38406'

    We are getting 40s (not ms) lag! It's insane. I am not sure of the cause. I am using 1GB local cache and left the buffer delay at default.

    Could it be because our connections differ a lot? The main reason I was trying this proxy out was to help my friend who has a sucky connection to the server. (for some routing reason we cannot work out). Is it trying to balance the number of packets sent to us? If so, it's gonna have a hard time as his computer and connection is rather slow.
     
  10. Offline

    Raphfrk

    Sorry should be fixed by the latest version. It was debug code for trying to track down where the latency was happening.
     
  11. Offline

    rtcabooservb

    ETA? :p

    Also, is there any real benefits by setting the buffer to 0 and compression ratio 1 vs 9?
     
  12. Offline

    Raphfrk

    Hmm.

    Btw, the cache is entirely client side. It doesn't matter what limit you set for the server.

    It should only balance packets of each type. The idea is to limit chunk updates so that they don't account for more than 50% of the traffic.

    Can you try with 0 latency on both server and client and see if that helps.
     
  13. Offline

    rtcabooservb

    I'll keep craftproxy disabled until the next update. Getting around 40-100ms of chat lag. 0ms of chat lag with it off. Keep up the good work though.
     
  14. Offline

    Raphfrk

    Now

    Buffer at 9 will improve compression from 70% to 80% and buffer = 0 gives better latency.

    Is that with the latest version and the latency set to 0?

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  15. Offline

    Simanova

    CraftProxy (v78) is now working really good - with 1 person online, using this tool
    If 2 ppl are online, after 3 sec no data is transfered between clients and server anymore.
    It seems CraftProxy is dead.

    20 sec. later:
    "Read timed out" / "End of Stream", all players are disconnected

    second login after crash:
    > "Login in" takes about 3 to 7 sec
    > "Downloading terrain" takes about 12 sec via LAN and 20 sec via Internet
    > Every player is falling through the map
    > Some players are disconnected


    some debug messages
     
  16. Offline

    Kizzard

    Yeah Sorry I meant client side options



    Ok we tested with 0 buffering and it is MUCH better!
    However our lag rises appreciably when downloading new chunks, I believe more so than than ever happened in vanilla. Do you think the fairness manager could be responsible? Perhaps an option to disable it would be useful for testing?

    Thanks

    EDIT: Done some more play testing and it I can confirm that when my friend is downloading new chunks, my lag (and presumably everyone else who might be on at the time)'s lag skyrockets. I'm talking like 5000-10000ms.

    EDIT: Teleported to my friend, and I get stuck floating in the void, no chunks loading. Re-logging in doesn't help. Console says: Fairness Manager Interrupted when waiting for timer to close

    EDIT: Forgot to mention, this is v77
     
  17. Offline

    Simanova

    Both confirmed as described above!
     
  18. Offline

    Faust

    The latency issue has been resolved for me with version 77, and my users have said the same thing, great job Raph!
     
  19. Offline

    Raphfrk

    I updated again

    • Removed output buffering
    • Changed how fairness manager works, low priority packets always beat high priority ones
     
  20. Offline

    Simanova

    Nothing changed in 79 - same issue as version 78

    Perhaps your changes are well working - but doesnt affect our server :(
     
  21. Offline

    Raphfrk

    Hmm, so updated again.

    • Bypassed fairness manager.

    Updated again

    • Made previous change thread safe.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  22. Offline

    Simanova

    Fixed my issue. Good work
     
  23. Offline

    Raphfrk

    Hmm, wonder what the issue is. Anyway, glad it is working now.

    @Kizzard , does it fix your issue too?
     
  24. Offline

    Simanova

    I noticed, that the fairness manager is limiting all connections to the level of slowest conection.

    eg.

    1 connected via LAN ( 2ms latency )
    1 connected via internet / dsl 16k ( 58ms latency )
    1 connected via umts ( 16 kb/s download ) ( 1200 ms latency )

    all clients are now limited to umts speed
    this could also had happened with latency
     
  25. Offline

    Raphfrk

    Need to give it more thought.
     
  26. Offline

    ledhead900

    What does the buffer latency do?

    @ others

    So it lags when connecting via lan thats strange I never expeirenced that until this last version.
    it may have been minor issues but nothing id call noticable lag.

    But this is a great plugin/tool. What I dont understand is why a one man can do this but notch cant even get simple network optimizations correct for 4 game versions with a team that apparently do nothing but go out on excursions.

    I have actually spoken to notch via twitter and so have my server staff about the light of more seriouse issues then just silly graphical bugs and so on and I can safely say that man is in a fantasy world or just plainly does not care at all about the end product.

    for instance there is this exploit you can currently do on the client that invloved opening the minecraft.jar and removing the maps.txt from the achivement folder on slower rigs this gives 5-40 fps increase.

    When we confronted notch about this he blunty denied any such file did anything other then look pretty then confronted by the 10 other who tested and confirmed it got into more denial telling us that the maps.txt has no part in the game and does nothing and was only used once.

    If this was the case why would it yeild even 4fps increase on even the best of machines while having no negative impact at all why is it even in the game to start with like a lot of his other redundent files and code.

    When confronted about the underlying network issues flawing smp thats when he called bail and stoped responding to us peroid.

    How long are people going to put up with that I paid for a game in alpha so what I paid for it knowing it was going to get BETTER not worse, so far BETA peforms worse then late Alpha and on recent times is an all time High for bug reports and SMP connection issues.

    Sorry about this lengthy thing its kinda relevant as we are talking about network optimization.
     
  27. Offline

    Kizzard

    Yes! Wonderful, so much better! Thank you.


    Yeah, this is exactly what I experienced
     
  28. Offline

    rtcabooservb

    Latest build is working muuuuuuuuuuuch better. Thank you :) Still haven't seen a donation button for your plugins.
     
  29. Offline

    Raphfrk

    Nothing now, since it is bypassed.

    I might add it back in this evening, since it looks like it was the fairness manager than was causing issues.

    What it did was buffer the output stream. If the server sends 10 (network) packets of 5 bytes each, it would combine them into one (network) packet and then send that.

    The max latency was how long it would wait between when it received the first piece and when it flushed the stream. Once the buffer filled (512 bytes), it would auto-flush.

    Notch doesn't use a BufferedOutputStream. In theory, he should write a packet and then use the .flush() method.

    Even without the buffer, I deal with (Minecraft) packets as units, so at minimum each packet would be combined into a unit.

    It looks like the lag occurred when at least one of the players had a slow connection.

    Thanks.
     
  30. Offline

    Simanova

    And this could happen when server bandwidth is used by 100% maybe?
     
  31. Offline

    Raphfrk

    That was what it was trying to solve. It only allowed data to be sent to one player at a time.

    However, that just meant that the server's uplink wasn't been used at 100%.
     

Share This Page