[MISC] CraftProxy 0.2.0 - Reduce bandwidth use by caching chunk data [1.6]

Discussion in 'Archived: Plugin Releases' started by Raphfrk, May 1, 2011.

  1. Offline

    Raphfrk

    CraftProxy - Reduce bandwidth use by caching chunk data
    Version: 0.2.0

    Description

    This plugin and local client allows chunk data to be cached locally by players. This reduces the bandwidth required for hosting the server. It also helps users who are on slower connections.

    I am not sure what the status of this system is. It doesn't count as a plugin, since it has 2 parts.

    Even when running the plugin, players who don't use the client proxy can still connect. However, they will use the full bandwidth.

    The system can reduce bandwidth by 70-90% (after the 2nd login).

    Setup

    Server

    Add the plugin file to the plugins folder

    Client

    Start minecraft client and login
    Double click on the client jar file
    Enter login details
    Enter the server location/port in the GUI
    Press start on the GUI
    Connect to localhost on the minecraft client

    Stable Builds

    None yet

    Dev Builds

    Warning: These may not be stable

    Client
    Plugin

    Had it get the compress/decompression gain backwards.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
    Jushy, Martin1704, Nathan C and 24 others like this.
  2. Offline

    Raphfrk

    If you interrupt the thread, it should close eventually.

    Something like

    Code:
    thread.interrupt();
    
    while (thread.isAlive()) {
        thread.interrupt();
        thread.join(100);
    }
    
    You would need to somehow call that when the client is closing.
     
    Hwo likes this.
  3. Offline

    but2002

    I've been using this for a while now, and I must say it's been a tremendous help.

    Suggestion though?
    Client-Side mod that integrates this, and only this..
     
  4. Offline

    Raphfrk

    Which part, caching or teleporting between servers?

    Spout already does caching and reconnecting. However, it doesn't support single port operation, that fundamentally needs a proxy.
     
  5. Offline

    but2002

    Caching / Compression whatnot.
    I don't want to install Sprout server or client mod, hence why I asked.
     
  6. Offline

    JohnEBH

    Got a question.
    If I have both Craftproxy and Spout, can they work together?
    Server is on 25564, Craftproxy 25565.
    If a Spout client connects to 25565 with Craftproxy also running, would it do double the bandwidth reduction awesomeness?
    BTW. It's just a thought, I'm not doing that... For now.
     
  7. Offline

    Raphfrk

    No, they both do the same thing. Caching twice is just a waste of disk space. However, if you connect to the server proxy directly with the spout client, then it will act as a passthrough and spout will do the caching.

    The spout client integration represents me integrating the caching into the client.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  8. Offline

    JohnEBH

    Ah I see, and if my friends are still using the normal client w/ craftproxy they'll have the same/similar caching effect. And when spout client connect to the craftproxy port, Spout should take over caching? Right?
     
  9. Offline

    Raphfrk

    Yeah. The can get caching using either Spout or the local proxy.
     
  10. Offline

    Takel

    Heh, you're probably going to get asked the same question a fair few times, Raphfrk. You might want to update the OP with a quick FAQ about the CPL + Spout interaction...
     
  11. Offline

    but2002

    After the update to the latest RB, CraftProxy connections are going EXTREMELY slow, with an extreme amount of lag, and I don't get that with RB 1000 or a normal connection.
     
  12. Offline

    Raphfrk

    CB did something around CB 1039 which is causing major problems with Craft Proxy.

    Is the memory usage for the the CB process rising? I am hoping to get a memory dump.

    Can you start craft proxy with something like:

    Code:
    java -Xms256M -Xmx256M -XX:+HeapDumpOnOutOfMemoryError  -XX:HeapDumpPath=dump2/dump.hprof -jar CraftProxyLiter.jar 20000 25565
    
    change dump2/dump.hprof to some path that can save the file.

    When the Craft Proxy process runs out of memory, it will write a file that can be used in analysis.

    Ok, I added the request to the OP.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 14, 2016
  13. Offline

    but2002

    Sorry, we're on a production server so it could be a bit long before I can do something like that. :\
     
  14. Offline

    Raphfrk

    Fair enough, I added it to the OP, maybe someone can do it :). Simanova is looking into it, but would be better to get the actual memory dump
     
  15. Offline

    Hwo

    Can i change "defaultHostname" on fly when thread of client-side-CPL start already but client not connect to CPL yet?
     
  16. Offline

    Raphfrk

    No not with the default code. The default hostname is a final variable.

    You would need to change it in ProxyListener from being final to be changable and also thread safe.

    I haven't tested this, but it should allow you to change the defaultHostname using

    ProxyListener.defaultHostname.set("some.hostname:12345");

    In ProxyListener.java

    Code:
    private final String defaultHostname;
    
    to

    Code:
    public final static AtomicReference<String> defaultHostname = new AtomicReference();
    
    change

    Code:
    this.defaultHostname = defaultHostname;
    
    to

    Code:
    ProxyListener.defaultHostname.set(defaultHostname);
    
    and

    change

    Code:
    PassthroughConnection ptc = new PassthroughConnection(socket, defaultHostname,  listenHostname, fairnessManager, this);
    
    to

    Code:
    PassthroughConnection ptc = new PassthroughConnection(socket, defaultHostname.get(),  listenHostname, fairnessManager, this);
     
     
    Hwo likes this.
  17. Offline

    Hwo


    Thank you very much.


    When a client is connected through the CPL, it saves the chunks in memory and writes them to disk when finished. When you run it reads all the chunks from the disk or not all?

    Then checks the hash of the local chunks and on the server and loads if the same and gives the chunks to the client? Or first gives the chunks to the client, then checks whether the chunks are different on the server?


    Is there a chek or batch processing and moving the player's actions?
    I understand that the amount of information transmitted is reduced. But as far as increasing the delay?
    Will it affect comfort in pvp mode?
     
  18. Offline

    Raphfrk



    It writes them always to disk and only reads as required. However, it won't delete the files until you properly close down.


    When you close down, it deletes the older files, but only if you have gone above the cache size limit.



    It sends the chunk pieces that are stored in the same file to the server (just the hashes of them). Each chunk piece is around 2.5% of a chunk. There are 512 cache pieces stored in each file.


    If you look in the cache directory, it is a long list of files. The files with the lower numbers are older, it uses that to decide which one to delete.


    It checks each map chunk as it is sent. It should only delay map updates.

    However, since it makes them smaller, it is probably overall faster for map updates. It does load the CPU more than would normally happen.

    One issue is that it doesn't do packet reordering, so there could be higher latency while the big packets are being processed.
     
    Hwo likes this.
  19. Offline

    Hwo

    Ok thx.
    But what occur firstly - getting data from local cache or cheсking server hash and getting data from server?

    For example: player walk and client need 21 chunks (view-distance=10) but all world's cache exist in local folder of CPL.
    Will player see local cached chunks firstly and than server's chunks if difference? Or he will wait hash cheking and than see current server's chunks anyway (if equals with local - load from local, else - from server)
     
  20. Offline

    Raphfrk

    The client can send a list of known hashes.

    For example

    File1: Pieces 1 - 512
    File2: Pieces 513 - 1023

    and so on

    There is a map

    hash (8 bytes) -> piece (2048 bytes)

    The process is:

    - Player logs in
    - Server sends the first chunk
    - Client scans it and finds it uses hashes 12,13,14 ...
    - Client sends all hashes in file1 (1 - 512)
    - Server updates list of known hashes
    - Server scans the second chunk.
    -> this has hashes 100,101,102,.....
    - It knows the client has them, so only sends the hash reference instead of the entire piece

    Each time the server sends the client a chunk that contains a piece that the server didn't know about, the client sends all the hashes in that file to the server. However, it doesn't send the hash if the server already knew about it.

    Also, the hash for any piece the server sends is automatically added to the known list. This means the even for the first chunk, there is some compression.

    The first chunk might be

    Piece 1: all air
    Piece 2: all air (sends hash for piece 1)
    Piece 3: all air (sends hash for piece 1)
    Piece 4: all air (sends hash for piece 1)
    ...
    Piece 20: hits ground level (send in full)

    The idea is that since all hashes in a particular file were recorded at the same time, they are probably related, so might as well inform the server of them all.
     
    Hwo likes this.
  21. Offline

    Hwo

    Good algoritm. But i can't understand one thing.
    Client get chunks from cache and then from server if difference or client get last version of chunks anyway? And never get old chunks?

    I know dude, who make local-only caching without protocol editing and without server side plugin.
    Chunks loads from local folder and u see it (stone=diamond on video if load from cache and stone=stone if load frome server)
    http://www.youtube.com/watch?v=VnEOddvRnPc

    It's good for very laggy server with delay ~ 10 secs. And i have sources and can share if u want.
     
  22. Offline

    Raphfrk

    No, the server controls caching or not.

    The server scans a piece and if it is has a known hash, then the server swaps it for a reference. If the client has a piece but never tells the server, then the server won't swap it, so it gets sent as full data.

    The client scans the chunk and replaces any references with the actual piece data. This is why it is important that the client never deletes pieces while connected. Otherwise, it might delete a piece and then the server might send a reference to that piece.

    The server probably still has to send the chunks, it is just that they are loaded from the local cache first?

    My way means that it reduces bandwidth and should reduce lag.
     
    Hwo likes this.
  23. Offline

    Maeyanie

    Ok, so, I realize this isn't really your problem, but I'm fairly stuck so figured it wouldn't hurt to ask...

    I was trying to add support for packet 230 (used by ModLoaderMP) to CraftProxy, and after a few tries I think I got it right; it can parse through one and repeat it without anything giving errors or crashing, and keeps going fine.

    However, after a minute or so, the client just stops sending data. The server still sends fine, and the client keeps getting it, but downstream thread reads just time out repeatedly until the client finally gives up and exits.

    Here's a patch with the code I came up with, if it helps any, either for suggestions or if you wanted to add it. Still a bit of debugging in there though.
    Show Spoiler

    Code:
    diff --git a/com/raphfrk/protocol/PacketScan.java b/com/raphfrk/protocol/PacketScan.java
    index bddcf95..7500999 100644
    --- a/com/raphfrk/protocol/PacketScan.java
    +++ b/com/raphfrk/protocol/PacketScan.java
    @@ -245,6 +245,37 @@ public class PacketScan {
                     }
                     break;
                 }
    +            case INT_ARRAY:
    +            case FLOAT_ARRAY: {
    +                int count = getInt(buffer, position, mask);
    +                if (count > 65535) {
    +                    System.out.println("Numeric too long: "+count);
    +                    return null;
    +                }
    +                if (count * 4 > maxPacketSize) return null;
    +                System.out.println("Numeric array: "+count+" @ "+position);
    +                position += (count+1) * 4;
    +                break;
    +            }
    +            case STRING_ARRAY: {
    +                int count = getInt(buffer, position, mask);
    +                if (count > 65535) {
    +                    System.out.println("String array too long: "+count);
    +                    return null;
    +                }
    +                if (count * 4 > maxPacketSize) return null;
    +                System.out.println("String array: "+count+" @ "+position);
    +                position += 4;
    +                for (int x = 0; x < count && position - start <= dataLength; x++) {
    +                    int len = getInt(buffer, position, mask);
    +                    if (len > 65535) {
    +                        System.out.println("String too long: "+len);
    +                        return null;
    +                    }
    +                    position += len + 4;
    +                }
    +                break;
    +            }
                 default: {
                     if(position - start <= dataLength) {
                         System.err.println("Unknown enum type " + ops[cnt]);
    diff --git a/com/raphfrk/protocol/ProtocolUnitArray.java b/com/raphfrk/protocol/ProtocolUnitArray.java
    index 94ea141..fc5aa1b 100644
    --- a/com/raphfrk/protocol/ProtocolUnitArray.java
    +++ b/com/raphfrk/protocol/ProtocolUnitArray.java
    @@ -14,7 +14,10 @@ public class ProtocolUnitArray {
             META_DATA,
             OPTIONAL_MOTION,
             ITEM,
    -        ITEM_ARRAY
    +        ITEM_ARRAY,
    +        INT_ARRAY,
    +        FLOAT_ARRAY,
    +        STRING_ARRAY
         }
     
         public static Op[][] ops;
    @@ -32,6 +35,9 @@ public class ProtocolUnitArray {
         
         public static OpPair item             = new OpPair(Op.ITEM, 0);
         public static OpPair itemArray        = new OpPair(Op.ITEM_ARRAY, 0);
    +    public static OpPair intArray         = new OpPair(Op.INT_ARRAY, 0);
    +    public static OpPair floatArray       = new OpPair(Op.FLOAT_ARRAY, 0);
    +    public static OpPair stringArray      = new OpPair(Op.STRING_ARRAY, 0);
     
         static {
     
    @@ -94,6 +100,7 @@ public class ProtocolUnitArray {
             opPairs[0x82] = new OpPair[] {jump(11), string16, string16, string16, string16};    
             opPairs[0x83] = new OpPair[] {jump(5), byteSized};
             opPairs[0xC8] = new OpPair[] {jump(6)};    
    +        opPairs[0xE6] = new OpPair[] {jump(9), intArray, floatArray, stringArray};
             opPairs[0xFF] = new OpPair[] {jump(1), string16};    
             
             ops = new Op[256][];
    
     
  24. Offline

    Hwo

    Yes it is.

    But what about servers with good connection and low ping, but with bad hardware or bad plugin's thread optimisation? Can u include option for load local chunks first?
    So if player have all world in local cache - he can fly with huge speed and never see void chunks with any server hardware and conect.
     
  25. Offline

    Raphfrk

    The local cache doesn't work that way. It just stores lots of pieces, it doesn't say where they are placed.

    For example, there would be 1 piece that is all air, and it is just placed lots of times.
     
    Hwo likes this.
  26. Offline

    Aox

    I'm really pulling for you on this one, Maey. I tried to solve my multiworld buildcraft problem with separate servers, but none of the serverport/transport/servertoserver mods are working properly for me. CPL always crashes on these Modloader packets, and your post here gives me hope.

    Thanks again for all you work, and you too, Raph. Lots of us hoping for this one to come together.

    -Aox
     
  27. Offline

    Raphfrk

    It is possible that you are running out of buffer space. You could try increasing the maxPacketSize variable at the top of the file and see if that helps.
     
  28. Offline

    Maeyanie

    Doesn't seem like it. I turned it up to 1 MB (in both PacketScan and LocalSocket) and tried that... the first time it crashed. Then the second time, when I had it log to a file so I could read the debug spam more easily, it worked. But only worked that once; tried several more times and they all got disconnected.

    Seems to have enough buffer space available though, I added a println inside ProtocolInputStream's SocketTimeoutException handler and when it's disconnecting it gives a few dozen identical lines like this:

    Code:
    Read timed out. available=279115 startMod=769461 endMod=769461 length=1048576 packetId=-1
    (Length there is buffer.length, others are the local variables of the same names, both client and server ends of the proxy give very similar (but not identical) messages.)

    The numbers vary from one run to another, but in general, there's several hundred KB free space. But (if I'm interpreting the code right) having several hundred KB sitting in the buffer already without being able to find a packet in it suggests something is going very wrong somewhere.

    Edit:
    Hmm... actually on closer review it looks like I was wrong, and it's the server->client stream that's freezing, not the other way around.
     
  29. Offline

    Raphfrk

    I was going to have a look, but I seem to be downloading the wrong client files, or something.

    I downloaded mod-loader from here and mod-loader-mp from here. The server seems to work, but I am getting bad packet type on the client (230), so I assume at least the server is modified correctly.
     
  30. Offline

    Maeyanie

    The latest version of ModLoaderMP is here. The original author has gone missing, but it's been unofficially kept up-to-date by that thread's owner.
    Older versions used a lower packet number (I forget what), but then real Minecraft took it so they increased the number to 230, so it could have been an older version or something.

    But, yeah, a packet 230 error is a sign of ModLoaderMP (or, at least, the 1.7.x version) not being installed into the client properly, as that's the packet used... and the one I'm trying to get CraftProxy to handle. :)

    If you need it, the source for a packet 230 (in its Bukkit flavour) is here.

    And if you need any mods to try it with, the Bukkit conversions of several of the most popular are here.
     
  31. Offline

    Tueurdragon

    Hello,

    I installed craftproxy, and I would set up a whitelist with the plugin,

    I create a white list file which is implemented by my website.

    can you tell me how to configure the file craftproxy
    to set up the whitelist.

    thank you very much
     

Share This Page