Util Persistent Data Past Reload (v0.1)

Discussion in 'Resources' started by MrAwellstein, Jan 27, 2015.

Thread Status:
Not open for further replies.
  1. Offline

    MrAwellstein

    So today I bring you something special. It can be extremely powerful, or extremely harmful, depending on how it is used and the user behind it. I don't know why I am releasing this tool, but it may be because I'm tired of plugins that cause my server to freeze for 10 full seconds during a reload. Or maybe it's because I'm a jerk and I like being first to publicly create stuff. Anyways, this tool, dubbed, for a lack of a better name, UnsecureClassloader, allows for some special stuff to happen. It allows for classes to be loaded preplugin in the ClassLoader Hierarchy. The significance of this, is that it allows devs to reconnect to threads created and recover data during reloads without major reflection. The simplicity of this tool allows even a beginner to use it (correctly I hope).

    UnsecureClassloader.java

    Example Uses:
    ReloadCounter.jar (Source)

    How to Use:

    Create a custom (Data)Thread. This thread is used to store data, and then reconnect to the thread and retrieve the data. This class created is special in the fact that it needs to be loaded with our utility or else we wont be able to reconnect to the thread and our data will be forever lost (unless you take the long way and use reflection to get the fields and such). Make sure it has a run method which makes it sleep, and a end method which interrupts its sleeping, and a way to store information. Be aware of what you store in here, because anything that is loaded with your plugin (ex: other classes), wont be accessible because of casting exceptions in the Class Loader. If you don't understand what I mean by anything loaded with your plugin: look at your project in Eclipse. See the files that you created? Anything that isn't preloaded with the Utility, will be inaccessible when you try reattach to the thread.

    DataThread.java (open)

    Code:
    package com.deadmen.example.dataplugin.resources;
    
    public class DataThread extends Thread{
    
        public DataThread(String name){
            super(name);
        }
      
        public DataThread(String name, DataObject dO){
            super(name);
            this.dO = dO;
        }
      
        private DataObject dO;
      
        public DataObject getDataObject(){return dO;}
      
        @Override public void run(){
            while(!this.isInterrupted()){
                try{
                    Thread.sleep(10000000);
                }catch(Exception e){break;}
            }
        }
      
        public void end(){
            this.interrupt();
        }
      
        public static DataThread retrieveDataThread(String name){
            try{
                for(Thread t: Thread.getAllStackTraces().keySet()){
                    if(t.getName().equals(name)) return (DataThread)t;
                }
            }catch(Exception e){e.printStackTrace();}
            return null;
        }
      
    }


    Next we want to create a "DataObject" class. The point of this class is to allow an easy mount to retrieve data. You can add any field that you would like, that is already loaded (ex: what you preload, and bukkit classes). NOTE TO NOT USE THIS CLASS FOR METHODS. IT SHOULD REALLY ONLY BE USED FOR STORING FIELDS. IF YOU USE METHODS IN THIS CLASS, MAKE SURE TO HAVE AT LEAST A BASIC UNDERSTANDING OF CLASSLOADER HIERARCHY!

    DataObject.java (open)

    Code:
    package com.deadmen.example.dataplugin.resources;
    
    public class DataObject {
    
        //...
    }


    Finally we come to our Main method, which most of the programmers here seem to name their plugin's name (I name it Main so its easy to find, and because it houses the main starting point of the program). onLoad, we want to check if the classes are loaded. To do this, we use the tool's checker methods, which return a boolean. If they aren't loaded, then load them using the proper methods and init the DataObject field. Make sure not to init the DataObject field before the class is loaded with the Util, or else it will cause you to crash. If the classes are loaded, then try to reconnect to the thread which stores the information. If you connect to it, then get the information and store it in the DataObject field. Then end the thread. You want the thread to die so its taking up less memory and processing power than needed. We only needed it to be persistent past a reload, so it's done it's job. onDisable, we want to wrap the DataObject in a new Thread, and start it. This will restart the Cycle. Take note, that you can't reuse threads, and once they are terminated, they are dead for good.

    Main.java (open)

    Code:
    package com.deadmen.bukkit.main;
    
    import org.bukkit.plugin.java.JavaPlugin;
    
    public class Main extends JavaPlugin {
    
        private static final String threadName = "DataThreadForPlugin";
      
        private static DataObject DO;
        public static DataObject getDataObject(){return DO;}
        public static void setDataObject(DataObject dO){DO = dO;}
      
        public void onDisable() {
            new DataThread(threadName,getDataObject()).start();;
        }
        public void onLoad() {
            if(!UnsecureClassloader.classDefined("com.deadmen.example.dataplugin.resources.DataObject")) { //check to see if one of, or more than one, of the classes were loaded in.
                try{
                    UnsecureClassloader.loadPackage(this.getFile(), "com.deadmen.example.dataplugin.resources"); //Load the package that houses the resources that are needed
                    DO = new DataObject();
                }catch(Exception e) {e.printStackTrace();}
            }else {
                DataThread d;
                if((d = DataThread.retrieveDataThread(threadName))!= null) {
                    DO = d.getDataObject();
                    d.end();
                }else {
                    DO = new DataObject();
                }
            }
        }
    }


    That's all there really is to it. If anyone wants to improve on anything here go right ahead. The Utility, created by CyanFrost and I, is free for anyone to use, but I would love to see one plugin be used as a master persistence store so we don't have a large amount of threads when they aren't really needed.
     
    Last edited: Jan 27, 2015
    ChipDev likes this.
  2. Offline

    fireblast709

    zDylann, Bammerbom and API_Tutorials like this.
  3. Offline

    Not2EXceL

    Last edited by a moderator: Jan 27, 2015
  4. Offline

    MrAwellstein

    Please only have constructive criticism, instead of just blind, hateful comments, if you will. If you want to say something bad about my Util, then please support your claim with reason and fact.
     
  5. Cleaned up thread
     
  6. Offline

    MrAwellstein

    Thank you.
     
  7. Offline

    fireblast709

    It's not a blind, hateful comment. You leak memory. Your whole concept is about leaking memory. And the fact that your plugin works proves that:
    • Bukkit is supposed to unload your classes and load them from disk.
    • This also means your class is loaded from disk, and is assigned to another part of memory than your first load.
    • In turn, assigning the ReloadCounter from the Thread to your field should cause a ClassCastException
    • But it doesn't. This means that your initial ClassLoader and the new ClassLoader (reloads create new ClassLoaders) are both in memory => nice memory leak you got there.
     
  8. Offline

    MrAwellstein

    My ClassLoader isn't a real ClassLoader. My classloader, is actually unloaded with the server on each reload. The purpose of it, is to put one or two classes into the Main JVM classloader, which, isn't a memory leak because they aren't constantly readded and they get removed when the JVM turns off. The purpose of this is to prevent the ClassCastException, by making the Class declared before the plugin is, making it like a dependency. Your statement makes me think that you posted this without actually looking at my code :/

    Edit: I wasn't the one who named it a "ClassLoader", but I was under the impression that it was named UnsafeClassloader because it was a virtual ClassLoader which used reflection to use classes.
     
    Last edited: Jan 27, 2015
    ChipDev likes this.
  9. Offline

    fireblast709

    I wasn't talking about yours, I was talking about Bukkit's ;).
    And they will stay there even if you remove the plugin => memory leak. Plugins are not supposed to have memory claimed after they are removed, it's called bad memory management
    While the trick itself is pretty smart, the fact that you can't remove the plugin is pretty bad and should be enough reason not to resort to your trick. If you so desire to be able to reload a plugin without that 10 second delay, you could also start by writing it in a way that allows you to load data without freezing the main thread (decent asynchronous programming, perhaps?)
    Such a shame that people think I would criticize without reading source code...
     
  10. Offline

    Not2EXceL

    You forgot to mention that #onLoad() is actually called after plugin instantiation, which makes the claim of declaring the class before the plugin invalid. static blocks will execute before #onLoad() will

    if you want true persistence, be a real programmer (yes i totally jacked this from a convo with cyan and another user), and use bytecode injection (eg. instrumentation) to attach a persistence framework to bukkit as to allow plugins to have persistent data, without all the overhead and hacky solutions (one of which you provided)
     
  11. Offline

    MrAwellstein

    I never actually said that this would actually load before the plugin. But in technicality it does load before the plugin, after the first reload. Also bytecode injection isn't needed here. Oh, and FYI, I am a real programmer.

    It's supposed to stay there. Its intentional, so its not technically a leak. Also you can remove the plugin, just not the classes that are injected, which is also intentional. Also your remark about Asynchronous Programming, the main purpose of this was to keep Sockets alive and running, which, don't like to be abruptly ended and restarted without interrupting the user. And yes I do feel like you criticized without reading the source.

    I meant in the classloader hierarchy, not in the actual time.
     
    Last edited by a moderator: Jan 31, 2015
    ChipDev likes this.
  12. Offline

    FerusGrim

    I totally respect @fireblast709 (I don't really know @Not2EXceL), but this utility is badass. I'd like to do some more in-depth testing with it, but just having something like this publicly available (to those who would know how to properly utilize it), is very nice.
     
    ChipDev likes this.
  13. Offline

    fireblast709

    Well, wikipedia disgreed with you on that one.
    Well implement a reconnect function on the client side...

    Clean up your garbage. If you don't want to, well, don't create garbage in the first place.
    (note, because I know you are going to take this offensive, garbage refers to the memory leak, not the resource)
     
    Skionz likes this.
  14. Offline

    Not2EXceL

    I never said you said that it loaded before the plugin. You stated that this declared your class before the plugin's declared, which according to classloader heirarchy the class extending Plugin, will be loaded, then static blocks ran, and finally the instantiation, which then your #onLoad() function runs after all of that happens.

    *the real programmer was taken from a convo between your partner on this util and adam, seriously you too that way too seriously. It wasnt meant to be direct or attacking in any way.

    The classes aren't injected at all. Use proper terminology. You're just using a hackish solution to load the class into Bukkit's classloader instead of the classloader bukkit creates for each plugin.

    And regarding your statement on bytecode injection isn't needed. Your right, it isn't, but then again neither is a hackish solution. If anything one should use what will in the end be most stable and instrumenting a codecave into bukkit will not only be safer memory wise, but also easier extend a framework for others to use, without the reliance upon a plugin loaded by bukkit.

    On the side note, make the UnsafeClassloader (which really should be named to something like BukkitClassLoaderUtil or something) only able to load classes that implement a specific empty interface. I shouldn't have to explain why (as its extremely simple) or how you leave the option of loading any class into the bukkit classloader can cause problems.

    (note, because I know you are going to take this offensive, everything hearby written not only contains opionions and not implications, the inferences made from said post is completely up to the reader and therefore the author is not at fault of anything) *idea totally not stolen from mark
     
    Last edited: Jan 27, 2015
  15. Offline

    MrAwellstein

    I know that you meant the memory, when you meant garbage.
    ALSO: that quote said "when isn't needed", which, it is always needed.
    Also even reconnect functions disrupt the user.

    I'll try to convince my partner to change the name.
    I was thinking that too, but then I realized something: If someone uses my tool, they can simply remove any safety features I add: IE: preventing org.bukkit or net.minecraft packaged classes in.

    I feel as if this hackish solution (which technically isn't all that hacky) safer than bytecode injection. I'll try to change to correct terminology

    You really don't understand how much I agree with you on this one.


    Furthermore, I want to know a better way to do this, if you have one, because I don't think one exist without modifying bukkit. But correct me if I'm wrong ^_^

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Jun 13, 2016
  16. Offline

    fireblast709

    My main point being if I remove the plugin, how is it still needed ;). That's where your memory management fails - it will never be GCed (yes, after the JVM has shut down. But that's not what should happen, you should not keep that memory allocated when it's not used)
     
  17. Offline

    Not2EXceL

    The only thing i gotta say here, is y not just PR a framework to bukkit?
     
  18. Offline

    FerusGrim

    Lol. Who do you think is around, still accepting PRs?
     
  19. Offline

    MrAwellstein

    From my understanding, you cant remove the jar until the JVM has shutdown and is no longer using the Jar file itself, because Bukkit is still technically using it (unless you do it during a reload which I think you have low chances of doing it). Anyways, there is a cost to use certain plugins, and server admins must understand how to use them and how to properly manage them (shut down the server before removing a plugin).
     
  20. Offline

    Not2EXceL

    It was more of a thing about how if you're gonna make a plugin, but it aint gonna be a plugin, and should just be moved to the core, then wat. makes less sense not to, and also didnt 1.8.1 bukkit come out after everything hit the fan?
     
  21. Offline

    MrAwellstein

    No.
     
  22. Offline

    fireblast709

    Well that's a flaw in Bukkit then, no need to abuse that in your argumentation to use hacky tools :p. I've written a plugin that allows me to dynamically load modules before (and no, the source was never published) that allows me to unload them dynamically, so this issue you are referring to is strictly Bukkit's issue.
    See my previous statement. A decent version of this, also the challenge of your fellow developer CyanFrost (make sure you inform him about this), would be using the Bukkit API (modified or not).

    Here ya go, all persistent within the scope of a reload, let us all sing the Metadata song. O yes, Metadata all the way.
     
    Hawktasard likes this.
  23. Offline

    RawCode

    unsafe.defineclass(rootloader, yourclass) that all you need - single line.
     
    xTrollxDudex likes this.
  24. Offline

    MrAwellstein

    I'll do testing with that and update my code. But my Util allows an easy way to get the bytes and actually load them from a jar, rather than having suspicious byte[] somewhere in your code.
     
  25. Offline

    RawCode

    native layout of all data types in java is byte[], you can fool yourself by making multiple layers of wrappers, but this wont change how jvm works.
     
Thread Status:
Not open for further replies.

Share This Page