Solved Help Dynamically Compiling Source Code

Discussion in 'Plugin Development' started by Orcane, Aug 7, 2018.

Thread Status:
Not open for further replies.
  1. Offline

    Orcane

    I'm working on a project that requires source code to be extracted from a text file, compiled then run. I have this working properly in debug mode. However this is only because of the fact that I have access to a common ground for all my plugin's byte code and my later dynamically compiled byte code, Eclipse's temporary bin folder. This makes it easy because the class path for all of the compiled byte code is accessible.

    The issue arises when you want to run the byte code run on a server. Since all the plugins are pre-compiled and compressed into their .jars, I don't have this common ground for my compiled byte code and the plugin's byte code, Meaning I can still have my compiled code run, however can't access any classes from other plugins, which in my case, is not useful.

    This is the code I used to read a text file, convert it into Java code, and compile it.
    Code:
    public static final ClassA createClassAFromFile(File file) {
        try {
            String simpleClassName = file.getName().replaceAll("\\.[^.]+$", "");
            String className = "this.is.a.package." + simpleClassName;
            StringBuffer codeBuilder = new StringBuffer();
            BufferedReader reader = new BufferedReader(new FileReader(file));
            String line;
            while ((line = reader.readLine()) != null) {
                codeBuilder.append(line);
            }
            reader.close();
            SourceCode source = new SourceCode(className, codeBuilder.toString());
    
            // This is the class path of the classes within the plugin that are already compiled. This will just be the path to your plugin.jar, since you can't refer to compressed directories.
            String classpath = URLDecoder.decode(ClassB.class.getProtectionDomain().getCodeSource().getLocation().getPath(), "UTF-8");
    
            // Requires the server to be ran using JDK
            JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
    
            // This is the line that actually compiles the code, I believe it doesn't know what to do with the classpath, as it's not a directory. However no exceptions are thrown here.
            compiler.getTask(null, null, null, Arrays.asList("-classpath", classpath), null, Collections.singleton(source)).call();
    
            // And then a ClassNotFoundException is thrown here, because it didn't manage to save the byte code into the compiled byte code to the same directory as ClassB's byte code, so ClassB's ClassLoader can't find it.
            Class<?> newClass = ClassB.class.getClassLoader().loadClass(className);
            return (ClassA) newClass.newInstance();
        } catch (Exception e) {
            e.printStackTrace();
            return null;
        }
    }
    
    private static class SourceCode extends SimpleJavaFileObject {
    
        private final String source;
    
        public SourceCode(String name, String source) {
            super(URI.create("string:///" + name.replace('.', '/') + Kind.SOURCE.extension), Kind.SOURCE);
            this.source = source;
        }
    
        public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
            return source;
        }
    }
    So that's where I'm stuck. Anyone have any ideas on how I can dynamically compile and run java source? Perhaps I'm taking the wrong approach? I'd love to hear your ideas.

    Cheers,
    ~ Orcane (Jeremy)
     
  2. Offline

    Zombie_Striker

    The issue is that you're not checking for dependancies. You need to make sure that the plugins that need to rely on other plugins are loaded last. Bukkit/Spigot normally uses the plugin.yml for this, so if your loader also reads the plugin.ymls, you should be able to load them in order.
     
  3. Offline

    Orcane

    No, It's a bit more complicated than that, I'm trying to access my own plugin's byte code from the class loader that dynamically loaded different byte code. It's really confusing.
     
  4. Offline

    Zombie_Striker

    @Orcane
    Yes, that will be pretty complicated to work around/fix.

    What is the ultimate goal of this? Why is your project required to load the code as byte code instead of just using a jar?
     
  5. Offline

    Orcane

    The idea is to easily be able to develop new modular scripts without having to recompile the .jar each time a new one is added, and instead of developing my own scripting language, I thought I would just execute java code.

    What I'm trying to do is use the same ClassLoader that that loads all of my plugin's .class files to load my compiled .class file.
     
    Last edited: Aug 7, 2018
  6. Offline

    Zombie_Striker

    @Orcane
    But again, why is it required? If it is simply because you want it done this way/ to simply have it exist, from what I can tell, you're going to have to rewrite/refactor your code to make sure they are loaded in a specific order so only classes that have been loaded can be referenced.

    Unless you're rewriting most of minecraft, with plugin file size that is larger than a Mb, recompiling jars should not take more than a couple seconds for mode IDEs, so you should not have to worry about downtime created by loading a new version of the plugin.
     
  7. Offline

    Orcane

    It's easy to loaded any required classes that are unloaded, the problem is just loading them in the same place so that they have access to one an other. And it goes deeper than having it just exist, for my project, it is required.
     
  8. @Orcane
    No, no, no, no. This is a classic case of doing something insane just because you think it will be marginally better than just doing it the way everyone else does it.

    If you were to do this, you would have to create your own custom ClassLoader to first of all load the new classes (just because you have compiled source code somewhere does not automatically mean that it can be loaded by a ClassLoader) and this classloader will have to have the PluginClassLoader (for one of the loaded plugins on the server) as a parent, so that any classes loaded by the JavaPluginLoader (IE regular plugins) will be available to the classes your ClassLoader loads.
     
  9. Offline

    Orcane

    I know what I need, and seek help on that topic.

    As for your reply, that's what I feared when I was thinking of ways to do it. I'll only need to refer to classes in the local plugin, so if I do, do it that way, I'll only need to recreate the ClassLoader for it. I was hoping someone would have done it before, or had an advanced knowledge about the topic. But I guess I'll just have to do more research.

    I'll make a post if I ever find a way to do it.

    Cheers
     
  10. @Orcane
    Well, there are people who have done similar things in the past (myself included), but I just don't see the point. It's not a system that you can deploy to users, so why not just recompile the plugin?

    I didn't mean you have to recreate the PluginClassLoader, you just have to create your own ClassLoader which has the PluginClassLoader as a parent (you just have to pass it as an argument to the constructor). Every PluginClassLoader has access to all of the plugins on the server, not just the plugin it loads (this is because if the classes aren't found it asks the JavaPluginLoader, which has control over all the ClassLoaders on the server), so you should have no worries there.

    You could probably even get away with a URLClassLoader and point it to the directory where you keep your compiled bytecode.
     
  11. Offline

    Orcane

    That's literally exactly what I was just trying. No luck yet. But I'll keep going.
    Thank the heavens for StackOverflow

    @AlvinB
    One thing that does confuse me though, you can specify a class path when you're compiling your source to byte code, and I'd presume that class path is where you want your compiled class to look for dependency classes. Must that class path be the same as the location that your byte code is saved to? If that's the case, I have a problem, as I can't save my byte code into it's plugin's .jar with all the other classes' byte code.

    If that's not the case, theoretically you're right, I can just use a URLClassLoader to load my byte code with the class path pointing to my plugin's .jar.
     
    Last edited: Aug 8, 2018
  12. @Orcane
    Well, mind showing us the code? It's not very easy to help you if you don't show us what you're doing. :p

    You still haven't explained the reason for wanting to create such a system, but I guess you won't be convinced not to use it (which is a shame since being a good programmer is not just about being able to write proper code, it's also about choosing a sensible approach to a problem).
     
  13. Offline

    Orcane

    I'm writing the plugin for a client that will play out a RPG-like scenario with different trigger events and actions, however my client wants to be able to be able to edit certain aspects of the scenario long after the project is finished. So I've made a really user-friendly scenario-building API, and want to utilize this API in dynamically compiled code to run modular scenarios. This way, my client can easily edit the code, from text files, without having to FTP the jar to the server, and doesn't need the knowledge of compiling jars.

    This is my code as of now. I'm not entirely sure that the path to my compiled byte code is in the correct format.
    Code:
    public static final ClassA readLesson(File file) {
        try {
            String name = file.getName().replaceAll("\\.[^.]+$", "");
            String className = "package.name." + name;
            DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();
            JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
            StandardJavaFileManager fileManager = compiler.getStandardFileManager(diagnostics, null, null);
    
            List<String> optionList = new ArrayList<String>();
            optionList.add("-classpath");
            optionList.add(createClassPath());
    
            Iterable<? extends JavaFileObject> compilationUnit = fileManager.getJavaFileObjectsFromFiles(Arrays.asList(file));
            JavaCompiler.CompilationTask task = compiler.getTask(null, fileManager, diagnostics, optionList, null, compilationUnit);
            if (task.call()) {
                String byteCodeDirectory = "file:///plugins/PluginName/scenarios/";
                URLClassLoader classLoader = new URLClassLoader(new URL[] { new URL(byteCodeDirectory) });
                Class<?> loadedClass = classLoader.loadClass(className);
                Object obj = loadedClass.newInstance();
                classLoader.close();
                fileManager.close();
                if (obj instanceof ClassA)
                    return (ClassA) obj;
            } else {
                for (Diagnostic<? extends JavaFileObject> diagnostic : diagnostics.getDiagnostics()) {
                    Utilities.info("&4Diagnostic&8: &7" + diagnostic.getMessage(Locale.US).replace("\n", "").replaceAll(" +", " "));
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }
     
  14. @Orcane
    Well alright, you must have found the first person who can write proper Java but can't compile a jar. Not to mention that I don't see why a server would be running a JDK instead of a JRE, but that's not my can of worms.

    Looking at your code, you didn't pass the PluginClassLoader as a parent to the URLClassLoader. This means the loaded classes won't be able to access any classes in your plugin or any other.

    Also, I recommend having the ClassLoader as a field in the class. It's not very resource-efficient to have to open a new ClassLoader each time you want to load a class.
     
  15. Offline

    Orcane

    My 'createClassPath()' returns a classpath that has the location to my plugin's jar, I believe that's why the my plugin's classes that are referred to in my compiled code ARE FOUND. The problem currently is my URLClassLoder can't find my compiled byte code (java.lang.ClassNotFoundException). Even with this modification to the code.

    Code:
    File byteCodeDirectory = new File("C:/path/to/byte/code/directory");
    TNClassLoader classLoader = new TNClassLoader(new URL[] { byteCodeDirectory.toURI().toURL() });
    Class<?> loadedClass = classLoader.loadClass(className);
    And yep, I'm just trying to get it working before I worry about efficiency.
     
  16. @Orcane
    You need to have the PluginClassLoader as a parent. Otherwise, any classes that you load with the URLClassLoader will throw errors as soon as they reference a class in your regular code.

    But you seem to have another issue as well. What if you try putting a slash at the end of the path too? That should make sure that the path is treated as a directory.

    Also, can you send a screenshot of how the compiled files look?
     
  17. Offline

    Orcane

    Added a slash, and made the PluginClassLoader the parent of the class loader.

    This is what the compiled code looks like.
    https://gyazo.com/5a417b2e3f03e89c60d952e7a1846d8a

    Update:

    Ahh, I got it working. The problem was I was telling the class loader to read from that 'scripts' directory, but didn't generate the required directories for my compiled classes' package declaration. So it was looking in the right place, but my compiled code wasn't.

    Anyway, this is the working, but not cleaned or optimized code for the the 2 people who ever want to attempt this stupid project.
    Code:
    public static final ClassA readLesson(File file) {
        try {
            String name = file.getName().replaceAll("\\.[^.]+$", "");
            String className = "com.package." + name;
            File byteCodeDirectory = new File("C:/Location/To/Compiled/Code");
            DiagnosticCollector<JavaFileObject> diagnostics = new DiagnosticCollector<JavaFileObject>();
            JavaCompiler compiler = ToolProvider.getSystemJavaCompiler();
            StandardJavaFileManager fileManager = compiler.getStandardFileManager(diagnostics, null, null);
    
            List<String> optionList = new ArrayList<String>();
            optionList.add("-classpath");
            optionList.add(createClassPath());
            optionList.add("-d");
            optionList.add(byteCodeDirectory.getPath());
    
            Iterable<? extends JavaFileObject> compilationUnit = fileManager.getJavaFileObjectsFromFiles(Arrays.asList(file));
            JavaCompiler.CompilationTask task = compiler.getTask(null, fileManager, diagnostics, optionList, null, compilationUnit);
            if (task.call()) {
                URLClassLoader classLoader = new URLClassLoader(new URL[] { byteCodeDirectory.toURI().toURL() },
                        Main.class.getClassLoader());
                Class<?> loadedClass = classLoader.loadClass(className);
                Object obj = loadedClass.newInstance();
                classLoader.close();
                fileManager.close();
                if (obj instanceof ClassA)
                    return (ClassA) obj;
            } else {
                for (Diagnostic<? extends JavaFileObject> diagnostic : diagnostics.getDiagnostics()) {
                    System.out.println("Diagnostic: " + diagnostic.getMessage(Locale.US).replace("\n", "").replaceAll(" +", " "));
                }
            }
        } catch (Exception e) {
            e.printStackTrace();
        }
        return null;
    }
    Thanks for the help @AlvinB
     
    Last edited: Aug 8, 2018
  18. @Orcane
    The class needs to be in folders corresponding to its package name. In the following example:
    Code:java
    1. try {
    2. ClassLoader classLoader = new URLClassLoader(new URL[] {new File(this.getDataFolder(), "classes/").toURI().toURL()}, this.getClassLoader());
    3. Class.forName("com.bringholm.testplugin.LoadingTestClass", true, classLoader);
    4. e.printStackTrace();
    5. }

    I would need the LoadingTestClass.class file to be located in the folders com/bringholm/testplugin/ (where the com folder would be in the folder that you point the ClassLoader too).
     
    Orcane likes this.
  19. Offline

    Orcane

    Haha, Thank you ;)
     
  20. @Orcane
    Nice that you solved it. Just don't go too crazy with ClassLoaders, will ya? There can be a serious security risk with dynamically loaded code if you're not careful. :)
     
  21. Offline

    Orcane

    I'll try. ;)
     
Thread Status:
Not open for further replies.

Share This Page