Solved Loading to Memory vs Reading From Disk

Discussion in 'Plugin Development' started by savagesun, Jun 9, 2013.

Thread Status:
Not open for further replies.
  1. Offline

    savagesun

    I'm in the middle of developing right now, and I came to thinking what the best way to handle decent sizes of data in Minecraft/Bukkit was. Currently when I have to save/load data I use GSON and load any required data from a file on start up or whenever someone logs in and save it to a map or list of data. This would eventually cause the maps/lists to get rather large, taking up more memory than necessary. Although I haven't necessarily run into any issues as of yet, I was wondering if streaming the data from the disk using SQLite to store/query data would be more favorable than just loading things in to memory. Would it cause the plugin to slow down? If so would the slow down be negligible in the face of reduced memory usage? Would it reduce memory usage by a significant enough amount to go through all the programming? Any input is appreciated. Thanks!
     
  2. Offline

    Hoot215

    It really depends on what kind of data you're accessing. If the operation isn't incredibly time critical (databases tend to have a bit of latency) and there's a lot of data (and I mean a lot), then you might consider using a database. However, if this data only needs to be used for a short while (say, for example, when a player is online), you can simply store it in memory and have it removed when you don't need it anymore. If there's no event hook for when your data doesn't need to be accessed anymore, you might consider using a WeakHashMap, but once again, it depends on the type of data you're accessing.

    Also, consider the fact that databases can have quite a bit of memory overhead on their own. MySQL, for example, can use several hundred MBs of RAM, depending on how finely-tuned the database is.
     
    savagesun likes this.
  3. Offline

    savagesun

    Instead of creating a new thread, I'm choosing to bump this thread since I haven't really come to a conclusion yet (so yelling necro is not needed). What I need to do right now, is to access data of both offline and online players. Right now I use GSON to serialize the player data structure (not Bukkit's Player entity) and then save it to a plain text file when data needs to be saved. To load data I just read a new object into a hashmap of data every time someone logs in (and it empties/saves when someone leaves). The issue here is that I can't access (meaning read and write) offline player data without reading the file into memory making changes and then saving the file again. I was looking into the Factions/MCore source and it looks like the way they handle retrieving and changing player data is reading GSON data from some kind of database, or YAML file or something and then saving it. So, since it's important to maintain an easy way to read/write data for players that may or may not be online, would this be an appropriate instance for using some kind of database? If so how should I start, and what should I use? (SQLite, some kind of YAML thing?). Any input is appreciated!

    Bump

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Jun 1, 2016
  4. Offline

    chris4315

    If it's something big, I'd prefer the use of MySQL databases or JSON.
     
  5. Offline

    savagesun

    Okay.. great. I already use GSON to serialize data. I was thinking I could just save it to some kind of file and stream the data from it when I need to. I'm really not all that certain of how that works. I feel like constantly opening/reading/writing/closing files would be inefficient. Having a database open would allow to read and write without the overhead of opening/closing all the time right? Would serializing the data into GSON and saving it in a database help at all? Or should the data be put into different fields? This is all very confusing.
     
  6. Offline

    Wingzzz

    savagesun
    I use YAML format, very easy and already highly supported with SnakeYaml and Bukkit's YamlConfiguration. I can post a tutorial on it as I don't see too many advanced tutorials for it out (and yet it's still easy although a bit more advanced than the .getConfig() bukkit has). Although like most things, this is situational. Loading a file onEnable(), isn't really much overhead, then access it whenever you wish within runtime. You will barely see a performance issue unless you have an extremely large amount of reading/writing that is needing to be done. If this is the case, then GSON would have the same problem, but it's always situational. With this I would recommend the use of GSON or YAML as they work great and are simple to use. Although if you do begin to experience issues, switch over to an SQL database (MySQL, SQLite- search and compare differences). These sort of require a bit of SQL knowledge for things like statements, but it's super easy to learn, and you can find tutorials here on the forums!

    tl;dr
    GSON and YAML are not bad! An SQL database will out perform, but not for minor - medium calls. Unless you're running a network of servers or just have a very popular server (or your plugins are demanding) then I wouldn't worry. Worst comes to worst, experience some lag, then decide to change to SQL(MySQL/SQLite).

    Good luck! Feel free to pm me for any information on YAML/SQL(not an expert on SQL, but I'm not too shabby) although GSON I haven't dove into just yet.... Then again, I haven't needed to with the use of YAML ;)
     
    savagesun likes this.
  7. Offline

    savagesun

    I have experience with MySQL queries, but not much Bukkit's YAML thing, I'll do a little googling. If I use YAML, I would just open the file and then read/write to it in runtime, then save when necessary (shutdown or some kind of weird exception)?
     
  8. Offline

    Wingzzz

    Save after every change with files, or if there are many calls, have a scheduled task run saves. YAML files are extremely easy to create and manipulate. Let me know if you have any questions about them :)
     
  9. Offline

    savagesun

    Thanks man!
     
Thread Status:
Not open for further replies.

Share This Page