[WEB/ADMIN/API] BukGet: The Unofficial BukkitDev JSON API

Discussion in 'Bukkit Tools' started by SteveMcGrath, Dec 4, 2012.

Thread Status:
Not open for further replies.
  1. Offline

    SteveMcGrath

    BukGet is a API into BukkitDev and aims to alleviate issues with trying to programmatically get updates from BukkitDev and download/manage plugins. Full documentation is available on the bukget,org website (linked below).

    Bukget Main Site: http://bukget.org
    Github Repository: https://github.com/BukGet/bukget
    Development Blog updates: http://chigeek.com/category/bukget.html

    Please keep in mind, we need your help to keep this service online. It's up to you folks to tell us when they think things are breaking. Post here, send a message on IRC, or post an issue ticket on github. If I don't hear any complaints, I assume everything is working as expected. I do not have the time to watch over this every day, so I generally assume everything is a-ok unless people tell me otherwise.
     
    Bertware, lol768 and hawkfalcon like this.
  2. Offline

    Sushi

    Add yaml as an output format, it's our fave.

    Also, how did you write this... flask?
     
  3. Offline

    SteveMcGrath

    I use bottle, not flask. As for yaml, I'll have to see if it'll export to that format without too many changes. Personally JSON makes a lot more sense however as whitespace isn't important, which means you can sent a lot less bits over the wire for every response.

    Here is an example output. it looks pretty messy: http://pastebin.com/561mFLip

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 30, 2016
  4. Offline

    Sushi

    Not really, but it has JSON style dictionaries in it?
     
  5. Offline

    Bertware

    http://api.bukget.org/api2/bukkit/plugins?fields=name,plugname,description
    I'd be nice to have this, but with plugin versions.
    I'd like to be able to quickly look up the current version of a plugin, eventually with the bukkit version.
    Will this API be backwards compatible with v2 (both URL's and output), or will it be hosted on api.bukget.org/api3 for example?
    In general it'd be nice if we can define what excavtly we want to see. e.g. If I want to see name, version, date updated, description, bukkit version, author, I can enter this in a url like the one above, ?fields=...

    Anyway, It's already extremely useful, so keep up the good work :)
     
  6. Offline

    SteveMcGrath

    Understood, and now thanks to using MongoDB as the backend, this is possible by just specifying the fields we want using dot-notation. Example: fields=versions.date,versions.version,versions.download,plugin_name

    This new version will be the first one to be breaking backwards compatibility. I will have a live dev version up once everything is coded for a while to give everyone some time to switch over. This is mainly as the backend is completely different, it would 3 times as long to code up 3 different APIs. I expect this one to be in dev for a while as I want to incorporate a lot of the requests people have been asking for, and I also want to get everyone on the same page. Its been obvious that not pushing everyone to /api2 after it was released was a bad idea. The vast majority of people are still using the rev1 api and stil even using the bukget.org/api redirect I had setup to ease the transition. These were supposed to be temporary hacks to move people into the new frameworks, however it instead allowed people to just continue along without making changes.

    That being said, depending on how BukGet usage trends, I may be forced to push this newer codebase into production earlier than I currently intent to, and in those cases I may write the wrappers.

    That was the main idea of releasing it like that in the first place, however I was limited by what I could do with the current SQL implementation without adding a ton of code. this switch to MongoDB on the backend is opening up a lot of possibilities for the API, however it's also requiring me to shift gears in my head from SQL to mongo, which is just taking some time while I test and play.
     
  7. Offline

    Bertware

    Looks awesome! Well I guess I'll hear when the dev version is ready in this topic :)
     
  8. Offline

    SteveMcGrath

    Yup. Still working out all of the bugs in the parser. It's a slow process that can only be worked out by running several generations on dev.bukkit.org until I can get a clean generation. I'm hoping that gets weeded out in this current generation, however we will only know after a couple of days (it takes that long for a generation with the Curse-compliant limiters in place).
     
  9. Offline

    hawkfalcon

    Awesome, thank you for the logo addition.
    The best thing about the current service in my opinion is dynamic links to the latest version of the plugin. As long as that is maintained, thats awesome:)
    Is there a way you can hook into the pm notification system?
     
  10. Offline

    Bertware

    That's something else I guess. He's getting public data right now, PM's are personal, different for each user.
     
    hawkfalcon likes this.
  11. Offline

    SteveMcGrath

    The question is what are you looking to get out of the PM notification system? I had originally built in a linking system before DBO came into play. The link below is the pyXenForo module I wrote to interface at that time.

    https://github.com/SteveMcGrath/pyxenforo
     
  12. Offline

    hawkfalcon

    I would only be looking at the current amount of bukkitdev pms you have. I am able to do that on the forums through an xhr... but not bukkitdev:)
     
  13. Offline

    SteveMcGrath

    Thats outside the scope of this project.

    Update: Currently reaching page 123 of 360 without issue. As we havent seen any issues yet I have a good feeling this version of the parser may make a full generation. Once thats done I'll start working on the API proper. The parser is the major hurdle as its a complete rewrite and does a lot more than the old one did.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 30, 2016
  14. Offline

    SteveMcGrath

    The generation failed at 161 pages. I made a minor adjustment to the parser, and also wrapped the permissions and commands parsers into try blocks. This should mean a complete generation and we can see how many warning we get from bad plugin.yml files.

    We have a website again! Yay! http://bukget.org

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 30, 2016
    lol768 likes this.
  15. Offline

    Sushi

    >The API is written entirely in Python, a fast & flexible language with a lot of capability.
    >Python, a fast and flexible language
    >Python, a fast
    >Python, __fast__

    >Python
    >implying slow
     
  16. Offline

    SteveMcGrath

    Lets not throw stones now. :-p

    I have yet to see a speed issue in Python with anything I have coded that wasn't artificially imposed. The current API code for example can easily handle 20 million API calls a month on the current hardware without breaking a sweat.

    Currently: 2M calls a month = 1.3 calls a second
    My Benchmarking tool = 12 calls a second.

    The big slowdown there is that the DB (mysql) and the code run on the same box, and that the virtual disk is shared with some of my production lab that cranks away at the same VMFS volume. There is also an ORM in the way as well, so just under a tenth of a second (on average) isn't bad IMHO.
     
  17. Offline

    Sushi

  18. Offline

    SteveMcGrath

    Sushi Honestly that graph is pretty arbitrary. Sure under ideal [and well defined] conditions language X is faster than Y. However, generally speaking in the real world a lot of these benchmarks are just that, benchmarks. Ofcourse a compiled language like C, or a VM-type language like C# or Java will be faster in some cases, however if you want to start saying language X is better than Y, its just like saying my wang is bigger than your wang because of X.

    Python performs pretty well when stacked up to other interpreted languages. Furthermore, the vast majority of slowdowns in a program is the result of programmer error or inefficient code verses language barriers. Every language has its advantages and disadvantages. In my case, I use python for a few reasons:

    1. It has some of the best HTML parsers out there (I heart you BeautifulSoup)
    2. It's more than fast enough for 999 cases out of 1000, and typically when that edge case does happen, your in a much more complex environment at that point.
    3. Python is very flexible. I'm given the latitude to organize the code as I see fit, and prototype code additions as its running. The latter is really important for quickly developing an HTML scraper like the BukkitDev parser.
    4. Python is what I'm most familiar with. While I'm generally pretty pragmatic about what tools I use for a given job, Python has everything I need to get this off the ground and is more than capable.

    So why dont we drop this BS argument here, Python is the language that BukGet is written in, if ya think you can do better, please feel free to do so.
     
    QuantumDev likes this.
  19. Offline

    Sushi

    I didn't say Python is a bad language, because it isn't. I said it's slow compared to other languages.

    Also, I've used a lot of HTML parsers (including BeautifulSoup) and my favorite by far is nokogiri. Nokogiri is a Ruby library. Ruby is an interpreted language that runs at around the same speed as Python. It's great. It has amazing syntax and it's intuitive, but I would also maintain Ruby is slow. Ruby could be my favorite programming language, as a matter of fact.

    Just because I said a language is slow it doesn't mean it's a bad language. I pointed out that it's slow because your website says it's fast.

    The graph isn't arbitrary, Python is just slow.
     
  20. Offline

    SteveMcGrath

  21. Offline

    Sushi

    Actually, they do. Especially when it comes to load testing benchmarks when it comes to webapps and APIs.

    They can tell you how well you fare during a DDOS attack or a large stream of visitors, since they simulate just that.
     
  22. Offline

    SteveMcGrath

    Sure they can, however from my own experiences in those cases typically the bottleneck will be I/O bound and not CPU/process bound, unless the code in question was written by this guy:

    [​IMG]
     
    zack6849 likes this.
  23. Offline

    md_5

    Complete database download is something I would want, even if you say it can be catered for via the api, its much easier to do one request than 7000.
     
  24. Offline

    SteveMcGrath

    md_5 Can you also tell me what you would need a full dump for? What specifically is the API currently not doing that you would like it to do?
     
  25. Offline

    md_5

    Getting all the information in one place, and quickly - much better than visiting 7000 different pages.
    Also it would be useful if you decide to run your own web service using the API and need low latency responses.
     
  26. Offline

    SteveMcGrath

    Again, why would you want to parse every single plugin at once?
     
  27. Offline

    Sushi

    Perhaps speed? A static data resource that can be parsed is faster than scraping html on every call.
     
  28. Offline

    SteveMcGrath

    Sushi The API isn't scraping realtime. It scrapes every 6hrs or so.

    md_5 I don't think anyone is using the API in that way. All of the major sysadmin tools that use BukGet (like MCMA, SpaceBukkit, Multicraft) are all using the API directly and not caching the data for any period of time. Most responses from the API happen in under 100ms, well within tolerable times. I'm hoping that by going to Mongo those times will get even lower. Furthermore, if we can start to get more geographically dispersed parent servers, then round trip latency would drop as well.
     
  29. Offline

    Sushi

    Okay, in that case, a static resource that is local would be faster than the python code and database calls that need to be executed, along with the tcp and network latency overhead.

    The speed difference of mongo and a relational database should be comparable.
     
  30. Offline

    SteveMcGrath

    Sushi, I'll concede on that note, however the original question still remains, what is needed that would require the need to pull the full database instead of simply searching for what is needed?
     
Thread Status:
Not open for further replies.

Share This Page