World Backup and Mapping Utility (windows only)

Discussion in 'Bukkit Tools' started by jasonznack, Feb 8, 2011.

Thread Status:
Not open for further replies.
  1. Offline

    instinx

    Greetz Dateranoth... FYI I am working on a fork to the script that eliminates the mapping tools, and handles the multiple worlds properly. I may also go a different route altogether and build it into a bukkit plugin but it may not be needed. Glad to see you are still supporting it.
     
  2. Offline

    Dateranoth

    The multiple worlds are supported as far as backing up unless you want them backed up to their own archives. The current revision can backup the entire Server directory and thus the worlds with it. The mapping stuff can all be disabled. It's pretty bloated when used simply, but it can be used just to backup. However, if you make a bukkit plugin that would be great. I would like to see a decent plugin for backups. The biggest downfall mine has is you have no access from minecraft and the saveall.vbs which has to be ran in order to force the world to save.

    Anyhow, hope some of it can be useful to you :) Glad it's still floating out there and getting used.
     
  3. Offline

    Phaedrus

    It's easy to support multiple worlds already, just run an instance of the batch file for each world, and another just for the server files if you want.
     
  4. Offline

    Dateranoth

  5. Offline

    Phaedrus

    @Dateranoth
    I'm running into an issue with incremental backups failing. The full backup is working however. I noticed this recently when I unfortunately needed to restore from a backup only to discover that the most recent backup was 12 hours old. I watched the backup process fly by and noticed a system error msg flash by. So I logged the output of the backup process overnight.

    It looks like it's running out of disk space, which is odd, because the target disk has 400+ gigs available. The server runs on a ramdisk though, which only has 256 megs, and 50 megs free. I think this might be the issue. I've stopped using incrementals for now. Is this something that can be fixed, or do I just need to reserve a certain amount of overhead room on the ramdisk?

    Here is my backup script: http://pastebin.com/csrmCF73
    and here is the error log: http://pastebin.com/tJv37H2y

    Thanks
     
  6. Offline

    Dateranoth

    Sorry to hear you're having trouble. The way the temporary backup works is it updates the IncrMaster in the cache directory and then compresses those for an incremental file in your backup folder. The cached IncrMaster has to be there so there is something to update against. 7zip is SUPPOSE to be creating a temporary IncrMaster in the same directory as your cache when it's updating, but the best I can figure is it's not in your case. I'm rolling out a patch with a fix in the next 15 or 20 minutes.

    I will update here when I get done and you can let me know if it fixes the problem or not
    --- merged: Mar 5, 2011 8:22 PM ---
    Ok. took longer than I thought. The latest version is up. Give it a try and let me know if you're still having problems.

    ::V2.9-Updated Incremental backup code for 7zip. It should have been placing the-------::
    ::-----temporary file when updating the backup in the same directory but it wasn't.-----::
    ::-----Added command line to force temp files to goto backup directory so people running::
    ::-----on RAMDisks or low disk space won't encounter errors.----------------------------::
    ::-----INFORMATION UPDATE: PLEASE READ: Regards Backuping up Entire Server and Python---::
    ::-----If backing up your entire server directory you CANNOT place the backup location--::
    ::-----inside your Server Directory. This creates a loop that xcopy won't allow and----::
    ::-----thus fails to create the backup.-------------------------------------------------::
    ::-----Second, if you use overviewer and Python you MUST have your Python Directory in--::
    ::-----your windows PATH environment. Instructions can be found here:------------------::
    ::-----http://docs.python.org/faq/windows.html------------------------------------------::
    --- merged: Mar 5, 2011 8:28 PM ---
    Almost forgot, the saveall.vbs was updated as well. Just rearranged a little bit to make sure the save-all command gets through.
     
  7. Offline

    Phaedrus

    Great, I'll try it out tonight. Thanks for the quick update!
    --- merged: Mar 7, 2011 1:46 AM ---
    Incrementals are working properly once again. Thanks!
     
  8. Offline

    Dateranoth

    Great. I'm glad that did the trick. Not sure why 7zip wasn't doing it by default. Oh well, works now :)
     
  9. Offline

    Dateranoth

    ::V3.0-Now supporting retrieval and backup of a remote server via ftp. If you are------::
    ::-----using a server host such as xenonservers.com and you need a way to backup your---::
    ::-----server and run the mapping tools you now have a solution. As long as you have a-::
    ::-----machine that can run this script you can pull the remote server files, via ftp---::
    ::-----run the backup and mapping, and upload the map to the webserver. The process----::
    ::-----treats the remote files just like you were hosting it locally and supports all---::
    ::-----features EXCEPT save-all. You will need to run a plugin such as SimpleCronClone-::
    ::-----to force save-all ever X minutes ahead of when this script is going to run-------::
    ::-----My latest release will include the two .scc scripts needed to do this. You will-::
    ::-----need to download and install the plugin however.---------------------------------::
    ::-----The option to do remote retrieval can be found under FTP Options-----------------::
    ::-----The only other change is that the xcopy exclude file MUST have the full path-----::
    ::-----if being used. Example below.---------------------------------------------------::

    V3.0 Supports Backing up Remote servers. Do you have a server host such as xenonservers.com with no way to backup your files or run mapping software? Now you can. Simply follow the directions below and the script will connect to your server via FTP, download the files and then run the backup and mapping process like you were running the server on your own machine. Not the best solution, but it's better than nothing. If using this for remote server backup, saveall should be disabled and an alternative method such as SimpleCronClone should be used.

    If performing a REMOTE Server Backup Make sure the variable GetRemoteServerFiles is set to yes and all the Retrieve Server Files from FTP Options have been properly set. Also, make sure WinSCP is installed and the path is set in your variables.
    Also, you will not use the VBS Script when backing up a remote server because it will have no effect. Instead you will want to install SimpleCronClone and using the two scripts I included have it run save-all every X minutes. You will want to force your server to save a few minutes more often then you backup. So, if you are backing up every 10 minutes, then force save every 7. It's not the best solution, but the only one available for remote backup atm.
     
  10. Offline

    h0us3cat

    suggestion:
    Add option to render night mode from overviewer?
     
  11. Offline

    Dateranoth

    If you are just wanting to always render in night mode then you can change this line:

    Code:
    Python gmap.py --cachedir="%OVWRCACHEDIR%" -p "%Procs%" "%TempWorldDir%" "%OVWROUTDIR%"
    to this

    Code:
    Python gmap.py --night --cachedir="%OVWRCACHEDIR%" -p "%Procs%" "%TempWorldDir%" "%OVWROUTDIR%"
    Next time I update I will add a variable field for entering the options in Overviewer.
     
  12. Offline

    drakhri

    Rather than using sleep.exe or ping, you can use timeout /T (time in seconds) /NOBREAK. Comes with win7.
     
  13. Offline

    Dateranoth

    Appreciate the heads up. I started this script on a windows XP machine. I don't think it had another method other than ping, but I might be wrong.
     
  14. Offline

    h0us3cat

    I want to render in all 3 modes,
    this is what i use in a bat script i execute manual.

    Code:
    overviewer.exe -p 1 "D:\world" "C:\output"
    overviewer.exe -p 1 --rendermodes=lighting "D:\world" "C:\output"
    overviewer.exe -p 1 --rendermodes=night "D:\world" "C:\output"
     
  15. Offline

    Dateranoth

    I actually just updated to the new DTT version of Overviewer today. I'll be updating the script tonight or tomorrow and I'll definitely add in a parameter for the changing modes ( and all the options ) and I'll see about doing more than one overviewer pass each run.
     
  16. Offline

    Dateranoth

    Ok, the latest version is out. Give it a try and let me know how it works for you. Overviewer DTT natively supports multiple pass rendering. In the never version just add this to the overviewer OPTIONS variable: --rendermodes=lighting,night etc.


    V3.1 is out and has adopted the Direct to Tile version of Overviewer. Support for older versions have been dropped though they can still be used. If you wish to keep the older versions you will need to rename gmap.py or gmap.exe to overviewer.py or overviewer.exe . You will then need to change the overviewer options to comply with the older version. I highly suggest updating to the DTT version as it is much faster and doesn't require the first caching stage. It also allows you to render more than one mode at a time. Refer to the settings for more information. There is now an Overviewer OPTIONS setting allowing you to customize the commands. Finally, support for sessions in WinSCP has been added. This is only necessary if you are Backing up from a Remote Server who's timezone is different from yours.

    ::V3.1-The older versions of Overviewer have been dropped. Only the Direct to Tile-----::
    ::-----version is supported now. It can be found at the link below---------------------::
    ::-----https://github.com/brownan/Minecraft-Overviewer/tree/dtt-c-render----------------::
    ::-----If you are using an older version of Overviewer it will still work but you'll----::
    ::-----need to change gmap.py or gmap.exe to overviewer.py or overviewer.exe.-----------::
    ::-----You will also have to change the OVWROPTION sections below to comply with the old::
    ::-----Overviewer settings.-------------------------------------------------------------::
    ::-----Finally, I have added an option to the Remote Server Download. This option------::
    ::-----will allow you to open sessions saved in WinSCP, which is useful for syncing the-::
    ::-----server time zone with your local computer. If you do not then the files will not-::
    ::-----sync properly. Refer to the WinSCP website for instructions on the time offset.-::

    If you Downloaded this in the last 35 minutes please redownload. I had implemented something unnecessary and removed it.

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: May 8, 2016
  17. Offline

    h0us3cat

    On first time it does the job backup/overviewer.
    But then when it starts the 2nd time backup/viewer,
    I get a error "file not found" or something. Its scrolls too fast to read.
    it creates a 1kb.7z and then goes back to the countdown for next run.
     
  18. Offline

    Dateranoth

    Can you post your user variables section? Omit out usernames/passwords. Also, there should have been a file with it called MCDebug.bat, can you place that in the same folder as MCBackup.bat and run it. Let it get to the error section and then post the log file it creates.
     
  19. Offline

    h0us3cat

    I fixed it, my "exclude.txt" was not in full path. just weird it works on first run :/
     
  20. Offline

    Dateranoth

    I think it was because I was lazy. When I initially wrote the batch I changed directories in the batch instead of opening the files using full path. So, on the first pass through the batch is running xcopy from the same location as your exclude.txt . On the next pass it's probably changed to a different directory and running xcopy from a location that is different from your exclude.txt
     
  21. Offline

    h0us3cat

    its ok ;)
     
  22. Offline

    Phaedrus

  23. Offline

    Dateranoth

    No current plans as of this moment. Everything has been working well so I haven't thought of any changes. I'm open to suggestions though.
     
  24. Offline

    Phaedrus

    Thing's have been running really smoothly here. I've been running 4 instances of the batch file. One for each of the 3 worlds, and another for just the server and plugin files. The only thing I can think of adding might be some additional things to overviewer, such as the biome color extractor. It would be cool if it could be run before each render so any new chunks that are generated will have the proper biome colours.

    http://www.minecraftforum.net/topic...e-extractor-add-biome-support-to-your-mapper/

    Right now I use this manually every once in a while to refresh the biome data.

    Code:
    @ ECHO OFF
    TITLE BiomeExtractor
    START java -jar MinecraftBiomeExtractor.jar -nogui R:\craftbukkit\survivalworld -flush
    START java -jar MinecraftBiomeExtractor.jar -nogui R:\craftbukkit\freebuildworld -flush
    EXIT
    
    I've also experimented with PNGOUT to reduce the map data size a bit. It works good, about 20% overall. I ended up saving a couple hundred megs, which is nothing to sneeze at, but takes forever cause it can only work on a single file at a time. I broke it down a bit to work on each of overviewers 4 folders in parallel, and that helped, but still takes a couple hours. So by the time it finished, overviewer has already generated some new tiles if something has changed. you'd have to run it almost constantly depeding on how big your world is and how often its updated, or have a lot more cores to work with. I'm using a Core i7 with 8 threads runnning at 4.8 ghz.

    I used a starter bat to launch the 4 seperate pngout processes for each world.

    Code:
    @ ECHO OFF
    TITLE STARTING PNGOUT
    ECHO Starting PNGOUT crushing of overviewer map data...
    start pngout_freebuild_0.bat
    start pngout_freebuild_1.bat
    start pngout_freebuild_2.bat
    start pngout_freebuild_3.bat
    start pngout_survival_0.bat
    start pngout_survival_1.bat
    start pngout_survival_2.bat
    start pngout_survival_3.bat
    EXIT
    
    and each instance is a variation of this...
    Code:
    @ ECHO OFF
    TITLE PNGOUT of Survival 0
    ECHO Crushing PNGs for Survival folder 0
    FOR /R C:\minecraft\Overviewer\Maps\survival\lighting\0 %%v IN (*.png) DO pngout %%v /y
    EXIT
    
    There's probably a better way to do that, but I'm not sure what it is...

    Anyway, just thought I'd share what I've been working on.
     
  25. Offline

    Dateranoth

    I like what you have setup. I had thought about including the biome extractor, but the last time I had looked into it the only one I found specifically stated not to run the extractor while minecraft was running. That was a few revisions ago, and now everything is moved first anyhow. I'll have to look into adding it. I had also thought of adding multiple world support, but at the moment it is easier to run multiple instances.
     
  26. Offline

    Phaedrus

    I think the BiomeExtractor might be currently broken with 1.6. I haven't upgraded yet to test it.
     
  27. Offline

    h0us3cat

    Multiworld for overviewer would be nice.
     
  28. Offline

    Phaedrus

    Run a separate batch file instance for each world. You get the benefit of more modular world backups as well.
     
  29. Offline

    Reid458

    I've got a bit of a problem. It seems to be working fine, and I see the 'world' folder copy into the directory I specified, but once the backup is completed and it starts counting down to the next backup, the world file it just copied disappears... D:
     
  30. Offline

    Phaedrus

    Do you mean the zip archive it creates?

    Can you post some more details about how you have it set up? Maybe even post the relevent config section of the batch file.
     
Thread Status:
Not open for further replies.

Share This Page