Solved Problem with File(In/Out)putStream and Object(In/Out)putStream

Discussion in 'Plugin Development' started by MrAwellstein, May 21, 2014.

Thread Status:
Not open for further replies.
  1. Offline

    MrAwellstein

    So I am working on sending a file over the internet through Sockets, and I want to do it with objects because it'll be a lot easier to do it with objects since my entire Server to Client program works off of sending Objects. I've determined (I think) that my main issue is that I am not properly reading the bytes from the file correctly, or more or less, I am not getting the file into a proper byte[][] by buffering. Anyways, could anyone take a look at my code and see what could possibly be wrong, or maybe tell me how to properly get an array of an array of bytes of a file? (Please do not give me other ideas with how to send files, such as using just plain FileIn/OutputStreams with anything other than with Objects).

    Reading the File:
    Code:java
    1.  
    2.  
    3. public void readFile(Socket sock){
    4. try{
    5. FileStart s = new FileStart("C:\\Java" ,"test2.exe");
    6. File f = new File("C:\\Java\\test1.exe");
    7. ObjectOutputStream o = new ObjectOutputStream(sock.getOutputStream());
    8. o.writeObject(s);
    9. byte[] data = new byte[4000];
    10. int length = 0;
    11. while((length = fis.read(data))> 0){
    12. o.writeObject(new FilePart(data, 0, length));
    13. }
    14. o.writeObject(null);
    15. fis.close();
    16. o.close();
    17. }catch(Exception e){}
    18. }
    19.  


    Writing the File:
    Code:java
    1.  
    2. public void createFile(Socket sock){
    3. try{
    4. FileOutputStream fos = null;
    5. ObjectInputStream stream = new ObjectInputStream(sock.getInputStream());
    6. Object o;
    7. while((o = stream.readObject())!= null){
    8. if(o instanceof FileStart){
    9. fos = new FileOutputStream(((FileStart)o).path+File.separator+((FileStart)o).name);
    10. }
    11. if(o instanceof FilePart){
    12. if(fos != null){
    13. FilePart fp = (FilePart) o;
    14. fos.write(fp.data, fp.offset, fp.length);
    15. }
    16. }
    17. }
    18. fos.flush();
    19. fos.close();
    20. stream.close();
    21. sock.close();
    22. }catch(Exception e){System.out.println("Broke");}
    23. }
    24.  


    FileStart.java
    Code:java
    1.  
    2. public class FileStart implements Serializable{
    3.  
    4. public String path;
    5. public String name;
    6.  
    7. public FileStart(String path, String name){
    8. this.path = path;
    9. this.name = name;
    10. }
    11.  
    12. }
    13.  


    FilePart.java
    Code:java
    1.  
    2. public class FilePart implements Serializable{
    3.  
    4. public byte[] data;
    5. public int length;
    6. public int offset;
    7.  
    8. public FilePart(byte[] data,int offset ,int length){
    9. this.data = data;
    10. this.length = length;
    11. this.offset = offset;
    12. }
    13. }
    14.  
     
  2. Offline

    nlthijs48

    MrAwellstein Why do you want to do it with byte arrays? Can't you just use the .writeObject() on the ObjectOutputStream? And then use .readObject() on the other side.

    Am I right that you want to split the message into parts? I don't think you have to do this yourself, the TCP layer of the network card will do that for you. And as the last question, what kind of error do you have? not receiving the object correctly or something else?
     
    NathanWolf likes this.
  3. Offline

    MrAwellstein


    I dont think you understand what java.io.file is.... Anyways data from a file is generally read with byte arrays. I want to use arrays of byte arrays because it would be less stressful on the entire program, and I dont have to do anything extra to the parsing of the objects or the stream of the server or client. In case you didn't read, the only thing I am having problems with is getting the byte arrays from the file (although I should have specifically said that the file being created isn't the original file for some odd reason). When I use a test file, such as Minecraft.exe, I dont get the same file created unless I transfer all of the bytes at once in one object, which I really don't want to do because that could possibly put stress on the ram of the server and client, if the file is too big.

    Edit:
    I don't think that I can do it with just byte arrays as I've tried that, and if I create my own custom object then I can add more features to it as an easy implemented progress bar, or other options.

    I'm not splitting a message, I'm reading a file, buffering it into byte arrays, sending byte arrays through the Socket in a custom Object storing the bytes that implements Serializable, and then writing the bytes to a server.

    The only error I get is that the file created isn't the file I sent. If you want I can post the results of the current program for you to see.
     
  4. Offline

    nlthijs48

    MrAwellstein I understand what the File class does, I used it quite some times already. But with splitting up the message I meant splitting up the thing you want to transfer, sorry that I was not clear, English is not my native language.

    Did you try it with a small file that is only 1 part? Just try with a .txt with 'hello world' in it or something and see if that transfers correctly. Maybe you need to write the fileparts in the opposite order at the recieving side. And why do you set the offset to 0 all the time in the while loop of your readFile() method? That should count up depending on the total length of the previous parts right?
     
  5. Offline

    Everdras

    There's no reason to try to split the file into parts.

    Use these simple steps:

    1. Open the file
    2. Read all its contents into a byte[]
    3. Close the file
    4. Wrap this btye[] in a serializable object
    5. Get the socket's object output stream
    6. Write the object to the stream with writeObject()

    On the server side, just have it readObject() the wrapper, unwrap the byte[], and write it to a file.
     
  6. Offline

    MrAwellstein

    You can't do that with larger files. Imagine something possibly a few gigs. Loading all of that into memory isn't a good idea. I'll probably have to resort to this though...

    My mistake I thought you were implying doing writeObject(File file); which you can do but the bytes won't be sent.

    I tried a one part file and it worked fine

    I'm not entirely sure but I think it should always be 0 because that's saying where the start is and then the length is saying the line to append it on possibly...

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Jun 8, 2016
  7. Offline

    nlthijs48

    MrAwellstein The offset indicates where the part belongs, this should not be set to 0 all the time. If the first part is 4000 bytes then the second part should have an offset of 4000, then it will append it correctly. The third packet should have offset 8000, etcetera. Best is just to add the length of the previous parts together and use that as offset, that also works if not all parts are not the same length. Just try it out and see if that fixes some problems. What your method on the recieving side does right know is just copying the recieved data on the start of the file, overwriting previous bytes. What size is the recieved file in comparison to the file you tried to send?

    I was not trying to say you should send the File object, that is just the path to the file as you also know.
     
  8. Offline

    MrAwellstein

    It will be appending. I'll test it when I get home. But isn't the offset supposed to be each line and not yeah byteset? But I'll still test it when I can.
     
  9. Offline

    MrAwellstein

    I couldn't get it to work. What's weird is when I do the code not through a socket, when I do it with just FileStreams, it works...
     
  10. Offline

    NathanWolf

    I know you immediately shot down nlthijs48 for some reason- but really, why are you not using Object streams? Or are you now (I see it in the title).

    I'm just thinking all of your code ought to boil down to a single read or write call. There is no need to deal with raw data, break up packets, etc.

    You seriously don't have some multi-gigabyte object you want to send, right? I mean if you're sending objects, they're objects- you have to fit them in memory anyway. If you have a multi-gigabyte object, that's not really an object anymore, it's a data store, and you should be sending raw data back and forth. Maybe some use cases would help.

    One quick note on object streams, though. It's not really a great pick for client/server communication because it doesn't handle versioning well (or at all). If you ever add a new field to one of the classes you're serializing, you have to update the client and the server or it'll break. If you plan on distributing clients (like your plugin is the client...) then Object serialization (either doing it yourself or via an object stream) may be a real bad idea for forward and backward compatibility.

    As much as I hate to say it, you could use YamlConfiguration to do the serialization, if you wanted. A JSON library might be a better pick though (side-note ... can Bukkit please switch to json now?? XD)
     
    nlthijs48 likes this.
  11. Offline

    MrAwellstein

    I am using them. I shot him down because what he suggested didn't work.

    I want to deal with break ups because it would help with bigger files, and possibly with time. I could also create an easy way to track the progress of the file being sent if its in parts.

    I would be sending raw data, but I picked Objects because there are some advantages to them.

    Versioning can be done with just creating new objects. The backwards compatibility part would just allow the users to use the same old client, but they wouldn't have any new features, just like how normal updates are. Using older packets should be fine as long as I dont have any severe issues, and with Serialized Objects, its quite easy to deal with simple data. If I was doing it with anything else, so to say just raw bytes or plantext or anything else, I would probably have to deal with the same exact thing, and would have to update the entire parser of both the client and the server, leaving less room for backwards compatibility. The only downside I can think of, when using objects, is that it takes a little bit longer to send (possibly 1-5 ms more, depending on the size). Also I can't think of any normal Server to Client program that has done a major update with the networking aspect and hasn't needed an update of both Server and Client (when I mean major I mean change in the protocal/packets).


    YamlConfiguration Serialization vs regular Java Serialization...
    I'm not entirely sure, but the Java Serialization was more meant to be used for transmitting objects through sockets, and Yaml was more meant for Files (the reason why its so easily readable and meant to be user friendly).
    Also Yaml is a bit heaver than Java Serialization, (I checked doing my own test, storing two objects into files, with 12 bytes of data, using both Yaml and regular Serialization).

    ----
    I've done the test and here's what happens. Below is a proper way of copying a file. It works perfectly fine. The file created is valid and an exact copy. But, when I store the byte[] into an object, something gets mixed up, the first 4000 bites of the file is exactly the same, but when I scroll down things start to get out of order I think. I'm going to do more test.

    Code:java
    1.  
    2. FileInputStream fis = new FileInputStream("C:\\Java\\test1.exe");
    3. FileOutputStream fos = new FileOutputStream("C:\\Java\\test2.exe");
    4. int count=0;
    5. byte[] b = new byte[4000];
    6. while((count = fis.read(b))>0){
    7. FilePart p = new FilePart(b, 0, count);
    8. fos.write(p.buffer,p.offset,p.count);
    9. }
    10. fis.close();
    11. fos.close();
    12.  
     
  12. Offline

    NathanWolf


    Ok, are you copying files/data or objects?

    If you're just trying to move data around you shouldn't be breaking anything up or wrapping anything- this is just stream piping. Use buffered streams, trust the underlying code and TCP stack to partition data appropriately - you don't need to load the whole file into memory to send it.

    There is a big difference between "I want to make a client/server communication protocol" and "I want to copy a file from one server to another". It's still not really clear (to me, anyway) what you're trying to do here.

    So (and this is assuming you are making a real protocol with real objects, not just wrapping wrap data in an Object wrapper for some reason) - every time you add a new field to your protocol, you're going to version the object? And the server is going to have to be able to appropriately deal with every version of every object ever made, so it can support any client that may be out there?

    Sounds like a nightmare to me, but I may be failing to communicate the issue properly.
     
    nlthijs48 likes this.
  13. Offline

    MrAwellstein

    For those who are interested, I solved this. It was because the value of the read Byte wasn't being copied, and the object was just being reused, which caused it to be the same byte over and over. I fixed this with Byte.clone();

    Code:java
    1.  
    2. public static class FilePart implements Serializable{
    3. /**
    4.   *
    5.   */
    6. private static final long serialVersionUID = -3136972845486999599L;
    7. public byte[] buffer;
    8. public int offset;
    9. public int count;
    10.  
    11. public FilePart(byte[] b, int o, int c){
    12. this.buffer = b.clone();
    13. this.offset = o;
    14. this.count = c;
    15. }
    16. }
    17.  
    18. }
    19.  


    I know I don't need to load the entire file into memory to send it, that's why I'm buffering it into parts. I've already created my client<->server protocal.


    It's not actually a nightmare. The new objects just are extended from the old objects, unless something extremely major needs to be done which would mean that it would be a completely major update (such as v.1 to v.2). When you extend the objects, you use "instanceof" (which I am already doing to parse the different objects), and then a simple version check is done by another object parser doing (instanceof object1 or instanceof object2). When the major versions hit, such as a full new release (like in mc going 1.6.x to going 1.7.x) that's when the older objects are removed if needed, or a legacy mode would be put in). Anyways the issue was solved ^_^

    EDIT by Moderator: merged posts, please use the edit button instead of double posting.
     
    Last edited by a moderator: Jun 8, 2016
Thread Status:
Not open for further replies.

Share This Page