Posts Comments

Home > Programming > Java > File copy in Java – Benchmark

File copy in Java – Benchmark

August 2, 2010 By 45 Comments

Conclusion

In conclusion, the NIO Transfer method is the best one for big files but it’s not the fastest for little files (< 5 MB). But the custom buffer strategy (and the NIO Buffer too) are also really fast methods to copy files. We’ve also see that the method using the native utility tools to make the copy is faster as NIO for big files (< 1 GB) but it’s really slow for little files because of the cost of invoking an external program.

So perhaps, the best method is a method that make a custom buffer strategy on the little files and a NIO Transfer on the big ones and perhaps use the native executable on the really bigger ones. But it will be interesting to also make the tests on an other computer and operating system.

We can take several rules from this benchmark :

  1. Never made a copy of file byte by byte (or char by char)
  2. Prefer a buffer in your side more than in the stream to make less invocations of the read method, but don’t forget the buffer in the side of the streams
  3. Pay attention to the size of the buffers
  4. Don’t use char conversion if you only need to tranfer the content of a file, so don’t use Reader if you need only streams.
  5. Don’t hesitate to use channels to make file transfer, it’s the fastest way to make a file transfer.
  6. Consider the native executable invocation only for really bigger files.
  7. The new Path method of Java 7 is really fast except for the transfer of an enormous file between two disks.

I hope this benchmark (and its results) interested you.

Here are the sources of the benchmark : File Copy Benchmark Version 3

Here are the informations complete for the benchmark between two disks : Complete results of first two benchmarks

Related posts:

  1. Java File Copy Benchmarks Update
  2. Java File Copy Benchmark Updates (once again)
  3. Generate graphs benchmarks easily
  4. Java Synchronization (Mutual Exclusion) Benchmark
  5. My Java Benchmarks on GitHub

Pages: 1 2 3 4 5

Filed Under: Java Tagged With: , , ,
  • Peter

    Hi, interesting post, thanks! Please, could you post, for a comparison, the time needed to copy the same files with the native copy utility (eg. copy/xcopy on Windows, cp on Unix). I am just wondering how Java compares with low-level copy.
    I remember that some years ago I figured out that the fastest way to transfer large files (> 1Gb) was to invoke the native copy command (copy.exe).

    • Baptiste Wicht

      Yes, I can try to add it to bench, but it takes a long time to make the bench. I’ll try to post the results tomorrow.

  • Peter

    Hi, interesting post, thanks! Please, could you post, for a comparison, the time needed to copy the same files with the native copy utility (eg. copy/xcopy on Windows, cp on Unix). I am just wondering how Java compares with low-level copy.
    I remember that some years ago I figured out that the fastest way to transfer large files (> 1Gb) was to invoke the native copy command (copy.exe).

    • Baptiste Wicht

      Yes, I can try to add it to bench, but it takes a long time to make the bench. I’ll try to post the results tomorrow.

  • http://www.tracemodeler.com Yanic

    It might be interesting to add an entry for the maximum throughput you can achieve on that SATA hard disk of yours.. Just to see how close the results are to the maximum achievable score.

    Couple of things caught my eye :

    The NIO transfer of a 50MB text file takes 400ms, while a binary file of the same size requires +- 450ms?

    Judging from the 50MB binary file benchmark, NIO transfer has a throughput of about 111 MB/s (50MB / 0.450s) in your test. In the test for the 1.3GB binary file the throughput is about 55MB (1331MB / 24s). Why such a big difference?

    Any ideas on the above?

    • Baptiste Wicht

      On my PC, I cannot benchmark the Disk for Write throughput. If you have a way to make a simple read/write bench on hard disk on Ubuntu, you are welcome.

      For the huge difference, I don’t know. Perhaps, the OS can make more efficient write on little files. I don’t know.

  • http://www.tracemodeler.com Yanic

    It might be interesting to add an entry for the maximum throughput you can achieve on that SATA hard disk of yours.. Just to see how close the results are to the maximum achievable score.

    Couple of things caught my eye :

    The NIO transfer of a 50MB text file takes 400ms, while a binary file of the same size requires +- 450ms?

    Judging from the 50MB binary file benchmark, NIO transfer has a throughput of about 111 MB/s (50MB / 0.450s) in your test. In the test for the 1.3GB binary file the throughput is about 55MB (1331MB / 24s). Why such a big difference?

    Any ideas on the above?

    • Baptiste Wicht

      On my PC, I cannot benchmark the Disk for Write throughput. If you have a way to make a simple read/write bench on hard disk on Ubuntu, you are welcome.

      For the huge difference, I don’t know. Perhaps, the OS can make more efficient write on little files. I don’t know.

  • Craig

    I noticed that the benchmarking framework you are using collects statistics. Can you publish the confidence intervals around your results so we can see which benchmark differences are statistically significant? TIA

    • Baptiste Wicht

      Yes, I will publish them tomorrow. I will launch the benchmark tonigh with new bench :

      First phase on the same disks
      Second phase between two disks
      Add a method that just invokes /bin/cp to make the copy

      • Peter

        Please consider the operating system and the hard disk cache when running such benchmarks.
        After you read a file the first time, the second read will be faster if still in cache.
        It would be good to empty the cache before each test (I don’t know how that could be done with the hard disk cache without a system reboot).

        • Baptiste Wicht

          I know there is a file system cache and a hard disk cache, but I don’t think they are invalidating the results.

          The cache are for all the different methods. So OK, the first tests will be slower than the other, but the benchmarking library I use make a JVM Warmup executing the taks several times, so the caches will also be loaded (normally) at this time. And all the benchmarks will benefit of the cache. No ?

          • Peter

            You are right, the warmup will fill the cache.

      • Baptiste Wicht

        The new results are not online. Feel free to comment them.

        I hope this is better ?

  • Craig

    I noticed that the benchmarking framework you are using collects statistics. Can you publish the confidence intervals around your results so we can see which benchmark differences are statistically significant? TIA

    • Baptiste Wicht

      Yes, I will publish them tomorrow. I will launch the benchmark tonigh with new bench :

      • First phase on the same disks
      • Second phase between two disks
      • Add a method that just invokes /bin/cp to make the copy
      • Peter

        Please consider the operating system and the hard disk cache when running such benchmarks.
        After you read a file the first time, the second read will be faster if still in cache.
        It would be good to empty the cache before each test (I don’t know how that could be done with the hard disk cache without a system reboot).

        • Baptiste Wicht

          I know there is a file system cache and a hard disk cache, but I don’t think they are invalidating the results.

          The cache are for all the different methods. So OK, the first tests will be slower than the other, but the benchmarking library I use make a JVM Warmup executing the taks several times, so the caches will also be loaded (normally) at this time. And all the benchmarks will benefit of the cache. No ?

          • Peter

            You are right, the warmup will fill the cache.

      • Baptiste Wicht

        The new results are not online. Feel free to comment them.

        I hope this is better ?

  • Kees

    You should also record the CPU-usage.
    nio will surpass CPU if it can.

    • Baptiste Wicht

      Sorry, but sadly, the framework I use does not collect CPU Usage data.

  • Kees

    You should also record the CPU-usage.
    nio will surpass CPU if it can.

    • Baptiste Wicht

      Sorry, but sadly, the framework I use does not collect CPU Usage data.

  • http://javamexico.org Enrique

    It would be interesting to use bigger buffers. 4096 bytes is not much; When I’ve had to copy bytes between files or sockets or whatever I use buffers between 16384 and 65536 bytes. A 64K buffer might make a big difference compared to the 4K buffer, because there will be far fewer reads.

    • Baptiste Wicht

      Of course, the results will vary using different buffer sizes as said in the conclusion, but I cannot made all the the benchmarks with several buffer sizes. The actual benchmarks takes several hours to finish. But I will try to start some more tests this week-end including some other buffer sizes.

  • http://javamexico.org Enrique

    It would be interesting to use bigger buffers. 4096 bytes is not much; When I’ve had to copy bytes between files or sockets or whatever I use buffers between 16384 and 65536 bytes. A 64K buffer might make a big difference compared to the 4K buffer, because there will be far fewer reads.

    • Baptiste Wicht

      Of course, the results will vary using different buffer sizes as said in the conclusion, but I cannot made all the the benchmarks with several buffer sizes. The actual benchmarks takes several hours to finish. But I will try to start some more tests this week-end including some other buffer sizes.

  • Alan

    Benchmarking file copying is very hard and I assume all these tests are actually run with the source file in the file cache. To do a more complete test probably would require dropping the cache before each copy. Maybe copy hundreds of different files of the same size and reboot/drop cache before running each test?

    In jdk7 there is java.nio.file.Path with a copyTo method that is a real file copy. When I say “real file copy” then it does a bit more than just copy file contents. It can copy directories, sym links, and other types of files too. It knows not to override the target file by default, knows can copy meta-data (file timestamps, file permissions, ACLs, etc.), and more. It might be good to include this in your tests. The method should make use of the best mechanism on each platform.

    A couple of comments on the code in FileCopyBenchmark. In nioBufferCopy the return from out.write(buffer) isn’t checked. Short writes are possible so this code should be: while (buffer.hasRemaining()) { out.write(buffer); }. As the method is just copying bytes then you may find that using a direct buffer is faster – try changed the ByteBuffer.allocate to ByteBuffer.allocateDirect. It looks like nioTransferCopy has the same issue as nioBufferCopy – you need to check the return value from transferTo.

    • Baptiste Wicht

      Thanks for your interesting reply :)

      I know that the cache can change the results, but I cannot really reboot before each test, it’s already enough long to run all tests without being aside my computer to reboot it before each test.

      I know there is a new version of file copy in Java 7, but at the time of the benchmark, I had no JDK7 installed. The next time I will run the tests, I will include it.

      Thanks for the corrections, I didn’t know that. I will change that for the next run of the tests. I will run all that tests after my vacation in one week and an half. So if somebody else has request for tests, make them before that point.

      • Baptiste Wicht

        The benchmarks have been updated :)

        I’ve made some corrections on the code (Thanks to Alan) and included the Path Java 7 version

  • Alan

    Benchmarking file copying is very hard and I assume all these tests are actually run with the source file in the file cache. To do a more complete test probably would require dropping the cache before each copy. Maybe copy hundreds of different files of the same size and reboot/drop cache before running each test?

    In jdk7 there is java.nio.file.Path with a copyTo method that is a real file copy. When I say “real file copy” then it does a bit more than just copy file contents. It can copy directories, sym links, and other types of files too. It knows not to override the target file by default, knows can copy meta-data (file timestamps, file permissions, ACLs, etc.), and more. It might be good to include this in your tests. The method should make use of the best mechanism on each platform.

    A couple of comments on the code in FileCopyBenchmark. In nioBufferCopy the return from out.write(buffer) isn’t checked. Short writes are possible so this code should be: while (buffer.hasRemaining()) { out.write(buffer); }. As the method is just copying bytes then you may find that using a direct buffer is faster – try changed the ByteBuffer.allocate to ByteBuffer.allocateDirect. It looks like nioTransferCopy has the same issue as nioBufferCopy – you need to check the return value from transferTo.

    • Baptiste Wicht

      Thanks for your interesting reply :)

      I know that the cache can change the results, but I cannot really reboot before each test, it’s already enough long to run all tests without being aside my computer to reboot it before each test.

      I know there is a new version of file copy in Java 7, but at the time of the benchmark, I had no JDK7 installed. The next time I will run the tests, I will include it.

      Thanks for the corrections, I didn’t know that. I will change that for the next run of the tests. I will run all that tests after my vacation in one week and an half. So if somebody else has request for tests, make them before that point.

      • Baptiste Wicht

        The benchmarks have been updated :)

        I’ve made some corrections on the code (Thanks to Alan) and included the Path Java 7 version

  • Ronald

    To understand NIO you should learn about DMA, http://en.wikipedia.org/wiki/Direct_memory_access. It doesn’t make IO faster, but it offloads the copying of bytes from the CPU. That is why the channels need a ByteBuffer. It is a memory buffer to which the DMA-chip copies the data. It can also copy the data directly from disk to network or from disk to disk as long as the sending and receiving device both understand DMA.
    But as long as the CPU or the speed of your memory is not a bottleneck there is nothing which gets faster.

    • Anonymous

      Hi Ronald,

      I know what is DMA, but I didn’t know that NIO used it.

      Thanks for the information.

      Baptiste

  • http://c-o-f.tumblr.com/ Zammbi

    I wouldn’t mind seeing what the speed of renameTo method is even though its not a copy.

  • http://www.escortsmeet.com/ escorts

    If your language is Assembly or Python like, you could benefit from creating a list of strings, where one string is a copy of one line. …

  • vl

    I use KillCopy to transfer my files and folders to other locations. It is a powerful copy manager with job resuming and secure file erasing features.

    http://bstdownload.com/reviews/killcopy-2/

  • Vicne

    Very interesting posts, and thanks for sharing the source

    A few thoughts :

    You were asking for comments regarding same disk vs different disk difference. I guess your disks are not SSD, so when working on the same disk, the heads have to physically move back and forth between the areas to read from and write to, which is much slower than linear read on one disk and linear write on another. Same disk copying benefits highly from big memory buffers of course.
    I also noticed that native method seems to be slower than any java method for small files, which is a nonsense of course. I guess the reason is that the overhead due to creating an environment to run a native process from java (Runtime.getRuntime().exec()) is predominant in this case. This influence could be reduced by performing a “cp src/* dst/” on a folder containing a thousand of small files.
    Anyway, I’m more interested in large files (20GB+) across network shares so I’ll try to adapt your source to this case.

    Best regards,

    Vicne

    • Anonymous

      Hi Vicne,

      Yes, none of my disk were a SSD. You’re right that working on the same is slower than with two disks, but based on my benchmark results, it doesn’t look much slower as we should think.

      You’re absolutely right about the native method. For very little files, it’s only slower because of the overhead from Java invocation of a command.
      If you run this kind of benchmarks on your own, I’m very interested of the results.

      Thanks

      Best Regards

      Baptiste

    • Niharika

      Hi Vicne .Are you able to find any good solution for large files(20GB+)
      I am working on the same thing.please reply .

  • ab0032

    I think your effort is great, but to really test things, especially for small files, is that it is not enough to measure the time to copy one file, but to measure how things behave it you cp small medium and large files for say 1000 sec and see how much you get done. 

    • ab0032

      The reason is, that now all the different things you tested behave relativly equal, which is due to the fact, that your hard disk is probably the bottleneck, and it doesnt matter how the java copies things, the cpu is idle and it doesnt matter if it copies small files around one or ten times. But what if you have an application, that really needs the CPU? Then coping around buffers in Java is a bad idea, because it takes time and produces garbage that has to be collected. Hence even if your results show that actually reading and writing is as fast as native io, it can still be a good idea to use native IO.

      • wichtounet

        You’re right, that could be interesting to add to the benchmark the throughput of each technique. And yes, again, depending on the situation, there are cases were a method should be avoided and another preferred. 

        I do not have time for now to improve this benchmark to test more things, but I will try to do it in the future.