File copy in Java – Benchmark

August 2, 2010 by Baptiste Wicht 46 Comments

Yesterday I wondered if the copyFile method in JTheque Utils was the best method or if I need to change. So I decided to do a benchmark.

So I searched all the methods to copy a File in Java, even the bad methods and found 5 methods :

Native Copy : Make the copy using the cp executable of Linux
Naive Streams Copy : Open two streams, one to read, one to write and transfer the content byte by byte.
Naive Readers Copy : Open two readers, one to read, one to write and transfer the content character by character.
Buffered Streams Copy : Same as the first but using buffered streams instead of simple streams.
Buffered Readers Copy : Same as the second but using buffered readers instead of simple readers.
Custom Buffer Stream Copy : Same as the first but reading the file not byte by byte but using a simple byte array as buffer.
Custom Buffer Reader Copy : Same as the fifth but using a Reader instead of a stream.
Custom Buffer Buffered Stream Copy : Same as the fifth but using buffered streams.
Custom Buffer Buffered Reader Copy : Same as the sixth but using buffered readers.
NIO Buffer Copy : Using NIO Channel and using a ByteBuffer to make the transfer.
NIO Transfer Copy : Using NIO Channel and direct transfer from one channel to other.
Path (Java 7) Copy : Using the Path class of Java 7 and its method copyTo()

I think, this is the principal methods to copy a file to another file. The different methods are available at the end of the post. Pay attention that the methods with Readers only works with text files because Readers are using character by character reading so it doesn’t work on a binary file like an image. Here I used a buffer size of 4096 bytes. Of course, use a higher value improve the performances of custom buffer strategies.

For the benchmark, I made the tests using different files.

Little file (5 KB)
Medium file (50 KB)
Big file (5 MB)
Fat file (50 MB)
And an enormous file (1.3 GB) only binary

And I made the tests first using text files and then using binary files. I made the tests using in three modes :

On the same hard disk. It’s an IDE Hard Disk of 250 GB with 8 MB of cache. It’s formatted in Ext4.
Between two disk. I used the first disk and an other SATA Hard Disk of 250 GB with 16 MB of cache. It’s formatted in Ext4.
Between two disk. I used the first disk and an other SATA Hard Disk of 1 TB with 32 MB of cache. It’s formatted using NTFS.

I used a benchmark framework, described here, to make the tests of all the methods. The tests have been made on my personal computer (Ubuntu 10.04 64 bits, Intel Core 2 Duo 3.16 GHz, 6 Go DDR2, SATA Hard Disks). The Java version used is a Java 7 64 bits Virtual Machine.

I’ve cut the post into several pages due to the length of the post :

Introduction about the benchmark
Benchmark on the same disk
Benchmark between Ext4 and Ext4
Benchmark between Ext4 and NTFS
Conclusions about the benchmark results

Filed under: Java Tagged with Benchmarks, I/O, Java, Performances

Peter

Hi, interesting post, thanks! Please, could you post, for a comparison, the time needed to copy the same files with the native copy utility (eg. copy/xcopy on Windows, cp on Unix). I am just wondering how Java compares with low-level copy.
I remember that some years ago I figured out that the fastest way to transfer large files (> 1Gb) was to invoke the native copy command (copy.exe).
- Baptiste Wicht
  
  Yes, I can try to add it to bench, but it takes a long time to make the bench. I’ll try to post the results tomorrow.
Peter

Hi, interesting post, thanks! Please, could you post, for a comparison, the time needed to copy the same files with the native copy utility (eg. copy/xcopy on Windows, cp on Unix). I am just wondering how Java compares with low-level copy.
I remember that some years ago I figured out that the fastest way to transfer large files (> 1Gb) was to invoke the native copy command (copy.exe).
- Baptiste Wicht
  
  Yes, I can try to add it to bench, but it takes a long time to make the bench. I’ll try to post the results tomorrow.
http://www.tracemodeler.com Yanic

It might be interesting to add an entry for the maximum throughput you can achieve on that SATA hard disk of yours.. Just to see how close the results are to the maximum achievable score.

Couple of things caught my eye :

The NIO transfer of a 50MB text file takes 400ms, while a binary file of the same size requires +- 450ms?

Judging from the 50MB binary file benchmark, NIO transfer has a throughput of about 111 MB/s (50MB / 0.450s) in your test. In the test for the 1.3GB binary file the throughput is about 55MB (1331MB / 24s). Why such a big difference?

Any ideas on the above?
- Baptiste Wicht
  
  On my PC, I cannot benchmark the Disk for Write throughput. If you have a way to make a simple read/write bench on hard disk on Ubuntu, you are welcome.
  
  For the huge difference, I don’t know. Perhaps, the OS can make more efficient write on little files. I don’t know.
http://www.tracemodeler.com Yanic

It might be interesting to add an entry for the maximum throughput you can achieve on that SATA hard disk of yours.. Just to see how close the results are to the maximum achievable score.

Couple of things caught my eye :

The NIO transfer of a 50MB text file takes 400ms, while a binary file of the same size requires +- 450ms?

Judging from the 50MB binary file benchmark, NIO transfer has a throughput of about 111 MB/s (50MB / 0.450s) in your test. In the test for the 1.3GB binary file the throughput is about 55MB (1331MB / 24s). Why such a big difference?

Any ideas on the above?
- Baptiste Wicht
  
  On my PC, I cannot benchmark the Disk for Write throughput. If you have a way to make a simple read/write bench on hard disk on Ubuntu, you are welcome.
  
  For the huge difference, I don’t know. Perhaps, the OS can make more efficient write on little files. I don’t know.
Craig

I noticed that the benchmarking framework you are using collects statistics. Can you publish the confidence intervals around your results so we can see which benchmark differences are statistically significant? TIA
- Baptiste Wicht
  
  Yes, I will publish them tomorrow. I will launch the benchmark tonigh with new bench :
  
  First phase on the same disks
  Second phase between two disks
  Add a method that just invokes /bin/cp to make the copy
  - Peter
    
    Please consider the operating system and the hard disk cache when running such benchmarks.
    After you read a file the first time, the second read will be faster if still in cache.
    It would be good to empty the cache before each test (I don’t know how that could be done with the hard disk cache without a system reboot).
    - Baptiste Wicht
      
      I know there is a file system cache and a hard disk cache, but I don’t think they are invalidating the results.
      
      The cache are for all the different methods. So OK, the first tests will be slower than the other, but the benchmarking library I use make a JVM Warmup executing the taks several times, so the caches will also be loaded (normally) at this time. And all the benchmarks will benefit of the cache. No ?
      - Peter
        
        You are right, the warmup will fill the cache.
  - Baptiste Wicht
    
    The new results are not online. Feel free to comment them.
    
    I hope this is better ?
Craig

I noticed that the benchmarking framework you are using collects statistics. Can you publish the confidence intervals around your results so we can see which benchmark differences are statistically significant? TIA
- Baptiste Wicht
  Yes, I will publish them tomorrow. I will launch the benchmark tonigh with new bench :
  - First phase on the same disks
  - Second phase between two disks
  - Add a method that just invokes /bin/cp to make the copy
  - Peter
    
    Please consider the operating system and the hard disk cache when running such benchmarks.
    After you read a file the first time, the second read will be faster if still in cache.
    It would be good to empty the cache before each test (I don’t know how that could be done with the hard disk cache without a system reboot).
    - Baptiste Wicht
      
      I know there is a file system cache and a hard disk cache, but I don’t think they are invalidating the results.
      
      The cache are for all the different methods. So OK, the first tests will be slower than the other, but the benchmarking library I use make a JVM Warmup executing the taks several times, so the caches will also be loaded (normally) at this time. And all the benchmarks will benefit of the cache. No ?
      - Peter
        
        You are right, the warmup will fill the cache.
  - Baptiste Wicht
    
    The new results are not online. Feel free to comment them.
    
    I hope this is better ?
Kees

You should also record the CPU-usage.
nio will surpass CPU if it can.
- Baptiste Wicht
  
  Sorry, but sadly, the framework I use does not collect CPU Usage data.
Kees

You should also record the CPU-usage.
nio will surpass CPU if it can.
- Baptiste Wicht
  
  Sorry, but sadly, the framework I use does not collect CPU Usage data.
http://javamexico.org Enrique

It would be interesting to use bigger buffers. 4096 bytes is not much; When I’ve had to copy bytes between files or sockets or whatever I use buffers between 16384 and 65536 bytes. A 64K buffer might make a big difference compared to the 4K buffer, because there will be far fewer reads.
- Baptiste Wicht
  
  Of course, the results will vary using different buffer sizes as said in the conclusion, but I cannot made all the the benchmarks with several buffer sizes. The actual benchmarks takes several hours to finish. But I will try to start some more tests this week-end including some other buffer sizes.
http://javamexico.org Enrique

It would be interesting to use bigger buffers. 4096 bytes is not much; When I’ve had to copy bytes between files or sockets or whatever I use buffers between 16384 and 65536 bytes. A 64K buffer might make a big difference compared to the 4K buffer, because there will be far fewer reads.
- Baptiste Wicht
  
  Of course, the results will vary using different buffer sizes as said in the conclusion, but I cannot made all the the benchmarks with several buffer sizes. The actual benchmarks takes several hours to finish. But I will try to start some more tests this week-end including some other buffer sizes.
Alan

Benchmarking file copying is very hard and I assume all these tests are actually run with the source file in the file cache. To do a more complete test probably would require dropping the cache before each copy. Maybe copy hundreds of different files of the same size and reboot/drop cache before running each test?

In jdk7 there is java.nio.file.Path with a copyTo method that is a real file copy. When I say “real file copy” then it does a bit more than just copy file contents. It can copy directories, sym links, and other types of files too. It knows not to override the target file by default, knows can copy meta-data (file timestamps, file permissions, ACLs, etc.), and more. It might be good to include this in your tests. The method should make use of the best mechanism on each platform.

A couple of comments on the code in FileCopyBenchmark. In nioBufferCopy the return from out.write(buffer) isn’t checked. Short writes are possible so this code should be: while (buffer.hasRemaining()) { out.write(buffer); }. As the method is just copying bytes then you may find that using a direct buffer is faster – try changed the ByteBuffer.allocate to ByteBuffer.allocateDirect. It looks like nioTransferCopy has the same issue as nioBufferCopy – you need to check the return value from transferTo.
- Baptiste Wicht
  
  Thanks for your interesting reply
  
  I know that the cache can change the results, but I cannot really reboot before each test, it’s already enough long to run all tests without being aside my computer to reboot it before each test.
  
  I know there is a new version of file copy in Java 7, but at the time of the benchmark, I had no JDK7 installed. The next time I will run the tests, I will include it.
  
  Thanks for the corrections, I didn’t know that. I will change that for the next run of the tests. I will run all that tests after my vacation in one week and an half. So if somebody else has request for tests, make them before that point.
  - Baptiste Wicht
    
    The benchmarks have been updated
    
    I’ve made some corrections on the code (Thanks to Alan) and included the Path Java 7 version
Alan

Benchmarking file copying is very hard and I assume all these tests are actually run with the source file in the file cache. To do a more complete test probably would require dropping the cache before each copy. Maybe copy hundreds of different files of the same size and reboot/drop cache before running each test?

In jdk7 there is java.nio.file.Path with a copyTo method that is a real file copy. When I say “real file copy” then it does a bit more than just copy file contents. It can copy directories, sym links, and other types of files too. It knows not to override the target file by default, knows can copy meta-data (file timestamps, file permissions, ACLs, etc.), and more. It might be good to include this in your tests. The method should make use of the best mechanism on each platform.

A couple of comments on the code in FileCopyBenchmark. In nioBufferCopy the return from out.write(buffer) isn’t checked. Short writes are possible so this code should be: while (buffer.hasRemaining()) { out.write(buffer); }. As the method is just copying bytes then you may find that using a direct buffer is faster – try changed the ByteBuffer.allocate to ByteBuffer.allocateDirect. It looks like nioTransferCopy has the same issue as nioBufferCopy – you need to check the return value from transferTo.
- Baptiste Wicht
  
  Thanks for your interesting reply
  
  I know that the cache can change the results, but I cannot really reboot before each test, it’s already enough long to run all tests without being aside my computer to reboot it before each test.
  
  I know there is a new version of file copy in Java 7, but at the time of the benchmark, I had no JDK7 installed. The next time I will run the tests, I will include it.
  
  Thanks for the corrections, I didn’t know that. I will change that for the next run of the tests. I will run all that tests after my vacation in one week and an half. So if somebody else has request for tests, make them before that point.
  - Baptiste Wicht
    
    The benchmarks have been updated
    
    I’ve made some corrections on the code (Thanks to Alan) and included the Path Java 7 version
Ronald

To understand NIO you should learn about DMA, http://en.wikipedia.org/wiki/Direct_memory_access. It doesn’t make IO faster, but it offloads the copying of bytes from the CPU. That is why the channels need a ByteBuffer. It is a memory buffer to which the DMA-chip copies the data. It can also copy the data directly from disk to network or from disk to disk as long as the sending and receiving device both understand DMA.
But as long as the CPU or the speed of your memory is not a bottleneck there is nothing which gets faster.
- Anonymous
  
  Hi Ronald,
  
  I know what is DMA, but I didn’t know that NIO used it.
  
  Thanks for the information.
  
  Baptiste
http://c-o-f.tumblr.com/ Zammbi

I wouldn’t mind seeing what the speed of renameTo method is even though its not a copy.
http://www.escortsmeet.com/ escorts

If your language is Assembly or Python like, you could benefit from creating a list of strings, where one string is a copy of one line. …
vl

I use KillCopy to transfer my files and folders to other locations. It is a powerful copy manager with job resuming and secure file erasing features.

http://bstdownload.com/reviews/killcopy-2/
Vicne

Very interesting posts, and thanks for sharing the source

A few thoughts :

You were asking for comments regarding same disk vs different disk difference. I guess your disks are not SSD, so when working on the same disk, the heads have to physically move back and forth between the areas to read from and write to, which is much slower than linear read on one disk and linear write on another. Same disk copying benefits highly from big memory buffers of course.
I also noticed that native method seems to be slower than any java method for small files, which is a nonsense of course. I guess the reason is that the overhead due to creating an environment to run a native process from java (Runtime.getRuntime().exec()) is predominant in this case. This influence could be reduced by performing a “cp src/* dst/” on a folder containing a thousand of small files.
Anyway, I’m more interested in large files (20GB+) across network shares so I’ll try to adapt your source to this case.

Best regards,

Vicne
- Anonymous
  
  Hi Vicne,
  
  Yes, none of my disk were a SSD. You’re right that working on the same is slower than with two disks, but based on my benchmark results, it doesn’t look much slower as we should think.
  
  You’re absolutely right about the native method. For very little files, it’s only slower because of the overhead from Java invocation of a command.
  If you run this kind of benchmarks on your own, I’m very interested of the results.
  
  Thanks
  
  Best Regards
  
  Baptiste
- Niharika
  
  Hi Vicne .Are you able to find any good solution for large files(20GB+)
  I am working on the same thing.please reply .
ab0032

I think your effort is great, but to really test things, especially for small files, is that it is not enough to measure the time to copy one file, but to measure how things behave it you cp small medium and large files for say 1000 sec and see how much you get done.
- ab0032
  
  The reason is, that now all the different things you tested behave relativly equal, which is due to the fact, that your hard disk is probably the bottleneck, and it doesnt matter how the java copies things, the cpu is idle and it doesnt matter if it copies small files around one or ten times. But what if you have an application, that really needs the CPU? Then coping around buffers in Java is a bad idea, because it takes time and produces garbage that has to be collected. Hence even if your results show that actually reading and writing is as fast as native io, it can still be a good idea to use native IO.
  - wichtounet
    
    You’re right, that could be interesting to add to the benchmark the throughput of each technique. And yes, again, depending on the situation, there are cases were a method should be avoided and another preferred.
    
    I do not have time for now to improve this benchmark to test more things, but I will try to do it in the future.
Lian

Just to complete the list of possible file copy methods: you can also use java.io.RandomAccessFile.

Best regards,
Lian

File copy in Java – Benchmark

Related Posts

Recent posts

Most commented posts

Latest Tweets

Blogroll

File copy in Java – Benchmark

Related posts:

Related Posts

Recent posts

Most commented posts

Latest Tweets

Blogroll