54

I am a graduate student, and the group in which I work maintains a Linux cluster. Each node of the cluster has its own local disk, but these local disks are relatively small and are not equipped with automatic backup. So the group owns a fileserver with many TBs of storage space. I am a relative Linux novice, so I am not sure what are the specs of the fileserver in terms of speed, networking ability, etc. I do know from experience that the local disks are significantly faster than the fileserver in terms of I/O. About a dozen or so people use the fileserver.

Using cp to copy a ~20 GB file from the fileserver to one of the local disks takes about 11.5 minutes in real time on average (according to time). I know that this cp operation is not very efficient because (1) time tells me that the system time for such a copy is only ~45 seconds; and because (2) when I examine top during the copy, %CPU is quite low (by inspection, roughly 0-10% on average).

Using cp to copy the same ~20 GB file from one folder on the local disk to another folder on the same local disk takes less time -- about 9 minutes in real time (~51 seconds in system time, according to time). So apparently the fileserver is somewhat slower than the local disk, as expected, but perhaps not significantly slower. I am surprised that copying from local to same local is not faster than 9 minutes.

I need to copy ~200 large files -- each ~20 GB -- from the fileserver to one of the local disks. So, my question is: Is there a faster alternative to cp for copying large files in Linux? (Or are there any flags within cp that I could use which would speed up copying?) Even if I could somehow shave a minute off this copying time, that would help immensely.

I am sure that buying new, faster hardware disks, but I don't have access to such resources. I am also not a system administrator -- I am only a (novice) user -- so I don't have access to more detailed information on the load that is on the disks. I do know that while about a dozen people use the fileserver daily, I am the only person using this particular node/local disk.

Jeff Schaller
  • 66,199
  • 35
  • 114
  • 250
Andrew
  • 16,315
  • 34
  • 73
  • 77
  • 30
    That makes around 29MB/s, which is pretty fast if you ask me. I don't think there's any command that will speed this up, the "bottleneck" is most likely a) the network or b) the file-server. – tink Jun 17 '13 at 20:08
  • 5
    tink is 100% correct. I've never seen anything that can improve this. The only thing that I've done in the past is compress the data prior to sending it, but that means you're adding time with the compression step and decompression steps, but sometimes that's worth it if the data is a good candidate to be compressed! – slm Jun 17 '13 at 20:10
  • 3
    You can also try `dd` and `rsync` to compare which one works faster in your environment – Raza Jun 17 '13 at 20:11
  • @Salton Thanks. I have not yet tried `dd`, but I just tried `rsync`. The real time was about 11.5 minutes and the system time was about 1.5 minutes, according to `time`. – Andrew Jun 17 '13 at 20:57
  • I've seen type of question asked and answered too many times? Did you search in the archives to see if anything could help you before asking the question? – mdpc Jun 18 '13 at 06:18
  • How are you mounting the data on the file server (NFS, CIFS, AFS, SSHFS?). Have you got shell access on the file server? Are ssh or rsync available there? – Stéphane Chazelas Jun 18 '13 at 06:50
  • Do you compy the same files many times, or files taht have been updated ? For example, computer A updates some files, then copy it again to the second computer? If so, I've edited my answer. – woliveirajr Jun 18 '13 at 13:07
  • If this is a one-time endeavor, do it however you'd like (and Thomas Padron-McCarthy's sneakernet answer below will probably be the fastest overall, with the least effect on the network). If this is going to be a recurring task (backup?) then you should figure out how to enact the transfers at night (and if it is for backup, come up with a good backup plan). – Chris Gregg Jun 19 '13 at 09:45
  • 2
    I'm surprised nobody has pointed out that the local disk to local disk copy could be made more efficient by having multiple disks mounted. Copying from `/dev/sda1` to `/dev/sdb1` is going to be faster than copying from one location on `/dev/sda1` to another location on `/dev/sda1` or another partition on `/dev/sda` because the hard drive won't have to do additional seeks between reads and writes (assuming traditional hard drives with spinning disks and moving heads; SSD is obviously different). – tripleee Jun 27 '13 at 05:09

12 Answers12

54

%CPU should be low during a copy. The CPU tells the disk controller "grab data from sectors X–Y into memory buffer at Z". Then it goes and does something else (or sleep, if there is nothing else). The hardware triggers an interrupt when the data is in memory. Then the CPU has to copy it a few times, and tells the network card "transmit packets at memory locations A, B, and C". Then it goes back to doing something else.

You're pushing ~240mbps. On a gigabit LAN, you ought to be able to do at least 800mbps, but:

  1. That's shared among everyone using the file server (and possibly a connection between switches, etc.)
  2. That's limited by the speed the file server can handle the write, keeping in mind its disk I/O bandwidth is shared by everyone using it.
  3. You didn't specify how you're accessing the file server (NFS, CIFS (Samba), AFS, etc.). You may need to tune your network mount, but on anything half-recent the defaults are usually pretty sane.

For tracking down the bottleneck, iostat -kx 10 is going to be a useful command. It'll show you the utilization on your local hard disks. If you can run that on the file server, it'll tell you how busy the file server is.

The general solution is going to be to speed up that bottleneck, which of course you don't have the budget for. But, there are a couple of special cases where you can find a faster approach:

  • If the files are compressible, and you have a fast CPU, doing a minimal compress on-the-fly might be quicker. Something like lzop or maybe gzip --fastest.
  • If you are only changing a few bits here and there, and then sending the file back, only sending deltas will be much faster. Unfortunately, rsync won't really help here, as it will need to read the file on both sides to find the delta. Instead, you need something that keeps track of the delta as you change the file... Most approaches here are app-specific. But its possible that you could rig something up with, e.g., device-mapper (see the brand new dm-era target) or btrfs.
  • If you're copying the same data to multiple machines, you can use something like udpcast to send it to all the machines at once.

And, since you note you're not the sysadmin, I'm guessing that means you have a sysadmin. Or at least someone responsible for the file server & network. You should probably ask him/her/them, they should be much more familiar with the specifics of your setup. Your sysadmin(s) should at least be able to tell you what transfer rate you can reasonably expect.

derobert
  • 107,579
  • 20
  • 231
  • 279
18

This could, possibly, be a faster alternative, and you won't clog the network for two days: Take one or two large USB (USB 3 if you have it) or FireWire disks, connect it to the server and copy the files to the disk. Carry the disk to your local machine. Copy the files to the machine.

Thomas Padron-McCarthy
  • 369
  • 1
  • 7
  • 30
    Sneakernet (http://en.wikipedia.org/wiki/Sneakernet) can be very fast: Never underestimate the bandwidth of a station wagon full of tapes hurtling down the highway. – SplinterReality Jun 18 '13 at 05:52
9

Your definition of efficient is backwards. A more efficient implementation wastes less cpu time. On the local copy you are averaging about 74 MB/s of throughput ( read + write ), which is about as good as a single hard disk is going to get.

psusi
  • 17,007
  • 3
  • 40
  • 51
9

If you have direct SSH (or SFTP) access (ask your sysadmin), you can use scp with compression (-C):

scp -C you@server:/path/to/yourfile .

Of course, that's only useful if the file is compressible, and this will use more CPU time, since it will be using encryption (because it's over SSH), and compressing.

Brendan Long
  • 542
  • 4
  • 16
  • In this case, it would be useful to disable the encryption. Remember that we are trying to make the copy _faster_. – lgeorget Jun 18 '13 at 02:23
  • 3
    @lgeorget I suspect the overhead of the encryption won't be significant, considering how slow hard drives are. I considered adding something about `-c none`, but that [seems to be non-standard](http://www.mail-archive.com/[email protected]/msg62497.html). – Brendan Long Jun 18 '13 at 02:32
  • 1
    We're dealing with ~20G files so it _is_ pretty inefficient to use encryption if not needed. – lgeorget Jun 18 '13 at 02:44
  • 1
    @lgeorget Encryption can be done far faster than the throughput he's getting, so it won't slow anything down. But it does seem unnecessary to go through SSH here. If you just need compression surely there are other tools? – Thomas Jun 18 '13 at 02:56
  • @Thomas The advantage of SSH is that if you're supposed to have access to the remote server, then it's almost certainly running SSH. Another option would be to compress the file locally, copy it to the server, then `ssh` in and decompress it.. – Brendan Long Jun 18 '13 at 15:47
8

The cp implementation is most likely not a bottleneck. Try to observe IO usage via iotop on both server and cluster node. This will give you an idea where you can improve the performance.

Another tip, is to avoid copying same data from same host. For example, if you have identical 20G file to distribute from fileserver over the network to all cluster nodes, it will work much faster to if you copy files in peer-to-peer fashion rather than one-server-to-all-clients. It's bit more complicated to implement, but you can even try to use some commandline p2p like direct connect hub.

If within that 20G files, some part is common, and some are cluster node specific, consider splitting it into common, and specific parts, and then distribute common part in p2p way.

Michał Šrajer
  • 2,808
  • 17
  • 17
  • 1
    If you're on a LAN, you should be able to do multicast instead of peer-to-peer. Which should be faster, and less load on the network. – derobert Jun 17 '13 at 20:45
8

The nature / contents of those files can make some difference. I understood that you need to copy 200 files, ~20 GB each, from one computer to another, is that it ?

If those files are compressible or with similar / identical pieces , you have two approaches:

  • zip them before copying, or create a tunnel between the computers with zip enable on it. So, if the network is the bottleneck, it will be a bit faster

  • if the files are very similar, or share some pieces of common content among them, try using rsync. It'll spend some time finding what is common among the files, and won't need to copy it literally, because it'll reconstruct it based on what is common.

edit

Will you need to copy those files many times ?? (like a copy -> use those files -> change something in the files in the computer A -> copy files again to computer B)

If so, rsync will be helpfull, because it'll try to detect what is equal among the versions and don't copy what is unchanged.

And a third method: if the above is correct (changes in file, then copy all files again to the second computer) you could try some binary diff to just change in the second computer what was changed in the first computer.

woliveirajr
  • 291
  • 2
  • 10
6

I see the following here, encryption is not a good idea as it might possibly INCREASE the amount of data to be transferred.

If you are copying between two systems, then the bottleneck is of course the connection between the servers.

If you are copying locally, look at how the process goes, it is SINGLE threaded, thus standard Linux utilities use:

- for all blocks in a file
      read a block
      write a block

There is NO concurrency to this operation.

To speed things up you can use something like this:

  buffer -i infile -o outfile -m size-of-shared-memory-default-1MByte

See the buffer(1) man page for more information.

The buffer command sets up two processes to run the copy process concurrently: one for reading, and the other for writing, and it uses a shared memory buffer to communicate the data between the two processes. The shared memory buffer is your classic circular buffer which prevents overwrite of unwritten data and writing of data already written. I have used this program to cut off about 10-20% of copy time in transfers from disk to tape.

mdpc
  • 6,736
  • 3
  • 32
  • 46
  • Actually, there is concurrency in "read a block/write a block" because "write a block" actually just puts it in the kernel's buffer, and the kernel handles the actual block write in the background (at least, until you start running out of RAM). Or if you are using O_DSYNC/O_SYNC for some reason. – derobert Jun 18 '13 at 18:49
3

Why not try a P2P propagation algorithm, if you need to update your entire cluster at the same time?

https://github.com/lg/murder is what twitter uses

There's BTSync that you can try as well.

Gui13
  • 168
  • 2
  • 11
1

If you're copying the same sets of files frequently from your local computer to the server with minor changes here and there. You can speed up the transfer by using rsync or a DVCS (e.g. hg or git).

git or hg can keep track and detect deltas and only transfer those deltas. In case of using a git, since both sides have full history of the repository, figuring out the delta is very cheap.

rsync uses a form of rolling checksumming algorithm to detect deltas without prior knowledge of what's on the other side. While it takes more work for rsync to calculate the deltas, it does not need to store the whole file history.

Lie Ryan
  • 1,208
  • 7
  • 12
1

You might want to try packaging all the files into a single archive (need not be compressed). In my experience, copying that one archive is faster than copying a large number of individual files

Michael Mrozek
  • 91,316
  • 38
  • 238
  • 232
Munim
  • 111
  • 3
  • 3
    Good generic observation, but as the question says “~200 large files -- each ~20 GB”, I don't believe this can be considered an actual answer to _this_ problem. – manatwork Jun 21 '13 at 08:50
  • @manatwork ah.. i didn't read clearly. I thought he had 200 files totaling 20gb – Munim Jun 22 '13 at 06:52
0

Try bbcp. Testing in our environment revealed that cp had some sort of built in governer. Just be careful because when you take off the governer, you can red-line your server and cause an outage. In our case we were taking the server offline to do the copy, so faster was better. This improved transfer time several hours.

James Shewey
  • 1,126
  • 2
  • 11
  • 18
0

Make sure the target files don't exist before copying.

Sometimes it is surprising how much time is spent even just copying on the same host (no network involved).

See my answer to another cp question here. Long story short, overwriting an existing file is much slower than truncating it or unlinking it first, and then copying. The latter is 8x faster for a 1.2GB file.

Pierre D
  • 645
  • 1
  • 5
  • 14