3

I am a graduate student and a relative Linux novice. My institution has an in-house Linux cluster on which I run many scientific simulations. I have a Windows desktop computer from which I access the Linux cluster via SSH.

I have a large amount (~1 TB) of simulation results data on the Linux cluster's file server. When the project is finished, the research group probably will not have the space to save the simulation results. However, I would like to save the files (with the group's permission, of course) on an external drive that I myself will purchase.

My question is, if I purchase a standard Windows external hard drive with a USB connection, will I be able to copy the files from the Linux cluster's files server to the external drive? (I am assuming that the Linux cluster has a USB port, but this is something that I will need to verify.)

It looks like many standard Windows external hard drives are formatted in either NTFS or FAT32, whereas our Ubuntu Linux file server uses NFS. Here are some examples from Amazon:

Do you think any or all of the above hard drives will be able to be easily reformatted in NFS for use with the Linux cluster?

On the other hand, Amazon does have a section for "Linux platform support" external hard drives, such as:

But, even if standard Windows external hard drives are easily reformatted, the problem is that I may subsequently want to copy the files from the external hard drive to a Windows computer, which is NTFS. This part of the question may require a separate question or a question on SuperUser, but is it possible to copy NFS files from an external hard drive to a Windows NTFS computer? Thanks for your time.

Andrew
  • 16,315
  • 34
  • 73
  • 77
  • 5
    As Hennes explains, data files are filesystem agnostic -- you can transfer a file from NFS to ext4 to NTFS, etc, and it does not matter. So you are probably best off just leaving the hard drive as NTFS, if that is mostly what you use normally. – goldilocks Jul 13 '13 at 18:23
  • 3
    ext4 might have better performance than NTFS (which runs in userspace). But the bottleneck in external HDDs is usually the USB interface. – Renan Jul 13 '13 at 19:13

2 Answers2

6

My question is, if I purchase a standard Windows external hard drive with a USB connection, will I be able to copy the files from the Linux cluster's files server to the external drive?

Yes, there is no technical problem to this, however:

The hardware us not a "standard windows hard drive with USB connection". Please scrap the windows part from that sentence. And external USB HDD will work equally well with or without windows as the OS.

I am assuming that the Linux cluster has a USB port, but this is something that I will need to verify.

For a large amount of data (and 1TB is a lot) connecting the drive locally is probably a lot faster. However with USB2 you are still limited to 35-ish MB/sec. That means that copying 1TB over USB2 takes about 8-9 hours.*

You can speed that up a lot if the drive is locally mounted (via plain SATA), if the cluster and your drive have eSATA, if both have USB3 or if both have firewire.

Alternatively you can connect the drive to your own desktop and copy the files. In this case the network might be the speed limit. You also risk an angry administrator asking why you are making the network so slow for other users. :-)

It looks like many standard Windows external hard drives are formatted in either NTFS or FAT32, whereas our Ubuntu Linux file server uses NFS.

uhm, no.

The hard disk does not care which filesystem is used. It may come pre-formatted with NTFS (which is a sensible choice for most people who buy them), but nothing stop you from changing the filesystem and reformatting. That should only take a few minutes.

Also, your file server does not use NFS on its hard disks. It is probably using ext2, ext4 or ZFS. Neither of which you need to worry about. As long as you can read the data you can write it in any format.

(Consider the analogy: You copy the text written in a notebook. Do not worry about the form or the colour of the original notebook. As long as you can read it and have a large enough notebook of your own you can copy the content from one notebook to another).

*: 8-9 hours estimated based on this:

35 MiB/second
100 MiB per 3 seconds.
1000 MiB per 30 seconds, which is the same as 1GiB per 30 seconds.

1GiB per 30 seconds
1000GiB per 30000 seconds
1TiB per 30000 seconds. 30000/3600=8.3 (3600 seconds per hour)

Hennes
  • 1,958
  • 14
  • 14
  • 4
    *The hardware us not a "standard windows hard drive"* To be fair to the OP, hard drives may be packaged that way, and to the extent that they are NTFS pre-formatted, it is sort of true. But you are correct to say this does not matter and that any "windows" hard drive can be reformatted easily and is usable without windows. – goldilocks Jul 13 '13 at 18:19
  • 1
    I agree they are packaged that way and usually I would just ignore that. However I feel that it is relevant to the OP and that it is useful to people who later come looking for information. So I mentioned it. (Maybe I did mention it rather heavily with the notebook copy example). – Hennes Jul 13 '13 at 19:52
-1

A wrote up an elaborate blog post on how to do exactly this.

http://blog.championswimmer.in/2014/saving-linux-data-on-your-fat32-external-hdd-using-loop-mount/

Basically you cannot save Linux data to FAT32 without loosing permissions and having it screwed up. What you can do is create an Ext4 loop device inside it and store there.

Create a 64GB file

cd /media/myUserName/HugeExternalDrive/my_favourite_folder/
dd if=/dev/zero of=./my.disk bs=33554432 count=2048

Mount and format 

sudo losetup /dev/loop0 ./my.disk
sudo mkfs.ext4 /dev/loop0

Now mount to a suitable directory

mkdir /home/myUserName/mount_point
sudo mount /dev/loop0 /home/myUserName/mount_point
sudo chown -R myUserName:myUserName /home/myUserName/mount_point

Now you are ready to use the created disk image

The linked blog post describes in more detail how that can be done.

Arnav Gupta
  • 101
  • 1
  • Since links often go stale, [it is always preferred](http://meta.stackoverflow.com/q/8259) to include the essential parts of the answer here, and provide the link for reference. – drs Aug 07 '14 at 13:30
  • This won't work for the asker's needs: you can't make a file larger than 4GB on a FAT32 disk, so a disk image large enough to store 1TB of data certainly won't fit. – Mark May 11 '15 at 07:59