8

POSIX filenames may contain all characters except /, but some filesystems reserve characters like ?<>\\:*|". Using pax, I can copy files while replacing these reserved characters:

$ pax -rw -s '/[?<>\\:*|\"]/_/gp' /source /target

But pax lacks an --delete option like rsync and rsync cannot substitute characters. I'm looking for a simple way to backup my music collection to an external hard drive on a regular basis.

slm
  • 363,520
  • 117
  • 767
  • 871
Frank
  • 348
  • 2
  • 7
  • Have you looked at this method using rsync? http://serverfault.com/questions/54949/how-can-i-use-rsync-with-a-fat-file-system – slm Jan 11 '14 at 17:15
  • Thanks, good to know about the `--modify-window` option for `rsync`. But it does not solve the character substitution. – Frank Jan 11 '14 at 19:56
  • I could not find any other method to do what you're asking. Not saying it doesn't exist, but while researching this I found nothing promising. – slm Jan 11 '14 at 20:00
  • I have no idea how to solve this, but I will comment that I think your list of reserved characters is incomplete. AFAIK `+` and possibly `;` should be in there. Only way to be sure is to test though... – Graeme Jan 11 '14 at 20:25
  • What does rsync do when it meets these characters, anyway. Do you get an IO error? – Graeme Jan 11 '14 at 20:33
  • `rsync` says: Invalid argument. `strace` gave me: `open(".foo\\bar.VVfqTX", O_RDWR|O_CREAT|O_EXCL, 0600) = -1 EINVAL (Invalid argument)`. I guess this is because of the underlying filesystem telling the kernel not to accept the backslash character. – Frank Jan 11 '14 at 21:13

2 Answers2

6

You can make a view of the FAT filesystem with POSIX semantics, including supporting file names with any character other than / or a null byte. POSIXovl is a relatively recent FUSE filesystem for this.

mkdir backup-fat
mount.posixovl -S /media/sdb1 backup-fat
rsync -au /source backup-fat/target

Characters in file names that VFAT doesn't accept are encoded as %(XX) where XX are hexadecimal digits. As of POSIXovl 1.2.20120215, beware that a file name like %(3A) is encoded as itself, and will be decoded as :, so there is a risk of collision if you have file names containing substrings of the form %(XX).

Beware that POSIXovl does not cope with file names that are too long. If the encoded name doesn't fit in 255 characters, the file can't be stored.

POSIXovl stores unix permissions and ownership in files called .pxovl.FILENAME.

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
2

I was curious about this so I did a bit of digging.

There are a couple of patches to do this that never made it into rsync, if you fancy compiling from source. Although both are old so they may not still work. This one specifically adds a --fat-filenames option and this one for a --tr option.

These questions also cover something similar - Rsyncing files with special characters to USB FAT32 and https://askubuntu.com/questions/11634/how-can-i-substitute-colons-when-i-rsync-on-a-usb-key. The best bet is probably rdiff-backup which does the substitution, but I can't see any options to have any control over it. There are some other good ideas too though.

A rough idea for a script that would delete and recreate the files each time, if there are not too many, would be something like:

[sudo] updatedb      # doing this as root will update the system database
locate -r '^/full/path/to/source/.*[?<>\\:*|\"]' |
  sed 's\^/full/path/to/source/\\' >bad_names
rsync --delete --exclude-from=bad_names -av source/ target
cd source
<bad_names pax -rw -s '/[?<>\\:*|\"]/_/gp' target

locate would be faster than find if the database is reasonably up to date.

Finally you could just do this so that the files don't cause any more problems (at least the currently existing ones):

find source/ -regex '[?<>\\:*|\"]' -exec rename 's/'[?<>\\:*|\"]'/_/g' {} +

I would recommend trying that with rename -n first though in case something goes wrong.

Community
  • 1
Graeme
  • 33,607
  • 8
  • 85
  • 110
  • thanks for investigating. A pity rsync didn't include those pataches. If a skript will be the solution, your ideas will come in very handy. – Frank Jan 12 '14 at 00:13