61

I have (well, I had) a directory:

/media/admin/my_data

It was approximately 49GB in size and had tens of thousands of files in it. The directory is the mount point of an active LUKS partition.

I wanted to rename the directory to:

/media/admin/my_data_on_60GB_partition

I didn't realise at the time, but I issued the command from home directory so I ended up doing:

~% sudo mv /media/admin/my_data my_data_on_60GB_partition

So then the mv program started to move /media/admin/my_data and its contents to a new directory ~/my_data_on_60GB_partition.

I used Ctrl + C to cancel the command part way through, so now I have a whole bunch of files split across directories:

~/my_data_on_60GB_partition    <---  about 2GB worth files in here

and 

/media/admin/my_data           <---- about 47GB of orig files in here

The new directory ~/my_data_on_60GB_partition and some of its subdirectories are owned by root.
I'm assuming the mv program must have copied the files as root initially and then after the transfer chown'ed them back to my user account.

I have a somewhat old backup of the directory/partition.
My question is, is it possible to reliably restore the bunch of files that were moved?

That is, can I just run:

sudo mv ~/my_data_on_60GB_partition/*  /media/admin/my_data

or should I give up trying to recover, as the files are possibly corrupted and partially complete, etc.?

  • OS - Ubuntu 16.04 
mv --version  
mv (GNU coreutils) 8.25
Wildcard
  • 35,316
  • 26
  • 130
  • 258
the_velour_fog
  • 11,840
  • 16
  • 64
  • 109
  • 37
    Get into the habit when panicking to type `Control-Z` (to pause) rather than `Control-C`. In this case, you would then be able to see which file was being transferred at that time and so know which file was only partially copied. You can then decide calmly on how to proceed. (Use `kill -stop` for processes not in the tty). – meuh Oct 27 '16 at 18:07
  • 1
    2GB + 47GB = 60GB??? – tbodt Oct 27 '16 at 23:08
  • 7
    @tbodt `(2GB + 47GB) < 60GB`. partition capacity is 60GB, size of folder and its contents: 49GB. – the_velour_fog Oct 27 '16 at 23:12

3 Answers3

90

When moving files between filesystems, mv doesn't delete a file before it's finished copying it, and it processes files sequentially (I initially said it copies then deletes each file in turn, but that's not guaranteed — at least GNU mv copies then deletes each command-line argument in turn, and POSIX specifies this behaviour). So you should have at most one incomplete file in the target directory, and the original will still be in the source directory.

To move things back, add the -i flag so mv doesn't overwrite anything:

sudo mv -i ~/my_data_on_60GB_partition/* /media/admin/my_data/

(assuming you don't have any hidden files to restore from ~/my_data_on_60GB_partition/), or better yet (given that, as you discovered, you could have many files waiting to be deleted), add the -n flag so mv doesn't overwrite anything but doesn't ask you about it:

sudo mv -n ~/my_data_on_60GB_partition/* /media/admin/my_data/

You could also add the -v flag to see what's being done.

With any POSIX-compliant mv, the original directory structure should still be intact, so alternatively you could check that — and simply delete /media/admin/my_data... (In the general case though, I think the mv -n variant is the safe approach — it handles all forms of mv, including e.g. mv /media/admin/my_data/* my_data_on_60GB_partition/.)

You'll probably need to restore some permissions; you can do that en masse using chown and chmod, or restore them from backups using getfacl and setfacl (thanks to Sato Katsura for the reminder).

Community
  • 1
Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164
  • Thank you Stephen Kitt, thats a huge help! I can use `find` to find and set permissions. There are alot of files in the new directory that have spaces in the filenames, but no hidden files - that I know of. Do you think the glob in the command `sudo mv -i ~/my_data_on_60GB_partition/* /media/admin/my_data/` would expand the filename without word splitting issues? I was thinking alternatively I could use `sudo rsync ~/my_data_on_60GB_partition/ /media/admin/my_data/` which I believe would handle the file paths with spaces? – the_velour_fog Oct 27 '16 at 10:29
  • 6
    Just to be sure, when things like OP described happens, I use `rsync` instead, so it would also check integrity of all files. But it is nice to know that I do not need that. – Hauleth Oct 27 '16 at 10:29
  • 1
    @the_velour_fog globbing handles spaces in filenames without issues. – Stephen Kitt Oct 27 '16 at 10:34
  • 5
    I'd `su command mv -i ...` (or `su /bin/mv -i ...` ), instead of `sudo mv -i ...` ), in case some (weird) admin made "mv" a function that does "mv -f" *at the system level* (ie, /etc/profile or such system wide files)... command something : start the command somthing, and not a function or alias of the same name. (for ex: one could be (very!) unlucky and have a (very, very bad!) `function mv { /bin/mv -f -- "$@" }` in a file that is always sourced, and then "rm -i something" will not ask anything (and just protest that "-i" file doesn't exist!) ... [I've see such things... *shiver*] – Olivier Dulac Oct 28 '16 at 12:18
  • 3
    @OlivierDulac - a perfect example of why it's bad practice to use aliases or scripts with the same names as standard programs. – Joe Oct 28 '16 at 21:15
21

After getting Stephen Kitt's answer and discussing this command as a potential solution:

sudo mv -i ~/my_data_on_60GB_partition/* /media/admin/my_data/

I decided to hold off on running it until I got my head around what was happening, this answer describes what I found out and ended up doing.

I'm using Gnu mv which copies files to the target, then only if the copy operation is successful, it deletes the original.
However I wanted to confirm whether mv performs this sequence one file at a time, if that was true, the original folder contents would have cleanly been sliced into two parts, one part shifted to the destination, the other still left behind at the source. And possibly there would have one file that was interrupted during the copy which would common between the two directories - and it would likely be malformed.

To discover files that were common between the two directories, I ran:

~% sudo diff -r --report-identical-files my_data_on_60GB_partition/. /media/admin/mydata/. | grep identical | wc -l
14237

This result suggested there were 14,237 instances of the same files in both the source and target directories, I confirmed by checking the files manually - yes there were many of the same files in both directories. This suggests that only after mv copies great swathes of files does it perform the deletion of the source files. A quick lookup in info on mv command showed

It [mv] first uses some of the same code that's used by cp -a to copy the requested directories and files, then (assuming the copy succeeded) it removes the originals. If the copy fails, then the part that was copied to the destination partition is removed.

I didn't run the command but I suspect if I tried to run

sudo mv -i ~/my_data_on_60GB_partition/* /media/admin/my_data/

The -i prompt before overwrite likely would have triggered more than 14,000 times.

So then to find out how many total files in the newly created directory:

~% sudo find my_data_on_60GB_partition/ -type f -a -print | wc -l                                                                    
14238

So then if there was a total of 14238 regular files in the new directory and 14237 had identical originals back in the source, that means there was only one file in the new directory that didn't have a corresponding identical file back on the source. To find out what that file was, I ran rsync back in the direction of the source:

~% sudo rsync -av --dry-run my_data_on_60GB_partition/ /media/admin/my_data
sending incremental file list
./
Education_learning_reference/
Education_learning_reference/Business_Education/
Education_learning_reference/Business_Education/Business_education_media_files/
Education_learning_reference/Business_Education/Business_education_media_files/Jeff Hoffman - videos/
Education_learning_reference/Business_Education/Business_education_media_files/Jeff Hoffman - videos/Jeff and David F interview/
Education_learning_reference/Business_Education/Business_education_media_files/Jeff Hoffman - videos/Jeff and David F interview/018 business plans-identifying main KPIs.flv

sent 494,548 bytes  received 1,881 bytes  330,952.67 bytes/sec
total size is 1,900,548,824  speedup is 3,828.44 (DRY RUN)

A quick check confirmed that this was the malformed file, where the file existed on both the source and the destination, destination file=64MB, original=100MB. This file and its directory hierarchy was still owned by root and had not yet had the original permissions restored.

So in summary:

  • all the files which mv never reached are still back in their original locations (obviously)
  • all the files which mv did copy completely still had their original copies in the source directory
  • the file which was only partially copied still had the original back in the source directory

In other words all the original files were still intact and the solution in this case was to simply delete the new directory!

the_velour_fog
  • 11,840
  • 16
  • 64
  • 109
  • Wow... I've updated my answer, `-n` would be better in the general case. I checked the `mv` source code, it deletes the source one argument at a time. – Stephen Kitt Oct 28 '16 at 06:50
  • @StephenKitt ah nice. I was wondering when `mv` does the deletion on the source. So the command `mv foo bar baz` would move `foo` to `baz/foo` *then* delete the original `foo` then move `bar` to `baz/bar` ..? – the_velour_fog Oct 28 '16 at 07:10
  • Yes, that's right; in fact that's what [POSIX specifies](http://pubs.opengroup.org/onlinepubs/9699919799/utilities/mv.html) (basically, so that any error affecting any source argument leaves the entire source hierarchy intact). – Stephen Kitt Oct 28 '16 at 07:34
  • I think you could have used diff to find the one unfinished file also. – Weaver Oct 29 '16 at 05:15
  • @StarWeaver I think `diff` would have reported something like `file X only in /media/admin/my_data/...` for the 47Gb of files still in the original directory that `mv` never reached. I used `rsync` without the `--delete` option so that it would only try to add files from `~/my_data_on_60GB_partition/` that it detected were changed or absent in `/media/admin/my_data` whilst at the same time ignoring the fact that the target actually had thousands of files that the source didn't have. – the_velour_fog Oct 29 '16 at 05:33
  • 1
    You should use `cmp` rather than `diff` to compare binary files. Also, your discussion above makes sense only when moving files across different filesystems. There is no copying involved when moving files within the same filesystem. – Satō Katsura Oct 29 '16 at 08:43
  • @SatoKatsura what is the advantage of `cmp`? does it give more details than`diff`? yes dumb luck it went across to a different file system :) – the_velour_fog Oct 29 '16 at 08:49
  • `cmp` is faster and is guaranteed to work with binary files. `diff` may or may not stop early. And no, it's really hard to lose files with `mv`. :) – Satō Katsura Oct 29 '16 at 08:54
4

I just thought I'd comment that some people may be tempted to toss 'xargs' into the mix to run things in parallel. That gives me the willies and I really like the rsync solution above.

As to the filesystem stuff about moving and copying and when exactly the original gets deleted, the VFS and the underlying filesystem(s) coordinate to guarantee per-file atomicity before getting to that delete step. So even if it gets interrupted before the target file is fully written, all of the locking in the VFS is real strict and protects against stuff like random data interleaving even in parallel cases. (I worked on Linux VFS and NFS4 stuff)

Adding 'xargs' to the mix would probably have made the double-sanity-checking step a headache, with multiple files in mid-transit. I wish I'd had more system level-scripting. Good reminders for me!

Loved the question, good for cobwebs, and makes me love rsync again. Cheers!

david richter
  • 41
  • 1