Move the oldest files into a directory, source directory has more than 10,000 files constantly incoming

Question

I am working on a command to move the x number of oldest files(FIFO) into a directory, is it better idea to pipe the "find" result into "ls" or just use the "ls" .. please suggest.

ls -ltr `/bin/find ${in}/* -prune -name "*.txt"  -type f` | head -10 |     
 while read -r infile ; do                    
     -move the file 
done

or should i just use ls. the reason i am using find is: i read some online content that ls should be avoided in scripting. but in my code at the end i have to pipe the find results into the ls. i am worried if find result is too big would it might cause any problem.

No, our OS is hp-ux and we have ksh scripts in that i had to implement this part. — user3342678, Dec 28 '18 at 20:08
https://unix.stackexchange.com/a/449771/117549 may be useful — Jeff Schaller, Dec 28 '18 at 20:32
10k files per ...? If it's 10k files per second I suspect we're going to have problems! Mind you apparently a directory can hold 4 billion files. — pbhj, Dec 29 '18 at 01:22
If any of the answers solved your problem, please [accept it](https://unix.stackexchange.com/help/someone-answers) by clicking the checkmark next to it. Thank you! — Jeff Schaller, Jan 06 '19 at 14:34

score 0 · Answer 1 · answered Dec 28 '18 at 20:45

While perl is probably going to be faster than looping through a large directory "x" times, here's a brute force simple solution.

The outer loop determines how many files will be moved -- 3 in this example. Inside that loop, we initialize an "oldest" file to the first filename that results from globbing *. The inner loop then compares every file's timestamp to see if it is older than (-ot) the currently-oldest file. If it is, we update the "oldest" filename. At the end of the inner loop, we report and move that file.

for((count=0; count < 3; count++))
do
  set -- *
  oldest=$1
  for f in ./*
  do
    if [ "$f" -ot "$oldest" ]
    then
      oldest=$f
    fi
  done
  echo mv -- "$oldest" ./.archive
  mv -- "$oldest" ./.archive
done

Thank you, But in your solution you are looping to every file in that directory how could this improve the performance compared to just `ls command` , i could just use ls -ltr | head -x then for each file will do the move operation. — user3342678, Dec 28 '18 at 21:52

score 0 · Answer 2 · answered Dec 29 '18 at 01:07

Python `heapq.nsmallest`

bash:

find -printf '%T@ %p\n' |
python noldest.py 1000 |
xargs -Ixxx mv xxx directory_for_old_files

Here find command lists files in format "seconds from 1970 (%T@), space, filename (%p)". Trailing xargs command takes filenames from stdin one by one and applies mv xxx directory_for_old_files command substituting xxx with them.

noldest.py implementation could be:

import sys
from heapq import nsmallest
from datetime import datetime


n = int(sys.argv[1])
date_file_pairs = (line.split(' ', maxsplit=1) for line in sys.stdin)  # generators are lazy

def parse_date(date_and_filename):
    seconds_from_1970 = float(date_and_filename[0])
    return datetime.fromtimestamp(int(seconds_from_1970))

for _, filename in nsmallest(n, date_file_pairs, key=parse_date):
    print(filename.rstrip('\n'))  # lines in sys.stdin have trailing newlines

The perfomance depends on implementation of nsmallest algorithm from python standard library.

Move the oldest files into a directory, source directory has more than 10,000 files constantly incoming

2 Answers2

Python heapq.nsmallest

Python `heapq.nsmallest`