0

This list will then be used to create tar balls for easy movement. This is what I came up with so far. Add up files that are just greater than 10GB list them to file. 

find .  -type f -exec ls -al {} + | awk '{ total += $5 }; {print $9}; {if (total > 10000000000) {print total; total = 0  }}; END { print total }' > /tmp/testfile.txt

What I would like to do name each tar ball with a unique name but I am putting all results in the /tmp/testfile.txt

Tried this but seems a little cumbersome. 

find . -type f -exec ls -al {} + | awk '{ filenumber };{ total += $5 }; {print $9}; {if (total > 10000000000) {print total; total = 0; filenumber++;print "This is filenumber"filenumber".tar" }}; END { print total }' > /tmp/testfile.txt

This is what I came up with - 

 find ./ -type f -exec ls -al {} + | awk 'BEGIN { filenumber };{ total += $5 }; {print $9 > "filename"filenumber".txt"}; {if (total > 10000000000) {print total; total = 0; filenumber++;print "This is filename"filenumber".txt" }}; END { print total }' > filecheck.txt

After that command I feed results to this command. 

for file in $( find . -name "filename*.txt" -type f );do tar -cvf "$file.tar" -T "$file"; done
LCJ
  • 1
  • 2
  • 1
    This doesn't directly answer your question, but.... I'm not sure about your basic premise. What about [Create a tar archive split into blocks of a maximum size](https://unix.stackexchange.com/questions/61774/create-a-tar-archive-split-into-blocks-of-a-maximum-size)? – mattdm May 22 '19 at 20:28
  • 1
    How are you transferring these files when you have them? Why 10GB? – mattdm May 22 '19 at 20:29
  • 1
    What exactly do you want to accomplish? – RalfFriedl May 22 '19 at 20:37
  • I have 100GB of data to move, splitting a large tar is not an option due to technical constraints. There is a data checker that finds the tar splits and dumps them as corrupted files. 10GB is a manageable size for NFS and rsyncing. So what I did was create a list of files that sum up in size of 10GB, that list is then fed to a tar command. – LCJ May 23 '19 at 15:03
  • Look up knapsack algorithms – muru May 23 '19 at 15:30
  • Let me try the knapsack algorithm, sounds promising. – LCJ May 23 '19 at 18:26
  • @LCJ Actually, what you want is "bin packing" not "knapsack". – Kaz Aug 09 '22 at 08:51
  • @Kaz Thanks for the info. I never heard of "bin packing" and fits the description. – LCJ Aug 10 '22 at 19:49

0 Answers0