Use Find to list files and include counter value in output

Question

For performance reasons, I want to use find to list a large number of files, but also include a counter on each line. The following is what I have so far:

local root_dir="."
local ouptut_file="/tmp/foo.txt"

File_Number=99              ## Initial value

echo "" > "${ouptut_file}"  ## Initialize file to empty

find "$root_dir"   \
    -type f        \
    -depth 2       \
    -name '*.xxx'  \
    -print0        \
| sort -z          \
| xargs -0  printf "DoSomething %5d '%s'\n" $[File_Number++] -- \
> "${ouptut_file}"

This outputs

DoSomething    99 '--'
DoSomething     0 'dirA/dirB/file1.xxx'
DoSomething     0 'dirA/dirB/file2.xxx'
DoSomething     0 'dirA/dirB/file3.xxx'

what I want is

DoSomething     100 'dirA/dirB/file1.xxx'
DoSomething     101 'dirA/dirB/file2.xxx'
DoSomething     102 'dirA/dirB/file3.xxx'

How do I include the counter and also eliminate the first spurious line in the output.

System: Am using bash on macOS Ventura 13.0.1

You never need to initialize a file. You can just do `find ... > file` and the file will be created if it doesn't exist and will be emptied and its contents replaced with the output of the command if it did exist. — terdon, Nov 30 '22 at 12:10
... or if the `noclobber` option is set (with `set -o noclobber`in `~/.bashrc`), use `find ... >| file` which will clobber any already existing file upon redirecting output to them, something practical but unsafe by design ... — Cbhihe, Nov 30 '22 at 18:21
The easiest way to number lines is to use the `nl` command like that `find ... | nl --starting-line-number=100`. It doesn't support zero terminated strings and also has no way of prepending output lines with custom text. — legolegs, Nov 30 '22 at 19:09

score 5 · Answer 1 · answered Nov 30 '22 at 06:55

A couple of notes:

$[...] is a really old form of arithmetic expansion. These days the bash documentation doesn't even mention it. Use $((...)) instead.
xargs will start a new process for each invocation of the command, so any changes to variables in one won't affect other invocations.
For printf(1), -- is just like any another argument. What's happening is that xargs runs printf "DoSomething %5d '%s'\n" $[File_Number++] dirA/dirB/file1.xxx dirA/dirB/file2.xxx ..., and printf loops over the arguments to fit the format string requirements. So every alternate filename is actually being treated as a number, resulting in the 0 in the output.

You might want to use something like:

find "$root_dir"   \
    -type f        \
    -depth 2       \
    -name '*.xxx'  \
    -print0        \
| sort -z          \
| while read -r -d '' file; do
     printf "DoSomething %5d '%s'\n" $((File_Number++)) "$file"
  done > "${ouptut_file}"

Also note that if you're actually using this to print commands for running in a shell, you should use %q instead of '%s' for proper quoting. From help printf:

In addition to the standard printf(1) formats, %b means to expand backslash escape sequences in the corresponding argument, and %q means to quote the argument in a way that can be reused as shell input.

Using `printf %q` is not the safest if it's to be interpreted by a Bourne-like shell. As the OP's on macos, they have the option to use [`zsh` which has more reliable quoting operators](/a/600214). — Stéphane Chazelas, Nov 30 '22 at 09:34
@StéphaneChazelas is right about the safety of `printf %q`, but if you can always know the conditions of your file system and can be assured that you'll never be sending it anything risky, that would mitigate the risk. — Br.Bill, Nov 30 '22 at 19:35
Point 2 isn't relevant here, because the variable here is expanded by the shell before `xargs` sees anything. So there's only ever one increment of `$File_number` regardless of how many processes are started there. Otherwise, a great explanation of _why_ the script doesn't do what Peter was expecting. — Toby Speight, Dec 01 '22 at 09:08

Stéphane Chazelas · Answer 2 · 2022-11-30T16:51:52.623

I'd use perl:

find "$root_dir"   \
    -type f        \
    -depth 2       \
    -name '*.xxx'  \
    -print0        \
| sort -z          \
| perl -l -0ne 'printf "something %05d %s\n", $., $_' > "$output_file"

If you need to quote the files, putting ' on either side may not be enough depending on what file names may contain and what language those quotes are intended for. For instance, in several languages including some shells, the \ and newline characters are still special within quotes and would need to be escaped. In any case, the ' character cannot be embedded asis in a single quoted string.

If the quoting is for a Bourne-like shell, you can do quoting with a:

sub shquote {return "'" . ($_[0] =~ s/'/'\\''/gr).  "'"}

function which is the safest form of quoting in those shells¹. So here:

perl -l -0ne '
  sub shquote {return "'\''" . ($_[0] =~ '"s/'/'\\\\''/gr"').  "'\''"}
  printf "something %05d %s\n", $., shquote($_)'

Or if you struggle to keep track of all those quotes:

perl -l -0nse '
  sub shquote {return $q . ($_[0] =~ s/$q/$q\\$q$q/gr) . $q}
  printf "something %05d %s\n", $., shquote($_)' -- -q="'"

Also note that outputting to files with fixed name in world writable areas is bad practice from a security standpoint. Someone could create a symlink with the same name to some sensitive file you have write access to for instance. Creating files in /tmp should only be done with randomly generated file names and with O_NOFOLLOW like mktemp does, or in freshly made private directories there (bailing out if it can't be created, like for instance if it already existed).

^{¹ Safer than the String::ShellQuote CPAN module in any case which has a few issues.}

I am getting `No such file or directory` on the line with the `perl` statement. I am using the `bash` shell that comes with Mac so that is probably the issue. — Peter Grill, Dec 01 '22 at 00:59

Use Find to list files and include counter value in output

2 Answers2