How to find files that don’t have a suffixed version?

Question

I have a few million .jpg files and I want to generate a .jpg.webp version for each one (foo.jpg -> foo.jpg.webp). For this, I need to find all files ending in .jpg for which there’s no .jpg.webp version.

Right now, I do it like this:

find "$path" -type f -iname "*.jpg" |
  while read -r image_path; do
      if [ ! -f "$image_path.webp" ]; then
        echo "$image_path"
      fi
  done |
  # treat only 10000 files per run
  head -n 10000 |
  ...

However because I’m using a pipe, this creates a subshell. I wonder if there would be a more efficient way to do so, especially because the more WebP images I generate, the more time the script spends filtering paths to find candidates. Would there be some way to do this using find only?

I’m using Ubuntu 20.04. Files are spread across sub directories.

Answer 1

Try something like this:

find "$path" -type f -iname "*.jpg" -exec \
  sh -c 'for f; do [ -e "$f.webp" ] || echo "$f" ; done' find-sh {} +

This executes sh as few times as possible (depending how many .jpg files found by find), limited by ARG_MAX (around 2 million bytes on Linux), and avoids the need for an excruciatingly slow while read ... loop by passing all the filenames as command line arguments. See Why is using a shell loop to process text considered bad practice? and Why is looping over find's output bad practice?

To efficiently process batches of these files, I'd redirect the output to a file and then split that into batches of 10,000 (or however many you need), e.g. with split -l 10000.

NOTE: If any of your .jpg filenames contain newlines then you will need to use NUL as the separator between them, otherwise use newline as the separator. To use NUL separators, replace echo "$f" with printf "%s\0" "$f". BTW, split supports NUL-separated input with -t '\0'.

The script that process the batches should read in the filenames, and check again that the corresponding .jpg.webp file doesn't exist (in case one was generated after the list was generated) before running whatever it needs to generate the .jpg.webp version.

If you had to use NUL as the filename separator, it would be easiest to use readarray (AKA mapfile) to read the entire batch's list into an array and iterate over the array of filenames. Or use awk or perl to process the filenames.

Actually, using an array would be better than a while-read loop, even with newlines as separators.

Answer 2

I'd do the following:

1. Find all suffixed (i.e., *.jpg.webp) files, put them in a sorted list removing their suffix;
2. Find all files without suffix (i.e., *.jpg), put them in a second sorted list
3. compare the two list, removing those entries that are in the first list.
4. operate your conversion on the "set difference" list that results from it.

So, something like

#!/bin/bash
comm -z -1 -3 \
   <(find -name '*.jpg.webp' -print0 | sed 's/\.webp\x0/\x0/g' | sort -z) \
   <(find -name '*.jpg'      -print0 | sort -z) \
| parallel -0 gm convert '{}' '{}.webp'

assuming you're using GraphicsMagick gm for conversion (in my experience, speed- and reliability-wise much preferable to ImageMagick's convert), and assuming you have GNU parallel installed (if not, xargs might work just as well).

Answer 3

This sounds like a job for make. It will only generate the files that are missing, or have older modification time than the files they're generated from.

.PHONY: all
all: $(addsuffix .webp,$(shell find . -name '*.jpg'))

%.jpg.webp: %.jpg
    cwebp $< -o $@   #Some command that generates $@ from $<

Save this to a file named Makefile, and run make.
Or make -j $(nproc) to run as many parallel jobs as you have logical cores. Or pick an explicit number, perhaps tthe number of physical cores, to leave some idle logical cores for other work.)

This will break if any files or subdirectories have spaces in their names.

%.jpg.webp: %.jpg is a pattern rule.

Answer 4

The problem isn't the subshell, per se, it's the fact that you are looping over the output using the shell to iterate over and check the files. The shell is slow, so it would be faster to just run find to get the files once, and then iterate over the list of target files. Something like this:

find . -name '*jpg' > jpgs
find . -name '*jpg.webp' > webps
awk 'NR==FNR{webps[$0]++; next} !($0".webp" in webps)' webps jpgs | 
  head -n 10000 | ...

Or even

awk 'NR==FNR{webps[$0]++; next} !($0".webp" in webps)' webps jpgs > target.files
## split into lists of 10000 file names each (assuming no newlines in your file names)
split -l 10000 target.file list
## Process each list
for list in list*; do
  while read jpg; do
    whatever "$jpg" > "$jpg".webp
  done < "$list"
done

But since you're dealing with millions of files, you probably want to use GNU parallel, assuming you have access to it, so you can run the commands in parallel. Assuming the command you use to create the .webp is command foo.jpg > foo.webp, you would do:

find . -name '*jpg' > jpgs
find . -name '*jpg.webp' > webps
awk 'NR==FNR{webps[$0]++; next} 
    !($0".webp" in webps){ 
       printf "command \"%s\" > \"%s\".webp\n",$0,$0 
}' webps jpgs > script.sh
parallel -j 30 < script.sh

That will create the script script.sh with all the conversions you want to run, and then would execute them running 30 conversions in parallel at a time.

Answer 5

Would there be some way to do this using find only?

Without considering performance and timing, this is the simplest find command to perform this task:

find "$path" -type f -iname "*.jpg" ! -exec test -e '{}.webp' \; -print

It probably won't be as fast as the other answers, but just as a reference.

By the way, if you're only looking for files that end with lowercase jpg, it's better to use -name (case sensitive) instead of -iname (case insensitive) which might be be a little bit slower, especially for millions of files.

Answer 6

If you use zsh, you can do this with a glob:

echo **/*.jpg(.e:'! [ -e "$REPLY.webp" ]':)

Or with the max count, if you want that:

echo **/*.jpg(.Y10000e:'! [ -e "$REPLY.webp" ]':)

How exactly you use it depends on what you want to do, how your generation script/binary args work, and whether you hit the max command line length, so give details for followup questions.