Use shell's "*" glob, but exclude one file and don't match directories?

Question

I've got a makefile rule that builds a zip/tarbar for distribution. In the recipe, it does some "value added" things, like ensure CR/LF's are correct, and ensures execute bits are correct before packaging.

The project has a buffet of files, but here are the requirements: (1) all files except GNUmakefile need CR/LF, (3) GNUmakefile needs LF only, (3) all files except GNUmakefile needs a-x.

Here's what the recipe looks like:

.PHONY: convert
convert:
    chmod a-x *[^GNUmakefile*] TestData/*.dat TestVectors/*.txt
    unix2dos --keepdate --quiet *[^GNUmakefile*] TestData/*.dat TestVectors/*.txt
    dos2unix --keepdate --quiet GNUmakefile

I'm using * and trying to avoid explicitly listing all the files in the buffet because some are non obvious, like IDE specific files. (*[^<somefile>*] is a neat trick; I got that from Exclude one pattern from glob match).

The problem is I'm matching TestData and TestVectors when performing chmod a-x, so I exclude myself from the directories.

I need to refine things, but I'm not sure how. I want to use the shell's "*" glob, but exclude one file and don't match directories.

How should I proceed?

score 5 · Answer 1 · answered Oct 05 '15 at 15:52

I'd solve this problem by using GNU Make's filter-out and wildcard functions. The only part of your task that you can't do with them is filter out directories; that has to be done via the shell. Code below is untested and assumes (a) no whitespace or shell metacharacters in any filename, (b) TestData/*.dat and TestVectors/*.txt do not need to be checked for directories.

NORM_TOPLEVEL := $(shell for f in $(filter-out GNUMakefile,$(wildcard *)); \
                   do [ -d "$$f" ] || printf '%s\n' "$$f"; done)
NORM_TESTDIRS := $(wildcard TestData/*.dat) $(wildcard TestVectors/*.txt)

convert:
    chmod a-x $(NORM_TOPLEVEL) $(NORM_TESTDIRS)
    unix2dos --keepdate --quiet $(NORM_TOPLEVEL) $(NORM_TESTDIRS)
    dos2unix --keepdate --quiet GNUmakefile

.PHONY: convert

Stéphane Chazelas · Answer 2 · 2015-10-05T13:19:33.673

Only zsh has globs where you can select files by type, so, assuming GNU make, you'd need something like:

SHELL = zsh
.SHELLFLAGS = -o extendedglob -c

test:
        echo ^GNUmakefile(^/)

^GNUmakefile (with extendedglob) is for non-hidden files other than GNUmakefile. (^/) is a glob qualifier that selects files of any type other than directory. See also (.) for files of type regular (excludes directories and all other non-regular types of files like fifos, symlinks, sockets...) which seems more like what you're looking for. Add the D glob qualifier (^GNUmakefile(.D)) to include hidden (Dot) files like .gitignore.

Note that *[^GNUmakefile*] expands to the list of non-hidden file names that end in a character other than G, N, U, m, a, k, e, f, i, l, or *. So it will indeed exclude GNUmakefile (since it ends in e), but also foo.a or file.html or bar.exe.

To do the same without changing the shell, you'd need to resort to a loop like (here for the equivalent of ^GNUmakefile(.)):

test:
        set -- *; \
        for i do \
          [ -f "$$i" ] && \
            [ ! -L "$$i" ] && \
            [ "$$i" != GNUmakefile ] && \
            set -- "$$@" "$$i"; \
          shift; \
        done; \
        [ "$$#" -gt 0 ] && echo "$$@"

(replace set -- * with set -- .* * to include hidden files).

Best there would probably be to resort to find instead of shell globs if you can't guarantee availability of zsh:

test:
        find . ! -name . -prune ! -name '.*' ! -name GNUmakefile \
          -type f -exec echo {} +
        find TestData/. ! -name . -prune -name '*.dat' ! -name '.*' \
          -type f -exec echo {} +

(remove ! -name '.*' to include hidden files).

I have to be careful of calling out shells (maybe I'm implicitly doing it already...). I would be OK with something like `SHELL ?= /bin/zsh`, but GNUmakefile takes away that freedom (how ironic; see [Where is /bin/bash being changed to /bin/sh?](http://stackoverflow.com/q/32474818)). — , Oct 05 '15 at 10:25
@jww, here you **do** want to force the shell to be `zsh`, so `SHELL ?= /bin/zsh` would not make sense. Note that `make`'s `$SHELL` is never inherited from the environment's `$SHELL`. Those two variables have different meanings (one for the shell used to interpret shell command lines in `make` (`sh` by default), the other one for the user's preferred choice of interactive shell (for things like `xterm` or `vi`...)). — Stéphane Chazelas, Oct 05 '15 at 10:40
Be careful with using zsh as shell for makefiles. zsh is not POSIX or Bourne Shell compatible when called as `zsh`. As a result, you cannot even run configure with zsh. — schily, Oct 05 '15 at 11:15
@schily, that's true, though here we explicitly use it for a non-POSIX feature that does break POSIX compliance so it should be quite obvious (`echo ^GNUmakefile` is required by POSIX to output `^GNUmakefile\n` which `zsh` obviously fails on here (as would the Bourne shell btw)). — Stéphane Chazelas, Oct 05 '15 at 11:29

Alexander · Answer 3 · 2015-10-05T12:49:13.043

0

Using Python makes it easy to expand the expression if you wish to filter for more advanced criteria. Python is installed almost everywhere. It's a bit verbose, but it works.

Example Makefile:

all:
    chmod a-x $(shell python -c "import glob; import os.path; print(' '.join([x for x in glob.glob('TestData/*.dat') + glob.glob('TestVectors/*.txt') if not (os.path.isdir(x) or x == 'GNUmakefile')]))")

edited Oct 05 '15 at 12:49

answered Oct 05 '15 at 11:47

Alexander

9,607
3
40
59

Its not clear to me how this works in a GNUmakefile. Could you provide the makefile recipe? – Oct 05 '15 at 11:53
Sure. Updated my answer. – Alexander Oct 05 '15 at 11:58
Thanks Alexander. I'll have to think about adding the Python dependency. – Oct 05 '15 at 13:00
That assumes none of the file names contain characters special to the shell (blanks, `$`, `&`, globs...). Could be interesting if there was a file called `$(rm -rf "$HOME")` for instance. – Stéphane Chazelas Oct 05 '15 at 13:34
Also not that that `$(shell...)` is expanded at the time that target is processed. So if you have a `all:cc -o file file.cchmod...`, that `file` created by `cc -o file file.c` won't get included. – Stéphane Chazelas Oct 05 '15 at 13:48
Stéphane Chazelas, adding filters for the filenames would be easy. Do you have an example for how to create a file named `$(rm -rf "$HOME")`? My system would not allow the creation of a file with that filename (at least not using `touch` or `vim`). The second issue you raise should be possible to solve either with Make or, in worst case, by wrapping the line in a small shell script. – Alexander Oct 07 '15 at 12:47

Use shell's "*" glob, but exclude one file and don't match directories?

3 Answers3