The best way to expand glob pattern?

Question

I need to expand a glob pattern (like ../smth*/*, or /etc/cron*/) into a list of files, programmatically. What would be the best way to do it?

You don't need to do anything special, just don't quote the `*`. — Kevin, Mar 12 '12 at 14:32
Though if you're going to be trying to parse it, use an array like the answer says. — Kevin, Mar 12 '12 at 14:34

score 75 · Accepted Answer · edited Oct 04 '21 at 15:31

75

Just let it expand inside an array declaration's right side:

list=(../smth*/)          # grab the list
echo "${#list[@]}"        # print array length
echo "${list[@]}"         # print array elements
for file in "${list[@]}"; do echo "$file"; done  # loop over the array

Note that the shell option `nullglob` needs to be set. **It is not set by default.** It prevents an error in case the glob (or one of multiple globs) does not match any name.

Set it in bash with

shopt -s nullglob

or in zsh or yash with

set -o nullglob

though in zsh (where the nullglob initially came from), you'd rather use the (N) glob qualifier to avoid having to change a global setting:

list( ../smth*/(N) )

The ksh93 equivalent:

list=( ~(N)../smth*/ )

edited Oct 04 '21 at 15:31

Stéphane Chazelas

522,931
91
1,010
1,501

answered Mar 12 '12 at 13:37

manatwork

30,549
7
101
91

And how do I print that list afterwards? – Rogach Mar 12 '12 at 13:38
It is just a regular array. You can do whatever you can with any array. Added some examples. – manatwork Mar 12 '12 at 13:44
1

There is a problem. If pattern matches no files, it prints itself - which is not very good. – Rogach Mar 12 '12 at 14:21
6

Ah, shopt -s nullglob solves it. – Rogach Mar 12 '12 at 14:25
How do I undo `setopt` in zsh? – lindhe Dec 05 '15 at 18:38
1

@lindhe, `unsetopt` and the same parameter as used for `setopt`. – manatwork Dec 05 '15 at 18:48
1

Having the pattern print itself is probably what is wanted in some cases. Using the option `failglob` might be more appropriate in others. See [this answer](https://unix.stackexchange.com/a/204944/85414) for an in-depth discussion. – SpinUp __ A Davis May 06 '22 at 20:41

Walf · Answer 2 · 2023-06-28T03:29:50.670

8

compgen is a Bash built-in that you can pass an escaped(!) pattern to, and it outputs matches, returning true or false based on whether there were any. This is especially useful if you need to pass the glob pattern from a variable/script argument.

glob_pattern='../smth*/*'
while read -r file; do
    # your thing
    echo "read $file"
done < <(compgen -G "$glob_pattern" || true)

adding the || true prevents a false return from compgen causing any problems. This method avoids issues with no matches and does not require changing nullglob options.

If you need the items in an array, just initialise one with files=() before the loop, and files+=("$file") inside the loop. You can then see if there were any matches by simply checking the length of the array with if [[ ${#files[@]} -gt 0 ]]; then.

edited Jun 28 '23 at 03:29

answered Aug 12 '20 at 03:34

Walf

1,254
1
14
19

I used to think this was a satisfactory solution, but it turned out not to be. For instance, `compgen` does not work properly for files with "composed extensions"; i.e. if you have a file `file.txt.bin`, with two extensions, expanding `**/*.bin` will weirdly not match them. In my case I was trying to expand the glob in the context of a Git script. Luckily, `git ls-files "$glob"` works just as I'd expect. While not a general solution outside of Git trees, I thought I might as well point it out here. – resolritter Feb 24 '21 at 11:40
2

@resolritter I just ran `compgen -G '/**/*.gz'` (on bash v4.2.46) and it found several files with composed extensions. I also have the shell option `globstar` `off`. Are you saying `compgen -G '**/*.bin'` produces no output where `echo **/*.bin` does? – Walf Feb 25 '21 at 03:26

score 2 · Answer 3 · answered Oct 04 '21 at 14:39

I wanted to use a standard input (pipe) in case a resulting command exceeds a command line length limit. The following command worked for me:

echo "../smth*/*" "/etc/cron*/" | xargs -n1 -I{} bash -O nullglob -c "echo {}" | xargs -n1

or for a list of globs:

cat huge_glob_list.txt | xargs -n1 -I{} bash -O nullglob -c "echo {}" | xargs -n1

score 1 · Answer 4 · answered Feb 08 '19 at 17:28

1

No need to overcomplicate things:

echo your/stuff*

answered Feb 08 '19 at 17:28

Alexei Averchenko

135
2

3

this doesn't work. For example: `TEST=$(echo your/stuff*) && eval \"$TEST\"` will output: `your/stuff*: No such file or directory` – Sebastian Jun 27 '19 at 17:41
2

@Sebastian this is a nullglob issue, see above – Alexei Averchenko Jul 04 '19 at 13:52
7

No, it's not a nullglob issue. Using scape characters is evaluating `TEST` variable as a string including `*` and not being expanded. – Sebastian Jul 05 '19 at 20:14

score 0 · Answer 5 · answered Jan 16 '23 at 08:10

Nowadays most linux distributions have python included, so you can just run the following command in shell

python -c 'from glob import glob; print(glob("*"))'

You are free modify the python script to meet your requirement, for example, dump to json format string.

python -c 'from glob import glob; from json import dumps; print(dumps(glob("*")))'

The best way to expand glob pattern?

5 Answers5