Pattern matching on path names in bash

Question

I want to act on a list of subdirectories in a directory. Consider:

for x in x86-headers/*/C/populate.sh; do echo $x; done

This gives

x86-headers/elf/C/populate.sh
x86-headers/gl/C/populate.sh
x86-headers/gmp/C/populate.sh
x86-headers/gnome2/C/populate.sh
x86-headers/gtk2/C/populate.sh
x86-headers/jni/C/populate.sh
x86-headers/libc/C/populate.sh

But I want values that corresponds to the second part of the path, elf, gl, etc. I know how to strip off the leading x86-headers.

for x in x86-headers/*/C/populate.sh; do i=${x##x86-headers/}; echo $i; done

which gives

elf/C/populate.sh
gl/C/populate.sh
gmp/C/populate.sh
gnome2/C/populate.sh
gtk2/C/populate.sh
jni/C/populate.sh
libc/C/populate.sh

I also know how to strip off later terms in the path. I.e. going down one level:

cd x86-headers
for x in */C/populate.sh; do i=${x%%/*}; echo $i; done

gives

elf
gl
gmp
gnome2
gtk2
jni
libc

But, trying to combine these doesn't work. I.e.

for x in x86-headers/*/C/populate.sh; do i=${${x##x86-headers}%%/*}; echo $i; done

gives

bash: ${${x##x86-headers}%%/*}: bad substitution

No doubt this is incorrect syntax, but I don't know the correct syntax. Of course there might be better ways to do this. If I was using Python, I'd use split on /to break each path into a list, and then pick the second element, but I don't know how to do that in bash.

EDIT: Thanks for the answers. I should also have asked, is it possible to do this portably, and if so, how?

Answer 1

You can't combine them with bash (or POSIXly), you have to do it in two steps.

i=${x#*/}; i=${i%%/*}

That's portable to any POSIX shell. If you need portability to the Bourne shell (but why would you tag your question /bash then?) as well in case you're porting to Solaris and are forced to use /bin/sh instead of the standard sh there, or porting to 20 year old systems), you could use this approach (which will work as well with POSIX shells):

IFS=/; set -f
set x $x
i=$3

(above is one of the very rare cases where it makes sense to leave a variable unquoted).

Just for the record, with zsh:

print -rl -- x86-headers/*/C/populate.sh(:h:h:t)

(the tail of the head of the head of the file).

Or for your python approach:

x=elf/C/populate.sh
i=${${(s:/:)x}[2]}

Answer 2

Parameter expansion doesn't allow you to nest expressions, so you're forced to do it in two steps, with an assignment:

i=${x##x86-headers/}
i=${i%%/*}

You do have the option of using arrays here, but I'd think it'd be more cumbersome than the above PE:

IFS=/
set -f # Disable globbing in case unquoted $x has globs in it
i=( $x )
set +f
echo "${i[1]}"

Then there's the third option of using external commands. With a short list of files this is usually the least efficient option, but it will scale the best as the number of files grows:

# For clarity, assume no names have linebreaks in them
find . -name populate.sh | cut -d / -f 3

Answer 3

regex="x86-headers/([^/]*)/.*"
for f in x86-headers/*/C/populate.sh; do [[ $f =~ $regex ]] && echo "${BASH_REMATCH[1]}"; done
elf
gl
gmp
gnome2
gtk2
jni
libc

Answer 4

Be very careful using a construct as bellow:

# What happen if no files match ?
for x in x86-headers/*/C/populate.sh; do
  x=${x##x86-headers/}
  x=${x%%/*}
  echo $x
done

As you will could end up doing an echo *. In this case, it is only funny, but it could be much worst if your are root, your CWD is / and you want to delete some sub directories of /tmp instead of echoing them !

To fix that in bash, you could do:

shopt -s nullglob
for x in x86-headers/*/C/populate.sh; do
  x=${x##x86-headers/}
  x=${x%%/*}
  echo $x
done
shopt -u nullglob

Notice the shopt -u nullglob at the end to restore default bash behavior after the loop.

A more portable solution could be:

for x in x86-headers/*/C/populate.sh; do
  [ -e $x ] || break
  x=${x##x86-headers/}
  x=${x%%/*}
  echo $x
done

(The ## and %% Parameter Substitutions are valid in ksh88).

Follow this link for a more complete discussion of nullglob.

Notice that the 3 solutions above fail if you have an intermediary directory with a space in its name. To solve it, use double-quotes in the variable substitution:

for x in x86-headers/*/C/populate.sh; do
  [ -e "$x" ] || break
  x="${x##x86-headers/}"
  x="${x%%/*}"
  echo "$x"
done

Answer 5

To do pattern matching on path names portably you may set the IFS shell variable to / and then use shell parameter expansion.

Doing all this in a subshell helps keeping the parent shell's environment unchanged!

# cf. "The real Bourne shell problem",
# http://utcc.utoronto.ca/~cks/space/blog/programming/BourneShellLists

paths='
x86-headers/elf/C/populate.sh
x86-headers/gl/C/populate.sh
x86-headers/gmp/C/populate.sh
x86-headers/gnome2/C/populate.sh
x86-headers/gtk2/C/populate.sh
x86-headers/jni/C/populate.sh
x86-headers/libc/C/populate.sh
'

IFS='
'

# version 1
for x in $paths; do 
   ( IFS='/'; set -- ${x}; echo "$2" ); 
done


# version 2
set -- ${paths}
for x in $@; do 
   ( IFS='/'; set -- ${x}; echo "$2" ); 
done