How to extract folders, split filenames in key and value(s) and save it in an associative array?

Question

I have following script which is supposed to solve the problem from the title, but obviously it isn't working to assign the values to the keys. Is the reason an accidental error or is there a substantial mistake in the script?

The foldernames are names of gem-files like gem-file-foo-1.2.3

The key is supposed to be gem-file-foo in this example and the value(s) the version number 1.2.3 or a string of multiple version numbers if there are multiple versions of the same gem.

It doesn't output any keys with echo "${!my_gems[@]}" ... why not?

#!/bin/bash

directory=$GEM_HOME/gems
declare -A my_gems

get_gemset_versions () {

last_key=""
values=""

FIND_RESULTS=$(find $directory -maxdepth 1 -type d -regextype posix-extended -regex "^${directory}\/[a-zA-Z0-9]+([-_]?[a-zA-Z0-9]+)*-[0-9]{1,3}(.[0-9]{1,3}){,3}\$")

 printf "%s\n" $FIND_RESULTS | sort | 
  while read -r line; do
    line=${line##*/}
            
    KEY="${line%-*}"
            
    VALUE="${line##*-}"
  
        if [[ $last_key -eq "" ]]; then
            last_key=$KEY
        fi
        
        if [[ $last_key -eq $KEY ]]; then
            values="$values ${VALUE}"
        else
            values="${VALUE}"
            last_key=$KEY
        fi
        
        my_gems[$KEY]=$values
    done
 
    echo "${!my_gems[@]}"

 }


get_gemset_versions

Also, the logic with $last_key and $key to summerize equal gem-packages seems to be faulty. This is not necessarily part of the question, but it would be nice if you would point out if I am applying some faulty logics here.

Thanks

`-eq` is an *integer* equality operator; string equality is `=` (POSIX) or `==` (bash extended test) — steeldriver, May 29 '21 at 12:41
Thanks. It seems like the input to `$my_gems[key]` happened just locally and isn't available anymore after the while loop ??? — von spotz, May 29 '21 at 13:33
Indeed - see for example [Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?](https://unix.stackexchange.com/questions/9954/why-is-my-variable-local-in-one-while-read-loop-but-not-in-another-seemingly) — steeldriver, May 29 '21 at 13:41
Ok thanks. Could you maybe please tell me how I can save the output from `printf "%s\n" $FIND_RESULTS | sort` in a variable such that I can use the redirecting syntax to give input to the `while` loop? Or how I can manipulate the process substitution syntax to sort the result-set of `find` (see the second solution here https://stackoverflow.com/a/67739521/4307872) ? — von spotz, May 29 '21 at 13:51
@vonspotz, there's a few workarounds in the answers to the question steeldriver linked. (Also, `$(...)` is command substitution, not process substitution which is a different thing.) — ilkkachu, May 29 '21 at 13:53
Yes I read those. But just doing command grouping isn't really a solution. — von spotz, May 29 '21 at 13:55
You could use a *process substitution* `while IFS= read -r line; do ; done < <(find | sort)`. Best practice would make the whole thing null terminated to work with all legal filenames i.e. `find -print0` / `sort -z` / `read -d ''` — steeldriver, May 29 '21 at 13:56
@vonspotz, I'm pretty sure I wrote [an answer there](https://unix.stackexchange.com/a/407802/170373) that mentions `lastpipe` and process substitution. :) But then they're not standard features. — ilkkachu, May 29 '21 at 14:08

ilkkachu · Accepted Answer · 2021-05-29T14:23:34.240

You have:

printf "%s\n" $FIND_RESULTS | sort | 
  while read -r line; do
    ...
    done
 
    echo "${!my_gems[@]}"

where, regardless of the indentation, the echo is outside the pipeline. Bash runs all parts of a pipeline in subshells by default, so the assignments within the while loop aren't visible after the pipeline ends. Shellcheck.net also warns about that:

Line 32:
        my_gems[$KEY]=$values
        ^-- SC2030: Modification of my_gems is local (to subshell caused by pipeline).

Sadly it doesn't give workarounds.

In Bash, you can either enable the lastpipe option, or replace the pipe with process substitution:

shopt -s lastpipe
echo test | while read line; do
    out=$line
  done
echo "out=$out"

or

while read line; do
  out=$line
done < <(echo test)
echo "out=$out"

(lastpipe probably won't work if you try it in an interactive shell, as it's tied to job control not being enabled.)

See: Why is my variable local in one 'while read' loop, but not in another seemingly similar loop?

In any case, this seems a bit odd:

FIND_RESULTS=$(find ...)

 printf "%s\n" $FIND_RESULTS

find outputs filenames separated by newlines, which is fine as long as you know no filenames contain any. But here, the round-trip through the variable and the word splitting from the unquoted expansion also splits any filenames with spaces.

You could just run find ... | while ... directly. Or while ...; done < <(find...).

Also, note that you almost always want to use while IFS= read -r line; do, to prevent read from breaking leading and tailing whitespace. Well, I hope your filenames don't contain those either, but in any case.

I can't find a good reference question right now, but that's specific to IFS containing whitespace. Other leading and trailing separators are not removed with a read to just one field. E.g. IFS=": " read -r foo <<< "::foobar " leaves foo with the literal ::foobar. The colons are kept, but the trailing spaces gone.

you can use grouping braces: `pipeline | { while read line; do out=$line; done; echo $line; }` but that only extends the availability of the variable a little bit. — glenn jackman, May 31 '21 at 15:24

How to extract folders, split filenames in key and value(s) and save it in an associative array?

1 Answers1