what does paste -d@ mean in this context. also confused with IFS=@

Question

context: https://stackoverflow.com/questions/52299524/duplicate-output-looping-through-multiple-values-in-while-loop-bash/52301168#52301168

Nota: this will work simply while each file do contain 1 word by line.

And if you plan to use space in your source files, you could use another delimiter:

while IFS=@ read -r aws_user_name aws_key aws_account_num ;do
    printf "User: %-16s Key: %-22s AccNum: %s\n" \
        "$aws_user_name" "$aws_key" "$aws_account_num"
    # ....
done < <(
    paste -d@ "$aws_users_all" "$aws_env_list" "$aws_account_numbers"
)

I am not sure IFS=@ paste -d@ mean? I understand paste -d "|,", I can aslo figure it out IFS=";" mean. I also following https://stackoverflow.com/questions/7427262/how-to-read-a-file-into-a-variable-in-shell https://man7.org/linux/man-pages/man1/bash.1.html

roaima · Answer 1 · 2021-12-24T11:17:36.100

4

Since you understand paste -d "|," then you can also infer the meaning of paste -d@ - it's the same process - instead of separating fields with |, they're separated with @. (In common with many UNIX/Linux tools a flag and its argument can be joined together or separated with a space; standard parsing libraries don't care - see POSIX Utility Argument Syntax, point 2a.)

In this situation the IFS variable determines the way that the read verb parses its input. The variables aws_user_name, aws_key, and aws_account_num are read from the output of paste -d@ and split at the @ symbol that paste has just added.

You can partially disect the command to see what's going on; running the paste command (with appropriate filenames for the three variables) will show you what the while read … loop is going to process

paste -d@ "$aws_users_all" "$aws_env_list" "$aws_account_numbers"

Be aware that this kind of processing will break if any of the source values contains the delimiter (@).

edited Dec 24 '21 at 11:17

answered Dec 24 '21 at 11:06

roaima

107,089
14
139
261

Something maybe worth mentioning is that it's important that the delimiter used here be a single-byte and non-whitespace one. Single-byte because several `paste` implementations including GNU `paste` don't support multibyte ones (and some shells, not bash, don't support them in IFS either), and non-white space ones because the whitespace ones receive a special treatment when it comes to `$IFS` splitting. So `@` is a good choice because it's single byte and non whitespace (and presumably cannot occur in the data) but `paste`'s default delimiter (TAB) would be a poor choice. – Stéphane Chazelas Dec 24 '21 at 11:35
Ah, yes. I do appreciate your knowledge here; I'd not seen (i.e. _read and understood the detail of_), "_If IFS has a value other than the default, then sequences of the whitespace characters space, tab, and newline are ignored at the beginning and end of the word, as long as the whitespace character is in the value of IFS_" until now. – roaima Dec 24 '21 at 14:07
Since 5.0, it's all single byte characters classified as whitespace that receive the special treatment like in ksh93. POSIX requires it to be all whitespace characters, but AFAIK, yash is the only shell that honours that requirement. Traditionally (in ksh which introduced that behaviour and most other shells), that was limited to SPC, TAB and NL. ksh93 and zsh allow the special treatment to be removed by doubling the character in `$IFS` (with `IFS=$'\t\t'`, you would be able to use the default `paste` delimiter there). – Stéphane Chazelas Dec 24 '21 at 14:20

Stéphane Chazelas · Accepted Answer · 2021-12-24T11:36:46.180

FWIW, more as a comment than an answer (which has already been given by @roaima), it would have made more sense to write it:

while
  IFS= read -r aws_user_name <&3 &&
    IFS= read -r aws_key <&4 &&
    IFS= read -r aws_account_num <&5
do
  printf "User: %-16s Key: %-22s AccNum: %s\n" \
    "$aws_user_name" "$aws_key" "$aws_account_num"
    # ....
done 3< "$aws_users_all" 4< "$aws_env_list" 5< "$aws_account_numbers"

Instead of joining with @ to later split again on @.

That way, having @ in values doesn't cause problems, and missing elements in the 3 lists are correctly detected. It also avoids the <(...) kshism, making it standard sh code.

If all the loop does is print stuff, it would even have made more sense to use awk / perl or some proper text processing utility than running several invocations of some clunky tools in a loop like that. More on that at Why is using a shell loop to process text considered bad practice?

what does paste -d@ mean in this context. also confused with IFS=@

2 Answers2

Linked