0

I had wrote a simple script with should read a list of URL string from a file into a variable and prepare it for the future processing.

Content of the dl.txt file:

https://some-url.com/site1
https://some-url.com/site2

and so on...

Script:

#!/bin/bash

IFS="\n"

while read -a line; do
    rl=.${line}

    echo -e "\n"
    echo $rl
    echo -e "\n"

#   for a in ${line}; do
#       echo -e $a
#   done
done <<< $(cat "./dl.txt")

As the result, the each from the read strings is beeing cut to exactly 10 characters from the begining.

Any idea how to solve it ?

Mcgiwer
  • 61
  • 3
  • As a general note, it is unnecessarily convoluted to feed to a `while` loop the content of a variable that captures the output of `cat`ing a text file - just say `while read ... done < dl.txt`. – AdminBee Nov 15 '21 at 12:17
  • What is the "future processing" that you are planning to do? It is almost never necessary to store all the data in intermediate variable(s) before processing it. If you are going to call `curl` with these URLs, just do `xargs curl – Kusalananda Nov 22 '21 at 13:03
  • Why are you using `read -a`? The `-a` option makes `read` create an array by splitting the line on the characters in `$IFS`, but AIUI you're trying to set that to newline, and newline will never occur within a line (by definition, since it defined the end of the line). Furthermore, you then use the variable as if it weren't an array at all (`${line}` just gets the first entry of an array), making it doubly useless. What are you actually trying to accomplish with the `-a`? – Gordon Davisson Nov 22 '21 at 17:17
  • Running this script through `[shellcheck.net](https://www.shellcheck.net/)` will point out several of the problems/confusions involved. I strongly recommend running all your scripts through it and fixing the things it points out. – Gordon Davisson Nov 22 '21 at 17:19

2 Answers2

2
IFS="\n"

This sets IFS to the two characters \ (backslash) and n (lowercase letter N). Which means that later on, read will split the input it gets on either of those. Now, the only line in your example with an n is the and so on..., but that's enough to see the result.

You probably meant to use

IFS=$'\n'

but as rightly noted in the comments, that wouldn't make much sense either, since read would read only one line ending in a newline, then try to split into fields using newline as the separator. There won't be any in the later step.

If you want to loop over the lines of the input file, use

while IFS= read -r line; do ...

or if you want to read them to an array and loop over that:

mapfile -t lines < ...
for line in "${lines[@]}"; do ...

but see Why is using a shell loop to process text considered bad practice?

Also, this is a bit suspect:

 done <<< $(cat "./dl.txt")

In general, the unquoted expansion would split the output from cat into multiple words, but I think word splitting doesn't happen on the right-hand side of <<<. But it would happen on the right side of a < (possibly erroring with "ambiguous redirect"), so I would suggest putting that in quotes.

Or just use a process substitution:

... < <(cat "./dl.txt")

or just drop the cat:

... < dl.txt
ilkkachu
  • 133,243
  • 15
  • 236
  • 397
  • `IFS=$'\n'` wouldn't make sense either. There's no way `read` can split on newline characters since it's reading only one line (could read more than one as `-r` was omitted, but even then the newlines separating those would not be included in the resulting text). Here, it looks like the OP wants `IFS= read -r line`, but that looks a lot like a case of [Why is using a shell loop to process text considered bad practice?](//unix.stackexchange.com/q/169716) – Stéphane Chazelas Nov 15 '21 at 12:28
  • 1
    About *but I think word splitting doesn't happen on the right-hand side of <<<*. It used to happen in older versions of bash (where the text would be split (not globbed) and resulting words joined back with spaces). That was later fixed. The `<<<` operator itself is from zsh which never had the problem. – Stéphane Chazelas Nov 15 '21 at 12:30
  • @StéphaneChazelas, d'oh, thanks again. – ilkkachu Nov 15 '21 at 12:33
  • the "< dl.txt" line you had typed overwrites the readed value and it display only one line. – Mcgiwer Nov 22 '21 at 08:23
  • @Mcgiwer, overwrites? what? – ilkkachu Nov 22 '21 at 09:13
  • @ikkachu I had tested it and wasn't working at it should. The problem was solved by replacing double quotes with single one in IFS variable – Mcgiwer Dec 03 '21 at 19:54
-1

For bash:

while read line; do
        array+=(${line:10})
done < dl.txt
echo ${array[@]}
nezabudka
  • 2,376
  • 5
  • 15
  • 1
    You're dropping whitespace all over the place. Use `while IFS= read -r line`; double-quote variables whenever you use them `"${line:10}"` and `"${array[@]}"`. But I don't see the difference between all that complexity and `cut -c11- dl.txt` – roaima Nov 23 '21 at 20:54
  • First post, not clear enough. I assumed that there are actions in the loop that are not demonstrated. From his answer, it is clear that the output needs an array. This is a concept - a template for further actions. – nezabudka Nov 23 '21 at 21:09