1

So, i'm using bash to read a file (that has no newlines, space or tab).

Something like this:

aababcbbcbckqkkqkqhddhkehkjhqkjhsdk
skjhqkdjhqkzdhkzhdkjqzhdhqkjhzdkqzh

though there is a newline in this example, there isn't on the data i'm working on... So i found that, since everything was technically seen as on "the same line" (since there isn't any newline or any delimiter i could use), i tried to read every Nth chars:

while read -N129999 character; do
  program "$character"
done < <(cat file | tr -d '\n')

(I'm aware of the "useless use of cat")

The number i'm using here is just the maximum i found that i could make read use, (which also i prefer doing since it process the file faster) The program here is just an example and for the sake of illustration. And i'm purposely removing newline, tabs and space on the aforementioned data.

Now while what i provided work, it doesn't quite work for the last part of the file, which contain less than the aforementioned number...and while i'm aware that, if there was a delimiter provided to IFS, the -n option could continue after that instead of ignoring the rest of file which doesn't fit the range of characters...

How do i (in bash, sed, or any posix tools?) read every Nth range of characters, while including the rest of the file/input, which doesn't fit that range?

Nordine Lotfi
  • 2,200
  • 12
  • 45
  • Manual for bash read says -N terminates if EOF is encountered. How big is the file? awk can read the whole thing into memory and substr it to pass to shell, but doing the whole job in bash would be way faster for a huge file. – Paul_Pedant Jun 10 '20 at 22:34
  • Yeah, it is decently huge (think in a couple gigabytes). I tried different method to read the file (xargs, pure bash, etc) and found that, a while loop was the fastest. – Nordine Lotfi Jun 10 '20 at 22:36
  • Also, i did try the -N option, but encountered the same problem in my case @Paul_Pedant ... – Nordine Lotfi Jun 10 '20 at 22:36
  • 1
    @NordineLotfi, there's also a Perl option: `perl -e 'system("somecmd", $a) while (sysread STDIN, $a, 9999)' < <(...)` Bash's `read` only ever reads 128 bytes at a time for some reason (even with `-N`, I tested), so it can suffer from a bit more overhead than most utils there. I have no idea if that makes any meaningful difference here, though. – ilkkachu Jun 10 '20 at 22:56
  • Thanks @ilkkachu didn't consider a perl option! Though, didn't know that bash did that (i noticed that `read` accept no more than 129999 characters though, probably related). And yes, it does have a meaningful difference i believe (at least for me), since, more chars read, faster processing! – Nordine Lotfi Jun 10 '20 at 23:03

1 Answers1

5

You're seeing the same issue that comes up when reading text files that are missing the newline at the end of the last line. read returns a falsy status when it hits end-of-file before seeing the delimiter, or here, without reading the required number of characters. That is, even if it did read something before that. It does set the output variable though, so you can check if it's non-empty.

$ printf abc | while read -N2 x || [ "$x" ] ; do echo "read: $x"; done ;
read: ab
read: c

If you don't have that condition, you could still see the final part of the input appear in the output variable, just after the loop exited.

$ printf abc | ( while read -N2 x; do echo "read: $x"; done ; echo "finally: $x" )
read: ab
finally: c

Related: What does `while read -r line || [[ -n $line ]]` mean?

ilkkachu
  • 133,243
  • 15
  • 236
  • 397