How to avoid losing my escape characters when reading in variables from a file with bash

Question

I have a lab ntlm-extract.ntds file which has usernames and hashes in the format: domain\username:integer:hash:hash2

For example: somedomain.local\jcricket:5201:0020cfaecd41954fb9c9da8c61ccacd7:0020cfaecd41954fb9c9da8c61ccacd7

I'm comparing the hashes in the LINE[3]/hash2 column with hashes in the NTLM HIBP database, then I'd like to print out usernames that have matches, but the domain\username keeps losing the \ whatever I try, and I'm not sure if it's on the read or write that it loses it.

The script I have so far is:

#!/usr/bin/bash

while read line
do
    IFS=':' read -ra "LINE" <<< ${line}
    HASH=${LINE[3]}
    HASH=${HASH^^}
    printf "Checking for %s\n" $HASH
    found=(`grep "$HASH" "./pwned-passwords-ntlm-ordered-by-hash-v7.txt"`)
    if [ -n $found ]; then
        printf "Match on username %s\n" "${LINE[0]}"
    fi
done < "ntlm-extract.ntds"

Following the recommendations the final working script ended up being:

#!/usr/bin/bash

numoflines=(`wc -l ntlm-extract.ntds`)
numcomp=0
while IFS= read -r line; do
    IFS=: read -ra hashline <<< "${line}"
    passhash="${hashline[3]}"
    printf "Checking for %s\n" $passhash
    printf "Line %d of %d\n" $numcomp $numoflines
    numcomp=$((numcomp+1))
    found=''
    found=(`grep -m 1 -i "$passhash" "./pwned-passwords-ntlm-ordered-by-hash-v7.txt"`)
    wait 
    if [ -z "$found" ]; then
        continue
    else
        printf "found return value is %s\n" "$found"
        printf "%s\n" "${hashline[0]}" >> ./hibp-busted.txt
    fi
done < "ntlm-extract.ntds"

Not sure if the "wait" is necessary though – flerb Jun 06 '21 at 01:29 — flerb, Jun 06 '21 at 01:29

score 2 · Accepted Answer · answered Jun 05 '21 at 23:47

2

You need -r on the outer read, not just on the inner read -a. You should also quote "${line}" and (probably) want IFS= unless you explicitly want to strip leading whitespace:

while IFS= read -r line; do 
  IFS=: read -ra LINE <<< "${line}"; printf '%s\n' "${LINE[@]}"
done < ntlm-extract.ntds
somedomain.local\jcricket
5201
0020cfaecd41954fb9c9da8c61ccacd7
0020cfaecd41954fb9c9da8c61ccacd7

I'd also suggest changing the name of variable LINE to something that is not all uppercase.

answered Jun 05 '21 at 23:47

steeldriver

78,509
12
109
152

Hmm... now I'm second guessing myself about quoting of the herestring variable; [When is double-quoting necessary?](https://unix.stackexchange.com/a/68748/65304) doesn't appear to mention this case but `var='foo * bar'; read -ra arr <<< $var; printf '%s\n' "${arr[@]}"` does not seem to expand the `*` – steeldriver Jun 05 '21 at 23:59

How to avoid losing my escape characters when reading in variables from a file with bash

1 Answers1