Add a line if it's missing

Question

I have a data gathering device which records per second. Sometimes, randomly it drops a second or more. For example

166,1.09424,240,76,132
168,1.10088,215,76,132
169,1.10765,213,78,131
170,1.11458,198,79,131

It can gather 1M data points in a session, and might miss >100

Looking around I found this AWK command

awk '{ while (NR + shift < $1) { print (NR + shift) " NA"; shift++ }; print } END { shift++; while (NR + shift < 13) { print (NR + shift) " NA"; shift++ } }' /tmp/test1

from this answer

but that gives new lines when they aren't needed:

166 NA
167 NA
168 NA
169 NA
170 NA

What am I doing wrong?

Answer 1

A few issues with your command line:

1. Your CSV uses comma delimiters, but your awk command assumes the default space delimiter. This is the primary reason you get unnecessary fills.
2. The END case in your command exists only to fill the end result to a total minimum of 12 lines. This should therefore be removed, or the test should be changed to a higher minimum, perhaps user-specified.
3. Most tools used to process CSV don't like lines with varying numbers of fields. Your padding lines should therefore contain the same number of fields as normal lines.

Taken all together, and assuming your first record starts with second 1, we have this:

awk -F, -v min_secs=2000 '
{ 
  while (NR + shift < $1) {
    print (NR + shift) ",NA,NA,NA,NA"
    shift++ 
  }
  print
}
END {
  while (NR + shift <= min_secs) {
    print (NR + shift) ",NA,NA,NA,NA"
    shift++ 
  }
}
' /tmp/test1