0

I have a csv file from a deduplicating tool (rmlint). I want to use awk to add a diving line after groups of rows that share the same value for a given field (like checksum) .

File:

type,path,size,checksum
emptydir,"/home/user/tree2/b",0,00000000000000000000000000000000
duplicate_dir,"/home/user/test/b",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/test/a",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/tree/b",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree/a",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree2/a",4,311095bc5669453990cd205b647a1a00

Desired output:

type,path,size,checksum
------------------------
emptydir,"/home/user/tree2/b",0,00000000000000000000000000000000
------------------------
duplicate_dir,"/home/user/test/b",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/test/a",4,f8772f6fda08bbc826543334663d6f13
------------------------
duplicate_dir,"/home/user/tree/b",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree/a",8,62202a79add28a72209b41b6c8f43400
------------------------
duplicate_dir,"/home/user/tree2/a",4,311095bc5669453990cd205b647a1a00

how could I go about doing this with awk?

ricardo3889
  • 43
  • 6

2 Answers2

2

Store the previous value, and output a line whenever the current value doesn’t match:

awk -F, 'NR > 1 && $NF != prev { print "------------------------" } { prev = $NF } 1'

($NF is the last field, adjust to suit.)

The number of dashes can be made easier to customise as follows:

awk -F, 'BEGIN { line = sprintf("%20s", ""); gsub(/ /, "-", line) }
         NR > 1 && $NF != prev { print line } { prev = $NF } 1'

or, with GNU AWK:

awk -F, 'BEGIN { line = gensub(/ /, "-", "g", sprintf("%20s", "")) }
         NR > 1 && $NF != prev { print line } { prev = $NF } 1'

Change the “20” to whatever value is appropriate.

Stephen Kitt
  • 411,918
  • 54
  • 1,065
  • 1,164
0

Here is another version where you can choose the length of your divider (20 in this example):

$ awk -F, -v i=$(printf -- '-%.0s' {1..20}) '{if(a==$NF){print;next}else{print i};print;a=$NF}' inputfile

type,path,size,checksum
--------------------
emptydir,"/home/user/tree2/b",0,00000000000000000000000000000000
--------------------
duplicate_dir,"/home/user/test/b",4,f8772f6fda08bbc826543334663d6f13
duplicate_dir,"/home/user/test/a",4,f8772f6fda08bbc826543334663d6f13
--------------------
duplicate_dir,"/home/user/tree/b",8,62202a79add28a72209b41b6c8f43400
duplicate_dir,"/home/user/tree/a",8,62202a79add28a72209b41b6c8f43400
--------------------
duplicate_dir,"/home/user/tree2/a",4,311095bc5669453990cd205b647a1a00
elmo
  • 105
  • 1
  • 6