Gnuplot: Plotting from *txt files from different directories by reading column header

Question

My folder structure looks kind of like this

$ tree
.
├── Original_folder
│   └── cat.txt
├── folderCD
│   └── cat.txt
├── folderGK
│   └── cat.txt
├── folderFE
    └── cat.txt

Each cat.txt file has 5 line before starting the column header. Sample cat.txt file is like this

Version LRv1.10.0
Build date 2017-12-06
MOL-calc
PRESSURE
!                       
      Time[s]     InletT[K]   InletP[Pa]   O2_GasOut     C_GasOut
       100         0.000885   1000000       0.0007       0.2111
and so on....

I want to plot first column along with the column which have column header with a keyword "_GasOut". (There are unknown number of header with this keywords, For each column I would like to have a separate graph). Additionally, the graphical results of Original_folder should be plotted in the same graph for all the plots from folderCD, folderGK, folderFE...... and so on.

Corresponding graph should be saved in the corresponding folders with a title same as column header. In each graph there should be two legend one is "original_folder" and another is "folderCD/folderGK/......"

I got all the output plot commands for Original_folder in one txt file and plot commands for all the other folders into another txt file. After that I am not finding a way to go ahead.. How can I do this for all other cases ? And how to make the column header as title?

LATEST UPDATE

for dir in folder* ; do
 echo "Preparing Post_processing files for ${dir}"
 mkdir "$dir"/Post_processing                        
 gawk -F  $'\t' '                                    
    /_GasOut/{                                      
       for(f=1;f<=NF;f++){                          
          hdr=$f                                    
         colhdr[f]=hdr                              
         if(index(hdr,"_GasOut"))wanted[f]=1
       }
    }
    ENDFILE{                                         

          print "reset\nset terminal pngcairo size 1024,768\nset encoding utf8\nset termoption dash\nset termopt enhanced"  
          print "set key top right"                                                                                         
          print "set xlabel '"'Time[s]'"';"    
       for(f in wanted){                           
          if(length(cmds)) cmds = cmds ",\n"
          hdr = colhdr[f]                        
          gsub(/^[[:space:]]+|[[:space:]]+$/,"",hdr)  
          printf("set ylabel '"'%s'"';\n",hdr)
          printf("set output '"'"$dir/Post_processing"/%s.png'"'\n",hdr)
          cmds = cmds "plot ""\"" FILENAME "\" using 1:" f " with lines" ","
          #print "plot " FILENAME using 1:" f " with lines" ",""
          cmds=cmds"'"'Original_folder/cat.txt'"' using 1:" f " with lines"        

        }      
       delete wanted  
    }
    END{              
       print cmds     
    }
    ' "$dir"/cat.txt>"$dir"/plot.gpl

   gnuplot "$dir"/plot.gpl
done

Current output is like this

reset
set terminal pngcairo size 1024,768
set encoding utf8
set termoption dash
set termopt enhanced
set xlabel 'Time[s]';
set ylabel 'H2_GasOut';
set output 'folderCD/Post_processing/H2_GasOut.png'
set ylabel 'O2_GasOut';
set output 'folderGK/Post_processing/O2_GasOut.png'
set ylabel 'H2O_GasOut';
set output 'folderFE/Post_processing/H2O_GasOut.png'
plot "folderCD/cat.txt" using 1:28 with lines,'Original_folder/cat.txt' using 1:28 with lines,
plot "folderGK/cat.txt" using 1:29 with lines,'Original_folder/cat.txt' using 1:29 with lines,
plot "folderGK/cat.txt" using 1:30 with lines,'Original_folder/cat.txt' using 1:30 with lines

Desired output

reset
set terminal pngcairo size 1024,768
set encoding utf8
set termoption dash
set termopt enhanced
set xlabel 'Time[s]';
set ylabel 'H2_GasOut';
set output 'folderCD/Post_processing/H2_GasOut.png'
plot "folderCD/cat.txt" using 1:28 with lines,'Original_RedKinMec/cat.txt' using 1:28 with lines,
set ylabel 'O2_GasOut';
set output 'folderGK/Post_processing/O2_GasOut.png'
plot "folderGK/cat.txt" using 1:29 with lines,'Original_folder/cat.txt' using 1:29 with lines,
set ylabel 'H2O_GasOut';
set output 'folderFE/Post_processing/H2O_GasOut.png'
plot "folderGK/cat.txt" using 1:30 with lines,'Original_folder/cat.txt' using 1:30 with lines

It's also good to have an output like this

set terminal pngcairo size 1024,768
   set encoding utf8
   set termopt dash
   set termopt enhanced
   set key top right
   set xlabel "Time[s]"
   set ylabel "O2_GasOut"
   set output "Post_processing/O2_GasOut.png"
   plot "folder1/cat.txt" using 1:22 with lines,\
   plot "folder2/cat.txt" using 1:22 with lines,\
   plot "folder3/cat.txt" using 1:22 with lines,\
   plot "folder4/cat.txt" using 1:22 with lines
   set ylabel "H2O_GasOut"
   set output "Post_processing/H2O_GasOut.png"
   plot "folder1/cat.txt" using 1:23 with lines,\
   plot "folder2/cat.txt" using 1:23 with lines,\
   plot "folder3/cat.txt" using 1:23 with lines,
   plot "folder4/cat.txt" using 1:23 with lines
   set ylabel "H2_GasOut"
   set output "Post_processing/H2_GasOut.png"
   plot "folder1/cat.txt" using 1:24 with lines,\
   plot "folder2/cat.txt" using 1:24 with lines,\
   plot "folder3/cat.txt" using 1:24 with lines,\
   plot "folder4/cat.txt" using 1:24 with lines




N.B: folder numbers are not fixed.
I added one of the cat.txt file for reference. https://1drv.ms/t/s!Aoomvi55MLAQh1wMmpnPGnliFmgg

These are several question in once and a mere request for someone else to do all the work for you. What have you tried so far? Where are you stuck? If you don't know where to start, here is one of many ways: use `find` to get all data file paths. Then you can loop over these value and replace a data_file and output_graph variable in your `gnuplot` input to match the current input file. Skipping lines before plotting is implemented in `gnuplot`. To match the correct column, `awk` can be used to select by name. — FelixJN, Jun 03 '19 at 12:41
Good start! I'd suggest looping over dirs (with `bash`) and executing `gawk` once per dir, creating a gnuplot command file: `for dir in * ; do gawk -f gawkfile "$dir"/cat.txt > "$dir"/plot.gpl ; gnuplot "$dir"/plot.gpl ; done`. Add more prints in the gawk file to make a full gnuplot script. Get the output image file [similar to this](https://unix.stackexchange.com/questions/134212/extract-file-name-from-path-in-awk-program) (it could be just replacing `cat.txt` with `plot.png`). For the titles: a line like `set key autottitle columhead` in the `gnuplot` script will work. Skip one less line. — FelixJN, Jun 03 '19 at 14:52
Can you be a bit more elaborate. It would be great if you edit my existing code. I am not so handy with bash scripting. — Mr. 206, Jun 03 '19 at 15:15
I edited my code like you said.. `for dir in *folder* ; do gawk -F $'\t' ' /_GasOut/{ for(f=1;f<=NF;f++){ bla bla bla ....... } ' "$dir"/cat.txt>"$dir"/plot.gpl gnuplot "$dir"/plot.gpl done ` It is printing an error like `"folderCD/plot.gpl", line 1: invalid command"` — Mr. 206, Jun 03 '19 at 15:30
Could you update the question above with the code? Seems like a missing newline or so. And I'd suggest putting the `awk` script as separate file, calling it from the bash script (`awk -f get_columns.awk "$dir"/cat.txt`). — FelixJN, Jun 03 '19 at 17:16
Pretty good! Check my answer for a few simplifications and corrections. — FelixJN, Jun 06 '19 at 23:06

FelixJN · Accepted Answer · 2019-06-06T23:12:29.843

First let's split the script so that we have a bash script and an awk script file. That way you'll need less escapes in the bash script and can forward variables into awk with the -v-option.

for dir in folder* ; do
  echo "Preparing Post_processing files for ${dir}"
  mkdir "${dir}"/Post_processing
  gawk -f make_gpl.awk -v dirname="${dir}" "${dir}"/cat.txt > "${dir}"/plot.gpl
  gnuplot "${dir}"/plot.gpl
done

The bash script is now fairly simple.

A few corrections and simplifications in the awk script - I hope the comments explain well enough:

#inserted field separator definition into script
BEGIN { FS="\t" }
/_GasOut/{
   for(f=1;f<=NF;f++){
      # $a ~ "B" matches if string B is part of field $a
      # only these elements are taken to array colhdr
      if ($f ~ "_GasOut") colhdr[f]=$f
   }
}
ENDFILE{
      #split prints with newlines into separate splits for readability
      print "set terminal pngcairo size 1024,768
      print "set encoding utf8"
      print "set termopt dash"
      print "set termopt enhanced"  
      print "set key top right"                                                                                         
      print "set xlabel \"Time[s]\""
      #for loop only matches if element of array colhdr is set
      for(f in colhdr){
        #it looks like there are only preceding spaces
        gsub(/^ +/,"",colhdr[f])
        #printing strings only - no printf needed
        #escaping quotes if they need to be printed
        #removed semicolons and commas at end of plot command - newline will do
        print("set ylabel \""colhdr[f]"\"")
        print("set output \""dirname"/Post_processing/"colhdr[f]".png\"")
        print("plot \""FILENAME"\" using 1:"f" with lines")
      }      
}

Example plotfile created with the script and your cat.txt:

set terminal pngcairo size 1024,768
set encoding utf8
set termopt dash
set termopt enhanced
set key top right
set xlabel "Time[s]"
set ylabel "O2_GasOut"
set output "folder1/Post_processing/O2_GasOut.png"
plot "folder1/cat.txt" using 1:22 with lines
set ylabel "H2O_GasOut"
set output "folder1/Post_processing/H2O_GasOut.png"
plot "folder1/cat.txt" using 1:23 with lines
set ylabel "H2_GasOut"
set output "folder1/Post_processing/H2_GasOut.png"
plot "folder1/cat.txt" using 1:24 with lines
set ylabel "N2_GasOut"
set output "folder1/Post_processing/N2_GasOut.png"
plot "folder1/cat.txt" using 1:25 with lines
set ylabel "NO_GasOut"
set output "folder1/Post_processing/NO_GasOut.png"
plot "folder1/cat.txt" using 1:26 with lines
set ylabel "NO2_GasOut"
set output "folder1/Post_processing/NO2_GasOut.png"
plot "folder1/cat.txt" using 1:27 with lines
set ylabel "N2O_GasOut"
set output "folder1/Post_processing/N2O_GasOut.png"
plot "folder1/cat.txt" using 1:28 with lines

Note that the formatting of the y labels in your plots might be wrong but I am unsure of the desired format. _ will make the following character subscript with termopt enhanced. For making more characters go subscript, use parentheses, e.g. C_6H_{12}O_6 for the molecular formula of sugar.

Perfect it worked super! I wonder whether I can arrange the plot.gpl like I edited in the Question? Like for example all O2_GasOut profile from folderCD, folderGK....folderFE in one graph? — Mr. 206, Jun 07 '19 at 08:34
In that case all values seem to be from the same columns. Run the first script, then just use e.g. `grep -h 'plot.*.1:22.*' folder*/plot.gpl | sed -e 's:$:,\\:' -e '2,$s/plot//' -e '$s:,\\$::'` and you have all the plot lines for column 22. Add settings and output file before that and then you have the gnuplot file you need. — FelixJN, Jun 07 '19 at 08:58

Gnuplot: Plotting from *txt files from different directories by reading column header

1 Answers1