Convert list of hdf5 files to netcdf files with same name using shell scripting

Question

I have a list of datasets containing satellite data arranged in monthly folders as follows:

01  02  03  04  05  06  07  08  09  10  11  12

These folder are further divided into daily data folder for example for first month 01, daily files are arranged in folder as:

01  02  03  04  05  06  07  08  09  10  11  12  13  14  15  16  17  18  19  20  21  22  23  24  25  26  27  28  29  30  31

Which eventually has data sets:

OMI-Aura_L2-PROFOZ_2016m0101t0059-o60970_v003-2016m0107t153711.he5.met  OMI-Aura_L2-PROFOZ_2016m0101t1410-o60978_v003-2016m0107t153715.he5
OMI-Aura_L2-PROFOZ_2016m0101t0237-o60971_v003-2016m0107t153714.he5      OMI-Aura_L2-PROFOZ_2016m0101t1410-o60978_v003-2016m0107t153715.he5.met
OMI-Aura_L2-PROFOZ_2016m0101t0237-o60971_v003-2016m0107t153714.he5.met  OMI-Aura_L2-PROFOZ_2016m0101t1549-o60979_v003-2016m0107t153713.he5
OMI-Aura_L2-PROFOZ_2016m0101t0416-o60972_v003-2016m0107t153715.he5      OMI-Aura_L2-PROFOZ_2016m0101t1549-o60979_v003-2016m0107t153713.he5.met
OMI-Aura_L2-PROFOZ_2016m0101t0416-o60972_v003-2016m0107t153715.he5.met  OMI-Aura_L2-PROFOZ_2016m0101t1727-o60980_v003-2016m0107t153718.he5
OMI-Aura_L2-PROFOZ_2016m0101t0555-o60973_v003-2016m0107t153709.he5      OMI-Aura_L2-PROFOZ_2016m0101t1727-o60980_v003-2016m0107t153718.he5.met
OMI-Aura_L2-PROFOZ_2016m0101t0555-o60973_v003-2016m0107t153709.he5.met  OMI-Aura_L2-PROFOZ_2016m0101t1906-o60981_v003-2016m0107t153716.he5
OMI-Aura_L2-PROFOZ_2016m0101t0734-o60974_v003-2016m0107t153717.he5      OMI-Aura_L2-PROFOZ_2016m0101t1906-o60981_v003-2016m0107t153716.he5.met
OMI-Aura_L2-PROFOZ_2016m0101t0734-o60974_v003-2016m0107t153717.he5.met  OMI-Aura_L2-PROFOZ_2016m0101t2045-o60982_v003-2016m0107t153719.he5
OMI-Aura_L2-PROFOZ_2016m0101t0913-o60975_v003-2016m0107t153711.he5      OMI-Aura_L2-PROFOZ_2016m0101t2045-o60982_v003-2016m0107t153719.he5.met
OMI-Aura_L2-PROFOZ_2016m0101t0913-o60975_v003-2016m0107t153711.he5.met  OMI-Aura_L2-PROFOZ_2016m0101t2224-o60983_v003-2016m0107t153717.he5

I want to extract only files with .he5 extension and convert it to netcdf file using following code where file name is preserved.

ncks inputfile.he5 inputfile.nc

I am trying to process every file so I wrote a shell script as follow

shopt -s globstar
   for f in ./**; do
       echo "$f" |grep -v .met | grep ".he5"
       echo "$f" |grep -v .met |ncks $(grep".he5") $(echo $(grep  -o 'OMI-Aura_L2-PROFOZ_[0-9]\{4\}m[0-9]\{4\}t[0-9]\{4\}-o[0-9]\{5\}_v003-[0-9]\{4\}m[0-9]\{4\}t[0-9]\{6\}').nc)
   done
         

                                                                                                                                                                                           

It is able to extract files names but, I am not getting the ouput. How can I convert all files eventually?

Answer 1

Try using find and its execdir option which it allows find to switch the directory to the one that a file found then execute your command within that file's parent directory as following:

find /path/to/source/directory -type f -name '*.he5' -execdir sh -c '
    for file; do
        ncks "${file}" ${file%.he5}.nc"
    done
' sh_ncks {} +

The ${parameter%word} syntax is the Parameter Expansion, that strips the shortest suffix from its parameter.

Answer 2

Your command is overly complicated, you don't need any grep or alike:

shopt -s globstar
for f in ./**/*.he5; do
    ncks "$f" "${f%.*}.nc"
done

• ${f%.*} remove all characters beginning with the last .