Get rid of UNIX redirection newly created file without using rm

Question

Using below function to filter out text file and organize to new format.

Using unix redirection > command to create new files.

How can I get rid of creating all new files

(e.g. ${file}_final_1.txt,${file}only${pattern}.txt)

without deleting it afterword using rm command ?

filter(){
  cat ${file}_1.txt | grep -v ${pattern} > ${file}_final_1.txt                            
  cat ${file}_1.txt | grep ${pattern} > ${file}_only_${pattern}.txt                        
  cat ${file}_only_${pattern}.txt  | nawk -F '|' '{ print $NF}' > ${pattern}_TS.txt        
  paste ${pattern}_TS.txt ${file}_only_${pattern}.txt > ${pattern}_TS_file.txt
  cat ${pattern}_TS_file.txt | grep "|${DT}|" | grep ${pattern} | sort -r | head -1 > ${file}_f.txt
  cat ${file}_f.txt >> ${file}_final_1.txt
  dos2unix ${file}_final_1.txt ${file}_final.txt
}

(1) You can avoid creating so many files if you build longer pipes instead of writing to a file only to read from it in the next step. This won't work with `paste` though, where in your case you need at least one regular file as input (unless you use process substitution, but not really useful in your case where `${file}_only_${pattern}.txt` is used twice). (2) Creating temporary files in a temporary directory (`mktemp -d`) allows you to remove them all by removing the directory (`rm -r …`). // This is not an answer because (1) does not "get rid of creating all" and (2) is not "without `rm`". — Kamil Maciorowski, Aug 23 '22 at 15:03
[I hope you're using zsh](https://unix.stackexchange.com/a/68748/108618). — Kamil Maciorowski, Aug 23 '22 at 15:06
As a starting point, copy/paste that script into http://shellcheck.net and fix the many issues it'll tell you about. — Ed Morton, Aug 23 '22 at 20:45

score 1 · Answer 1 · answered Aug 23 '22 at 15:09

1

You can chain pipes as long as you like:

  cat ${file}_1.txt | grep ${pattern} > ${file}_only_${pattern}.txt                        
  cat ${file}_only_${pattern}.txt  | nawk -F '|' '{ print $NF}' > ${pattern}_TS.txt

could be replaces with

cat ${file}_1.txt | grep ${pattern} | nawk -F '|' '{ print $NF}' > ${pattern}_TS.txt

And so on... There is a limit on the command line, of course, but it is ridiculously large. Do getconf ARG_MAX and you'll get how many characters it is.

If some app does not accept input from a pipe, or it is easier to work with multiple files... Then you can always dump temp files into a temp directory. So your filter() function could follow a pattern:

filter(){
   TEMP_DIR=/tmp/dir_for_this_task_`date +%s`
   mkdir $TMP_DIR
   CUR_DIR=`pwd`
   cd $TMP_DIR

   .... your code with temp files

   cd $CUR_DIR
   rm -r $TMP_DIR
}

you can also use pushd/popd to remember current directory and return to it from temp.

answered Aug 23 '22 at 15:09

White Owl

4,511
1
4
15

2

1. Use `mktemp` to create temporary files and directories 2. Double-quote your variables when you use them 3. Use `$(...)` instead of backticks – roaima Aug 23 '22 at 15:16
@roaima There are infinite ways to improve any script. – White Owl Aug 23 '22 at 15:21
2

There are indeed, but writing incorrect code is poor – roaima Aug 23 '22 at 20:25
@roaima `$(...)` is a bashism, backticks work in every POSIX shell. – allo Aug 23 '22 at 20:43
3

@allo no, it's part of the spec for all POSIX shells, see https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_02_03 and other parts of the spec. – Ed Morton Aug 23 '22 at 20:47

Get rid of UNIX redirection newly created file without using rm

1 Answers1