I would like to do something equivalent to this:

some-expensive-command > /tmp/mytempfile
grep -v "pattern" /tmp/mytempfile >> output.txt
grep "pattern" /tmp/mytempfile | yet-another-command

Preferably elegant and without the need for the tempfile. I was thinking about piping through tee, but the best I can think of might is to combine two of the three lines and still require the intermediate storage:

some-expensive-command | tee /tmp/mytempfile | grep -v "pattern" >> output.txt
grep "pattern" /tmp/mytempfile | yet-another-command

1

2 Answers

The way question reads it sounds like you want one stdin redirected to two different commands. If that's the case, take advantage of tee plus process substitution:

some-expensive-command | tee >(grep 'pattern' > output.txt) >(grep -v 'pattern' | another-command)

Process substitutions are in fact anonymous pipelines implemented within bash itself ( on subprocess level ). We can also make the use of a named pipeline + tee. For instance, in terminal A do

$ mkfifo named.fifo
$ cat /etc/passwd | tee named.fifo | grep 'root'

And in another terminal B do

$ grep -v 'root' named.fifo

---

Another way to look at this is by recognizing that grep is line pattern matching tool, so by reading line at a time and using that same line in multiple commands we can achieve exactly the same effect:

rm output.txt # get rid of file so that we don't add old and new output
some-expensive-command | while IFS= read -r line || [ -n "$line" ]; do printf "%s\n" "$line" | grep 'pattern' >> output.txt printf "%s\n" "$line" | grep -v 'pattern' | another-command
done
# or if another-command needs all of the output,
# place `| another-comand` after `done` clause

---

Yet another way is to abandon grep and use something more powerful, like awk:

some-expensive-command | awk '/pattern/{print >> "output.txt"}; !/pattern/{print}' | another-command.

Practically speaking, don't worry about using temporary files, so long as you clean them up after using. If it works, it works.

Use bash Process Substitution:

some-command | tee >(grep "pat" | another-command >>out1) | grep -v "pat" >>out2

The process substitution assigns some-command’s output to grep "pat"’s input, thus saving you the tempfile. Of course the data is still saved in a file (it’s always), just that you don’t have to take care of that. If you don’t want to save another-command’s output in a file but rather print it I recommend to simply switch the two command lists.

Another nice source of information: man bash/EXPANSION

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy