How to make this search faster in fgrep/Ag?

Question

I am thinking methods to make the search faster and/or better which principally uses fgrep or ag. Code which searches the word and case-insensitively at $HOME, and redirects a list of matches to vim

find -L $HOME -xtype f -name "*.tex" \
   -exec fgrep -l -i "and" {} + 2>/dev/null | vim -R -

It is faster with ag because of parallelism and ack

find -L $HOME -xtype f -name "*.tex" \
   -exec ag -l -i "and" {} + 2>/dev/null | vim -R -

Statistics

Small group average statistics with fgrep and ag by time

        fgrep   ag     terdon1  terdon2  terdon3  muru 
user    0.41s   0.32s  0.14s    0.22s    0.18s    0.12s
sys     0.46s   0.44s  0.26s    0.28s    0.30s    0.32s

Cases terdon1 and terdon3 can be equal fast. I get great fluctuations with those two. Some Ranking by sys time (not best criteria!)

1. terdon1
2. terdon2
3. terdon3
4. muru
5. ag
6. fgrep

Abbreviations

• terdon1 = terdon-many-find-grep
• terdon2 = terdon-many-find-fgrep
• terdon3 = terdon-many-find-ag (without F because not exists in ag)

Other codes

muru's proposal in comments

grep -RFli "and" "$HOME" --include="*.tex" | vim -R -

OS: Debian 8.5
Hardware: Asus Zenbook UX303UA

Answer 1

Since you're using ack and The Silver Searcher (ag), it seems that you are OK with using additional tools.

A new tool in this space is ripgrep (rg). It is designed to be fast in both finding files to search (like ag) and also fast in searching files themselves (like plain old GNU grep).

For the example in your question, you might use it something like this:

rg --files-with-matches --glob "*.tex" "and" "$HOME"

The author of ripgrep posted a detailed analysis of how the different searching tools work, along with benchmark comparisons.

One of the benchmarks, linux-literal-casei, is somewhat similar to the task you describe. It searches over a large number of files in a lot of nested directories (the Linux codebase), searching for a case-insensitive string literal.

In that benchmark, rg was fastest when using a whitelist (like your "*.tex" example). The ucg tool also did well on this benchmark.

rg (ignore)         0.345 +/- 0.073 (lines: 370)
rg (ignore) (mmap)  1.612 +/- 0.011 (lines: 370)
ag (ignore) (mmap)  1.609 +/- 0.015 (lines: 370)
pt (ignore)        17.204 +/- 0.126 (lines: 370)
sift (ignore)       0.805 +/- 0.005 (lines: 370)
git grep (ignore)   0.343 +/- 0.007 (lines: 370)
rg (whitelist)      0.222 +/- 0.021 (lines: 370)+
ucg (whitelist)     0.217 +/- 0.006 (lines: 370)* 

* - Best mean time. + - Best sample time.

The author excluded ack from the benchmarks because it was much slower than the others.

Answer 2

You could probably make it a little bit faster by running multiple find calls in parallel. For example, first get all toplevel directories and run N find calls, one for each dir. If you run the in a subshell, you can collect the output and pass it to vim or anything else:

shopt -s dotglob ## So the glob also finds hidden dirs
( for dir in $HOME/*/; do 
    find -L "$dir" -xtype f -name "*.tex" -exec grep -Fli and {} + & 
  done
) | vim -R -

Or, to be sure you only start getting output once all the finds have finished:

( for dir in $HOME/*/; do 
    find -L "$dir" -xtype f -name "*.tex" -exec grep -Fli and {} + & 
  done; wait
) | vim -R -

I ran a few tests and the speed for the above was indeed slightly faster than the single find. On average, over 10 runs, the single find call tool 0.898 seconds and the subshell above running one find per dir took 0.628 seconds.

I assume the details will always depend on how many directories you have in $HOME, how many of them could contain .tex files and how many might match, so your mileage may vary.