0

Production logs file are huge in size and are stored in compressed format.

I use the following to find specific log line 

zgrep searchString LargeLogFiles*.gz

Typically this takes huge amount of time, when this runs on multiple files at once.

In my case, I am aware that the search string I'm trying to find (if present) will lie in 50% to 70% of the part of the file for sure. Is there any way I can leverage this information for faster search?

Possible solution zcat the file and filter the part using other unix commands. This is a possbility, but interested to know if there is a good/efficient way to achieve the above said problem.

mtk
  • 26,802
  • 35
  • 91
  • 130
  • @Inian Had pasted incorrect command. Updated the post with correct command that is used. – mtk Jan 22 '20 at 10:14
  • Not really. You cannot (efficiently) seek in a gzip file, unless it was compressed via a special method (`Z_FULL_FLUSH` from time to time, and add a table with the offsets -- dictionaries do that). Change your workflow to make the LargeLogFiles*.gz files smaller ;-) –  Feb 09 '20 at 20:21

0 Answers0