1

I wrote a script to save logs of request to its individual files. Over time the number of files in the folder seems to have grown in number by ALOT. Now I need to access some logs of some time range and I am not able to do it at all.

The log filename format is this: 2016-02-11_124805_4269_170784456256bc8305a7e8b1455194885_req.xml

So far I have tried

  • ls
  • tar -cvzf 2016-02-20.tar.gz 2016-02-20*.xml to compress files
  • cat 2016-02-20_01* | grep -nr "text_to_search"
  • ls -lah >> ../list.log
  • find ./dirname -type f >> list2.log

  • And a script to search the 

    for each_file in ./dirname/*
do
echo ls -lah $each_file >> ./dirname_file.log
done

The tar and cat returns with arguments list too long, and others have not given me a response yet.

Any help is appreciated.

Starx
  • 187
  • 1
  • 9

3 Answers3

1

find ./dirname -type f -name "2016-02-20_01*" may work, although it still likely won't be fast.

You could then do,

find ./dirname -type f -name "2016-02-20_01*" -print | xargs grep -n "text_to_search"

EightBitTony
  • 20,963
  • 4
  • 61
  • 62
  • Trying this, no response yet. – Starx Mar 03 '16 at 12:44
  • I got a response when I ran `find ./dirname -type f -name "2016-02-20_0101*", the number of entries I got is `564` its not too much, why is this so hard to browse or view such files? – Starx Mar 03 '16 at 13:37
  • @Starx because it has to search through every single directory entry and it sounds like you have many, many thousands of files. There's no way for it to magically know which files you want without checking every single filename, unless you give it the full and explicit filename in the first instance. – EightBitTony Mar 03 '16 at 20:58
  • I managed to get a full list of files and they were around 6500. This is very less number, I thought there were millions LOL. I would assume this is nothing for `ext4` file system. [Any thoughts?](http://serverfault.com/questions/761381/why-would-a-folder-with-around-6500-in-a-ext4-partition-become-inaccesible?noredirect=1#comment957762_761381) – Starx Mar 04 '16 at 07:27
  • I suggest, in that case, asking a new question, because now you know how many files you have, you can ask specifics about the performance of that (it sounds like an issue to me, shouldn't take that long, but a new question is your best bet). – EightBitTony Mar 04 '16 at 08:37
  • I asked one on serverfault, the link to it is in my earlier comment :) if you are interested. – Starx Mar 04 '16 at 09:07
  • @Starx apologies, didn't notice the link. – EightBitTony Mar 04 '16 at 09:09
1

With 'ls', I have the impression that the slowness is mostly caused by the sorting.

'ls -f' will disable the sorting, and in my experience (with ext3 and xfs a few years back when I ran a nntp server) works a lot faster.

jgaa
  • 111
  • 3
0

Try this:

perl -le 'opendir $d,".";while($f=readdir($d)){print $f if $f =~ /^2016-02-20_01/}'

This reads the directory entries directly and doesn't try to do anything with them as such. If you want to actually look at the contents of the file, you could use:

perl -le 'opendir $d,".";while($f=readdir($d)){if($f =~ /^2016-02-20_01/){print "=== $f ===";open(my $i,"<",$f);while(<$i>){print};close($i)}}'

EDIT: to clarify, this assumes that you are already cd'd into the directory. (The dot refers to the current directory). Otherwise, replace "." with the directory path.

Uditha Desilva
  • 141
  • 4