Using sed/awk to retrieve first "non-whitspace token" in each line

Question

I have a big file of N-Triples/N-Quads
I want to be left with a collection of only subjects.
On smaller files I could have achieved this using:

awk '{print $1}' | uniq

but awk fails for large ~~files~~ lines. (several MB).

How can I achieve something similar with sed/grep -o/etc'...?

EDIT:

awk fails with:

awk: program limit exceeded: maximum number of fields size=32767
    FILENAME="file.nq" FNR=308254 NR=308254

How does `awk` fail on large files? Have you tried piping the input in (`awk '{print $1}' < file`)? — Stephen Kitt, Dec 21 '16 at 12:37
@StephenKitt yes. I accidently wrote large files (it is a large file, but that's not the problem), I should have written long lines. question is edited. — gilad hoch, Dec 21 '16 at 12:43

xhienne · Accepted Answer · 2016-12-21T12:54:48.157

2

[update] Some lines have too many (blank-separated) fields. Try grep instead of awk:

grep -E -o '^[^[:space:]]+' your_input_file | uniq

I would advise against using sed for this, as it would do a lot of extra-work on each line (to remove the end of the line) on a very big file. Same for awk: the line parsing is unnecessary.

edited Dec 21 '16 at 12:54

answered Dec 21 '16 at 12:38

xhienne

17,075
2
52
68

sorry, it's not the file size, I had a (mental) typo, some of the lines are very long... I edited the question – gilad hoch Dec 21 '16 at 12:42
@giladhoch I have updated my answer – xhienne Dec 21 '16 at 12:55
accepted as it is the fastest working snippet among all other answers. thanks :-) – gilad hoch Dec 21 '16 at 13:05

Stephen Kitt · Answer 2 · 2016-12-21T13:17:35.630

1

With sed:

sed 's/^ *\([^ ]*\) .*$/\1/g' | uniq

This replaces each line with the first sequence of non-spaces.

A faster variant using two greps (to handle lines with leading spaces, as AWK does):

grep -o "^[[:space:]]*[^[:space:]]*" | grep -o "[^[:space:]]*" | uniq

edited Dec 21 '16 at 13:17

answered Dec 21 '16 at 12:48

Stephen Kitt

411,918
54
1,065
1,164

works fine, but slower than the `grep` based answer (at least for me). – gilad hoch Dec 21 '16 at 13:06
Yes, it would be slower; the advantage of this approach is that it matches AWK's behaviour (including on lines which start with spaces). – Stephen Kitt Dec 21 '16 at 13:15
luckily, I don't have such lines on my file. but good to know :+1: – gilad hoch Dec 21 '16 at 16:06

icarus · Answer 3 · 2016-12-21T15:54:28.913

1

Answering the edited question, with long lines.

A trick you can use for this case is to use tr to interchange spaces and newlines. There are various ways you can get the first record out of the first line. Your problem then becomes one of finding lines which contain a space

 { echo # output a newline to get the first record
   cat file
 } | tr ' \n' '\n ' |
   sed -n '/ /s/.* //p'

Or

 tr ' \n' '\n ' < file | sed -ne '1p' -e '/ /s/.* //p'

The idea is that you change

this is a long line
and this is another

to

this
is
a
long
line and
this
is
another

so then tools with line length limits don't have problems. If you have tab characters between the fields then you probably want tr ' \t\n' '\n\n '

edited Dec 21 '16 at 15:54

answered Dec 21 '16 at 12:58

icarus

17,420
1
37
54

I don't understand everything you did there, but when I ran it on my file, it only output one line and halted (killed it after a while) – gilad hoch Dec 21 '16 at 13:07
Oops, fixed program. – icarus Dec 21 '16 at 15:49
and now it works :-) – gilad hoch Dec 21 '16 at 16:05

Using sed/awk to retrieve first "non-whitspace token" in each line

3 Answers3