56

Say I have a file with the following

bob
john
sue

Now these directly corrospond to (in this case) URL pattern such as http://example.com/persons/bob.tar, john.tar, sue.tar.

I would like to take these lines and run them through xargs. I don't know what is passed to the command being executed though. How do I access the parameter either from the prompt (say I want to simply echo each line like cat file | xargs echo $PARAM) or from a bash script.

Josh K
  • 3,866
  • 4
  • 22
  • 15
  • I'm not quite following the question, sorry. In particular I'm not sure what "I don't know what is passed to the command being executed though" means – Michael Mrozek Oct 29 '10 at 02:07
  • 1
    @Michael: When you run a list through `xargs` it breaks it up by line and feeds each line into a command, right? How do I access that if I need to say something like `cat file | xargs curl http://example.com/[PASSED FROM FILE].tar`? – Josh K Oct 29 '10 at 02:40

8 Answers8

91

Michael's answer is right, and should sort out your problem. Running

cat file | xargs -I % curl http://example.com/persons/%.tar

will download files bob.tar john.tar. sue.tar as expected.

BUT: cat here is useless

rather use:

<file xargs -I % curl http://example.com/persons/%.tar
Community
  • 1
Stefan
  • 24,830
  • 40
  • 98
  • 126
  • 1
    In my one file example it may not be ideal, however, `cat xaa xab xac xad ... xargs ...` – Josh K Oct 29 '10 at 10:33
  • 14
    @Josh For some reason people tend to take unnecessary uses of cat really seriously here; I've been downvoted for it twice now – Michael Mrozek Oct 29 '10 at 12:20
  • 3
    for the record, you got a +1 from me. I think using `cat` is fine really, just like to add more info... :) – Stefan Oct 29 '10 at 12:25
  • Surely it should be `xargs -I % curl …` (*xargs* option before `curl` and its options/arguments). At the very least `xargs curl -I %` (where `-I %` is meant as an option to *xargs*, not *curl*) is not portable. – Chris Johnsen Dec 05 '10 at 18:04
  • @Chris, nice catch, tnx. fixed. – Stefan Dec 05 '10 at 19:59
18

I think you're asking how to insert the individual lines pulled from xargs' stdin in the middle of a command, instead of just pasting it on the end always. If so, the -I flag takes a replacement-string argument; xargs will then replace replacement-string in the command with the line read from stdin:

$ cat file | xargs -I foobar curl http://example.com/foobar.tar
Michael Mrozek
  • 91,316
  • 38
  • 238
  • 232
  • Okay, how about `curl http://example.com/foobar.tar > foobar.tar`? – Josh K Oct 29 '10 at 03:02
  • 2
    @Josh K: `>` is a shell constructs, and won't work for `xargs`. On the other hand, `curl -o` will write to a named file instead of stdout, like what `wget` does, so that's probably what you would like to use here. – ephemient Nov 01 '10 at 07:23
16
$ man xargs
...
       --arg-file=file
       -a file
              Read items from file instead of standard input.  If you use this
              option,  stdin  remains unchanged when commands are run.  Other-
              wise, stdin is redirected from /dev/null.
...

You may want to set --delimiter=/-d to '\n' as well.

On the other hand, if you are just trying to turn each line in the file into a URL,

$ sed -e 's#.*#http://example.com/persons/&.tar#' file

will do, and if you want to fetch all of them, just pipe that into | wget -i.

ephemient
  • 15,640
  • 5
  • 49
  • 39
9

another way with shell looping:

for i in `cat file`; do curl -I http://foo.com/$i; done

you can also run each iteration in the background by appending & prior to the last semicolon - for very large downloads this might be handy

Brad Clawsie
  • 269
  • 1
  • 4
  • 2
    You've [fallen for one of the classic blunders](http://partmaps.org/era/unix/award.html#arg-max)! <-- broken link, here's a cached version: [Useless Use of Cat Award](http://archive.is/fBdSf). – Sorpigal Dec 06 '10 at 17:41
  • @Sorpigal That url is broken. What is the 'Classic Blunder'? – starbeamrainbowlabs Jul 13 '15 at 06:40
  • @starbeamrainbowlabs - useless use of cat, I've put a cached version of that link in the comment above. – slm Jul 13 '15 at 12:24
  • @starbeamrainbowlabs: I specifically was linking to [Dangerous Backticks](http://archive.is/fBdSf#arg-max), but the use of cat is also needless if you change to a while construct: `while IFS= read i ; do curl ... ; done < file` – Sorpigal Jul 14 '15 at 20:28
6

With GNU Parallel you can do:

cat urls | parallel curl {} ">" {/}

Or:

cat persons | parallel curl http://example.com/persons/{}.tar ">" {}.tar

Watch the intro video for GNU Parallel to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ

Ole Tange
  • 33,591
  • 31
  • 102
  • 198
2

while read VAR; do ... done loop is simple yet very versatile:

while read word; do wget http://example.com/persons/$word; done < file
musiphil
  • 1,601
  • 2
  • 14
  • 15
2

Eventually, you can use as well:

xargs -n1 -a file -I % curl http://example.com/persons/%.tar
Enginer Rod
  • 21
  • 1
  • 1
    Does not using `-I` imply `-n 1`? Also, the `-a` option is [non-standard](https://pubs.opengroup.org/onlinepubs/9699919799/utilities/xargs.html), so you would have to specify what implementation of `xargs` you're using. – Kusalananda Oct 05 '20 at 06:52
  • Using `-I` imply `-x` and `-L` (max-lines), but not `-n` ... if I am not mistaken. In *xargs 4.7.0 GNU* `-a file ` is equal to `--arg-file=file ` – Enginer Rod Oct 06 '20 at 18:18
0

This is a more general version of Stefan:s answer but I'm using awk in the middle to prepare the exact "string" that I would like xargs to execute. And then xargs is using bash to do the actual "work".

It is a little bit overkill for this example, but it is a general solution that with some modifications can solve many problems...

cat file | awk '{print "curl http://example.com/persons/"$1".tar"}' | xargs -0 bash -c
Johan
  • 4,523
  • 2
  • 26
  • 33