How to use defined function with xargs

Question

This is my code

#!/bin/bash

showword() {
  echo $1
}

echo This is a sample message | xargs -d' ' -t -n1 -P2 showword

So I have a function showword which echoes whatever string you pass as a parameter to the function.

Then I have xargs trying to call the function and pass one word at a time to the function, and run 2 copies of the function in parallel. The thing that is not working is xargs doesn't recognize the function. How can I achieve what I am trying to do, how can I make xargs work with the function showword?

Also discussed in https://stackoverflow.com/questions/11003418/calling-shell-functions-with-xargs — Joshua Goldberg, Feb 28 '19 at 17:38

score 49 · Accepted Answer · edited Oct 15 '22 at 17:07

49

Try exporting the function, then calling it in a subshell:

showword() {
  echo $1
}

export -f showword
echo This is a sample message | xargs -d' ' -t -n1 -P2 bash -c 'showword "$@"' _

This causes xargs to execute

bash -c 'showword "$@"' _ This
bash -c 'showword "$@"' _ is
bash -c 'showword "$@"' _ a
            ︙

The arguments passed to the bash command are, well, passed into the bash environment, but starting from 0. So, inside the function,

$0 is “_” and $1 is “This”
$0 is “_” and $1 is “is”
$0 is “_” and $1 is “a”
︙

See Bash -c with positional parameters.

Note that export -f works only in Bash, and -Pn (--max-procs=max-procs) works only in GNU xargs.

edited Oct 15 '22 at 17:07

G-Man Says 'Reinstate Monica'

22,130
27
68
117

answered Oct 01 '14 at 06:13

cuonglm

150,973
38
327
406

7

One more question, why is the trailing _ required? Is it to stop xargs from processing anything else? – GMaster Oct 01 '14 at 06:51
4

@FazleA.: read http://unix.stackexchange.com/questions/152391/bash-c-with-positional-parameters – cuonglm Oct 01 '14 at 07:08
2

I am getting "export: Illegal option -f" when I do export -f – Sagar Chamling Nov 22 '18 at 06:35
@SagarChamling export -f works in bash only – cuonglm Nov 22 '18 at 08:06
1

@cuonglm this does invoke a new `bash` process for every `xargs` item, right? – NarūnasK May 10 '19 at 19:42
@NarūnasK No, for each xargs invocation. – cuonglm May 11 '19 at 13:27
2

Why does this work? Or, more specifically, why is the bash -c subshell needed (why can't xargs just detect and use the function just defined in the same shell process)? – Magnus Mar 13 '21 at 22:26
1

@Magnus because `xargs` run the command in new process, not with the original one. That's why you need `export f` to make the function visible in sub-shells. – cuonglm Mar 15 '21 at 02:25

GMaster · Answer 2 · 2021-12-09T16:57:20.933

Just adding an alternative solution using parallel which I have started using in place of xargs. The task is much easier with parallel

#!/bin/bash

showword() {
  echo $1
}
export -f showword

parallel -j2 showword {} ::: This is a sample message

-j2 makes sure 2 copies of the function is run in parallel
::: anything after this is passed as separate arguments to parallel, separation is whitespace
{} is replaced by the argument passed into parallel and fed into the showword function

If you are using zsh shell this solution will not work since zsh does not have any feature to export functions. You will need something like this:

#!/usr/bin/zsh

showword() {
  echo $1
}

# add the following to your .zshrc if you want env_parallel in your shell permanently
source /usr/bin/env_parallel.zsh

env_parallel -j2 --env showword showword {} ::: This is a sample message

Where does `/usr/bin/env_parallel.zsh` come from? I don't have it on my system... — friederbluemle, Oct 19 '22 at 21:18
@friederbluemle On Ubuntu, if I run the command `dpkg -S /usr/bin/env_parallel.zsh`, it says the file `/usr/bin/env_parallel.zsh` comes from the package `parallel` — GMaster, Oct 20 '22 at 02:55

How to use defined function with xargs

2 Answers2

Linked