/usr/bin/cp(&find): Argument list too long

Question

Error: Argument list too long

sudo cp and find... -exec sudo cp yield

/usr/bin/sudo(or find): Argument list too long

I get this error in my Travis CI build output. My .travis.yml file uses a shell script (deploy.sh) to run a git diff against branch2's force-app folder, and the output (several thousand files) is cp into a new directory:

sudo cp --parents $(git diff --name-only branch2 force-app/) $DEPLOYDIRECTORY;

The error is two lines:

./deploy.sh: line 122: /usr/bin/git: Argument list too long
./deploy.sh: line 122: /usr/bin/sudo: Argument list too long

I have the same issue with the subsequent find command. I use a naming convention to find and copy a file that corresponds with each file from the git diff:

for FILE in $GIT_DIFF_FILES; do
    if [[ $FILE == *Test.cls ]]; then
        find $classPath -maxdepth1 -samefile "$FILE-meta.xml" -exec sudo cp --parents {} $DEPLOY_DIRECTORY +
    fi;
done;

Again, I get the error: ./deploy.sh: line 142: /usr/bin/find: Argument list too long.

Note: Travis CI is running an Ubuntu Linux (Xenial) virtual environment for this build.

---

What I've Tried

1. Resetting the stack size

Travis CI automatically sets the stack size to 8192 and the arg max to 8388608 for each new build. In my deploy.sh file, I ulimit -s 9999999 to change the stack size to 9999999 and the arg max to 2559999744 for each new build.

2. Command Variations

For the first sudo cp command, I've tried:

1. formatting command as a for loop
2. tar -cf - -C files... | tar xpf - -C target directory...
3. formatting command as git diff ... | xargs cp ...
   

For the find command, I've tried:

1. find ... | xargs -n 1000 sudo cp ....
   • Any different flags added to this command don't work either.
2. find ... -exec cp ...
   • Any flags or syntax changes return the same error

Changing the command doesn't appear to be the solution

After reproducing the error locally, I found that my commands already work fine when interacting with a smaller number of files (~1000 or less).

However, running the cp and find commands when the output of git diff is approximately 1200-1500 files or more then returns:

Argument list too long

In addition, simply running find or cp from my shell script also returns:

Argument list too long

It does not seem to matter what I do to these commands, therefore. Something more fundamental is causing the problem, but only when the file count of the git diff output exceeds approximately 1200 files.

How do I fix my cp and find commands?

---

How to Reproduce the Error

Here are the steps to take to reproduce this error according to my use case. I used VS Code, so I recommend you do as well to replicate this as similarly to me as possible. Note that you will need to replace USERNAME in the below code with your username.

Step 1: Fork this repo.

Step 2: Run each line of code below in your terminal (one at a time)

cd force-app/main/default

Create a new folder "diff". Then continue below.

git checkout -b branch2
cd force-app/main/default/classes

Add //comment to the bottom of myclass.cls and myclass.cls-meta.xml files. Save changes.

for n in {001..1500}; do cp myclass.cls myclass$n.cls; done
for n in {001..1500}; do cp myclass.cls-meta.xml myclass$n.cls-meta.xml; done
git add .
git commit -m “first commit”
git push --set-upstream origin branch2
git checkout master
for n in {001..1500}; do cp myclass.cls myclass$n.cls; done
for n in {001..1500}; do cp myclass.cls-meta.xml myclass$n.cls-meta.xml; done
git add .
git commit -m “second commit”
git push

cd back to the sfdx-travisci folder.

sudo cp -p $(git diff --name-only branch2 /Users/USERNAME/sfdx-travisci/force-app/main/default/classes) /Users/USERNAME/sfdx-travisci/force-app/main/default/diff

for file in $(sudo cp -p $(git diff --name-only branch2 /Users/USERNAME/sfdx-travisci/force-app/main/default/classes) /Users/USERNAME/sfdx-travisci/force-app/main/default/diff); do if [[ $file == *.cls ]]; then find /Users/USERNAME/sfdx-travisci/force-app/main/default/classes -samefile “$file-meta.xml” -exec sudo cp -p {} /Users/USERNAME/sfdx-travisci/force-app/main/default/diff +; fi; done;

Answer 1

A complex copy command replacement:

tar -cf - -C /home/auser/data . | tar xpf - -C /home/auser/data2

Answer 2

Instead of trying some workarounds it is necessary to understand what the error is all about. The error is because the given argument exceeded the maximum argument length before invoking the command.

This may happen because

• shell expansion produced a too long argument list
• A process (like find) invoked a function of the exec family passing a too long argument list.

The command getconf ARG_MAX tells you the maximum length of arguments.

The command

cp $(git diff --name-only branch2 force-app/) ...

means that the command git is executed and then its output is passed as arguments to cp. If this output exceeds the maximum length of arguments, you get the said error.

The command

find ... -exec cp -t target {} +

is not much different because the trailing + to the find command tells find to pass the names of all found files in place of the curly braces.

So, the solution is to find ways to formulate your commands in ways that do not lead to huge argument strings. xargs or command substituion may be your friends.

As @AaronD.Marasco mentioned in their comment:

find ... -print0 | xargs -0 cp {} target

(xargs takes care of maximum argument length. Note that the curly braces here are an argument to xargs, not to find, but they have the same meaning).

Or with process substitution (less efficient in this case):

while IFS= read -d $'\0' -r Filename; do
  # do something with $Filename:
  cp "$Filename" target
done < <(find ... -print0)

For further details, please refer to the man pages of find and xargs.