0

I have a simple bash script goes like

#!/bin/bash
set -e
SPARK_CONF="--master ... \
...
--conf spark.driver.extraJavaOptions='-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps' \
--conf ...."
SPARK_CMD="spark-submit $SPARK_CONF"
echo $SPARK_CMD
exec $SPARK_CMD

I'll get an error

Error: Unrecognized option: -XX:+PrintGCDetails

I can see that I print the whole command before exec it, so I copied the printed command, and paste in my shell then I found it can be executed successfully.

I tried to replace the single quote with \" but still the same error.

Also, if I replace exec with eval, it will also work. I googled around the difference between them, but didn't found why exec will terminate inside the quotes...

I just want to understand why exec fail in this case.

Update: According to https://unix.stackexchange.com/a/444949/524235, I changed my related part to

set -e -x

SPARK_CONF="--master ... \
...
--conf spark.driver.extraJavaOptions='-verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps' \
--conf ...."
SPARK_CMD=($SPARK_HOME/bin/spark-submit $SPARK_CONF)
"${SPARK_CMD[@]}"

It's still failing with the same error message mentioned above, but I can see the executed command becomes:

.../bin/spark-submit ...
--conf 'spark.driver.extraJavaOptions='\''-verbose:gc' -XX:+PrintGCDetails '-XX:+PrintGCDateStamps'\'''
...

I can see there're some extra single quotes and backslash added into the command, how can I progress now?

Alain ux
  • 1
  • 1
  • This may help: [How can we run a command stored in a variable?](https://unix.stackexchange.com/questions/444946/how-can-we-run-a-command-stored-in-a-variable) – steeldriver May 01 '22 at 02:23
  • @steeldriver Thanks a lot for the useful link! I am trying with the solution menioned there, but I am still failing with the same error, I've updated my description, could you take another look? Respect, man. – Alain ux May 01 '22 at 02:54
  • The shell parses quotes and escapes before expanding variables, so putting quotes in a variable doesn't do anything useful; if you need to store a command or argument list, use an array instead. This applies to `SPARK_CONF` as well as `SPARK_CMD`. Or better yet, don't store them at all; just execute the command directly. (Or maybe use a function, depending on the situation.) See [BashFAQ #50: "I'm trying to put a command in a variable, but the complex cases always fail!"](http://mywiki.wooledge.org/BashFAQ/050) – Gordon Davisson May 01 '22 at 05:45
  • "_Update: According to [[this](https://unix.stackexchange.com/a/444949/524235)], I changed my related part to..._" The parameters should go into the array. You've still got them as a string – roaima May 01 '22 at 08:35

0 Answers0