curl --data-binary $' inside a shell script

Question

I have tried everything to do a curl from within a shell script. This curl has a

--data-binary $'

part which I just can not get my script to recreate. It will move the $ inside the single quote for some weird reason. Any ideas how to fix it?

--data-binary '$

curl -s  -H 'Connection: keep-alive' \
  -H 'Cache-Control: max-age=0' \
  -H 'Upgrade-Insecure-Requests: 1' \
  -H 'Origin: https://url.com' \
  -H 'Content-Type: multipart/form-data; boundary=----WebKitFormBoundary6AiBMw0TyIybreRa' \
  -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36' \
  -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
  -H 'Sec-Fetch-Site: same-origin' \
  -H 'Sec-Fetch-Mode: navigate' \
  -H 'Sec-Fetch-User: ?1' \
  -H 'Sec-Fetch-Dest: document' \
  -H 'Referer: https://url.com' \
  -H 'Accept-Language: en-US,en;q=0.9,es;q=0.8' \
  -H "Cookie: JSESSIONID=$jsession; __utmc=35664418; __utmz=35664418.1593348512.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); BALANCEID=balancer.apphpgal01; acceta_default=im10snhiask7qic85vrr1dr4i2; __utma=35664418.1302973562.1593348512.1593372701.1593378049.6; __utmt=1; __utmb=35664418.5.10.1593378049" \
  --data-binary $"------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"$var1\"\r\n\r"$var2"\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"$var3\"\r\n\r\n\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"g-recaptcha-response\"\r\n\r\n$recaptcha\r\n------WebKitFormBoundary6AiBMw0TyIybreRa--\r\n" \
  --compressed "https://url.com"

It is driving me crazy. I have tried everything. Escaping the $, putting it in single quotes. Double quotes. Changing every " for '. I'm definitely missing something guys.

EDIT: Someone pointed out that I'm not explaining myself that clearly. Rightly so: I'm running the script like this "bash curl.sh". The content of curl.sh would be

curl -s -H "Cookie: JSESSIONID=$jsession" \
  --data-binary $"------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"$var1\"\r\n\r"$var2"\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name=\"g-recaptcha-response\"\r\n\r\n$recaptcha\r\n------WebKitFormBoundary6AiBMw0TyIybreRa--\r\n" \
  --compressed "https://url.com"

However when I execute it, the --data-binary part puts the dollar sign $ inside the quotes

I have also tried this per suggestion of @Stéphane Chazelas without success

--data-binary \$'------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name="'"$var1"'"\r\n\r'"$var2"'\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name="'"$var3"'"\r\n\r\n\r\n------WebKitFormBoundary6AiBMw0TyIybreRa\r\nContent-Disposition: form-data; name="g-recaptcha-response"\r\n\r\n'"$recaptcha"'\r\n------WebKitFormBoundary6AiBMw0TyIybreRa--\r\n' \
  --compressed "https://url.com"

Will run

--data-binary '$------WebKitFormBoundary

Same thing happens with

--data-binary "$"'------WebKitFormBoundary

as well as

--data-binary '$''------WebKitFormBoundary

Well, first, you don't have `$'` or `'$` in that command you've shown. Second, what do you mean it gets moved inside the single quote? What's the value you're actually trying to give to `curl`? What's the value it's getting? — ilkkachu, Jun 29 '20 at 17:22
Ah, right. Can you show exactly how you're running that script? And can you show the exact contents of the script? With the whole `curl` command changed to something shorter to demonstrate the problem, e.g. just an `echo`? — ilkkachu, Jun 29 '20 at 17:25
I never wrote `\$'...'` in my answer. Do you want a literal `$` to be included in the argument passed to `curl`, or do you want to use the `$'...'` quoting operator to allow the `\r` to be expanded to a CR character? — Stéphane Chazelas, Jun 30 '20 at 14:27

Stéphane Chazelas · Answer 1 · 2020-07-03T09:25:15.210

The $"..." ksh93 quoting operator (also found in bash) is for internationalising messages, it's not what you want here.

To expand the \r\n, you need $'...' (also from ksh93 but more widely supported).

Parameters are not expanded within though, so you'd need:

--data-binary $'....\r\n...name="'"$var1"$'"...\r\n...'

For --data-binary and ....<CR><LF>...name="<contents-of-var1>"...<CR><LF>... (where <CR>, <LF>, <contents-of-var1> stand respectively for the Carriage Return control character, the Line Feed aka Newline control character and the contents of the $var1 variable) arguments to be passed to curl.

That is, use $'...' for the parts where you want \x sequences to be expanded and "..." for the parts that need variables to be expanded.

Another option is to define:

CR=$'\r' LF=$'\n' CRLF=$'\r\n'

Or using standard sh syntax:

eval "$(printf 'CR="\r" LF="\n" CRLF="\r\n"')"

and use standard "..." for the whole thing:

--data-binary "...${CRLF}...name=\"$var1\"..."

See How to use a special character as a normal one? for details on various quoting operators in various shells.

Hello @Stéphane, I have tried some of what you stated as well as the linked on your answer. Executing `--data-binary \$'------WebKitFormBoundary` returned `--data-binary '$------WebKitFormBoundary` — user1079378, Jun 30 '20 at 09:37

ilkkachu · Answer 2 · 2020-06-30T12:14:11.493

So, what I think you want is basically this:

$ cat quote.sh
#!/bin/bash
str=foobar
printf "%s\n" $'str='"$str"$'\nanother line\n'
#             <-----><----><----------------->  see below

$ bash quote.sh
str=foobar
another line

I.e. the \r\n changed to CRLF, and the variables expanded. As Stéphane mentioned, you have to use both $'...' and "..." for that. $'...' interprets the backslash-escapes and the regular double quotes expand variables within. Neither does both, and $"..." is different still (and not useful here). I marked the three distinct quoted strings above. Note that if you need to put literal double-quotes in the string going to curl, you can just put them as-is within single quotes. $'"\n"' is two double-quotes separated by a newline.

I asked about running the script, because $'...' isn't a standard feature, and not supported in e.g. Dash, which Debian and Ubuntu have as /bin/sh. So using that shell would fail:

$ sh quote.sh
$str=foobar$\nanother line\n

The result is a bit similar to what you'd get in Bash if you put the $ inside the single quotes, like echo '$str=...'.

In all of these, the $ is quoted or escaped, and gets interpreted as a regular character, not as the start of the $'...' quoting.

--data-binary \$'------Web...'
--data-binary '$------Web...'
--data-binary "$"'------Web...'
--data-binary '$''------Web...'

And if you used any of those with set -x in effect, Bash would indeed output the resulting string as '$----...':

$ echo --data-binary \$'------Web...'
+ echo --data-binary '$------Web...'
--data-binary $------Web...

Actually, since that string is rather unwieldy and over multiple lines, you might also want to build it in pieces.

Something like this:

data=
data+=$'------WebKitFormBoundary6AiBMw0TyIybreRa\r\n'
data+=$'Content-Disposition: form-data; name="'"$var1"$'"\r\n'
#      ^                                     *^^     ^^^*    ^
#   ^ = shell quotes, * = quotes in the resulting string
data+=$'------WebKitFormBoundary6AiBMw0TyIybreRa\r\n'
...

and then

curl ... --data-binary "$data"

curl --data-binary $' inside a shell script

2 Answers2