62

What would be the most straightforward way of making a GET request to a url over HTTPS, and getting the raw, unparsed response?

Could this be achieved with curl? If so, what options would you need to use?

Acorn
  • 825
  • 1
  • 7
  • 11

4 Answers4

78

If you want to use curl, this should work:

curl -D - https://www.google.com/

Note, however, that this is not exactly the raw response. For instance chunked transfer encoding will not be visible in the response. Using --raw solves this, also verbose mode (-v) is useful, too and -i shows the headers before the response body:

curl -iv --raw https://www.google.com/

If you want to use a pager like less on the result, it is also necessary to disable the progress-bar (-s):

curl -ivs --raw https://www.google.com/ | less

Depending on what you want to do this may or may not be a problem.

What you do get is all HTTP response headers and the document at the requested URL.

hakre
  • 409
  • 1
  • 6
  • 20
Mika Fischer
  • 2,212
  • 1
  • 16
  • 16
17

Here's a simple way that comes to mind

echo 'GET / HTTP/1.1
Host: google.com

' | openssl s_client -quiet -connect google.com:443 2>/dev/null
phemmer
  • 70,657
  • 19
  • 188
  • 223
  • note that some servers (e.g github.com) only work with `HTTP/1.0` – Zombo Jan 23 '18 at 01:14
  • I have a question based on your answer. Please check https://unix.stackexchange.com/questions/622274/why-does-echo-works-but-cat-does-not-when-pipeing-to-openssl – Ahmad Ismail Nov 30 '20 at 22:52
7

It's not curl, but it should be available on almost all Unices:

wget -S --spider https://encrypted.site

If the status messages bother you:

wget -S --spider https://encrypted.site 2>&1 | awk '/^  /'

If you want CRLF line endings:

wget -S --spider https://encrypted.site 2>&1 | awk '/^  / { sub(/$/,"\r"); print }'
Chris Down
  • 122,090
  • 24
  • 265
  • 262
  • Do you know if this would preserve the `\r\n` characters in the response? – Acorn Jan 19 '12 at 00:04
  • @Acorn - First of all, not all webservers will respond with `\r\n` as a line ending. I don't believe this will preserve it in any case, but if that matters to you, I'll add a way of getting that result in the answer. – Chris Down Jan 19 '12 at 08:05
  • any idea how to parse the response headers in a { json: "object" } ? – assayag.org Mar 13 '22 at 06:58
3
$ GET -e https://www.google.com

On Debian/Ubuntu distros belongs to the package lwp-request.

funollet
  • 241
  • 1
  • 2