There is this webpage I'm trying to scrape but it is behind a login form that needs user/password.
I used Firefox debug mode to find HTTP requests. Upon entering the credentials, there is a POST request which return a set-cookie in Response Headers. There is subsequent GET that uses this cookie (instead of user/pass) to finally access the actual page. The cookie has an expiration date.
I assume I have to 1- get that cookie via curl POST dumping it in a cookie.txt, 2- then again use curl GET with -b cookie.txt option (without needing to pass along user/pass).
I want to fetch that set-cookie via curl in command line (step 1).
I tried to replicate what Firefox did with Copy as cURL command on POST. It forms the curl request (with user/pass inside a --data-raw switch and a short ASP.NET_SessionId inside a -H 'Cookie: ....
Problem is, it does return a header but cookie is not there. The downloaded website (HTML) indicated something went wrong.
Now, in forming up the --data-raw, there are fields like __VIEWSTATE, __VIEWSTATEGENERATOR and __EVENTVALIDATION which seem to change on each GET of the login page. I filled them up via a little scrpting. It didn't fix the issue.
I'm using the copied curl command from firefox alongside -c cookie.txt -D - -v ...It does not print the cookie in headers or dump the cookie in the file.
