Ho to insert cookies value in curl? from firebug request headers I can see in the following
Cookie: PHPSESSID=gg792c2ktu6sch6n8q0udd94o0; was=1; uncheck2=1; uncheck3=1; uncheck4=1; uncheck5=0; hd=1; uncheck1=1"
I have tried the following:
curl http://site.com/ -s -L -b cookie.c -c cookie.c -d "was=1; uncheck2=1; uncheck3=1; uncheck4=1; uncheck5=0; hd=1; uncheck1=1" > comic
and the only thing i see in cookie.c is
PHPSESSID=gg792c2ktu6sch6n8q0udd94o0; was=1;
To pass keys/values to cURL cookie, you need the -b switch, not -d.
For the forms -d, the data will be separated by & and not by ; in your curl command.
So :
curl http://site.com/ \
-s \
-L \
-b cookie.c \
-c cookie.c \
-b "was=1; uncheck2=1; uncheck3=1; uncheck4=1; uncheck5=0; hd=1; uncheck1=1"
> comic
If you need to know the names of the forms to be POSTed, you can run the following command :
mech-dump --forms http://site.com/
It comes with libwww-mechanize-perl package with debian or derivated.
Related
I use this curl command to download a setup.exe :
curl -k -u login:Password -O "https://url/to/my/setup.exe"
I want to get the time total :
curl -s -w 'total : %{time_total}\n' -k -u login:Password -O "https://url/to/my/setup.exe"
total : 41.165108
I want to redirect the output ( total : 41.165108 ) in an external file. I try :
curl -o test.txt -s -w 'total : %{time_total}\n' -k -u login:Password -O "https://url/to/my/setup.exe"
curl -s -w 'total : %{time_total}\n' -k -u login:Password -O "https://url/to/my/setup.exe" > test.txt
But it doesn't work...
Someone to show me ?
Thanks !
Just combine the two things you've tried:
curl -s -w 'total : %{time_total}\n' -k -u login:Password \
-O "https://url/to/my/setup.exe" -o setup.exe > total.txt
# ~~~~~~~~~~~~ ~~~~~~~~~~~
setup.exe will contain the downloaded file
total.txt will contain the total
The -o doesn't influence where the output of -w goes.
The -o parameter is in lowercase, and the file to output to, is to be entered after the parameter, an example is below,
curl -K myconfig.txt -o output.txt
What I'm looking for
I want to scrape a website and have an alert as soon as there are changes.
But I have a little trouble to understand the authentication mechanism set up by this website.
I tried almost all tricks to be able to connect via wget, but none worked.
The website : offre.astria.com
What I tried
username=XXXXX
password=XXXXX
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'username=$username&password=$password' \
--delete-after \
https://cas.astria.com/cas-ext/login?service=https://offre.astria.com
wget --load-cookies cookies.txt \
https://offre.astria.com
After your comments, I tried this too :
username=XXXX
password=XXXX
code=`wget -qO- https://cas.astria.com/cas-ext/login?service=https://offre.astria.com | cat | grep 'name="lt"' | cut -d"_" -f2`
hidden_code=_$code
wget --save-cookies cookies.txt \
--keep-session-cookies \
--post-data 'username=$username&password=$password<=$hidden_code&_eventId=submit' \
--delete-after \
https://cas.astria.com/cas-ext/login?service=https://offre.astria.com
wget --load-cookies cookies.txt \
https://offre.astria.com
And the error message goes from http 302 to http 500 servlet internal error.
Maybe, because the hidden field change the value between the first attempt and the second ...
I tried this too
username=XXXX
password=XXXX
code=`wget -qO- --save-cookies cookies.txt --keep-session-cookies --delete-after https://cas.astria.com/cas-ext/login?service=https://offre.astria.com | cat | grep 'name="lt"' | cut -d"_" -f2,3 | cut -d"\"" -f1`
wget --load-cookies cookies.txt \
--post-data 'username=$username&password=$password<=$hidden_code&_eventId=submit' \
--delete-after \
https://cas.astria.com/cas-ext/login?service=https://offre.astria.com
wget --load-cookies cookies.txt \
https://offre.astria.com
And same results :s
If you look at the source of the web page, you will find the login form has a hidden field:
<input type="hidden" name="lt" value="_c72D26258-FFFE-C561-D3EC-9BAC7F85B9A4_kCB5ECD54-57E0-1FD5-8A2F-BF72E114C604" />
The value of this field is generated server side, and changes every time you refresh the page.
This is a mechanism to prevent exactly what you are trying to do.
You may be able to bypass it, by having a script read the page, extract the value, and then use wget to submit a proper form.
It is however possible that you will encounter other defensive methods on the site.
I am attempting to run the following script on a virtual instance I have created on the google cloud:
#!/bin/bash
set -eu
DS=$(date "+%Y-%m-%d" -d "7 days ago")
DE=$(date "+%Y-%m-%d" -d "1 day ago")
account=123
## above specifies last weeks delivery
rm -f cookiejar
curl /dev/null -s -S -L -f -c cookiejar 'https://url.io/auth/authenticate' -d name=usr -d passwd='pwd'
curl -o /dev/null -s -S -L -f -b cookiejar -c cookiejar 'https://adloox.io/auth/adminaccounts' -d account=$account
curl -s -S -L -f -o "report1.xlsx" -J -b cookiejar -c cookiejar "https://url.io/adquality/ajax-adblocking?categoryFw=&platform_id[]=7&id1=All&id2=&id3=All&id4=All&id5=&id11=&date=2019-12-09&date_start=$DS&date_end=$DE&website=&keywords=&zfTablePage=1&zfTableColumn=&zfTableOrder=desc&zfTableQuickSearch=&zfTableItemPerPage=100&zfTableDataTablesMaxRows=2628&zfTableItemPerPage=10000&zfTableExport=xlsx"
curl -s -S -L -f -o "report2.xlsx" -J -b cookiejar -c cookiejar "https://url.io/report/ajax-by-tag2?platform_id[]=7&id1=All&id2=&id3=All&id4=All&id5=&id11=&date=2019-12-09&date_start=$DS&date_end=$DE&website=&zfTablePage=1&zfTableColumn=&zfTableOrder=desc&zfTableQuickSearch=&zfTableItemPerPage=100&zfTableDataTablesMaxRows=10000&zfDetails=true&by_viewability=imps_sivt&device_id[]=all&tag_type_id[]=all&support_id[]=all&by_website=1&zfTableItemPerPage=10000&zfTableExport=xlsx"
Set Up
I have curl installed on my local desktop (running windows 10 with `Ubuntu 18.04.3 LTS)
I have curl installed on my remote (google cloud virtual instance set up with Ubuntu 18.04.3 LTS`)
Both have version curl 7.58.0
Issue
When run on my local desktop there is no issue and the files download. When run on my remote I am able to log in however I receive the following error for the next curl line:
+ curl -o /dev/null -s -S -L -f -b cookiejar -c cookiejar https://url.io/auth/adminaccounts -d account=123
curl: (22) The requested URL returned error: 500 Internal Server Error
Can someone confirm what else I should be looking at here? I would have thought if my linux and curl versions are the same, there wouldn't be an issue. Sorry if there are some other straight forward checks to do. This is my first time setting up a server.
If your virtual instance does not have a public IP address, this might help you out.
#!/bin/bash
set -eu
DS=$(date "+%Y-%m-%d" -d "7 days ago")
DE=$(date "+%Y-%m-%d" -d "1 day ago")
account=123
## above specifies last weeks delivery
rm -f cookiejar
curl /dev/null -s -S -L -f -c cookiejar 'https://url.io/auth/authenticate' -d name=usr -d passwd='pwd'
curl -o /dev/null -s -S -f -b cookiejar -c cookiejar 'https://adloox.io/auth/adminaccounts' -d account=$account
curl -s -S -L -f -o "report1.xlsx" -J -b cookiejar -c cookiejar "https://url.io/adquality/ajax-adblocking?categoryFw=&platform_id[]=7&id1=All&id2=&id3=All&id4=All&id5=&id11=&date=2019-12-09&date_start=$DS&date_end=$DE&website=&keywords=&zfTablePage=1&zfTableColumn=&zfTableOrder=desc&zfTableQuickSearch=&zfTableItemPerPage=100&zfTableDataTablesMaxRows=2628&zfTableItemPerPage=10000&zfTableExport=xlsx"
curl -s -S -L -f -o "report2.xlsx" -J -b cookiejar -c cookiejar "https://url.io/report/ajax-by-tag2?platform_id[]=7&id1=All&id2=&id3=All&id4=All&id5=&id11=&date=2019-12-09&date_start=$DS&date_end=$DE&website=&zfTablePage=1&zfTableColumn=&zfTableOrder=desc&zfTableQuickSearch=&zfTableItemPerPage=100&zfTableDataTablesMaxRows=10000&zfDetails=true&by_viewability=imps_sivt&device_id[]=all&tag_type_id[]=all&support_id[]=all&by_website=1&zfTableItemPerPage=10000&zfTableExport=xlsx"
This works:
curl -k -d 'VARTEST=vartest2&TEXTPARAM=text1%0Atext2&COMMENT=abc22' http://localhost:8080/job/TEST1/buildWithParameters?delay=0sec
curl -X POST -k 'http://localhost:8080/job/TEST1/buildWithParameters?delay=0sec&VARTEST=vartest2&TEXTPARAM=text1%0Atext2&COMMENT=abc22'
I want to be able to write the parameters to -d into a file and run the command line like this
curl -k -d #persargfile 'http://localhost:8080/job/TEST1/buildWithParameters?delay=0sec' -o abc.html
OR like this
curl -k --data-urlencode #persargfile 'http://localhost:8080/job/TEST1/buildWithParameters?delay=0sec' -o abc.html
Question:
What should be the format of the persargfile?
As the man page states, there are several formats available:
ascii (--data-ascii)
urlencoded (--data-urlencoded)
binary (--data-binary)
It looks like for your application the ascii option is required. Meaning your file should contain:
VARTEST=vartest2&TEXTPARAM=text1%0Atext2&COMMENT=abc22
I need to login to a page and then submit an entry.
i'm able to login with the following code
curl $HOST -s -L -b $COOKIE -c $COOKIE -d "login=submit&login_name=$USER&login_password=$PASS" -o $OUTPUT
but when i try to submit an entry i'm not logged in anymore
curl $HOSTADDNEWS -L -b $COOKIE -c $COOKIE -A $HEADER -d "title=$TITLE" -o $OUTPUT.add.news
How to retain cookies between requests?
Thanks to Daniel Stenberg! $COOKIE variable had a "slash" in it... it still saved to file but wasn't able to use it.