When using AIOHTTP to cookies of a session, it doesnt show the complete cookies.
I have tried to index through cookies, but it still just shows very few of them. I believe its using something called simple cookie but i could be wrong.
async with s.get(endpoint) as add:
print (endpoint.url)
sitetext = (await endpoint.text())
cookies = s.cookie_jar.filter_cookies(endpoint)
for each in cookies.items():
print (each)
Which yields results like:
('__cfduid', <Morsel: __cfduid=d2719c3095fb81ad69c19507dbe8bd7a01566353842>)
('_orig_referrer', <Morsel: _orig_referrer=>)
('_landing_page', <Morsel: _landing_page=productname>)
But this isn't the full cookies of the session, when i go and view the headers there are more cookies too, but im unsure of how to extract cookies from headers.
I tried getting cookies through headers by doing:
async with s.get(endpoint) as add:
print (endpoint.url)
sitetext = (await endpoint.text())
cookies = s.cookie_jar.filter_cookies(endpoint)
print (endpoint.headers['Set-Cookie'])
But this only gives me the first Set-Cookie and not the rest.
Edit: Im using AIOHTTP as this program has to be async and non blocking. (i know requests is much easier)
I am actually not sure, but I had similar problem. Try this:
cookies = response.cookies
cookie_string = "; ".join([str(value) for _, value in cookies.items()]).replace('Set-Cookie: ', '')
I have such format of answer:
__Host-ariregweb=g4q8icpEmmIdHnQmFSNsfHdNhLLsgISKE7XRzphrLiscb4pSIzrpRLm9aiLCHqXa; Domain=ariregister.rik.ee; HttpOnly; Path=/; SameSite=lax; Secure; __cf_bm=GIYOJNShc1kY79bPk7GG1U6T.jV6K4BG8DLQoc70NT8-1671428340-0-AcxNKFuuhvkmblvy/q4WPGEPezvLQUL8/k6NeylOmaX5awlf1L7eOWnc55DGMsyPzpv5YKUDL6w100KlzzjsJVE=; Domain=ariregister.rik.ee; expires=Mon, 19-Dec-22 06:09:00 GMT; HttpOnly; Path=/; SameSite=None; Secure; _cfuvid=0EIfdSb4ltpOgYzKuMthWVvN0x5L3kn_uCfhyvBEk34-1671428340155-0-604800000; Domain=ariregister.rik.ee; HttpOnly; Path=/; SameSite=None; Secure
As I see you try to bypass the CloudFlare protection. CloudFlare detects your scripted requests and gives 403 Forbidden status. So that you get non-completed cookies.
I recommend you to use Insomnia and Fiddler to understand what is happening with your request in browser and script. The only way to bypass the protection system is to define the key moments in headers or the character of request.
By the way, I have this problem currently. If you send the request through Fiddler in Python, you might be able to recieve completed cookies.
Good luck. But let me know if the solution was useful for your case.
Related
I recently deployed Juiceshop on Heroku and am trying to solve the challenge of forging an unsigned JWT token using Burpsuite. I followed the hints provided in the ebook starting from using burpsuite to get a valid JWT token from the request header using the proxy tab on the /rest/user/whoami URL and sent it to the repeater. In the repeater tab, I changed the alg property in the token to "none" and the email to "jwtn3d#juice-sh.op". I then encoded them in base64 and put a period after the header and payload respectively in the token value. When I send the request, I get a HTTP 200 OK status and the jwtn3d#juice-sh.op email is reflected as well, but the juiceshop challenge doesn't seem to be solved. When I reload the website, I'm not logged in as jwtn3d#juice-sh.op either.
Server: Cowboy
Connection: close
Access-Control-Allow-Origin: *
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Feature-Policy: payment 'self'
Set-Cookie: token=ewogICJ0eXAiOiAiSldUIiwKICAiYWxnIjogIm5vbmUiCn0.ewogICJzdGF0dXMiOiAic3VjY2VzcyIsCiAgImRhdGEiOiB7CiAgICAiaWQiOiAxLAogICAgInVzZXJuYW1lIjogIiIsCiAgICAiZW1haWwiOiAiand0bjNkQGp1aWNlLXNoLm9wIiwKICAgICJwYXNzd29yZCI6ICIwMTkyMDIzYTdiYmQ3MzI1MDUxNmYwNjlkZjE4YjUwMCIsCiAgICAicm9sZSI6ICJhZG1pbiIsCiAgICAiZGVsdXhlVG9rZW4iOiAiIiwKICAgICJsYXN0TG9naW5JcCI6ICIwLjAuMC4wIiwKICAgICJwcm9maWxlSW1hZ2UiOiAiYXNzZXRzL3B1YmxpYy9pbWFnZXMvdXBsb2Fkcy9kZWZhdWx0QWRtaW4ucG5nIiwKICAgICJ0b3RwU2VjcmV0IjogIiIsCiAgICAiaXNBY3RpdmUiOiB0cnVlLAogICAgImNyZWF0ZWRBdCI6ICIyMDIyLTAyLTA2IDExOjUyOjAxLjQwOSArMDA6MDAiLAogICAgInVwZGF0ZWRBdCI6ICIyMDIyLTAyLTA2IDExOjUyOjAxLjQwOSArMDA6MDAiLAogICAgImRlbGV0ZWRBdCI6IG51bGwKICB9LAogICJpYXQiOiAxNjQ0MTQ4NDcxLAogICJleHAiOiAxNjQ0MTY2NDcxCn0.; Path=/
Content-Type: application/json; charset=utf-8
Content-Length: 133
Etag: W/"85-PWOL8M3mYJv9caLvST18mm7K+2g"
Vary: Accept-Encoding
Date: Sun, 06 Feb 2022 11:57:25 GMT
Via: 1.1 vegur
{"user":{"id":1,"email":"jwtn3d#juice-sh.op","lastLoginIp":"0.0.0.0","profileImage":"assets/public/images/uploads/defaultAdmin.png"}}
I tried opening DevTools on the website and manually changing the cookie value to the forged token, and when I reloaded the website it reflected the jwtn3d#juice-sh.op email on the profile. But it still didn't solve the challenge. I've looked for tutorials everywhere and the challenges all seem to have just been solved once they sent the forged token over using the repeater, I'm at a loss as to why it isn't working in my case.
I found an alternative to using repeater, I used proxy intercept instead and changed the token value to the forged token on the /rest/user/whoami requests. That seemed to do the trick, though the notif pop up for the solved challenge didn't appear but it did show as solved in the scoreboard.
Locally, my cookies are set fine, but on Heroku with a cross-site call, they are not set in Chrome or Safari, the two browsers I've tried so far. I can only think that it is either because of the cross-site call or because of Heroku's proxy setup.
My Set-Cookie header looks like this:
CookieName=cookieValue; Max-Age=864; Domain=.myPurchasedDomain.net; Path=/; Expires=Tue, 21 Dec 2021 22:43:28 GMT; HttpOnly; Secure; SameSite=None
I have set my BE server to trust proxies, and both my FE and BE should only be available via https. I do not have any (visible) CORS issues. I am setting withCredentials: true in my Express BE and credentials: true in my FE axios options. I'm using next.js on my FE in case that might impact anything.
I don't know how to determine why my cookie is not being set. Any ideas of what I should try next or what might be causing the problem?
Edit: My FE axios call looks like this:
axios
.post(
"herokuUrl/login",
{
/* no body */
},
{
headers: {
Authorization: jwtToken, // Variable from elsewhere
},
withCredentials: true,
}
)
I found the answer:
My FE React app was calling a BE Express server at a different domain. Browsers block setting cookies from different domains. To fix, put both behind the same domain.
I am doing a POST request in python using requests library then i am getting multiple set-cookies in the response headers out of 4 i got only 2 are being sent as cookies in my subsequent request.
response = session.request("POST", url1, data=payload1, headers=headers1,verify=False,cookies=previous_response.cookies)
When i send request as mentioned above all the set cookies from previous request headers must set as cookies but only few of then are being set
For eg
Set-Cookie: sp_ac=AQB1uO6eg3fKauhLNID7uLdT9wuD_9Qy-FtXYhfl68sGO4YkUL2tKpY_EXlV_SKvngDccQOI5BaKSdycCA5U-7h1N5LQ7HH5wjQbGXSB6o7pcKBvhRXsBne4zHSSFsdExwBQ0m_AwVo9d8UjkfUXiGtStI8vvF-p9ZJctNSrqf14DFh1juqZpK3cV_AplvJDVGgZEnALUa6JrBQJLZLXrUnDM4aBvPT9qNc;Version=1;Domain=accounts.spotify.com;Path=/;Secure;HttpOnly
Set-Cookie: sp_dc=AQDCboRdqnDFFtXA8px3gqjA3UkFXu5ikby1DsKg6D3v3LkholwIGSZizDHnSGuFHieuTsitpr8ubYApjQRaH2asYAuQGdzJ69zuzjPU8g;Version=1;Domain=spotify.com;Path=/;Max-Age=31536000;Secure;HttpOnly;Expires=Mon, 22 Jun 2020 13:03:15 GMT
Set-Cookie: csrf_token=AQB0q1XS7kZ5saul8QVL-7NIZVqrrAeeXW5OQUzjys8SQXTLUBf9M2wNOOQeyGU2cB-assn4XKqB9vRx;Version=1;Domain=accounts.spotify.com;Path=/;Secure
Set-Cookie: sp_key=59071bcb-664c-4302-b98b-6ba15d54f605;Version=1;Domain=spotify.com;Path=/;Max-Age=31536000;Secure;Expires=Mon, 22 Jun 2020 13:03:15 GMT
these are my previous response headers
In my next request only 2 are being sent, i dont know the reason behind it and to how to solve it
Cookie: sp_dc=AQDCboRdqnDFFtXA8px3gqjA3UkFXu5ikby1DsKg6D3v3LkholwIGSZizDHnSGuFHieuTsitpr8ubYApjQRaH2asYAuQGdzJ69zuzjPU8g; sp_key=59071bcb-664c-4302-b98b-6ba15d54f605
this is my new cookie header in new request
Well i figured out the answer after lot if research the hosts seems to be different
So what i have done is make a small notification in the post
Instead of this
response = session.request("POST", url1, data=payload1, headers=headers1,verify=False,cookies=previous_response.cookies)
I changed the cookies to dict so that it would be host independent as follows
response = session.request("POST", url1, data=payload1, headers=headers1,verify=False,cookies=previous_response.cookies.get_dict())
I have:
1) A client side app that has its own domain: http://client.com
2) A server side app that has a separate domain: http://server.com
Now,
the scenario is:
1) Opening http://client.com/home in the browser, which displays an HTML page.
2) http://client.com/home redirects to http://server.com/login
3) http://server.com/login stores a cookie 'auth' and sends a redirect instruction to http://client.com/welcome
Response:
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Length: 104
Content-Type: text/html; charset=utf-8
Date: Wed, 16 Jan 2019 10:47:11 GMT
Location: http://client.com/welcome
Set-Cookie: auth=1479da80-197c-11e9-ba74-59606594e2fb; Path=/
Vary: Accept
X-Powered-By: Express
4) The browser receives the response, which does contain the cookie 'auth'
5) The browser redirects itself to http://client.com/welcome
6) 'auth' cookie is sent to http://client.com/welcome
Request:
Cookie: auth=1479da80-197c-11e9-ba74-59606594e2fb
7) http://client.com/welcome returns HTML but does not return the cookie 'auth'
8) http://client.com/welcome makes an AJAX request to http://server.com/data (CORS enabled), but the cookie 'auth' is not sent
9) http://server.com/data doesn't recognize the user because there is no cookie
The client side is an angular app hosted by Node.js
Edit:
As suggested, I've added to the response of server.com:
Access-Control-Allow-Credentials: true
but nothing has been changed.
Relevant client side code:
const headerOptions = new HttpHeaders({
'Content-Type': 'application/json', 'withCredentials': 'true', 'Access-Control-Allow-Origin': 'true', 'Access-Control-Allow-Credentials': 'true'
});
this.httpClient.get<any>(this.baseUrl + "data", { headers: headerOptions }).subscribe((res) => {
You should use the withCredentials option when sending your ajax request to your http://server.com and your server.com should have the Access-Control-Allow-Credentials set to true.
Example code in Node.JS server:
var cors = require('cors');
var corsOptions = {
origin: '*',
credentials: true };
app.use(cors(corsOptions));
More on this here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Credentials
Example code in Angular.JS client
import {RequestOptions, Request, RequestMethod} from '#angular/http';
const options = new RequestOptions({
method: RequestMethod.Post,
url: 'https://google.com',
withCredentials: true
});
More on this here: https://angular.io/api/http/RequestOptions
Also check this out: https://github.com/angular/angular/issues/24283 - it looks like a particular version of Angular had problems with this flag, so unless you're using a more up-to-date version, you might need to set the header explicitly.
The reasoning of this is, unless the server explicitly tells the client "I will accept cookies (set previously on my domain) passed by another domain" - accepting the cookies would be a security issue. More here: https://en.wikipedia.org/wiki/Cross-site_request_forgery
Your description of what is happening does not seem right.
http://server.com/login stores a cookie 'auth' and sends a redirect instruction to http://client.com/welcome
'auth' cookie is sent to http://client.com/welcome
That is not (or at least should not be) what is happening. When the browser requests http://server.com/login and gets back in the response a Set-Cookie header, the cookie is set on and restricted to the server.com domain, even if the response is a redirect. If you are seeing the 'auth' cookie sent to client.com then that is a cookie that client.com previously set.
Anyway, it seems that what you really care about is
http://client.com/welcome makes an AJAX request to http://server.com/data (CORS enabled), but the cookie 'auth' is not sent
There are a bunch of reasons this can happen.
CORS. You mentioned it was CORS enabled, but for the sake of others reading this, you must have the following CORS headers set on server.com
Access-Control-Allow-Origin: http://client.com
Access-Control-Allow-Credentials: true
Note that you cannot get away with using a wildcard for Access-Control-Allow-Origin when you are sending credentials. Also note that the origin has to be an exact match, including scheme (http or https). In practice, what servers generally do is read the Origin header of the request, check it against a white list, and if it allowed, copy the Origin header value from the request to the Access-Control-Allow-Origin header in the response.
You must set xhr.withCredentials = true in your XHR request. (See MDN for more details.)
Then after you have done all that, you have one other hurdle in your way. Because you are on client.com and trying to send a cookie to server.com, the server.com cookie is considered a "third-party" cookie. AFAIK all the major browsers have a setting that blocks third-party cookies for privacy, because they are most often used by trackers to gather marketing data for advertising. I believe most of them block third-party cookies by default, but I am not sure of that. For sure lots of people have set their browsers to block third-party cookies.
So you have to tell your visitors to configure their browser to allow third-party cookies from server.com.
BTW, it is not safe to set a cookie on a redirect to a different domain. While it is allowed under the specification AFAIK, there have been issues with browser support. See, for example, this Chrome bug.
From Access-Control-Allow-Origin spec:
For requests without credentials, the literal value "" can be specified, as a wildcard;*
Try to add specific domain to Access-Control-Allow-Origin field.
I think you should use proxy in angular angular app. For more info check this link: https://github.com/angular/angular-cli/blob/master/docs/documentation/stories/proxy.md
I have 2 separate apps, let's call them Login & Dashboard. Both apps have a UI written in react and an express server.
In my Login app, when I make a POST from my Login UI, it hits the Login Express server to authenticate. Once authenticated, I set a cookie and redirect to my Dashboard url:
res.cookie(cookie.key, cookie.access_token, {
path: '/',
domain: cookie.domain,
httpOnly: true,
maxAge: cookie.rememberExpiry
})
res.redirect(dashboard_url)
However when I use req.cookies in my dashboard app I don't see any cookies.
When I make the POST from my Login UI I do indeed see a network call stating response header:
Set-Cookie: mycookie=cookievalue; Max-Age=28800; Domain=.local.myurl.com; Path=/; Expires=Thu, 03 Nov 2016 19:20:39 GMT; HttpOnly
Note that as of this moment the time is Nov 3 2016, 11:28 GMT so its not an expiry issue.
To test I have edited my hosts file such that login.local.myurl.com & dashboard.local.myurl.com point to localhost.
Is there any reason why the req.cookies is not available in the Dashboard express app??
My problem was 2 fold.
First I needed to set credentials: 'same-origin' on fetch, which is to say that I had to allow cookies to persist on the request library I was using.
Second, because my server and my client are essentially separate, a redirect on the server did not have the intended effect on the client. Hence I could not just res.redirect from the server response. instead I replaced the res.redirect line with res.status(200).send() and in my client code, I simply did window.location.replace('http://dashboardurl.com').
Hope that helps anyone who has this issue in the future.