We recently set up a website (http://www.doverjewelry.com/) with hikashop, the domain has godaddy website protection so it scans the website and warns against vulnerabilities. The scan is currently reporting the the website is vulnerable to cross-site scripting attacks. This the scan output:
Using the GET HTTP method, Site Scanner found that :
+ The following resources may be vulnerable to XSS (on parameters names) :
/bands-and-settings/category/371-all-ring-settings/limit_hikashop_catego
ry_information_module_223_371-0/limitstart_hikashop_category_information
_module_223_371-0/filter_order_hikashop_category_information_module_223_
371-a.ordering/filter_order_Dir_hikashop_category_information_module_223
_371-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-atom?<<<<<<<<<<foo"bar'
314>>>>>=1
-------- request --------
GET /bands-and-settings/category/371-all-ring-settings/limit_hikashop_category_information_module_223_371-0/limitstart_hikashop_category_information_module_223_371-0/filter_order_hikashop_category_information_module_223_371-a.ordering/filter_order_Dir_hikashop_category_information_module_223_371-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-atom?<<<<<<<<<<foo"bar'314>>>>>=1 HTTP/1.1\r
Host: www.doverjewelry.com\r
Accept-Charset: iso-8859-1,utf-8;q=0.9,*;q=0.1\r
Accept-Language: en\r
Connection: Close\r
Cookie: 7eedc822c6dd39ecf3c8ab00003d56f9=764a229107bda6b48c2863965f50ca03\r
User-Agent: Mozilla/5.0 (compatible; MSIE 7.0; MSIE 6.0; Site Scanner Bot; +http://www.websiteprotection.com) Firefox/2.0.0.3\r
Pragma: no-cache\r
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
------------------------
-------- output --------
[...] bd44a6ec-1/type-atom?<<<<<<<<<<foo"bar'314>>>>>=1" method="post" name="ad [...]
<div class="hikashop_products_pagination hikashop_products_paginat [...]
------------------------
/engagement-rings/category/366-antique-engagement-rings/limit_hikashop_c
ategory_information_module_222_366-25/limitstart_hikashop_category_infor
mation_module_222_366-0/filter_order_hikashop_category_information_modul
e_222_366-a.ordering/filter_order_Dir_hikashop_category_information_modu
le_222_366-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-atom?<<<<<<<<<<fo
o"bar'314>>>>>=1
-------- request --------
GET /engagement-rings/category/366-antique-engagement-rings/limit_hikashop_category_information_module_222_366-25/limitstart_hikashop_category_information_module_222_366-0/filter_order_hikashop_category_information_module_222_366-a.ordering/filter_order_Dir_hikashop_category_information_module_222_366-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-atom?<<<<<<<<<<foo"bar'314>>>>>=1 HTTP/1.1\r
Host: www.doverjewelry.com\r
Accept-Charset: iso-8859-1,utf-8;q=0.9,*;q=0.1\r
Accept-Language: en\r
Connection: Close\r
Cookie: 7eedc822c6dd39ecf3c8ab00003d56f9=764a229107bda6b48c2863965f50ca03\r
User-Agent: Mozilla/5.0 (compatible; MSIE 7.0; MSIE 6.0; Site Scanner Bot; +http://www.websiteprotection.com) Firefox/2.0.0.3\r
Pragma: no-cache\r
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
------------------------
-------- output --------
[...] bd44a6ec-1/type-atom?<<<<<<<<<<foo"bar'314>>>>>=1" method="post" name="ad [...]
<div class="hikashop_products_pagination hikashop_products_paginat [...]
------------------------
/engagement-rings/category/366-antique-engagement-rings/limit_hikashop_c
ategory_information_module_222_366-25/limitstart_hikashop_category_infor
mation_module_222_366-0/filter_order_hikashop_category_information_modul
e_222_366-a.ordering/filter_order_Dir_hikashop_category_information_modu
le_222_366-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-rss?<<<<<<<<<<foo
"bar'314>>>>>=1
-------- request --------
GET /engagement-rings/category/366-antique-engagement-rings/limit_hikashop_category_information_module_222_366-25/limitstart_hikashop_category_information_module_222_366-0/filter_order_hikashop_category_information_module_222_366-a.ordering/filter_order_Dir_hikashop_category_information_module_222_366-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-rss?<<<<<<<<<<foo"bar'314>>>>>=1 HTTP/1.1\r
Host: www.doverjewelry.com\r
Accept-Charset: iso-8859-1,utf-8;q=0.9,*;q=0.1\r
Accept-Language: en\r
Connection: Close\r
Cookie: 7eedc822c6dd39ecf3c8ab00003d56f9=764a229107bda6b48c2863965f50ca03\r
User-Agent: Mozilla/5.0 (compatible; MSIE 7.0; MSIE 6.0; Site Scanner Bot; +http://www.websiteprotection.com) Firefox/2.0.0.3\r
Pragma: no-cache\r
Accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, image/png, */*
------------------------
-------- output --------
[...] abd44a6ec-1/type-rss?<<<<<<<<<<foo"bar'314>>>>>=1" method="post" name="ad [...]
<div class="hikashop_products_pagination hikashop_products_paginat [...]
------------------------
/engagement-rings/category/50-estate-engagement-rings/limit_hikashop_cat
egory_information_module_222_50-0/limitstart_hikashop_category_informati
on_module_222_50-0/filter_order_hikashop_category_information_module_222
_50-a.ordering/filter_order_Dir_hikashop_category_information_module_222
_50-ASC/688ae9879a2df0fc5b840aeabd44a6ec-1/type-atom?<<<<<<<<<<foo"bar'3
14>>>>>=1
We think it is refering to the pagination form at the bottom of the product pages. Here is the form code for one of the product pages:
<form action="http://www.doverjewelry.com/engagement-rings/category/50-estate-engagement-rings?filter_order_hikashop_category_information_module_222_50=%3C%3C%3C%3C%3C%3C%3C%3C%3C%3Cfoo%22bar'204%3E%3E%3E%3E%3E" method="post" name="adminForm_hikashop_category_information_module_222_50_bottom">
<div class="hikashop_products_pagination hikashop_products_pagination_bottom">
<div class="list-footer">
<div class="limit">Display #<select id="limit_hikashop_category_information_module_222_50" name="limit_hikashop_category_information_module_222_50" class="inputbox" size="1" onchange="this.form.submit()">
<option value="20" selected="selected">20</option>
<option value="5">5</option>
<option value="10">10</option>
<option value="15">15</option>
<option value="20" selected="selected">20</option>
<option value="25">25</option>
<option value="30">30</option>
<option value="50">50</option>
<option value="100">100</option>
<option value="0">all</option>
</select>
</div><span class="pagenav_start_chevron"><< </span><span class="pagenav pagenav_text">Start</span><span class="pagenav_previous_chevron"> < </span><span class="pagenav pagenav_text">Prev</span> <span class="pagenav">1</span> <a class="pagenav" title="2" onclick="javascript: document.adminForm_hikashop_category_information_module_222_50_bottom.limitstart_hikashop_category_information_module_222_50.value=20; document.adminForm_hikashop_category_information_module_222_50_bottom.submit();return false;">2</a> <a class="pagenav" title="3" onclick="javascript: document.adminForm_hikashop_category_information_module_222_50_bottom.limitstart_hikashop_category_information_module_222_50.value=40; document.adminForm_hikashop_category_information_module_222_50_bottom.submit();return false;">3</a> <a class="pagenav" title="Next" onclick="javascript: document.adminForm_hikashop_category_information_module_222_50_bottom.limitstart_hikashop_category_information_module_222_50.value=20; document.adminForm_hikashop_category_information_module_222_50_bottom.submit();return false;">Next</a><span class="pagenav_next_chevron"> ></span> <a class="pagenav" title="End" onclick="javascript: document.adminForm_hikashop_category_information_module_222_50_bottom.limitstart_hikashop_category_information_module_222_50.value=40; document.adminForm_hikashop_category_information_module_222_50_bottom.submit();return false;">End</a><span class="pagenav_end_chevron"> >></span>
<div class="counter">Page 1 of 3</div>
<input type="hidden" name="limitstart_hikashop_category_information_module_222_50" value="0">
</div>
<span class="hikashop_results_counter">
Results 1 - 20 of 48</span>
</div>
<input type="hidden" name="filter_order_hikashop_category_information_module_222_50" value="a.ordering">
<input type="hidden" name="filter_order_Dir_hikashop_category_information_module_222_50" value="ASC">
<input type="hidden" name="18aa959f74c6262cdb2863f0ffaff82e" value="1">
</form>
We have talked to the hikashop people about this and they say we need to update to their most recent version (our version is just one below the latest one) but we have made some major mods to the code to include some of the clients requests so we do not want to lose those changes (maybe in the future we will update to the latest version, but for now we just want to know if there is a quick fix for this).
Is the form really vulnerable to cross-site scripting attacks? what can we do to protect it or make godaddy site scanner stop showing this warning message?
From the output of scanner he thinks that when he issued a request with additional parameter:
<<<<<<<<<<foo"bar'314>>>>>=1
and this param got printed what we can see in output:
type-atom?<<<<<<<<<<foo"bar'314>>>>>=1
that could mean that your page is prone to XSS, but many of those scanners forgets encodings... the same issue is for example with scannig LifeRay with w3af. But your html code prints:
%3C%3C%3C%3C%3C%3C%3C%3C%3C%3Cfoo%22bar'204%3E%3E%3E%3E%3E
So it seems that the param althogh appended, is escaped... so it is not strictly prone to XSS. If you want to know more visit XSS - Cheat Sheet, and you can use some other vuln scanners/proxies to confirm this issue: ZAP, WebScarab, w3af.
Related
I am running a python based server inside a container. I can access it inside my host machine
curl --header "Content-Type: application/json" --request POST --data '{"uid":"admin","password":"admin"}' http://localhost:9000/auth
Result:
{"access_token": "eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJleHAiOjE1OTA4Mjk1NDAsImlhdCI6MTU5MDgyNTk0MCwibmJmIjoxNTkwODI1OTQwLCJzdWIiOiJhZG1pbiJ9.iTexlDupUMYYrodw44GI9ZnsTXnl5MurAXq6JCfqM0A"}
But now i am trying to do same curl inside another container, But It gives me access denied error.
Note: Unnecessary use of -X or --request, POST is already inferred.
* Expire in 0 ms for 6 (transfer 0x564809d7ff50)
* Uses proxy env variable http_proxy == 'http://10.223.4.20:911'
* Trying 10.223.4.20...
* TCP_NODELAY set
* Expire in 200 ms for 4 (transfer 0x564809d7ff50)
* Connected to 10.223.4.20 (10.223.4.20) port 911 (#0)
> POST http://localhost:9000/auth HTTP/1.1
> Host: localhost:9000
> User-Agent: curl/7.64.0
> Accept: */*
> Proxy-Connection: Keep-Alive
> Content-Type: application/json
> Content-Length: 34
>
* upload completely sent off: 34 out of 34 bytes
< HTTP/1.1 403 Forbidden
< Cache-Control: no-cache
< Pragma: no-cache
< Content-Type: text/html; charset=utf-8
< Proxy-Connection: Keep-Alive
< Connection: Keep-Alive
< Content-Length: 642
<
<HTML><HEAD>
<TITLE>Access Denied</TITLE>
</HEAD>
<BODY>
<FONT face="Helvetica">
<big><strong></strong></big><BR>
</FONT>
<blockquote>
<TABLE border=0 cellPadding=1 width="80%">
<TR><TD>
<FONT face="Helvetica">
<big>Access Denied (policy_denied)</big>
<BR>
<BR>
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
Your system policy has denied access to the requested URL.
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica">
</FONT>
</TD></TR>
<TR><TD>
<FONT face="Helvetica" SIZE=2>
<BR>
For assistance, contact your network support team.
</FONT>
</TD></TR>
</TABLE>
</blockquote>
</FONT>
</BODY></HTML>
* Connection #0 to host 10.223.4.20 left intact
All the containers are mapped as network_mode: host.
Here is my Docker-compose.yml
version: '2'
services:
tacotron:
image: tacotron-image
network_mode: host
command: python3 runserver.py
tts_driver:
image: tts_driver
privileged: true
network_mode: host
environment:
- ASR_PUB_PORT=5555
- ASR_PUB_TOPIC=subnlptopic
- TTS_DRIVER_PUB_PORT=5556
- TTS_DRIVER_PUB_TOPIC=pubttstopic
command: python3 /app/TTSDriver.py
What i am doing wrong here?
Thanks
Akshay
I am trying to scrape formularylookup.com, a site with information on the market for pharmaceuticals.
It requires a login:
username: -
password: -
I need the information for the medicine called Rybelsus.
When I look into the Inspect-> Network -> XHR I suspect there could be an easy way to get the required data form this page:
https://formularylookup.com/Formulary/Coverage?ProductId=237171&ProductName=Rybelsus&ChannelId=1&DrugTypeId=3&StateId=all&Options=SummaryCoverages
I identified this site, which might give an idea of how to connect to formularylookup.com, but I am very inexperienced with connecting to API's.
Here's my code:
import requests
from bs4 import BeautifulSoup
url ="https://api.mmitnetwork.com/Formulary/v1/Products?Name=rybelsus"
params = {
"ProductId":"237171",
"productSearch":"Rybelsus"}
headers = {
"authorization":"Bearer H-oa4ULGls2Cpu8U6hX4myixRoFIPxfj",
"Access-Token":"H-oa4ULGls2Cpu8U6hX4myixRoFIPxfj",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36",
"X-Requested-With": "XMLHttpRequest",
"Host": "formularylookup.com",
"X-NewRelic-ID": "XAYCVFZSGwcGU1lXBAI="
}
res = requests.get(url ,params=params ,headers = headers)
soup = BeautifulSoup(res.content, "lxml")
print(soup.prettify())
Which gives me the following response:
<!DOCTYPE html>
<html>
<head>
<title>
The resource cannot be found.
</title>
<meta content="width=device-width" name="viewport"/>
<style>
body {font-family:"Verdana";font-weight:normal;font-size: .7em;color:black;}
p {font-family:"Verdana";font-weight:normal;color:black;margin-top: -5px}
b {font-family:"Verdana";font-weight:bold;color:black;margin-top: -5px}
H1 { font-family:"Verdana";font-weight:normal;font-size:18pt;color:red }
H2 { font-family:"Verdana";font-weight:normal;font-size:14pt;color:maroon }
pre {font-family:"Consolas","Lucida Console",Monospace;font-size:11pt;margin:0;padding:0.5em;line-height:14pt}
.marker {font-weight: bold; color: black;text-decoration: none;}
.version {color: gray;}
.error {margin-bottom: 10px;}
.expandable { text-decoration:underline; font-weight:bold; color:navy; cursor:hand; }
#media screen and (max-width: 639px) {
pre { width: 440px; overflow: auto; white-space: pre-wrap; word-wrap: break-word; }
}
#media screen and (max-width: 479px) {
pre { width: 280px; }
}
</style>
</head>
<body bgcolor="white">
<span>
<h1>
Server Error in '/' Application.
<hr color="silver" size="1" width="100%"/>
</h1>
<h2>
<i>
The resource cannot be found.
</i>
</h2>
</span>
<font face="Arial, Helvetica, Geneva, SunSans-Regular, sans-serif ">
<b>
Description:
</b>
HTTP 404. The resource you are looking for (or one of its dependencies) could have been removed, had its name changed, or is temporarily unavailable. Please review the following URL and make sure that it is spelled correctly.
<br/>
<br/>
<b>
Requested URL:
</b>
/Formulary/v1/Products
<br/>
<br/>
<hr color="silver" size="1" width="100%"/>
<b>
Version Information:
</b>
Microsoft .NET Framework Version:4.0.30319; ASP.NET Version:4.6.1590.0
</font>
</body>
</html>
<!--
[HttpException]: The controller for path '/Formulary/v1/Products' was not found or does not implement IController.
at System.Web.Mvc.DefaultControllerFactory.GetControllerInstance(RequestContext requestContext, Type controllerType)
at System.Web.Mvc.DefaultControllerFactory.CreateController(RequestContext requestContext, String controllerName)
at System.Web.Mvc.MvcHandler.ProcessRequestInit(HttpContextBase httpContext, IController& controller, IControllerFactory& factory)
at System.Web.Mvc.MvcHandler.BeginProcessRequest(HttpContextBase httpContext, AsyncCallback callback, Object state)
at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
-->
<!--
This error page might contain sensitive information because ASP.NET is configured to show verbose error messages using <customErrors mode="Off"/>. Consider using <customErrors mode="On"/> or <customErrors mode="RemoteOnly"/> in production environments.-->
Update: I get an 404 error. Not sure why.
Below code will help you,
import requests
headers = {
'Accept': '*/*',
'X-Requested-With': 'XMLHttpRequest',
'Access-Token': '7Lq-KkDx2fCO_3kG90pLEpBS9Ssh62IQ',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36',
'Is-Session-Expired': 'false',
'Referer': 'https://formularylookup.com/',
}
response = requests.get('https://formularylookup.com/Formulary/Coverage?ProductId=237171&ProductName=Rybelsus&ChannelId=1&DrugTypeId=3&StateId=AL&Options=SummaryCoverages', headers=headers)
print(response.json())
Note: 'Is-Session-Expired': 'false' is very important in the header otherwise you'll get 404 error.
See it in action here
I'm trying to make an HTTPS proxy server, but I cant make a connection to any server.
Edit: this is the part of the code that, after a client connect to my server, I get the message and try to send to the web. This is the message of a FireFox client trying to connect to Google:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock = socket.create_connection((google.com, 443))
ssl_sock = ssl.wrap_socket(sock)
fullData=b''
ssl_sock.send(b'CONNECT www.google.com:443 HTTP/1.1\r\nUser-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:67.0) Gecko/20100101 Firefox/67.0\r\nProxy-Connection: keep-alive\r\nConnection: keep-alive\r\nHost: www.google.com:443\r\n\r\n')
while 1:
# receive data from web server
data = ssl_sock.recv(4026)
print(data)
if (len(data) > 0):
fullData+=data
else:
break
clientSock.send(fullData)
Google should got me a ok message but its getting me an error
HTTP/1.1 405 Method Not Allowed
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer
Content-Length: 1592
Date: Fri, 24 May 2019 05:28:17 GMT
Alt-Svc: quic=":443"; ma=2592000; v="46,44,43,39"
Connection: close
<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
<title>Error 405 (Method Not Allowed)!!1</title>
<style>
*{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}#media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}#media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}#media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
</style>
<a href=//www.google.com/><span id=logo aria-label=Google></span></a>
<p><b>405.</b> <ins>That\xe2\x80\x99s an error.</ins>
<p>The request method <code>CONNECT</code> is inappropriate for the URL <code>/</code>. <ins>That\xe2\x80\x99s all we know.</ins>
Trying hard to submit the form to no success.
This form is supposed to redirect and return new url with PDF.
Here is how to access the page in question:
Start with Search Page
Click on Document Type tab
Enter LP, click Search
Click View
Click Get Image
View PDF button is the one that Im interested in.
I need to mimic multipart formdata which looks like this:
<form name="courtform" action="http://oris.co.palm-beach.fl.us:8080/PdfServlet/PdfServlet27" method="post" enctype="multipart/form-data">
<input type="hidden" name="hostURL" value="http://oris.co.palm-beach.fl.us/or_web1/" size="60">
<input type="hidden" name="pdfPath" value="\\wcp01zfs-03.clerk.local\files2\ORISPDF\" size="60">
<input type="hidden" name="pdfURL" value="http://oris.co.palm-beach.fl.us/pdf/" size="60">
<input type="hidden" name="pages" value="1" size="60">
<!--<input type="hidden" name="pages" value="1" size="60">-->
<input type="hidden" name="id" value="22590889" size="60">
<input type="hidden" name="mpages" value="1" size="60">
<input type="hidden" name="doc_id" value="22590889" size="60">
<input type="hidden" name="page1" value="image_from_file.asp?imageurl=\\ors_fs\ORImage\O\30336\O.30336.1200.0001.tif" size="60">
<input type="hidden" name="WaterMarkText" value="1" size="60">
<input name="button" type="button" value="View PDF" onclick="javascript:ValidateAndSubmit(this.form)">
Here is part of my Scrapy code responsible for this request:
def get_image(self, response):
# inspect_response(response, self)
url = 'http://oris.co.palm-beach.fl.us:8080/PdfServlet/PdfServlet27'
headers = { 'Connection': 'keep-alive',
'origin': "http://oris.co.palm-beach.fl.us",
'upgrade-insecure-requests': "1",
'dnt': "1",
'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36",
'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
'cache-control': "max-age=0",
'Accept-Encoding': 'gzip,deflate',
}
id = response.xpath("//input[#name='doc_id']/#value").extract_first()
body = {'WaterMarkText': '0',
'hostURL': 'http://oris.co.palm-beach.fl.us/or_web1/',
'mpages': '1',
'page1': 'image_from_file.asp?imageurl=\\ors_fs\\ORImage\\O\\30338\\O.30338.0268.0001.tif',
'pages': '1',
'pdfPath': '\\wcp01zfs-03.clerk.local\\files2\\ORISPDF\\',
'pdfURL': 'http://oris.co.palm-beach.fl.us/pdf/',
}
body['doc_id'] = id
body['id'] = id
me = MultipartEncoder(fields=body, boundary='------WebKitFormBoundarygGHlhpHs08goICxO')
me_body = me.to_string()
headers['Content-Type'] =me.content_type
headers['Content-Length'] = me.len
yield scrapy.Request(url, method = 'POST', body = me_body, callback = self.get_pdf, headers = headers)
yield {'body':me_body}
def get_pdf(self, response):
inspect_response(response, self)
Whenever I run the code Im getting Response 400.
How do I mimic this form correctly?
UPDATE:
It appears I do not need to provide Content-Length manually.
After I removed it worked just one time. And then reverted to 404
error.
Is Boundary supposed to be new for every request? From what I read it
looks like it does not, since it is just a divider with no other
purpose.
I had to automate the entire process of filling the form and now it seems to work just fine.
def get_image(self, response):
# inspect_response(response, self)
item = response.meta['item']
url = 'http://oris.co.palm-beach.fl.us:8080/PdfServlet/PdfServlet27'
headers = {
'Connection': 'keep-alive',
'origin': "http://oris.co.palm-beach.fl.us",
'upgrade-insecure-requests': "1",
'dnt': "1",
'user-agent': "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36",
'accept': "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3",
'cache-control': "max-age=0",
'Accept-Encoding': 'gzip,deflate',
}
body={}
# Generate body from form
for i in response.xpath("//form[#name='courtform']/input"):
name = i.xpath(".//#name").extract_first()
val = i.xpath(".//#value").extract_first()
body[name] = val
# Remove watermakr from PDF
body['WaterMarkText'] = '0'
me = MultipartEncoder(fields=body, boundary='----WebKitFormBoundarygGHghpHs08goICxO')
me_body = me.to_string()
headers['Content-Type'] =me.content_type
yield scrapy.Request(url, method = 'POST', body = me_body, callback = self.get_pdf, headers = headers, meta={'item' : item})
I've deployed an application on Wildfly 10, that is working correctly on GlassFish, but I'm getting a blank page /<context_path>/j_security_check when I try to login.
I've looked some posts suggesting to include cache control request headers, but it didn't resolve the problem.
The logs do not show any kind of error or relevant information and I really don't know what to try next.
Has anyone experienced any similar issue?
EDIT 1
The authentication is working correctly. If, afterwards, I try to access a protected resource, I'm able to do so. It's just the redirect after the login that is not being triggered.
EDIT 2
The Request/Response dump:
----------------------------REQUEST---------------------------
URI=/ecc
characterEncoding=null
contentLength=-1
contentType=null
cookie=EPMSID=sfTmDLw92HjAhwfY7HUei5fzlUbwKjxUg3EhyTMk.d014349
header=Accept=text/html, application/xhtml+xml, */*
header=Connection=Keep-Alive
header=Accept-Language=pt-PT
header=Accept-Encoding=gzip, deflate
header=Cookie=EPMSID=sfTmDLw92HjAhwfY7HUei5fzlUbwKjxUg3EhyTMk.d014349
header=User-Agent=Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
header=Host=localhost:8443
locale=[pt_PT]
method=GET
protocol=HTTP/1.1
queryString=
remoteAddr=/127.0.0.1:62990
remoteHost=sibshare
scheme=https
host=localhost:8443
serverPort=8443
--------------------------RESPONSE--------------------------
contentLength=0
contentType=null
header=Connection=keep-alive
header=X-Powered-By=Undertow/1
header=Server=WildFly/10
header=Location=https://localhost:8443/ecc/
header=Content-Length=0
header=Date=Mon, 09 May 2016 08:18:33 GMT
status=302
==============================================================
2016-05-09 09:18:33,890 INFO [stdout] (default task-5) [DEBUG] ecc_src - NoCacheFilter:Initializing filter
2016-05-09 09:18:33,902 INFO [io.undertow.request.dump] (default task-5)
----------------------------REQUEST---------------------------
URI=/ecc/
characterEncoding=null
contentLength=-1
contentType=null
cookie=EPMSID=sfTmDLw92HjAhwfY7HUei5fzlUbwKjxUg3EhyTMk.d014349
header=Accept=text/html, application/xhtml+xml, */*
header=Connection=Keep-Alive
header=Accept-Language=pt-PT
header=Accept-Encoding=gzip, deflate
header=Cookie=EPMSID=sfTmDLw92HjAhwfY7HUei5fzlUbwKjxUg3EhyTMk.d014349
header=User-Agent=Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
header=Host=localhost:8443
locale=[pt_PT]
method=GET
protocol=HTTP/1.1
queryString=
remoteAddr=/127.0.0.1:62989
remoteHost=sibshare
scheme=https
host=localhost:8443
serverPort=8443
--------------------------RESPONSE--------------------------
contentLength=239
contentType=text/html
header=Expires=Thu, 01 Jan 1970 00:00:00 GMT
header=Cache-Control=no-cache, no-store, must-revalidate
header=X-Powered-By=Undertow/1
header=Server=WildFly/10
header=Pragma=no-cache
header=Accept-Ranges=bytes
header=Date=Mon, 09 May 2016 08:18:33 GMT
header=Connection=keep-alive
header=ETag=W/"239-1462554016000"
header=Last-Modified=Fri, 06 May 2016 17:00:16 GMT
header=Content-Type=text/html
header=Content-Length=239
status=200
==============================================================
2016-05-09 09:18:34,112 INFO [io.undertow.request.dump] (default task-6)
----------------------------REQUEST---------------------------
URI=/ecc/secure/home.jsf
characterEncoding=null
contentLength=-1
contentType=null
cookie=EPMSID=sfTmDLw92HjAhwfY7HUei5fzlUbwKjxUg3EhyTMk.d014349
header=Accept=text/html, application/xhtml+xml, */*
header=Connection=Keep-Alive
header=Accept-Language=pt-PT
header=Accept-Encoding=gzip, deflate
header=Cookie=EPMSID=sfTmDLw92HjAhwfY7HUei5fzlUbwKjxUg3EhyTMk.d014349
header=User-Agent=Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
header=Host=localhost:8443
locale=[pt_PT]
method=GET
protocol=HTTP/1.1
queryString=
remoteAddr=sibshare/127.0.0.1:62990
remoteHost=sibshare
scheme=https
host=localhost:8443
serverPort=8443
--------------------------RESPONSE--------------------------
contentLength=2897
contentType=text/html;charset=UTF-8
cookie=EPMSID=KcblRlqogTv4hCuVtjeL27onM3Nbp04k--DDZfnt.d014349; domain=null; path=/ecc
header=Expires=0
header=Expires=0
header=Cache-Control=no-cache, no-store, must-revalidate
header=Cache-Control=no-cache, no-store, must-revalidate
header=X-Powered-By=Undertow/1
header=Set-Cookie=EPMSID=KcblRlqogTv4hCuVtjeL27onM3Nbp04k--DDZfnt.d014349; path=/ecc; secure; HttpOnly
header=Server=WildFly/10
header=Pragma=no-cache
header=Pragma=no-cache
header=Date=Mon, 09 May 2016 08:18:34 GMT
header=Connection=keep-alive
header=Content-Type=text/html;charset=UTF-8
header=Content-Length=2897
status=200
==============================================================
2016-05-09 09:18:44,841 INFO [io.undertow.request.dump] (default task-13)
----------------------------REQUEST---------------------------
URI=/ecc/j_security_check
characterEncoding=null
contentLength=68
contentType=[application/x-www-form-urlencoded]
cookie=EPMSID=KcblRlqogTv4hCuVtjeL27onM3Nbp04k--DDZfnt.d014349
header=Accept=text/html, application/xhtml+xml, */*
header=Accept-Language=pt-PT
header=Cache-Control=no-cache
header=Accept-Encoding=gzip, deflate
header=User-Agent=Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; WOW64; Trident/5.0)
header=Connection=Keep-Alive
header=Content-Type=application/x-www-form-urlencoded
header=Content-Length=68
header=Cookie=EPMSID=KcblRlqogTv4hCuVtjeL27onM3Nbp04k--DDZfnt.d014349
header=Referer=https://localhost:8443/ecc/secure/home.jsf
header=Host=localhost:8443
locale=[pt_PT]
method=POST
protocol=HTTP/1.1
queryString=
remoteAddr=sibshare/127.0.0.1:62993
remoteHost=sibshare
scheme=https
host=localhost:8443
serverPort=8443
--------------------------RESPONSE--------------------------
contentLength=0
contentType=null
cookie=EPMSID=JtIoopj1u-p_Ko95XwYi45HqkdzNBVRxSklVFQEL.d014349; domain=null; path=/ecc
header=Expires=0
header=Cache-Control=no-cache, no-store, must-revalidate
header=X-Powered-By=Undertow/1
header=Set-Cookie=EPMSID=JtIoopj1u-p_Ko95XwYi45HqkdzNBVRxSklVFQEL.d014349; path=/ecc; secure; HttpOnly
header=Server=WildFly/10
header=Pragma=no-cache
header=Date=Mon, 09 May 2016 08:18:44 GMT
header=Connection=keep-alive
header=Content-Length=0
status=200
==============================================================
I eventually figured out what is wrong.
In my login page I've the following listener configured for invalidating the active session:
<f:metadata>
<f:event type="preRenderView" listener="#{manager.invalidateActiveSession}" />
</f:metadata>
This listener simply invalidates the session (if it exists):
HttpSession session = (HttpSession) FacesContext.getCurrentInstance().getExternalContext().getSession(false);
if (session != null ) {
synchronized( session ) {
session.invalidate();
}
}
And this is what is causing the strange behavior. This same code works fine on GlassFish.
I've changed the code to, additionally, verify if the Principal is also not null.