I am attempting to scrape some data from an internal GUI for a modem in the field. Here is the code I am using:
## import the requests python library
import requests
## defines the login URL and the Post URL
post_loginURL = 'http://100.255.255.255/cgi-bin/login.cgi'
## defines the request URL to scrape from
requestURL = 'http://100.255.255.255/cgi-bin/stat_eth0.cgi'
## define the login data paramaters
payload = {
'NAME': 'username',
'PWD': 'password',
'CMD': '1' ## hidden value being passed in the post request
}
## define the content type as text/html
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
## creates the session object
with requests.Session() as session:
## sends the post request, headers and login data to the server
post = session.post(post_loginURL, data=payload, headers=headers)
r = session.get(requestURL)
print(r.text)
Here is the HTML Error message I receive in the IDLE console:
<em><u><p align="right">ERROR: You must log in</u></em>
Here are the details from the Chrome Dev Tools:
screenshot
I am unsure if I am just missing a crucial step, or if this is an issue with syntax, etc. Any constructive advice is greatly appreciated.
Thanks in advance!
Related
I'm trying to login to a website via python to print the info. So I don't have to keep logging into multiple accounts.
In the tutorial I followed, he just had a login and password, but this one has
Website Form Data
Does the _wp attributes change each login?
The code I use:
mffloginurl = ('https://myforexfunds.com/login-signup/')
mffsecureurl = ('https://myforexfunds.com/account-2')
payload = {
'log': '*****#gmail.com',
'pdw': '*****'
'''brandnestor_action':'login',
'_wpnonce': '9d1753c0b6',
'_wp_http_referer': '/login-signup/',
'_wpnonce': '9d1753c0b6',
'_wp_http_referer': '/login-signup/'''
}
r = requests.post(mffloginurl, data=payload)
print(r.text)
using the correct details of course, but it doesn't login.
I tried without the extra wordpress elements and also with them but it still just goes to the signin page.
python output
different site addresses, different login details
Yeah the nonce will change with every new visit to the page.
I would use request.session() so that it automatically stores session cookies and all that good stuff.
Do a session.GET('some_login_page.com')
Parse with the response content with BeautifulSoup to retrieve the nonce.
Then add that into the payload of your POST request when you login.
A very quick and dirty example:
import requests
from bs4 import BeautifulSoup as bs
email = 'test#email.com'
password = 'password1234'
url = 'https://myforexfunds.com/account-2/'
# Start a session
with requests.session() as session:
# Send a GET request to the login page
r = session.get(url)
# Check if the request was successful
if r.status_code != 200:
print("Get Request Failed")
# Parse the HTML content of the page
soup = bs(r.content, 'lxml')
# Extract the value of the nonce from the HTML
nonce = soup.find(id='woocommerce-login-nonce')['value']
# Set up the login form data
params ={
"username": email,
"password": password,
"woocommerce-login-nonce": nonce,
"_wp_http_referer": "/account-2/",
"login": "Log+in"
}
# Send a POST request with the login form data
r = session.post(url, params=params)
# Check if the request was successful
if r.status_code != 200:
print("Login Failed")
I'm trying to use requests to login to a site, navigate to a page, and scrape some data. This question is about the first step (to get in).
I cannot fetch the token from the site:
import requests
URL = 'https://coderbyte.com/sl'
with requests.Session() as s:
response = s.get(URL)
print([response.cookies])
Result is empty:
[<RequestsCookieJar[]>]
This make sense according to the response I'm seeing in Chrome's dev tools. After I login with my username and password, I see four tokens, three of them deleted, but one valid:
How can I fetch the valid token?
you can use the post method to the url you want in order to fetch the token (to pass the login first). For example :
url = "url-goes-here"
url_login = "login-url-goes-here"
with requests.Session() as s:
# get the link first
s.get(url)
payload = json.dumps({
"email" : "your-email",
"password" : "your-password"
})
headers = {
'Content-Type': 'application/json'
}
response = s.post(url=url_login, data=payload, headers=headers)
print(response.text)
Based on your question, i assume that if you only use username or password to login, then you can use HTTPBasicAuth() which is provided by requests package.
I am trying to access swagger url that gives response upon sending request in json format. I have disabled to avoid SSLCertVerificationError using verify=False. How to access that please help me to read it using python code. You can get details what I have tried till now.
Installed requests library by using cmd in windows = pip install requests
I am able to get the 200 response by using code mentioned
When I am trying to use "params= payload" where. I am not able to get the response the same I am getting in the swagger url.
url : https:///#!/prepaid-estimate-controller/getAccountDetailsUsingPOST
payload={'searchCriteria': 'string', 'value': 'string'}
r4 = requests.get('https://<domain>/#!/prepaid-estimate-controller/getAccountDetailsUsingPOST', verify=False)
print(r4)
Output : 200
payload={'searchCriteria': 'string', 'value': 'string'}
r4 = requests.get('https://<domain>/#!/prepaid-estimate-controller/getAccountDetailsUsingPOST',params=payload, verify=False)
print(r4)
print(r4.text)
Output: DOM structure returns
This issue has been fixed by using below code generated in the postman.
import requests
url = "https:///bsi/prepaid/getAccountDetails"
payload = "{ \r\n "searchCriteria": "BAID", \r\n "value": "PPB006788"\r\n}"
headers = {
'Content-Type': 'application/json'
}
response = requests.request("POST", url, headers=headers, data = payload, verify=False)
print(response.text.encode('utf8'))
This question has been asked many times but every single accepted answer utilises other librarys. I'm developing within an environment where i cannot use urllib2, http, or requests. My only option is to use urllib or write my own.
I need to send get and post requests to a server locally that requires authentication. I have no problem with the requests and this was all working until the latest security update enforced authentication. Authentication is done via cookies only.
I can send my authentication post and receive a status 200 with successful response. What i'm struggling with is pulling the cookie values out of this response and attaching them to all future post requests using urllib only.
import urllib.request, json
url = "serverurl/login"
data = {
"name" : "username",
"password" : "password"
}
jsonData = json.dumps(data).encode('utf-8')
req = urllib.request.Request(url, data=jsonData, headers={'content-type': 'application/json'})
response = urllib.request.urlopen(req).read().decode('utf8')
print(response)
For others reference, After a few hours of trial and error and cookie research the following got a working solution.
import urllib.request, json
url = "serverurl/login"
data = {
"name" : "username",
"password" : "password"
}
jsonData = json.dumps(data).encode('utf-8')
req = urllib.request.Request(url, data=jsonData, headers={
'content-type': 'application/json'
})
response = urllib.request.urlopen(req)
cookies = response.getheader("Set-Cookie")
then in future posts you add "Cookie" : cookies to the request
req = urllib.request.Request(url, data=jsonData, headers={
"content-type" : "application/json",
"Cookie" : cookies
})
I am trying to create a new table on Airtable with the aid of the post method. I have the following code :
# importing the requests library
import requests
# defining the api-endpoint
API_ENDPOINT = "https://api.airtable.com/v0/appa3r2UUo4JxpjSv/Table%201?api_key=MYKEY"
# data to be sent to api
data = {
'fields': {
'Name': 'Andromachis Row'
}
}
# sending post request and saving response as response object
r = requests.post(url = API_ENDPOINT, data = data)
# extracting response text
print(r.text)
Despite that when I run the script I get an error saying:
(mypyth) PS C:\Users\andri\PythonProjects\mypyth> py post_API.py
{"error":{"type":"INVALID_REQUEST_UNKNOWN","message":"Invalid request: parameter validation failed. Check your request data."}}
Does anyone understand why this happens? I am really desperate! Thanks in advance