How to use urllib with username/password authentication in python 3? - python-3.x

Here is my problem with urllib in python 3.
I wrote a piece of code which works well in Python 2.7 and is using urllib2. It goes to the page on Internet (which requires authorization) and grabs me the info from that page.
The real problem for me is that I can't make my code working in python 3.4 because there is no urllib2, and urllib works differently; even after few hours of googling and reading I got nothing. So if somebody can help me to solve this, I'd really appreciate that help.
Here is my code:
request = urllib2.Request('http://mysite/admin/index.cgi?index=127')
base64string = base64.encodestring('%s:%s' % ('login', 'password')).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib2.urlopen(request)
resulttext = result.read()

Thankfully to you guys I finally figured out the way it works.
Here is my code:
request = urllib.request.Request('http://mysite/admin/index.cgi?index=127')
base64string = base64.b64encode(bytes('%s:%s' % ('login', 'password'),'ascii'))
request.add_header("Authorization", "Basic %s" % base64string.decode('utf-8'))
result = urllib.request.urlopen(request)
resulttext = result.read()
After all, there is one more difference with urllib: the resulttext variable in my case had the type of <bytes> instead of <str>, so to do something with text inside it I had to decode it:
text = resulttext.decode(encoding='utf-8',errors='ignore')

What about urllib.request ? It seems it has everything you need.
import base64
import urllib.request
request = urllib.request.Request('http://mysite/admin/index.cgi?index=127')
base64string = bytes('%s:%s' % ('login', 'password'), 'ascii')
request.add_header("Authorization", "Basic %s" % base64string)
result = urllib.request.urlopen(request)
resulttext = result.read()

An alternative using OpenerDirector that installs the auth headers for all future urllib requests
login_pass = base64.b64encode(f'{login}:{password}'.encode()).decode()
opener = urllib.request.build_opener()
opener.addheaders = [('Authorization', f'Basic {login_pass}')]
urllib.request.install_opener(opener)
response = urllib.request.urlopen(API_URL)
print(response.read().decode())
A further example using HTTPBasicAuthHandler although a bit more work required if need to send credentials unconditionally:
password_mgr = urllib.request.HTTPPasswordMgrWithPriorAuth()
password_mgr.add_password(None, API_URL, login, password, is_authenticated=True)
auth_handler = request.HTTPBasicAuthHandler(password_mgr)
opener = request.build_opener(auth_handler)
request.install_opener(opener)
response = urllib.request.urlopen(API_URL)
print(response.read().decode())

Related

Bytes and base64 formatting (Python3)

My scripting skills are very basic. I'm trying to adapt a Python 2 snippet I found on here to Python 3.
I'm having difficulty formatting this line (I understand it needs a bytes-object but can't get it to work).
headers["Authorization"] = "Basic {0}".format(
base64.b64encode("{0}:{1}".format('auth', 'login')))
Full snippet:
import base64, http.client
headers = {}
body = '/api/2.1/xml-in'
headers["Authorization"] = "Basic {0}".format(
base64.b64encode("{0}:{1}".format('auth', 'login')))
headers["Content-type"] = "application/xml"
# the XML we ll send to Freshbooks
XML = """<?xml version="1.0" encoding="utf-8"?>
<request method="task.list">
<page>1</page>
<per_page>15</per_page>
</request>"""
# Enable the job
conn = http.client.HTTPSConnection('sample.freshbooks.com')
conn.request('POST', body, None, headers)
resp = conn.getresponse()
print(resp.status)
conn.send(XML)
print(resp.read())
conn.close()
I've tried the following but then get an error about formatting the str:
headers["Authorization"] = "Basic {0}".format(
base64.b64encode("%b:%b" % b'auth', b'login'))
Any help would be greatly appreciated. Thanks.
It's a bit of a mouthful but the following works:
headers["Authorization"] = "Basic {0}".format(base64.b64encode(("{0}:{1}".format('string', 'string').encode('utf-8'))).decode())

Sending URL inside another URL

So I'm building a telegram bot with python and I need to send to the user an URL. I'm using telegram send_text URL:
https://api.telegram.org/bot{bot_token}/sendMessage?chat_id={chat_id}&parse_mode=Markdown&text={message}
but the URL that I'm using:
https://www.amazon.es/RASPBERRY-Placa-Modelo-SDRAM-1822096/dp/B07TC2BK1X/ref=sr_1_3?__mk_es_ES=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=YJ6X8FN3V801&keywords=raspberry+pi+4&qid=1577853490&sprefix=raspberr%2Caps%2C195&sr=8-3
has a special character like & that prevents the message to be sent with the full URL. In the case of this URL I only receive this:
https://www.amazon.es/RASPBERRY-Placa-Modelo-SDRAM-1822096/dp/B07TC2BK1X/ref=sr13?mkesES=ÅMÅŽÕÑ
I tried using utf-8 to replace the characters like & but python transforms them back to "real character" so I had to throw the idea off.
In case you want to check out what I tried here is the code snippet:
url = url.replace('&', u"\x26")
So is there any way I could fix this?
Encode the URL with urlencode()
import requests
import urllib.parse
link = "https://www.amazon.es/RASPBERRY-Placa-Modelo-SDRAM-1822096/dp/B07TC2BK1X/ref=sr_1_3?__mk_es_ES=%C3%85M%C3%85%C5%BD%C3%95%C3%91&crid=YJ6X8FN3V801&keywords=raspberry+pi+4&qid=1577853490&sprefix=raspberr%2Caps%2C195&sr=8-3"
markdownMsg = "[Click me!](" + urllib.parse.quote(link) + ")"
url = "https://api.telegram.org/bot<TOKEN>/sendMessage?chat_id=<ID>&text=" + markdownMsg + "&parse_mode=MarkDown"
response = requests.request("GET", url, headers={}, data ={})
print(response.text.encode('utf8'))
This also works for &parse_mode=HTML
htmlMsg = "Click me!"

Python 3.6.3 - Send MozillaCookieJar File and read HTML source code

I'm very fresh about python (i'm learning just about 1 day long).
I need to send cookies (i got them from my Google Chrome browser to a *.text file) and be redirected after login to my account page, to after read a source HTML code do what i wanna do. With much searches allong internet, i already have this piece of code:
import os
import time
import urllib.request
import http.cookiejar
while 1:
cj = http.cookiejar.MozillaCookieJar('cookies.txt')
cj.load()
print(len(cj)) # output: 9
print(cj) # output: <MozillaCookieJar[<Cookie .../>, <Cookie .../>, ... , <Cookie .../>]>
for cookie in cj:
cookie.expires = time.time() + 14 * 24 * 3600
cookieProcessor = urllib.request.HTTPCookieProcessor(cj)
opener = urllib.request.build_opener(cookieProcessor)
request = urllib.request.Request(url='https://.../')
response = opener.open(request, timeout=100)
s = str(response.read(), 'utf-8')
print(s)
if 'class' in s:
os.startfile('test.mp3')
time.sleep(5)
With this code i believe, hope i'm not be mistaken, have sending the cookies correctly. My main question is: How can i wait and catch the source HTML code after server redirect my login to personal page? I can't call again my Request with the same URL.
Thank you in advance.

Unable to get the response in POST method in Python

I am facing a unique problem.
Following is my code.
url = 'ABCD.com'
cookies={'cookies':'xyz'}
r = requests.post(url,cookies=cookies)
print(r.status_code)
json_data = json.loads(r.text)
print("Printing = ",json_data)
When I use the url and cookie in the POSTMAN tool and use POST request I get JSON response . But when I use the above code with POST request method in python I get
404
Printing = {'title': 'init', 'description': "Error: couldn't find a device with id: xxxxxxxxx in ABCD: d1"}
But when I use the following code i .e with GET request method
url = 'ABCD.com'
cookies={'cookies':'xyz'}
r = requests.post(url,cookies=cookies)
print(r.status_code)
json_data = json.loads(r.text)
print("Printing = ",json_data)
I get
200
Printing = {'apiVersion': '0.4.0'}
I am not sure why POST method works with JSON repsone in POSTMAN tool and when I try using python it is not work. I use latest python 3.6.4
I finally found what was wrong following is correct way
url = 'ABCD.com'
cookies={'cookies':'xyz'}
r = requests.post(url,headers={'Cookie'=cookies)
print(r.status_code)
json_data = json.loads(r.text)
print("Printing = ",json_data)
web page was expecting headers as cookie and i got the response correctly

Allow user to download ZIP from Django view

My main task is to have the user press a Download button and download file "A.zip" from the query directory.
The reason I have a elif request.POST..... is because I have another condition checking if the "Execute" button was pressed. This execute button runs a script. Both POST actions work, and the dir_file is C:\Data\Folder.
I followed and read many tutorials and responses as to how to download a file from Django, and I cannot figure out why my simple code does not download a file.
What am I missing? The code does not return any errors. Does anybody have any documentation that can explain what I am doing wrong?
I am expecting an automatic download of the file, but does not occur.
elif request.POST['action'] == 'Download':
query = request.POST['q']
dir_file = query + "A.zip"
zip_file = open(dir_file, 'rb')
response = HttpResponse(zip_file, content_type='application/zip')
response['Content-Disposition'] = 'attachment; filename=%s' % 'foo_zip'
zip_file.close()
I found out my answer.
After reading through many documentation about this, I left out the most important aspect of this feature which is the url.
Basically, the function download_zip is called by the POST and runs script where the zip is downloaded.
Here is what I ended up doing:
elif request.POST['action'] == 'Download':
return(HttpResponseRedirect('/App/download'))
Created a view:
def download_zip(request):
zip_path = root + "A.zip"
zip_file = open(zip_path, 'rb')
response = HttpResponse(zip_file, content_type='application/zip')
response['Content-Disposition'] = 'attachment; filename=%s' % 'A.zip'
response['Content-Length'] = os.path.getsize(zip_path)
zip_file.close()
return response
Finally in urls.py:
url(r'^download/$', views.download_zip, name='download_zip'),

Resources