Trying to store/re-use cookies between code-executions, similar to this question, but using only python3's urllib.
Thanks to this answer for the process for creating a cookiejar for automatic use across urllib.request.Request calls:
cookie_jar = http.cookiejar.CookieJar()
opener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(cookie_jar))
urllib.request.install_opener(opener)
Initial searches on cookie storage led to SO questions regarding requests (the module) + cookies, but unfortunately, http.cookiejar objects cannot be pickled (below), as opposed to their requests brethren, which can:
>>> pickle.dumps(cookie_jar)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: can't pickle _thread.RLock objects
The other queries' results mostly pointed back the python3 docs: https://docs.python.org/3/library/http.cookiejar.html#http.cookiejar.FileCookieJar
Playing around with the FileCookieJar class led to noticing in the description: A CookieJar which can load cookies from, and *perhaps* save cookies to, a file on disk. and the associated error:
>>> FCJ=cookiejar.FileCookieJar("unversioned.cookies")
>>> FCJ.save()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.7/http/cookiejar.py", line 1785, in save
raise NotImplementedError()
Both of which were clarified by this answer - which suggested using LWPCookieJar (among others) instead, but I couldn't find any existing examples on how to use this class in practice.
LWPCookieJar was the answer:
Create the CookieJar + try loading from any existing cookie file:
cookie_filename = "unversioned.cookies"
cookie_jar = cookiejar.LWPCookieJar(cookie_filename)
try:
cookie_jar.load()
except FileNotFoundError as fnfe:
# No existing/adjacent cookie file
pass
opener = request.build_opener(request.HTTPCookieProcessor(cookie_jar))
request.install_opener(opener)
complete any urllib.request.Request call that populates cookies
store the cookies to file (named in the constructor):
cookie_jar.save()
And to confirm, after having saved some cookies in a previous execution-run:
prepare the cookiejar, but do not call .load()
test-request the page -> logged-out version received
call cookiejar.load()
test-request the page -> logged-in version received
Related
this is my code
import requests
r = requests.get('https://www.reddit.com/r/Art.json').json()
print(r['data'])
this code sometimes work but sometimes get fail
Exception is
Traceback (most recent call last):
File "c:/Users/SAMET/Desktop/python/a.py", line 5, in <module>
print(r['data'])
KeyError: 'data'
You must use r.data. r['data'] could work in Javascript, but in Python you can't access objects as dictionaries.
I tried to login into the site through RoboBrowser but I'm facing below error.
import config
from robobrowser import RoboBrowser
br = RoboBrowser()
br.open('https://stackoverflow.com')
form = br.get_form(id="login-form")
form['Email address']=config.username
form['Password']=config.password
br.submit_form(form)
Error
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "C:\Users\test\AppData\Local\Programs\Python\Python37\lib\site-packages\robobrowser\forms\form.py", line 216, in __setitem__
self.fields[key].value = value
File "C:\Users\test\AppData\Local\Programs\Python\Python37\lib\site-packages\werkzeug\datastructures.py", line 784, in __getitem__
raise exceptions.BadRequestKeyError(key)
werkzeug.exceptions.BadRequestKeyError: 400 Bad Request: The browser (or proxy) sent a request that this server could not understand.
Through any other module, we are able to login into site and search data in the next screen. Example if I login into site and next page it will display as "Hello User".
I want to bring the output and see "Hello User" is displayed or not through the python script
I think you are trying to login to the sing up form. Move to the login page and then login.
import config
from robobrowser import RoboBrowser
br = RoboBrowser()
br.open('https://stackoverflow.com/users/login')
form = br.get_form(id="login-form")
form['email']=config.username
form['password']=config.password
br.submit_form(form)
This error, surprisingly, would happen if you are setting a form field which is not actually a part of the form. Tomasz's answer goes to the point of what was wrong, but a couple additional notes:
robobrowser is not supported and you should look into better-maintained libraries like MechanicalSoup or mechanize
StackExchange has an API
I have code that sets up an environment for running and logging scientific experiments. Some of the initial setup involves using the built in input() method to query the user for values. I keep getting a I/O operation on closed file error whenever I try to call input however.
Code flow: Control.py calls Analyzer.py which calls a specific method in Prompts.py (the code for which is below).
def prompt_instruments(message):
res = input(message) # query user with arg message
print("done")
if '.' in res:
print("User input not cool. Use comma-separated values.")
return None # to continue prompting
...
I have searched all over the internet and have been unable to find anything remotely related. Thank you so much!!
The code you posted seems ok, and the error is probably in one of your other files.
The input() function uses sys.stdout to display the prompt text, and sys.stdin to get the user's input text.
The error message you get is probably caused by one of these files being closed, e.g.:
>>> import sys
>>> input('test: ')
test: hello
'hello'
>>> sys.stdin.close()
>>> input('test: ')
test: Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file.
or:
>>> import sys
>>> input('test: ')
test: hi
'hi'
>>> sys.stdout.close()
>>> input('test: ')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ValueError: I/O operation on closed file.
I can't tell you exactly where to fix this issue, but look for things that might close one of these file, either directly or indirectly (e.g. context manager).
This is for an internal project. My end goal is to get details of my connections. who are in same city as I am.
I am new in using LinkedIn API . I have used code mentioned in answer here to generate the access token. Now I am using below line to get my LinkedIn profile.
application.get_profile(access_token['oauth_token'])
But I am getting below error.
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
application.get_profile(access_token['oauth_token'])
File "C:\Python34\lib\site-packages\python_linkedin-2.0-py3.4.egg\linkedin\linkedin.py", line 189, in get_profile
response = self.make_request('GET', url, params=params, headers=headers)
File "C:\Python34\lib\site-packages\python_linkedin-2.0-py3.4.egg\linkedin\linkedin.py", line 169, in make_request
params.update({'oauth2_access_token': self.authentication.token.access_token})
AttributeError: 'str' object has no attribute 'token'
Can someone please help me?
It is a bit late I guess. But for future reference, this error comes from the fact that you need to call explicitly the token element from the linkedin application.
For instance, to generate the application, you need to enter the following:
from linkedin_v2 import linkedin
application = linkedin.LinkedInApplication(token=access_token['oauth_token'])
From there, you can then call the different function from application (like "application.get_profile()").
BR
I don't think you need to pass the auth token again to get your profile. I am also using the python lib.
application.get_profile()
Should return the basic information for your profile. Then look into using selectors to specify what endpoints you want.
I try to get the custom json from my OpsWorks stacks with python and boto3. Getting the name is ok but if I want to get the CustomJson - KeyError. Don't have a clue why.
import boto3
import traceback
client = boto3.client('opsworks')
response = client.describe_stacks()
max_elements = len(response['Stacks'])
for i in range(max_elements):
stack_Name = response['Stacks'][i]['Name'] # works
try:
stack_CustomJson = response['Stacks'][i]['CustomJson'] # KeyError
except:
traceback.print_exc()
That's the console output:
$ python3 get_custom_json.py
Traceback (most recent call last):
File "get_custom_json.py", line 27, in get_opsworks_details
stack_CustomJson = response['Stacks'][i]['CustomJson']
KeyError: 'CustomJson'
Reading the docs from http://boto3.readthedocs.org/en/latest/reference/services/opsworks.html#OpsWorks.Client.describe_stacks I don't see a difference between 'Name' and 'CustomJson' except that CustomJson is a JSON object. Do I have to transform it?
Thx in advance
You are getting a KeyError occasionally because the CustomStack element in the response is optional. If a custom stack is specified for the stack, it will be returned. Otherwise, the CustomStack key will not be there at all. You should do something like:
if 'CustomStack' in stack:
# do your thing
Had a quick chat with a developer in my company. Got some basic introductions for getting better in coding and python and whatever (must loose some of my admin thinking).
Don't iterate about max_elements, better iterate above 'stack in stacks'.
for stack in response['Stacks']:
print(stack['CustomJson'])
Now it works - I'll get the custom json from the OpsWorks stacks. But still there is the KeyError.
Traceback (most recent call last):
File "get_custom_json.py", line 22, in <module>
get_opsworks_details()
File "get_custom_json.py", line 18, in get_opsworks_details
print(stack['CustomJson'])
KeyError: 'CustomJson'
I'll check if I can fetch him again to see why that's happening.
[EDIT] Blind spot - if a stack has no custom json then the KeyError will occur.