How can I add credential(username and password) and then fetch the URL of that particular website in python?
I need help to implement this in my project
Example: suppose the first page is the login page and when we provide the credential and validate that and then we are re-directed into the homepage and then pick the url of that homepage
You can use package os
import os
url = os.environ['HTTP_HOST']
uri = os.environ['REQUEST_URI']
return url+uri
Related
There is one public html webpage where some content is available. With urllib I am able to fetch contents. But there another version where if I am logged in as user, additional content is available. in order to fetch logged-in page I use solution provided on portal and also urllib basic auth
import urllib.request
passman = urllib.request.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, username, password)
authhandler = urllib.request.HTTPBasicAuthHandler(passman)
opener = urllib.request.build_opener(authhandler)
urllib.request.install_opener(opener)
res = urllib.request.urlopen(url)
res_body = res.read()
res_body.decode('utf-8')
When I check contents I do not see what is additionally available for logged-in user . I would like to know how to correct this . I am using python 3.
Thanks
I have authorized.html page which needs a admin login to view so when I go to http://127.0.0.1:8000/authorized/ it takes me to http://127.0.0.1:8000/admin/login/?next=/admin/%3Fnext%3D/authorized/, which is expected. I used the below code in view.py file to built this functionality:
class authorizedView(LoginRequiredMixin,TemplateView):
template_name = 'home/authorized.html'
login_url = '/admin/'
But after the successful admin login it didn't take me back to authorized.html instead it directs to http://127.0.0.1:8000/admin/?next=/authorized/ which is just admin page and not the page that I want authorized.html. authorized.html is in home/templates/home/ where home is the django app that i created. How to do this? Please provide the detailed steps, i'am new to django!
In your settings.py, make sure you specify where to navigate to after successful login, for example:
LOGIN_REDIRECT_URL = '/authorized/'
See https://docs.djangoproject.com/en/3.2/ref/settings/#login-redirect-url for more details.
Also, you could decorate your view with #login_required and let Django handle the login portion of things and redirect you back there too. See https://docs.djangoproject.com/en/3.2/topics/auth/default/#the-login-required-decorator for details of this.
I am trying to make a web scraper. I would like to get the destination URL from a query URL. But it redirects many times.
This is my URL:
https://data.jw-api.org/mediator/finder?lang=INS&item=pub-jwb_201812_16_VIDEO
Destination url should be:
https://www.jw.org/ins/library/videos/#ins/mediaitems/VODOrgLegal/pub-jwb_201812_16_VIDEO
But I am getting https://www.jw.org/ins/library/videos/?item=pub-jwb_201812_16_VIDEO&appLanguage=INS this as the redirected URL.
I tried this code:
import requests
url = 'https://data.jw-api.org/mediator/finder?lang=INS&item=pub-jwb_201812_16_VIDEO'
s = requests.get(url)
print(s.url)
The redirect is made using JavaScript
It is not a server redirect so requests is not following it.
You can get the URL using Selenium
from selenium import webdriver
import time
browser = webdriver.Chrome()
url = 'https://data.jw-api.org/mediator/finder?lang=INS&item=pub-jwb_201812_16_VIDEO'
browser.get(url)
time.sleep(5)
print (browser.current_url)
browser.quit()
Outputs
https://www.jw.org/ins/library/videos/#ins/mediaitems/VODOrgLegal/pub-jwb_201812_16_VIDEO
If you are building a scraper I would suggest you check out scrapy-splash https://github.com/scrapy-plugins/scrapy-splash or requests-html https://github.com/psf/requests-html
You can do this pretty easily using requests:
import requests
destination = requests.get("http://doi.org/10.1080/07435800.2020.1713802")
#this link redirects the user to another link with a research paper of a given DOI code
print(destination.url)
#this returns "https://www.tandfonline.com/doi/full/10.1080/07435800.2020.1713802", the redirect of the initial doi.org link
I am new in Python3 and I am trying to download a doc after login to a website.
I have 2 url which can let me instantly login to the page and download the doc. after login which are:
https://www.xxxcompany.com/login.action?loginname=name&password=psw
https://www.xxxcompany.com/doc_download_all.action?ID=37887&edition=PD&Year=2018&Month=10&Day=5&&CLI=&transferNumber=&inOut=C&deviceType=A&minDuration=0&maxDuration=0&sortType=0&sortAsc=1&showAdv=0&viewtype=0&subPage=M&RMID=-1&updateRMID=&updateRecordID=&customField1=
Here is my code. Its definitely not work and it doesn't print me the status code. Did I misunderstand some concept? Please hep me to solve the problem. Thank you so much!
from lxml import html
import webbrowser
import requests
def login():
with requests.session() as s:
# fetch the login page
s.get(url1)
print(s.status_code) #check whether its successfully login
s.get(url2) #download the doc
You need to write data to file.
url = "http://www.xxxx.com/xxx/xxxx/sample.doc"
import requests
with requests.Session() as se:
req = se.get(url)
with open(url.split("/")[-1],"wb") as doc:
doc.write(req.content)
Right now, in gmail appscript we don't have any option to add a password type field.
Gmail Card Service for add-on has a very good ability to show any thing in it. We can integrate with any app which has basic REST api. We need authentication for that which commonly need password type field.
Any work around to show password type field?
As of now, there is no support for password field in Gmail add-on.
But we can build a hack for it. I hope password is needed only in registration forms. So, we can build a registration form using HTML and that can be served through authorization action.
CardService.newAuthorizationAction().setAuthorizationUrl(loginUrl)
Here, host registration HTML in a web server and pass this URL as "loginUrl" in the above snippet. We have to supply AuthorizationAction for the signup/register button. So, when the user clicks on this button, a new popup page is launched, the user will give the username, password, etc... onsubmit, we can encode all the form data and pass it to the parent Gmail add-on by redirecting it to a script redirection URL which you can generate an add-on. Once the redirection to the script URL comes, there will be a callback in our add-on code from there you can get the form fields which were encoded from registration HTML page.
function generateNewStateToken(callbackName, payload) {
return ScriptApp.newStateToken()
.withMethod(callbackName)
.withArgument("payload", JSON.stringify(payload))
.withTimeout(3600)
.createToken();
}
function getRedirectURI() {
return "https://script.google.com/macros/d/" + ScriptApp.getScriptId() + "/usercallback";
}
var state = generateNewStateToken("registerCallback", {"signup": true});
var reg_url = <reg_url> + "?redirect_uri=" + getRedirectURI() + "&state=" + state;
function registerCallback(cbResp) {
// to access payload which passed in state token: cbResp.parameter.payload;
// in the html serialize all the form fields or data which you want to pass to plugin as query params like: <redirect_uri>?form_data=<encoded_data>&state=<state>
//Note: here the registration HTML page should parse the URL to get the state & redirect_uri from URL.
// to access form_data: cbResp.parameter.form_data
}
I hope this will help you. This is how we are doing the signup/signin flow now.
Looks like you are authorizing a non google service . Please refer to Authorizing custom google services .