In short, I'm interested in downloading search results from this website, but I'm running into a pop-up that asks for input. How do I submit that form using Python 3 (no Machanize) so that I can get to the search results?
This page: http://www.oreillyauto.com/site/c/search/Brake+Pads+&+Shoes/C0068/C0009.oap?model=G6&vi=1432754&year=2006&make=Pontiac
...has this form:
<div id="forcedVehicleQuestions" class="forcedUserInput" style="display: block; position: absolute; left: 50%; top: 40px; z-index: 6000; margin-left: -199px; margin-top: 0px;">
<div class="forcedContents clearfix">
<a class="btn-remove" onclick="closeForced('Search','question');">
<svg><use xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="#shape-remove"></use></svg>
</a>
<form name="forcedQuestionsForm" id="forcedQuestionsForm">
<h2 class="sans">
More Product Info Required
</h2>
<p id="questionText" class="questionText">
Brake Pads - Position
</p>
<div id="forceQuestionsRadio">
<div class="form-row">
<label class="questionRadio checkbox-radio" id="questionRadio" for="Front">
<input type="radio" id="Front" name="answer" value="10219">
Front
</label>
</div>
<div class="form-row">
<label class="questionRadio checkbox-radio" id="questionRadio" for="Rear">
<input type="radio" id="Rear" name="answer" value="10290">
Rear
</label>
</div>
<div class="form-row">
<label class="questionRadio checkbox-radio" id="questionRadio" for="Show all">
<input type="radio" id="Show all" checked="" name="answer" value="-1">
Show all
</label>
</div>
</div>
<input id="questionSubmit" type="button" class="btn btn-green btn-shadow" value="Continue" onclick="setQuestionAnswer('Brake Pads - Position',document.forms['forcedQuestionsForm'].elements['answer'],'Show all');">
<div id="forcedVehicleQuestionsLoading" class="loading load-sm">
<div class="spinner"></div>
</div>
</form>
</div>
</div>
...but there's no method= so POST and GET don't seem to work. I'm pretty sure I understand how to set the radio buttons, so all I'm really interested in is how to get the onclick= behavior to happen.
So far, this is what I'm using:
#import HTTP libraries
import requests
#import HTML parsing libraries
import bs4
url = 'http://www.oreillyauto.com/site/c/search/Brake+Pads+&+Shoes/C0068/C0009.oap?model=G6&vi=1432754&year=2006&make=Pontiac'
answerURL = 'http://www.oreillyauto.com/site/ConditionSelectServlet?answer=-1'
print("Making request")
session = requests.Session()
session.headers.update({'referer': url})
r = session.get(answerURL)
print(r.status_code)
oreillyList = bs4.BeautifulSoup(r.text, "lxml")
print("Writing response...")
logfile = 'C:/Users/mhurley/Portable_Python/notebooks/' + output + '.log'
with open(logfile, 'w') as file:
file.write(oreillyList.prettify())
print("...done writing "+logfile)
This code retrieves the HTML for the page, but without any search results.
Related
I'm new in Python. I'm trying to log in to the site, but it doesn’t work out (remain unauthorized). What am I doing wrong?
import requests
with requests.Session() as c:
url = 'https://kartoteka.ru'
LOGIN = 'somelogin'
PWD = 'secretpassword'
PROMO = ""
c.get(url)
loginData = dict(login=LOGIN, pwd=PWD, promo=PROMO)
c.post(url, data=loginData, headers = {"Referer": "https://kartoteka.ru"})
page=c.get(url)
print(page.content)
The first this is it looks like you've missed this input in the login form:
<input type="checkbox" id="scratchcard_checkbox" name="scratchcard" value="scratchcard" style="top: 2px; position: relative;margin-right: 4px;">
So you should add a scratchcard option to your login data.
Edit: Here's the whole form:
<form action="" method="post" class="form-horizontal">
<div class="auth-ep-select" id="auth-cert-list-area"></div>
<div class="form-group no-margin-bottom">
<label class="control-label col-sm-12 col-xs-12 col-md-12 auth-login-label">Промо-код</label>
<div class="col-sm-12 col-xs-12 col-md-12 auth-text-field">
<span class="fa fa-lock user-auth-icon"></span>
<input id="auth_promo" class="form-control in-auth-control" value="" type="text" name="promo" placeholder="Промо-код"/>
</div>
</div>
<div class="form-group">
<div class="col-xs-12 auth-btn-block text-center" style="margin-top:15px">
<button type="button" data-loading-text="Войти" class="btn btn-md btn-info auth-btn js-dlg-auth-btn">Войти</button>
</div>
</div>
<div class="form-group text-center">
<a class="auth-registration-link" onsubmit="yaCounter23391682.reachGoal('registraciya'); return true;" href="/reguser/" style="color:#ffffff;">Регистрация </a>
<span>/</span>
<a class="success auth-buy-ep" href="/uc/" role="button" onsubmit="yaCounter23391682.reachGoal('kypit_ep'); return true;" title="Купить ЭП"> Купить ЭП</a>
</div>
</form>
Why do you bother GETting the url at the start? I don't think that's necessary.
How to you know you remain unauthorised? You're not saving the Request object from when you POST with the login data, so how do you know it returns an error code.
BTW, it's much quicker to declare a literal dictionary that create one like that, and you don't need it within the session.
import requests
ROOT_URL = 'https://kartoteka.ru'
USERNAME, PASSWORD = 'login', 'pass'
PROMO = ''=
with requests.Session() as c:
login_data = {'login': USERNAME, 'pwd': PASSWORD,
'promo': PROMO, 'scratchcard': false}
login_request = c.post(ROOT_URL,
data=login_data,
headers={"Referer": "https://kartoteka.ru"})
login_request.raise_for_status()
print(c.get(ROOT_URL).content)
I don't know if that will fix anything, but it's probably a start.
When I access the website and log-in manually, it works well.
However, when I use selenium webdriver and try to log in, this login slide bar shows, so I can't log in.
This is my code as followed.
id = driver.find_element_by_css_selector('#TPL_username_1').send_keys("prayjung50")
pw = driver.find_element_by_xpath('//*[#id="TPL_password_1"]').send_keys("xxxxxxxxxxxx")
time.sleep(10)
login = driver.find_element_by_xpath('//*[#id="J_SubmitStatic"]').click()
This is the image of the website as followed.
This is the website code of slide bar as followed.
<!-- use new slide checkcode -->
<div id="nocaptcha" class="nc-container tb-login nc-tm-min-fix" data-nc-idx="1" style="display: block;">
<div id="nc_1_wrapper" class="nc_wrapper">
<div id="nc_1_n1t" class="nc_scale">
<div id="nc_1__bg" class="nc_bg"></div>
<span id="nc_1_n1z" class="nc_iconfont btn_slide"></span>
<div id="nc_1__scale_text" class="scale_text slidetounlock"><span class="nc-lang-cnt" data-nc-lang="_startTEXT" data-spm-anchor-id="a2107.1.0.i0.3e3e11d9wfV1qj">请按住滑块,拖动到最右边</span></div>
<div id="nc_1_clickCaptcha" class="clickCaptcha" style="top: -202px; height: 380px; min-height: 0px;">
<div class="clickCaptcha_text">
<b id="nc_1__captcha_text" class="nc_captch_text"></b>
<i id="nc_1__btn_2" class="nc_iconfont nc_btn_2 btn_refresh"></i>
</div>
<div class="clickCaptcha_img"></div>
<div class="clickCaptcha_btn"></div>
</div>
<div id="nc_1_imgCaptcha" class="imgCaptcha" style="top: -202px; height: 339px; min-height: 0px;">
<div class="imgCaptcha_text"><input id="nc_1_captcha_input" maxlength="6" type="text" style="ime-mode:disabled"></div>
<div class="imgCaptcha_img" id="nc_1__imgCaptcha_img"></div>
<i id="nc_1__btn_1" class="nc_iconfont nc_btn_1 btn_refresh" onclick="document.getElementById('nc_1__imgCaptcha_img').children[0].click()"></i>
<div class="imgCaptcha_btn">
<div id="nc_1__captcha_img_text" class="nc_captcha_img_text"></div>
<div id="nc_1_scale_submit" class="nc_scale_submit"></div>
</div>
</div>
<div id="nc_1_cc" class="nc-cc"></div>
<i id="nc_1__voicebtn" tabindex="0" role="button" class="nc_voicebtn nc_iconfont" style="display:none"></i>
<b id="nc_1__helpbtn" class="nc_helpbtn"><span class="nc-lang-cnt" data-nc-lang="_learning">了解新功能</span></b>
</div>
<div id="nc_1__voice" class="nc_voice"></div>
</div>
</div>"a2107.1.0.i0.3e3e11d9wfV1qj">请按住滑块,拖动到最右边</span>
My html looks like this:
<div class="row">
<div class="col-md-7">
<ul class="breadcrumb">
<li id="get_data">Get data</li>
<li id="sampling_task">Sampling</li>
<li id="confirm_task">Confirmation</li>
</ul>
<div class="container-fluid">
<form action="#" method="post" enctype="multipart/form-data" role="form" class="form-horizontal">
{% csrf_token %}
<div class="form-group">
<label for="id_fileA" class="col-sm-3 control-label" style="text-align: left">
Select dataset A<span class='required_label'>*</span>
</label>
<div class="col-sm-9">
{{ form.fileA|attr:"class:form-control" }}
</div>
</div>
<div class="form-group"></div>
<div class="form-group">
<label for="id_fileB" class="col-sm-3 control-label" style="text-align: left">
Select dataset B<span class='required_label'>*</span>
</label>
<div class="col-sm-9">
{{ form.fileB|attr:"class:form-control" }}
</div>
</div>
<div class="form-group"></div>
<div class="form-group" id="sample_btn" style="display: none">
<label class="col-sm-3 control-label"></label>
<div class="col-sm-9">
<input type="button" name="theButton" id="sample-step" value="Start Sampling" class="btn btn-success btn-large disabled" style="border-radius: 5px;">
</div>
</div>
<div class="form-group">
<div class="col-sm-9 col-sm-offset-3" id="sample_msg" style="display: none;">
<p id="sample_text" style="font-size: medium">
Some ABCD message
</p>
</div>
</div>
<div class="form-group">
<div class="col-sm-9 col-sm-offset-3" id="get_sample_confirm_msg" style="display: none;">
<p>
<input type="button" style="height: 40px; width: 140px; border-radius: 5px" name="YesButton" id="accept-step" value="Accept & Continue" class="btn btn-success disabled">
<input type="button" style="height: 40px; width: 140px; border-radius: 5px" name="NoButton" id="cancel-step" value="Cancel Sampling" class="btn btn-danger disabled">
</p>
</div>
</div>
</form>
</div>
</div>
<div class="col-md-5">
</div>
</div>
The backend is a javascript and on button click, it do processing and if something fails the javascript code update the message.
var error = "<div class='alert alert-danger'><p>We encountered an error while sampling: <br /><strong>Sampling failed!!</strong></p>"; error += "<p>Please <a href='mailto:abcd#gmail.com'>contact us</a> if this error persists.</p>";
error += "</div>";
$('#sample_msg').html(error);
Now, in my selenium code if I do this:
sample_msg = self.driver.find_element_by_id('sample_msg')
I get an empty list result. What I want is to do is read the error in the "sample_msg" class if any and I have tried few things but its not working out. Help is appreciated. Thanks.
If you want to get access to hidden text you might need to use below code:
sample_msg = self.driver.find_element_by_id('sample_msg').get_attribute('textContent').strip()
Note that text property allows to get text from visible elements only
As you are trying to extract the error message Some ABCD message it is contained within a <p> tag which have parent <div> tag with style attribute set as display: none;. So to extract the text you can use the following code block :
element = driver.find_element_by_xpath("//div[#class='form-group']/div[#id='sample_msg']")
driver.execute_script("arguments[0].removeAttribute('style')", element)
print(driver.find_element_by_xpath("//div[#class='form-group']/div[#id='sample_msg']/p").get_attribute("innerHTML"))
So, you have to wait for sometime after clicking the button and once some message appears then you need to extract that message.
Try below:-
WebDriverWait wait = new WebDriverWait(webDriver, timeoutInSeconds);
wait.until(ExpectedConditions.visibilityOfElementLocated(By.id));
or
wait.until(ExpectedConditions.elementToBeClickable(By.id));
to be precise.
See also:
org.openqa.selenium.support.ui.ExpectedConditions for similar shortcuts for various wait scenarios.
org.openqa.selenium.support.ui.WebDriverWait for its various constructors.
You can check the python syntax also on the above links.
I am writing scripts for automating login action with credentials . I followed the standard process of finding the element through name , class , xpath . The input fields are not taking the credentials . I tried giving time interval of 10 seconds in between each process , still the issue continues.
Below I have attested the code .
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.select import Select
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Firefox()
driver.get() #WEBSITE HERE
time.sleep(10)
elem = driver.find_element_by_xpath('//*[#id="ssousernameUI"]')
elem.clear()
elem.send_keys() # USERNAME HERE
elem.send_keys(Keys.RETURN)
time.sleep(10)
passwd = driver.find_element_by_xpath('//*[#id="password"]')
passwd.clear()
passwd.send_keys() #PASSWORD HERE
passwd.send_keys(Keys.RETURN)
time.sleep(10)
login = driver.find_element_by_link_text("Login")
login.click()
ERROR MESSAGE :
Traceback (most recent call last):
File "/home/think201/PycharmProjects/pat/pat-cbt/login.py", line 13,
in <module>
elem = driver.find_element_by_xpath('//*[#id="ssousernameUI"]')
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 368, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 858, in find_element
'value': value})['value']
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 311, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate element: //*[#id="ssousernameUI"]
HTML code
<form name="loginForm" action="https://login.asdfghjk.in/oam/server/auth_cred_submit" target="_top" id="loginForm" method="POST">
<!-- Taksh End -->
<input name="ipRandomValue" value="709381701517861" id="ipRandomValue" type="hidden"><input name="deviceTypeHidden" value="Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:57.0) Gecko/20100101 Firefox/57.0" id="deviceTypeHidden" type="hidden">
<input name="ssousername" value="" id="ssousername" type="hidden">
<input name="hashnumber" value="" id="hashnumber" type="hidden">
<!-- Start mainWrapper -->
<div class="overlay"></div>
<div style="top: 20%;left: 50%;position: fixed;z-index: 5;color: white;display:none;" class="wait">
<h1>Please wait...</h1>
</div>
<div class="overlay1"></div>
<div class="mainWrapper">
<!-- Start Header -->
<!-- End Header -->
<!-- Start Middle Container -->
<article class="midContainer">
<div class="container">
<div class="breadcrums">
<p>You are here:</p>
<ul>
<li>Home</li>
<li class="last"> Login</li>
</ul>
</div>
<h1> Login</h1>
<section class="RHSsection standardWide">
<div class="pageWrap">
<div class="loginWrp">
<div class="box vfAcc">
<!-- displaying error message -->
<div class="cl c414042 bold fnt16"><p style="color:red;margin-left:10px;"><br><strong class="errMsgClass"></strong></p></div>
<div class="cl c414042 bold fnt16" id="SrvError"></div>
<div style="display:none;" class="cl c414042 bold fnt16 lockClass"><p style="color:red;margin-left:10px;"><br><strong>Your account is locked since it is
not used for a long time. Please click here to activate it</strong></p></div>
<!-- displaying error message end -->
<h3 class="accor">Login to manage your Account <span class="drop"></span></h3>
<div class="setWrp">
<div class="row">
<label>Enter your username or mobile number</label>
<div class="inputArea">
<div class="inputFieldBox">
<input name="ssousernameUI" autocomplete="off" class="login_input width255" value="" onfocus="if
(this.value==this.defaultValue)this.value='';" id="ssousernameUI" onblur="if(this.value=='')this.value=this.defaultValue;" type="text">
<div class="error_msg"></div>
</div>
</div>
</div>
<div class="row">
<label>Enter your password</label>
<div class="inputArea">
<div class="inputFieldBox">
<input name="password" autocomplete="off" id="password" class="login_input width255" onkeydown="if ((event.keyCode ==
13) && (document.getElementById('ssousernameUI').value!='') && (document.getElementById('passwordsrv').value!='')){LoginValidation();}" onblur="if(this.value=='')this.value=this.defaultValue;" value="" onfocus="if
(this.value==this.defaultValue)this.value='';" type="password">
<div class="error_msg"></div>
</div>
</div>
</div>
<div class="row">
<div class="checkboxBtn"> <span class="uncheked"></span>
<input name="checkbox" id="RememberMe" type="checkbox">
<label for="RememberMe"> Remember Me </label>
</div>
<div class="forgot"> Forgot your password?</div>
</div>
Login<!-- Add Here : Taksh -->
<!-- <div class="col orline"><span class="or">Or</span></div>
<ul class="smoShare">
<li class="facebook">Facebook</li>
<li class="gplus">Google Plus</li>
</ul> -->
</div>
</div>
<div class="box myvf">
<h3 class="accor">Why should I register ? <span class="drop"></span></h3>
<div class="setWrp">
<p>Here are 5 great reasons! </p>
<ul class="redbult">
<li>Simplified, dashboard view of voice & Internet usage</li>
<li> Easy update of profile & contact info</li>
<li> Quick change of voice & plans</li>
<li> Hassle-free access to old bills & statements of accounts</li>
<li> Instant activation/deactivation of services</li>
</ul>
<p>It takes just a few moments!</p>
Register Now
</div>
</div>
<div class="box vfApp">
<h3 class=""> App</h3>
<div class="">
<div class="vfappSlide">
<div class="owl-vfapp owl-carousel owl-theme owl-loaded">
<!-- <div class="apps"> <img src="/images/appimages/windows_logo.jpg" alt="" />
<p>Manage your account on the go !</p>
<div class="getNow"><a onclick="ga('send', 'event', 'My Account', 'Click', 'Windows - Get Now'); utag.link({ Custom_Links : 'MyAccount: Login: windows App - Get Now' });" href="http://qwertyu" class="purpleBtn">Get Now</a> </div>
</div>
<div class="apps"> <img src="/images/appimages/blackberry_logo.png" alt="" />
<p>Manage your Vodafone account on the go !</p>
<div class="getNow"><a onclick="ga('send', 'event', 'My Account', 'Click', 'Blackberry - Get Now'); utag.link({ Custom_Links : 'MyAccount: Login: Blackberry App - Get Now' });" href="http://qwerty" class="purpleBtn">Get Now</a> </div>
</div>
<div class="apps"> <img src="/images/appimages/nokia_ovi_logo.jpg" alt="" />
<p>Manage your account on the go !</p>
<div class="getNow"><a onclick="ga('send', 'event', 'My Account', 'Click', 'OVI - Get Now'); utag.link({ Custom_Links : 'MyAccount: Login: Ovi App - Get Now' });" href="http://qwertyu" class="purpleBtn">Get Now</a> </div>
</div>
<div class="apps"> <img src="/images/appimages/java_logo.jpg" alt="" />
<p>Manage your account on the go !</p>
<div class="getNow"><a onclick="ga('send', 'event', 'My Account', 'Click', 'Java - Get Now');utag.link({ Custom_Links : 'MyAccount: Login: Java App - Get Now' });" href="http://qwertyui" class="purpleBtn">Get Now</a> </div>
</div> -->
<div class="owl-stage-outer"><div class="owl-stage" style="transform: translate3d(-837px, 0px, 0px); transition: all 0.25s ease 0s; width: 1674px;"><div class="owl-item cloned" style="width: 279px; margin-right: 0px;"><div class="apps"> <img alt="" src="/images/appimages/android_logo1.jpg">
<p>Manage your Vodafone account on the go !</p>
<div class="getNow">Get Now </div>
</div></div><div class="owl-item cloned" style="width: 279px; margin-right: 0px;"><div class="apps"> <img alt="" src="/images/appimages/apple_logo1.jpg">
<p>Manage your Vodafone account on the go !</p>
<div class="getNow">Get Now </div>
</div></div><div class="owl-item" style="width: 279px; margin-right: 0px;"><div class="apps"> <img alt="" src="/images/appimages/android_logo1.jpg">
<p>Manage your Vodafone account on the go !</p>
<div class="getNow">Get Now </div>
</div></div><div class="owl-item active" style="width: 279px; margin-right: 0px;"><div class="apps"> <img alt="" src="/images/appimages/apple_logo1.jpg">
<p>Manage your account on the go !</p>
<div class="getNow">Get Now </div>
</div></div><div class="owl-item cloned" style="width: 279px; margin-right: 0px;"><div class="apps"> <img alt="" src="/images/appimages/android_logo1.jpg">
<p>Manage your account on the go !</p>
<div class="getNow">Get Now </div>
</div></div><div class="owl-item cloned" style="width: 279px; margin-right: 0px;"><div class="apps"> <img alt="" src="/images/appimages/apple_logo1.jpg">
<p>Manage your account on the go !</p>
<div class="getNow">Get Now </div>
</div></div></div></div><div class="owl-controls"><div class="owl-nav"><div class="owl-prev" style="display: none;">prev</div><div class="owl-next" style="display: none;">next</div></div><div style="" class="owl-dots"><div class="owl-dot"><span></span></div><div class="owl-dot active"><span></span></div></div></div></div>
</div>
</div>
</div>
</div>
</div>
</section>
</div>
</article>
<!-- End Middle Container -->
</div>
<!-- End mainWrapper -->
</form>
NoSuchElementException is because webdriver is not able to find the element in the webpage. Check if there are any frames in the webpage. Switch to it, if any.
You will need to click the element to get that element to focus and then enter username/password details.
If that also does not work try wrapping your find element statement with explicit wait.
Let me know if that helps.
EDIT1:
elem.click();
elem.send_keys();
time.sleep(50);
EDIT2:
Instead of:
elem = driver.find_element_by_xpath('//*[#id="ssousernameUI"]')
you can try:
driver.find_element_by_xpath('//*[#id="ssousernameUI"]').click()
For the first look, it seems that you xpath is not a correct one.
Instead of using:
elem = driver.find_element_by_xpath('//*[#id="ssousernameUI"]')
Try this:
elem = driver.find_element_by_xpath("//*[#id='ssousernameUI']")
As per the HTML you have shared to fill up the credentials you can use the following code block :
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("your_website") #WEBSITE HERE
username = driver.find_element_by_xpath("//input[#id='ssousernameUI']").click()
username.clear()
username.send_keys("user_name") # USERNAME HERE
password = driver.find_element_by_xpath("//input[#id='password']").click()
password.clear()
password.send_keys("Pa$$w0rd") #PASSWORD HERE
driver.find_element_by_xpath("//a[#class='purpleBtn' and contains(.,'Login')]").click()
Thank You . I resolved it .
The website had multiple login pages , hence they have used iframe to suffice the Login access in multiple pages.
The issue was , the login was inside a iframe . I referred to the documentation for switching through frame.
I used
driver.switch_to_frame("frameName")
Thank you #Arun Prakash and everyone.
I'm trying to make a simple Bootstrap searchbar (with bootstrap 3.3.7), but always the glyphicon-search is more little than the form...
With all Bootstrap navbar examples, the result is the same, as the following :
<div class="col-sm-3 col-md-3">
<form class="navbar-form" role="search">
<div class="input-group">
<input type="text" class="form-control" placeholder="Search" name="q">
<div class="input-group-btn">
<button class="btn btn-default" type="submit"><i class="glyphicon glyphicon-search"></i></button>
</div>
</div>
</form>
</div>
Do I really need to add some CSS to that ?
To change size and color of your glyphicon:
.glyphicon-search.white {
font-size: 1.2em; // slightly bigger
color: #fff; // white
}
In your html
<i class="glyphicon glyphicon-search white"></i>