Hey I am trying to make a simple Insta bot using selenium.I reached up to my following list by automating but now I don't know how I can scroll down the list of my following/followers. I want to grab the account from my following and followers list and compare it and make a list of the accounts who havenot followed me back.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from time import sleep
from secrets import password,username
class Instabot:
def __init__(self,username,password):
self.driver = webdriver.Chrome(executable_path="C:\\Users\\user\\PycharmProjects\\chromedriver_win32\\chromedriver.exe")
self.driver.get("https://www.instagram.com")
sleep(5)
self.driver.find_element_by_xpath('//*[#id="react-root"]/section/main/article/div[2]/div[1]/div/form/div[2]/div/label/input').send_keys(username) #searching the username box and giving it username
self.driver.find_element_by_xpath('//*[#id="react-root"]/section/main/article/div[2]/div[1]/div/form/div[3]/div/label/input').send_keys(password)#searching the password box and giving it password
self.driver.find_element_by_xpath('//*[#id="react-root"]/section/main/article/div[2]/div[1]/div/form/div[4]/button').click() #clicking login button
wait=WebDriverWait(self.driver,10)
notNowButton = wait.until(
lambda d: d.find_element_by_xpath('//button[text()="Not Now"]'))#it will click on first notnow button
notNowButton.click()
next_not_now=wait.until(lambda notnow:notnow.find_element_by_xpath('//button[text()="Not Now"]'))
next_not_now.click() #it will click on second not now button
def get_unfollowers(self):
wait = WebDriverWait(self.driver, 10)
clickprofile= wait.until(lambda a: a.find_element_by_xpath('//*[#id="react-root"]/section/nav/div[2]/div/div/div[3]/div/div[5]/a'))
clickprofile.click()
following_list = wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(#href,'/following')]")))
following_list.click()#IT clicks the following and gives window of following listy
print("CLicked following list")
my_bot=Instabot(username,password)
my_bot.get_unfollowers()
I saw about execute_script() But I dont know what to put under those brackets.
Well I figure it out. This code works for me now!
fBody = self.driver.find_element_by_css_selector("div[class='isgrP']")
scrolling_times=(numoffollowers/4)
scroll=0
scroll_count = scrolling_times+5 # You can use your own logic to scroll down till the bottom
while scroll < scroll_count:
self.driver.execute_script(
'arguments[0].scrollTop = arguments[0].scrollTop + arguments[0].offsetHeight;',
fBody)
sleep(2)
scroll += 1
You can use something like that :
self.driver.execute_script("window.scrollTo(0, Y)")
with Y : how much you want to scroll in pixel.
It's up to you to see how far down you have to go down.
Related
I wrote a python script that goes to a site, and interacts with some dropdowns. It works perfectly fine if after I run the script, quickly make the browser instance full screen so that the elements are in view. If I don't do that, I get the error "Element could not be scrolled into view".
Here is my script:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://example.com")
driver.implicitly_wait(5)
yearbtn = driver.find_element("id", "dropdown_year")
yearbtn.click()
year = driver.find_element("css selector", '#dropdown_ul_year li:nth-child(5)')
year.click()
makebtn = driver.find_element("id", "dropdown_make")
makebtn.click()
make = driver.find_element("css selector", '#dropdown_ul_make li:nth-child(2)')
make.click()
modelbtn = driver.find_element("id", "dropdown_model")
modelbtn.click()
model = driver.find_element("css selector", '#dropdown_ul_model li:nth-child(2)')
model.click()
trimbtn = driver.find_element("id", "dropdown_trim")
trimbtn.click()
trim = driver.find_element("css selector", '#dropdown_ul_trim li:nth-child(2)')
trim.click()
vehicle = driver.find_element("css selector", '#vehiclecontainer > div > p')
vdata = driver.find_element("css selector", '.top-sect .tow-row:nth-child(2)')
print("--------------")
print("Your Vehicle: " + vehicle.text)
print("Vehicle Data: " + vdata.text)
print("--------------")
print("")
driver.close()
Like I said, it works fine if I make the browser full-screen (or manually scroll) so that the elements in question are in view. It finds the element, so what's the issue here? I've tried both Firefox and Chrome.
Generally firefox opens in maximized mode. Incase for specific version of os it doesn't, the best practice is to open it in maximized mode as follows:
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("start-maximized")
driver = webdriver.Firefox(options=options)
However in somecases the desired element may not get rendered within the due coarse, so you need to induce WebDriverWait for the visibility_of_element_located() and you can use either of the following locator strategies:
vehicle = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "#vehiclecontainer > div > p")))
vdata = WebDriverWait(driver, 20).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".top-sect .tow-row:nth-child(2)")))
print("--------------")
print("Your Vehicle: " + vehicle.text)
print("Vehicle Data: " + vdata.text)
print("--------------")
print("")
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
You can find a relevant discussion in How to retrieve the text of a WebElement using Selenium - Python
I'm not sure what all is happening on the page in question. I use a generic method to take care of common actions like click(), find(), etc. that contain a wait, scroll, and whatever else is useful for that action. I've put an example below. It waits for the element to be clickable (always a best practice), scrolls to the element, and then clicks it. You may want to tweak it to your specific use but it will get you pointed in the right direction.
def click(self, locator):
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable(locator))
driver.execute_script("arguments[0].scrollIntoView();", element)
element.click()
I would suggest you try this and then convert all of your clicks to use this method and see if that helps, e.g.
click((By.ID, "dropdown_year"))
or
locator = (By.ID, "dropdown_year")
click(locator)
I am trying to create an Account on Walmart using selenium python. I successfully opened https://www.walmart.com/ and successfully go to create an account button under Sign In tab. Moreover, I also successfully entered the details of First name, Last name, Email Address and Password. However, once I clicked on Create account button, I got TimeoutException error despite using EC.visibility_of_element_located().click () method.
Can anyone kindly guide me what is wrong with my approach. Thanks in advance.
The source code of the website for Create Account button is as follows:
<button class="button m-margin-top text-inherit" type="submit" data-automation-id="signup-submit-btn" data-tl-id="signup-submit-btn" aria-label="Create Account, By clicking Create Account, the user is acknowledging that they have read and agreed to the Terms of Use and Privacy Policy">Create account</button>
My Python code is as follows:
import time
import requests
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.action_chains import ActionChains
url = "https://www.walmart.com/"
first_name = "chuza"
last_name = "123"
email_id = "chuza123#gmail.com"
password = "Eureka1#"
options = Options()
s=Service('C:/Users/Samiullah/.wdm/drivers/chromedriver/win32/96.0.4664.45/chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.execute_script("Object.defineProperty(navigator, 'webdriver', {get: () => undefined})")
driver.execute_cdp_cmd("Page.addScriptToEvaluateOnNewDocument", {
"source":
"const newProto = navigator.__proto__;"
"delete newProto.webdriver;"
"navigator.__proto__ = newProto;"
})
wait = WebDriverWait(driver, 20)
actions = ActionChains(driver)
driver.get(url)
sign_in_btn = wait.until(EC.visibility_of_element_located((By.XPATH, "//div[text()='Sign In']")))
actions.move_to_element(sign_in_btn).perform()
time.sleep(0.5)
wait.until(EC.visibility_of_element_located((By.XPATH, '//button[normalize-space()="Create an account"]'))).click()
f_name = driver.find_element(By.ID, 'first-name-su')
l_name = driver.find_element(By.ID, 'last-name-su')
email = driver.find_element(By.ID, 'email-su')
pswd = driver.find_element(By.ID, 'password-su')
f_name.send_keys(first_name)
driver.implicitly_wait(2)
l_name.send_keys(last_name)
driver.implicitly_wait(1.5)
email.send_keys(email_id)
driver.implicitly_wait(2)
pswd.send_keys(password)
driver.implicitly_wait(0.5)
###
wait.until(EC.visibility_of_element_located((By.XPATH, '//button[normalize-space()="Create account"]'))).click()
I see this css selector that represent the desired webelement:
button[data-automation-id='signup-submit-btn']
and xpath would be:
//button[#data-automation-id='signup-submit-btn']
there are 3 matching nodes for each CSS and XPath and Selenium will look for the first match, the CSS and XPath basically are first matching node.
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-automation-id='signup-submit-btn']"))).click()
or
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[#data-automation-id='signup-submit-btn']"))).click()
It makes more sense to use element_to_be_clickable when trying to click on a web element instead of visibility_of_element_located. Also, CSS are much better locator as compared to XPath.
//button[normalize-space()="Create account"] locator matches 3 elements on that page, you need to use more precise locator.
This locator is unique: //form[#id='sign-up-form']//button[#data-tl-id='signup-submit-btn']
So, this should work:
wait.until(EC.visibility_of_element_located((By.XPATH, "//form[#id='sign-up-form']//button[#data-tl-id='signup-submit-btn']"))).click()
This xpath based Locator Strategy...
//button[normalize-space()="Create account"]
...identifies three(3) elements within the DOM Tree and your desired element is the second in the list.
Solution
The desired element is a dynamic element so to click on the clickable element instead of visibility_of_element_located() you need to induce WebDriverWait for the element_to_be_clickable() and you can use the following Locator Strategy:
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//form[#id='sign-up-form']//button[normalize-space()='Create account']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
This issue is because of selenium, you can fix this by creating it manually, follow the steps:
Create an account on Walmart IO platform here by clicking on the man icon just before the search box.
Login to the account and accept "Terms of Use"
Click on "Create Your Application" to create a new application and fill in appropriate details.
You can follow this tutorial to generate two sets of public/private keys -one set will be used for production and the other set will be used for stage.
Upload both public keys using this: https://walmart.io/key-upload?app_name=<your app name>
Consumer ID will be generated for both sets for prod and stage which can be seen on the dashboard
Click on "Request Access" for OPD APIs at here and fill out the form
Im using Selenium with Python3 on a Service Now Website.
So the process is as follows: selenium loads up the ServiceNow URL and then I use sendKeys to automate typing in of username and password, then the page is loaded which has a table of incidents I need to extract. Unfortunately I have to login in every single time because of the group policy I have.
This works up until I have to find the dynamically rendered Javascript table with data and I can't for the life of me seem to find it. I even tried to put a sleep in there for 15 seconds to allow it to load.
I also double checked the XPaths and Id / Class names and they match up. When I print query.page_source I don't see anything rendered by JS.
I've used beautiful soup too but that also doesn't work.
Any ideas?
from time import sleep
from collections import deque
from selenium import webdriver
from selenium.webdriver.support.ui import Select # for <SELECT> HTML form
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
query = webdriver.Firefox()
get_query = query.get("SERVICENOW URL")
query.implicitly_wait(10)
login_username = query.find_element_by_id('username')
login_password = query.find_element_by_id('password')
login_button = query.find_element_by_id('signOnButton')
username = "myUsername"
password = "myPassword"
login_username.send_keys(username)
login_password.send_keys(password)
login_button.click()
sleep(10)
incidentTableData = []
print(query.page_source)
// *** THESE ALL FAIL AND RETURN NONE ***
print(query.find_elements())
tableById = query.find_element_by_id('service-now-table-id')
tableByXPath = query.find_element_by_xpath('service-now-xpath')
tableByClass = query.find_element_by_id('service-now-table-class')
Since it's a dynamically rendered Javascript table, I would suggest you to implement explicit wait in your code.
so instead of this :
tableById = query.find_element_by_id('service-now-table-id')
tableByXPath = query.find_element_by_xpath('service-now-xpath')
tableByClass = query.find_element_by_id('service-now-table-class')
re-write these lines like this :
wait = WebDriverWait(query, 10)
service_now_with_id = wait.until(EC.element_to_be_clickable((By.ID, "service-now-table-id")))
service_now_with_xpath = wait.until(EC.element_to_be_clickable((By.XPATH, "service-now-xpath")))
service_now_with_class = wait.until(EC.element_to_be_clickable((By.ID, "service-now-table-class")))
You are gonna need to use the below imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as E
PS :- service_now_with_id, service_now_with_xpath, service_now_with_class, these are web elements returned by explicit waits. you may wanna have to interact with them as per your requirement meaning, clicking on it or sending keys or whatever.
The code is working itself, but when thrown to loop not working anymore.
driver.get("https://www.esky.pl/okazje/16578/WMI-EDI-FR")
i = 1
departure_date_clickable = False
while departure_date_clickable == False:
try:
time.sleep(5)
xpath ="/html/body/div[4]/div[3]/div/div[1]/div[1]/div[2]/div/div[4]/div/div[{}]".format(i)
find_ele = driver.find_element_by_xpath(xpath)
find_ele.click()
print("Departure:Found clickable date on " + str(i))
departure_date_clickable = True
except WebDriverException:
print("Departure date not clickable, checking next day")
i += 1
continue
I expect to click first element able to be clicked from the calendar. But for some reason it's problem for selenium when in loop.
Code that's working:
xpath = "/html/body/div[4]/div[3]/div/div[1]/div[1]/div[2]/div/div[4]/div/div[{}]".format("4")
find_ele = driver.find_element_by_xpath(xpath)
time.sleep(2)
find_ele.click()
I recommend identifying clickable divs by a class attribute unique to those divs. It looks like all the "clickable" links have a class called "offer" so you could add an if else condition to check each element for that class. I also added a termination condition to your loop since there are 35 divs in the block.
driver.get("https://www.esky.pl/okazje/16578/WMI-EDI-FR")
i = 1
departure_date_clickable = False
while departure_date_clickable == False and i <= 35:
try:
time.sleep(1)
xpath ="/html/body/div[4]/div[3]/div/div[1]/div[1]/div[2]/div/div[4]/div/div[{}]".format(i)
find_ele = driver.find_element_by_xpath(xpath)
if "offer" in find_ele.get_attribute("class").split(" "):
find_ele.click()
print("Departure:Found clickable date on " + str(i))
departure_date_clickable = True
else:
raise error()
except:
print("Departure date not clickable, checking next day")
i += 1
continue
Please find below solution, There are few issues observed while iterating through a list element. Kindly note if you want to select any specific departure date then you need to put that condition in the for loop. At the moment as per your above code we are just click all departure dates
from selenium.common.exceptions import WebDriverException
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support.ui import WebDriverWait as Wait
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome('C:\New folder\chromedriver.exe')
driver.maximize_window()
driver.get("https://www.esky.pl/okazje/16578/WMI-EDI-FR")
i = 1
departure_date_clickable = False
while departure_date_clickable == False:
try:
xpath = WebDriverWait(driver, 15).until(EC.presence_of_all_elements_located((By.XPATH, "//div[#id='departure-calendar']//div[#class='month-days']/div/div")))
for value in xpath:
value.click()
departure_date_clickable = True
except WebDriverException:
I had the same problem in R while using selenium, I found that running the command to click twice sequentially within the loop works.
When the click was only present once it seemed to highlight the element, but not click on it.
While the language is different, the logic might hold.
I have provided the code below in a kind of pseudo form so the logic can be seen.
The loop would iterate over different webpages within a list (represented by 'list') with the same element (represented by '[xpath]') to be clicked in each webpage in turn.
library(RSelenium) # load package
# set up the driver
fprof <-getFirefoxProfile('[pathtoprofile]', useBase = TRUE)
rd <- rsDriver(browser = "firefox", port = 4567L,
extraCapabilities = fprof
)
ffd <- rd$client
for (i in 1:length(list)){
ffd$open() # open the browser
ffd$navigate(list[i]) # navigate to page
Sys.sleep(10) # wait
ffd$findElement("xpath", '[xpath]')$clickElement() # click element first time
ffd$findElement("xpath", '[xpath]')$clickElement() # click element second time
Sys.sleep(5) # wait
ffd$close() # close the browser
}
You can get all departure days with offer using:
driver.find_elements_by_css_selector("#departure-calendar .cell-day.number.offer")
For arrival calendar:
driver.find_elements_by_css_selector("#arrival-calendar .cell-day.number.offer")
Code below click on each day with offer in departure calendar and print the date:
driver.get("https://www.esky.pl/okazje/16578/WMI-EDI-FR")
departure_days = driver.find_elements_by_css_selector("#departure-calendar .cell-day.number.offer")
for departure_day in departure_days:
departure_day.click()
# print selected date
print(departure_day.get_attribute("data-qa-date"))
If you are just looking for the first available date, you can use the CSS selector below
div.cell-day.offer
Your code would look like
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://www.esky.pl/okazje/16578/WMI-EDI-FR")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.cell-day.offer"))).click()
I want to enter text into a textarea. The problem is the textarea is only created and shows up in the html after clicking twice on a visual element in the browser.
For this visual element Dubbelklik om je tekst te typen, I cannot find an element in the html that refers to it and perform a double_click(). There are only containers, that either are unclickable elements, or allow clicks, but don't perform an action.
Up until here it works and the visual cue to double click appears slightly below the center:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait as wait
driver = webdriver.Chrome()
driver.get("https://www.hallmark.nl/kaarten/verjaardag-man/")
#wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='https://www.hallmark.nl:443/kaarten/verjaardag-man/grappig-m/make-that-the-cat-wise/happy-bursdeej-to-jou-3415094.aspx']"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='https://www.hallmark.nl:443/kaarten/verjaardag-man/grappig-m/hallmark/een-jaguar-voor-je-verjaardag-3346861.aspx']"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.ID, "btnShowSizepicker"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//span[text()='Standaard']"))).click()
wait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class, 'showDesktop')]//button[contains(text(),'Binnenkant')]"))).click()
But from then on I can't seem to find anything to make selenium click on the element.
I have tried clicking the things I can locate in the html:
# elem = driver.find_element_by_xpath("//div[contains(#class, 'canvasAnchor')]").click()
This gives an element not interactable error. I have also tried using an offset for a findable element:
elem = driver.find_element_by_class_name("canvas-container")
print(elem.location)
print(elem.size)
action = webdriver.common.action_chains.ActionChains(driver)
action.move_to_element_with_offset(elem, 0.5*elem.location.get('x'), (0.5*elem.location.get('y'))) #should click in the middle of this container
action.double_click()
action.perform()
But for different elements and values of x and y, I can only get it to do either nothing, or click on the contact sidebar (which highlights the word 'Neem').
If I click twice manually, a textarea appears in the html and I can execute send_keys('hello world')
The element where you have to click is actually canvas, so what you have to do is that move cursor to the position where double click is needed and then click there.
I tried this over your code and worked for me:
from selenium.webdriver.common.action_chains import ActionChains
elem = driver.find_element_by_xpath('//*[#class="canvasWrapper active"]//*[#class="canvas-container"]')
ActionChains(driver).move_to_element_with_offset(elem, 70, 60).double_click().perform()
Note:You may have to wait for the canvas element to load properly. I just tried from prompt.
You can read about this mouse action here.
UPDATE:
I noticed that somehow the double click was behaving as single click for me, so changed to this.
from selenium.webdriver.common.action_chains import ActionChains
import time
elem = driver.find_element_by_xpath('//*[#class="canvasWrapper active"]//*[#class="canvas-container"]')
ActionChains(driver).move_to_element_with_offset(elem, 70, 60).click().perform()
time.sleep(1)
ActionChains(driver).move_to_element_with_offset(elem, 70, 60).click().perform()