trouble getting the current url on selenium - python-3.x

I want to get the current url when I am running Selenium.
I looked at this stackoverflow page: How do I get current URL in Selenium Webdriver 2 Python?
and tried the things posted but it's not working. I am attaching my code below:
from selenium import webdriver
#launch firefox
driver = webdriver.Firefox()
url1='https://poshmark.com/search?'
# search in a window a window
driver.get(url1)
xpath='//input[#id="user-search-box"]'
searchBox=driver.find_element_by_xpath(xpath)
brand="freepeople"
style="top"
searchBox.send_keys(' '.join([brand,"sequin",style]))
from selenium.webdriver.common.keys import Keys
#EQUIValent of hitting enter key
searchBox.send_keys(Keys.ENTER)
print(driver.current_url)
my code prints https://poshmark.com/search? but it should print: https://poshmark.com/search?query=freepeople+sequin+top&type=listings&department=Women because that is what selenium goes to.

The issue is that there is no lag between your searchBox.send_keys(Keys.ENTER) and print(driver.current_url).
There should be some time lag, so that the statement can pick the url change. If your code fires before url has actually changed, it gives you old url only.
The workaround would be to add time.sleep(1) to wait for 1 second. A hard coded sleep is not a good option though. You should do one of the following
Keep polling url and wait for the change to happen or the url
Wait for a object that you know would appear when the new page comes
Instead of using Keys.Enter simulate the operation using a .click() on search button if it is available
Usually when you use click method in selenium it takes cared of the page changes, so you don't see such issues. Here you press a key using selenium, which doesn't do any kind of waiting for page load. That is why you see the issue in the first place

I had the same issue and I came up with solution that uses default explicit wait (see how explicit wait works in documentation).
Here is my solution
class UrlHasChanged:
def __init__(self, old_url):
self.old_url = old_url
def __call__(self, driver):
return driver.current_url != self.old_url:
#contextmanager
def url_change(driver):
current_url = driver.current_url
yield
WebDriverWait(driver, 10).until(UrlHasChanged(current_url))
Explanation:
At first, I created my own wait condition (see here) that takes old_url as a parameter (url from before action was made) and checks whether old url is the same like current_url after some action. It returns false when both urls are the same and true otherwise.
Then, I created context manager to wrap action that I wanted to make, and I saved url before action was made, and after that I used WebDriverWait with created before wait condition.
Thanks to that solution I can now reuse this function with any action that changes url to wait for the change like that:
with url_change(driver):
login_panel.login_user(normal_user['username'], new_password)
assert driver.current_url == dashboard.url
It is safe because WebDriverWait(driver, 10).until(UrlHasChanged(current_url)) waits until current url will change and after 10 seconds it will stop waiting by throwing an exception.
What do you think about this?

I fixed this problem by clicking on the button by using href. Then do driver.get(hreflink). Click() was not working for me!

Related

Selenium Web driver cannot find css selector even if its present (python)

I am trying to scrape data from seetickets.us. I am clicking on each org and then all events by that org. The scraper correctly scrape data from each event but the problem is that when I come back to all events page web driver cannot find the css selector.
Here is the site structure:
https://ibb.co/WBjMDJf
clicking on World Cafe Live get me here:
https://ibb.co/cLbMP19
clicking on any event will move me toward further info about event.
Now when the driver is coming back from extracting each event , It is not able to go into other event. I have also tried explicit wait anf time.sleep()
Here is my code:
#this is the func click on each event and extract data then come back to all event page
def get_all_events_in_each_event(self):
inner_events = self.get_all_inner_events()
print(len(inner_events))
for event in inner_events:
self.click_inner_event(event)
self.get_event_loc()
self.get_talent()
self.get_facebook()
self.get_date()
self.get_showtime_city()
self.get_ticket_doors()
self.back()
try:
WebDriverWait(self, 10).until(
EC.element_to_be_clickable((By.CLASS_NAME, "event-images-box")))
except Exception as e:
print("Wait Timed out")
print(e)
#this is the func to click on each event in all event pages
def click_inner_event(self , inner_event):
link = inner_event.find_element_by_css_selector('div[class="event-info"]')
link.click()
Here is HTML of all events page:
https://ibb.co/wcKWc68
Kindly help me with finding what's wrong here.
Thanks
As #Arundeep Chohan , correctly pointed that web driver loses reference when moving back and forth so I had to re grab all the elements.
Correct code is:
def get_all_events_in_each_event(self):
inner_events = self.get_all_inner_events()
for i in range(len(inner_events)):
self.click_inner_event(inner_events[i])
self.get_event_loc()
self.get_talent()
self.get_facebook()
self.get_date()
self.get_showtime_city()
self.get_ticket_doors()
self.back()
inner_events = self.get_all_inner_events() #regrabbing the elements
Thanks arundeep for the answer.

How to set window.alert when redirecting

I'm crawlling some web pages for my research.
I want to inject javascript code below when redirecting to other page:
window.alert = function() {};
I tried to inject the javascript code below using WebDriverWait, so that selenium may execute the code as soon as the driver redirect to new page. But It doesn't work.
while (some conditions) :
try:
WebDriverWait(browser, 5).until(
lambda driver: original_url != browser.current_url)
browser.execute_script("window.alert = function() {};")
except:
//do sth
original_url = browser.current_url
It seems that the driver execute javascript code after the page loaded because the alert that made in the redirected page is showing.
Chrome 14+ blocks alerts inside onunload (https://stackoverflow.com/a/7080331/3368011)
But, I think the following questions may help you:
JavaScript before leaving the page
How to call a function before leaving page with Javascript
JavaScript before leaving the page
I solved my problem in other way.
I tried and tried again with browser.switch_to_alert but it didn't work. Then I found that it was deprecated, so not works correctly. I checked the alert and dismiss it in every 1 second with following code :
while *some_condition* :
try:
Alert(browser).dismiss()
except:
print("no alert")
continue
This works very fine, in Windows 10, python 3.7.4

What is making the webbrowser close before it finishes?

I have the code bellow which I know has worked before but for some reason seems to be broken now. The code is mean't to open a search engine, search for a query and return a list of results by the href tag. The webbrowser will open and navigate to http://www.startpage.com success fully, it then puts the term I have entered at the bottom into the search box but then just closes the browser. No error, no links. Nothing.
import selenium.webdriver as webdriver
def get_results(search_term):
url = "https://www.startpage.com"
browser = webdriver.Firefox()
browser.get(url)
search_box = browser.find_element_by_id("query")
search_box.send_keys(search_term)
search_box.submit()
try:
links = browser.find_elements_by_xpath("//ol[#class='web_regular_results']//h3//a")
except:
links = browser.find_elements_by_xpath("//h3//a")
results = []
for link in links:
href = link.get_attribute("href")
print(href)
results.append(href)
browser.close()
return results
get_results("dog")
Does anyone know what is wrong with this? Basically it gets to search_box.submit() then skips everything until browser.close().
Unlike find_element_by_xpath (single WebElement) If find_elements_by_xpath won't find any results it won't throw an exception, it will return an empty list. links is empty so the for loop is never executed. You can change the try except to if condition, and check if it has values
links = browser.find_elements_by_xpath("//ol[#class='web_regular_results']//h3//a")
if not links:
links = browser.find_elements_by_xpath("//h3//a")
It is not recommended to use browser close function within the function that you are testing. Instead you can use after the get_results("dog") function and keep the test logic away.
get_results("dog")
browser.close()
By doing this way selenium will complete the execution of the function first and then close the browser window.
The problem with your solution is that the method is returning the result set after the browser is closing the window due to which you are facing logical problem with your script.

Multithreading with Selenium using Python and Telpot

I'm coding my first telegram bot, but now I have to serve multiple user at the same time.
This code it's just a little part, but it should help me to use multithread with selenium
class MessageCounter(telepot.helper.ChatHandler):
def __init__(self, *args, **kwargs):
super(MessageCounter, self).__init__(*args, **kwargs)
def on_chat_message(self, msg):
content_type, chat_type, chat_id = telepot.glance(msg)
chat_id = str(chat_id)
browser = browserSelenium.start_browser(chat_id)
userIsLogged = igLogin.checkAlreadyLoggedIn(browser, chat_id)
print(userIsLogged)
TOKEN = "***"
bot = telepot.DelegatorBot(TOKEN, [
pave_event_space()(
per_chat_id(), create_open, MessageCounter, timeout=10),
])
MessageLoop(bot).run_as_thread()
while 1:
time.sleep(10)
when the bot recive any message it starts a selenium session calling this function:
def start_browser(chat_id):
global browser
try:
browser.get('https://www.google.com')
#igLogin.checkAlreadyLoggedIn(browser)
#links = telegram.getLinks(24)
#instagramLikes(browser, links)
except Exception as e:
print("type error: " + str(e))
print('No such session! starting webDivers!')
sleep(3)
# CLIENT CONNECTION !!
chrome_options = Options()
chrome_options.add_argument('user-data-dir=/home/ale/botTelegram/users/'+ chat_id +'/cookies')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--lang=en')
print("Starting WebDrivers")
browser = webdriver.Chrome(options=chrome_options)
start_browser(chat_id)
return browser
and then this one check if the user is logged:
def checkAlreadyLoggedIn(browser, chat_id):
browser.get('https://www.instagram.com/instagram/')
try:
WebDriverWait(browser, 5).until(EC.element_to_be_clickable(
(By.XPATH, instagramClicks.buttonGoToProfile))).click()
print('User already Logged')
return True
except:
print('User not Logged')
userLogged = login(browser, chat_id)
return userLogged
and if the user is not logged it try to log the user in whit username and password
so, basically, when I write at the bot with one account everithing works fine, but if I write to the bot from two different account it opens two browser, but it controll just one.
What I mean it's that for example, one window remain over the google page, and then the other one recive two times the comand, so, even when it has to write the username, it writes the username two times
How can I interract with multiple sessions?
WebDriver is not thread-safe. Having said that, if you can serialise access to the underlying driver instance, you can share a reference in more than one thread. This is not advisable. But you can always instantiate one WebDriver instance for each thread.
Ideally the issue of thread-safety isn't in your code but in the actual browser bindings. They all assume there will only be one command at a time (e.g. like a real user). But on the other hand you can always instantiate one WebDriver instance for each thread which will launch multiple browsing tabs/windows. Till this point it seems your program is perfect.
Now, different threads can be run on same Webdriver, but then the results of the tests would not be what you expect. The reason behind is, when you use multi-threading to run different tests on different tabs/windows a little bit of thread safety coding is required or else the actions you will perform like click() or send_keys() will go to the opened tab/window that is currently having the focus regardless of the thread you expect to be running. Which essentially means all the test will run simultaneously on the same tab/window that has focus but not on the intended tab/window.
Reference
You can find a relevant detailed discussion in:
Chrome crashes after several hours while multiprocessing using Selenium through Python

Is there a way to slow down execution of Watir Webdriver under Cucumber?

Is there any way we can slow down the execution of Watir WebDriver under Cucumber?
I would like to visually track the actions performed by Watir. At the moment, it goes too fast for my eyes.
While Watir itself does not have an API for slowing down the execution, you could use the underlying Selenium-WebDriver's AbstractEventListener to add pauses before/after certain types of actions.
Given you want to see the result of actions, you probably want to pause after changing values and clicking elements. This would be done by creating the following AbstractEventListener and passing it in when creating the browser:
class ActionListener < Selenium::WebDriver::Support::AbstractEventListener
def after_change_value_of(element, driver)
sleep(5)
end
def after_click(element, driver)
sleep(5)
end
end
browser = Watir::Browser.new :firefox, :listener => ActionListener.new
For a full list of events that you can listen for, see the
Selenium::WebDriver::Support::AbstractEventListener documentation.
Not universally. You could Monkey Patch the element_call method to add a sleep after every interaction with a Selenium Element. Import this code after requiring watir-webdriver.
module Watir
class Element
alias_method :watir_element_call, :element_call
def element_call &block
watir_element_call &block
sleep 1
end
end
end
Also note, that Monkey Patching is generally a bad idea, and when I change the implementation (which I plan to), this code will break.

Resources