Internet Clicks using Python Module - python-3.x

PROBLEM:
Power BI, under the pro-license, only allows data sources to be refreshed at most 8 times in a day and in 30 minute increments (using AM/PM timing).
This takes away my ability to make near real time decisions. So currently, my visualization is only updating every hour starting at 9:30 and the update happens 8 times, i.e. the total update times are 9:30, 10:30, 11:30, 12:30, 1:30, 2:30, 3:30, 4:30.
WORK AROUND:
So in order to bypass the pro license limitation on refreshes, I created code using PYAUTOGUI that will login to my Power BI server and click on that refresh button for me and I run this every 5 minutes.
WORK AROUND ISSUE:
The problem, is that this PYAUTOGUI will only work if the computer is active, i.e. I am logged in.
REQUEST:
What modules exist so that I can perform this same functionality in the background (without needing to have computer logged in or awake)?
NOTE:
I have done a search for packages from the command prompt using the code pip search mouse, pip search click, etc. but this is not the best use of time.

This is very much possible using selenium.
Firstly, ensure you have selenium installed in your environment. Run the following command:
pip install selenium
Then you'll need to make sure you have the correct chrome executable installed either in your PATH variable or explicitly written in your code (the latter option implies that you may use a config file as I have below, to avoid posting credentials). You can download the chrome executable for your version of Chrome here:
https://chromedriver.chromium.org/downloads
(You can do this with any browser, my choice is Chrome, though you will have to adjust the code as necessary)
Now the below code should work for you:
NOTE: The time module is essential here because if your code executes faster than the DOM loads, then selenium/python will return an error usually that error is saying it cannot find the elements you are looking for (because they haven't loaded yet)
'''REQUIRED PACKAGES'''
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
import time
'''CONFIG FILE FOR CHROME EXECUTABLE & LOGIN INFO'''
from _config import chrome_executable, user_name, user_pass
'''LINK TO SIGN IN PAGE FOR POWER BI'''
powerbi = 'https://app.powerbi.com/home?redirectedFromSignup=1&noSignUpCheck=1&response=AlreadyAssignedLicense'
'''SET CHROME OPTIONS TO RUN IN INCOGNITO AND HEADLESS(no browser window)'''
option = webdriver.ChromeOptions()
option.headless = True
option.incognito = True
'''CREATE BROWSER OBJECT / NAVIGATE TO POWER BI / MAXIMIZE WINDOW'''
browser = webdriver.Chrome(executable_path=chrome_executable, options=option)
browser.get(powerbi)
browser.maximize_window()
'''WAIT / FIND AND FILL IN EMAIL FIELD / FIND AND CLICK NEXT BUTTON'''
time.sleep(4)
bi_email = WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.ID, 'i0116')))
bi_email.send_keys(user_name)
bi_email.send_keys(Keys.ENTER)
'''WAIT / FIND AND FILL IN PASSWORD FIELD / FIND AND CLICK ON LOGIN BUTTON'''
time.sleep(4)
bi_pass = WebDriverWait(browser, 5).until(EC.presence_of_element_located((By.ID, 'i0118')))
bi_pass.send_keys(user_pass)
bi_pass.send_keys(Keys.ENTER)
'''THIS IS THE PROMPT THAT ASKS IF YOU WANT TO REMAIN LOGGED IN'''
'''WAIT / STAY ON CURRENT SELECTION / MOVE AND CLICK ON ADJACENT SELECTION'''
time.sleep(4)
yes_choice = browser.find_element_by_id('idSIButton9')
yes_choice.send_keys(Keys.SHIFT, Keys.TAB, Keys.ENTER)
'''WAIT / FIND AND CLICK ON MY WORKSPACE'''
time.sleep(7)
my_workspace_button = browser.find_element_by_class_name('workspaceName')
my_workspace_button.click()
'''CREATE MOUSE OBJECT'''
action = ActionChains(browser)
'''WAIT / FIND CORRECT ROW / CLICK ON REFRESH'''
time.sleep(4)
stock_alerts = browser.find_elements_by_xpath("//*[ text() = 'NAME OF YOUR DATA SOURCE' ]")
action.move_to_element(stock_alerts[-1]).click().send_keys(Keys.TAB).send_keys(Keys.ENTER).perform()
'''WAIT / CLOSE BROWSER'''
time.sleep(3)
browser.close()
This is the poor mans way of bypassing Power BIs strict 8 refreshes for a basic account. It also helps avoid having to buy a costly plan just to increase the number of data source scheduled updates.
Hope it helps

Related

How to check if Open File dialog has been open after pressing a button in a Chrome Browser tab on Python?

I'm trying to automate a process within the OpenSea Create page after having logged in with Metamask, and so far, I have managed to develop a simple program that chooses a particular image file using a path which passes to the Open File dialog "implicitly", here's the code:
import pyautogui
import time
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
def wait_xpath(code): #function to wait for the xpath of an element to be located
WebDriverWait(driver, 60).until(EC.presence_of_element_located((By.XPATH, code)))
opt = Options() #the variable that will store the selenium options
opt.add_experimental_option("debuggerAddress", "localhost:9222") #this allows bulk-dozer to take control of your Chrome Browser in DevTools mode.
s = Service(r'C:\Users\ResetStoreX\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe') #Use the chrome driver located at the corresponding path
driver = webdriver.Chrome(service=s, options=opt) #execute the chromedriver.exe with the previous conditions
nft_folder_path = r'C:\Users\ResetStoreX\Pictures\Cryptobote\Cryptobote NFTs\Crypto Cangrejos\SANDwich\Crabs'
start_number = 3
if driver.current_url == 'https://opensea.io/asset/create':
print('all right')
print('')
print(driver.current_window_handle)
print(driver.window_handles)
print(driver.title)
print('')
nft_to_be_selected = nft_folder_path+"\\"+str(start_number)+".png"
wait_xpath('//*[#id="main"]/div/div/section/div/form/div[1]/div/div[2]')
imageUpload = driver.find_element(By.XPATH, '//*[#id="main"]/div/div/section/div/form/div[1]/div/div[2]').click() #click on the upload image button
print(driver.current_window_handle)
print(driver.window_handles)
time.sleep(2)
pyautogui.write(nft_to_be_selected)
pyautogui.press('enter', presses = 2)
Output:
After checking the URL, the program clicks on the corresponding button to upload a file
Then it waits 2 seconds before pasting the image path into the Name textbox, for then pressing Enter
So the file ends up being correctly uploaded to this page.
The thing is, the program above works because the following conditions are met before execution:
The current window open is the Chrome Browser tab (instead of the Python program itself, i.e. Spyder environment in my case)
After clicking the button to upload a file, the Name textbox is selected by default, regardless the current path it opens with.
So, I'm kind of perfectionist, and I would like to know if there's a method (using Selenium or other Python module) to check if there's an Open File dialog open before doing the rest of the work.
I tried print(driver.window_handles) right after clicking that button, but Selenium did not recognize the Open File dialog as another Chrome Window, it just printed the tab ID of this page, so it seems to me that Selenium can't do what I want, but I'm not sure, so I would like to hear what other methods could be used in this case.
PS: I had to do this process this way because send_keys() method did not work in this page
The dialog you are trying to interact with is a native OS dialog, it's not a kind of browser handler / dialog / tab etc. So Selenium can not indicate it and can not handle it. There are several approaches to work with such OS native dialogs. I do not want to copy - paste existing solutions. You can try for example this solution. It is highly detailed and looks good.

Login to Google with Selenium via stackoverflow error

I am trying to create a script that will login to my google slide presentations and update them. In order to do this with Selenium it appears I have to login each time which causes the 'This browser or app may not be secure' error page, shown below.
I researched online that you should be able to work around this by logging in through stackoverflow or another site that uses 'log in with google'. However my below attempt seems to still provide the same results. Following up on this, I wasn't able to find out if this is still doable or if this approach no longer works.
Could someone confirm if my approach is outdated or if there is something I've failed to grasp from my approach? Sorry if I've missed something basic, I've just started writing Python so I'm deep in the learning curve stage.
I'm using Ubuntu 20.04 if that's of any use.
Code
#!/usr/bin/env python3
import webbrowser
import pyautogui
import time
import sys
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
driver=webdriver.Firefox()
driver.implicitly_wait(30)
url='https://accounts.google.com/o/oauth2/auth/identifier?client_id=717762328687-iludtf96g1hinl76e4lc1b9a82g457nn.apps.googleusercontent.com&scope=profile%20email&redirect_uri=https%3A%2F%2Fstackauth.com%2Fauth%2Foauth2%2Fgoogle&state=%7B%22sid%22%3A1%2C%22st%22%3A%2259%3A3%3Abbc%2C16%3A7667a52eee989fc7%2C10%3A1611306467%2C16%3Ad5e975ab3e71c656%2Cb45db9f242e90237a65d3c06754d76a1f8a7bbfc93dffe8b19376a3a573f700e%22%2C%22cdl%22%3Anull%2C%22cid%22%3A%22717762328687-iludtf96g1hinl76e4lc1b9a82g457nn.apps.googleusercontent.com%22%2C%22k%22%3A%22Google%22%2C%22ses%22%3A%22451ca6a7a6c349248b04db280731cf23%22%7D&response_type=code&flowName=GeneralOAuthFlow'
driver.get(url)
username = driver.find_element_by_id("identifierId")
username.send_keys("fake.user#gmail.com")
my_btn = driver.find_element_by_id('identifierNext')
my_btn.click()
This is because automation testing frameworks are disabled by Google for security reasons
However, there are a few things you can do:
The easiest way: Open browser controlled by selenium, open stack overflow and perform authorisation manually. When you next run the script, it will log you in automatically.
Add arguments to the script that block web security:
from selenium.webdriver.firefox.options import Options
options = FirefoxOptions()
options.add_argument('--disable-web-security')
options.add_argument('--user-data-dir')
options.add_argument('--allow-running-insecure-content')
You can also do this in the terminal.
Login to stackoverflow with your google account, once logged in go to your email.
driver.get("https://accounts.google.com/signin/oauth/identifier?client_id=717762328687-iludtf96g1hinl76e4lc1b9a82g457nn."
+ "apps.googleusercontent.com&scope=profile%20email&redirect_uri=https%3A%2F%2Fstackauth"
+ ".com.com%2Fauth%2Foauth2%2Fgoogle&state=%7B%22sid%22%3A1%2C%22st%22%3A%2259%3A3%3Abbc%2C16%3Af343579f103b7116%2C10%3A1611315201%2C16%3A360adcabd84812cf%2Cc1c668aab33e5327311ff2c3a0cb356918a99d0d6cb65a0da11008340199a802%22%2C%22cdl%22%3Anull%2C%22cid%22%3A%22717762328687-iludtf96g1hinl76e4lc1b9a82g457nn.apps.googleusercontent.com%22%2C%22k%22%3A%22Google%22%2C%22ses%22%3A%225b28397546b247c7994c2d3cd480cc31%22%7D&response_type=code&flowName=GeneralOAuthFlow");
driver.find_element_by_id("identifierId")\ #entering username
.send_keys("email#gmail.com")
sleep(2)
driver.find_element_by_id("identifierNext").click() #clicking next
driver.find_element_by_name("password")\ #entering password
.send_keys("password")
driver.find_element_by_id("passwordNext").click()#clicking next
sleep(2)
driver.get("https://mail.google.com/mail/u/0/#inbox") #going to email inbox
Click Insert → Link or press Ctrl + K (Cmd + K in Mac). You can also click the Insert link button on the toolbar.and your connection problem or email id you can please refresh and restart.

Selenium: Edge webdriver not waiting for page to load before executing next step (python)

I am writing some tests using selenium with Python. So far my suite works perfectly with Chrome and Firefox. However, the same code is not working when I try with the Edge (EdgeHTML). I am using the latest version at the time of writing which is release 17134, version: 6.17134. My tests are running on Windows 10.
The problem is that Edge is not waiting for the page to load. As part of every test, a login is first performed. The credentials are entred and the form submitted. Firefox and Chrome will now wait for the page we are redirected to, to load. However, with Edge, the next code is executed as soon as the login submit button is clicked which of course results in a failed test.
Is this a bug with Edge? It seems a bit too fundamental to be the case. Does the browser need to be configured in a certain manner? I cannot see anything in the documentation.
This is the code run with the last statement resulting in a redirect as we have logged in:
self.driver.find_element_by_id("login-email").send_keys(username)
self.driver.find_element_by_id("login-password").send_keys(password)
self.driver.find_element_by_id("login-messenger").click()
Edge decides it does not need to wait and will then execute the next code which is to navigate to a protected page. The code is:
send_page = SendPage(driver)
send_page.load_page()
More concisely:
self.driver.find_element_by_id("login-messenger").click()
# should wait now for the login redirect before excuting the line below but it does not!
self.driver.get(BasePage.base_url + self.uri)
I can probably perform a workaround by waiting for an element on the extent page to be present thus making Edge wait. This does not feel like the right thing to do. I certainly don't want to have to keep making invasive changes just for Edge.
Any advice please on what I should do?
Is this a bug with Edge? It seems a bit too fundamental to be the
case. Does the browser need to be configured in a certain manner? I
cannot see anything in the documentation.
No, I think it is not a bug with Edge browser. Because of the difference between the browser's performance, perhaps Edge browser will spend more time to load the page.
Generally, we could use the time.sleep(secs) method, WebDriverWait() method and implicitly_wait() method to wait the page load.
The code block below shows you how to wait for a page load to complete. It uses a timeout. It waits for an element to show on the page (you need an element id).
Then if the page is loaded, it shows page loaded. If the timeout period (in seconds) has passed, it will show the timeout error.
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get('https://pythonbasics.org')
timeout = 3
try:
element_present = EC.presence_of_element_located((By.ID, 'main'))
WebDriverWait(driver, timeout).until(element_present)
except TimeoutException:
print("Timed out waiting for page to load")
finally:
print("Page loaded")
More detail information, please check the following articles.
Wait until page is loaded with Selenium WebDriver for Python
How to wait for elements in Python Selenium WebDriver

Avoid download pop up in python3 using serelinum Download pdf

Object:
automate following process.
1. Open particular web page, fill the information in search box, submit.
2. from search results click on first result and download the PDF
Work done:
To reach to this object I have written a code as first step. The code works fine but opens up download pop up. Till the time I can't get rid of it, I can not automate the process further. Searched for very many solutions. But none has worked.
For instance, This solution is hard for me to understand and I think its more to do with Java then Python. I changed fire fox profile as suggested by many. This dose matches though not exactly same. I haven't tried as there is no much difference. Even this speaks about changing fire fox profile but that doesn't work.
My code is as below
import selenium.webdriver as webdriver
import selenium.webdriver.support.ui as ui
from time import sleep
import time
import wget
from wget import download
import os
#set firefox Profile
profile = webdriver.FirefoxProfile()
profile.set_preference('browser.download.folderList', 2)
profile.set_preference('browser.download.manager.showWhenStarting', False)
profile.set_preference("browser.download.manager.showAlertOnComplete", False)
profile.set_preference('browser.download.dir', os.getcwd())
profile.set_preference('browser.helperApps.neverAsk.saveToDisk', 'application/pdf')
#set variable driver to open firefox
driver = webdriver.Firefox(profile)
#set variable webpage to open the expected URL
webpage = r"https://documents.un.org/prod/ods.nsf/home.xsp" # edit me
#set variable to enter in search box
searchterm = "A/HRC/41/23" # edit me
#open the webpage with get command
driver.get(webpage)
#find the element "symbol", insert data and click submit.
symbolBox = driver.find_element_by_id("view:_id1:_id2:txtSymbol")
symbolBox.send_keys(searchterm)
submit = driver.find_element_by_id("view:_id1:_id2:btnRefine")
submit.click()
#list of search results open up and 1st occarance is clicked by coppying its id element
downloadPage = driver.find_element_by_id("view:_id1:_id2:cbMain:_id135:rptResults:0:linkURL")
downloadPage.click()
#change windiows. with sleep time
window_before = driver.window_handles[0]
window_after = driver.window_handles[1]
time.sleep(10)
driver.switch_to.window(window_after)
#the actual download of the pdf page
theDownload = driver.find_element_by_id("download")
theDownload.click()
Please guide.
The "Selections" popup is not a different window/tab, it's just an HTML popup. You can tell this because if you right click on the dialog, you will see the normal context menu. You just need to make your "Language" and "File type(s)" selections and click the "Download selected" button.

How to click the Continue button using Selenium and Python

I'm trying to automate some tedious copy / paste I do monthly from my bank's online service via Selenium and Python 3. Unfortunately, I can't get Selenium to click the log-in link.
It's the blue continue button at https://www1.bmo.com/onlinebanking/cgi-bin/netbnx/NBmain?product=5.
Strangely, when I try to click that link manually in the browser launched by Selenium, it doesn't work either - whereas it does work in a browser I launch manually.
I suspect the issue is that the bank's website is smart enough to detect that I'm automating the browser activity. Is there any way to get around that?
If not, could it be something else?
I've tried using Chrome and Firefox - to no avail. I'm using a 64 bit Windows 10 machine with Chrome 73.0.3683.103 and Firefox 66.0.
Relevant code is below.
#websites and log in information
bmo_login_path = 'https://www1.bmo.com/onlinebanking/cgi-bin/netbnx/NBmain?product=5'
bmo_un = 'fake_user_name'
bmo_pw = 'fake_password'
#Selenium setup
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
chrome_driver_path = 'C:\\Path\\To\\Driver\\chromedriver.exe'
gecko_driver_path = 'C:\\Path\\To\\Driver\\geckodriver.exe'
browswer_bmo = webdriver.Firefox(executable_path = gecko_driver_path)
#browswer_bmo = webdriver.Chrome(executable_path = chrome_driver_path)
#log into BMO
browswer_bmo.get(bmo_login_path)
time.sleep(5)
browswer_bmo.find_element_by_id('siBankCard').send_keys(bmo_un)
browswer_bmo.find_element_by_id('regSignInPassword').send_keys(bmo_pw)
browswer_bmo.find_element_by_id('btnBankCardContinueNoCache1').click()
Sending the keys works perfectly. I may actually have the wrong element ID (I was trying to test that in Chrome when I realized I couldn't click the link manually) - but I think the bigger issue is that I can't manually click the link in the browser launched by Selenium. Thank you for any ideas.
EDIT
This is a screenshot that I get of all I get when I try to click the continue button.
Ultimately the error message I get in my IDE (Jupyter Notebook) is:
TimeoutException: Message: timeout
(Session info: chrome=74.0.3729.108)
(Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729#{#29}),platform=Windows NT 10.0.17134 x86_64)
To click on the button with text as Continue you can fill up the Card Number and Password field inducing WebDriverWait for the element_to_be_clickable() and you can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument('start-maximized')
options.add_argument('disable-infobars')
options.add_argument('--disable-extensions')
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get('https://www1.bmo.com/onlinebanking/cgi-bin/netbnx/NBmain?product=5')
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.dijitReset.dijitInputInner#siBankCard[name='FBC_Number']"))).send_keys("1234567890112233")
driver.find_element_by_css_selector("input.dijitReset.dijitInputInner#regSignInPassword[name='FBC_Password']").send_keys("fake_password")
driver.find_element_by_css_selector("span.dijitReset.dijitInline.dijitIcon.dijitNoIcon").click()
# driver.quit()
Browser Snapshot:
I was able to fix this issue and solve the problem by adding the following line below the options variables. This disables the chrome check for automation. I used the whole sale code and then added the following line in the correct location before starting the driver.
options.add_experimental_option("excludeSwitches", ['enable-automation'])
ref: https://help.applitools.com/hc/en-us/articles/360007189411--Chrome-is-being-controlled-by-automated-test-software-notification

Resources