How to Infinitely Update Labels with while loop tkinter - python-3.x

Hi I have some simple code here. https://pastebin.com/97uuqKQD
I would simply like to add something like this, to the button function, so while the root window is open the datetime and exrates are constantly regotten from the xe website and displayed.
amount = '1'
def continuousUpdate():
while amount == '0':
results()
def results():
#Get xe data put to labels etc here
btnConvert = tk.Button(root, text="Get Exchange Rates",command=continuousUpdate).place(x=5,y=102)
Once I input the two exrates and they then display on there respective labels I would like the program to continuously grab the data from xe over and over again.
Like this code here which runs in IPython no problems,
import requests
from bs4 import BeautifulSoup
from datetime import datetime
amount = '1'
while amount != '0':
t = datetime.utcnow()
url1 = "http://www.xe.com/currencyconverter/convert/" + "?Amount=" + amount + "&From=" + cur1 + "&To=" + cur2
url2 = "http://www.xe.com/currencyconverter/convert/" + "?Amount=" + amount + "&From=" + cur2 + "&To=" + cur1
#url = "http://www.xe.com/currencycharts/" + "?from=" + cur1 + "&to=" + cur2
html_code1 = requests.get(url1).text
html_code2 = requests.get(url2).text
soup1 = BeautifulSoup(html_code1, 'html.parser')
soup2 = BeautifulSoup(html_code2, 'html.parser')
i = i + 1
rate1 = soup1.find('span', {'class', 'uccResultAmount'})
rate2 = soup2.find('span', {'class', 'uccResultAmount'})
print ('#',i, t,'\n', cur1,'-',cur2, rate1.contents[0], cur2,'-',cur1, rate2.contents[0], '\n')
I thought I would just be able to throw the entire results function into a while loop function then simply call that function but no luck any help would be appreciated.???

Put this at the end of your results function without the while loop:
after_id = root.after(milliseconds,results)
This way it just keeps running itself after that time you specified. And this code will cancel it.
root.after_cancel(after_id)
Answering your other question in the comments:
To make a cancel button make sure after_id is global. Also if the time you specify is very low (very fast recall) it might restart again before you can cancel it. So to be safe better make a global boolean and put .after in an if boolean==True. And you can set that boolean to False whenever you hit cancel button or to True whenever you hit the start button, here's how you can do it:
# button_call default will be True so if you click on the button
# this will be True (no need to pass var through). You can use this to set
# your restart boolean to True
def func(button_call=True):
global restart
global after_id
if button_call:
restart = True
if restart:
after_id = root.after(ms,lambda: func(button_call=False) )
# This way you know that func was called by after and not the button
Now you can put this in your cancel button function:
def cancel():
global restart
global after_id
root.after_cancel(after_id)
restart = False
Let me know if this works (haven't tested it myself).

Related

How to automate an API to change the URL every hour and append the new data to a csv

I have successfully implemented an API that generates a unique URL to grab data from a database and downloads it into a csv. I am now attempting to automate this API so that it can generate the unique URL every hour and then append the csv file with the new data. I have no idea where to begin to automate this but the working API is pasted below so any help would be truly appreciated. Thank you.
import os
import sys
from datetime import datetime
from os.path import expanduser
import urllib.request
def main():
# API parameters
options = {}
options["url"] = "https://airnowapi.org/aq/data/"
options["start_date"] = "2020-01-01"
options["start_hour_utc"] = "01"
options["end_date"] = "2020-01-01"
options["end_hour_utc"] = "05"
options["parameters"] = "pm25"
options["bbox"] = "-76,38,-72,42"
options["data_type"] = "b"
options["format"] = "text/csv"
options["ext"] = "csv"
options["api_key"] = "" #NotIncludedforProtectionOfUniqueAPIkey
# API request URL
REQUEST_URL = options["url"] \
+ "?startdate=" + options["start_date"] \
+ "t" + options["start_hour_utc"] \
+ "&enddate=" + options["end_date"] \
+ "t" + options["end_hour_utc"] \
+ "&parameters=" + options["parameters"] \
+ "&bbox=" + options["bbox"] \
+ "&datatype=" + options["data_type"] \
+ "&format=" + options["format"] \
+ "&api_key=" + options["api_key"]
try:
# Request AirNowAPI data
print("Requesting AirNowAPI data...")
print(REQUEST_URL)
# User's home directory.
home_dir = expanduser("E:\SPRING2021\AIRNOWAPI\AIRNOWFILES")
download_file_name = "AirNowAPI" + datetime.now().strftime("_%Y%M%d%H%M%S." + options["ext"])
download_file = os.path.join(home_dir, download_file_name)
# Perform the AirNow API data request
api_data = urllib.request.URLopener()
api_data.retrieve(REQUEST_URL, download_file)
# Download complete
print ("Download URL: %s" % REQUEST_URL)
print("Download File: %s" % download_file)
except Exception as e:
print("Unable perform AirNowAPI request. %s" % e)
sys.exit(1)
if __name__ == "__main__":
main()
I find most of your code is well documented. There are many ways to automated your task. Here are the steps I would recommend you to do.
Create a Config file.
Try to separate your code from config(All the options data). You can even pickle it.
Make your code command line executable like python main.py config.yml, where you can pass config.file.
Checkpoint: Here your code should be one/multiple file and config is in another file.
Use a cronjob or any scheduler to trigger & step 3.
Shared Variables/Data?: If you have variables that need to be passed from first execution to another then you can use a static file, where you dump this data and using it for next interation

python requests.get() does not refresh the page

I have a piece of Python 3 code that fetches a webpage every 10 seconds which gives back some JSON information:
s = requests.Session()
while True:
r = s.get(currenturl)
data = r.json()
datetime = data['Timestamp']['DateTime']
value = data['PV']
print(str(datetime) + ": " + str(value) + "W")
time.sleep(10)
The output of this code is:
2020-10-13T13:26:53: 888W
2020-10-13T13:26:53: 888W
2020-10-13T13:26:53: 888W
2020-10-13T13:26:53: 888W
As you can see, the DateTime does not change with every iteration. When I refresh the page manually in my browser it does get updated every time.
I have tried adding
Cache-Control max-age=0
to the headers of my request but that does not resolve the issue.
Even when explicitely setting everything to None after loop, the same issue remains:
while True:
r = s.get(currenturl, headers={'Cache-Control': 'no-cache'})
data = r.json()
datetime = data['Timestamp']['DateTime']
value = data['PV']
print(str(datetime) + ": " + str(value) + "W")
time.sleep(10)
counter += 1
r = None
data = None
datetime = None
value = None
How can I "force" a refresh of the page with requests.get()?
It turns out this particular website doesn't continuously refresh on its own, unless the request comes from its parent url.
r = s.get(currenturl, headers={'Referer' : 'https://originalurl.com/example'})
I had to include the original parent URL as referer. Now it works as expected:
2020-10-13T15:32:27: 889W
2020-10-13T15:32:37: 889W
2020-10-13T15:32:47: 884W
2020-10-13T15:32:57: 884W
2020-10-13T15:33:07: 894W

How to remove list element using while loop

I open a web page to get the names of the cluster1(p1, p2). I am not sure how many times I need to open a web page to get these cluster1 names. So, I am using a while loop, it will remove the p1 or p2 whichever the value is obtained from web page.
When I open a web page, I'll get p1 or p2 and that value will be stored in new[-1]. If this value is in cluster1, it will execute the other test functions and that value will be removed from cluster1.
new = ['some', 'list' 'items', 'p1'] # last element of list is either p1 or p2. So, new[-1] will give p1 or p2.
cluster1 =[ 'p1', 'p2']
while len(cluster1) != 0:
print("Length of cluster1 before:", len(cluster1))
# for i in range(10):
if new[-1] in cluster1:
print(new[-1] + " is in cluster1.")
test1()
test2()
new_ver_names.append(new[-1])
cluster1.remove(new[-1])
print("Length of cluster1 after:", len(cluster1))
else:
print(new[-1] + " portal version is not listed.")
driver.quit()
break
My exception is, when value is removed, control should go back to while loop and start again until len(cluster1) is 0. And, if the value is not in cluster1, else past should execute. But, when I remove cluster1.remove(new[-1]), else part also gets executed.
I checked other answers where it is mentioned we can't remove items from a list while iterating over it and tried list comprehension. But, couldn't make it work.
I tried lst = [(teset1(), test2()) for i in range(len(cluster1)) if new[-1] in cluster1]
Any help is really appreciated.
Thank you.
Edit:
def login1():
ChromeDriver = 'C:\\PortalTesting\\Drivers\\chromedriver.exe'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--incognito")
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(executable_path=ChromeDriver, chrome_options=chrome_options)
driver.maximize_window()
driver.implicitly_wait(130)
driver.get("MY_URL")
print("session id ", driver.session_id)
username = driver.find_element_by_css_selector("#uid")
username.send_keys("username")
password = driver.find_element_by_css_selector("#pid")
password.send_keys("password")
login_button = driver.find_element_by_class_name("secondarybtnlabel")
login_button.click()
# time.sleep(10)
cluster1 = ['p1', 'p2']
dc_elm = driver.find_element_by_xpath('/html/body/div[4]/div/div[2]/span[2]').text
new = unicodedata.normalize('NFKD', dc_elm).encode('ascii', 'ignore').split()
print("Portal version: ", new[-1])
logout_btn = driver.find_element_by_xpath('/html/body/div[4]/div[2]/div/div[4]/div/div[2]/div/header/div[2]/table/tbody/tr/td[3]/div/li')
logout_btn.click()
driver.delete_all_cookies()
print("Clearing cookies")
new_ver_names = []
time.sleep(3)
while len(cluster1) != 0:
print("Length of cluster1 before:", len(cluster1))
# for i in range(10):
if new[-1] in cluster1:
print(new[-1] + " is in cluster1.")
test1()
test2()
new_ver_names.append(new[-1])
# cluster1.remove(new[-1])
print("Length of cluster1 after:", len(cluster1))
else:
print(new[-1] + " portal version is not listed.")
driver.quit()
break
# cluster1.remove(new[-1])
It is hard (for me) to understand what happens in the current code, but why not try sets?
new = {'some', 'list' 'items', 'p1'}
cluster1 ={'p1', 'p2'}
in_both = new & cluster1
not_found = cluster1 - new
# do stuff with values in `in_both` and `not_found`....

Selenium (Python) - waiting for a download process to complete using Chrome web driver

I'm using selenium and python via chromewebdriver (windows) in order to automate a task of downloading large amount of files from different pages.
My code works, but the solution is far from ideal: the function below clicks on the website button that initiating a java script function that generating a PDF file and then downloading it.
I had to use a static wait in order to wait for the download to be completed (ugly) I cannot check the file system in order to verify when the download is completed since i'm using multi threading (downloading lot's of files from different pages at once) and also the the name of the files is generated dynamically in the website itself.
My code:
def file_download(num, drivervar):
Counter += 1
try:
drivervar.get(url[num])
download_button = WebDriverWait(drivervar, 20).until(EC.element_to_be_clickable((By.ID, 'download button ID')))
download_button.click()
time.sleep(10)
except TimeoutException: # Retry once
print('Timeout in thread number: ' + str(num) + ', retrying...')
.....
Is it possible to determine download completion in webdriver? I want to avoid using time.sleep(x).
Thanks a lot.
You can get the status of each download by visiting chrome://downloads/ with the driver.
To wait for all the downloads to finish and to list all the paths:
def every_downloads_chrome(driver):
if not driver.current_url.startswith("chrome://downloads"):
driver.get("chrome://downloads/")
return driver.execute_script("""
var items = document.querySelector('downloads-manager')
.shadowRoot.getElementById('downloadsList').items;
if (items.every(e => e.state === "COMPLETE"))
return items.map(e => e.fileUrl || e.file_url);
""")
# waits for all the files to be completed and returns the paths
paths = WebDriverWait(driver, 120, 1).until(every_downloads_chrome)
print(paths)
Was updated to support changes till version 81.
I have had the same problem and found a solution. You can check weither or not a .crdownload is in your download folder. If there are 0 instances of a file with .crdownload extension in the download folder then all your downloads are completed. This only works for chrome and chromium i think.
def downloads_done():
while True:
for filename in os.listdir("/downloads"):
if ".crdownload" in i:
time.sleep(0.5)
downloads_done()
Whenever you call downloads_done() it will loop itself untill all downloads are completed. If you are downloading massive files like 80 gigabytes then i don't recommend this because then the function can reach maximum recursion depth.
2020 edit:
def wait_for_downloads():
print("Waiting for downloads", end="")
while any([filename.endswith(".crdownload") for filename in
os.listdir("/downloads")]):
time.sleep(2)
print(".", end="")
print("done!")
The "end" keyword argument in print() usually holds a newline but we replace it.
While there are no filenames in the /downloads folder that end with .crdownload
sleep for 2 seconds and print one dot without newline to console
I don't really recommend using selenium anymore after finding out about requests but if it's a very heavily guarded site with cloudflare and captchas etc then you might have to resort to selenium.
With Chrome 80, I had to change the answer from #florent-b by the code below:
def every_downloads_chrome(driver):
if not driver.current_url.startswith("chrome://downloads"):
driver.get("chrome://downloads/")
return driver.execute_script("""
return document.querySelector('downloads-manager')
.shadowRoot.querySelector('#downloadsList')
.items.filter(e => e.state === 'COMPLETE')
.map(e => e.filePath || e.file_path || e.fileUrl || e.file_url);
""")
I believe this is retro-compatible, I mean this shall be working with older versions of Chrome.
There are issues with opening chrome://downloads/ when running Chrome in headless mode.
The following function uses a composite approach that works whether the mode is headless or not, choosing the better approach available in each mode.
It assumes that the caller clears all files downloaded at file_download_path after each call to this function.
import os
import logging
from selenium.webdriver.support.ui import WebDriverWait
def wait_for_downloads(driver, file_download_path, headless=False, num_files=1):
max_delay = 60
interval_delay = 0.5
if headless:
total_delay = 0
done = False
while not done and total_delay < max_delay:
files = os.listdir(file_download_path)
# Remove system files if present: Mac adds the .DS_Store file
if '.DS_Store' in files:
files.remove('.DS_Store')
if len(files) == num_files and not [f for f in files if f.endswith('.crdownload')]:
done = True
else:
total_delay += interval_delay
time.sleep(interval_delay)
if not done:
logging.error("File(s) couldn't be downloaded")
else:
def all_downloads_completed(driver, num_files):
return driver.execute_script("""
var items = document.querySelector('downloads-manager').shadowRoot.querySelector('#downloadsList').items;
var i;
var done = false;
var count = 0;
for (i = 0; i < items.length; i++) {
if (items[i].state === 'COMPLETE') {count++;}
}
if (count === %d) {done = true;}
return done;
""" % (num_files))
driver.execute_script("window.open();")
driver.switch_to_window(driver.window_handles[1])
driver.get('chrome://downloads/')
# Wait for downloads to complete
WebDriverWait(driver, max_delay, interval_delay).until(lambda d: all_downloads_completed(d, num_files))
# Clear all downloads from chrome://downloads/
driver.execute_script("""
document.querySelector('downloads-manager').shadowRoot
.querySelector('#toolbar').shadowRoot
.querySelector('#moreActionsMenu')
.querySelector('button.clear-all').click()
""")
driver.close()
driver.switch_to_window(driver.window_handles[0])
import os
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
class MySeleniumTests(unittest.TestCase):
selenium = None
#classmethod
def setUpClass(cls):
cls.selenium = webdriver.Firefox(...)
...
def test_download(self):
os.chdir(self.download_path) # default download directory
# click the button
self.selenium.get(...)
self.selenium.find_element_by_xpath(...).click()
# waiting server for finishing inner task
def download_begin(driver):
if len(os.listdir()) == 0:
time.sleep(0.5)
return False
else:
return True
WebDriverWait(self.selenium, 120).until(download_begin) # the max wating time is 120s
# waiting server for finishing sending.
# if size of directory is changing,wait
def download_complete(driver):
sum_before=-1
sum_after=sum([os.stat(file).st_size for file in os.listdir()])
while sum_before != sum_after:
time.sleep(0.2)
sum_before = sum_after
sum_after = sum([os.stat(file).st_size for file in os.listdir()])
return True
WebDriverWait(self.selenium, 120).until(download_complete) # the max wating time is 120s
You must do these thing
Wait for server to finish inner business( for example, query from database).
Wait for server to finish sending the files.
(my English is not very well)
To obtain the return of more than one item, I had to change the answer of #thdox by the code below:
def every_downloads_chrome(driver):
if not driver.current_url.startswith("chrome://downloads"):
driver.get("chrome://downloads/")
return driver.execute_script("""
var elements = document.querySelector('downloads-manager')
.shadowRoot.querySelector('#downloadsList')
.items
if (elements.every(e => e.state === 'COMPLETE'))
return elements.map(e => e.filePath || e.file_path || e.fileUrl || e.file_url);
""")
This may not work for all usecases but for my simple need to wait for one pdf to download it works great. Based off of Walter's comment above.
def get_non_temp_len(download_dir):
non_temp_files = [i for i in os.listdir(download_dir) if not (i.endswith('.tmp') or i.endswith('.crdownload'))]
return len(non_temp_files)
download_dir = 'your/download/dir'
original_count = get_non_temp_len(download_dir) # get the file count at the start
# do your selenium stuff
while original_count == get_non_temp_len(download_dir):
time.sleep(.5) # wait for file count to change
driver.quit()
I had the same problem and this method worked for me.
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import ElementClickInterceptedException
from threading import Thread
import os
import datetime
def checkFilePresence(downloadPath, numberOfFilesInitially, artistName,
songTitle):
timeNow = datetime.datetime.now()
found = False
while not found:
numberOfFilesNow = len(os.listdir(downloadPath))
if numberOfFilesNow > numberOfFilesInitially:
for folders, subfolders, files in os.walk(downloadPath):
for file in files:
modificationTime = datetime.datetime.fromtimestamp\
(os.path.getctime(os.path.join(folders, file)))
if modificationTime > timeNow:
if file.endswith('.mp3'):
return
This code work in headless mode and return downloaded file name (based on
#protonum code):
def wait_for_downloads(download_path):
max_delay = 30
interval_delay = 0.5
total_delay = 0
file = ''
done = False
while not done and total_delay < max_delay:
files = [f for f in os.listdir(download_path) if f.endswith('.crdownload')]
if not files and len(file) > 1:
done = True
if files:
file = files[0]
time.sleep(interval_delay)
total_delay += interval_delay
if not done:
logging.error("File(s) couldn't be downloaded")
return download_path + '/' + file.replace(".crdownload", "")
def wait_for_download_to_be_don(self, path_to_folder, file_name):
max_time = 60
counter = 0
while not os.path.exists(path_to_folder + file_name) and time_counter < max_time:
sleep(0.5)
time_counter += 0.5
if time_counter == max_time:
assert os.path.exists(path_to_folder + file_name), "The file wasn't downloaded"
When using test automation, its crucial that developers make the software testable. It is your job to check the software combined with the testability, meaning that you need to request a spinner or a simple HTML tag which indicates when the download is done successfully.
In a case as yours, where you cannot check it in the UI and you cannot check in system, this is the best way to solve it.

Exiting Python While True loop

I am using a temperature sensor (HS18B20) on my raspberry pi. Every second or so, temperature is recorded onto a spreadsheet and sent to dropbox using a while True loop. It will do this forever until ctrl + c is pressed. In order to upload to dropbox, I cannot have the same file name as another file. The program will give an error and stop. Obviously, I could simply create a file with a different name everytime but to combat having hundreds of different files on dropbox, I delete the file on dropbox and immediately upload the new spreadsheet to dropbox. I have no problem with this technique, except for one thing...
If the program is stopped via ctrl + c right after the file is deleted and during the uploading of the file, the new file doesn't get uploaded and only gets deleted on dropbox (the local .xls file is never deleted; just replaced).
toc = time.strftime("%b %-d, %Y. %H:%M:%S")
dbx.files_delete('/Temperature Data ' + toc + '.xls')
with open(time.strftime("%m_%d_%Y") + '.xls', "rb") as f:
dbx.files_upload(f.read(), '/Temperature Data ' + toc + '.xls')
print("Uploaded to Dropbox")
I thought I set up a fail safe, but it still does not work.
Here is a snippet from my code.
def signal_handler(signal, frame):
signal interrupted
interrupted = True
signal.signal(signal.SIGINT, signal_handler)
interrupted = False
...
while True:
print("Temperature Taken")
a += 1
c += 1
if temp_f < input_bad:
ws.write(a,0,temp_f)
ws.write(a,1,time.strftime("%H:%M:%S %p"))
ws.write(a,2,"YES")
while c % 6 == 0:
c += 1
dbx.files_delete('/Temperature Data ' + toc + '.xls')
with open(time.strftime("%m_%d_%Y") + '.xls', "rb") as f:
dbx.files_upload(f.read(), '/Temperature Data ' + toc + '.xls')
print("Uploaded to Dropbox")
if interrupted:
print("Saving...")
dbx.files_delete('/Temperature Data ' + toc + '.xls')
with open(time.strftime("%m_%d_%Y") + '.xls',"rb") as f:
dbx.files_upload(f.read(), '/Temperature Data ' + toc + '.xls')
quit()
If I ctrl + c the program while it's taking a temperature readings, no problem! The program finishes its loop until c % 6 == 0 and then saves and closes.
If I ctrl + c the program while its saving, yes problem...
I could just print("Do not close the program") when its saving, but I would like to make this idiot proof.
Any tips? Thanks!
You can register a signal handler and the quit the loop when at the end of processing.
import signal
import sys
userQuitRequested = False
def signal_handler(signal, frame):
print('You pressed Ctrl+C!')
userQuitRequested = True
signal.signal(signal.SIGINT, signal_handler)
print('Press Ctrl+C to quit')
You can then check for userQuitRequested in your loop at the appropriate time, and break if True.

Resources