Example i was try use progress.bar, alive_progress.
from alive_progress import alive_bar
with alive_bar(100) as bar:
for i in range(100):
for i in combinations_with_replacement(['a','b','c','a','b','c','a','b'], 8):
b = (''.join(i))
bar()
and i hv many annoing problems or it slow down script , not work as expecter with print ( print simple print over this bar or its creates new bar after any print.
If i run it with no bar/counting it finish at ~0.001 second .
as exmaple i was try with tqdm
from itertools import *
from tqdm import trange
for i in trange(100):
for i in combinations_with_replacement(['a','b','c','a','b','c','a','b','a','b','c','a','b','c','a','b','a','b','c','a','b','c','a','b'], 8):
b = (''.join(i))
if i dissble it it finish less than second , if i use it , it expect time is 3 m , how to this this and not slowdown script?
UPD : if belive info what find at goodle that progress bars and counters for pyhon is not usable , only if you dont care bout 1000% slowdown script.
but some how progress bars is used at any brutforce scripts , generators etc...
i dont undestand .
from itertools import *
from collections import Counter
from tqdm import *
#for i in tqdm(Iterable):
limit = product(['0','1','2','3','4','5','6','7','8','9','a','b','c','d','e','f'], repeat = 8)
for i in trange(100):
for j in limit:
b = (''.join(j))
if b in ['bbbbbbbb','11111111','aaaaaaaa','00001111']:
print (b)
its static before script is and so to not work. ( stay at 0% all runing and go instantly to 100% when it end)
You can commnent logic "if b in and print" notting no change .
The reason the script is slowing down is that the combinations_with_replacement() function is being called for each progress bar update.
I calculated the value outside the loop and the output was instant.
The rate of execution went from 2 s/it (0.5 it/s) -> 60 it/s
from itertools import *
from tqdm import trange
limit = combinations_with_replacement(['a','b','c','a','b','c','a','b','a','b','c','a','b','c','a','b','a','b','c','a','b','c','a','b'], 8)
for i in trange(100):
for j in limit:
b = (''.join(j))
So in python you can do from foo import bar, this gets the bar from foo but how can I get something from inside of the bar object
I.e
from foo import bar.childDef # gets childDef() from inside of bar
Because normally you would have to type bar.childDef() with a regular import statement using from,
however I just wanna use childDef() instead of bar.childDef()
Sorry if this is a confusing or just bad question in general, I'm just curious.
you could use:
import datetime
now = datetime.datetime.now
or
def now():
import datetime
return datetime.datetime.now()
now()
## foo.py
class bar:
def childDef ():
print('Child Def')
And import from another script,
from foo import bar
# import method
imported_fun = bar.childDef
#call method
bar.childDef() # or imported_fun()
i'm new to python and i wrote a simple code just to run in sequence time and send email when is trigger.
i use threading timer for counting and import outbound.py (with import datetime) just to clean up the main scrip so it will not have email scrip in main.py. however when the first email received, the message time reflect correctly but the next email remain the same time from first email.
below is the main.py scrip
import threading
from outbound import outbound
def run ():
threading.Timer(10,run).start()
outbound()
run()
outbound.py scrip
from datetime import datetime
now=datetime.now()
dt_string = now.strftime("%d/%m/%Y %H:%M:%S")
def outbound():
print (dt_string)
You created dt_string when you imported the module, and never updated it again. Move the string initialization into the function body.
def outbound():
dt_string = datetime.now().strftime("%d/%m/%Y %H:%M:%S")
print (dt_string)
I want to be able to force spark to execute my code in the order I want.
In the example below, foo and bar functions does data manipulation but send_request is just a web trigger unaffected by those functions. When Spark executes below code it runs the send_request the first and foo and bar later.
It does not work for me because after foo and bar completed, I have timeout from my request. If the request ran after foo, the result would be ready at the same time bar ends.
How could I achieve this in spark.
I could have separate scripts for each step but cluster starting times adds up hence I would like to be able modify execution order.
I am using databricks on Azure if it helps.
import os
import base64
import requests
import pyspark
sc.addFile("dbfs:/bar.py")
from bar import bar
sc.addFile("dbfs:/foo.py")
from foo import foo
if __name__ == '__main__':
foo()
response = send_request(request=request_json)
bar()
the content of foo and bar and send_request are as follows
def foo():
df = spark.read.parquet(file_1_path)
df = df.filter(F.col('IDType') == 'E') \
.select(F.col('col1'),F.col('col2')).distinct()
df.repartition(10).write.parquet(file_1_new_path)
logger.info('1 foo is done')
and
def bar():
df = spark.read.parquet(file_2_path)
df = df.filter(F.col('IDType') == 'M') \
.select(F.col('col1'),F.col('col2')).distinct()
df.repartition(10).write.parquet(file_2_new_path)
logger.info('3 bar is done')
and
def send_request():
response_json = http_response.json()
logger.info('2 request is sent')
I will try to be more clear. When I run above code in spark the output I get is as follows
2 request is sent
1 foo is done
3 bar is done
But I want it to be in this order
1 foo is done
2 request is sent
3 bar is done
I'm writing a script where a user has to provide input for each element of a large list. I'm trying to use tqdm to provide a progress bar for the user, but I can't find a good way to get input within the tqdm loop without breaking the output.
I'm aware of tqdm.write() for writing to the terminal during a tqdm loop, but is there a way of getting input?
For an example of what I'm trying to do, consider the code below:
from tqdm import tqdm
import sys
from time import sleep
def do_stuff(x): sleep(0.5)
stuff_list = ['Alpha', 'Beta', 'Gamma', 'Omega']
for thing in tqdm(stuff_list):
input_string = input(thing + ": ")
do_stuff(input_string)
If I run this code, I get the following output:
0%| | 0/4 [00:00<?, ?it/s]Alpha: A
25%|█████████████████████ | 1/4 [00:02<00:07, 2.54s/it]Beta: B
50%|██████████████████████████████████████████ | 2/4 [00:03<00:04, 2.09s/it]Gamma: C
75%|███████████████████████████████████████████████████████████████ | 3/4 [00:04<00:01, 1.72s/it]Omega: D
100%|████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:05<00:00, 1.56s/it]
I've tried using tqdm.external_write_mode, but this simply didn't display the progress bar whenever an input was waiting, which is not the behaviour I'm looking for.
Is there an easy way of doing this, or am I going to have to swap libraries?
It isn't possible to display the progress bar while inside the input() function, because once a line is finished, it cannot be removed any more. It's a technical limitation of how command lines work. You can only remove the current line until you wrote a newline.
Therefore, I think the only solution is to remove the status bar, let the user input happen and then display it again.
from tqdm import tqdm
import sys
from time import sleep
def do_stuff(x): sleep(0.5)
stuff_list = ['Alpha', 'Beta', 'Gamma', 'Omega']
# To have more fine-control, you need to create a tqdm object
progress_iterator = tqdm(stuff_list)
for thing in progress_iterator:
# Remove progress bar
progress_iterator.clear()
# User input
input_string = input(thing + ": ")
# Write the progress bar again
progress_iterator.refresh()
# Do stuff
do_stuff(input_string)
If you don't like the fact that the progress_iterator object exists after the loop, use the with syntax:
with tqdm(stuff_list) as progress_iterator:
for thing in progress_iterator:
...
EDIT:
If you are willed to sacrifice platform independence, you can freely move the cursor and delete lines with this:
from tqdm import tqdm
import sys
from time import sleep
def do_stuff(x): sleep(0.5)
stuff_list = ['Alpha', 'Beta', 'Gamma', 'Omega']
# Special console commands
CURSOR_UP_ONE = '\x1b[1A'
# To have more fine-control, you need to create a tqdm object
progress_iterator = tqdm(stuff_list)
for thing in progress_iterator:
# Move the status bar one down
progress_iterator.clear()
print(file=sys.stderr)
progress_iterator.refresh()
# Move the cursor back up
sys.stderr.write('\r')
sys.stderr.write(CURSOR_UP_ONE)
# User input
input_string = input(thing + ": ")
# Refresh the progress bar, to move the cursor back to where it should be.
# This step can be omitted.
progress_iterator.refresh()
# Do stuff
do_stuff(input_string)
I think this is the closest you will get to tqdm.write(). Note that the behaviour of input() can never be identical to tqdm.write(), because tqdm.write() first deletes the bar, then writes the message, and then writes the bar again. If you want to display the bar while being in input(), you have to do some platform-dependent stuff like this.