I have an ldap query that I am using to populate a database. I cant process more than 1000 at a time. Currently I am looping through each user and building a string using a counter. This seems inefficient . Is there a way to parse the 1000 at a time without using a counter? This is what I currently have.
users=conn.extend.standard.paged_search(ldapqeuery,paged_size=1000,generator=true)
for user in users:
if(counter<999)
appendToQuery(user)
else:
counter=0
executeSQL()
executeSQL() # this one is to do any remaining
There is no problem with using an auxiliar "counter" variable as you do - and it is part of Python "success formula" to allow one to just type-in what he is thinking about to solve a problem.
It can be made "prettier" by the use of the built-in enumerate, which can provide an index along with each item in a for iteration, though:
users=conn.extend.standard.paged_search(ldapqeuery,paged_size=1000,generator=true)
for counter, user in enumerate(users):
appendToQuery(user)
if counter % 1000 == 0 and counter > 0:
executeSQL()
if counter % 1000:
executeSQL()
Related
I have a uniform 2D coordinate grid stored in a numpy array. The values of this array are assigned by a function that looks roughly like the following:
def update_grid(grid):
n, m = grid.shape
for i in range(n):
for j in range(m):
#assignment
Calling this function takes 5-10 seconds for a 100x100 grid, and it needs to be called several hundred times during the execution of my main program. This function is the rate limiting step in my program, so I want to reduce the process time as much as possible.
I believe that the assignment expression inside can be split up in a manner which accommodates multiprocessing. The value at each gridpoint is independent of the others, so the assignments can be split something like this:
def update_grid(grid):
n, m = grid.shape
for i in range (n):
for j in range (m):
p = Process(target=#assignment)
p.start()
So my questions are:
Does the above loop structure ensure each process will only operate
on a single gridpoint? Do I need anything else to allow each
process to write to the same array, even if they're writing to
different placing in that array?
The assignment expression requires a set of parameters. These are
constant, but each process will be reading at the same time. Is this okay?
To explicitly write the code I've structured above, I would need to
define my assignment expression as another function inside of
update_grid, correct?
Is this actually worthwhile?
Thanks in advance.
Edit:
I still haven't figured out how to speed up the assignment, but I was able to avoid the problem by changing my main program. It no longer needs to update the entire grid with each iteration, and instead tracks changes and only updates what has changed. This cut the execution time of my program down from an hour to less than a minute.
I just want to know the logic within this problem. I just want my program to execute another loops once the first loop was done.
for i in range(1,5):
for j in range(1,5):
print(i*j,end="\t")
print("\n")
After the code above was done I want my program to perform another loop which is this one:
for i in range(5,1,-1):
for j in range(5,1,-1):
print(i*j,end="\t")
print("\n")
Just order them how you want them executed (in this case, one after the other). Python, among many other languages will just execute code sequentially unless you specifically introduce control structures like loops:
for i in range(1,5):
for j in range(1,5):
print(i*j,end="\t")
print("\n")
for i in range(5,1,-1):
for j in range(5,1,-1):
print(i*j,end="\t")
print("\n")
Just be aware that the second loop isn't a perfect reversal of the first. Since ranges are inclusive at the start and exclusive at the end, the tow complementary loops would be:
for i in range(1,5): # 1..4 inclusive
for i in range(4,0,-1): # 4..1 inclusive
Is there any way to Python pool an iteration itself?
For example, I tried the following:
def x():
for i in range(10):
...
The number of iterations is 10 (0 to 9), can we create a pool which creates 10 separate processes for iteration i=0, i=1, ... i=9?
The language does not have it in that simple form. But I remember seeing a small package that would provide an iterator for just that once. I will try to find it,but I doubt it is still maintained.
Here it is: https://github.com/npryce/python-parallelize
And present in Pypi: https://pypi.python.org/pypi/python-parallelize/1.0.0.0
If it works, it may be a matter of 'unmainained due to being complete' - just test your stuff - a lot.
I'm trying to make a simple slot machine python program for an assignment and I am having trouble getting the slot machine images to update. What I want to happen is for the user to click the button and have the three labels dynamically update with a different picture every 0.1 seconds for 2 seconds. But what is happening is my randint is generating random index numbers for the array, but the label only show a new image on the last randint instance. Here is the code I have:
def update_image():
global images
time.sleep(0.1)
y = images[randint(0, 3)]
slot1.delete()
slot1.configure(image = y)
print(y)
def slot_machine():
x = 0
while x != 10:
slot1.after(0,update_image)
x = x + 1
The problem is that you are calling after(0, ...) which adds a job to the "after" queue to be run as soon as possible. However, your while loop runs extremely quickly and never gives the event loop the chance to process queued events, so the entire loop ends before a single image changes.
Once the event loop is able to process the events, tkinter will try to process all pending events that are past due. Since you used a timeout of zero, they will all be past due so tkinter will run them all as fast as possible.
The better solution is to have the function that updates the image also be responsible for scheduling the next update. For example:
def update_image(delay, count):
<your existing code here>
# schedule the next iteration, until the counter drops to zero
if count > 0:
slot1.after(delay, update_image, delay, count-1)
With that, you call it once and then it calls itself repeatedly after that:
update_image(100, 10)
I have a piece of code which has to get executed every 100ms and update the GUI. When I am updating the GUI - I am pressing a button, which calls a thread and in turn it calls a target function. The target function gives back the message to the GUI thread using pub sub as follows.
wx.CallAfter(pub.sendMessage, "READ EVENT", arg1=data, arg2=status_read) # This command line is in my target function
pub.subscribe(self.ReadEvent, "READ EVENT") # This is in my GUI file - whihc calls the following function
def ReadEvent(self, arg1, arg2):
if arg2 == 0:
self.MessageBox('The program did not properly read data from MCU \n Contact the Program Developer')
return
else:
self.data = arg1
self.firmware_version_text_control.Clear()
#fwversion = '0x' + ''.join('{:02X}'.format(j) for j in reversed(fwversion))
self.firmware_version_text_control.AppendText(str(SortAndDecode(self.data, 'FwVersion')))
# Pump Model
self.pump_model_text_control.Clear()
self.pump_model_text_control.AppendText(str(SortAndDecode(self.data, 'ModelName')))
# Pump Serial Number
self.pump_serial_number_text_control.Clear()
self.pump_serial_number_text_control.AppendText(str(SortAndDecode(self.data, 'SerialNum'))[:10]) # Personal Hack to not to display the AA , AB and A0
# Pressure GAIN
self.gain_text_control.Clear()
self.gain_text_control.AppendText(str(SortAndDecode(self.data, 'PresGain')))
# Pressure OFFSET Offset
self.offset_text_control.Clear()
self.offset_text_control.AppendText(str(SortAndDecode(self.data, 'PresOffset')))
#Wagner Message:
#self.status_text.SetLabel(str(SortAndDecode(self.data, 'WelcomeMsg')))
# PUMP RUNNING OR STOPPED
if PumpState(SortAndDecode(self.data, 'PumpState')) == 1:
self.led6.SetBackgroundColour('GREEN')
elif PumpState(SortAndDecode(self.data, 'PumpState')) == 0:
self.led6.SetBackgroundColour('RED')
else:
self.status_text.SetLabel(PumpState(SortAndDecode(self.data, 'PumpState')))
# PUMP RPM
self.pump_rpm_text_control.Clear()
if not self.new_model_value.GetValue():
self.pump_rpm_text_control.AppendText("000")
else:
self.pump_rpm_text_control.AppendText(str(self.sheet_num.cell_value(self.sel+1,10)*(SortAndDecode(self.data, 'FrqQ5'))/65536))
# PUMP PRESSURE
self.pressure_text_control.Clear()
self.pressure_text_control.AppendText(str(SortAndDecode(self.data, 'PresPsi')))
# ON TIME -- HOURS AND MINUTES --- EDITING IF YOU WANT
self.on_time_text_control.Clear()
self.on_time_text_control.AppendText(str(SortAndDecode(self.data, 'OnTime')))
# JOB ON TIME - HOURS AND MINUTES - EDITING IF YOU WANT
self.job_on_time_text_control.Clear()
self.job_on_time_text_control.AppendText(str(SortAndDecode(self.data, 'JobOnTime')))
# LAST ERROR ----- RECHECK THIS AGAIN
self.last_error_text_control.Clear()
self.last_error_text_control.AppendText(str(SortAndDecode(self.data, 'LastErr')))
# LAST ERROR COUNT --- RECHECK THIS AGAIN
self.error_count_text_control.Clear()
self.error_count_text_control.AppendText("CHECK THIS")
As you can see my READEVENT is very big and it takes a while for the GUI to take enough time to successfully do all these things. My problem here is, when my GUI is updating the values of TEXTCTRL it is going unresponsive - I cannot do anything else. I cant press a button or enter data or anything else. My question is if there is a better way for me to do this, so my GUI wont go unresponsive. I dont know how I can put this in a different thread as all widgets are in the main GUI. But that also requires keep creating threads every 100ms - which is horrible. Any suggestions would be greatly helpful.
Some suggestions:
How long does SortAndDecode take? What about the str() of the result? Those may be good candidates for keeping that processing in the worker thread instead of the UI thread, and passing the values to the UI thread pre-sorted-and-decoded.
You can save a little time in each iteration by calling ChangeValue instead of Clear and AppendText. Why do two function calls for each text widget instead of just one? Function calls are relatively expensive in Python compared to other Python code.
If it's possible that the same value will be sent that was sent on the last iteration then adding checks for the new value matching the old value and skipping the update of the widget could potentially save lots of time. Updating widget values is very expensive compared to leaving them alone.
Unless there is a hard requirement for 100ms updates you may want to try 150 or 200. Fewer updates per second may be fast enough for most people, especially since it's mostly textual. How much text can you read in 100ms?
If you are still having troubles with having more updates than the UI thread can keep up with, then you may want to use a different approach than pubsub and wx.CallAfter. For example you could have the worker thread receive and process the data and then add an object to a Queue.Queue and then call wx.WakeUpIdle(). In the UI thread you can have an EVT_IDLE event handler that checks the queue and pulls the first item out of the queue, if there are any, and then updates the widgets with that data. This will give the benefit of not flooding the pending events list with events from too many wx.CallAfter calls, and you can also do things like remove items from your data queue if there are too many items in it.