I have a simple python script that creates docker network with threads simultaneously:
import os
import threading
i = 0
def c(a):
global i
while True:
if i:
print(a + " ---- " + str(os.system('docker network create test1')))
break
t1 = threading.Thread(target=c, args=('t1',))
t2 = threading.Thread(target=c, args=('t2',))
t1.start()
t2.start()
i = int(input("enter i: "))
t1.join()
t2.join()
In this case docker does not complain about network names and it is possible to have 2 networks with the same name. Of course it causes problems while using.
Should Docker be able to handle this type of cases or it is under the users' responsibility ?
Related
The following code is of a shop that has 5 items and three customers each demanding one item.
import multiprocessing as mp
class Shop:
def __init__(self, stock=5):
self.stock = stock
def get_item(self, l, x):
l.acquire()
if self.stock >= x:
self.stock -= x
print(f"{self.stock} = remaining")
l.release()
if __name__ == "__main__":
l = mp.Lock()
obj = Shop()
p1 = mp.Process(target=obj.get_item, args=(l, 1))
p2 = mp.Process(target=obj.get_item, args=(l, 1))
p3 = mp.Process(target=obj.get_item, args=(l, 1))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock)
The output that I got is as follows
4 = remaining
4 = remaining
4 = remaining
Final: 5
However, since I'm using Lock I was expecting it to be
4 = remaining
3 = remaining
2 = remaining
Final: 2
Question: How to achieve the above output just with Locks(and no process communication i.e without Pipe/Queue)?
The reason this code is not working as you expect it to is because multiprocessing does not share its state with child processes. This means that each of the process you start, p1, p2 and p3, get a copy of the object of class Shop. It is NOT the same object. There are two ways you can fix this, share the instance attribute stock with the processes, or share the whole object itself. The second way is probably better for your larger use case if the shop object holds other data that needs to be shared between the processes to.
Method 1:
To share the value of only the stock instance variable, you can use multiprocessing.Value. The way to create shared integers using this and also access their value is here:
shared_int = multiprocessing.Value('i', 5)
print(f'Value is {shared_int.value}') # 5
Adapting to your use case, the code will then become:
import multiprocessing
class Shop:
def __init__(self, stock=5):
self.stock = multiprocessing.Value('i', stock)
def get_item(self, l, x):
l.acquire()
if self.stock.value >= x:
self.stock.value -= x
print(f"{self.stock.value} = remaining")
l.release()
if __name__ == "__main__":
l = multiprocessing.Lock()
obj = Shop()
p1 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p2 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p3 = multiprocessing.Process(target=obj.get_item, args=(l, 1))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock.value)
Output
4 = remaining
3 = remaining
2 = remaining
Final: 2
Method 2
Sharing the whole complex object is a more involved process. I had recently answered a similar question in detail about sharing complex objects (like the object of class Shop in this case), which also covered the reasoning behind the code provided below. I recommend that you give it a read since it explains the logic behind the code provided at the bottom in greater detail. The only major difference for this use-case is that you will want to use multiprocess, a fork of multiprocessing, instead of multiprocessing. This library works identically to the built-in multiprocessing except for the fact that it offers better pickling support which we will need.
Basically, you will want to use multiprocessing.Managers to share the state, and a suitable proxy to access the state. The ObjProxy provided in below code is one such proxy which shares the namespace as well as instance methods (apart from protected/private attributes). Once you have these, you just need to create the objects of class Shop using the manager and the proxy. This is done using the newly added create method of class Shop. This is a class constructor and all objects of Shop should be created using this method only rather than directly calling the constructor. Full code:
import multiprocess
from multiprocess import Manager, Process
from multiprocess.managers import NamespaceProxy, BaseManager
import types
class ObjProxy(NamespaceProxy):
"""Returns a proxy instance for any user defined data-type. The proxy instance will have the namespace and
functions of the data-type (except private/protected callables/attributes). Furthermore, the proxy will be
pickable and can its state can be shared among different processes. """
def __getattr__(self, name):
result = super().__getattr__(name)
if isinstance(result, types.MethodType):
def wrapper(*args, **kwargs):
return self._callmethod(name, args, kwargs)
return wrapper
return result
class Shop:
def __init__(self, stock=5):
self.stock = stock
#classmethod
def create(cls, *args, **kwargs):
# Register class
class_str = cls.__name__
BaseManager.register(class_str, cls, ObjProxy, exposed=tuple(dir(cls)))
# Start a manager process
manager = BaseManager()
manager.start()
# Create and return this proxy instance. Using this proxy allows sharing of state between processes.
inst = eval("manager.{}(*args, **kwargs)".format(class_str))
return inst
def get_item(self, l, x):
with l:
if self.stock >= x:
self.stock -= x
print(f"{self.stock} = remaining")
def k(self, l, n):
pass
if __name__ == "__main__":
manager = Manager()
l = manager.Lock()
obj = Shop.create()
p1 = Process(target=obj.get_item, args=(l, 1, ))
p2 = Process(target=obj.get_item, args=(l, 1, ))
p3 = Process(target=obj.get_item, args=(l, 1, ))
p1.start()
p2.start()
p3.start()
p1.join()
p2.join()
p3.join()
print("Final: ", obj.stock)
Output
4 = remaining
3 = remaining
2 = remaining
Final: 2
Note : Explanation for these 2 lines:
manager = Manager()
l = manager.Lock()
The reason why we didn't need to create a manager (and subsequently a proxy) for the lock before in your example is outlined here. The reason why it does not work with the above code without creating a proxy is because we are no longer creating the processes in the main process, and the lock does not exist in the current processes memory space (since creating a manager for our complex object to share its state spawned its own server process)
I was just reading through the threading API in Python 3 and I don't see a method of syncing threads. In my case I'm creating performance testing in a Server and Client environment so I might want to use the thread class call this test with N users:
def tDoTest( ):
doSync() # Wait for all thread creation
doLogin()
doSync() # Wait for all users to login
startTest()
create100Rows()
endTest()
doSync() # Wait for completion
doLogout()
I was hoping there was a built in way to handle this that I missed.
You could use a blocking queue.
import queue
q = queue.Queue()
If you want one thread to wait for all members of a group of three other threads to do some task, the waiting thread could get() three tokens from the queue:
for i in range(3):
q.get()
Each of the three awaited threads could signify that it has completed its task by putting an informationless token into the queue:
q.put(())
The waiting thread will not exit its for loop until it has collected all three tokens.
You could wrap it up in a class:
import queue
import threading
import time
class CountdownLatch:
def __init__(self, N):
self.N = N
self.q = queue.Queue()
def check_in(self):
self.q.put(())
def await_all_checkins(self):
for i in range(self.N):
self.q.get()
def demo_CountDownLatch():
cdl = CountdownLatch(3)
def task(cdl,delay):
time.sleep(delay)
print(delay)
cdl.check_in()
threading.Thread(target=task, args=(cdl,2.0)).start()
threading.Thread(target=task, args=(cdl,1.5)).start()
threading.Thread(target=task, args=(cdl,1.0)).start()
cdl.await_all_checkins()
print("nighty night.")
if __name__ == "__main__":
demo_CountDownLatch()
OK after a bunch of searching this is the answer. Never expected this functionality to be named Barrier(). The is a built in function for it.
b = Barrier(2, timeout=5)
def doSync():
ct = threading.currentThread()
b.wait( )
print('Sync done: ' + ct.getName())
The output looks like as expected now:
0.75
1.0
Sync done: 1
Sync done: 2
0.75
1.0
Sync done: 1
Sync done: 2
0.75
1.0
Sync done: 1
Sync done: 2
Consider the following snippet:
import numpy as np
import multiprocessing as mp
import time
def work_standalone(args):
return 2
class Worker:
def __init__(self):
self.data = np.random.random(size=(10000, 10000))
# leave a trace whenever init is called
with open('rnd-%d' % np.random.randint(100), 'a') as f:
f.write('init called\n')
def work_internal(self, args):
return 2
def _run(self, target):
with mp.Pool() as pool:
tasks = [[idx] for idx in range(16)]
result = pool.imap(target, tasks)
for res in result:
pass
def run_internal(self):
self._run(self.work_internal)
def run_standalone(self):
self._run(work_standalone)
if __name__ == '__main__':
t1 = time.time()
Worker().run_standalone()
t2 = time.time()
print(f'Standalone took {t2 - t1:.3f} seconds')
t3 = time.time()
Worker().run_internal()
t4 = time.time()
print(f'Internal took {t3 - t4:.3f} seconds')
I.e. we have an object containing a large variable that uses multiprocessing to parallelize some work that has nothing to do with that large variable, i.e. does not read from or write to. The location of the worker process has a huge impact on the runtime:
Standalone took 0.616 seconds
Internal took 19.917 seconds
Why is this happening? I am completely lost. Note that __init__ is only called twice, so the random data is not created for every new process in the pool. The only reason I can think of why this would be slow is that data is copied around, but that would not make sense since it is never used anywhere, and python is supposed to use copy-on-write semantics. Also note that the difference disappears if you make run_internal a static method.
The issue you have is due to the target you are calling from the pool. That target is the function with the reference to Worker instance.
Now, you're right that the __init__() is only called twice. But remember, when you send anything to and from the processes, python will need to pickle the data first.
So, because your target is self.work_internal(), python has to pickle the Worker() instance every time the imap is called. This leads to one issue, self.data being copied over again and again.
The following is the proof. I just added 1 "input" statements, and fixed the last time of time calculation.
import numpy as np
import multiprocessing as mp
import time
def work_standalone(args):
return 2
class Worker:
def __init__(self):
self.data = np.random.random(size=(10000, 10000))
# leave a trace whenever init is called
with open('rnd-%d' % np.random.randint(100), 'a') as f:
f.write('init called\n')
def work_internal(self, args):
return 2
def _run(self, target):
with mp.Pool() as pool:
tasks = [[idx] for idx in range(16)]
result = pool.imap(target, tasks)
input("Wait for analysis")
for res in result:
pass
def run_internal(self):
self._run(self.work_internal)
# self._run(work_standalone)
def run_standalone(self):
self._run(work_standalone)
def work_internal(target):
with mp.Pool() as pool:
tasks = [[idx] for idx in range(16)]
result = pool.imap(target, tasks)
for res in result:
pass
if __name__ == '__main__':
t1 = time.time()
Worker().run_standalone()
t2 = time.time()
print(f'Standalone took {t2 - t1:.3f} seconds')
t3 = time.time()
Worker().run_internal()
t4 = time.time()
print(f'Internal took {t4 - t3:.3f} seconds')
You can run the code, when it shows up "wait for analysis", go and check the memory usage.
Like so
Then on the second time you see the message, press enter. And observe the memory usage increasing and decreasing again.
On the other hand, if you change self._run(self.work_internal) to self._run(work_standalone) you would notice that the speed is very fast, and the memory is not increasing, as well as the time taken is a lot shorter than doing self.work_internal.
Solution
One way to solve your issue is to set self.data as a static class variable. In normal cases, this would prevent instances from having to copy/reinit the variable again. This also prevented the issue from occuring.
class Worker:
data = np.random.random(size=(10000, 10000))
def __init__(self):
pass
...
Iam trying to improve the performance of my application. It is a Python3.6 asyncio.Protocol based TCP server (SSL wrapped) handling a lot of requests.
It works fine and the performance is acceptable when only one connection is active, but as soon as another connection is opened, the client part of the application slows down. This is really noticeable once there are 10-15 client connection.
Is there a way to properly handle requests in parallel or should I resort to running multiple server instances?
/edit Added code
main.py
if __name__ == '__main__':
import package.server
server = package.server.TCPServer()
server.join()
package.server
import multiprocessing, asyncio, uvloop
asyncio.set_event_loop_policy(uvloop.EventLoopPolicy())
from package.connection import Connection
class TCPServer(multiprocessing.Process):
name = 'tcpserver'
def __init__(self, discord_queue=None):
multiprocessing.Process.__init__(self)
self.daemon = True
# some setup in here
self.start()
def run(self):
loop = uvloop.new_event_loop()
self.loop = loop
# db setup, etc
server = loop.create_server(Connection, HOST, PORT, ssl=SSL_CONTEXT)
loop.run_until_complete(server)
loop.run_forever()
package.connection
import asyncio, hashlib, os
from time import sleep, time as timestamp
class Connection(asyncio.Protocol):
connections = {}
def setup(self, peer):
self.peer = peer
self.ip, self.port = self.peer[0], self.peer[1]
self.buffer = []
#property
def connection_id(self):
if not hasattr(self, '_connection_id'):
self._connection_id = hashlib.md5('{}{}{}'.format(self.ip, self.port, timestamp()).encode('utf-8')).hexdigest()
return self._connection_id
def connection_lost(self, exception):
del Connection.connections[self.connection_id]
def connection_made(self, transport):
self.transport = transport
self.setup(transport.get_extra_info('peername'))
Connection.connections[self.connection_id] = self
def data_received(self, data):
# processing, average server side execution time is around 30ms
sleep(0.030)
self.transport.write(os.urandom(64))
The application runs on Debian 9.9 and is started via systemd
To "benchmark" I use this script:
import os, socket
from multiprocessing import Pool
from time import time as timestamp
def foobar(i):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('127.0.0.1', 60000))
while True:
ms = timestamp()*1000
s.send(os.urandom(128))
s.recv(1024*2)
print(i, timestamp()*1000-ms)
if __name__ == '__main__':
instances = 4
with Pool(instances) as p:
print(p.map(foobar, range(0, instances)))
To answer my own question here. I went with a solution that spawned multiple instances which were listening on base_port + x and I put a nginx TCP loadbalancer in front of it.
The individual TCPServer instances are still spawned as own process and communicate among themselves via a separate UDP connection and with the main process via multiprocessing.Queue.
While this does not "fix" the problem, it provides a somewhat scalable solution for my very specific problem.
I'm new to python, multithreading, queue, and locking. What I'm trying to achieve is to login to many devices all at once, check for new image, download an image, then install it on all devices, in multi-threading process, download has to be done before start installing. I tried the following concept but I have no clue how to accomplish or connect/glue them together, any help will be appreciated in advance!
from threading import Thread
import Queue
import re
from LoginCli import Device_Under_test
import sys, os, string, threading
import time
# some global vars
num_threads = 25
ips_q = Queue.Queue()
out_q = Queue.Queue()
outlock = threading.Lock()
# build IP array
ips = []
for i in range(100,200):
ips.append("10.10.10."+str(i))
user = 'root'
pword = 'password'
log = 'output.log'
def login_to_device (i, q):
"""get hosts in queue"""
while True:
# get an IP item form queue
ip = q.get()
cmd = "check for image"
#login to device:this works just fine using my loginCli script
LoginCli = Device_Under_test(str(ip), user, pword)
print_cmd= LoginCli.send_exp(cmd)
with outlock:
print print_cmd
q.task_done()
def image_download (i, q):
while True:
#get an IP item form queue
ip = q.get()
cmd = "download for image"
#NOT sure if I need to login again!
LoginCli = Device_Under_test(str(ip), user, pword)
print_cmd= LoginCli.send_exp(cmd)
search = re.search(r'image download completed 100%', LoginCli)
#if is done 100% completed then start installing the image
#Once installing is completed then reboot all devices
with outlock:
print print_cmd
q.task_done()
# Not sure if I need these functions?
def image_downoad (i, q):
def image_install (i, q):
# start the thread pool
for i in range(num_threads):
worker = Thread(target=login_to_device, args=(i, ips_q))
worker.setDaemon(True)
worker.start()
for i in range(num_threads):
worker = Thread(target=image_downoad, args=(i, ips_q))
worker.setDaemon(True)
worker.start()
# fill queue
for ip in ips:
ips_q.put(ip)
# wait until worker threads are done to exit
ips_q.join()
# print result
while True:
try:
msg = out_q.get_nowait()
except Queue.Empty:
break
print msg