I'm trying to convert an AVI video into a series of images, saving only the most recent eight seconds of video into the 'buffer'. I'm only saving two bits of information: the timestamp for when the image was pulled from the AVI, and the image itself.
My problem is that even though the program is written for the buffer to overwrite itself, it is still having memory crashes - about 2 minutes in, for an 8s buffer. (error: (-4) Failed to allocate ###### bytes in function cv::OutofMemoryError) The code for this thread is below the aside for context.
(Context: I'm trying to create an emulator for a frame grabber. It constantly/endlessly converts a saved AVI, and restarts itself when it reaches the end of the video. The buffer is supposed to be analogous to whatever hardware - an HDD or SSD or whatever- that the frame grabber is actually saving to, in practice. This emulator runs as its own thread, parallel to another thread that processes & sends images as they are requested. The crash occurs without invoking the second thread.)
import base64, os, cv2, re, time, math
from PIL import Image
from threading import Thread
lobal numbertosave, framearray, sessionlist
framearray = []
vidfile = os.path.normpath("videolocation")
savepath = os.path.normpath("whereIsaverequestedimages"
class FrameInfo:
def __init__(self, frame, time):
self.f = frame
self.t = time
def VidEmulator():
global numbertosave, framearray, hit, fps
video = cv2.VideoCapture(vidfile)
fps = video.get(5)
##### Change the integer in the expression below to the number of seconds you want ot save
numbertosave = int(fps*5)
######## numbertosave = framerate * (# of seconds to store in buffer)
totalframes = video.get(7)
atframe = 1
count = 1
framearray = [0] * numbertosave
deltat = []
converting = True
while converting == True:
try:
ret, frame = video.read()
timestamp = time.time()
x = FrameInfo(frame, timestamp)
framearray[count] = ((x.f, x.t))
atframe = atframe + 1
count = count + 1
if count == numbertosave:
count = 1
VidEmulator()
except KeyboardInterrupt:
#### edited to remove mention of other thread, which isn't
#### invoked when testing this one (it is on request; see
#### comments) I honestly haven't tested this block since I
#### usually ctrl+printbreak
t1.join()
print "threads killed"
exit()
print "hopefully you don't see this"
if __name__ == '__main__':
t1 = Thread(target = VidEmulator)
t1.start()
I'm guessing there's probably a very simple logic problem I'm having here from preventing the buffer from actually overwriting itself, or some other variable is 'hanging', accumulating with consecutive runs and crashing the program. Otherwise, is there a better way to handle this problem, that will evade this error?
Thanks!
Related
I have a python program that reads from a serial port, accesses registers then writes data to the CSV. Id like to perform a writing operation to the csv at a set frequency of 100Hz. Below is sample code related to the timing which attempts to limit writing to the csv. For simplicity it will only print to console rather than write to csv.
import datetime
from datetime import timezone
import time
FREQ = 5
CYCLETIME = 1/FREQ
def main():
while(1):
### Code that will read message bytes from a port
# Time start for Hz
start = time.monotonic()
delta = time.monotonic() - start
if delta < CYCLETIME:
time.sleep(CYCLETIME - delta)
# in the full program we write to a csv but in this simple program we will just print it
milliseconds_since_epoch = datetime.datetime.now(timezone.utc)
print(milliseconds_since_epoch)
if __name__ == "__main__":
main()
2020-08-24 18:57:15.572637+00:00
2020-08-24 18:57:15.773183+00:00
2020-08-24 18:57:15.973637+00:00
2020-08-24 18:57:16.174117+00:00
2020-08-24 18:57:16.374569+00:00
2020-08-24 18:57:16.575058+00:00
2020-08-24 18:57:16.775581+00:00
2020-08-24 18:57:16.976119+00:00
2020-08-24 18:57:17.176627+00:00
2020-08-24 18:57:17.377103+00:00
2020-08-24 18:57:17.577556+00:00
The output seems consistent for 5Hz but if I change it to 100Hz it seems inconsistent. It sounds like this could be a time accuracy drift associated to time.monotonic. Is this a good approach? Is time.montonic appropriate?
My knowledge with pythons thread library is limited but in the near future, I plan to create 2 child threads for each task. One which constantly reads from serial and another that will write (or in our case print to console every 100Hz).
Edit:
I took the solution below and modified it. The original solution seemed to only print once. Here is my new attempt:
import datetime
from datetime import timezone
import time
FREQ = 5
CYCLETIME = 1/FREQ
def main():
t0 = time.perf_counter() # Time ref point in ms
time_counter = t0 # Will be incremented with CYCLETIME for each iteration
while 1:
### Code that will read message bytes from a port
now = time.perf_counter()
elapsed_time = now - time_counter
if elapsed_time < CYCLETIME:
target_time = CYCLETIME - elapsed_time
time.sleep(target_time)
# In the full program we write to a csv but in this simple program we will just print it
milliseconds_since_epoch = datetime.datetime.now(timezone.utc)
print(milliseconds_since_epoch)
time_counter += CYCLETIME
if __name__ == "__main__":
main()
Output:
I used matplot lib to determine the frequency to create this output. I take the rolling window difference of the current and previous value and inverse it since frequency=1/(time diff).
There are 5 problems in your algorithm:
Using time.sleep() has low accuracy, sometimes with an error around 10-13ms, depending on your system. (ref)
start and delta variables do not track the time spent on print(), as delta is set right after start.
start and delta variables use two different time.monotonic(). You should call the function only once, and pass the value to the other variable to make sure the same time value is being used.
Tick rate of time.monotonic() is 64 per second. Use time.perf_counter() instead. (ref)
You are doing time.sleep() on a delta time, which has low accuracy. If you have two threads running together, the cycles will hardly be synchronized. You need a time tracker which makes it sleep based on elapsed time.
Fix:
The following code make use of time.perf_counter(), and uses time_counter to keep track of the previous loop iteration timestamp.
The next loop time is the previous loop timestamp + cycle time - elapsed time. Thus, we will make sure the time.sleep() sleeps the program until then.
def main():
t0 = time.perf_counter() # Time ref point
time_counter = t0 # Will be incremented with CYCLETIME for each iteration
while 1:
### Code that will read message bytes from a port
now = time.perf_counter()
elapsed_time = now - t0
target_time = time_counter + CYCLETIME
if elapsed_time < target_time:
time.sleep(target_time - elapsed_time)
# In the full program we write to a csv but in this simple program we will just print it
milliseconds_since_epoch = datetime.datetime.now(timezone.utc)
print(milliseconds_since_epoch)
time_counter += CYCLETIME
I have a gzipped file spanning (compressed 10GB, uncompressed 100GB) and which has some reports separated by demarcations and I have to parse it.
The parsing and processing the data is taking long time and hence is a CPU bound problem (not an IO bound problem). So I am planning to split the work into multiple processes using multiprocessing module. The problem is I am unable to send/share data to child processes efficiently. I am using subprocess.Popen to stream in the uncompressed data in parent process.
process = subprocess.Popen('gunzip --keep --stdout big-file.gz',
shell=True,
stdout=subprocess.PIPE)
I am thinking of using a Lock() to read/parse one report in child-process-1 and then release the lock, and switch to child-process-2 to read/parse next report and then switch back to child-process-1 to read/parse next report). When I share the process.stdout as args with the child processes, I get a pickling error.
I have tried to create multiprocessing.Queue() and multiprocessing.Pipe() to send data to child processes, but this is way too slow (in fact it is way slower than doing it in single thread ie serially).
Any thoughts/examples about sending data to child processes efficiently will help.
Could you try something simple instead? Have each worker process run its own instance of gunzip, with no interprocess communication at all. Worker 1 can process the first report and just skip over the second. The opposite for worker 2. Each worker skips every other report. Then an obvious generalization to N workers.
Or not ...
I think you'll need to be more specific about what you tried, and perhaps give more info about your problem (like: how many records are there? how big are they?).
Here's a program ("genints.py") that prints a bunch of random ints, one per line, broken into groups via "xxxxx\n" separator lines:
from random import randrange, seed
seed(42)
for i in range(1000):
for j in range(randrange(1, 1000)):
print(randrange(100))
print("xxxxx")
Because it forces the seed, it generates the same stuff every time. Now a program to process those groups, both in parallel and serially, via the most obvious way I first thought of. crunch() takes time quadratic in the number of ints in a group, so it's quite CPU-bound. The output from one run, using (as shown) 3 worker processes for the parallel part:
parallel result: 10,901,000,334 0:00:35.559782
serial result: 10,901,000,334 0:01:38.719993
So the parallelized run took about one-third the time. In what relevant way(s) does that differ from your problem? Certainly, a full run of "genints.py" produces less than 2 million bytes of output, so that's a major difference - but it's impossible to guess from here whether it's a relevant difference. Perahps, e.g., your problem is only very mildly CPU-bound? It's obvious from output here that the overheads of passing chunks of stdout to worker processes are all but insignificant in this program.
In short, you probably need to give people - as I just did for you - a complete program they can run that reproduces your problem.
import multiprocessing as mp
NWORKERS = 3
DELIM = "xxxxx\n"
def runjob():
import subprocess
# 'py' is just a shell script on my box that
# invokes the desired version of Python -
# which happened to be 3.8.5 for this run.
p = subprocess.Popen("py genints.py",
shell=True,
text=True,
stdout=subprocess.PIPE)
return p.stdout
# Return list of lines up to (but not including) next DELIM,
# or EOF. If the file is already exhausted, return None.
def getrecord(f):
result = []
foundone = False
for line in f:
foundone = True
if line == DELIM:
break
result.append(line)
return result if foundone else None
def crunch(rec):
total = 0
for a in rec:
for b in rec:
total += abs(int(a) - int(b))
return total
if __name__ == "__main__":
import datetime
now = datetime.datetime.now
s = now()
total = 0
f = runjob()
with mp.Pool(NWORKERS) as pool:
for i in pool.imap_unordered(crunch,
iter((lambda: getrecord(f)), None)):
total += i
f.close()
print(f"parallel result: {total:,}", now() - s)
s = now()
# try the same thing serially
total = 0
f = runjob()
while True:
rec = getrecord(f)
if rec is None:
break
total += crunch(rec)
f.close()
print(f"serial result: {total:,}", now() - s)
I have this situation where I use OpenCV to detect faces in front of the camera and do some ML on those faces. The issue that I have is that once I do all the processing, and go to grab the next frame I get the past, not the present. Meaning, I'll read whats inside the buffer, and not what is actually in front of the camera. Since I don't care which faces came in front of the camera while processing, I care what is now in front of the camera.
I did try to set the buffer size to 1, and that did help quite a lot, but I still will get at least 3 buffer reads. Setting the FPS to 1, also dose not help remove this situation 100%. Bellow is the flow that I have.
let cv = require('opencv4nodejs');
let camera = new cv.VideoCapture(camera_port);
camera.set(cv.CAP_PROP_BUFFERSIZE, 1);
camera.set(cv.CAP_PROP_FPS, 2);
camera.set(cv.CAP_PROP_POS_FRAMES , 1);
function loop()
{
//
// <>> Grab one frame from the Camera buffer.
//
let rgb_mat = camera.read();
// Do to gray scale
// Do face detection
// Crop the image
// Do some ML stuff
// Do whats needs to be done after the results are in.
//
// <>> Release data from memory
//
rgb_mat.release();
//
// <>> Restart the loop
//
loop();
}
My question is:
Is it possible to remove the buffer all-together? And if so, how. If not, a why would be much appreciated.
Whether CAP_PROP_BUFFERSIZE is supported appears quite operating system and backend-specific. E.g., the 2.4 docs state it is "only supported by DC1394 [Firewire] v 2.x backend currently," and for backend V4L, according to the code, support was added only on 9 Mar 2018.
The easiest non-brittle way to disable the buffer is using a separate thread; for details, see my comments under Piotr Kurowski's answer. Here Python code that uses a separate thread to implement a bufferless VideoCapture: (I did not have a opencv4nodejs environment.)
import cv2, Queue, threading, time
# bufferless VideoCapture
class VideoCapture:
def __init__(self, name):
self.cap = cv2.VideoCapture(name)
self.q = Queue.Queue()
t = threading.Thread(target=self._reader)
t.daemon = True
t.start()
# read frames as soon as they are available, keeping only most recent one
def _reader(self):
while True:
ret, frame = self.cap.read()
if not ret:
break
if not self.q.empty():
try:
self.q.get_nowait() # discard previous (unprocessed) frame
except Queue.Empty:
pass
self.q.put(frame)
def read(self):
return self.q.get()
cap = VideoCapture(0)
while True:
frame = cap.read()
time.sleep(.5) # simulate long processing
cv2.imshow("frame", frame)
if chr(cv2.waitKey(1)&255) == 'q':
break
The frame reader thread is encapsulated inside the custom VideoCapture class, and communication with the main thread is via a queue.
This answer suggests using cap.grab() in a reader thread, but the docs do not guarantee that grab() clears the buffer, so this may work in some cases but not in others.
I set cap value after reading each frame to None and my problem solved in this way:
import cv2
from PyQt5.QtCore import QThread
if __name__ == '__main__':
while True:
cap = cv2.VideoCapture(0)
ret, frame = cap.read()
cv2.imshow('A', frame)
cv2.waitKey(0)
print('sleep!')
QThread.sleep(5)
print('wake up!')
cap = None
I have the same problem but in C++. I didn't find proper solution in OpenCV but I found a workaround. This buffer accumulates a constant number of images, say n frames. So You can read n frames without analysis and read frame once more. That last frame will be live image from camera. Something like:
buffer_size = n;
for n+1
{
// read frames to mat variable
}
// Do something on Mat with live image
So, basically I want to make timer but I don't want to use thread.Timer for
efficiency
Python produces thread by itself, it is not efficient and better not to use it.
I search the essay related to this. And checked It is slow to use thread.
e.g) single process was divided into N, and made it work into Thread, It was slower.
However I need to use Thread for this.
class Works(object):
def __init__(self):
self.symbol_dict = config.ws_api.get("ASSET_ABBR_LIST")
self.dict = {}
self.ohlcv1m = []
def on_open(self, ws):
ws.send(json.dumps(config.ws_api.get("SUBSCRIPTION_DICT")))
everytime I get the message form web socket server, I store in self.dict
def on_message(self,ws,message):
message = json.loads(message)
if len(message) > 2 :
ticker = message[2]
pair = self.symbol_dict[(ticker[0])]
baseVolume = ticker[5]
timestmap = time.time()
try:
type(self.dict[pair])
except KeyError as e:
self.dict[pair] = []
self.dict[pair].append({
'pair':pair,
'baseVolume' : baseVolume,
})
def run(self):
websocket.enableTrace(True)
ws = websocket.WebSocketApp(
url = config.ws_api.get("WEBSOCK_HOST"),
on_message = self.on_message,
on_open = self.on_open
)
ws.run_forever(sslopt = {"cert_reqs":ssl.CERT_NONE})
'once in every 60s it occurs. calculate self.dict and save in to self.ohlcv1m
and will sent it to db. eventually self.dict and self.ohlcv1m initialized again to store 1min data from server'
def every60s(self):
threading.Timer(60, self.every60s).start()
for symbol in self.dict:
tickerLists = self.dict[symbol]
self.ohlcv1m.append({
"V": sum([
float(ticker['baseVolume']) for ticker in tickerLists]
})
#self.ohlcv1m will go to database every 1m
self.ohlcv1 = [] #init again
self.dict = {} #init again
if __name__ == "__main__":
work=Works()
t1 = threading.Thread(target=work.run)
t1.daemon = True
t1.start()
work.every60s()
(sorry for the indention)
I am connecting to socket by running run_forever() and getting realtimedata
Every 60s I need to check and calculate the data
Is there any way to make 60s without thread in python27?
I will be so appreciate you answer If you give me any advice.
Thank you
The answer comes down to if you need the code to run exactly every 60 seconds, or if you can just wait 60 seconds between runs (i.e. if the logic takes 5 seconds, it'll run every 65 seconds).
If you're happy with just a 60 second gap between runs, you could do
import time
while True:
every60s()
time.sleep(60)
If you're really set on not using threads but having it start every 60 seconds regardless of the last poll time, you could time the last execution and subtract that from 60 seconds to get the sleep time.
However, really, with the code you've got there you're not going to run into any of the issues with Python threads you might have read about. Those issues come in when you've got multiple threads all running at the same time and all CPU bound, which doesn't seem to be the case here unless there's some very slow, CPU intensive work that's not in your provided code.
I wrote a continuous script that collects some data from the internet every few seconds, keeps it in memory for a while, periodically stores it all to db and then deletes it. To keep everything running smoothly I use threads to collect the data from several sources at the same time. To minimize db operations and to avoid conflict with other db processes, I only write every now and then.
The memory from the deleted variables is never returned and eventually becomes so large the script crashes (shown by tracemalloc and pympler). I guess I'm handling the data coming out of the threads wrong but I don't know how I could do it differently. Minimal example below.
Addition: I don't think I can use a queue because in reality multiple functions are threaded from this point, modifying different local variables.
import threading
import time
import tracemalloc
import pympler.muppy, pympler.summary
import gc
tracemalloc.start()
def a():
# collect data
collection.update({int(time.time()): list(range(1,1000))})
return
collection = {}
threads = []
start = time.time()
cycle = 0
while time.time() < start + 60:
cycle += 1
t = threading.Thread(target = a)
threads.append(t)
t.start()
time.sleep(1)
for t in threads:
if t.is_alive() == False:
t.join()
# periodically delete data
delete = []
for key, val in collection.items():
if key < time.time() - 10:
delete.append(key)
for delet in delete:
print('DELETING:', delet)
del collection[delet]
gc.collect()
print('CYCLE:', cycle, 'THREADS:', threading.active_count(), 'COLLECTION:', len(collection))
print(tracemalloc.get_traced_memory())
all_objects = pympler.muppy.get_objects()
sum1 = pympler.summary.summarize(all_objects)
pympler.summary.print_(sum1)