When I try to open a webcam (FLIR Boson) with OpenCV on a Jetson TX2 it gives the following error:
libv4l2: error set_fmt gave us a different result then try_fmt!
VIDEOIO ERROR: libv4l unable convert to requested pixfmt
I am using this python script:
import numpy as np
import cv2
cap = cv2.VideoCapture(0)
while(True):
# Capture frame-by-frame
ret, frame = cap.read()
# Our operations on the frame come here
# Display the resulting frame
cv2.imshow('frame',frame)
if cv2.waitKey(1) & 0xFF == ord('q'):
break
# When everything done, release the capture
cap.release()
cv2.destroyAllWindows()
Although it does display the video it shows those errors. The reason that is relevant is I am trying to get the FLIR Boson to work with a Jetson TX2 running this program https://github.com/naisy/realtime_object_detection
I have it working with a regular webcam but with the FLIR Boson it gives
libv4l2: error set_fmt gave us a different result then try_fmt!
VIDEOIO ERROR: libv4l unable convert to requested pixfmt
VIDEOIO ERROR: V4L: Initial Capture Error: Unable to load initial memory buffers.
Segmentation fault (core dumped)
the above error and closes. In my research on the error, it seems to come up with people who use webcams that are monochrome, looking at this https://www.flir.com/support-center/oem/is-there-a-way-to-maximize-the-video-display-on-the-boson-app-for-windows-pc-to-full-screen/ I am wondering if I need to configure OpenCV or the V4L2 driver to choose the right format for the webcam to prevent the errors.
I also have a Jetson Xavier and the same object detection program works on it (it just has a different build of OpenCV and Tensorflow), so I am guessing that there is a slightly different configuration related to webcam format compatibility on that OpenCV install on the Xavier VS the TX2. I am new to all of this so forgive me if I ask for more clarification.
One last bit of info, this is out of the FLIR Boson manuel related to USB:
8.2.2 USB
Boson is capable of providing digital data as a USB Video Class (UVC) compliant device. Two output options are provided. Note the options are not selected via the CCI but rather by the video capture or viewing software selected by the user. The options are:
■ Pre-AGC (16-bit): The output is linearly proportional to the flux incident on each pixel in the array; output resolution is 320x256 for the 320 configuration, 640x512 for the 640 configuration. Note that AGC settings, zoom settings, and color-encoding settings have no effect on the output signal at this tap point. This option is identified with a UVC video format 4CC code of “Y16 ” (16-bit uncompressed greyscale image)
■ Post-Colorize, YCbCrb: The output is transformed to YCbCr color space using the specified color palette (see Section 6.7). Resolution is 640x512 for both the 320 and 640 configurations. Three options are provided, identified via the UVC video format 4CC code:
• I420: 8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes
• NV12: 8-bit Y plane followed by an interleaved U/V plane with 2x2 subsampling
• NV21: same as NV12 except reverse order of U and V planes
I have tried reinstalled everything several times, although it takes a few hours to reflash the TX2 and re-install open CV and Tensorflow. I have tried two different "builds" of opencv. I have tried to view the webcam with cheese and have never had a problem.
I don't work with Python but you need disable conversion to RGB:
cap.set(cv.CAP_PROP_CONVERT_RGB, 0)
See you v4l example from OpenCV.
I was able to find a way to get it to work, using the below code worked. It seemed to be a problem with open CV interacting with the v4l2.
pipeline = "v4l2src device=/dev/video1 ! video/x-raw,width=640,height=512,format=(string)I420,pixel-aspect-ratio=1/1, interlace-mode=(string)progressive, framerate=30/1 ! videoconvert ! appsink"
cap = cv2.VideoCapture(pipeline, cv2.CAP_GSTREAMER)
https://github.com/FLIR/BosonUSB/issues/13
Related
Situation is very weird, when I got it over a year ago, it was fine. I have updated the sdk, updated the windows 10, updated the firmware, all are latest.
behavior in the official "Azure Kinect Viewer" is the following:
device shown, opens fine.
if I hit start on the defaults I get a "camera failed: timed out" after a few seconds and all other sensors (depth, IR) show nothing, I think they are not the issue here (see 3)
if I de-select the color camera (after stop and start) the depth and IR are shown and getting 30fps as expected.
So, my problem is with the color camera connection, since everything runs on the same wires, I'm assuming the device is fine and that the cable works.
I have tried the color camera with and without the depth, on all options with respect to frame rate and resolution, none work at all, missing camera fails. In some cases I get an encrypted error message in the log, something with libUSB, I rarely get it so I'm not including the exact output.
When I open the "camera app" (standard windows 10) just to see if this app sees the color camera, I do get an image (Eureka!), but the refresh rate is like a frame every 5 seconds or more (!!!!), I've never seen anything like that.
I'm very puzzled with this behavior, I'm also including my device manager layout for the device perhaps that hints someone, as far as i can see, this should be a supported specification (I tried switching ports but in most cases the device was never detected or had the same color image behavior).
Any hints on how to move forward, much appreciated.
Bonus question:
If I solved this issue with the color camera, is there a way to work with the dk camera from a remote desktop session? (the microphones seem to not be detected when doing remote desktop). My USB device manager looks fine as far as I can see
ps.
I had also tried "disable streaming LED" and instead of camera failed in the dk viewer, I can get a color image but with a frame rate of about 1 frame per 4 seconds or more.
I am building a project wherein I capture video from webcam or usb camera or from url and perform object detection on the video using machine learning tensorflow API. Everything works fine if I take the input video from webcam or external usb camera but when I take input from IP camera using url the code fails after running for 30-40 seconds.
My code looks like this
import cv2
vid = cv2.VideoCapture(“rtsp://x.xx.xx.xx:554”)
While(True)
_,img = vid.read()
img = cv2.cvtColor(img,cv2.COLOR_BGR2RGB)
final_img = show_inference(detection_model , img)
final_img = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)
cv2.imshow(‘frame’, final_img)
If cv2.waitkey(1)
break
Vid.release()
cv2.destroyAllWindows()
This works fine when i execute it with webcam or usb camera using below lines:
cv2.VideoCapture(0) or cv2.VideoCapture(1)
But when i run using url it shows me frame for 30-40 seconds and then fails with the below error
OpenCV(4.4.0)\source\color.cpp:182: error:(-215:Asertion failed)!_src.empty() in function ‘cv::cvtColor’
It appears to me that the opencv library fails to capture live feed from url and then the code fails.
Anyone any idea how to resolve this issue, below are the versions and specifications i am using:
Using Tensorflow 2.0 on i5 machine without gpu
Hikvision PTZ IP camera
Python version 3.7
Opencv version 4.4
code:
check vid.isOpened(). if it isn't, do not read.
say rv, img = vid.read()
check that rv is True, otherwise break loop
are you throttling the reception of frames in any way? does your inference step take much time?
set the camera to a lower FPS value. the camera will produce frames at its own rate. it will not stop or slow down for you.
when you don't read frames, they queue up. they do not disappear. that will eventually cause you to crash or give you other types of failure. you absolutely must consume frames as fast as they are made.
I am working on ubuntu 16.04 and using a USB 2.0 webcam. I want to decrease the frame rate somehow since the project I'm working on requires face detection which really lags the video hence want to decrease the frame rate.
I've tried implementing the following code
import cv2
cap = cv2.VideoCapture(0)
cap.set(cv2.CAP_PROP_FPS, 15)
fps = int(cap.get(5))
print("fps:", fps)
while(cap.isOpened()):
ret,frame = cap.read()
if not ret:
break
cv2.imshow('frame', frame)
k = cv2.waitKey(1)
if k == 27:
break
I get the following error
(python3:24100): GStreamer-CRITICAL **: gst_element_get_state: assertion 'GST_IS_ELEMENT (element)' failed
If i set the frame rate in the above mentioned code to 30 (default frame rate) then i get a proper video, but if I change it I get the above mentioned error.
How can i decrease the frame rate either through code or even manually through settings (if there's a way)
Okay, there is several ways you can do this but I would suggest first checking the capabilities of the webcam. You can do this by installing:
sudo apt-get install v4l-utils
And run:
v4l2-ctl --list-formats-ext
If the desired frame rate is not listed you can increase the value in cv2.waitKey() and time it with time.time() to get the frame rate you want.
I am using eclipse 2018-09 (4.9.0) and PyDev 9.0.3. I am running a "standard" motion detect pipeline with python 3.6 and Opencv 3 on windos 10. I attached a snapshop of pstats from a run of 500 frames. The algorithm runs the usual mask, gray frame, Gaussian blur, absdiff, dilation and contour detection. I made sure there was movement in all frames. I save the movement frames (with moments). Next to file writing with imwrite, cv2.waitKey(1) is the largest consumer of time, to my surprise, slower than running the opencv pipeline (must be good code in opencv). I can't display using cv2.imshow without the cv2.waitKey(1). Is there work to remove this dependency, and, put the keyboard event in a callback?
!
For a computer vision project that I am working on I need to grab images using a Logitech C920 webcam. I am using OpenCV's VideoCapture to do that, but the problem that I am facing is that the image that I take at a certain moment does not show the latest thing that the camera sees. That is, if I take an image at timestamp t, it shows what the camera saw at timestamp (t - delta), so to say.
I did this by writing a program that increments a counter and shows it on the screen. I pointed the camera at the screen and let it record. When the counter reached a certain value, say 10000, it would grab an image and save it with the filename "counter_value.png" (e.g. 10000.png). That way I was able to compare the current value of the counter with the current value seen by the camera. I noticed that most of the time the delay is about 4-5 frames, but it is not a fixed value.
I saw similar posts about this issue, but none of them really helped. Some people recommended putting the frame grabbing routine into a separate thread and updating a "current_frame" Mat variable. I tried that, but for some reason the issue is still present. Someone else mentioned that the camera worked well on Windows (but I need to use Linux, though). I tried running the same code on Windows and indeed the delay was only about 1 frame (which might as well be that the camera did not see the counter because the screen did not update fast enough).
I then decided to run a simple webcam viewer based only on V4L2 code, thinking that the issue might be coming from OpenCV. I again experienced the same delay, which makes me believe that the driver is using some sort of buffer to cache the images.
I am new to V4L2 and I really need to solve this problem as soon as possible, so my questions to you guys are:
Has anyone found a solution for getting the latest image using V4L2 (and maybe OpenCV)?
If there is no way to solve it using V4L2, does anyone know another alternative to fixing this issue on Linux?
Regards,
Mihai
It looks like that there will be always a delay between the VideoCapture::grab() call and when the frame is actually taken. This is because of frame buffering that is done at hardware/SO level and you cannot avoid that.
OpenCV provides the VideoCapture::get( CV_CAP_PROP_POS_MEC) ) method to give you the exact time a frame was captured, but this is only possible if the camera supports it.
Recently a problem has been discovered in V4L OpenCV implementation:
http://answers.opencv.org/question/61099/is-it-possible-to-get-frame-timestamps-for-live-streaming-video-frames-on-linux/
And a few days ago a fix has been pulled:
https://github.com/Itseez/opencv/pull/3998
In the end, if you have the right setup, you can know what is the time a frame was taken (and therefore compensate).
It is possible the problem is with the Linux UVC driver, but I have been using Microsoft LifeCam Cinemas for machine vision on Ubuntu 12.04 and 14.04 machines, and have not seen a 4-5 frame delay. I operate them in low light conditions, though, in which case they reduce the frame rate to 7.5 fps.
One other possible culprit is a delay in the webcam depending what format is used. The C920 appears to support H.264 (which few webcams do), so Logitech may have put most effort to make this work well, yet OpenCV appears not to support H.264 on Linux; see this answer for what formats it supports. The same question also has an answer with a kernel hack(!) to fix an issue with the UVC driver.
PS: to check the format actually used in my case, I added
fprintf(stderr, ">>> palette: %d\n", capture->palette);
at this line in the OpenCV code.