I'm using OpenGL via OpenTK and I'm rendering to a FramebufferObject in a background thread.
Now after each rendered frame, I want to display (part(s) of) the FBO in one or more OpenGL controls in my UI.
How does that work ?
You have two choices:
If your drivers support context sharing, you can bind the FBO texture on your OpenGL controls and display that directly (bind texture, render quad, done). Simple and fast - just make sure to synchronize your rendering with the display.
If your drivers don't, you'll have to readback the results of the rendering into a Bitmap object (or equivalent) via GL.ReadPixels. You can then re-upload them to your other OpenGL controls as textures or display them directly on non-OpenGL controls.
By default, OpenTK will always try to share contexts. Unfortunately, Intel drivers don't support context sharing, so you cannot use the first approach there.
Related
I was searching for a syscall that would draw a pixel on a given coordinate on the screen on something similar. But I couldn't find any such syscalls in this site.
I came to know that OS interacts with monitors using graphic drivers. But these drivers may be different on different machines. So is there a common native API provided by linux for handling these?
Much like how there are syscalls for opening, closing, reading, writing to files. Even though underlying file systems maybe different, these syscalls provide an abstract API for user programs to simplify things. I was searching something similar for drawing onto the screen.
Typically a user is running a display server and window system which organizes the screen into windows which applications draw to individually using the API provided by that system. The details will depend on the architecture of this system.
The traditional window system on Linux is the X window system and the more modern Wayland display server/protocol is also in common use. For example X has commands to instruct the X server to draw primitives to the screen.
If no such system is in use, you can directly draw to a display either via a framebuffer device or using the DRM API. Both are not accessed by special syscalls, but instead by using normal file syscalls like open, read, write, etc., but also ioctl, on special device files in /dev, e.g. /dev/dri/card0 for DRM to the first graphics card or /dev/fb0 for the first framebuffer device. DRM is also used for applications to render directly to the screen or a buffer when under a display server or window system as above.
In any case DRM is usually not used directly to draw e.g. pixels to the screen. It still is specific to the graphics card. Typically a library like Mesa3D is used to translate the specific details into a common API like OpenGL or Vulkan for applications to use.
How to get screenshot of graphical application programmatically? Application draw its window using EGL API via DRM/KMS.
I use Ubuntu Server 16.04.3 and graphical application written using Qt 5.9.2 with EGLFS QPA backend. It started from first virtual terminal (if matters), then it switch display to output in full HD graphical mode.
When I use utilities (e.g. fb2png) which operates on /dev/fb?, then only textmode contents of first virtual terminal (Ctrl+Alt+F1) are saved as screenshot.
It is hardly, that there are EGL API to get contents of any buffer from context of another process (it would be insecure), but maybe there are some mechanism (and library) to get access to final output of GPU?
One way would be to get a screenshot from within your application, reading the contents of the back buffer with glReadPixels(). Or use QQuickWindow::grabWindow(), which internally uses glReadPixels() in the correct way. This seems to be not an option for you, as you need to take a screenshot when the Qt app is frozen.
The other way would be to use the DRM API to map the framebuffer and then memcpy the mapped pixels. This is implemented in Chromium OS with Python and can be translated to C easily, see https://chromium-review.googlesource.com/c/chromiumos/platform/factory/+/367611. The DRM API can also be used by another process than the Qt UI process that does the rendering.
This is a very interesting question, and I have fought this problem from several angles.
The problem is quite complex and dependant on platform, you seem to be running on EGL, which means embedded, and there you have few options unless your platform offers them.
The options you have are:
glTexSubImage2D
glTexSubImage2D can copy several kinds of buffers from OpenGL textures to CPU memory. Unfortunatly it is not supported in GLES 2/3, but your embedded provider might support it via an extension. This is nice because you can either render to FBO or get the pixels from the specific texture you need. It also needs minimal code intervertion.
glReadPixels
glReadPixels is the most common way to download all or part of the GPU pixels which are already rendered. Albeit slow, it works on GLES and Desktop. On Desktop with a decent GPU is bearable up to interactive framerates, but beware on embedded it might be really slow as it stops your render thread to get the data (horrible framedrops ensured). You can save code as it can be made to work with minimal code modifications.
Pixel Buffer Objects (PBO's)
Once you start doing real research PBO's appear here and there because they can be made to work asynchronously. They are also generally not supported in embedded but can work really well on desktop even on mediocre GPU's. Also a bit tricky to setup and require specific render modifications.
Framebuffer
On embedded, sometimes you already render to the framebuffer, so go there and fetch the pixels. Also works on desktop. You can enven mmap() the buffer to a file and get partial contents easily. But beware in many embedded systems EGL does not work on the framebuffer but on a different 'overlay' so you might be snapshotting the background of it. Also to note some multimedia applications are run with UI's on the EGL and media players on the framebuffer. So if you only need to capture the video players this might work for you. In other cases there is EGL targeting a texture which is copied to the framebuffer, and it will also work just fine.
As far as I know render to texture and stream to a framebuffer is the way they made the sweet Qt UI you see on the Ableton Push 2
More exotic Dispmanx/OpenWF
On some embedded systems (notably the Raspberry Pi and most Broadcom Videocore's) you have DispmanX. Whichs is really interesting:
This is fun:
The lowest level of accessing the GPU seems to be by an API called Dispmanx[...]
It continues...
Just to give you total lack of encouragement from using Dispmanx there are hardly any examples and no serious documentation.
Basically DispmanX is very near to baremetal. So it is even deeper down than the framebuffer or EGL. Really interesting stuff because you can use vc_dispmanx_snapshot() and really get a snapshot of everything really fast. And by fast I mean I got 30FPS RGBA32 screen capture with no noticeable stutter on screen and about 4~6% of extra CPU overhead on a Rasberry Pi. Night and day because glReadPixels got was producing very noticeable framedrops even for 1x1 pixel capture.
That's pretty much what I've found.
I am considering different ways to stream a massive number of live videos to the screen in linux/X11, using multiple independent processes.
I started the project initially with openGL/GLX and openGL textures, but that was a dead end. The reason: "context switching". It turns out that (especially nvidia) performs poorly when several (independent multi-)processes are manipulating at fast pace textures, using multiple contexts. This results in crashes, freezes, etc.
( see the following thread: https://lists.freedesktop.org/archives/nouveau/2017-February/027286.html )
I finally turned into Xvideo and it seems to work very nicely. My initial tests show that Xvideo handles video dumping ~ 10 times more effectively than openGL and does not crash. One can demonstrate this running ~ 10 vlc clients with 720p#25fps and trying both Xvideo and OpenGL output (remember to put all fullscreen).
However, I am suspecting that Xvideo uses, under the hood, openGL, so let's see if I am getting this right ..
Both Xvideo and GLX are extension modules of X11, but:
(A) Dumping video through Xvideo:
XVideo considers the whole screen as a device port and manipulates it directly (it has these god-like powers, being an extension to X11)
.. so it only needs a single context from the graphics driver. Lets call it context 1.
Process 1 requests Xvideo services for a certain window .. Xvideo manages it into a certain portion of the screen, using context 1.
Process 2 requests Xvideo services for a certain window .. Xvideo manages it into a certain portion of the screen, using context 1.
(B) Dumping video "manually" through GLX and openGL texture dumping:
Process 1 requests a context from glx, gets context 1 and starts dumping textures with it.
Process 2 requests a context from glx, gets context 2 and starts dumping textures with it.
Am I getting this right?
Is there any way to achieve, using openGL directly, situation (A) ?
.. one might have to drop GLX completely, which starts to be a bit hard-core.
It's been a while but I finally got it sorted out, using OpenGL textures and multithreading. This seems to be the optimal way:
https://elsampsa.github.io/valkka-core/html/process_chart.html
(disclaimer: I did that)
Is it possible to directly modify frame buffer of a GPU without using any operating system or OpenGl/directX calls but just GPU driver calls?
Basically I want to overlay some animations on my screen and don't want operating system to overwrite any pixels of it.
Yes, it's possible by using the Dumb-Buffer API of the DRM (Direct Rendering Manager) driver.
You should definitely see David Herrmann's tutorials and download his examples. He only forgot to initialize the gamma ramps.
I set a texture2d as a device render target. After draw, how can i read and write pixels from render target directly , and present it then.
If your intent is to use it as an input to another shader:
- simply ensure you created the texture2D with the D3D11_BIND_RENDER_TARGET and D3D11_BIND_SHADER_RESOURCE.
Bind the texture as render target and render to it.
Unbind it as a render target and Bind it as a shader resource and use it in the next shader.
Just note that a texture cannot be bound as a target and a resource at the same time.
If your intent is to access the texture on the CPU using C++ as an array of pixels, then you have to do some work. Unfortunately, due to current GPU architectures, it is not possible to directly access a texture2D's pixels since the pixels actually live in GPU memory potential in a special format(swizzled format).
Create a texture the GPU can render into.
Create a staging texture(D3D11_USAGE_STAGING) that will be used to receive the output of the GPU.
Render to the GPU texture.
Issue a ID3D11DeviceContext::CopyResource() or ID3D11DeviceContext::CopySubresource()
Call ID3D11DeviceContext::Map() on the staging resource to get access to the pixels.
Call ID3D11DeviceContext::Unmap() on the staging resource.
Call ID3D11DeviceContext::UpdateSubresource() to update the version of the resource the GPU has.
As you can see this is certainly not a trivial set of operations and goes against what the GPU architecture of today is optimized to do. I certainly would not recommend it.
If you do end up going down this path, be sure to also read about all the perf concerns GPU memory read back comes with: http://msdn.microsoft.com/en-us/library/windows/desktop/bb205132(v=vs.85).aspx#Performance_Considerations