Is it possible to access and modify live stream / pixel buffers /video frames on MAC OS X, so that the modified frame (i.e. change the orientation, sharpness etc.) can be accessed by applications such as FaceTime/Skype? If yes, how?
What are the frameworks that will be used to modify the frames as specified in point [1]? Core Video/ AV Foundation / Core Media / OpenGL?
Is it even possible to do this from the Application Layer itself , without writing a kernel extension driver (KEXT) for my camera device ?
Related
I'm working on a fullscreen Windows desktop application that's moderately graphics-intensive, it uses OpenGL but only renders 2D content. Nothing fancy, mostly pushing pixels to the screen (up to 4K, single monitor) and uploading textures. We're using VSync to control the rendering framerate, ie. calling SwapBuffers() at the end of rendering to block until the next VBlank.
The main requirement we have is that the app runs at a solid 60FPS as it's used with a touchscreen, and interactions need to be as fluid as possible.
Because it's pretty basic, the app runs just fine on a 8th gen Intel i7 CPU with integrated Intel HD Graphics 630 GPU. Neither the CPU or GPU are anywhere near peak usage, and we can see that we're hitting a comfortable 60FPS through our in-app FPS meter. I also have it running with similar results on my Surface Book 2 with Intel i7 and integrated Intel UHD Graphics 620 GPU.
However, what I've recently started noticing is that the app sometimes starts dropping to 30FPS, then staying there either for long periods of time or sometimes even permanently. Through our FPS meter, I can tell that we're not actually spending any time rendering, it's just our SwapBuffers() call that blocks arbitrarily for 2 frames, capping us at 30FPS. The only way to get back to 60FPS is to alt-tab with another app and back to ours, or simply bringing the Windows menu up then going back to the app.
Because of the app going back to 60FPS afterwards, I'm positive that this is an intended behavior of the Intel driver, probably meant for gaming (gamers prefer a stable 30FPS rather than irregular/occasional dropped frames which make the game look choppy).
However in our case, dropping an occasional frame isn't a big deal, however being capped at 30FPS makes our UI and interactions far less pleasing to the eye, especially when it could easily render at a smooth 60FPS instead.
Is there any way to switch the driver behavior to prefer pushing 60FPS with occasional drops rather than capping at 30FPS?
OK so I was able to figure this out with a little bit of tweaking and reverse-engineering: The answer is that yes this is an intended but unfortunate default behavior of the Intel driver, and it can be fixed via the Intel HD Graphics Control Panel app if available, or directly in the registry otherwise (which is the only way to fix the issue on the Surface Book and other Surface devices, where the custom Intel driver doesn't expose the Intel HD Graphics Control Panel app anymore).
Starting with the simple solution: In the Intel HD Graphics Control Panel app, go to "3D", then "Application Settings". You'll first need to create an application profile, by selecting the file on disk for the process that creates the OpenGL window. Once that's done, the setting you want to adjust is "Vertical Sync". By default, "Use Application Default Settings" is selected. This is the setting that causes the capping at 30FPS. Select "Use Driver Settings" instead to disable that behavior and always target 60FPS:
This should've been pretty obvious, if it wasn't for Intel's horrible choice of terms and incomprehensible documentation. To me it looks like the choices for the settings are inverted: I would expect the capping to happen when I select "Use Driver Settings", which then implies the driver is free to adjust buffer swapping as it sees fit. Similarly, "Use Application Default Settings" implies that the app decides when to push frames, which is precisely the opposite of what the setting does. Even the little help bubbles in the app seem to contradict what these settings do...
ps: I'll post the registry-based solution in a separate answer to keep it short
Here is the registry-based answer, if your driver does not expose the Intel HD control panel (such as the driver used on the Surface Book and possibly other Surface laptops), or if you want to make that fix programmatically via regedit.exe or the Win32 API:
The application profiles created by the Intel HD control panel are saved in the registry under HKCU\Software\Intel\Display\igfxcui\3D using a key with the process file name (e.g. my_game.exe) and a REG_BINARY value with a 536-byte data blob divided like this:
Byte 0-3: Anisotropic Filtering (0 = use app default, 2-10 = multiplier setting)
Byte 4-7: Vertical Sync (0 = use app default, 1 = use driver setting)
Byte 8-11: MSAA (0 = use app default, 1 = force off)
Byte 12-15: CMAA (0 = use app default, 1 = override, 2 = enhance)
Byte 16-535: Application Display Name (wide-chars, for use in the control panel's application list)
Note: all values are stored in little-endian
In addition, you need to make sure that the Global value under the same key has its first byte set to 1, which is a sort of global toggle.(the control panel sets it to 1 when one or more entries are added to the applications list, then back to zero when the last entry is deleted from the list).
The format of value is also a REG_BINARY value with 8 bytes encoded like this:
Byte 0-3: Global toggle for application entries (0 = no entries, 1 = entries)
Byte 4-7: Application Optimal mode (0 = enabled, 1 = disabled)
For example:
I have a Dell Studio 1558, running x86_64 4.5.1-1-ARCH Linux. I noticed that it has a freefall sensor (setup as /dev/freefall) and I was wondering:
What's the easiest way of triggering an audio clip in linux when the sensor detects a fall? (as per https://xkcd.com/1363/)
I also note that a fall is supposed to trigger a shutdown of the HD for protection (the main reason for the sensor) so would I need to keep the sound clip in ram?
I have an app that delivers video content using HTTP Live Streaming. I want the app to retrieve the appropriate resolution based on the devices screen size (either 4x3 or 16x9). I ran Apple's tool to create the master .m3u8 playlist file (variantplaylistcreator) and got the following:
#EXTM3U
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=248842,BANDWIDTH=394849,CODECS="mp4a.40.2, avc1.4d4028",RESOLUTION=480x360
4x3/lo/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=384278,BANDWIDTH=926092,CODECS="mp4a.40.2, avc1.4d4028",RESOLUTION=480x360
4x3/mid/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=787643,BANDWIDTH=985991,CODECS="mp4a.40.2, avc1.42801e",RESOLUTION=480x360
4x3/hi/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=249335,BANDWIDTH=392133,CODECS="mp4a.40.2, avc1.4d4028",RESOLUTION=640x360
16x9/lo/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=384399,BANDWIDTH=950686,CODECS="mp4a.40.2, avc1.4d4028",RESOLUTION=640x360
16x9/mid/prog_index.m3u8
#EXT-X-STREAM-INF:AVERAGE-BANDWIDTH=780648,BANDWIDTH=987197,CODECS="mp4a.40.2, avc1.42801e",RESOLUTION=640x360
16x9/hi/prog_index.m3u8
This does cause my live stream to switch between video quality correctly, but it seems to randomly pick whether it uses a 4x3 or 16x9 resolution.
Is there a way to have it select the correct dimensions automatically, or do I need to have multiple playlist files and have a device request a specific one? For example, if on an iPad do I need to detect that it's screen has a 4x3 dimension and request a 4x3_playlist.m3u8 that only has the 480x360 resolution option?
Update 2017:
Keeping the same aspect-ratio is only a recommendation in the latest HLS authoring guide:
1.33. All video variants SHOULD have identical aspect ratios.
Original answer:
Audio/Video Stream Considerations:
Video aspect ratio must be exactly
the same, but can be different dimensions.
Apple Technical Note TN2224 - Best Practices for Creating and Deploying HTTP Live Streaming Media for the iPhone and iPad
Select a playlist based on the User-Agent instead.
I'm looking into rendering frames at a high rate (ideally next to the max monitor rate) and I was wondering if anyone had any idea at what level I should start looking into: kernel/driver level (OS space) ? X11 level ? svgalib (userspace) ?
On a modern computer, you can do it using the ordinary tools and APIs for graphics. If you have full frames full of random pixels, a simple bit blit from an in-memory buffer will perform more than adequately. Without any optimization work, I found that I could generate more than 500 frames per second on Windows XP using 2008 PCs.
I'm trying to get images from a minoru3d webcam, which is actually two Vimicro webcams plus a USB hub in a single package. The problem is, opencv always takes streams in maximum resolution, making simultaneous capture from two webcams impossible(due to usb constraints). How do I set resolution or FPS? For some reason, opencv calls
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_WIDTH, 320 );
cvSetCaptureProperty( capture, CV_CAP_PROP_FRAME_HEIGHT, 240 );
don't work. I don't need to work with opencv, any other library doing the same job is good for me. The webcam uses uvc drivers from kernel 2.6.30, with v4l2. I tried the custom module here: http://linuxtv.org/hg/~pinchartl/uvcvideo on my Ubuntu box with 2.6.27 kernel.
I used luvcview and v4l2cam for my purposes. 2 is specifically written for the Minoru.