So, Brad Larson is awesome. I'm using his GPUImage library since he optimized the CGContextCreateImage for video output to instead render straight into OpenGL. Then he rewrote it to be even more amazing, and half the questions are outdated. The other half have the new callbacks, Like this question, but for the life of me, I can't get the video frames callback to not be nil. (the CMSampleBuffer to CIImage functions)
I know I have to "tag the next frame" to be kept in memory, thanks to his blog. I also know I process it (but GPUImageVideo also does that), then I grab from the framebuffer. Still nill.
The capture command that's supposed to auto-filter it into a CGImage, from the CGImagePicture's processImageUpToFilter function seems to be what I want, and I've seen it mentioned, but I am lost as to how to hook up the output to its frameBuffer.
Or should I use GPUImageRawDataOutput, and how to hook up? I've been copying and pasting, editing, experimenting, but unsure if it's just the fact I don't know openGL enough to hook up the right stuff or?
Any help is appreciated. I wouldn't ask, since so many related questions are up here, but I use them and still get nil on the output.
Here is my current try:
func willOutputSampleBuffer(sampleBuffer: CMSampleBuffer!) {
gpuImageVideoCamera.useNextFrameForImageCapture()
//Next line seems like a waste, as this func is called in GPUImageVideoCamera already.
gpuImageVideoCamera.processVideoSampleBuffer(sampleBuffer);
if let image = gpuImageVideoCamera.imageFromCurrentFramebuffer()
{
//it's nil
}
}
It seems to use the filter instead, and useNextFrame should be AFTER processing to not go super-slow.
Inside of willOutputSampleBuffer, this is it.
if let image = transformFilter.imageFromCurrentFramebuffer()
{
// image not nil now.
}
transformFilter.useNextFrameForImageCapture(); //enusre this comes after
This has given us stunning speeds that beat Google's p2p library. Brad, thanks, everyone should support your efforts.
Related
I am trying to make myself Digital audio to Analog audio converter
I have STM32F769i Discovery0: https://www.st.com/en/evaluation-tools/32f769idiscovery.html
Which has SPDIFRX and SPDIFTX ports
I found a fearly good starting point here: http://www.tjaekel.com/DiscoveryF7Audio/index.html
Also the guy posted here: https://www.openstm32.org/forumthread921
But the guy used STM32746G Discovery: https://www.st.com/en/evaluation-tools/32f746gdiscovery.html
Instead
So I went and tried to just get his SPDIF audio portion working on my board
My Project can be found here (I hope it compiles, with CubeIDE you never know what will happen :)): https://www.mediafire.com/file/n0s2z9p6nn735qg/SPDIF_Example.zip/file
I have no idea what I am doing wrong, but for some reason SPDIF_RX_IRQHandler (in stm32f7xx_it.c) is never called (LED never turns green, yea my debugging tehniques are primitive, but will explain why later)
So because of that HAL_SPDIFRX_ReceiveDataFlow_IT (in spdifrx.c) always returns HAL_TIMEOUNT, and of course no audio is ever played on the speakers
I am not sure what I am doing wrong
When I start MCU I call BSP_SPDIF_Init() (defined in spdifrx.c) in main.c right after I take care of the clock
if (BSP_SPDIFRX_Init() != HAL_OK)
{
Error_Handler();
}
And it appears it initializes all right, because I get HAL_OK back
Maybe I am not inializing GPIO properly from HAL_MspInit in stm32f7xx_hal_msp.c inproperly
I am realy out of ideas, what I am doing wrong, because the analog side of the audio does init, I can hear that as pop pop from the speakers when I power up my MCU, its just that SPDIF side has problems
I am is my setup crocked?
I am using this component radio as my SPDIF transmitter (Hama DIT2000M): https://de.hama.com/webresources/article-documents/00054/man/00054821man_en.pdf
It says it has SPDIF Audio out (it says its digital over coaxial)
I know its optical side is working fine because on my component receiver it plays just fine (it reports as 48khz Stereo)
Is my cable to long? I am using this cable: https://i.imgur.com/JqAxePF.jpg
(not sure who made it)
Now why do I debug with blinking leds, because where my test subject is (my Hama receiver), there is no computer so…. Blinking leds it is, I would like to avoid aditional libraries and have a minimum working example, because you never know what problems they could bring so that's why LCD is not used right now
I hope someone has any advice, eather how to get any data in to SPDIF port (because right now for some reason, I don't get anything) or what I am doing wrong for my audio not to play, the usage of STM32F769i Discovery0 instead of STM32746G Discovery is probably not responsible
I hope that this is a proper place for this king of questions, because I did ask a question regarding SPDIF on the STM forum: https://community.st.com/s/feed/0D53W00001z0RaqSAE
But didn't get any usefull advice there
Now SPDIF realy does not have much examples, there is only a polling example which does work (with the same cable), there is no interupt example, my interupt example (you can read the post on the STM forum post I linked) is not working as well (interupts are probably not broken right?)
So yea, I am lost a bit not sure what to do, and who to ask, so I tried here
PS: I know stackvoverflow does not like links to code, but I believe something is wrong with my project (interupts don't fire for some reason), and its realy hard to put this all into the question
Thanks for Anwsering and Best Regards
I managed to solve this, I guess I did not initialize SPDIF GPIO properly
after setting this
GPIO_InitStructure.Pin = GPIO_PIN_7;
GPIO_InitStructure.Mode = GPIO_MODE_AF_PP;
GPIO_InitStructure.Pull = GPIO_NOPULL;
GPIO_InitStructure.Speed = GPIO_SPEED_FAST;
GPIO_InitStructure.Alternate = DISCOVERY_SPDIFRX_AF;
HAL_GPIO_Init(GPIOD, &GPIO_InitStructure);
to this
GPIO_InitStructure.Pin = GPIO_PIN_12;
GPIO_InitStructure.Mode = GPIO_MODE_AF_PP;
GPIO_InitStructure.Pull = GPIO_NOPULL;
GPIO_InitStructure.Speed = GPIO_SPEED_FAST;
GPIO_InitStructure.Alternate = GPIO_AF7_SPDIFRX;
HAL_GPIO_Init(GPIOG, &GPIO_InitStructure);
interupts started to fire
What i am trying to accomplish is to have a user click on the canvas and the sprite(player) will move to that location and STOP once it reaches that location.
Currently i have the player can click and it will continue in that path which is the default function.
update: function() {
this.player.rotation = this.game.physics.arcade.angleToPointer(this.player);
if (this.game.input.activePointer.justPressed()) {
// move on the direction of input
this.game.physics.arcade.moveToPointer(this.player, this.playerSpeed);
}
}
Maybe its simpler than i think but i cant seem to find a solution.
point in the right direction would be useful :)
I think you look for something like this:
Using Tweens (something very interesting about Phaser)
OR
Using
Arcade Physics (which is an example similar to yours)
I'm new to web development and I'm currently in the process of building my own website for my portfolio. My app uses node, express, and Heroku to launch it online. However, the page isn't as smooth as I'd like it to be. It drops a lot of frames when scrolling and viewing animations. I'm wondering what I can do to make my app feel buttery smooth. I've looked all around for solutions but I'm too new to development to really understand what to do. I've attempted to use the inspector to see if the css or javascript files are slowing down the processes via the waterfall insepctor, but came up dry. So far, I've compressed the files and lowered the scale of all images. Also, the entire website is static content so I can't imagine why its running so slowly. Any help would be much appreciated.
The website in question
The "lag" could be caused by the ScrollFire plugin. Every time you call Materialize.scrollFire(..) you actually add a JavaScript listener for the "scroll" event. The way you use it, you call Materialize.scrollFire for each of your target objects, so you actually create multiple "scroll" listeners. But the scrollFire options is actually an array of targets, so you could get away with only initializing it once. Like so:
var options = [
{selector: '.iphone1', offset: 300, callback: function(el) {
$('.iphone1').css('visibility', 'visible');
$('.iphone1').addClass('animated slideInLeft');
}},
{selector: '#paragraph_intro', offset: 300, callback: function(el) {
$('#paragraph_intro').css('visibility', 'visible');
$('#paragraph_intro').addClass('animated slideInRight');
}},
// ... And so on
];
Materialize.scrollFire(options);
It could also be caused by your own scroll listener at:
$(window).on('scroll', function(){
updateNavigation();
changeNavColor();
changeHeaderColor();
});
I would consider adding some kind of throttling, so these functions are called less frequent while the user is scrolling.
These may not be the problem, or the whole problem, as I cannot see how the page would behave without it, but it could have an impact, so it's worth investigating.
I have a very simple program that plays 4 different tones, depending on what button is pressed. I have found that if I play multiple tones or the same tone in rapid succession, there are unpleasant clicking noises produced. I have made sure that these clicks are not present in my audio samples; it is definitely caused by playing the clips quickly one after another.
After googling around, I'm fairly sure that the clicks are due to the rapid change in pitch between clips. Looking at the waveform of the playback from the offending audio, it looks like a clip is first cancelled for a fraction of a second before starting the next clip. I have highlighted the section where this seems particularly obvious.
The clip that showcases these audio clicks can also be downloaded here.
My code is very simple. I am using XInput to read input from a connected controller, which determines the tone to play, and I am using WinMM to output sound from wav files. It is written in the D programming language, but I have modified it to use no D-specific features to make it as C-like as possible and to avoid confusion.
SHORT keyPressed(int vkey)
{
enum highBit { val = 0x8000 }
return cast(SHORT)(GetKeyState(vkey) & highBit.val);
}
enum Button
{
DPAD_UP = 0x0001,
DPAD_DOWN = 0x0002,
DPAD_LEFT = 0x0004,
DPAD_RIGHT = 0x0008,
START = 0x0010,
BACK = 0x0020,
LEFT_THUMB = 0x0040,
RIGHT_THUMB = 0x0080,
LEFT_SHOULDER = 0x0100,
RIGHT_SHOULDER = 0x0200,
A = 0x1000,
B = 0x2000,
X = 0x4000,
Y = 0x8000,
}
struct XINPUT_GAMEPAD
{
WORD wButtons;
BYTE bLeftTrigger;
BYTE bRightTrigger;
SHORT sThumbLX;
SHORT sThumbLY;
SHORT sThumbRX;
SHORT sThumbRY;
}
struct XINPUT_STATE
{
DWORD dwPacketNumber;
XINPUT_GAMEPAD Gamepad;
bool isPressed(int button)
{
return cast(bool)(Gamepad.wButtons & button);
}
}
int main()
{
HANDLE xinputDLL = initXinput();
XINPUT_STATE oldState;
XINPUT_STATE newState;
while (!keyPressed(VK_ESCAPE))
{
oldState = newState;
XInputGetState(0, &newState);
enum flags { val = SND_ASYNC | SND_FILENAME | SND_NODEFAULT }
if (newState.isPressed(Button.A) && !oldState.isPressed(Button.A))
{
PlaySoundA(toStringz("Piano.ff.A4.wav"), null, flags.val);
}
if (newState.isPressed(Button.B) && !oldState.isPressed(Button.B))
{
PlaySoundA(toStringz("Piano.ff.B4.wav"), null, flags.val);
}
if (newState.isPressed(Button.X) && !oldState.isPressed(Button.X))
{
PlaySoundA(toStringz("Piano.ff.C5.wav"), null, flags.val);
}
if (newState.isPressed(Button.Y) && !oldState.isPressed(Button.Y))
{
PlaySoundA(toStringz("Piano.ff.F4.wav"), null, flags.val);
}
}
denitXinput(xinputDLL);
return 0;
}
Assuming that I'm correct in regards to the source of the clicking sounds, I think the solution is to have each sample fade into the next one. However, I am not sure how to do this as the WinMM documentation seems relatively sparse, and I am inexperienced with it.
Is the solution to my problem of clicks when playing audio samples to have each sample fade into the next one? If so, how can I accomplish this using WinMM? If not, is there another solution that I can try?
I know how we can solve this in theory, but I don't have actual working code yet for all cases. (When I do, I'll edit this.)
First, the simple case which kinda works: instead of using PlaySound, try mciSendStringA:
if(auto err = mciSendStringA("play test.wav", null, 0, null))
writeln(err);
I am not making that up, Windows actually has that function, and it actually works with a lot of little command strings and file formats (though if your program terminates, all sound stops, so make sure the program keeps running e.g. stay in your controller loop or call Sleep(something)).
I've used a lot of Win32 and sometimes I'm amazed by how much stuff it has. Prototype:
extern(Windows) uint mciSendStringA(in char*,char*,uint,void*);
found in winmm.lib.
That basically works, but in my test, playing the same file twice at the same time has no effect. Playing different files together mixes them though. So it is a partial solution.
Next step from that would be to use the mciSendCommand function - a bit lower level than send string, so you can open multiple devices and try to get more overlap that way:
http://msdn.microsoft.com/en-us/library/windows/desktop/dd743675%28v=vs.85%29.aspx
I haven't tried this yet, but it looks fairly simple and I suspect it might be good enough for you. Open up a few devices for each button so you can hit them a few times fast and it cycles through them, hopefully mixing the same sound more than once when needed.
The prototype to that is:
extern(Windows) uint /*MCIERROR*/ mciSendCommandA(MCIDEVICEID,UINT,DWORD,DWORD);
Yes, it casts to void* then to DWORD in the msdn example. Blargh. Relevant structs:
struct MCI_OPEN_PARMSA {
DWORD dwCallback;
MCIDEVICEID wDeviceID; // aka uint
LPCSTR lpstrDeviceType;
LPCSTR lpstrElementName;
LPCSTR lpstrAlias;
}
struct MCI_PLAY_PARMS {
DWORD dwCallback;
DWORD dwFrom;
DWORD dwTo;
}
and you can borrow some constants from here too:
https://github.com/AndrejMitrovic/DWinProgramming/blob/master/WindowsAPI/win32/mmsystem.d#L693
(if you are already using the win32 bindings, great! But I think they are kinda a pain for little things so I try to avoid them, preferring to copy/paste prototypes+structs+constants off MSDN as I need them.)
You should be able to get the MSDN example working with those definitions and core.sys.windows.windows. Don't forget pragma(lib, "winmm"); too.
I think a full solution that will certainly work, but is also quite a bit harder, will be using the low level interface to mix the sounds yourself as they happen and send that result to the device. I don't have this working yet and I'm out of time today, but hopefully I can get something to you tomorrow.
The basic steps are:
1) call waveOutOpen to get a device. Set up a callback function which it calls when it needs more data.
2) prepare a buffer - or perhaps more than one - with waveOutPrepareHeader
3) feed data with waveOutWrite when requested by your callback (might want this in a separate thread) with the current notes. Mixing two samples is simply a case of adding the values together (and clipping if they overflow - sounds awful btw but hopefully that won't actually happen) so if you are doing more than one sound, just add them as you go.
Don't forget extern(Windows) on any callback function!
4) Loading your samples probably means reading the .wav file. That's not super hard, Windows has helper functions or you can do it yourself. I'll show code for this too.
What I have so far is in my simpleaudio.d https://github.com/adamdruppe/arsd/blob/master/simpleaudio.d find struct AudioOutput and the WinMM version. It has a horrible API right now that must be radically changed - it was acceptable on Linux but sucks on Windows. A callback feeder instead of write(data) should work better on both platforms, so that's what I'll do.
Problem I'm having with the demo right now is gaps between buffers... leading to clicky sounds. Yeah. But I'm sure it is just latency that should be solved with the proper callback approach and buffer sizing.
That MCI function might work for you as a next step though, maybe even a final step if the multiple devices works.
BTW: you could also prolly make it do MIDI commands instead of playing wavs and get all kinds of cool stuff. Simpleaudio.d's low level midi is already functioning - the demo main even shows a piano scale. Rigging it into the xbox controller shouldn't be too hard... note on when the button is pressed, note off when released, and not even think about timing.. Not really an answer to the question but a cool thing to play with in the same vein!
I've embedded an nsIWebBrowser in my application. Because I'm just generating HTML for it on the fly, I'm using OpenStream, AppendToStream, and CloseStream to add content. What I need is to add event listeners for mouse movement over the web browser as well as mouse clicks. I've read documentation and tried lots of different things, but nothing I have tried has worked. For instance, the code below would seem to do the right thing, but it does nothing:
nsCOMPtr<nsIDOMWindow> domWindow;
mWebBrowser->GetContentDOMWindow(getter_AddRefs(domWindow));
if (!mEventTarget) {
mEventTarget = do_QueryInterface(domWindow);
if (mEventTarget)
mEventTarget->AddEventListener(NS_LITERAL_STRING("mouseover"), (nsIDOMEventListener *)mEventListener, PR_FALSE);
}
Perhaps it isn't working because this is run during initialization, but before any content is actually added. However, if I add it during AppendStream, or CloseStream, it segfaults.
Please tell me a straightforward way to do this.
Well, here's the answer:
nsCOMPtr<nsIDOMEventTarget> cpEventTarget;
nsCOMPtr<nsIDOMWindow> cpDomWin;
m_pWebBrowser->GetContentDOMWindow (getter_AddRefs(cpDomWin));
nsCOMPtr<nsIDOMWindow2> cpDomWin2 (do_QueryInterface (cpDomWin));
cpDomWin2->GetWindowRoot(getter_AddRefs(cpEventTarget));
rv = cpEventTarget->AddEventListener(NS_LITERAL_STRING("mousedown"),
m_pBrowserImpl, PR_FALSE);