I need to make a video of an audio equalizer.
So i need a script that analyses audio every frame, and extracts the frequency apectrum so i can draw that somehow and make an equalizer.
The first part of the problem is easily solvable on frontend as there is a myriad of open source equalizer visualisations in canvas.
The thing works nicely in browser but i have a problem to make an mp4 of that.
Ive tried using headless browsers(pupeteer and phantomjs) to capture frames from canvas, but i could not get the framerate above 10fps, resulting in unacceptable video quality and sync issues when connecting the jpg frames and mp3 via ffmpeg. The plan was to speed it up, so you dont have to wait for the full audio length to finish to get an mp4, but i cant even get it to show above 10fps on regular playback speed.
I feel the tech i thought would work is not there yet, and i might be in need of a different approach.
The only condition is that it has to run as a script on a linux server. So any programmimg language or any equalizer design will work.
Any ideas or resources are more than welcome. Thanks
Related
My Zoom H4n somehow decided it didn't want to properly save two recordings this weekend, leaving me with four zero byte files (which I have tried any which way to open/convert, but nothing was working).
I then used CardRescue to scan the SD card for any audio it could find, and - lo and behold - I got .wav files! However, instead of two files for each session (one was an XLR output from the desk, the other the on-Zoom mics), or even a nice stereo with one left, the other right, I have a mess.
In importing as raw data to Audacity (the rescued .wavs themselves do not open), the right channel has the on-Zoom mic audio, with intermittent silence. The left has the on-Zoom audio, followed by the same part of the XLR input audio. This follows the same pattern as the silences.
I have spent hours chopping up in Garageband, but as it is audio for a video, it needs to match what 'really' happened perfectly (I appreciate for a podcast/audio-only I could relatively simply take away the on-Zoom mic audio from the left channel). I began attempting to sync the mic audio to the on-camera audio (which, despite playing around with settings is as unusable as it always is) but because it's a pattern, can't help but wonder if there's a cleaner fix: either analysing the audio somehow as there are clean lines if I look at the spectral data, or a case of adding a couple of numbers to the wav's binary that'd click the two into place?
I've tried importing to Audacity with different settings, different offsets - this has ended up in either slow audio, fast audio, or heavily distorted audio (but always the same patterns with the files).
I use a Mac (and don't know any PC users close by!) so any software suggestions will need to run on Mac. However, I'm willing to try just about anything that's not dragging tiny clips.
I'm currently working on an application that uses the AudioContext api to control audio for both video clips and background audio. We would like to use the AudioContext (and therefore MediaElementAudioSourceNodes) so we can make adjustments to the audio programmatically.
Because the application is syncing up media to a timeline, this often means adjusting the playbackRate of the media element to catch up. In Chrome, this works fine: you adjust playbackRate and the media speeds up or slows down accordingly. Now in Safari, any audio piped through a MediaElementAudioSourceNode will not respect the changed media playbackRate, playing the audio at normal speed and will then sputter out after a few seconds. (Safari audio will respect the playbackRate when played directly from the media element, notably without pitch correction, but that is a separate, known, issue)
Here's a CodeSandbox that replicates the issue. The first player on the page will play audio back coming directly from the HTMLMediaElement, where as the second player will pipe it through a MediaElementAudioSourceNode.
We've tried a couple other avenues, such as using an AudioBufferSourceNode for the audio source, but due to the size of clip we're often working with, this is not a desired avenue. If at all possible, we would like to still use the AudioContext api for both Chrome and Safari as well.
Ok, so I found out that Blender has this really cool video-editing interface and I was beginning to love it. Until, I created this awesome project composition and when I exported the animation as a video file, the audio was out of sync :(.
Actual Problem
Audio is in-sync with video when the animation is played in Blender but is out-of-sync in the rendered video.
Solutions I tried out and failed
I used the 'Audio-Sync' option in the sequencer but that made no difference.
Then I thought that my scene audio frequency might have been an issue since it was initially 48kHz and my videos were at 24kHz, so I changed the scene audio frequency to 24kHz, this still failed to solve the issue.
Initially, I was combining videos with different frame rates and thought that might have been an issue (although animation played as expected in Blender), so I recreated the source videos to ensure all videos I was using in my project had the same frame rate, but this also did not work.
Someone online suggested exporting the video and audio separately and then combining them using a command-line tool like FFMPEG, this also failed.
What's really frustrating
This lag (audio is a few frames ahead of the video) is noticeable only in longer videos (>12 mins, my video is 1 hr long) suggesting a very small rendered rate difference between the video and the audio.
Also, note that the animation plays absolutely fine in Blender, so all I could figure out was that this was a rendering issue.
So if anyone figured this out please let me know. I am a noob in video/audio codecs so please forgive me if I used some incorrect nomenclature above.
I encountered this issue on OBS capture (a 13 minute clip) with Blender 2.93.3. OBS capture is constant framerate at 60 fps, I did try Handbrake conversion to 60 fps constant framerate also with no help. Workaround to solve the issue is to set Blender rendering fps to 59.94, sequencer shows audio track extending over video track but after render everything matches perfectly. Unfortunately you cannot edit the video in 59.94 fps mode, so you need to switch back to 60 fps for editing.
In case your video is 24 fps then use 23.98 fps preset and for 30 fps you can use the 29.97 fps preset.
May 2021. Blender v2.92.0 - I experienced the same as described out-of-sync problem with rendered videos that were over five minutes long. Source was as-is (3.6GB, 10mins) file from Canon EOS 5DMKII, which is an old camera, so pretty much any software can handle the encoding.
In Blender's preview mode everything looks in-sync. Audio and Video tracks are of the same length. I didn't even cut or merged any segments of the source video. I tried running rendering after a clean boot, gave Blender highest resource priority in Win10, allocated more memory to caching, etc. Source and output was on SSD. Rendered result still didn't match what GUI showed. Very frustrating, and a lot of wasted time.
What worked better for me is the following:
Change Video Codec to "FFmpeg video codec #1". This produces a lossless file that is about 27 times bigger (13.8GB for 10mins) than H.264 codec file (0.5GB). However, the audio remains in sync all the way through.
Use HandBrake open source video transcoder to convert FFmpeg file into H.264 (or H.265). End result produces a smallish-size file with A/V that is in sync.
This workaround is relatively painless and produces good-quality results because there's only a single lossy compression step. The time required to get to final file more than triples though. I believe the issue continues to be with the way H.264 rendering in Blender is implemented. I also experienced similar out-of-sync issues in ShotCut a year ago while working with cheap action cam H.265 files. I also found ShotCut to be less stable than Blender.
So after a lot of online searching, I did find an answer to fix this problem, but not in Blender. If you are like me and would like to use Blender for video editing and still get around the issue, then I found a workaround, but you need Shotcut for this. Shotcut is another great free and open-source video editor
Export the entire long video from Blender (the rendered video has desync issues as expected).
Open the video in Shotcut and detach the audio from it.
Use the audio properties to make very fine adjustments to the audio playback speed to suit your requirements (make fine adjustments until video and audio are in sync).
Follow the GIF attached.
(I am using a shorter video in the GIF but you get the idea)
Explanation
Blender has issues while rendering long videos and I noticed that the video is exported at 1.0x speed but the audio is sometimes faster (1.00400x or something like that) and hence the rendered video has audio not in sync with the video.
Another bad thing is that Blender does not really allow very fine playback speed adjustment just to the audio.
One trick is to adjust the pitch of the audio in Blender which in turn changes the playback speed but this is only allowed up to 2 decimal places (not what we want for long videos) and it makes the audio sound funny (since it actually changes the pitch).
Shotcut is a great tool that allows fine playback adjustment, and it also has a pitch compensation feature so that your pitch is kind of unaffected (since we don't want the characters to be sounding funny in our edited video).
Shotcut allows playback speed adjustment up to 6 decimal places.
I landed at this thread because of the same issue happening in a video that I have just finished. The "View animation CTRL F11" command starts an internal player that has sync issues with long videos. Opening the same video file on "Videos" in Fedora, it plays perfectly synchronized.
I have a bunch of video clips from a webcam (duration is 5, 10, 60 seconds), and I'm looking for a way to detect "does this video clip have movement", to decide whether the file should be saved or discarded in a future processing phase.
I've looked into motion and OpenCV, but motion seems to only want to work on the raw video stream, and OpenCV seems to be way too advanced for my use.
My ideal solution would be a linux command-line tool that I can feed video files into, and get a simple "does/doesn't contain movement" answer back, so I can discard the irrelevant files. False positives (in a reasonable quantity) are perfectly acceptable for my use.
Does such a tool exist? Or any simple examples of doing this with other tools?
You can check dvr-scan which is simple cross-platform command line tool based on OpenCV.
To just list motion events in csv format (scan only):
dvr-scan -i some_video.mp4 -so
To extract motion in single video:
dvr-scan -i some_video.mp4 -o some_video_motion_only.avi
For more examples and various other parameters see:
https://dvr-scan.readthedocs.io/en/latest/guide/examples/
I had the same problem and wrote the solution: https://github.com/jooray/motion-detection
Should be fairly easy to use from command-line.
If you would like to post-process already-captured video then motion can be useful.
VLC allow you to stream or convert your media for use locally, on your private network, or on the Internet. So an already-captured video can be streamed over HTTP, RTSP, etc. and motion can handle it as a network camera.
Furthermore:
How to Stream using VLC Media Player
If OpenCv is to advanced for you, maybe you should consider something easier which is... SimpleCV (wrapper for OpenCV) "This is computer vision made easy". There is even an example of motion detection using SimpleCV - https://github.com/sightmachine/simplecv-examples/blob/master/code/motion-detection.py Unfortunetely i can't test it(because my OpenCv version isn't compatible with SimpleCV), but generally it looks fine (and isn't complicated) - it just substract previous frame from current and calculate mean of the result. If this value is bigger than some threshold (which most likely you will have to adjust) than we can assume that there were some motion between those 2 frames. Note that setting threshold to 0 is really a bad idea, because always there is some difference between 2 consecuitve frames (changes of lighting, noises, etc).
I'm working on a project where we have many small audio files of around 500-600k. Then there are audio files of around 15M.
The 15M files are full narrated articles. The smaller ones are individual sentences within the article.
There are going to be many users and many articles in the future.
I want to be able to load the audio files relatively fast -- either through pre-loading or streaming or something of that nature. Basically if a user clicks on a button -- I want the audio to start more or less immediately.
What are my options here? Red5? Icecast?
EDIT:
I'd like to avoid flash if at all possible but not opposed to it -- I definitely can't use html5 audio as much as I'd like too.
I've already tried doing document onload to issue get requests for the files -- there are usually 15-20 per page. (19 small files, one big one). That doesn't seem to work as well as I thought it might.
In terms of latency -- I'm looking for push-button instant play -- right now I can count to 2 or 3 for the small files and 6-7 for the big one. Flash would be able to do this?
Streaming solutions such as Icecast are not appropriate here. All you need is simple HTTP.
You don't mention what you are playing these things on the client side with. If you are doing this in flash, it is relatively simple to preload or play while the download is still running.
For audio compression, you should be using MP3. For speech, you can easily get away with a lower bitrate. 48kbit 44.1kHz Mono is generally acceptable. This will load fine, even on decent mobile connections.
In any case, HTTP is the way to go. That way you can request the separate files easily. Icecast is for a single stream that runs for awhile, such as internet radio.
ok -- so i did some investigation and figured out what the competition was using
it was this:
http://www.schillmania.com/projects/soundmanager2/
basically what it does is try and use html5 audio tags with the ever so helpful 'preload=true' flag set and if it can't do that it fallsback on flash to preload the mp3