Apparently, the hip way to define a new image format is to combine the ISO base media file format with an existing image or intra-frame video codec, generalized in HEIF / MIAF with existing explicit HEVC, AVC and AV1 adaptations. JPEG 2000 employs ISOBMMF in a different way.
Can I expect platforms (i.e. operating systems, browsers, ...) that support HEIF in general and a random codec in general to support the specific combination of both as well? or does every codec need custom boxes/atoms?
For instance, could I simply put VP8 or VP9 streams, as known from WebM and WebP, into a HEIF container? or more ancient, exotic or futuristic video codecs like MPEG-1/2, ASP, Ogg Theora, Daala, Thor, Dirac, Cineform, VVC, EVC? What about JPEG XL/XR/XT/LS/2K, PNG or even BMP/RLE still images?
Related
So I have an esp32 which captures images and sound. The esp32-camera library already returns the jpeg encoded buffer. The audio however is uncompressed and is just a digital representation of signal strength at high sample rate.
I use esp32 to host a webpage which contains <image> element and a JavaScript snippet, which constantly sends GET requests to a branching url for image data and updates the element. This approach is not very good, especially that now I've added audio capabilities to the circuit.
I'm curious if it would be possible to combine jpeg encoded frames and some audio data into a chunk of h264 and then send it directly as a response to a GET request making it a stream?
This not only would simplify the whole serving multiple webpages thing, but also remove the issues of syncing the audio and video if they are sent separately.
In particular I'm also curious how easy would it be to do on esp32 since it doesn't have a whole bunch of ram and computational power. It would be challenging to find or port large libraries which could help as well, so i guess I would have to code it myself.
I also am not sure if h264 is the best option. I know its supported on most browser out of the box and is using jpeg compression behind the scenes for the frames, but perhaps a simpler format exists which is also widely supported.
So to sum it up: Is h264 a best bet in the provided context? Is combining jpeg and uncompressed mono audio into h264 possible in the provided context? If an answer to either of previous questions is a no, what alternatives do i have if any?
I'm curious if it would be possible to combine jpeg encoded frames and some audio data into a chunk of h264 and then send it directly as a response to a GET request making it a stream?
H.264 is a video codec. It doesn't have anything to do with audio.
I know its supported on most browser out of the box and is using jpeg compression behind the scenes for the frames
No, this isn't true. H.264 is its own thing. It's far more powerful than JPEG and is specifically designed for motion, whereas JPEG was not.
You need a few things:
A video codec, to efficiently handle your frames. Most of these embedded camera libraries can give you an MJPEG stream. I'd use that if possible. I don't think your ESP32 has other video encoding capability, does it? H.264 is a good choice, but only if you can actually encode it.
A container format, to aid in streaming your audio and video streams together. ISOBMFF/MP4 is common, as is WebM/Matroska.
If you're only streaming to a single client (which seems likely given the limited horsepower of the board), and if you have enough capability to do the audio/video encoding, you can generate a WebM stream on the fly that is directly playable in a <video> element. This seems exactly what you are asking for.
I'm trying to understand what does impose support of certain media file formats by different browsers.
Why there is no specifications for file format (e.g. HLS or DASH) in Media Source Extension? And if file format is not specified by MSE, then which standart specifies it? Is it up just to browser implementation?
HLS and DASH are not file formats, they are adaptive streaming standards. They can use various containers and codecs.
For MSE there's a registry of byte stream formats here:
https://www.w3.org/TR/mse-byte-stream-format-registry/
At the time of writing they are WebM, MP4, MPEG-2 TS and MPEG/AAC audio-only.
I have an audio file of File type - WAVE (.WAV), Mime Type - audio/wav
Codec - 0x0135 (Sipro Lab KELVIN)
Is it possible to convert this file to Mp3? If so, can you please provide pointers. Also, I'm not able to play this wav file in vlc player. Specific codec needs to be installed?
Short answer: Looks difficult unless you can find a codec for Sipro Lab KELVIN format.
Long Answer:
Most players rely on a system codec to decode the audio. So if you install the Sipro Lab Kelvin codec, then you will be able to play the audio in players, like Windows Media Player, that use underlying system codecs. If you can get such a codec, there is a complicated way to convert from any format to the MP3 format.
VLC, on the other hand, does not use codecs installed in the system. Based on VLC audio support page, VLC does not support codec type 0x135.
The other powerful codec tool ffmpeg also does not seem to support Sipro Lab Kelvin audio as per FFMPEG audio codec support page.
I am looking for a comprehensive API in Java that can convert audio across various formats and bitrates.
For example
WAV (6kHz to 48kHz) L16/audio ---TO--- WAV (RIFF header) 8KHz 8-bit mono A-law/U-law
AIFF (6kHz to 48kHz) L16/audio ---TO--- WAV (RIFF header) 8KHz 8-bit mono A-law/U-law
and other voice audio formats.
Any other suggestions about similar Java libraries on audio conversion are also entertained.
I was able to solve this problem by using Tritonus : Open Source Java Sound API and its wide range of sound convertor plugins.
Specifically the Tritonus miscellaneous plugins was very useful in my context.
I'm able to play .mp4, .m4v formats using MPMoviePlayerController, but its not playing .avi format. MPMoviePlayerController internally would use QuickTimePlayer and this QuicktimePlayer would support AVI format, then y not MPMoviePLayerController API?
Thank You,
Suse
AVI is a container format. It can contain many different audio/video formats internally, most of which iOS does not support natively.
You may be able to decode the contents of your .AVI using the ffmpeg library. Note, however, that decoding video using ffmpeg will drain the battery considerably faster than normal, as it cannot take advantage of the hardware acceleration that MPEG4 decoding uses, and using ffmpeg may subject your application to software licensing complications (as ffmpeg is licensed under LGPL, with some GPL and non-free components).