Crossdevice encoding static file to stream in browser using FFMPEG (segmented h264 ?) - node.js

Question

I'm building a mediacenter application in NodeJS which is going pretty ok.
(you can check it out on Github: https://github.com/jansmolders86/mediacenterjs )
I'm using FFMPEG to transcode local (static) movies to a stream which I then send to the browser.
At first I used h264 with Flash which worked in browsers, but I really need it to work on Android an iOS (so no Flash) and preferably working on a Raspberry Pi.
But getting it to play on all devices is driving me absolutely insane!
I have all these bits of the puzzle I've gathered from countless hours reading articles, tutorials and stack overflow posts, which led me to the conclusion that I need to produce the following:
Use video codec H264 to transcode to MP4
Move the moovatom '-movflags' to make a MP4 streamable
Segment the stream so Apple can play the stream as well.
But getting nowhere with this. Every time I produce a series of FFMPEG settings that either don't work, or work on some devices rather than all.
Some of my failed attempt where:
My flash attempt -> Main problem (not running in IOS):
'-y','-ss 0','-b 800k','-vcodec libx264','-acodec mp3'\
'-ab 128','-ar 44100','-bufsize 62000', '-maxrate 620k'\
metaDuration,tDuration,'-f flv
my HLS attempt -> Main problem (not running in browser):
'-r 15','-b:v 128k','-c:v libx264','-x264opts level=41'\
'-threads 4','-s 640x480','-map 0:v','-map 0:a:0','-c:a mp3'\
'-b:a 160000','-ac 2','-f hls','-hls_time 10','-hls_list_size 6'\
'-hls_wrap 18','-start_number 1'
My MP4 attempt -> Main problem (duration is shortened and the later part of the video is speeding by)
'-y','-ss 0','-b 800k','-vcodec libx264','-acodec mp3'\
'-ab 128','-ar 44100','-bufsize 62000', '-maxrate 620k'\
metaDuration,tDuration,'-f mp4','-movflags','frag_keyframe+empty_moov'
Second MP4 attempt: -> Main problem (duration is shortened and the later part of the video is speeding by)
'-y','-vcodec libx264','-pix_fmt yuv420p','-b 1200k','-flags +loop+mv4'\
'-cmp 256','-partitions +parti4x4+parti8x8+partp4x4+partp8x8+partb8x8'\
'-me_method hex','-subq 7','-trellis 1','-refs 5','-bf 3','-coder 1'\
'-me_range 16','-g 150','-keyint_min 25','-sc_threshold 40'\
'-i_qfactor 0.71','-acodec mp3','-qmin 10','-qdiff 4','-qmax 51'\
'-ab 128k','-ar 44100','-threads 2','-f mp4','-movflags','frag_keyframe+empty_moov'])
Here is an example of the FFMPEG log running with these settings:
file conversion error ffmpeg version N-52458-gaa96439 Copyright (c) 2000-2013 the FFmpeg developers
built on Apr 24 2013 22:19:32 with gcc 4.8.0 (GCC)
configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --e
nable-frei0r --enable-gnutls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable
-libgsm --enable-libilbc --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --ena
ble-libopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwola
me --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libxavs --enabl
e-libxvid --enable-zlib
libavutil 52. 27.101 / 52. 27.101
libavcodec 55. 6.100 / 55. 6.100
libavformat 55. 3.100 / 55. 3.100
libavdevice 55. 0.100 / 55. 0.100
libavfilter 3. 60.101 / 3. 60.101
libswscale 2. 2.100 / 2. 2.100
libswresample 0. 17.102 / 0. 17.102
libpostproc 52. 3.100 / 52. 3.100
[avi # 02427900] non-interleaved AVI
Guessed Channel Layout for Input Stream #0.1 : mono
Input #0, avi, from 'C:/temp/the avengers.avi':
Duration: 00:00:34.00, start: 0.000000, bitrate: 1433 kb/s
Stream #0:0: Video: cinepak (cvid / 0x64697663), rgb24, 320x240, 15 tbr, 15 tbn, 15 tbc
Stream #0:1: Audio: pcm_u8 ([1][0][0][0] / 0x0001), 22050 Hz, mono, u8, 176 kb/s
Please use -b:a or -b:v, -b is ambiguous
[libx264 # 02527c60] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 AVX
[libx264 # 02527c60] profile High, level 2.0
[libx264 # 02527c60] 264 - core 130 r2274 c832fe9 - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x26
4.html - options: cabac=1 ref=5 deblock=1:0:0 analyse=0x3:0x133 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16
chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=2 lookahead_threads=1 sliced_th
reads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=1 b_bias=0 direct=1 we
ightb=1 open_gop=0 weightp=2 keyint=150 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=40 rc=abr mbtree=1 bitrate=120
0 ratetol=1.0 qcomp=0.60 qpmin=10 qpmax=51 qpstep=4 ip_ratio=1.41 aq=1:1.00
Output #0, mp4, to 'pipe:1':
Metadata:
encoder : Lavf55.3.100
Stream #0:0: Video: h264 ([33][0][0][0] / 0x0021), yuv420p, 320x240, q=10-51, 1200 kb/s, 15360 tbn, 15 tbc
Stream #0:1: Audio: mp3 (i[0][0][0] / 0x0069), 44100 Hz, mono, s16p, 128 kb/s
Stream mapping:
Stream #0:0 -> #0:0 (cinepak -> libx264)
Stream #0:1 -> #0:1 (pcm_u8 -> libmp3lame)
Press [q] to stop, [?] for help
frame= 106 fps=0.0 q=10.0 size= 1kB time=00:00:06.94 bitrate= 1.4kbits/s
frame= 150 fps=149 q=14.0 size= 1kB time=00:00:09.87 bitrate= 1.0kbits/s
frame= 191 fps=126 q=16.0 size= 1kB time=00:00:12.61 bitrate= 0.8kbits/s
frame= 244 fps=121 q=16.0 size= 2262kB time=00:00:16.14 bitrate=1147.6kbits/s
frame= 303 fps=120 q=14.0 size= 2262kB time=00:00:20.08 bitrate= 922.2kbits/s
frame= 354 fps=117 q=15.0 size= 3035kB time=00:00:23.48 bitrate=1058.6kbits/s
frame= 402 fps=113 q=15.0 size= 3035kB time=00:00:26.67 bitrate= 932.1kbits/s
frame= 459 fps=113 q=16.0 size= 4041kB time=00:00:30.43 bitrate=1087.7kbits/s
frame= 510 fps=103 q=2686559.0 Lsize= 5755kB time=00:00:33.93 bitrate=1389.3kbits/s
video:5211kB audio:531kB subtitle:0 global headers:0kB muxing overhead 0.235111%
[libx264 # 02527c60] frame I:6 Avg QP:10.55 size: 25921
[libx264 # 02527c60] frame P:245 Avg QP:12.15 size: 14543
[libx264 # 02527c60] frame B:259 Avg QP:15.55 size: 6242
[libx264 # 02527c60] consecutive B-frames: 6.1% 73.7% 14.7% 5.5%
[libx264 # 02527c60] mb I I16..4: 19.9% 6.2% 73.9%
[libx264 # 02527c60] mb P I16..4: 6.0% 0.2% 12.0% P16..4: 35.4% 9.6% 16.3% 7.0% 5.6% skip: 7.8%
[libx264 # 02527c60] mb B I16..4: 0.7% 0.0% 4.3% B16..8: 27.6% 17.2% 17.0% direct:17.3% skip:15.9% L0:39.4% L1:43.2%
BI:17.4%
[libx264 # 02527c60] final ratefactor: 11.41
[libx264 # 02527c60] 8x8 transform intra:1.6% inter:4.0%
[libx264 # 02527c60] coded y,uvDC,uvAC intra: 93.0% 97.0% 94.9% inter: 58.4% 58.7% 50.6%
[libx264 # 02527c60] i16 v,h,dc,p: 15% 26% 54% 5%
[libx264 # 02527c60] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 16% 17% 39% 4% 4% 3% 1% 6% 9%
[libx264 # 02527c60] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 28% 34% 21% 4% 2% 2% 2% 2% 5%
[libx264 # 02527c60] i8c dc,h,v,p: 51% 24% 19% 6%
[libx264 # 02527c60] Weighted P-Frames: Y:4.1% UV:1.2%
[libx264 # 02527c60] ref P L0: 68.2% 9.8% 11.0% 5.6% 4.6% 0.8% 0.0%
[libx264 # 02527c60] ref B L0: 87.7% 8.0% 3.9% 0.4%
[libx264 # 02527c60] ref B L1: 97.8% 2.2%
[libx264 # 02527c60] kb/s:1255.36
Lastly this is my node code fireing up FFMPEG. (I use the module Fluent-ffmpeg: https://github.com/schaermu/node-fluent-ffmpeg )
var proc = new ffmpeg({ source: movie, nolog: true, timeout:15000})
.addOptions(['-r 15','-b:v 128k','-c:v libx264','-x264opts level=41','-threads 4','-s 640x480','-map 0:v','-map 0:a:0','-c:a mp3','-b:a 160000','-ac 2','-f hls','-hls_time 10','-hls_list_size 6','-hls_wrap 18','-start_number 1 stream.m3u8'])
.writeToStream(res, function(retcode, error){
if (!error){
console.log('file has been converted succesfully',retcode .green);
}else{
console.log('file conversion error',error .red);
}
});
So to conclude this very long and code heavy question:
I hope this does not come off as a lazy request, but could someone show/explain to me which FFMPEG settings could/should work on all platforms (modern browsers, Android and iOS) producing a stream of a static file which I can send to a HTML5 player.
[EDIT] what I need if a generic option isn't available
And if this is not possible as some posts might suggest, I would love to see a set of FFMPEG settings that get's the job done properly as far as mp4 streaming is concerned. (e.g encoding a streamable mp4).
The streaming mp4 needs the following
A shifted moovAtom
It needs to be h264
Thanks very much for your help!

Answer 1

There is no format that can play on every device and browser. HTML5 is getting us closer, but there is still debate on formats and codecs. My friends at Zencoder have a new blog post blog post (HERE) that addresses this exact issue.
EDIT: you asked for more specifics. Again, it depends on what platforms you wish to target. I will cover a couple here.
This should play on all modern browsers that support the h.264 codec. It should also play on iPhone4 and above:
ffmpeg -i ~/Dropbox/Test\ Content/bigbuckbunny/bigbuckbunny_1500.mp4 -vcodec libx264 -profile:v main -b:v 512k -s 1280x720 -r:v 30 -acodec libfdk_aac -b:a 128k -movflags faststart -y movie1.mp4
The iPhone 3gs does not support main profile, and its max supported resolution is 640x480. This command will encode for this older device.
ffmpeg -i ~/Dropbox/Test\ Content/bigbuckbunny/bigbuckbunny_1500.mp4 -vcodec libx264 -profile:v baseline -b:v 512k -s 640x432 -r:v 30 -acodec libfdk_aac -b:a 128k -movflags faststart -y movie2.mp4
I encoded some sample files and created a web page here:
http://szatmary.org/stackoverflow/18758133/
The HTML looks like this:
<!DOCTYPE html>
<html>
<body>
<br>ffmpeg -i ~/Dropbox/Test\ Content/bigbuckbunny/bigbuckbunny_1500.mp4 -vcodec libx264 -profile:v main -b:v 512k -s 1280x720 -r:v 30 -acodec libfdk_aac -b:a 128k -movflags faststart -y movie1.mp4<br>
<video controls>
<source src="movie1.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
<br>ffmpeg -i ~/Dropbox/Test\ Content/bigbuckbunny/bigbuckbunny_1500.mp4 -vcodec libx264 -profile:v baseline -b:v 512k -s 640x432 -r:v 30 -acodec libfdk_aac -b:a 128k -movflags faststart -y movie2.mp4<br>
<video controls>
<source src="movie2.mp4" type="video/mp4">
Your browser does not support the video tag.
</video>
</body>
</html>
Here the command broken down into what each element means:
command + input file (should be obvious):
ffmpeg -i ~/Dropbox/Test\ Content/bigbuckbunny/bigbuckbunny_1500.mp4
Use libx264 to encode the video:
-vcodec libx264
Set the h.264 profile to main. baseline will allow for playback on older devices, but you will sacrifice a little quality:
-profile:v main
Set the bitrate to 512 kilobits per second. Choose a value based on the available bandwidth. Higher for LAN/WiFi, lower for 3G/LTE
-b:v 512k
Scale the video to 720p resolution (Again depends on target platform)
-s 1280x720
Encode at 30 frames per second:
-r:v 30
Use libfdk_aac to encode the audio. Or use libmp3lame if you want mp3. AAC is highly recommended. It has much better support on ios and produces higher quality audio:
-acodec libfdk_aac
Set audio bitrate to 128 kilobits per second. You may adjust this for bandwidth as well. with AAC you can probably go as low as 32k
-b:a 128k
Set audio sampling rate to 48000 sample per second. If using mp3, do 44100 for ios
-r:a 48000
This tells ffmpeg to place the moov atom at be start of the mp4 file.
-movflags faststart
Output file (-y tells ffmpeg it can overwrite the file without asking)
-y movie1.mp4

Answer 2

As far as I know you won't find a setting which works on every device. I'd recommend you to check the user agent and then use different settings for different devices. This way you could also use device optimized settings.

Answer 3

I'm thinking about 2 workarounds to enable seeking. I'm assuming that, like me, you don't want to store files transcoded in advance.
The first one is just faking seek from the browser. Use a custom timeline control and when seeking change the video src to an URL including the desired start time, and pass it to ffmpeg. Of course this completely defeats browser prefetching. I implemented this in my project and it works fine.
The second option is a bit more technical and I'm not sure how to do it. The idea would be to transcode all available videos in advance, extract the moov atom, save it to disk and write it manually when streaming. This seems quite hard to do but not impossible.