I have around 15,000 music files stored on Ubuntu server (16.04), around 50% FLAC, 25% each mp3 and m4a (aac).
I think maybe 3-5% are corrupted due to HDD hardware failure. The problems accumulated gradually for some time before I noticed. Files are now recovered to new drives using ddrescue.
Original storage was two copies of each file on separate devices, and both drives gradually failed, but independently. Result is that a file which is bad in one copy may be ok in the other copy.
I am trying to find command line validation tools to use in a script to identify which titles have at least one good copy. In cases where both are bad I will need to re-rip from cd.
For FLAC, I have looped the command flac -t in a script which generates lists of good files and the bad files. I believe the flac -t command decodes without sending audio to any play device, and calculates an MD5 hash on the decoded audio and compares this to an original hash included in the file’s metadata. This is pretty fast and works fine.
I would like to achieve similar validation with the mp3 and the m4a files, but have not been able to find a suitable tool. I have looked at mp3val, but testing this against an mp3 where I deliberately damaged data in the audio does not show an error.
From what I can find researching mp3 and m4a it seems there is no hash stored, so I am not sure what other approaches to validation might be possible.
Ideally I would like to sort into definitely good / definitely bad. If this can't be done, I would still benefit from sorting into possibly good / definitely bad, or definitely good / possibly bad.
Can anyone suggest some linux tool that could achieve this, for either/both of mp3 and m4a/aac ?
Related
The common situation when the integrity of an MP3 file is not correct, is when the file has been partially uploaded to the server. In this case, the indicated audio duration doesn't correspond to what is really in the MP3 file: we can hear the beginning, but at some point the playing stops and the indicated duration of the audio player is broken.
I tried with libraries like node-ffprobe, but it seems they just read metadata, without making comparison with real audio data in the file. Is there a way to detect efficiently a corrupted or incomplete MP3 file from node.js?
Note: the client uploading MP3 files is a hardware (an audio recorder), uploading files on a FTP server. Not a browser. So I'm not able to upload potentially more useful data from the client.
MP3 files don't normally have a duration. They're just a series of MPEG frames. Sometimes, there is an ID3 tag indicating duration, but not always.
Players can determine duration by choosing one of a few methods:
Decode the entire audio file.This is the slowest method, but if you're going to decode the file anyway, you might as well go this route as it gives you an exact duration.
Read the whole file, skimming through frame headers.You'll have to read the whole file from disk, but you won't have to decode it. Can be slow if I/O is slow, but gives you an exact duration.
Read the first frame's bitrate and estimate duration by file size.Definitely the fastest method, and the one most commonly used by players. Duration is an estimate only, and is reasonably accurate for CBR, but can be wildly inaccurate for VBR.
What I'm getting at is that these files might not actually be broken. They might just be VBR files that your player doesn't know the duration of.
If you're convinced they are broken (such as stopping in the middle of content), then you'll have to figure out how you want to handle it. There are probably only a couple ways to determine this:
Ideally, there's an ID3 tag indicating duration, and you can decode the whole file and determine its real duration to compare.
Usually, that ID3 tag won't exist, so you'll have to check to see if the last frame is complete or not.
Beyond that, you don't really have a good way of knowing if the stream is incomplete, since there is no outer container that actually specifies number of frames to expect.
The expression for calculating the filesize of an mp3 based on duration and encoding (from this answer) is quite simple:
x = length of song in seconds
y = bitrate in kilobits per second
(x * y) / 1024 = filesize (MB)
There is also a javascript implementation for the Web Audio API in another answer on that same question. Perhaps that would be useful in your Node implementation.
mp3diags is some older open source software for fixing mp3s and which was great for batch processing stuff like this. The source is c++ and still available if you're feeling nosy and want to see how some of these features are implemented.
Worth a look since it has some features that might be be useful in your context:
What is MP3 Diags and what does it do?
low quality audio
missing VBR header
missing normalization data
Correcting files that show incorrect song duration
Correcting files in which the player cannot seek correctly
I have an audio streaming application that uses requests to download the audio file and then played using Gstreamer.
I want to trim the first few seconds of all the audio files that i have. I could use ffmpeg to trim but that would waste cpu resources on my embedded platform and also waste network bandwidth
(The number of songs are around 1000, and they get downloaded continously, so it does make a difference)
I have tried downloading partial file using the range header in requests but that doesn't work. I can't play the file.
Can someone please tell me how i can make this work?
The audio files are generally .m4a / .webm but they are extracted from youtube so can't say for sure.
This is an uneasy task.. there is no clean way how to do it..
you can probably use the valve element set it to drop by default..
and then put some timer which sets the drop to false..
not sure how this will work, you need to try.
Here are some hints:
I've written a script to "normalize" all my FLAC files by stripping unneeded tags, padding tracknumber/discnumber, removing pictures, etc. As part of the normalization process, my script re-compresses the FLAC file to level 8. Since re-compressing an already level-8 FLAC is pointless and time consuming, I want a way to know if the audio of the FLAC file has been changed since my last compression (I don't want to use file modification time because changing the metadata would change this as well). Is there an easy way to get the MD5 hash or something of the FLAC audio section so I can quickly check if it's been altered? Thanks!
I ended up using the python-audio-tools over at http://audiotools.sourceforge.net/. Here's the relevant code, for future reference:
track = audiotools.open('file.flac')
metadata = track.get_metadata()
raw_hash = metadata.get_block(audiotools.flac.Flac_STREAMINFO.BLOCK_ID).md5sum
print(audiotools.hex_string(raw_hash))
I would like to save a quiet audio file with more volume. Can you suggest me a program, method to do it?
The device I would like to use the audio file on is not very intelligent and the maximum volume setting is too quiet for me. Other audio files (that are louder) can be played fine on the device. So I thought, I open the file on PC, modify it to be louder, save it, then the device will play this fine too. I am aware of distortions and such, that is not the point now.
I have used VLC player and I can make a setting where the audio file is loud enough with little distortions, but I can not find the options to save the file with these settings. It is an MP3 file.
Thanks for the help,
Sziro
Increasing the volume of an audio file so that the peaks are at (or near) the maximum level is called normalisation. You can use an audio editior like audacity or there are dedicated solutions. Normally if saving to mp3 you normalise to slightly less than full volume (say -0.5 dB).
You might also want to consider compressing the audio. This will be useful if the peaks in your audio are much louder than the quiet passages, and the quiet passages are hard to hear as a result. Again, you can do this in audacity.
My code using NAudio to read one particular MP3 gets different results than several other commercial apps.
Specifically: My NAudio-based code finds ~1.4 sec of silence at the beginning of this MP3 before "audible audio" (a drum pickup) starts, whereas other apps (Windows Media Player, RealPlayer, WavePad) show ~2.5 sec of silence before that same drum pickup.
The particular MP3 is "Like A Rolling Stone" downloaded from Amazon.com. Tested several other MP3s and none show any similar difference between my code and other apps. Most MP3s don't start with such a long silence so I suspect that's the source of the difference.
Debugging problems:
I can't actually find a way to even prove that the other apps are right and NAudio/me is wrong, i.e. to compare block-by-block my code's results to a "known good reference implementation"; therefore I can't even precisely define the "error" I need to debug.
Since my code reads thousands of samples during those 1.4 sec with no obvious errors, I can't think how to narrow down where/when in the input stream to look for a bug.
The heart of the NAudio code is a P/Invoke call to acmStreamConvert(), which is a Windows "black box" call which I can't think how to error-check.
Can anyone think of any tricks/techniques to debug this?
The NAudio ACM code was never originally intended for MP3s, but for decoding constant bit rate telephony codecs. One day I tried setting up the WaveFormat to specify MP3 as an experiment, and what came out sounded good enough. However, I have always felt a bit nervous about decoding MP3s (especially VBR) with ACM (e.g. what comes out if ID3 tags or album art get passed in - could that account for extra silence?), and I've never been 100% convinced that NAudio does it right - there is very little documentation on how exactly you are supposed to use the ACM codecs. Sadly there is no managed MP3 decoder with a license I can use in NAudio, so ACM remains the only option for the time being.
I'm not sure what approach other media players take to playing back MP3, but I suspect many of them have their own built-in MP3 decoders, rather than relying on the operating system.
I've found some partial answers to my own Q:
Since my problem boils down to consuming too much MP3 w/o producing enough PCM, I used conditional-on-hit-count breakpoints to find just where this was happening, then drilled into that.
This showed me that some acmStreamConvert() calls are returning success, consuming 417 src bytes, but producing 0 "dest bytes used".
Next I plan to try acmStreamSize() to ask the codec how many src bytes it "wants" to consume, rather than "telling" it to consume 417.
Edit (followup): I fixed it!
It came down to passing acmStreamConvert() enough src bytes to make it happy. Giving it its acmStreamSize() requested size fixed the problem in some places but then it popped up in others; giving it its requested size times 3 seems to cure the "0 dest bytes used" result in all MP3s I've tested.
With this fix, acmStreamConvert() then sometimes returned much larger converted chunks (almost 32 KB), so I also had to modify some other NAudio code to pass in larger destination buffers to hold the results.