I am trying to do some wav processing using Lua, but have fallen a the first hurdle! I cannot find a function or library that will allow me to load a wav file and access the raw data. There is one library, but it onl allows playing of wavs, not access to the raw data.
Are there any out there?
Cheers,
Pete.
I don't think Lua is the right tool for raw audio data processing, mainly because Lua uses only a single numeric data type - doubles. Also, Lua cannot directly access the elements of a data stream, although you can use something like the struct library ( http://www.inf.puc-rio.br/~roberto/struct/ )
A better way to process the data would be to write the filters in C, with binding for Lua, and then use Lua for higher level processing, like (imaginary toolkit):
require 'wave'
-- load the wave
wav = wave.load('file.wav', 's16')
-- apply some filters
thresh = wave.threshold(wav, 0.5)
faded = wave.fadeout(thresh, 5)
Alternatively, you can load the data and view it like this. The ascii column will show you the the WAV header, which is stored in the first 44 bytes...
local f = assert(io.open(path, "rb"))
-- read in 16 bytes at a time
local block = 16
while true do
local bytes = f:read(block)
if not bytes then break end
for _, b in pairs{string.byte(bytes, 1, -1)} do
io.write(string.format("%02X ", b))
end
io.write(string.rep(" ", block - string.len(bytes)))
io.write(" ", string.gsub(bytes, "%c", "."), "\n")
end
Related
I've started taking a look at Nim for hobby game modding purposes.
Intro
Yet, I found it difficult to work with Nim compared to C when it comes to machine-specific low-level memory layout and would like to know if Nim actually has better support here.
I need to control byte order and be able to de/serialize arbitrary Plain-Old-Datatype objects to binary custom file formats. I didn't directly find a Nim library which allows flexible storage options like representing enum and pointers with Big-Endian 32-bit. Or maybe I just don't know how to use the feature.
std/marshal : just JSON, i.e. no efficient, flexible nor binary format but cross-compatible
nim-serialization : seems like being made for human readable formats
nesm : flexible cross-compatibility? (It has some options and has a good interface)
flatty : no flexible cross-compatibility, no byte order?
msgpack4nim : no flexible cross-compatibility, byte order?
bingo : ?
Flexible cross-compatibility means, it must be able to de/serialize fields independently of Nim's ABI but with customization options.
Maybe "Kaitai Struct" is more what I look for, a file parser with experimental Nim support.
TL;DR
As a workaround for a serialization library I tried myself at a recursive "member fields reverser" that makes use of std/endians which is almost sufficient.
But I didn't succeed with implementing byte reversal of arbitrarily long objects in Nim. Not practically relevant but I still wonder if Nim has a solution.
I found reverse() and reversed() from std/algorithm but I need a byte array to reverse it and turn it back into the original object type. In C++ there would be reinterprete_cast, in C there is void*-cast, in D there is a void[] cast (D allows defining array slices from pointers) but I couldn't get it working with Nim.
I tried cast[ptr array[value.sizeof, byte]](unsafeAddr value)[] but I can't assign it to a new variable. Maybe there was a different problem.
How to "byte reverse" arbitrary long Plain-Old-Datatype objects?
How to serialize to binary files with byte order, member field size, pointer as file "offset - start offset"? Are there bitfield options in Nim?
It is indeed possible to use algorithm.reverse and the appropriate cast invocation to reverse bytes in-place:
import std/[algorithm,strutils,strformat]
type
LittleEnd{.packed.} = object
a: int8
b: int16
c: int32
BigEnd{.packed.} = object
c: int32
b: int16
a: int8
## just so we can see what's going on:
proc `$`(b: LittleEnd):string = &"(a:0x{b.a.toHex}, b:0x{b.b.toHex}, c:0x{b.c.toHex})"
proc `$`(l:BigEnd):string = &"(c:0x{l.c.toHex}, b:0x{l.b.toHex}, a:0x{l.a.toHex})"
var lit = LittleEnd(a: 0x12, b:0x3456, c: 0x789a_bcde)
echo lit # (a:0x12, b:0x3456, c:0x789ABCDE)
var big:BigEnd
copyMem(big.addr,lit.addr,sizeof(lit))
# here's the reinterpret_cast you were looking for:
cast[var array[sizeof(big),byte]](big.addr).reverse
echo big # (c:0xDEBC9A78, b:0x5634, a:0x12)
for C-style bitfields there is also the {.bitsize.} pragma
but using it causes Nim to lose sizeof information, and of course bitfields wont be reversed within bytes
import std/[algorithm,strutils,strformat]
type
LittleNib{.packed.} = object
a{.bitsize: 4}: int8
b{.bitsize: 12}: int16
c{.bitsize: 20}: int32
d{.bitsize: 28}: int32
BigNib{.packed.} = object
d{.bitsize: 28}: int32
c{.bitsize: 20}: int32
b{.bitsize: 12}: int16
a{.bitsize: 4}: int8
const nibsize = 8
proc `$`(b: LittleNib):string = &"(a:0x{b.a.toHex(1)}, b:0x{b.b.toHex(3)}, c:0x{b.c.toHex(5)}, d:0x{b.d.toHex(7)})"
proc `$`(l:BigNib):string = &"(d:0x{l.d.toHex(7)}, c:0x{l.c.toHex(5)}, b:0x{l.b.toHex(3)}, a:0x{l.a.toHex(1)})"
var lit = LitNib(a: 0x1,b:0x234, c:0x56789, d: 0x0abcdef)
echo lit # (a:0x1, b:0x234, c:0x56789, d:0x0ABCDEF)
var big:BigNib
copyMem(big.addr,lit.addr,nibsize)
cast[var array[nibsize,byte]](big.addr).reverse
echo big # (d:0x5DEBC0A, c:0x8967F, b:0x123, a:0x4)
It's less than optimal to copy the bytes over, then rearrange them with reverse, anyway, so you might just want to copy the bytes over in a loop. Here's a proc that can swap the endianness of any object, (including ones for which sizeof is not known at compiletime):
template asBytes[T](x:var T):ptr UncheckedArray[byte] =
cast[ptr UncheckedArray[byte]](x.addr)
proc swapEndian[T,U](src:var T,dst:var U) =
assert sizeof(src) == sizeof(dst)
let len = sizeof(src)
for i in 0..<len:
dst.asBytes[len - i - 1] = src.asBytes[i]
Bit fields are supported in Nim as a set of enums:
type
MyFlag* {.size: sizeof(cint).} = enum
A
B
C
D
MyFlags = set[MyFlag]
proc toNum(f: MyFlags): int = cast[cint](f)
proc toFlags(v: int): MyFlags = cast[MyFlags](v)
assert toNum({}) == 0
assert toNum({A}) == 1
assert toNum({D}) == 8
assert toNum({A, C}) == 5
assert toFlags(0) == {}
assert toFlags(7) == {A, B, C}
For arbitrary bit operations you have the bitops module, and for endianness conversions you have the endians module. But you already know about the endians module, so it's not clear what problem you are trying to solve with the so called byte reversal. Usually you have an integer, so you first convert the integer to byte endian format, for instance, then save that. And when you read back, convert from byte endian format and you have the int. The endianness procs should be dealing with reversal or not of bytes, so why do you need to do one yourself? In any case, you can follow the source hyperlink of the documentation and see how the endian procs are implemented. This can give you an idea of how to cast values in case you need to do some yourself.
Since you know C maybe the last resort would be to write a few serialization functions and call them from Nim, or directly embed them using the emit pragma. However this looks like the least cross platform and pain free option.
Can't answer anything about generic data structure serialization libraries. I stray away from them because they tend to require hand holding imposing certain limitations on your code and depending on the feature set, a simple refactoring (changing field order in your POD) may destroy the binary compatibility of the generated output without you noticing it until runtime. So you end up spending additional time writing unit tests to verify that the black box you brought in to save you some time behaves as you want (and keeps doing so across refactorings and version upgrades!).
I made a record called automaton containing the 5 fields required to represent an automaton I'm supposed to use graphics to represent every state and transition from the transitions list without using for or while loops only recursive functions
transitions :(int*char*int) list;
It might be easiest to use graphviz to do the actual drawing. It automatically draws a graph from a list of nodes and edges, which is exactly what your input is. The function fmt_transition generates one edge, the function fmt_transitions lifts it via pp_print_list to lists of transitions and wraps it into the correct header.
let fmt_transition fmt (inedge,by,outedge) =
Format.fprintf fmt "#[%d -> %d [label=\"%c\"];#]" inedge outedge by
let fmt_transitions fmt =
Format.fprintf fmt "#[<v 2>digraph output {#,%a#,#]}#,#."
(Format.pp_print_list fmt_transition)
If we try it on some test data, we get the test file we wanted:
let test1 = [ (1,'a',2); (2,'b',1); (1,'b',1)] in
fmt_transitions Format.std_formatter test1;;
results in:
digraph output
{
1 -> 2 [label="a"];
2 -> 1 [label="b"];
1 -> 1 [label="b"];
}
The call to graphviz is:
dot -T pdf -o output.pdf < input.dot
If you call it directly from ocaml, you can also pipe the data directly into dot without writing it to disk first.
Alternatively, you can also use an external viewer like gv:
let call_dot data =
let cmd = "dot -Tps | gv -" in
let (sout, sin, serr) as channels =
Unix.open_process_full cmd (Unix.environment ()) in
let fmt = Format.formatter_of_out_channel sin in
Format.fprintf fmt "%a#." fmt_transitions data;
channels
let cleanup channels =
(* missing: flush channels, empty buffers *)
Unix.close_process_full channels
The call call_dot test1 |> cleanup will send the data to dot and open ghostview. To prevent hitting the limit of open files, one shout await the termination of the process and close the channels, which is done in cleanup.
If you want to use an image library instead, you might want to replace the definition of cmd by "dot -Tpng" and read the image data from stdin before you call cleanup.
I wanted to know what the best way to write a large fortran array ( 5000 x 5000 real single precision numbers) to a file. I am trying to save the results of a numerical calculation for later use so they do not need to be repeated. From calculation 5000 x 5000 x 4bytes per number number is 100 Mb, is it possible to save this in a form that is only 100Mb? Is there a way to save fortran arrays as a binary file and read it back in for later use?
I've noticed that saving numbers to a text file produces a file much larger than the size of the data type being saved. Is this because the numbers are being saved as characters?
The only way I am familiar with to write to file is
open (unit=41, file='outfile.txt')
do i=1,len
do j=1,len
write(41,*) Array(i,j)
end do
end do
Although I'd imagine there is a better way to do it. If anyone could point me to some resources or examples to approve my ability to write and read larger files efficiently (in terms of memory) that would be great.
Thanks!
Write data files in binary, unless you're going to actually be reading the output - and you're not going to be reading a 2.5 million-element array.
The reasons for using binary are threefold, in decreasing importance:
Accuracy
Performance
Data size
Accuracy concerns may be the most obvious. When you are converting a (binary) floating point number to a string representation of the decimal number, you are inevitably going to truncate at some point. That's ok if you are sure that when you read the text value back into a floating point value, you are certainly going to get the same value; but that is actually a subtle question and requires choosing your format carefully. Using default formatting, various compilers perform this task with varying degrees of quality. This blog post, written from the point of view of a games programmer, does a good job of covering the issues.
Let's consider a little program which, for a variety of formats, writes a single-precision real number out to a string, and then reads it back in again, keeping track of the maximum error it encounters. We'll just go from 0 to 1, in units of machine epsilon. The code follows:
program testaccuracy
character(len=128) :: teststring
integer, parameter :: nformats=4
character(len=20), parameter :: formats(nformats) = &
[ '( E11.4)', '( E13.6)', '( E15.8)', '(E17.10)' ]
real, dimension(nformats) :: errors
real :: output, back
real, parameter :: delta=epsilon(output)
integer :: i
errors = 0
output = 0
do while (output < 1)
do i=1,nformats
write(teststring,FMT=formats(i)) output
read(teststring,*) back
if (abs(back-output) > errors(i)) errors(i) = abs(back-output)
enddo
output = output + delta
end do
print *, 'Maximum errors: '
print *, formats
print *, errors
print *, 'Trying with default format: '
errors = 0
output = 0
do while (output < 1)
write(teststring,*) output
read(teststring,*) back
if (abs(back-output) > errors(1)) errors(1) = abs(back-output)
output = output + delta
end do
print *, 'Error = ', errors(1)
end program testaccuracy
and when we run it, we get:
$ ./accuracy
Maximum errors:
( E11.4) ( E13.6) ( E15.8) (E17.10)
5.00082970E-05 5.06639481E-07 7.45058060E-09 0.0000000
Trying with default format:
Error = 7.45058060E-09
Note that even using a format with 8 digits after the decimal place - which we might think would be plenty, given that single precision reals are only accurate to 6-7 decimal places - we don't get exact copies back, off by approximately 1e-8. And this compiler's default format does not give us accurate round-trip floating point values; some error is introduced! If you're a video-game programmer, that level of accuracy may well be enough. If you're doing time-dependant simulations of turbulent fluids, however, that might absolutely not be ok, particularly if there's some bias to where the error is introduced, or if the error occurs in what is supposed to be a conserved quantity.
Note that if you try running this code, you'll notice that it takes a surprisingly long time to finish. That's because, maybe surprisingly, performance is another real issue with text output of floating point numbers. Consider the following simple program, which just writes out your example of a 5000 × 5000 real array as text and as unformatted binary:
program testarray
implicit none
integer, parameter :: asize=5000
real, dimension(asize,asize) :: array
integer :: i, j
integer :: time, u
forall (i=1:asize, j=1:asize) array(i,j)=i*asize+j
call tick(time)
open(newunit=u,file='test.txt')
do i=1,asize
write(u,*) (array(i,j), j=1,asize)
enddo
close(u)
print *, 'ASCII: time = ', tock(time)
call tick(time)
open(newunit=u,file='test.dat',form='unformatted')
write(u) array
close(u)
print *, 'Binary: time = ', tock(time)
contains
subroutine tick(t)
integer, intent(OUT) :: t
call system_clock(t)
end subroutine tick
! returns time in seconds from now to time described by t
real function tock(t)
integer, intent(in) :: t
integer :: now, clock_rate
call system_clock(now,clock_rate)
tock = real(now - t)/real(clock_rate)
end function tock
end program testarray
Here are the timing outputs, for writing to disk or to ramdisk:
Disk:
ASCII: time = 41.193001
Binary: time = 0.11700000
Ramdisk
ASCII: time = 40.789001
Binary: time = 5.70000000E-02
Note that when writing to disk, the binary output is 352 times as fast as ASCII, and to ramdisk it's closer to 700 times. There are two reasons for this - one is that you can write out data all at once, rather than having to loop; the other is that generating the string decimal representation of a floating point number is a surprisingly subtle operation which requires a significant amount of computing for each value.
Finally, is data size; the text file in the above example comes out (on my system) to about 4 times the size of the binary file.
Now, there are real problems with binary output. In particular, raw Fortran (or, for that matter, C) binary output is very brittle. If you change platforms, or your data size changes, your output may no longer be any good. Adding new variables to the output will break the file format unless you always add new data at the end of the file, and you have no way of knowing ahead of time what variables are in a binary blob you get from your collaborator (who might be you, three months ago). Most of the downsides of binary output are avoided by using libraries like NetCDF, which write self-describing binary files that are much more "future proof" than raw binary. Better still, since it's a standard, many tools read NetCDF files.
There are many NetCDF tutorials on the internet; ours is here. A simple example using NetCDF gives similar times to raw binary:
$ ./array
ASCII: time = 40.676998
Binary: time = 4.30000015E-02
NetCDF: time = 0.16000000
but gives you a nice self-describing file:
$ ncdump -h test.nc
netcdf test {
dimensions:
X = 5000 ;
Y = 5000 ;
variables:
float Array(Y, X) ;
Array:units = "ergs" ;
}
and file sizes about the same as raw binary:
$ du -sh test.*
96M test.dat
96M test.nc
382M test.txt
the code follows:
program testarray
implicit none
integer, parameter :: asize=5000
real, dimension(asize,asize) :: array
integer :: i, j
integer :: time, u
forall (i=1:asize, j=1:asize) array(i,j)=i*asize+j
call tick(time)
open(newunit=u,file='test.txt')
do i=1,asize
write(u,*) (array(i,j), j=1,asize)
enddo
close(u)
print *, 'ASCII: time = ', tock(time)
call tick(time)
open(newunit=u,file='test.dat',form='unformatted')
write(u) array
close(u)
print *, 'Binary: time = ', tock(time)
call tick(time)
call writenetcdffile(array)
print *, 'NetCDF: time = ', tock(time)
contains
subroutine tick(t)
integer, intent(OUT) :: t
call system_clock(t)
end subroutine tick
! returns time in seconds from now to time described by t
real function tock(t)
integer, intent(in) :: t
integer :: now, clock_rate
call system_clock(now,clock_rate)
tock = real(now - t)/real(clock_rate)
end function tock
subroutine writenetcdffile(array)
use netcdf
implicit none
real, intent(IN), dimension(:,:) :: array
integer :: file_id, xdim_id, ydim_id
integer :: array_id
integer, dimension(2) :: arrdims
character(len=*), parameter :: arrunit = 'ergs'
integer :: i, j
integer :: ierr
i = size(array,1)
j = size(array,2)
! create the file
ierr = nf90_create(path='test.nc', cmode=NF90_CLOBBER, ncid=file_id)
! define the dimensions
ierr = nf90_def_dim(file_id, 'X', i, xdim_id)
ierr = nf90_def_dim(file_id, 'Y', j, ydim_id)
! now that the dimensions are defined, we can define variables on them,...
arrdims = (/ xdim_id, ydim_id /)
ierr = nf90_def_var(file_id, 'Array', NF90_REAL, arrdims, array_id)
! ...and assign units to them as an attribute
ierr = nf90_put_att(file_id, array_id, "units", arrunit)
! done defining
ierr = nf90_enddef(file_id)
! Write out the values
ierr = nf90_put_var(file_id, array_id, array)
! close; done
ierr = nf90_close(file_id)
return
end subroutine writenetcdffile
end program testarray
OPEN the file for reading and writing as "unformatted", and READ and WRITE the data without supplying a format, as demonstrated in the program below.
program xunformatted
integer, parameter :: n = 5000, inu = 20, outu = 21
real :: x(n,n)
integer :: i
character (len=*), parameter :: out_file = "temp_num"
call random_seed()
call random_number(x)
open (unit=outu,form="unformatted",file=out_file,action="write")
do i=1,n
write (outu) x(i,:) ! write one row at a time
end do
print*,"sum(x) =",sum(x)
close (outu)
open (unit=inu,form="unformatted",file=out_file,action="read")
x = 0.0
do i=1,n
read (inu) x(i,:) ! read one row at a time
end do
print*,"sum(x) =",sum(x)
end program xunformatted
Here's my program.
local t = {}
local match = string.gmatch
local insert = table.insert
val = io.read("*a")
for num in match(val, "%d+") do
insert(t, num)
end
I'm wondering if there is a faster way to load a large (16MB+) array of integers than this. Considering the data is composed of line after line of a single number can this be made faster? Should I be looking at io.read("*n") instead?
Given that your file size is 16MB, your loading routine's performance will be dominated by file IO. How long it takes you to process the loaded data will generally be irrelevant next to that.
Just try it; profile how long it takes to just load the file (stopping the script after io.read), then profile how long the whole script takes. The latter will be longer, but it's only going to be by some relatively small percentage, not vast amounts.
Loading the whole file at once the way you're doing will almost certainly be faster than doing it piecemeal. Filesystems like reading entire blocks of data all at once, rather than bits at a time. Beyond that, how to process the text is relatively irrelevant.
I'm not sure if its faster, but read("*n") is much simpler...
local t = { }
while true do
local n = io.stdin:read("*n")
if n == nil then break end
table.insert ( t , n )
end
Probably, this would be faster:
local t = {}
local match = string.match
for line in io.lines() do
t[#t+1] = match(line, '%d+')
end
Don't forget to convert strings to numbers.
I just started my path on Erlang and I'm facing a problem I can't sort out a solution about:
I wrote a metod to take a domain espressed as a binary string, i.e. <<"www.404pagenotfound.com">> and convert it in the domain format as required for DNS protocol, so: <<3,"www",15,"pagenotfound",3,"com">>.
In the following the code (I rewroted it many times in different ways):
domainbyte(Bin) ->
if byte_size(Bin) > 0 ->
Res = binary:split(Bin, <<".">>),
[Chunk|[RestList]] = Res,
ChunkSize = byte_size(Chunk),
if length(RestList) > 0 ->
Rest = domainbyte(RestList), %% <- Got "bad argument" here!
<<ChunkSize/binary,Chunk,Rest>>;
true ->
<<ChunkSize/binary,Chunk>>
end
end.
Thx in advance for any clues.
PS.
Thx to comments I've found the error in the code abobe:
if length(RestList) > 0 -> %% here RestList is binary data so length throw "bad argument" error.
I've rewroted the method this way, but still with no luck:
**NOTE: I was able to fix the code in the following, problem is that if you have a binary chunk and you want to use it in another binary string, you must specify /binary on it: something not obvious to me.
I.e.: consider this small code snip:
**
TT = <<"com">>,
SS = <<3, TT, 0>> %% <- you get error: bad argument
**
must be fixed this way:
**
TT = <<"com">>,
SS = <<3, TT/binary, 0>>
domainbyte(Bin) ->
if byte_size(Bin) > 0 ->
Res = binary:split(Bin, <<".">>),
if length(Res) > 1 ->
[Chunk|[RestList]] = Res,
ChunkSize = byte_size(Chunk),
Rest = domainbyte(RestList),
<<ChunkSize,Chunk,Rest>>;
true ->
[Chunk] = Res,
ChunkSize = byte_size(Chunk),
<<ChunkSize,Chunk>>
end
end.
MdP
I think the easiest solution is to define the function using a binary comprehension:
domainbyte(Bin) ->
Chunks = binary:split(Bin, <<".">>, [global]), %A list of all the chunks
<< <<byte_size(C),C/binary>> || C <- Chunks >>. %Build output binary
It might be slightly faster to build the output binary as a list of segments in a separate function then put them together in an iolist_to_binary/1. Note that if a '.' occurs outermost in the binary then this code will take that as an empty segment of length 0. If these should be discarded then you nee to add the option trim to binary:split/3. Note also that the size will occupy only one byte.
#Alnitak has the separate function but builds the binary one segment at time so it is not more efficient than the binary comprehension which does the same thing.
N.B. that if you have a binary segment Chunk/binary when constructing a binary this means that Chunk IS a binary, not that it should become one. Binaries are flat structures, think byte arrays, so everything becomes a binary. Or rather the binary.
EDIT: Though I see I missed the 0 which should be at the end. That is left as an exercise to the reader.
EDIT: Being in teaching mode, apart from the constructing binaries, a key to writing good Erlang code is understanding pattern matching. You use a bit but could do it more:
domainbyte(Bin) ->
case binary:split(Bin, <<".">>) of
[Chunk,Rest] -> %There was a '.'
RestBin = domainbyte(Rest),
Size = byte_size(Chunk),
<<Size,Chunk/binary,RestBin/binary>>;
[Chunk] -> %This was the last chunk
Size = byte_size(Chunk),
<<Size,Chunk/binary,0>> %Add terminating 0
end.
This is basically doing the same as your code but we are using pattern matching to select a clause, not only to pull apart a known structure. Pattern matching is the basic method for control, not just in case as here but also in functions and receive. This results in if being used quite sparingly.
Enough from me for now.
I'm a bit rusty on Erlang, but I think your problem is that RestList is an array of chunks from the output of binary:split.
So when you chuck it recursively back into domainbyte it's in the wrong format.
Also - don't forget the terminating NUL byte to represent the root label!
FWIW, here's my working version:
label([]) ->
<< 0 >>;
label([H|T]) ->
D = label(H),
P = label(T),
<< D/binary, P/binary>>;
label(A) ->
L = byte_size(A),
<< <<L>>/binary, A/binary>>.
domainbyte(A) ->
Res = binary:split(A, <<".">>, [global, trim]),
label(Res).
It correctly adds a trailing NUL byte, and trims any extra trailing dots.
The concat of binaries should be written like this:
<< <<ChunkSize>>/binary, Chunk/binary, Rest/binary>>
% and
<< <<ChunkSize>>/binary, Chunk/binary>>