How to convert an array to a cstring? - nim-lang

I found (here) the source of a program to compute the MD5 of a file using Nim.
This program no longer compiles (nim 1.4) as the implicit conversion between array to cstring has been disabled.
How can this be fixed?
import md5
import os
proc calculateMD5Incremental(filename: string) : string =
const blockSize: int = 8192
var
c: MD5Context
d: MD5Digest
f: File
bytesRead: int = 0
buffer: array[blockSize, char]
byteTotal: int = 0
#read chunk of file, calling update until all bytes have been read
try:
f = open(filename)
md5Init(c)
bytesRead = f.readBuffer(buffer.addr, blockSize)
while bytesRead > 0:
byteTotal += bytesRead
md5Update(c, buffer, bytesRead) # <--- HERE buffer should be cstring
bytesRead = f.readBuffer(buffer.addr, blockSize)
md5Final(c, d)
except IOError:
echo("File not found.")
finally:
if f != nil:
close(f)
result = $d
if paramCount() > 0:
let arguments = commandLineParams()
echo("MD5: ", calculateMD5Incremental(arguments[0]))
else:
echo("Must pass filename.")
quit(-1)
Note: I'm more interested in the general question and not in MD5, this was the example that came to hand.

In your answer you turned array into string with $. This though does not return "ciao" but "['c', 'i', 'a', 'o']". Not sure if you want that but proper way to do the conversion is as follows:
const blockSize: int = 4
var
c: cstring
buffer: array[blockSize, char]
buffer = ['c','i','a','o']
c = cast[cstring](create(char, blockSize + 1))
moveMem(c[0].addr, buffer[0].addr, blockSize)
assert c == "ciao"
Mind that this is optimal unsafe solution. If you want to be safe but little slower the you can use this code:
const blockSize: int = 4
var
c: cstring
buffer: array[blockSize, char]
temp: string
buffer = ['c','i','a','o']
temp.setLen(blockSize)
for i in 0..<blockSize:
temp[i] = buffer[i]
c = temp
assert c == "ciao"

UPDATE In this context this is completely wrong. As pointed out in the accepted answer, $array will render it in a "screen friendly" way.
Correct implementation
The working procedure for md5(fromFile):
proc calculateMD5Incremental(filename: string) : string =
const blockSize: int = 8192
var
c: MD5Context
d: MD5Digest
f: File
s: cstring
bytesRead: int = 0
buffer: array[blockSize, char]
byteTotal: int = 0
#read chunk of file, calling update until all bytes have been read
try:
f = open(filename)
md5Init(c)
bytesRead = f.readBuffer(buffer.addr, blockSize)
while bytesRead > 0:
byteTotal += bytesRead
s = cast[cstring](create(char, blockSize + 1))
moveMem(s[0].addr, buffer[0].addr, blockSize)
md5Update(c, s, bytesRead)
bytesRead = f.readBuffer(buffer.addr, blockSize)
md5Final(c, d)
except IOError:
stderr.writeLine("ERROR: File not found: ", filename)
finally:
if f != nil:
close(f)
result = $d
Wrong conversion
~I found that a regular string is accepted so $buffer will convert the array to a valid string:~
const blockSize: int = 4
var
c: cstring
buffer: array[blockSize, char]
buffer = ['c','i','a','o']
c = $buffer

Related

Have to count a character occurrence in Nim string type

How to count a character occurrence in string in Nim, mainly using its native statements prior go to module ? eg.
var
str = "Hello World"
c : int
c = numChar( "o", str ) # <- illustration only ?
The earlier answer is correct but if you do not want to import any modules you can write your own procedure:
proc count_char(value: string = "Hello World", ch: char = 'o'): int =
var cnt_c: int = 0
for c in value:
if c == ch:
cnt_c += 1
result = cnt_c
var
val: string = "Mother Goose"
ch: char = 'o'
echo $count_char(val, ch)
PS: Unrelated - Need syntax highlight for nim-lang on SO.
Use the count function from strutils:
import std/strutils
let str = "Hello World"
let count = count(str, 'o')
assert count = 1
There’s also a string overload for counting sub strings as well.

Changing the length of a string (allocating memory for characters)

def get_avail_mb(): int
f: FILE = FILE.open("/proc/meminfo","r")
s: string = ""
while s.length < 200 do s += " "
f.read(s, 200, 1)
var a = s.split("\n")
s = a[2]
t: string = ""
for var i = 0 to s.length
if s[i] <= '9' && s[i] >= '0'
t += s[i].to_string()
return int.parse(t)/1000
Notice how I allocate the string to 200 charaters with while s.length < 200 do s += " " to read bytes into this string from the file? Is there a better way to set the length of a string to N characters in Genie other than appending space character N times?
Probably the best way is to create a fixed size array as a buffer and cast the buffer to a string. This avoids some C warnings when compiled. Compile with valac --pkg posix example.gs:
[indent=4]
uses
Posix
init
print( #"Available memory: $(get_avail_mb()) MB" )
def get_avail_mb():int
f:FILE = FILE.open("/proc/meminfo","r")
buffer:uint8[200]
f.read(buffer, 200, 1)
result:int = 0
match_result:MatchInfo
if ( /^MemAvailable:\s*([0-9]*).*$/m.match(
(string)buffer,
0,
out match_result
))
result = int.parse( match_result.fetch( 1 ))/1000
return result
Alternatively you could try string.nfill ():
[indent=4]
uses
Posix
init
print( #"Available memory: $(get_avail_mb()) MB" )
def get_avail_mb():int
f:FILE = FILE.open("/proc/meminfo","r")
s:string = string.nfill( 200, ' ' )
f.read(s, 200, 1)
result:int = 0
match_result:MatchInfo
if ( /^MemAvailable:\s*([0-9]*).*$/m.match(
s,
0,
out match_result
))
result = int.parse( match_result.fetch( 1 ))/1000
return result
yes there is, just avoid the dreaded for-loop which cannot handle certain corner cases!

Compile-time Call Count in Nim

The following code does not compile but illustrates what I would like to do: totalTests should hold the number of time that assertEquals() is called (assertEquals() should probably be a macro for this to be possible, but I'm not familiar with this aspect of Nim yet).
Any idea how this code should be modified to enable the following code to print [1/2] and [2/2] at the beginning of each test report line?
from strutils import format
var countTested = 0
var countFailed = 0
var countPassed = 0
let totalTests = 0 # <-- need let or other compile-time type here (?)
# using proc here... macro may be needed to be able to count calls (?)
proc assertEquals*[T](testName: string, expected: T, p: (proc(): T)) =
countTested += 1
totalTests += 1 # <-- compilation error (how can we increase each time it is called?)
write(stdout, format("[$num/$total] $name: ", "num", countTested, "total", totalTests, "name", testName))
var val = p()
if val == expected:
write(stdout, "passed\n")
countPassed += 1
else:
write(stdout, "failed\n")
countFailed += 1
when isMainModule:
assertEquals("test #A", 12, proc(): int = 14-2)
assertEquals("test #B", 12, proc(): int = 12-2)
Edit: added questions in code
Here is one way to do it. You can execute code at compile time by using a macro or a static statement. Note that there's still no way to reliably count these across multiple modules.
import macros, strutils
proc beginTests()
var countTested = 0
var countFailed = 0
var countPassed = 0
var totalTests = 0
var totalTestsCT {.compiletime.} = 0
macro endTests(): stmt =
quote do:
proc beginTests() =
totalTests = `totalTestsCT`
proc assertEqualsImpl*[T](testName: string, expected: T, p: (proc(): T)) =
countTested += 1
write(stdout, format("[$num/$total] $name: ",
"num", countTested, "total", totalTests, "name", testName))
var val = p()
if val == expected:
write(stdout, "passed\n")
countPassed += 1
else:
write(stdout, "failed\n")
countFailed += 1
macro assertEquals*[T](testName: string, expected: T, p: (proc(): T)): stmt =
totalTestsCT += 1
quote do:
assertEqualsImpl(`testName`, `expected`, `p`)
when isMainModule:
beginTests()
assertEquals("test #A", 12, proc(): int = 14-2)
assertEquals("test #B", 12, proc(): int = 12-2)
endTests()
An alternative implementation would be to embed the tests in a custom block statement, e.g.
testSuite:
assertEquals("test #A", 12, proc(): int = 14-2)
assertEquals("test #B", 12, proc(): int = 12-2)
The testSuite macro would then count the assertions in the embedded code and initialize the variable accordingly.
Yet another solution would be to not execute the tests directly, but store them in a list and only execute them at the end.
Here is an implementation of Reimer's third suggestion, which worked best for me.
import macros, strutils
type
TestSuiteObj = object
countTested: int
countFailed: int
countPassed: int
totalTests: int
tests: seq[(proc (self: TestSuite))]
TestSuite* = ref TestSuiteObj
proc newTestSuite*(): TestSuite =
new(result)
result.countTested = 0
result.countFailed = 0
result.countPassed = 0
result.totalTests = 0
result.tests = #[]
proc assertEquals*[T](self: TestSuite, testName: string, expected: T, p: (proc(): T)) =
self.totalTests += 1
var testProc = proc(self: TestSuite) =
self.countTested += 1
write(stdout, format("[$num/$total] $name: ", "num", self.countTested, "total", self.totalTests, "name", testName))
var val = p()
if val == expected:
write(stdout, "passed\n")
self.countPassed += 1
else:
write(stdout, "failed\n")
self.countFailed += 1
self.tests.add(testProc)
proc run*(self: TestSuite) =
self.totalTests = self.tests.len
for p in self.tests:
p(self)
var verdict = case (self.countTested == self.countPassed)
of true: "PASSED"
of false: "FAILED"
echo format("$verdict. Passed [$passed/$total] tests.", "verdict", verdict, "passed", self.countPassed, "total", self.countTested)
# Sanity
assert(self.countTested == (self.countFailed+self.countPassed))
assert(self.countTested == self.totalTests)
when isMainModule:
var suite = newTestSuite()
suite.assertEquals("test #A", 12, proc(): int = 14-2)
suite.assertEquals("test #B", 12, proc(): int = 12-2)
suite.run()

How to read a C generated binary file in Lua

I want to read a 32 bit integer binary file provided by another program. The file contains only integer and no other characters (like spaces or commas). The C code to read this file is as follows:
FILE* pf = fopen("C:/rktemp/filename.dat", "r");
int sz = width*height;
int* vals = new int[sz];
int elread = fread((char*)vals, sizeof(int), sz, pf);
for( int j = 0; j < height; j++ )
{
for( int k = 0; k < width; k++ )
{
int i = j*width+k;
labels[i] = vals[i];
}
}
delete [] vals;
fclose(pf);
But I don't know how to read this file into array using Lua.
I've tried to read this file using io.read, but part of the array looks like this:
~~~~~~xxxxxxxxyyyyyyyyyyyyyyzzzzzzzz{{{{{{{{{|||||||||}}}}}}}}}}}~~~~~~~~~xxxxxxxyyyyyyyyyyyyyyzzzzzz{{{{{{{{{{|||||||||}}}}}}}}}}}~~~~~~~~~xxyyyyyyyyyyyyyzzzzz{{{{{{|||}}}yyyyyyyyyyyz{{{yyyyyyyyÞľūơǿȵɶʢ˺̤̼ͽаҩӱľǿجٴȵɶʢܷݸ˺໻⼼ӱľǿ
Also the Matlab code to read this file is like this:
row = image_size(1);
colomn = image_size(2);
fid = fopen(data_path,'r');
A = fread(fid, row * colomn, 'uint32')';
A = A + 1;
B = reshape(A,[colomn, row]);
B = B';
fclose(fid);
I've tried a function to convert bytes to integer, my code is like this:
function bytes_to_int(b1, b2, b3, b4)
if not b4 then error("need four bytes to convert to int",2) end
local n = b1 + b2*256 + b3*65536 + b4*16777216
n = (n > 2147483647) and (n - 4294967296) or n
return n
end
local sup_filename = '1.dat'
fid = io.open(sup_filename, "r")
st = bytes_to_int(fid:read("*all"):byte(1,4))
print(st)
fid:close()
But it still not read this file properly.
You are only calling bytes_to_int once. You need to call it for every int you want to read. e.g.
fid = io.open(sup_filename, "rb")
while true do
local bytes = fid:read(4)
if bytes == nil then break end -- EOF
local st = bytes_to_int(bytes:byte(1,4))
print(st)
end
fid:close()
Now you can use the new feature of Lua language by calling string.unpack , which has many conversion options for format string. Following options may be useful:
< sets little endian
> sets big endian
= sets native endian
i[n] a signed int with n bytes (default is native size)
I[n] an unsigned int with n bytes (default is native size)
The arch of your PC is unknown, so I assume the data to read is unsigned and native-endian.
Since you are reading binary data from the file, you should use io.open(sup_filename, "rb").
The following code may be useful:
local fid = io.open(sup_filename, "rb")
local contents = fid:read("a")
local now
while not now or now < #contents do
local n, now = string.unpack("=I4", contents, now)
print(n)
end
fid:close()
see also: Lua 5.4 manual

Reading beyond buffer

I have a buffer of size bufferSize from which I read in chunks of blockSize, however, this yields some (to me) unexpected behavior, when the blockSize goes beyond the bufferSize.
I've put the code here:
http://play.golang.org/p/Ra2jicYHPu
Why does the second chunk only give 4 bytes? What's happening here?
I'd expect Read to always give the amount of bytes len(byteArray), and if it goes beyond the buffer, it'll handle that situation by setting the pointer in the buffer to after byteArray, and putting the rest of the buffer + whatever is beyond until the new buffer pointer.
Your expectations are not based on any documented behavior of bufio.Reader. If you want "Read to always give the amount of bytes len(byteArray)" you must use io.ReadAtLeast.
package main
import (
"bufio"
"fmt"
"io"
"strings"
)
const bufSize = 10
const blockSize = 12
func main() {
s := strings.NewReader("some length test string buffer boom")
buffer := bufio.NewReaderSize(s, bufSize)
b := make([]byte, blockSize)
n, err := io.ReadAtLeast(buffer, b, blockSize)
if err != nil {
fmt.Println(err)
}
fmt.Printf("First read got %d bytes: %s\n", n, string(b))
d := make([]byte, blockSize)
n, err = io.ReadAtLeast(buffer, d, blockSize)
if err != nil {
fmt.Println(err)
}
fmt.Printf("Second read got %d bytes: %s\n", n, string(d))
}
Playground
Output:
First read got 12 bytes: some length
Second read got 12 bytes: test string
1.see the code of buffio.NewReaderSize
func NewReaderSize(rd io.Reader, size int) *Reader {
// Is it already a Reader?
b, ok := rd.(*Reader)
if ok && len(b.buf) >= size {
return b
}
if size < minReadBufferSize {
size = minReadBufferSize
}
return &Reader{
buf: make([]byte, size),
rd: rd,
lastByte: -1,
lastRuneSize: -1,
}
}
strings.NewReader return a strings.Reader,so the buffer's (returned by bufio.NewReaderSize ) buf has minReadBufferSize(val is 16)
2.see code of bufio.Read
func (b *Reader) Read(p []byte) (n int, err error) {
……
copy(p[0:n], b.buf[b.r:])
b.r += n
b.lastByte = int(b.buf[b.r-1])
b.lastRuneSize = -1
return n, nil
}
copy src is b.buf[b.r:],when your first Read,b.r=12,……

Resources