Swift 3 how to store a struct in a "Data" object - struct

What is the "right" way to stuff an arbitrary, odd sized struct into a swift 3 Data object ?
I think that I have got there, but it seems horribly convoluted for what from prior experience was no than
dataObject.append(&structInstance, sizeof(structInstance))
My case is as follows:
The structure of interest:
public struct CutEntry {
var itemA : UInt64
var itemB : UInt32
}
I have an array of these things that I want to stuff into a data object, in a specific manner as the data object becomes a file which is eventually read by a different application on a different architecture.
The function to put them into a Data object
open func encodeCutsData() -> Data
{
var data = Data()
for entry in cutsArray
{
// bigendian stuff, as a var, just so the you can get the address
var entryCopy = CutEntry(itemA: entry.itemA.bigEndian, itemB: entry.itemB.bigEndian)
// step 1 get the address of the item as a UnsafePointer
let d2 = withUnsafePointer(to: &entryCopy) { return $0}
// step 2 cast it to a raw pointer
let d3 = UnsafeRawPointer(d2)
// step 3 create a temp data object
let d4 = Data(bytes:d3, count: MemoryLayout<CutEntry>.size )
// step 4 add the temp to main data object
data.append(d4)
}
return data
}
Earlier when we only had NSMutableData it was
let item = NSMutableData()
for entry in cutsArray
{
var entryCopy = CutEntry(cutPts: entry.cutPts.bigEndian, cutType: entry.cutType.bigEndian)
item.append(&entryCopy, length: MemoryLayout<CutEntry>.size)
}
I've spent a few hours searching for examples of manipulating struct and Data objects. I though that I was close when I found references to unsafebufferpointer. That blew up in my face when I discovered that "buffer" bit uses core memory alignment (which can be useful) and it was stuffing 16 bytes into the data object instead of the expected 12.
I am quite prepared to say that I have missed the blindingly obvious bit of RTFM somewhere. Can anyone offer a cleaner solution ? or has Swift really gone backwards here ?
If I could find a way of getting a pointer to the item as a UInt8 pointer that would remove a couple of lines, but that looks just a difficult.

With checking the reference of Data, I can find two things which may be useful for you:
init(bytes: UnsafeRawPointer, count: Int)
func append(Data)
You can write something like this:
var data = Data()
for entry in cutsArray {
var entryCopy = CutEntry(cutPts: entry.cutPts.bigEndian, cutType: entry.cutType.bigEndian)
data.append(Data(bytes: &entryCopy, count: MemoryLayout<CutEntry>.size))
}

Related

Nim: work with read-only memory mapped files

I've only just started with Nim, hence it possibly is a simple question. We need to do many lookups into data that are stored in a file. Some of these files are too large to load into memory, hence the mmapped approach. I'm able to mmap the file by means of memfiles and either have a pointer or MemSlice at my hand. The file and the memory region are read-only, and hence have a fixed size. I was hoping that I'm able to access the data as immutable fixed size byte and char arrays without copying them, leveraging all the existing functionalities available to seqs, arrays, strings etc.. All the MemSlice / string methods copy the data, which is fair, but not what I want (and in my use case don't need).
I understand array, strings etc. types have a pointer to the data and a len field. But couldn't find a way to create them with a pointer and len. I assume it has something to do with ownership and refs to mem that may outlive my slice.
let mm = memfiles.open(...)
let myImmutableFixesSizeArr = ?? # cast[ptr array[fsize, char]](mm.mem) doesn't compile as fsize needs to be const. Neither could I find something like let x: [char] = array_from(mm.mem, fsize)
let myImmutableFixedSizeString = mm[20, 30].to_fixed_size_immutable_string # Create something that is string like so that I can use all the existing string methods.
UPDATE: I did find https://forum.nim-lang.org/t/4680#29226 which explains how to use OpenArray, but OpenArray is only allowed as function argument, and you - if I'm not mistaken - it is doesn't behave like a normal array.
Thanks for your help
It is not possible to convert a raw char array in memory (ptr UncheckedArray[char]) to a string without copying, only to an openArray[char] (or cstring)
So it won't be possible to use procs that expect a string, only those that accept openArray[T] or openArray[char]
Happily an openArray[T] behaves exactly like a seq[T] when sent to a proc.
({.experimental:"views".} does let you assign an openArray[T] to a local variable, but it's not anywhere near ready for production)
you can use the memSlices iterator to loop over delimited chunks in a memFile without copying:
import memfiles
template toOpenArray(ms: MemSlice, T: typedesc = byte): openArray[T] =
##template because openArray isn't a valid return type yet
toOpenArray(cast[ptr UncheckedArray[T]](ms.data),0,(ms.size div sizeof(T))-1)
func process(slice:openArray[char]) =
## your code here but e.g.
## count number of A's
var nA: int
for ch in slice.items:
if ch == 'A': inc nA
debugEcho nA
let mm = memfiles.open("file.txt")
for slice in mm.memSlices:
process slice.toOpenArray(char)
Or, to work with some char array represented in the middle of the file, you can use pointer arithmetic.
import memfiles
template extractImpl(typ,pntr,offset) =
cast[typ](cast[ByteAddress](pntr)+offset)
template checkFileLen(memfile,len,offset) =
if offset + len > memfile.size:
raise newException(IndexDefect,"file too short")
func extract*(mm: MemFile,T:typedesc, offset:Natural): ptr T =
checkFileLen(mm,T,offset)
result = extractImpl(ptr T,mm.mem,offset)
func extract*[U](mm: MemFile,T: typedesc[ptr U], offset: Natural): T =
extractImpl(T,mm.mem,offset)
let mm = memfiles.open("file.txt")
#to extract a compile-time known length string:
let mystring_offset = 3
const mystring_len = 10
type MyStringT = array[mystring_len,char]
let myString:ptr MyStringT = mm.extract(MyStringT,mystring_offset)
process myString[]
#to extract a dynamic length string:
let size_offset = 14
let string_offset = 18
let sz:ptr int32 = mm.extract(int32,size_offset)
let str:ptr UncheckedArray[char] = mm.extract(ptr UncheckedArray[char], string_offset)
checkFileLen(mm,sz[],string_offset)
process str.toOpenArray(0,sz[]-1)

Converting string into field names

I have a structure, where I store all the hours and minutes into mongodb. In this case, when I get a request to modify the value, I get the hour and minute as string. Is there a way to find the field name from the string that is given as input
You can see it here
package main
import "fmt"
type Min struct {
v01 int `bson:"01",json:"01"`
v02 int `bson:"02",json:"02"`
}
type Hour struct {
v01 Min `bson:"01",json:"01"`
v02 Min `bson:"02",json:"02"`
}
func main() {
fmt.Println("Hello, playground")
var h Hour
h.v01.v01 = 1
h.v02.v01 = 2
fmt.Println(h)
h.Set("01", "01", 10)
fmt.Println(h)
}
func (h *Hour) Set(hour string, min string, value int) {
h.v01.v01 = 10 //Here I have hardcoded it
// Is there a way to do this from the given input
// e.g. h.Set("01","01",100)
}
If you notice, the input is "01","01". I would like to change this input as h.v01.v01. Is it possible in Go?
Note: I am using currently maps in this case. I would like to change this into structure access, if possible, so that I can use goroutine to speed up my program. Currently goroutines are not safe for writing into maps.
Currently goroutines are not safe for writing into maps.
That might need to be revisited with Go 1.9 (Aug. 2017) and the concurrent map present in the sync package
The new Map type in the sync package is a concurrent map with amortized-constant-time loads, stores, and deletes.
It is safe for multiple goroutines to call a Map's methods concurrently.
Combine that with oleiade/reflections that I mentioned, as commented, in "GoLang: Access struct property by name", and you should be able to do both (access property name, and use maps)

Struct store large chunk of data

I have a data file with millions of rows and I wanted to read that and store in a struct.
public struct Sample
{
public int A;
public DateTime B;
}
Sample[] sample = new Sample[];
This definition gives me this error "Wrong number of indicies inside[]; expected 1"
How do I store data in struct (with less memory usage)? Array is that best of something else?
var reader = new StreamReader(File.OpenRead(#"C:\test.csv"));
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(';');
}
In order to allocate an array, you need to specify how many elements it holds. That's what the error is telling you.
If you don't know the size, you can use List<T> which will grow as needed. Internally List<T> is implemented using T[] which means that from a look-up stand point it acts like an array. However, the work of reallocating larger arrays as needed is handled by List<T>.

C++ : Strings, Structures and Access Violation Writing Locations

I'm attempting to try and use a string input from a method and set that to a variable of a structure, which i then place in a linked list. I didn't include, all of code but I did post constructor and all that good stuff. Now the code is breaking at the lines
node->title = newTitle;
node->isbn = newISBN;
So newTitle is the string input from the method that I'm trying to set to the title variable of the Book structure of the variable node. Now, I'm assuming this has to do with a issue with pointers and trying to set data to them, but I can't figure out a fix/alternative.
Also, I tried using
strcpy(node->title, newTitle)
But that had an issue with converting the string into a list of chars because strcpy only uses a list of characters. Also tried a few other things, but none seemed to pan out, help with an explanation would be appreciated.
struct Book
{
string title;
string isbn;
struct Book * next;
};
//class LinkedList will contains a linked list of books
class LinkedList
{
private:
Book * head;
public:
LinkedList();
~LinkedList();
bool addElement(string title, string isbn);
bool removeElement(string isbn);
void printList();
};
//Constructor
//It sets head to be NULL to create an empty linked list
LinkedList::LinkedList()
{
head = NULL;
}
//Description: Adds an element to the link in alphabetical order, unless book with
same title then discards
// Returns true if added, false otherwise
bool LinkedList::addElement(string newTitle, string newISBN)
{
struct Book *temp;
struct Book *lastEntry = NULL;
temp = head;
if (temp==NULL) //If the list is empty, sets data to first entry
{
struct Book *node;
node = (Book*) malloc(sizeof(Book));
node->title = newTitle;
node->isbn = newISBN;
head = node;
}
while (temp!=NULL)
{
... //Rest of Code
Note that your Book struct is already a linked list implementation, so you don't need the LinkedList class at all, or alternatively you don't need the 'next' element of the struct.
But there's no reason from the last (long) code snippet you pasted to have an error at the lines you indicated. node->title = newTitle should copy the string in newTitle to the title field of the struct. The string object is fixed size so it's not possible to overwrite any buffer and cause a seg fault.
However, there may be memory corruption from something you do further up the code, which doesn't cause an error until later on. The thing to look for is any arrays, including char[], that you might be overfilling. Another idea is you mention you save method parameters. If you copy, it's ok, but if you do something like
char* f() {
char str[20];
strcpy(str, "hello");
return str;
}
...then you've got a problem. (Because str is allocated on the stack and you return only the pointer to a location that won't be valid after the function returns.) Method parameters are local variables.
The answer you seek can be found here.
In short: the memory malloc returns does not contain a properly constructed object, so you can't use it as such. Try using new / delete instead.

JNA - Use structure array as byref argument

I know parts of this issue is covered by some posts here and I have looked at them and tested some but with no luck.
I have this native method signature which should populate the provided CBadgeData structure array with results:
int elc_GetBadges(int nHandle, char* cErr, int* nRecCount, CBadgeData** arr)
The CBadgeData structure is implemented as follows:
package test.elcprog;
import java.util.Arrays;
import java.util.List;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;
public class CBadgeData extends Structure{
public static class ByReference extends CBadgeData implements Structure.ByReference { }
public int nBadgeID, nTrigger, nExtraData;
public String cName;
public CBadgeData(Pointer pointer){
super(pointer);
}
public CBadgeData(){ }
public String ToString() {
return nBadgeID + "," + nTrigger + "," + nExtraData + "," + cName;
}
#Override
protected List getFieldOrder() {
String[] s = new String[]{"nBadgeID","nTrigger","nExtraData","cName"};
return Arrays.asList(s);
}
}
My last try to craft this argument and call the method looked like this:
CBadgeData.ByReference[] badges = new CBadgeData.ByReference[max_items];
new CBadgeData.ByReference().toArray(badges);
int ret = inst.elc_GetBadges(handle, err, recCount, badges);
It fails with segmentation error.
My Question is what Java type should be provided here as an argument for the native CBadgeData** in the call to elc_GetBadges?
EDIT -1-
Populating the array myself (with or without terminating null pointer) didn't work and caused further Seg crashes. I then used Pointer[] arg as technomage suggested:
Pointer[] pointers = new Pointer[max_items];
for(int i=0; i<max_items; i++){
pointers[i] = new CBadgeData.ByReference().getPointer();
}
int ret = inst.elc_GetBadges(handle, err, recCount, pointers);
This caused no error but seems to not make any changes to the returning struct which should have contain 4 items in this case:
int bid = new CBadgeData(pointers[i]).nBadgeID; // this returns null for all items
Using explicit read() / write() on the struct led to Seg crashes again (on the read):
Any idea what am I still missing here?
EDIT -2-
Interestingly enough - using the Memory.get directly, after calling the native method, gets the correct results:
Memory m= (Memory)pointers[0];
System.out.println("1st int: "+m.getInt(0)); // this gets 24289 which is 5ee1
System.out.println("2nd int: "+m.getInt(4)); // this gets 3
System.out.println("3rd int: "+m.getInt(8)); // this gets 255
System.out.println("String: "+m.getString(12)); // this gets "Badge[5EE1]" as supposed
But the read() still crashes. Any thoughts?
I'm inferring that CBadgeData** input is intended to be an array of pointer to CBadgeData.
As such, the Structure.ByReference tagging is correct.
Structure.toArray() is probably not appropriate here, or at least not necessary (it allocates a contiguous block of structs in memory). You can just populate your array with CBadgeData.ByReference instances.
Perhaps your callee is expecting a NULL pointer at the end of the array? I don't see another indicator of the array length to the callee.
CBadgeData.ByReference[] badges = new CBadgeData.ByReference[max_items+1];
for (int i=0;i < badges.length-1;i++) {
badges[i] = new CBadgeData.ByReference();
}
badges[badges.length-1] = null;
Pretty sure that works. If for whatever reason there's a bug handling Structure.ByReference[], I know that Pointer[] is reliable and will do the same thing.
EDIT
If you use Pointer[] instead of Structure.ByReference[] (please post a bug to the project site if Structure.ByReference[] does not work), you will have to manually call Structure.write/read before/after your native function call, since JNA will not know that the pointers reference structures that need to be synched with native memory. I'd bet, however, that the cause of your crashes when using Structure.ByReference[] was simply that JNA was automatically calling Structure.read() after the call and triggered the same error that you see when calling it explicitly.
If you get a segfault on read, it likely means that your structure fields aren't properly aligned or defined, or (less likely) that you have corrupt data that can't be read properly. To diagnose this, set jna.dump_memory=true and print out your struct after calling Structure.write() to see if the contents of the structure appear as you'd expect. It'd also help to post the native and JNA forms of your structure here, if possible.

Resources