I have a data file with millions of rows and I wanted to read that and store in a struct.
public struct Sample
{
public int A;
public DateTime B;
}
Sample[] sample = new Sample[];
This definition gives me this error "Wrong number of indicies inside[]; expected 1"
How do I store data in struct (with less memory usage)? Array is that best of something else?
var reader = new StreamReader(File.OpenRead(#"C:\test.csv"));
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(';');
}
In order to allocate an array, you need to specify how many elements it holds. That's what the error is telling you.
If you don't know the size, you can use List<T> which will grow as needed. Internally List<T> is implemented using T[] which means that from a look-up stand point it acts like an array. However, the work of reallocating larger arrays as needed is handled by List<T>.
Related
How much data can be processed using params, saved and session declarations?
How do these declaration affect performance and memory allocation/consumption (stack usage, data copy, etc.)?
What methods can be used in case of static/dynamic arrays with 10k-100k elements?
Params
An untyped param is expanded like a macro any time it is referenced, so resource consumption depends on its use. If you have a param with a large amount of data, then it usually means that the value is a compile-time list ([...]) with many elements, and you use a #foreach loop to process it. A #foreach loop is always unrolled, which gives long compile times and large generated code.
If a param is typed in a template, then that template evaluates the param once and stores a copy in heap-allocated memory. The data is shared between all instances of the device. Cost should be negligible.
Session
Data is heap-stored, one copy per device instance.
Saved
Pretty much like data, but adds a presumably negligible small per-module cost for attribute registration.
There's two more variants of data:
Constant C tables
header %{ const int data[10] = {0, 1, 2, 3, 4, 5, 6, 7, 8, 9}; %}
extern const int data;
Creates one super-cheap module-local instance.
Independent startup memoized method
independent startup memoized method data() -> (const int *) {
int *ret = new int[10];
for (local int i = 0; i < 10; i++) {
ret[i] = i;
}
return ret;
}
The data will be heap-allocated, initialized once, and shared across instances. Initialization is done by code, which saves size if it's easy to express the data programmatically, but can be cumbersome if it's just a table of irregular data.
What is the "right" way to stuff an arbitrary, odd sized struct into a swift 3 Data object ?
I think that I have got there, but it seems horribly convoluted for what from prior experience was no than
dataObject.append(&structInstance, sizeof(structInstance))
My case is as follows:
The structure of interest:
public struct CutEntry {
var itemA : UInt64
var itemB : UInt32
}
I have an array of these things that I want to stuff into a data object, in a specific manner as the data object becomes a file which is eventually read by a different application on a different architecture.
The function to put them into a Data object
open func encodeCutsData() -> Data
{
var data = Data()
for entry in cutsArray
{
// bigendian stuff, as a var, just so the you can get the address
var entryCopy = CutEntry(itemA: entry.itemA.bigEndian, itemB: entry.itemB.bigEndian)
// step 1 get the address of the item as a UnsafePointer
let d2 = withUnsafePointer(to: &entryCopy) { return $0}
// step 2 cast it to a raw pointer
let d3 = UnsafeRawPointer(d2)
// step 3 create a temp data object
let d4 = Data(bytes:d3, count: MemoryLayout<CutEntry>.size )
// step 4 add the temp to main data object
data.append(d4)
}
return data
}
Earlier when we only had NSMutableData it was
let item = NSMutableData()
for entry in cutsArray
{
var entryCopy = CutEntry(cutPts: entry.cutPts.bigEndian, cutType: entry.cutType.bigEndian)
item.append(&entryCopy, length: MemoryLayout<CutEntry>.size)
}
I've spent a few hours searching for examples of manipulating struct and Data objects. I though that I was close when I found references to unsafebufferpointer. That blew up in my face when I discovered that "buffer" bit uses core memory alignment (which can be useful) and it was stuffing 16 bytes into the data object instead of the expected 12.
I am quite prepared to say that I have missed the blindingly obvious bit of RTFM somewhere. Can anyone offer a cleaner solution ? or has Swift really gone backwards here ?
If I could find a way of getting a pointer to the item as a UInt8 pointer that would remove a couple of lines, but that looks just a difficult.
With checking the reference of Data, I can find two things which may be useful for you:
init(bytes: UnsafeRawPointer, count: Int)
func append(Data)
You can write something like this:
var data = Data()
for entry in cutsArray {
var entryCopy = CutEntry(cutPts: entry.cutPts.bigEndian, cutType: entry.cutType.bigEndian)
data.append(Data(bytes: &entryCopy, count: MemoryLayout<CutEntry>.size))
}
I'm attempting to try and use a string input from a method and set that to a variable of a structure, which i then place in a linked list. I didn't include, all of code but I did post constructor and all that good stuff. Now the code is breaking at the lines
node->title = newTitle;
node->isbn = newISBN;
So newTitle is the string input from the method that I'm trying to set to the title variable of the Book structure of the variable node. Now, I'm assuming this has to do with a issue with pointers and trying to set data to them, but I can't figure out a fix/alternative.
Also, I tried using
strcpy(node->title, newTitle)
But that had an issue with converting the string into a list of chars because strcpy only uses a list of characters. Also tried a few other things, but none seemed to pan out, help with an explanation would be appreciated.
struct Book
{
string title;
string isbn;
struct Book * next;
};
//class LinkedList will contains a linked list of books
class LinkedList
{
private:
Book * head;
public:
LinkedList();
~LinkedList();
bool addElement(string title, string isbn);
bool removeElement(string isbn);
void printList();
};
//Constructor
//It sets head to be NULL to create an empty linked list
LinkedList::LinkedList()
{
head = NULL;
}
//Description: Adds an element to the link in alphabetical order, unless book with
same title then discards
// Returns true if added, false otherwise
bool LinkedList::addElement(string newTitle, string newISBN)
{
struct Book *temp;
struct Book *lastEntry = NULL;
temp = head;
if (temp==NULL) //If the list is empty, sets data to first entry
{
struct Book *node;
node = (Book*) malloc(sizeof(Book));
node->title = newTitle;
node->isbn = newISBN;
head = node;
}
while (temp!=NULL)
{
... //Rest of Code
Note that your Book struct is already a linked list implementation, so you don't need the LinkedList class at all, or alternatively you don't need the 'next' element of the struct.
But there's no reason from the last (long) code snippet you pasted to have an error at the lines you indicated. node->title = newTitle should copy the string in newTitle to the title field of the struct. The string object is fixed size so it's not possible to overwrite any buffer and cause a seg fault.
However, there may be memory corruption from something you do further up the code, which doesn't cause an error until later on. The thing to look for is any arrays, including char[], that you might be overfilling. Another idea is you mention you save method parameters. If you copy, it's ok, but if you do something like
char* f() {
char str[20];
strcpy(str, "hello");
return str;
}
...then you've got a problem. (Because str is allocated on the stack and you return only the pointer to a location that won't be valid after the function returns.) Method parameters are local variables.
The answer you seek can be found here.
In short: the memory malloc returns does not contain a properly constructed object, so you can't use it as such. Try using new / delete instead.
I know parts of this issue is covered by some posts here and I have looked at them and tested some but with no luck.
I have this native method signature which should populate the provided CBadgeData structure array with results:
int elc_GetBadges(int nHandle, char* cErr, int* nRecCount, CBadgeData** arr)
The CBadgeData structure is implemented as follows:
package test.elcprog;
import java.util.Arrays;
import java.util.List;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;
public class CBadgeData extends Structure{
public static class ByReference extends CBadgeData implements Structure.ByReference { }
public int nBadgeID, nTrigger, nExtraData;
public String cName;
public CBadgeData(Pointer pointer){
super(pointer);
}
public CBadgeData(){ }
public String ToString() {
return nBadgeID + "," + nTrigger + "," + nExtraData + "," + cName;
}
#Override
protected List getFieldOrder() {
String[] s = new String[]{"nBadgeID","nTrigger","nExtraData","cName"};
return Arrays.asList(s);
}
}
My last try to craft this argument and call the method looked like this:
CBadgeData.ByReference[] badges = new CBadgeData.ByReference[max_items];
new CBadgeData.ByReference().toArray(badges);
int ret = inst.elc_GetBadges(handle, err, recCount, badges);
It fails with segmentation error.
My Question is what Java type should be provided here as an argument for the native CBadgeData** in the call to elc_GetBadges?
EDIT -1-
Populating the array myself (with or without terminating null pointer) didn't work and caused further Seg crashes. I then used Pointer[] arg as technomage suggested:
Pointer[] pointers = new Pointer[max_items];
for(int i=0; i<max_items; i++){
pointers[i] = new CBadgeData.ByReference().getPointer();
}
int ret = inst.elc_GetBadges(handle, err, recCount, pointers);
This caused no error but seems to not make any changes to the returning struct which should have contain 4 items in this case:
int bid = new CBadgeData(pointers[i]).nBadgeID; // this returns null for all items
Using explicit read() / write() on the struct led to Seg crashes again (on the read):
Any idea what am I still missing here?
EDIT -2-
Interestingly enough - using the Memory.get directly, after calling the native method, gets the correct results:
Memory m= (Memory)pointers[0];
System.out.println("1st int: "+m.getInt(0)); // this gets 24289 which is 5ee1
System.out.println("2nd int: "+m.getInt(4)); // this gets 3
System.out.println("3rd int: "+m.getInt(8)); // this gets 255
System.out.println("String: "+m.getString(12)); // this gets "Badge[5EE1]" as supposed
But the read() still crashes. Any thoughts?
I'm inferring that CBadgeData** input is intended to be an array of pointer to CBadgeData.
As such, the Structure.ByReference tagging is correct.
Structure.toArray() is probably not appropriate here, or at least not necessary (it allocates a contiguous block of structs in memory). You can just populate your array with CBadgeData.ByReference instances.
Perhaps your callee is expecting a NULL pointer at the end of the array? I don't see another indicator of the array length to the callee.
CBadgeData.ByReference[] badges = new CBadgeData.ByReference[max_items+1];
for (int i=0;i < badges.length-1;i++) {
badges[i] = new CBadgeData.ByReference();
}
badges[badges.length-1] = null;
Pretty sure that works. If for whatever reason there's a bug handling Structure.ByReference[], I know that Pointer[] is reliable and will do the same thing.
EDIT
If you use Pointer[] instead of Structure.ByReference[] (please post a bug to the project site if Structure.ByReference[] does not work), you will have to manually call Structure.write/read before/after your native function call, since JNA will not know that the pointers reference structures that need to be synched with native memory. I'd bet, however, that the cause of your crashes when using Structure.ByReference[] was simply that JNA was automatically calling Structure.read() after the call and triggered the same error that you see when calling it explicitly.
If you get a segfault on read, it likely means that your structure fields aren't properly aligned or defined, or (less likely) that you have corrupt data that can't be read properly. To diagnose this, set jna.dump_memory=true and print out your struct after calling Structure.write() to see if the contents of the structure appear as you'd expect. It'd also help to post the native and JNA forms of your structure here, if possible.
Iam have a DBC file, which is a database file for a game, containing ingame usable spell data, like ID, SpellName, Category etc...
Struct is something like this:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
public struct SpellEntry
{
public uint ID;
public uint Category;
public float speed;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8, ArraySubType = UnmanagedType.I4)]
public int[] Reagent;
public int EquippedItemClass;
[MarshalAs(UnmanagedType.LPStr)] // Crash here
public string SpellName;
}
Iam reading the file with a binary reader, and marshaling it to the struct. Snippet:
binReader.BaseStream.Seek(DBCFile.HEADER_SIZE + (index * 4 * 234), SeekOrigin.Begin);
buff = binReader.ReadBytes(buff.Length);
GCHandle handdle = GCHandle.Alloc(buff, GCHandleType.Pinned);
Spell.SpellEntry testspell = (Spell.SpellEntry)Marshal.PtrToStructure(handdle.AddrOfPinnedObject(), typeof(Spell.SpellEntry));
handdle.Free();
Now to be more complex, lets see how does the DBC file storing the strings, for example the SpellName. Its not in the records, strings are contained in the end of the file, in a "string table" block. The string data in the records contains a number (offset) to the string in the string table. (so its not really a string).
I managed to read all the strings from the string block (at the end of the file), to a string[]. (this is dont before start reading the records)
Then I would start reading the records, but first problem Is :
1.) I cant read it, because it "crashes" on the last line of my struct (because its not a string really)
2.) I cant assign a string to the number.
When I read it, it will be a number, but at the end, as a result, I have to assign that string to the SpellName, thats got pointed by the number, in the string table. Jeez .
public struct SpellEntry
{
//...
private int SpellNameOffset;
public string SpellName {
get { return Mumble.GetString(SpellNameOffset); }
}
}
This is hard to get right, Mumble must be a static class since you cannot add any members to SpellEntry. That screws up Marshal.SizeOf(), making it too large. You'll need to initialize Mumble so that its static GetString() method can access the string table. Moving the SpellName property into another class solves the problem but makes the code ugly too.
This is liable to confuse you badly. If you got a version going that uses BitConverter then you're definitely better off by using it instead. Separating the file format from the runtime format is in fact an asset here.