Structure Reading Theory Problem - string

Iam have a DBC file, which is a database file for a game, containing ingame usable spell data, like ID, SpellName, Category etc...
Struct is something like this:
[StructLayout(LayoutKind.Sequential, CharSet = CharSet.Ansi, Pack = 1)]
public struct SpellEntry
{
public uint ID;
public uint Category;
public float speed;
[MarshalAs(UnmanagedType.ByValArray, SizeConst = 8, ArraySubType = UnmanagedType.I4)]
public int[] Reagent;
public int EquippedItemClass;
[MarshalAs(UnmanagedType.LPStr)] // Crash here
public string SpellName;
}
Iam reading the file with a binary reader, and marshaling it to the struct. Snippet:
binReader.BaseStream.Seek(DBCFile.HEADER_SIZE + (index * 4 * 234), SeekOrigin.Begin);
buff = binReader.ReadBytes(buff.Length);
GCHandle handdle = GCHandle.Alloc(buff, GCHandleType.Pinned);
Spell.SpellEntry testspell = (Spell.SpellEntry)Marshal.PtrToStructure(handdle.AddrOfPinnedObject(), typeof(Spell.SpellEntry));
handdle.Free();
Now to be more complex, lets see how does the DBC file storing the strings, for example the SpellName. Its not in the records, strings are contained in the end of the file, in a "string table" block. The string data in the records contains a number (offset) to the string in the string table. (so its not really a string).
I managed to read all the strings from the string block (at the end of the file), to a string[]. (this is dont before start reading the records)
Then I would start reading the records, but first problem Is :
1.) I cant read it, because it "crashes" on the last line of my struct (because its not a string really)
2.) I cant assign a string to the number.
When I read it, it will be a number, but at the end, as a result, I have to assign that string to the SpellName, thats got pointed by the number, in the string table. Jeez .

public struct SpellEntry
{
//...
private int SpellNameOffset;
public string SpellName {
get { return Mumble.GetString(SpellNameOffset); }
}
}
This is hard to get right, Mumble must be a static class since you cannot add any members to SpellEntry. That screws up Marshal.SizeOf(), making it too large. You'll need to initialize Mumble so that its static GetString() method can access the string table. Moving the SpellName property into another class solves the problem but makes the code ugly too.
This is liable to confuse you badly. If you got a version going that uses BitConverter then you're definitely better off by using it instead. Separating the file format from the runtime format is in fact an asset here.

Related

constant expression required while using constants

private static int counter = 0; public static final int
CLIENT_REQUEST_TIME = counter++;
...
switch (command) { case Command.CLIENT_REQUEST_TIME:
...
but here it comes the
"Error:(20, 25) java: constant expression required",
for the case statement.
but why on earth? since CLIENT_REQUEST_TIME is constant
I know this is something that should be rather addressed to Oracle and I can choose from millions of workarounds... but just in case someone can see some logic behind, it will certainly make me sleep better tonight :)
This error comes because you set not constant property to constant field. I am not understand what this code must to do, but possibly you can change it to look like this:
private static int counter = 0;
public static int CLIENT_REQUEST_TIME = counter++;

Struct store large chunk of data

I have a data file with millions of rows and I wanted to read that and store in a struct.
public struct Sample
{
public int A;
public DateTime B;
}
Sample[] sample = new Sample[];
This definition gives me this error "Wrong number of indicies inside[]; expected 1"
How do I store data in struct (with less memory usage)? Array is that best of something else?
var reader = new StreamReader(File.OpenRead(#"C:\test.csv"));
while (!reader.EndOfStream)
{
var line = reader.ReadLine();
var values = line.Split(';');
}
In order to allocate an array, you need to specify how many elements it holds. That's what the error is telling you.
If you don't know the size, you can use List<T> which will grow as needed. Internally List<T> is implemented using T[] which means that from a look-up stand point it acts like an array. However, the work of reallocating larger arrays as needed is handled by List<T>.

C++ : Strings, Structures and Access Violation Writing Locations

I'm attempting to try and use a string input from a method and set that to a variable of a structure, which i then place in a linked list. I didn't include, all of code but I did post constructor and all that good stuff. Now the code is breaking at the lines
node->title = newTitle;
node->isbn = newISBN;
So newTitle is the string input from the method that I'm trying to set to the title variable of the Book structure of the variable node. Now, I'm assuming this has to do with a issue with pointers and trying to set data to them, but I can't figure out a fix/alternative.
Also, I tried using
strcpy(node->title, newTitle)
But that had an issue with converting the string into a list of chars because strcpy only uses a list of characters. Also tried a few other things, but none seemed to pan out, help with an explanation would be appreciated.
struct Book
{
string title;
string isbn;
struct Book * next;
};
//class LinkedList will contains a linked list of books
class LinkedList
{
private:
Book * head;
public:
LinkedList();
~LinkedList();
bool addElement(string title, string isbn);
bool removeElement(string isbn);
void printList();
};
//Constructor
//It sets head to be NULL to create an empty linked list
LinkedList::LinkedList()
{
head = NULL;
}
//Description: Adds an element to the link in alphabetical order, unless book with
same title then discards
// Returns true if added, false otherwise
bool LinkedList::addElement(string newTitle, string newISBN)
{
struct Book *temp;
struct Book *lastEntry = NULL;
temp = head;
if (temp==NULL) //If the list is empty, sets data to first entry
{
struct Book *node;
node = (Book*) malloc(sizeof(Book));
node->title = newTitle;
node->isbn = newISBN;
head = node;
}
while (temp!=NULL)
{
... //Rest of Code
Note that your Book struct is already a linked list implementation, so you don't need the LinkedList class at all, or alternatively you don't need the 'next' element of the struct.
But there's no reason from the last (long) code snippet you pasted to have an error at the lines you indicated. node->title = newTitle should copy the string in newTitle to the title field of the struct. The string object is fixed size so it's not possible to overwrite any buffer and cause a seg fault.
However, there may be memory corruption from something you do further up the code, which doesn't cause an error until later on. The thing to look for is any arrays, including char[], that you might be overfilling. Another idea is you mention you save method parameters. If you copy, it's ok, but if you do something like
char* f() {
char str[20];
strcpy(str, "hello");
return str;
}
...then you've got a problem. (Because str is allocated on the stack and you return only the pointer to a location that won't be valid after the function returns.) Method parameters are local variables.
The answer you seek can be found here.
In short: the memory malloc returns does not contain a properly constructed object, so you can't use it as such. Try using new / delete instead.

JNA - Use structure array as byref argument

I know parts of this issue is covered by some posts here and I have looked at them and tested some but with no luck.
I have this native method signature which should populate the provided CBadgeData structure array with results:
int elc_GetBadges(int nHandle, char* cErr, int* nRecCount, CBadgeData** arr)
The CBadgeData structure is implemented as follows:
package test.elcprog;
import java.util.Arrays;
import java.util.List;
import com.sun.jna.Pointer;
import com.sun.jna.Structure;
public class CBadgeData extends Structure{
public static class ByReference extends CBadgeData implements Structure.ByReference { }
public int nBadgeID, nTrigger, nExtraData;
public String cName;
public CBadgeData(Pointer pointer){
super(pointer);
}
public CBadgeData(){ }
public String ToString() {
return nBadgeID + "," + nTrigger + "," + nExtraData + "," + cName;
}
#Override
protected List getFieldOrder() {
String[] s = new String[]{"nBadgeID","nTrigger","nExtraData","cName"};
return Arrays.asList(s);
}
}
My last try to craft this argument and call the method looked like this:
CBadgeData.ByReference[] badges = new CBadgeData.ByReference[max_items];
new CBadgeData.ByReference().toArray(badges);
int ret = inst.elc_GetBadges(handle, err, recCount, badges);
It fails with segmentation error.
My Question is what Java type should be provided here as an argument for the native CBadgeData** in the call to elc_GetBadges?
EDIT -1-
Populating the array myself (with or without terminating null pointer) didn't work and caused further Seg crashes. I then used Pointer[] arg as technomage suggested:
Pointer[] pointers = new Pointer[max_items];
for(int i=0; i<max_items; i++){
pointers[i] = new CBadgeData.ByReference().getPointer();
}
int ret = inst.elc_GetBadges(handle, err, recCount, pointers);
This caused no error but seems to not make any changes to the returning struct which should have contain 4 items in this case:
int bid = new CBadgeData(pointers[i]).nBadgeID; // this returns null for all items
Using explicit read() / write() on the struct led to Seg crashes again (on the read):
Any idea what am I still missing here?
EDIT -2-
Interestingly enough - using the Memory.get directly, after calling the native method, gets the correct results:
Memory m= (Memory)pointers[0];
System.out.println("1st int: "+m.getInt(0)); // this gets 24289 which is 5ee1
System.out.println("2nd int: "+m.getInt(4)); // this gets 3
System.out.println("3rd int: "+m.getInt(8)); // this gets 255
System.out.println("String: "+m.getString(12)); // this gets "Badge[5EE1]" as supposed
But the read() still crashes. Any thoughts?
I'm inferring that CBadgeData** input is intended to be an array of pointer to CBadgeData.
As such, the Structure.ByReference tagging is correct.
Structure.toArray() is probably not appropriate here, or at least not necessary (it allocates a contiguous block of structs in memory). You can just populate your array with CBadgeData.ByReference instances.
Perhaps your callee is expecting a NULL pointer at the end of the array? I don't see another indicator of the array length to the callee.
CBadgeData.ByReference[] badges = new CBadgeData.ByReference[max_items+1];
for (int i=0;i < badges.length-1;i++) {
badges[i] = new CBadgeData.ByReference();
}
badges[badges.length-1] = null;
Pretty sure that works. If for whatever reason there's a bug handling Structure.ByReference[], I know that Pointer[] is reliable and will do the same thing.
EDIT
If you use Pointer[] instead of Structure.ByReference[] (please post a bug to the project site if Structure.ByReference[] does not work), you will have to manually call Structure.write/read before/after your native function call, since JNA will not know that the pointers reference structures that need to be synched with native memory. I'd bet, however, that the cause of your crashes when using Structure.ByReference[] was simply that JNA was automatically calling Structure.read() after the call and triggered the same error that you see when calling it explicitly.
If you get a segfault on read, it likely means that your structure fields aren't properly aligned or defined, or (less likely) that you have corrupt data that can't be read properly. To diagnose this, set jna.dump_memory=true and print out your struct after calling Structure.write() to see if the contents of the structure appear as you'd expect. It'd also help to post the native and JNA forms of your structure here, if possible.

NUnit - how to compare strings containing composite Unicode characters?

I'm using NUnit v2.5 to compare strings that contain composite Unicode characters.
Although comparison itself works fine, a caret indicating first difference seems to be misplaced.
UPD: I've ended up with overridden EqualConstraint that in turn invokes a custom TextMessageWriter, so I no longer need an answer. See for solution below.
Here's the snippet:
string s1 = "ใช้งานง่าย";
string s2 = "ใช้งานงาย";
Assert.That(s1, Is.EqualTo(s2));
Here's the output:
Expected: "ใช้งานงาย"
But was: "ใช้งานง่าย"
------------------^
The arrow indicating first different character seems to be off 2 positions (as many as there are tone marks above). For longer strings, it becomes a real pain.
I have attempted String.Normalize() but it wouldn't work either.
How can I overcome this problem? Thanks for your help. See my answer below.
When you are comparing Unicode strings, you must always normalize both sides of the comparison, and in the same way. It is not good enough to do binary compare of s1 and s2, because canonically equivalent strings would not test binary equivalent.
Positing the existence of four trivial normalization function, one for each of the four normalization forms, you would want to test NFD(s1) for binary eqality to NFD(s2). It doesn't matter whether you use NFD or NFC there, but you must do the same thing to both strings.
For the k-compat functions, NFKD and NFKD, those are useful when doing string searching, because they improve the recall at the cost of some precision. For example NFKD("™") would be equal to NFKD("TM"). This is what Adobe Reader does, for example, when you run searches on documents: it always runs the search in k-compat mode, so that your searches have a better chance at finding things. However, unlike NFC and NFD, the k-compat functions NFKC and NFKD lose information and are not reversible. With simple NFD and NFC, though, you can always get back to the other one.
You should be able to use the code from this answer to convert each string to an escaped version of the original string. Composite characters will become a single escaped \u codepoint, while combining characters will be a series of such escapes. Then run your Assert on these escaped versions of the string.
I think I cannot find any better answer, so answering my own question.
Cause.
There are many languages using non-spacing modifiers for characters. For European languages, there are substitutions, e.g. "u" (U+0075) + "¨" (U+00A8) = "ü" (U+00FC). In this case, solution by #tchrist is quite sufficient.
However, for complex writing systems, there is no substitution for non-spacing modifiers. Therefore, NUnit's TextMessageWriter.WriteCaretLine(int mismatch) treats mismatch parameter as a byte offset, while screen representation of Thai string may be shorter than the length of caret line ("-----^").
Solution.
Force WriteCaretLine(int mismatch) to respect non-spacing modifiers, reducing mismatch value to the number of non-spacing modifiers occurred before this offset.
Implement all supplementary classes that are actually needed only to make your new code invoked.
Along with Thai, I have tested it with Devanagari and Tibetan. It works as expected.
Yet another pitfall. If you're using NUnit with Visual Studio through ReSharper like I do, you have to configure your Internet Explorer's fonts (it cannot be managed with R#) so that it used proper monospaced fonts for Thai, Devanagari, etc.
Implementation.
Inherit TextMessageWriter and override its DisplayStringDifferences;
Implement your own ClipExpectedAndActual and FindMismatchPosition - here are non-spacing modifiers are respected; Proper clipping is needed since it may also impact calculation of non-spacing elements.
Inherit EqualConstraint and override its WriteMessageTo(MessageWriter writer) so that your MessageWriter was used;
Optionally, create a custom wrapper for simple invocation of custom constraint.
The source code goes below. About 80% of the code doesn't do anything useful, but it's included due to access levels in original code.
// Step 1.
public class ThaiMessageWriter : TextMessageWriter
{
/// <summary>
/// This method is merely a copy of the original method taken from NUnit sources,
/// except that it changes meaning of <paramref name="mismatch"/> before the caret line is displayed.
/// <remarks>
/// Originally passed <paramref name="mismatch"/> contains byte offset, while proper display of caret requires
/// it position to be calculated in character placeholder units. They are different in case of
/// over- or under-string Unicode characters like acute mark or complex script (Thai)
/// </remarks>
/// </summary>
/// <param name="clipping"></param>
public override void DisplayStringDifferences(string expected, string actual, int mismatch, bool ignoreCase, bool clipping)
{
// Maximum string we can display without truncating
int maxDisplayLength = MaxLineLength
- PrefixLength // Allow for prefix
- 2; // 2 quotation marks
int mismatchOffset = mismatch;
if (clipping)
MsgUtils2.ClipExpectedAndActual(ref expected, ref actual, maxDisplayLength, mismatchOffset);
expected = MsgUtils.EscapeControlChars(expected);
actual = MsgUtils.EscapeControlChars(actual);
// The mismatch position may have changed due to clipping or white space conversion
int mismatchInCharPlaceholders = MsgUtils2.FindMismatchPosition(expected, actual, 0, ignoreCase);
Write(Pfx_Expected);
WriteExpectedValue(expected);
if (ignoreCase)
WriteModifier("ignoring case");
WriteLine();
WriteActualLine(actual);
//DisplayDifferences(expected, actual);
if (mismatch >= 0)
WriteCaretLine(mismatchInCharPlaceholders);
}
// Copied due to private
/// <summary>
/// Write the generic 'Actual' line for a constraint
/// </summary>
/// <param name="constraint">The constraint for which the actual value is to be written</param>
private void WriteActualLine(Constraint constraint)
{
Write(Pfx_Actual);
constraint.WriteActualValueTo(this);
WriteLine();
}
// Copied due to private
/// <summary>
/// Write the generic 'Actual' line for a given value
/// </summary>
/// <param name="actual">The actual value causing a failure</param>
private void WriteActualLine(object actual)
{
Write(Pfx_Actual);
WriteActualValue(actual);
WriteLine();
}
// Copied due to private
private void WriteCaretLine(int mismatch)
{
// We subtract 2 for the initial 2 blanks and add back 1 for the initial quote
WriteLine(" {0}^", new string('-', PrefixLength + mismatch - 2 + 1));
}
}
// Step 2.
public static class MsgUtils2
{
private static readonly string ELLIPSIS = "...";
/// <summary>
/// Almost a copy of MsgUtil.ClipExpectedAndActual method
/// </summary>
/// <param name="expected"></param>
/// <param name="actual"></param>
/// <param name="maxDisplayLength"></param>
/// <param name="mismatch"></param>
public static void ClipExpectedAndActual(ref string expected, ref string actual, int maxDisplayLength, int mismatch)
{
// Case 1: Both strings fit on line
int maxStringLength = Math.Max(expected.Length, actual.Length);
if (maxStringLength <= maxDisplayLength)
return;
// Case 2: Assume that the tail of each string fits on line
int clipLength = maxDisplayLength - ELLIPSIS.Length;
int clipStart = maxStringLength - clipLength;
// Case 3: If it doesn't, center the mismatch position
if (clipStart > mismatch)
clipStart = Math.Max(0, mismatch - clipLength / 2);
// shift both clipStart and maxDisplayLength if they split non-placeholding symbol
AdjustForNonPlaceholdingCharacter(expected, ref clipStart);
AdjustForNonPlaceholdingCharacter(expected, ref maxDisplayLength);
expected = MsgUtils.ClipString(expected, maxDisplayLength, clipStart);
actual = MsgUtils.ClipString(actual, maxDisplayLength, clipStart);
}
private static void AdjustForNonPlaceholdingCharacter(string expected, ref int index)
{
while (index > 0 && CharUnicodeInfo.GetUnicodeCategory(expected[index]) == UnicodeCategory.NonSpacingMark)
{
index--;
}
}
static public int FindMismatchPosition(string expected, string actual, int istart, bool ignoreCase)
{
int length = Math.Min(expected.Length, actual.Length);
string s1 = ignoreCase ? expected.ToLower() : expected;
string s2 = ignoreCase ? actual.ToLower() : actual;
int iSpacingCharacters = 0;
for (int i = 0; i < istart; i++)
{
if (CharUnicodeInfo.GetUnicodeCategory(s1[i]) != UnicodeCategory.NonSpacingMark)
iSpacingCharacters++;
}
for (int i = istart; i < length; i++)
{
if (s1[i] != s2[i])
return iSpacingCharacters;
if (CharUnicodeInfo.GetUnicodeCategory(s1[i]) != UnicodeCategory.NonSpacingMark)
iSpacingCharacters++;
}
//
// Strings have same content up to the length of the shorter string.
// Mismatch occurs because string lengths are different, so show
// that they start differing where the shortest string ends
//
if (expected.Length != actual.Length)
return length;
//
// Same strings : We shouldn't get here
//
return -1;
}
}
// Step 3.
public class ThaiEqualConstraint : EqualConstraint
{
private readonly string _expected;
// WTF expected is private?
public ThaiEqualConstraint(string expected) : base(expected)
{
_expected = expected;
}
public override void WriteMessageTo(MessageWriter writer)
{
// redirect output to customized MessageWriter
var myMessageWriter = new ThaiMessageWriter();
base.WriteMessageTo(myMessageWriter);
writer.Write(myMessageWriter);
}
}
// Step 4.
public static class ThaiText
{
public static EqualConstraint IsEqual(string expected)
{
return new ThaiEqualConstraint(expected);
}
}

Resources