I have created the following C library for reading an image:
typedef struct {
unsigned int height;
unsigned int width;
unsigned char* red; //length=height*width
unsigned char* green;
unsigned char* blue;
} Contents;
Contents readJPEGFile(const char* inFilename);
I can't really find any info using arrays and structs with the Foreign Function Interface.
How would I proceed to be able to use my library in Haskell?
I tried to use the following example as a base: http://therning.org/magnus/archives/315 but then the hsc file was compiled down to a hs file that only contained the above c-code and nothing more (and of course it can't be compiled).
The basic FFI support includes only scalar types. Everything else you wind up doing with address arithmetic. The section on foreign types in the FFI documentation gives the basics, and you can find an example in the FFI Cookbook.
At one time you could use tools like Green Card and H/Direct to generate marshalling and unmarshalling code for you. For reasons I don't understand, these tools have not been updated in a long time. As far as I can tell the current tool of choice is hsc2hs.
Edit: As noted in comment (thanks ephemient), c2hs is also popular, and since c2hs is from Manuel Chakravarty it is likely to be good.
It sounds as if you have a build issue; I do seem to recall that I used the very page you reference as an example when I was writing an FFI interface into the Windows Win32 DDEML library. For example, one of the structures we use is
typedef struct tagHSZPAIR {
HSZ hszSvc;
HSZ hszTopic;
} HSZPAIR, *PHSZPAIR;
#include "ddeml.h" brings this in to the DDEML.hsc file. We access it with:
data HSZPair = HSZPair HSZ HSZ
instance Storable HSZPair where
sizeOf _ = (#size HSZPAIR)
alignment = sizeOf
peek ptr = do svc <- (#peek HSZPAIR, hszSvc) ptr
topic <- (#peek HSZPAIR, hszTopic) ptr
return $ HSZPair svc topic
poke ptr (HSZPair svc topic) = do (#poke HSZPAIR, hszSvc) ptr svc
(#poke HSZPAIR, hszTopic) ptr topic
Unfortunately, I can't show you what this compiles to at the moment because I don't have a Windows box handy, but the generated code was just as above, except with #size HSZPAIR replaced with (64) or whatever and so on.
(If you really want to see what was generated, or need help doing your build, e-mail me and I'll help you out.)
Hackage has several packages which use FFI which you could look at for examples.
Related
I've got the following piece of code which failed to compile on i386 with "gs" missing.
I've looked at the struct definition, and it's clearly there.
Any idea what I got wrong?
Thanks!
struct user_regs_struct regs_struct;
struct iovec pt_iov = {
.iov_base = ®s,
.iov_len = sizeof(regs),
};
if (ptrace(PTRACE_GETREGSET, tid, NT_PRSTATUS, &pt_iov) == 0) {
#if defined(__x86_64__)
return regs_struct.fs;
#elif defined(__i386__)
return regs_struct.gs; <<< Got an error about "gs" not being a field of user_regs_struct
}
P.S: I know I should produce a small test case but I could not. It didn't hit this error on a standalone app. (never mind that I don't have the hardware to test it locally). All I knew was this error popped up when the code is part of a larger system, being built remotely.
That's why I was hoping maybe somebody recognized this as a "known-issue" or have some intuition as to what might be the issue.
Looks like the i386 version of user_regs_struct calls it xgs for some reason.
In sys/user.h, there's an #ifdef __x86_64__. The #else
/* These are the 32-bit x86 structures. */ side of the file has this content:
struct user_regs_struct
{
long int ebx;
long int ecx;
long int edx;
long int esi;
long int edi;
long int ebp;
long int eax;
long int xds;
long int xes;
long int xfs;
long int xgs;
long int orig_eax;
long int eip;
long int xcs;
long int eflags;
long int esp;
long int xss;
};
Perhaps that changed in some glibc version? This is on x86-64 Arch GNU/Linux, so those are plain vanilla glibc headers (taken from the Linux kernel).
ack user_regs_struct /usr/include found the right file right away. (like grep -r).
Note the top of the file says:
/* The whole purpose of this file is for GDB and GDB only. Don't read
too much into it. Don't use it for anything other than GDB unless
you know what you are doing. */
I don't know exactly why there's such a stern warning, or if they really mean for ptrace in general. If it's more than that, I'd be cautious about using it blindly if reading the header and checking the struct member names wasn't obvious to you. Maybe it's fine, maybe it's not, just saying that I wouldn't rely on code you write using it for anything critical without more research.
I am working in a legacy codebase with a large amount of Objective-C++ written using manual retain/release. Memory is managed using lots of C++ std::shared_ptr<NSMyCoolObjectiveCPointer>, with a suitable deleter passed in on construction that calls release on the contained object. This seems to work great; however, when enabling UBSan, it complains about misaligned pointers, usually when dereferencing the shared_ptrs to do some work.
I've searched for clues and/or solutions, but it's difficult to find technical discussion of the ins and outs of Objective-C object pointers, and even more difficult to find any discussion about Objective-C++, so here I am.
Here is a full Objective-C++ program that demonstrates my problem. When I run this on my Macbook with UBSan, I get a misaligned pointer issue in shared_ptr::operator*:
#import <Foundation/Foundation.h>
#import <memory>
class DateImpl {
public:
DateImpl(NSDate* date) : _date{[date retain], [](NSDate* date) { [date release]; }} {}
NSString* description() const { return [&*_date description]; }
private:
std::shared_ptr<NSDate> _date;
};
int main(int argc, const char * argv[]) {
#autoreleasepool {
DateImpl date{[NSDate distantPast]};
NSLog(#"%#", date.description());
return 0;
}
}
I get this in the call to DateImpl::description:
runtime error: reference binding to misaligned address 0xe2b7fda734fc266f for type 'std::__1::shared_ptr<NSDate>::element_type' (aka 'NSDate'), which requires 8 byte alignment
0xe2b7fda734fc266f: note: pointer points here
<memory cannot be printed>
I suspect that there is something awry with the usage of &* to "cast" the shared_ptr<NSDate> to an NSDate*. I think I could probably work around this issue by using .get() on the shared_ptr instead, but I am genuinely curious about what is going on. Thanks for any feedback or hints!
There were some red herrings here: shared_ptr, manual retain/release, etc. But I ended up discovering that even this very simple code (with ARC enabled) causes the ubsan hit:
#import <Foundation/Foundation.h>
int main(int argc, const char * argv[]) {
#autoreleasepool {
NSDate& d = *[NSDate distantPast];
NSLog(#"%#", &d);
}
return 0;
}
It seems to simply be an issue with [NSDate distantPast] (and, incidentally, [NSDate distantFuture], but not, for instance, [NSDate date]). I conclude that these must be singleton objects allocated sketchily/misaligned-ly somewhere in the depths of Foundation, and when you dereference them it causes a misaligned pointer read.
(Note it does not happen when the code is simply NSLog(#"%#", &*[NSDate distantPast]). I assume this is because the compiler simply collapses &* on a raw pointer into a no-op. It doesn't for the shared_ptr case in the original question because shared_ptr overloads operator*. Given this, I believe there is no easy way to make this happen in pure Objective-C, since you can't separate the & operation from the * operation, like you can when C++ references are involved [by storing the temporary result of * in an NSDate&].)
You are not supposed to ever use a "bare" NSDate type. Objective-C objects should always be used with a pointer-to-object type (e.g. NSDate *), and you are never supposed to get the "type behind the pointer".
In particular, on 64-bit platforms, Objective-C object pointers can sometimes not be valid pointers, but rather be "tagged pointers" which store the "value" of the object in certain bits of the pointer, rather than as an actual allocated object. You must always let the Objective-C runtime machinery deal with Objective-C object pointers. Dereferencing it as a regular C/C++ pointer can lead to undefined behavior.
I am using Cython for fast parallel processing of data, adding items to a shared memory linked list from multiple threads. I use __sync_bool_compare_and_swap, which provides an atomic compare and swap (CAS) operation to compare if the value was not modified (by another thread) before replacing it with a new value.
cdef extern int __sync_bool_compare_and_swap (void **ptr, void *oldval, void *newval) nogil
cdef bint firstAttempt = 1
cdef type *next = NULL
cdef type *newlink = ....
while firstAttempt or not __sync_bool_compare_and_swap( <void**> c, <void*>next, <void*>newlink):
firstAttempt = 0
next = c[0]
newlink.next = next
This works very well. However, now I also want to keep track of the size of the linked list, and want to use the same CAS function for the updates, however, this time it is not pointers that need to be updated but an int. How can use the same external function twice in Cython, once with void** parameter and once with an int* parameter?
EDIT
What I have in mind is two separate atomic operations, in one atomic operation I want to update the linked list, in the other I want to update the size. You can do it in C, but for Cython it means you have to reference the same external function twice with different parameters, can that be done?
CONCLUSION
The answer suggested by DavidW works. In case anyone is thinking to use a similar construction, you should be aware that when using two separate update functions there is no guarantee these are processed in sequence (i.e. another thread can update in between), however, if the objective is to update a cumulative value for instance to monitor progress while multithreading or to create an aggregated result that is not used until all threads are finished, CAS does guarantee that all updates are done exactly once. Unexpectedly, gcc refuses to compile without casting to void*, so either define separate hard-typed versions, or you need to cast. A snippet from my code:
in some_header.h:
#define sync_bool_compare_and_swap_int __sync_bool_compare_and_swap
#define sync_bool_compare_and_swap_vp __sync_bool_compare_and_swap
in some_prog.pxd:
cdef extern from "some_header.h":
cdef extern int sync_bool_compare_and_swap_vp (void **ptr, void *oldval, void *newval) nogil
cdef extern int sync_bool_compare_and_swap_int (int *ptr, int oldval, int newval) nogil
in some_prog.pyx:
cdef void updateInt(int *value, int incr) nogil:
cdef cINT previous = value[0]
cdef cINT updated = previous + incr
while not sync_bool_compare_and_swap_int( c, previous, updated):
previous = value[0]
updated = previous + incr
So the issue (as I understand it) is that it's __sync_bool_compare_and_swap is a compiler intrinsic rather than a function, so doesn't really have a fixed signature, because the compiler just figures it out. However, Cython demands to know the types, and because you want to use it with two different types, you have a problem.
I can't see a simpler way than resorting to a (very) small amount of C to "help" Cython. Create a header file with a bunch of #defines
/* compare_swap.h */
#define sync_bool_compare_swap_voidp __sync_bool_compare_and_swap
#define sync_bool_compare_swap_int __sync_bool_compare_and_swap
You can then tell Cython that each of these is a separate function
cdef extern from "compare_swap.h":
int sync_bool_compare_swap_voidp(void**, void*, void*)
int sync_bool_compare_swap_int(int*, int, int)
At this stage you should be able to use them naturally as plain functions without any type casting (i.e. no <void**> in your code, since this tends to hide real errors). The C preprocessor will generate the code you want and all is well.
Edit: Looking at this a few years later I can see a couple of simpler ways you could probably use (untested, but I don't see why they shouldn't work). First you could use Cython's ability to map a name to a "cname" to avoid the need for an extra header:
cdef extern from *:
int sync_bool_compare_swap_voidp(void**, void*, void*) "__sync_bool_compare_and_swap"
int sync_bool_compare_swap_int(int*, int, int) "__sync_bool_compare_and_swap"
Second (and probably best) you could use a single generic definition (just telling Cython that it's a varargs function):
cdef extern from "compare_swap.h":
int __sync_bool_compare_and_swap(...)
This way Cython won't try to understand the types used, and will just defer it entirely to C (which is what you want).
I wouldn't like to comment on whether it's safe for you to use two atomic operations like this, or whether that will pass through a state with dangerously inconsistent data....
Can I somehow use inline assembly in Haskell (similar to what GCC does for C)?
I want to compare my Haskell code to the reference implementation (ASM) and this seems the most straightforward way. I guess I could just call Haskell from C and use GCC inline assembly, but I'm still interested if I can do it the other way around.
(I'm on Linux/x86)
There are two ways:
Call C via the FFI, and use inline assembly on the C side.
Write a CMM fragment that calls C (without the FFI), and uses inlined assembly.
Both solutions use inline assembly on the C side. The former is the most idiomatic. Here's an example, from the rdtsc package:
cycles.h:
static __inline__ ticks getticks(void)
{
unsigned int tbl, tbu0, tbu1;
do {
__asm__ __volatile__ ("mftbu %0" : "=r"(tbu0));
__asm__ __volatile__ ("mftb %0" : "=r"(tbl));
__asm__ __volatile__ ("mftbu %0" : "=r"(tbu1));
} while (tbu0 != tbu1);
return (((unsigned long long)tbu0) << 32) | tbl;
}
rdtsc.c:
unsigned long long rdtsc(void)
{
return getticks();
}
rdtsc.h:
unsigned long long rdtsc(void);
rdtsc.hs:
foreign import ccall unsafe "rdtsc.h" rdtsc :: IO Word64
Finally:
A slightly non-obvious solution is to use the LLVM or Harpy packages to call some generated assembly.
I've made a variant type to use instead of boost::variant. Mine works storing an index of the current type on a list of the possible types, and storing data in a byte array with enough space to store the biggest type.
unsigned char data[my_types::max_size];
int type;
Now, when I write a value to this variant type comes the trouble. I use the following:
template<typename T>
void set(T a) {
int t = type_index(T);
if (t != -1) {
type = t;
puts("writing atom data");
*((T *) data) = a; //THIS PART CRASHES!!!!
puts("did it!");
} else {
throw atom_bad_assignment;
}
}
The line that crashes is the one that stores data to the internal buffer. As you can see, I just cast the byte array directly to a pointer of the desired type. This gives me bad address signals and bus errors when trying to write some values.
I'm using GCC on a 64-bit system. How do I set the alignment for the byte array to make sure the address of the array is 64-bit aligned? (or properly aligned for any architecture I might port this project to).
EDIT: Thank you all, but the mistake was somewhere else. Apparently, Intel doesn't really care about alignment. Aligned stuff is faster but not mandatory, and the program works fine this way. My problem was I didn't clear the data buffer before writing stuff and this caused trouble with the constructors of some types. I will not, however, mark the question as answered, so more people can give me tips on alignment ;)
See http://gcc.gnu.org/onlinedocs/gcc-4.0.4/gcc/Variable-Attributes.html
unsigned char data[my_types::max_size] __attribute__ ((aligned));
int type;
I believe
#pragma pack(64)
will work on all modern compilers; it definitely works on GCC.
A more correct solution (that doesn't mess with packing globally) would be:
#pragma pack(push, 64)
// define union here
#pragma pack(pop)