I would like to receive the vehicle's CAN data to the socket CAN and send it to the server. (raw data)
Accordingly, CAN ID is very important in post-processing raw data sent to the server.
The CAN data ID received by Candump and wireshark is only 3 digits, so it is difficult to distinguish which data it is.
For example, although the CAN ID I need to receive is 0x1f532d10, only d10 is received and displayed from socketcan.
Is there any way to get the ID length as a whole?
If you're using SocketCAN to send frames you need to set the "Extended Frame Format" bit in the can_id field of the can_frame struct, e.g.:
frame.can_id |= CAN_EFF_FLAG;
I see that socket.io supports binary data. To send I could just set a Buffer object.
I want to send / receive a large number medium size files. I want to see if it can be optimized. When creating a Buffer from file and sending via socket.io, does it internally create any copy is the data or is it handled with zero-copy?
Similarly, when receiving, is it possible to receive the data as a Buffer that can be written to a file without creating a copy? I couldn't find an example of receiving data as a Buffer. Can someone point out examples of receiving binary data as a Buffer?
How can I generate an exception when the client writes a number out of range to my server? Also, I would like some vairables on my server to be read-only and other read / write. Any recommendation?
Pymodbus generally handles out of bound register access internally, a request to read/write a data block which is out of range would raise ExceptionResponse('Invalid Address') . On the client side , you can check if the received response is exception response by either checking isError() method on the response (pymodbus v1.5.0 and above).
Refer synchronous_server and synchronous_client
Also by default, Discrete Inputs and Input Registers are Read only and Holding registers and Coils and Read/Write. Any attempt to write to the read only datablocks raises an ExceptionResponse.
If you have specific requirements to make a range of holding registers/coils read only you can have a look at the various datablocks , to create a custom datablock refer custom datablock example
I am writing an application server that processes images (large data). I am trying to minimize copies when sending image data back to clients. The processed images I need to send to clients are in buffers obtained from jemalloc. The ways I have thought of sending the data back to the client is:
1) Simple write call.
// Allocate buffer buf.
// Store image data in this buffer.
write(socket, buf, len);
2) I obtain the buffer through mmap instead of jemalloc, though I presume jemalloc already creates the buffer using mmap. I then make a simple call to write.
buf = mmap(file, len); // Imagine proper options.
// Store image data in this buffer.
write(socket, buf, len);
3) I obtain a buffer through mmap like before. I then use sendfile to send the data:
buf = mmap(in_fd, len); // Imagine proper options.
// Store image data in this buffer.
int rc;
rc = sendfile(out_fd, file, &offset, count);
// Deal with rc.
It seems like (1) and (2) will probably do the same thing given jemalloc probably allocates memory through mmap in the first place. I am not sure about (3) though. Will this really lead to any benefits? Figure 4 on this article on Linux zero-copy methods suggests that a further copy can be prevented using sendfile:
no data is copied into the socket buffer. Instead, only descriptors
with information about the whereabouts and length of the data are
appended to the socket buffer. The DMA engine passes data directly
from the kernel buffer to the protocol engine, thus eliminating the
remaining final copy.
This seems like a win if everything works out. I don't know if my mmaped buffer counts as a kernel buffer though. Also I don't know when it is safe to re-use this buffer. Since the fd and length is the only thing appended to the socket buffer, I assume that the kernel actually writes this data to the socket asynchronously. If it does what does the return from sendfile signify? How would I know when to re-use this buffer?
So my questions are:
What is the fastest way to write large buffers (images in my case) to a socket? The images are held in memory.
Is it a good idea to call sendfile on a mmapped file? If yes, what are the gotchas? Does this even lead to any wins?
It seems like my suspicions were correct. I got my information from this article. Quoting from it:
Also these network write system calls, including sendfile, might and
in many cases do return before the data sent over TCP by the method
call has been acknowledged. These methods return as soon as all data
is written into the socket buffers (sk buff) and is pushed to the TCP
write queue, the TCP engine can manage alone from that point on. In
other words at the time sendfile returns the last TCP send window is
not actually sent to the remote host but queued. In cases where
scatter-gather DMA is supported there is no seperate buffer which
holds these bytes, rather the buffers(sk buffs) just hold pointers to
the pages of OS buffer cache, where the contents of file is located.
This might lead to a race condition if we modify the content of the
file corresponding to the data in the last TCP send window as soon as
sendfile is returned. As a result TCP engine may send newly written
data to the remote host instead of what we originally intended to
send.
Provided the buffer from a mmapped file is even considered "DMA-able", seems like there is no way to know when it is safe to re-use it without an explicit acknowledgement (over the network) from the actual client. I might have to stick to simple write calls and incur the extra copy. There is a paper (also from the article) with more details.
Edit: This article on the splice call also shows the problems. Quoting it:
Be aware, when splicing data from a mmap'ed buffer to a network
socket, it is not possible to say when all data has been sent. Even if
splice() returns, the network stack may not have sent all data yet. So
reusing the buffer may overwrite unsent data.
For cases 1 and 2 - does the operation you marked as // Store image data in this buffer require any conversion? Is it just plain copy from the memory to buf?
If it's just plain copy, you can use write directly on the pointer obtained from jemalloc.
Assuming that img is a pointer obtained from jemalloc and size is a size of your image, just run following code:
int result;
int sent=0;
while(sent<size) {
result=write(socket,img+sent,size-sent);
if(result<0) {
/* error handling here */
break;
}
sent+=result;
}
It is working correctly for blocking I/O (the default behavior). If you need to write a data in a non-blocking manner, you should be able to rework the code on your own, but now you have the idea.
For case 3 - sendfile is for sending data from one descriptor to another. That means you can, for example, send data from file directly to tcp socket and you don't need to allocate any additional buffer. So, if the image you want to send to a client is in a file, just go for a sendfile. If you have it in memory (because you processed it somehow, or just generated), use the approach I mentioned earlier.
I have bunch of records in my offcard application and I want to save them all in javacard,
The question is:
What is the best way of transferring data to Java Card?
Should I transfer all data record by record (each one with a APDU) or send all the records in just one APDU?
Of course I know the limitation size of APDU and I'm using extended APDU in order to send all data just in one extended APDU which is more than 255 bytes..
It does not matter much if you send your data in one extended length APDU or one single APDU security wise. It is however much better to send unrelated information using separate APDU's. This would make your application much more modular. Note that if you send related information using separate APDU's, you may need to keep state between those APDU's for validation purposes (e.g. you may have to send either none or all of them, or send the APDU's in specific order).
Furthermore, ISO 7816-4 only defines 2 byte status words to send back to the sender, e.g. 8A80 to indicate any error in the command data. This means that it is impossible to tell from the status word which of the records contains failure information.
Finally, there are certainly still readers and software out there that have issues handling extended length APDU's. So if your software is going to be used by other parties you may want to stick to normal length APDU's.