I write server that sends two types of messages: plain message (without encoding) and encrypted message (AES encryption). The transport is UDP. The message content is fixed-length header (2 bytes) and body (JSON string). The question: should I encode these messages using Base64 encoding? If so, what is the reason?
base64encoding has one reason to exist (and this one only): Make something, that is not safe for handling and/or transport in a text-based system (such as E-Mail, classic C strings) safe to do so.
UDP definitly has no such limit, so it depends on whether any other part of your application does. If not, I recommend you use the raw data.
base64 was meant to encode binary data (which is very compact) into group of 64 symbols in the ASCII table (which is less compact). base64 is very good for storing binary data inside text files, things like storing image data inside HTML document etc. I don't see any reason to use it in your case
Related
I have this using Buffer in node.
string hex_message = "7E01000036015211601385025E";
var chunk = Buffer.from(hex_message, 'hex'); //
var message = chunk.toString('utf-8'); //
var bu = Buffer.from(message, 'utf-8');
var message3 = bu.toString('hex').toUpperCase();
// 7E010000360152116013EFBFBD025E
Note that makes 7E010000360152116013EFBFBD025E
I want using string to pass buffer between microservices, and i want to convert all buffer in string, but in the other when I need to rebuild the buffer I got different information.
This step here:
var message = chunk.toString('utf-8');
creates a problem because after calling this:
var chunk = Buffer.from(hex_message, 'hex');
the Buffer contains binary data. If you do console.log(chunk), you will see this:
<Buffer 7e 01 00 00 36 01 52 11 60 13 85 02 5e>
which shows the binary representation of your original hex string:
"7E01000036015211601385025E"
Calling chunk.toString('utf-8') on that Buffer containing the binary data will attempt to interpret it as utf-8, but that won't be correct on purely binary data. In fact, it will take your binary data and try to interpret it as something other than the pure binary data. It will look at the incoming data and see utf-8 escape sequences in some places causing it to interpret the data wrongly.
When you say you want to pass the data between microservices, we need to know how you're passing it between the microservices and what the capabilities of that transport are. For some transports, you can just send the binary Buffer directly (like with TCP). For other transports, you have to encode it to something that transport can handle, in which case the original hex_message string is a safe way to send it. You could also put the binary into an array of numbers and the use JSON to encode/decode it.
The point is that you have to either just send the original hex_message string which is likely safe with any transport you're using or you have to properly encode that to something your transport can handle and then properly decode it on the receiving end to get back the binary it represents.
To summarize, here are the initial steps:
You first need to decide what encoding you're going to use for whatever transport you are using to communicate with the microservice. You need to pick an encoding that can properly represent your binary data. To help you pick that encoding, we would have to know how you're sending the data and what types of encoding it can handle.
Then, you need to take your ascii hex data and properly convert it to the desired encoding.
And here are some options for encoding:
Just send the original hex_message string you started with since the whole purpose of the "hex" encoding is to represent binary data in plain ascii strings. This is likely safe with any transport you're using between microservices. The receiving end can process the ascii string it receives to a binary buffer if required to use it.
Put the binary data into an array of numbers and encode that array as JSON and send the JSON. Though sending the hex string in option #1 will likely be more efficient as it is represented in fewer bytes and is already available.
If your transport is capable of sending/receiving binary (which TCP can), then you can do const chunk = Buffer.from(hex_message, 'hex'); and send the binary data in the Buffer directly. The receiving end will get the binary Buffer and can then treat that according to its needs.
I want to develop some very specific UDP protocol, with following workflow:
It is very simple in design, and very "realtim'ish", in contrast to stubs like HTTP MJPEG streaming technique: it could fully utulize even narrow channel to deliever as much updates as quickly as possible. Drawbacks are that delivery not guranteed, and frame start and end are not determined - with each packet you know just that "some blocks at that coordinates just have been updated"
But I am very new to libjpeg api. How is it possible to encode frame, but then split bitstream, to pack in packet only "integer" amount of DCT-ized and huffman-encoded blocks? (as it stated in article on how JPEG works)
So can reword the task - Given encoded jpeg file; the goal is to split data stream to chunks up to some maximum size in such a way, that each chunk would contain just suffient data to decode it. (No chunk concatenation is allowed)
Sometimes I receive this strange responses from other nodes. Transaction id match to my request transaction id as well as the remote IP so I tend to believe that node responded with this but it looks like sort of a mix of response and request
d1:q9:find_node1:rd2:id20:.éV0özý.?tjN.?.!2:ip4:DÄ.^7:nodes.v26:.ï?M.:iSµLW.Ðä¸úzDÄ.^æCe1:t2:..1:y1:re
Worst of all is that it is malformed. Look at 7:nodes.v it means that I add nodes.v to the dictionary. It is supposed to be 5:nodes. So, I'm lost. What is it?
The internet and remote nodes is unreliable or buggy. You have to code defensively. Do not assume that everything you receive will be valid.
Remote peers might
send invalid bencoding, discard those, don't even try to recover.
send truncated messages. usually not recoverable unless it happens to be the very last e of the root dictionary.
omit mandatory keys. you can either ignore those messages or return an error message
contain corrupted data
include unknown keys beyond the mandatory ones. this is not an error, just treat them as if they weren't there for the sake of forward-compatibility
actually be attackers trying to fuzz your implementation or use you as DoS amplifier
I also suspect that some really shoddy implementations are based on whatever string types their programming language supports and incorrectly handle encoding instead of using arrays of uint8 as bencoding demands. There's nothing that can be done about those. Ignore or occasionally send an error message.
Specified dictionary keys are usually ASCII-mappable, but this is not a requirement. E.g. there are some tracker response types that actually use random binary data as dictionary keys.
Here are a few examples of junk I'm seeing[1] that even fails bdecoding:
d1:ad2:id20:�w)��-��t����=?�������i�&�i!94h�#7U���P�)�x��f��YMlE���p:q9Q�etjy��r7�:t�5�����N��H�|1�S�
d1:e�����������������H#
d1:ad2:id20:�����:��m�e��2~�����9>inm�_hash20:X�j�D��nY��-������X�6:noseedi1ee1:q9:get_peers1:t2:�=1:v4:LT��1:y1:qe
d1:ad2:id20:�����:��m�e��2~�����9=inl�_hash20:X�j�D��nY���������X�6:noseedi1ee1:q9:get_peers1:t2:�=1:v4:LT��1:y1:qe
d1:ad2:id20:�����:��m�e��2~�����9?ino�_hash20:X�j�D��nY���������X�6:noseedi1ee1:q9:get_peers1:t2:�=1:v4:LT��1:y1:qe
[1] preserved char count. replaced all non-printable, ASCII-incompatible bytes with the unicode replacement character.
I need to exchange both protobuf-net objects and files between computers and am trying to figure out the best way to do that. Is there a way for A to inform B that the object that follows is a protobuf or a file? Alternately, when a file is transmitted, is there a way to know that the file has ended and the Byte[] that follows is a protobuf?
Using C# 4.0, Visual Studio 2010
Thanks, Manish
This has nothing to do with protobuf or files, and everything to do with your comms protocol, specifically "framing". This means simply: how you demark sub-messages in a single stream. For example, if this is a raw socket you might choose to send (all of)
a brief message-type, maybe a byte: 01 for file, 02 for a protobuf message of a particular file
a length prefix (typically 4 bytes network-byte-order)
the payload, consisting of the previous number of bytes
Then rinse and repeat for each message.
You don't state what comms you are asking, so I can be more specific.
Btw, another approach would be to treat a file as a protobuf message with a byte[] member - mainly suitable for small files, though
guys need some insight here.
I know the definition of a protocol, being new to this c++ programming is quite a challenging
task.I am creating a Multi-threaded chat using SDL/C++, this is a learning experience for me
and now i have encounter a hump in which I need to overcome but understanding it is a little more difficult than I had thought.I need to make a chat protocol of some sort, I think...but am stump. Up until this point i have been sending messages in strings of characters.Now that am improving the application to the point where clients can register and login, I need a better way to communicating with my clients and server.
thank you.
Create objects that represent a message, then serialize the object, send it over the network, then deserialize at the other end.
For example, you could create a class called LoginMessage that contains two fields. One for a user name, and one for a password. To login, you would do something like:
LoginMessage *msg = new LoginMessage();
msg->username = "Fred";
msg->password = "you'll never guess";
char *serialized_msg = serialize(msg);
// send the bytes over the network
You would do something similar at the other end to convert the byte stream back into an object.
There are APIs for creating message objects and serializing them for you. Here are two popular ones. Both should suit your needs.
Protocol Buffers by Google
Thrift By Facebook
If you want the serialized messages to be readable, you can use YAML. Google has an API called yaml-cpp for serializing data to YAML format.
UPDATE:
Those APIs are for making your own protocol. They just handle the conversion of messages from object form to byte stream form. They do have feature for the actual transport of the messages over the network, but you don't need to use those features. How you design your protocol it up to you. But if you want to create messages by hand, you can do that too.
I'll give you some ideas for creating your own message format.
This is one way to do it.
Have the first 4 bytes of the message represent the length of the message as an unsigned integer. This is necessary to figure out where one message ends and where the next one starts. You will need to convert between host and network byte order when reading and writing to/from these four bytes.
Have the 5th byte represent the message type. For example, you could use a 1 to indicate a login request, a 2 to indicate a login response, and 3 to indicate a chat message. This byte is necessary for interpreting the meaning of the remaining bytes.
The remaining bytes would contain the message contents. For example, if it was a login message, you would encode the username and password into these bytes somehow. If it is a chat message, these bytes would contain the chat text.