Best Practice to creating an custom expandable protocol architecture? [closed] - protocols

Question

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 2 years ago.
Improve this question
Disclaimer: I have 0 experience regarding creating custom, larger scaled protocols.
I am about to start a new project for fun (preferably in java), consisting of one Master-Server (MS), several smaller servers (SS) on the same network, and several clients. All of those three parties should be to communicate information to each other.
Examples:
A Client 'logs in' to the MS.
The MS sends a client to an SS. (SS has to be started, MS sends IP/PORT of SS to Client and tells him to connect, SS waiting for Client to connect, ...)
SS and Client communicate information to each other (e.g. game server and client)
The most experience with custom protocols and packets on a larger scale I have is from Minecraft Servers (Spigot, etc.). When reading the Servers packet system I still get confused a bit.
Whilst researching this most of the time I only found basic tutorials on how to create a TCP/UDP Server-Client model in various programming languages, which I am not interested in.
What I want to Know:
I want to create my own protocol 'architecture', but I have no idea where to start. I want it to be very expandable, but not to complex.
Are there any common practices to creating good packets -> "How should a packet message look like?"
A simple answer or a link recommendation could already help me quite a lot! I know it is a very broad question, however I need to start at some point.

Answer 1

Basically what you are describing is a proxy server.
For now, this is what has come to my mind. Let me know any doubts so that I can solve them by expanding the response.
What is a proxy server?
A proxy server is a server that routes incoming traffic to other servers (internal or external) and acts as an intermediary between the client and the final server.
There are multiple approaches to your problem.
Approach 1: Nginx + JSON
In this case, I would recommend you use some proxy server like Nginx that uses the HTTP protocol. The information would then be transferred as JSON strings instead of using raw binary packets which would simplify the problem quite much.
For more info about NGINX:
Main website
Official docs
Nice youtube tutorial series for beginners.
For more info about JSON:
Working with JSON. Nice introduction by Mozilla.
Working with JSON in java with Jackson.
Approach 2: Making your own proxy server and using binary packets
For the proxy part, you could use Java Sockets and a class that distributes the connections by reading and opening packet form the client where it specifies the wanted destination. Then you would have two options:
Redirect the (Client-Proxy) socket streams to the (Proxy-WantedDestination) socket.
Tell the WantedDestination to open a connection to the client. (ServerSocket on client and Socket on WantedDestination) So in this way, the WantedDestination would open a socket connection with the Client instead of the Client opening a connection with the Wanted destination.
The first method allows you to log all incoming and outgoing data. The second method allows you to keep the WantedDestination secure.
First method:
Client <--> Proxy <--> WantedDestination (2 Sockets)
Second method:
Step 1: Client <--> Proxy
Step 2: Proxy <--> WantedDestination
Step 3: Client <---------------> WantedDestination (1 socket)
How to structure packets
I usually structure packets in the following way:
Packet header
Packet length
Packet payload
Packet checksum
The packet header can be used to identify if the packet is coming from your software and that you are starting to read the data from the right position.
The packet length will indicate how many bytes the stream must read before trying to deserialize the packet into its wrapper class. Let's imagine that the header has a length of 2 bytes and that the length has a length of 3 bytes. Then if the length indicates that the packet is 30 bytes long, you will know that the end of the packet is (30 - 3 - 2) = 25 bytes away.
The packet payload will have a variable size and will contain some fixed size bytes at the beginning indicating the packet type. The packet type can be chosen arbitrarily. For example, you can determine that a packet of type (byte) 12 must be interpreted as a packet containing data about a pong match.
Finally, the packet checksum indicates the sum of the bytes of the packet you that you can verify the integrity of the packet. Java already provides some checksum algorithms, such as CRC32. If Packet Checksum = CRC32(Packet header, Packet length, and Packet Payload), then the data is not corrupted.
In the end, a packet is a byte array that can be transmitted using Java Input and Output streams. Despite this, working directly with byte arrays can be usually difficult and frustrating, so I would recommend that you use a wrapper class to represent a packet and then extend that class to create other packets. For example:
package me.PauMAVA.DBAR.common.protocol;
import java.util.Arrays;
import java.util.zip.CRC32;
import java.util.zip.Checksum;
import static me.PauMAVA.DBAR.common.util.ConversionUtils.*;
public abstract class Packet implements Serializable {
public static final byte[] DEFAULT_HEADER = new byte[]{(byte) 0xAB, (byte) 0xBA};
private final byte[] header;
private final byte packetType;
private byte[] packetParameter;
private byte[] packetData;
private byte[] packetCheckSum;
Packet(PacketType type, PacketParameter parameter) {
this(type, parameter, new byte[0]);
}
Packet(PacketType type, PacketParameter parameter, byte[] data) {
this.header = DEFAULT_HEADER;
this.packetType = type.getCode();
this.packetParameter = parameter.getData();
this.packetData = data;
recalculateChecksum();
}
public byte[] getParameterBytes() {
return packetParameter;
}
public PacketParameter getPacketParameter() {
return PacketParameter.getByData(packetParameter);
}
public byte[] getPacketData() {
return packetData;
}
public void setParameter(PacketParameter parameter) {
this.packetParameter = parameter.getData();
recalculateChecksum();
}
public void setPacketData(byte[] packetData) {
this.packetData = packetData;
recalculateChecksum();
}
public void recalculateChecksum() {
Checksum checksum = new CRC32();
checksum.update(header);
checksum.update(packetParameter);
checksum.update(packetType);
if (packetData.length > 0) {
checksum.update(packetData);
}
this.packetCheckSum = longToBytes(checksum.getValue());
}
public byte[] toByteArray() {
return concatArrays(header, new byte[]{packetType}, packetParameter, packetData, packetCheckSum);
}
And then a custom packet could be:
package me.PauMAVA.DBAR.common.protocol;
import java.nio.charset.StandardCharsets;
import static me.PauMAVA.DBAR.common.util.ConversionUtils.subArray;
public class PacketSendPassword extends Packet {
private String passwordHash;
public PacketSendPassword() {
super(PacketType.SEND_PASSWORD, PacketParameter.NO_PARAM);
}
public PacketSendPassword(String passwordHash) {
super(PacketType.SEND_PASSWORD, PacketParameter.NO_PARAM);
super.setPacketData(passwordHash.getBytes(StandardCharsets.UTF_8));
}
#Override
public byte[] serialize() {
return toByteArray();
}
#Override
public void deserialize(byte[] data) throws ProtocolException {
validate(data, PacketType.SEND_PASSWORD, PacketParameter.NO_PARAM);
PacketParameter packetParameter = PacketParameter.getByData(subArray(data, 3, 6));
if (packetParameter != null) {
super.setParameter(packetParameter);
}
byte[] passwordHash = subArray(data, 7, data.length - 9);
super.setPacketData(passwordHash);
this.passwordHash = new String(passwordHash, StandardCharsets.UTF_8);
}
public String getPasswordHash() {
return passwordHash;
}
}
Sending a packet over a stream would be as easy as:
byte[] buffer = packet.serialize();
dout.write(buffer);
You can take a look at a small protocol that I developed for a Bukkit server auto reloader here.
Be advised that this method will need you to convert between different data types and byte arrays, so you would need a good understanding of numeric and character representation in binary.