I'm looking to a design a protocol for a client-server application and need some links to some resources that may help me.
The big part is I'm trying to create my own "packet" format so I can minimize the amount of information being sent. I'm looking for some resources to dissect their protocol, but it seems some completely lack packet design, such as SMTP (which just sends strings terminated by CLRF). What are the advantages/disadvantages of using a system like SMTP over a system that uses a custom made packet? Couldn't SMTP use only a couple bytes to cover all commands through bit flags and save bandwidth/space?
Just trying to get my head around all this.
True, but SMTP wasn't particularly optimized for space, nor is it a packet-based protocol. It sits atop TCP, and uses the stream functionality of TCP. You need to decide what is desirable in your protocol: is it performance sensitive? latency? bandwidth?
Is it going to need to run as superuser? If not, you'll probably want to use UDP or TCP.
Are you going to need guarantees on delivery? If so, TCP is probably your best option, unless you are dealing with fairly extreme performance or size issues.
Few protocols these days design individual packets, though many do send very specific data structures across the wire using TCP, or, less commonly, UDP.
If you want to really optimize for space or bandwidth, consider condensing your data as much as possible into individual bits and byte, and defining and packing structures to send it across TCP. Modern network adapters are so optimized for TCP anyway, that there is often little advantage to other strategies.
First of all, are you about to implement an enhanced transport protocol (like RTP on top of UDP) or an application protocol (like HTTP/SMTP)?
There are several things you should think about in both cases concerning your design of the protocol or the demands of your application:
Stream based or packet based,
unidirectional / bi-directional,
stateful and sessionful or stateles,
reliable or best effort,
timing demands,
flow/congestion control,
secure or plain.
Towards an application layer protocol, you should also think about:
Textual or binary data, mapping of application data to network data units/packets, security demands and integrity, etc.
SMTP, HTTP and other TCP based protocols do not concern themselves with packet design because they are stream based. So it makes more sense to talk about protocol design.
As for why use text based protocols vs binary protocols...
Readability of the protocol by packet sniffing programs like Wireshark is very useful.
Also it is often very useful to be able to simply telnet into your port and be able to communicate with the server by specifying plain text.
Also with a protocol like HTTP the actual resource is usually the payload of the communication, the resource can be in binary or any other specified format. So having just the headers and status in plain text is not a bad thing.
TCP is a stream based protocol, not packet based.
Using text with lines makes ad hoc debugging a lot easier
Using text with lines makes it possible to exercise your protocol with telnet
Related
I feel like I am missing something, all of the VITA49 examples seem to be using TCP or UDP.
Is there a specification or standard way of providing VITA49 packets for consumption?
Should I be performing the conversion and providing standard complex samples with Keywords?
I have looked at the rh.vita49 loopback demo waveform, and the MSDD device source, as well as the sourceVITA49 and sinkVITA49 component. All of these use either a tcp or udp packet stream.
If the standard is to use sockets to pass VITA49 packets, then where should I be looking to understand how to construct a device that adheres to the standard?
ANSWER
I was able to talk to an experienced REDHAWK developer.
There is no standard, per-se, with that said the approach I took was to make use of the socket.sourceVita49 asset. This asset consumes the Vita49 packets and inserts appropriate keywords etc based upon context packet. This required me update my device to support setting the hardware up to send Vita49 via TCP. This actually provided an easier solution for me, as I wasn't having to bust the VRT apart.
Examples:
The best example I found of consuming Vita49 was the MSDD device asset.
NOTE:
After reviewing the MSDD, it does not look to be too difficult to create a device that consumes VITA49 VRL,VRT packets and produce time stamped samples. I will be investigating that in the future.
I need a SIP server, hopefully simpler, more efficient and more secure than Asterisk or other full-featured PBX, to just terminate SIP calls and play audio, based on the incoming number. This would be in a linux environment.
I'm concerned about long-term support, and much-reduced attack surface. I'm a programmer but a no0b in the SIP world. I want to avoid the tons of security hassles of a full-featured PBX, let alone the crazy big bills after a hacker gets into it. Thanks in advance!
Personally I would build one with pjsip. Assuming you do not need built in registration server all calls would be handled by so called local account. You can use command-line pjsua with auto-answer and auto-play options for initial tests and then add some logic for audio wave selection and perhaps audio player allocation/deallocation (audio players could be active all the time and shared by multiple calls for efficiency if they e.g. contain something like background music).
For security disable not needed parts (TCP, TLS transports, not needed codecs?) and use port other than 5060. In my application I'm also using kind of application-level firewall that blocks requests from hosts that generate suspicious traffic (registrations, malformed SIP requests).
I have designed a simple communications protocol, over raw TCP sockets, to enable simple messaging between some embedded devices. To put things in context, my embedded device is a box of electronics containing a relatively small microcontroller that is running a basic embedded RTOS (which essentially provides just task priority and message queueing) and TCP/IP stack. The intended use for the TCP protocol is to
Enable two or more 'boxes' to communicate with each other over a LAN in one building
To allow a box to exchange data with an external server over the Internet.
I now have a messaging protocol working between my metal boxes that I'm happy with. The basic messaging procedure between two boxes is basically:
Box 'A' initiates a socket connection to 'B'.
Box 'A' sends a message report or command sequence.
Box 'B' responds with an acknowledge and / or command response.
Box 'A' closes the socket.
What I would now like to do is to incorporate some level of security and authentication. The great restriction here is that I don't have any kind of OS or glorified TCP stack that can provide me with any security features; I just have simple TCP stack, and therefore I must implement security measures at application level, and with microcontroller limitations in mind.
The objectives I would like to meet are:
Authentication between devices. To achieve this, I'm intending to do the following measures:
Hold a table of known IPs from which connections shall only be accepted.
Each time a socket connection is established, unique identifiers are always exchanged first. The number might be a unique serial number for a given device, and must be known to the other devices.
Encryption of data, in the event that packets are somehow intercepted. Presumably I need some kind of cipher algorithm that isn't too 'expensive' to perform on a small microcontroller, used in conjuction with a unique key that is programmed into all devices. One such algorithm I've seen, which looks compact enough to implement in my code, is TEA (Tiny Encryption Algorithm).
I would be most grateful for any advice or pointers.
Check MatrixSSL - they boast a tiny size and embeddable capabilities. This is much better than re-inventing SSL/TLS yourself (which you would end up doing).
Tea looks to be quite simple and will probably do what you need.
compiling for thumb with -O2 or similar optimizations:
arm-none-linux-gnueabi-gcc (Sourcery G++ Lite 2011.03-41) 4.5.2
encrypt 136 bytes decrypt 128 bytes
llvm 29
encrypt 92 bytes decrypt 96 bytes
compiling for generic arm...
gnu encrypt 188 bytes, decrypt 184 bytes
llvm encrypt 112 bytes, decrypt 116 bytes
For authentication, is there a one to one relationship between the ip address table and the number of devices? Basically does there need to be more than one unique identifier per device? Are you wanting the other side that is making the connection to the embedded system to log in in some form or fashion? Or are you controlling the binaries/code on both ends and there are no users or no selection of devices (the programs know what to do), etc? If each device has a known ip address, that ip address could be the key to the encryption (plus other bytes common to all or derived in some fashion that everyone knows) If a connection coming in is 1) not from the approved list 2) encryption fails when the embedded systems ip address based key fails then reject the connection.
Your security can only go so far anyway, if you need something really robust you probably need more horsepower in the embedded system and then implement one of the common/standard systems like ssl.
I'm looking to write my own multiplayer game server (most likely in nodejs) and I was wondering what protocol I should be using to transfer data? Are Datagrams the norm to send information (i realize they don't confirm delivery like HTTP, that can written on top of the protocol)? Any suggestions of performant proven systems would be a real help.
I guess I'm looking for successful techniques in handling the data transfer quickly and effectively (maintaining state on the server and scaling are a separate issue that I have a solid understanding of).
I'm looking to initially support desktop/mobile games (MacOS, iOS, and Android).
For all your protocol needs take a look at socketIO.
Basically your best solution is to rely on websockets which are TCP sockets. socketIO is just a nice cross-browser compliant abstraction.
Either you use standard long pulling techniques or html5 websockets. There is no access to UDP for browser <-> server.
There is a technology that is called RTMFP that Adobe introduced in the latest version of Flash (Flash 10). It allows you to do P2P connection and transfer data directly from a client to an other client without passing by the server. On top of that, it's using UDP to transfer data. I believe that this was originally designed to do video and audio streaming, but you can use it to pass data around.
However the main downside on this technology is the mobile since most of them don't support Flash. In this case you can use Socket.IO and use the server as a router of information as a fallback.
If you want to build your application in Javascript, you can still use it by bridging the functionnality to Javascript. If you want to take a look at a simple version of a bridge, you can take a look at this github project (I am the author).
As a C++ developer of Massive Multiplayer games for 10 years, I can tell you that most of your more advanced games, such as ones in which I was involved (Legends Of Kesmai, Magic: The Gathering Online, Airwarrior II, AVP, NTN Triva) to name a few, TCP is used for most communication simply because you need an ACK / NACK to be sure the data was received from the client. That is not to say UDP doesn't have it's place. In Legends we wrote the the protocol code to use UDP for out of bandwidth delivery of data which wasn't imperative that it be received in proper packet order and complete. Use UDP when you want to do things like update graphic files in the background while the user is playing, etc. This type of delivery is often used for such purposes and allows your TCP packets to arrive as required by your server.
I have an application where I have about 10,000 pieces of monitoring equipment across the US that periodically dials into a bank of 32 phone lines. I have two receivers of 16 lines each that answer the call and temporarily stores a small alpha string. I then have a computer that polls the receivers and parses the string and copies it to a database.
I am looking to replace the phone lines and the receivers with a voip solution and rewrite the software to parse the data string.
Any ideas on where to get started?
Tom's suggestion about Asterisk is a good one for the overall system.
However you will still need to decode the data sent from your remote equipment from an audio signal to a data signal. That task is what the "dem" part of Modem stands for (Modulate/Demodulate). Either you do this with a canned hardware/software package (as you are currently doing with a commercial modem) or you have to emulate the modem in software yourself which will be extremely tricky to code at the very least if you attempt it yourself (heaps of standards that you have to comply with for a general modem solution, plus the solution needs to work in real time)
For the software approach could start with this page Linmodems.org (just a something I saw on google prompted by your question). Alternatively do lots of searches on google for software modems. Getting someone else's code is the best approach for this sort of code :)
Whatever you end up doing I suspect it will be rather custom.
A good place to start is probably Asterisk PBX.
I take it you don't want to replace the modems at the client sites (the easiest thing on the server side would be each clients had its own IP software stack, and used its modem to call an ISP and establish an internet connection, and then talk to your server using TCP or UDP or HTTP or whatever).
Assuming that you don't have IP capability on the client sites, Googling suggests that the relevent technology is called "Modem over IP" or "MoIP" (which Wikipedia seems to be confusing with "Mobile over IP").
VoIP consists of SIP for signalling (e.g. for call set-up and call tear-down) plus some codecs (e.g. H.323) for traffic (encoded voice) while the call is established.
I'm guessing that MoIP can keep the SIP signalling, but needs to use some different codecs.
V.150 Modem over IP White Paper looks like an introduction to the technologies. I don't know what vendors there are.
I presume you are looking to find a way to do this without mofidying the modem hardware at your remote sites. If this is the case you will have to find or write signal processing software to demodulate the encoded signal from the modem. Fortunately, signal encodings on a modem are designed to be easy to do this with.
Maybe somebody makes software modem libaries that do this sort of thing. The other parts of the problem will be emulating the handshaking on the modem so it plays nicely with the remote sites.
If you can modify the software (really just the number to dial, but it would have to include the data you want to transfer) at the 10000 sites (not likely!), you could in theory use DTMF in the "dial" string to key the data over into Asterisk. Ok, more than a bit hackey, but it would avoid having to have a software modem. Note: you'd want a checksum!! (and maybe send it multiple times) And a way to tell the caller if it was received correctly. Like I said, hackey but cute.