I have a Client and Server programs communicating on TCP, implementing IEC-60870-5-104 protocol on the application layer. I am sniffing a copy of all traffic between them,
How do I exactly replicate the IP and TCP layer functionalities on packets sniffed from libpcap in C ?. like IP-reassembly, Managing out-of-order segments, re-transmission, and duplicate TCP segments and separating PDUs, such that I get the same packet as if I was sniffing on the application layer at the Server. (Also please suggest any Frame-works which helps to do this).
This is a very difficult and involved task. It's what network intrusion detection / deep packet inspection systems do and AFAIK there is no drop-in library that handles it.
Your best bet is to build on an existing system such as Bro or Suricata (or maybe wireshark internals) that is already doing the session tracking, state management, re-assembly, re-ordering, duplicate detection, etc. You can then add your own application layer decoding to operate on the re-assembled data stream provided.
Related
I have recently been trying to learn the basics of the TLS protocol, and I am unsure about how TCP packets containing TLS data can be differentiated from those that don't.
Can someone please provide an explanation?
TCP - the Transmission Control Protocol - is the underlying protocol of many higher level protocols; it doesn't specify what it transports. It turns a best effort packet protocol (IP) into a two way connection that can transport any stream of data.
TLS - the Transport Layer Protocol - provides security on top of a protocol such as TCP. It has a specific protocol description where the handshake records can easily be distinguished.
However, as indicated, TCP may transport any data. So if you have a protocol that is, say, one bit different from TLS then it won't be easy to detect this small change. However, tools such as WireShark are pretty capable of detecting protocols with high certainty.
Separate data records are harder to detect, as encrypted data packets don't contain much distinguishing features. Encrypted data itself looks like random data by definition. So if you just have a few packets then you have just some structure and otherwise random data. Now such random data is probably encrypted, but other than that it isn't much use for determining the protocol.
For learning purposes I'm implementing TCP (for now just RFC 793) but I have no
idea how to test it. Most TUN/TAP stuff on the internet are out of date (e.g.
Linux API calls no longer work) and just doesn't explain enough. In addition, I
feel like a creating a device and forwarding packages etc. are not the best way
for learning purposes. For example, I'd rather only override socket(),
listen(), connect(), accept(), send(), recv() etc. in a program rather
than forwarding all ethernet traffic to a device/program that does the
bookeeping for the whole system rather than for a single program.
I'm wondering if this is possible. If not, I'd like to know the simplest way to
test a TCP implementation on Linux.
Because I'm following RFC 793, it'd be great if I could have an IP (Internet
Protocol as mentioned in the RFC) API in my application. Is this possible or do
I have to mess with TUN/TAP stuff?
Thanks..
If we talk about research I strongly recommend you read Engineering with Logic: Rigorous Test-Oracle Specification and
Validation for TCP/IP and the Sockets API
It contains section about testing TCP/IP implementation:
"EXPERIMENTAL VALIDATION: TESTING INFRASTRUCTURE"
You could try setting up two peers, one using a RAW socket and the other a TCP socket.
If they can communicate and packets are really delivered/recovered the same way TCP does, you know that your custom implementation is successful.
C sockets
C RAW sockets
C TCP implementation
All you need is to intercept all tcp packets with bits (syn, ack, fin, etc.) your application has sent and to see if it works properly:
It could simply be done with wireshark or other sniffer. When testing you will see all tcp packets with bits you've sent.
In order you want to see linux system calls which your application are calling, you can use GDB or any other debugger.
I recall reading an article about a proposed way to do this. If I recall correctly, the researchers successfully created a connection to a client on another network without port forwarding by sending HTTP packets to each other (Alice pretends that Bob is an HTTP web server while Bob pretends Alice is a web server).
I'm not sure if that makes sense, but does anyone know where I can find the article or does anyone have any other ideas how to connect two clients together without a central server or port forwarding?
Is it even possible?
Edit: I would know the IPs of both computers and port the program listens on.
It is possible. I see at least 2 parts to your question. (It is not going to be HTTP packet. It is a lot more complex than that.)
First off, I believe you might be talking about a concept called decentralized P2P network. The main idea behind a decentralized peer-to-peer network is the fact that nodes conjoint in such a network will not require central server or group of servers.
As you might already know, most common centralized peer-to-peer networks require such centralized system to exchange and maintain interconnectivity among nodes. The basic concept is such, a new node will connect to one of the main servers to retrieve information about other nodes on the network to maintain its connectivity and availability. The central system gets maintained through servers constantly synchronizing network state, relevant information, and central coordination among each other.
Decentralized network, on the other hand, does not have any structure or predetermined core. This peer-to-peer model is also called unstructured P2P networks. Any new node will copy or inherit original links from the "parent" node and will form its own list over time. There are several categories of decentralization of such unstructured networks.
Interestingly enough, the absence of central command and control system makes it solution of choice for modern malware botnets. A great example could be Storm botnet, which employed so-called Passive P2P Monitor (PPM). PPM was able to locate the infected hosts and build peer list regardless whether or not infected hosts are behind a firewall or NAT. Wikipedia's article Storm botnet is an interesting read. There is also great collaborative study called Towards Complete Node Enumeration in a Peer-to-Peer Botnet, which provides excellent conceptual analysis and techniques employed by Storm botnet network.
Second of all, you might be talking about UDP hole punching. This is a technique or algorithm used to maintain connectivity between 2 hosts behind NATed router/gateway using 3rd comment host by means of a third rendezvous server.
There is a great paper by Bryan Ford, Pyda Srisuresh, and Dan Kegel called Peer-to-Peer Communication Across Network Address Translators.
As answered, a peer-to-peer connection requires establishment of a connection between two (presumably) residential computers, which will necessitate punching holes through both of their firewalls. For a concrete example of hole punching, see pwnat: "The only tool to punch holes through firewalls/NATs without a third party". The process, put simply, goes like this:
The "server" (who doesn't know the client's IP address, but the client knows the server's) pings a very specific ICMP Echo Request packet to 1.2.3.4 every 30 seconds. The NAT, during translation, takes note of this packet in case it gets a response.
The client sends an ICMP Time Exceeded packet to the server, which is a type of packet that usually contains the packet that failed to deliver. The client, knowing in advance the exact packet that the server has been sending to 1.2.3.4, embeds that whole packet in the Data field.
The NAT recognizes the Echo Request packet and happily relays the whole Time Exceeded packet, source IP and all, to the correct user, i.e. the server. Voila, now the server knows the client's IP and port number.
Now that the server knows the address, it begins to continually send UDP packets to the client, despite the fact that the client's NAT did not expect them and will therefore ignore them all.
The client begins sending UDP packets to the server, which will be recognized by the server's NAT as a response to the server's packets and route them appropriately.
Now that the client is sending UDP packets to the server, the server's stream of UDP packets starts getting properly routed by the client's NAT.
And, in 6 easy steps, you have established a UDP connection between a client and a server penetrating two residential firewalls. Take that, ISP!
Hello I was wondering during the development, what are the differences between the types of protocols that I can use for my endpoint? the latest SDK had, HTTP, HTTPS, UDP and TCP. I certainly understand what the differences between the http and Https, I also understand the differences between the TCP and UDP.
what I don't understand what are the differences between TCP and HTTP from the development perspective?
TCP / UDP are lower level protocols in the OSI model than HTTP/ HTTPS. Actually HTTPS is combining two thing, HTTP over SSL.
Have a read through the Wikipedia article describing the Osi Model
HTTP is a layer 7 (Application) protocol and as such has a strict set of rules governing how the messages are constructed and what are considered valid responses. It is not concerned with how the actual connection takes place or how the messages are routed.
TCP and UDP are layer 5 which means they are concerned with addressing, establishing connection, packetization and sequencing. Things that are needed to exchange a series of bytes (payloads) between two endpoints.
Usually when developing software you want to implement open and established protocols that simplify the task of integrating with systems from other vendors or opening up end points for others to consume. In this scenario, HTTP or HTTPS make sense.
If your system is a closed one where you control both the client and server applications or where performance is of paramount importance then TCP might be a good choice. Operating at this level means you have to concern yourself with issues of defining your own payload structure, security, packet loss etc.
I need an EXPERT opinion please, and sorry if my question itself is a confused question.
I was reading around about structure of VOIP applications (Client/Server). And mostly UDP is recommended for voice streams. I also checked some voicechat applications like paltalk and inspeak and their sites mention they use udp voice stream which dont seem correct for below reasons.
I examined the traffic/ports used by paltalk and inspeak. They have UDP and TCP ports open and using a packet sniffer i can see there is not much UDP communication but mostly it is the TCP communication going on.
Also as far as i know, In UDP Protocol server can not send data to a client behind NAT (DSL Router). And "UDP Braodcast" is not an option for "internet" based voice chat applications. THATS WHY YAHOO HAVE MENTIONED in their documentation that yahoo messenger switch to tcp if udp communication is not possible.
So my question is ....
Am i understanding something wrong in my above statements ?
If UDP is not possible then those chat applications use TCP Stream for voice ?
Since i have experienced that TCP voice streams create delay, No voice breaking but Delay in voice, so what should be the best structure for a voice chat server/client communication ?
So far i think that , if Client send data as udp packets to server and server distribute the packets to clients over TCP streams, is this a proper solution ? I mean is this what commercial voicechat applications do ?
Thanks your answer will help me and a lot of other programmers .
JF
UDP has less overhead (in terms of total packet size), so you can squeeze more audio into the channel's bandwidth.
UDP is also unreliable - packets sent may never be received or could be received out of order - which is actually OK for voice applications, since you can tolerate some loss of signal quality and keep going. a small amount of lost packets can be tolerated (as opposed to downloading a file, where every byte counts).
can you use TCP? sure, why not... it's slightly more overhead, but that may not matter.
SIP is a voice/media standard that supports UDP and TCP. most deployments use UDP because of the lower overhead.
The Skype protocol prefers UDP where possible, and falls back to TCP.
in SIP situations, the NAT problem is solved by using a nat keep-alive packet (any request/response data) to keep the channel up and open, and by exploiting the fact that most NATs will accept replies on the same source port the connection was opened from... this isn't foolproof, and often requires a proxy server mediating the connection between 2 nat'd peers, but it's used in many deployments.
STUN, TURN, and ICE are additional methods that help with NAT scenarios, and especially in p2p (serverless) situations.
info regarding NAT issues and media:
http://www.voip-info.org/wiki/view/NAT+and+VOIP
http://en.wikipedia.org/wiki/UDP_hole_punching
http://www.h-online.com/security/features/How-Skype-Co-get-round-firewalls-747197.html
if you're implementing a voice service of some kind, a system like FreeSwitch provides most of the tools you need to deliver media to distributed clients:
http://www.freeswitch.org/
I see the question is 3 years overdue, but I see no answer accepted, so I'll take a shot at it
1- your statements are correct
2- correct, TCP or UDP can be used for audio stream.
3- Combining tcp and udp for the audio stream is not useful. If UDP is working for transmission to the server, it will work for reception, that's how all NAT firewalls work, i.e they send datagram received from internal host to remote host after they change the ip header to make the packet seem coming from them, and when they receive response, they forward it back to internal host. The difference between NAT firewalls is for how long the NAT tunnel will remain alive, but this does not matter for the audio part of the call, as there is constant flow of audio in both way during a call. This would matter more for the signalling part of the call, which uses the SIP protocol. So I would recommend using TCP for SIP as the TCP session has a default timeout of 900s, making the keep alive messages less frequently needed.
Now some applications you mentioned do not use SIP for session initiation, and hence have proprietary ways of signalling.
Other applications take advantage of something called 'hole punching' to allow client-to-client direct communication (or peer-to-peer) such as Skype. The advantage of these is that the server does not stay in the middle of the voice stream, and this can effectively reduce latency, making TCP a potential choice for the audio stream.
The guys behind development of Asterisk, the famous opensource PBX, have realized the problems in SIP which require having lots of ports open, and they have developed their own protocol, called IAX, to transmit signalling and media over one port. I would encourage you to consider implementing IAX for your client/server, because it ensures that if a client is able to connect (through signalling), then it's able to make calls.