I am familiar with the theory of how a Distributed Hash Table (DHT) works. Is it possible to write a program that stores data to an existing DHT (such as Kademlia or Mainline DHT) ? Is there a simple 'Hello World' type of program that would show the simplest possible way to do this?
The best hello world for DHT would be to send a 'ping' on Bittorrent's DHT to a bootstrap node. The steps are:
Bencode a KRPC PING message.
Send it over UDP to a bootstrap node.
Wait for a reply.
These are the steps I just took before I began working on my own DHT implementation.
The question is might be outdated but anyway.
As was mentioned, the simplest way to say "Hello" to an existing DHT is to send a ping message to one of DHT nodes. Let's consider Kademlia-based Mainline DHT (MDHT).
There is a bootstrap server at address router.bittorrent.com on port 6881. You can think about this server as a general DHT node which is permanently online. Also, you can use another node such as locally run torrent client, which uses DHT.
I've written a small example in Python:
import bencode
import random
import socket
# Generate a 160-bit (20-byte) random node ID.
my_id = ''.join([chr(random.randint(0, 255)) for _ in range(20)])
# Create ping query and bencode it.
# "'y': 'q'" is for "query".
# "'t': '0f'" is a transaction ID which will be echoed in the response.
# "'q': 'ping'" is a query type.
# "'a': {'id': my_id}" is arguments. In this case there is only one argument -
# our node ID.
ping_query = {'y': 'q',
't': '0f',
'q': 'ping',
'a': {'id': my_id}}
ping_query_bencoded = bencode.bencode(ping_query)
# Send a datagram to a server and recieve a response.
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.sendto(ping_query_bencoded,
(socket.gethostbyname('router.bittorrent.com'), 6881))
r = s.recvfrom(1024)
ping_response = bencode.bdecode(r[0])
print(ping_response)
I've used bencode module to bencode and bdecode messages.
More information on Mainline DHT protocol can be in this document. (Note that the protocol is slightly different from original Kademlia protocol.)
Related
How do i transfer a large file (video,audio) from my client to server in the local host using UDP sockets in python 3.5? I was able to send a small .txt file but not other file types. Please give me suggestions.
Thank you!
Here is my code to transfer a text file.
CLIENT CODE:
import socket
import sys
s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
host = '127.0.0.1'
port=6000
msg="Trial msg"
msg=msg.encode('utf-8')
while 1:
s.sendto(msg,(host,port))
data, servaddr = s.recvfrom(1024)
data=data.decode('utf-8')
print("Server reply:", data)
break
s.settimeout(5)
filehandle=open("testing.txt","rb")
finalmsg=filehandle.read(1024)
s.sendto(finalmsg, (host,port))
SERVER CODE:
import socket
host='127.0.0.1'
port=6000
s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
s.bind(("",port))
print("waiting on port:", port)
while 1:
data, clientaddr= s.recvfrom(1024)
data=data.decode('utf-8')
print(data)
s.settimeout(4)
break
reply="Got it thanks!"
reply=reply.encode('utf-8')
s.sendto(reply,clientaddr)
clientmsg, clientaddr=s.recvfrom(1024)
Don't use UDP for transferring large files, use TCP.
UDP does not garauntee all packets you send will arrive, or if they will arrive in order, they may even be duplicated. Furthermore UDP is not suited to large transfers because 1) it has no congestion control so you will just flood the network and the packets will be dropped, and, 2) you would have to break up your packets into smaller ones usually about 1400 bytes is recommended to keep under MTU otherwise if you rely on IP fragmentation and one fragment is lost your whole file is lost .. You would have to write custom code to fix all these issues with UDP since file transfers require everything to be sent reliably.
TCP on the other hand already does all this, it is reliable, has congestion control and is ubiquitous - you are viewing this web page over HTTP which is on top of TCP.
If you must use UDP instead of TCP or an application level protocol then, you should be able to 'generate redundant blocks' with a package like zfec so that you can reconstruct the original data even if not all of the packets are received.
Is it possible to send a SYN packet with self-defined payload when initiating TCP connections? My gut feeling is that it is doable theoretically. I'm looking for a easy way to achieve this goal in Linux (with C or perhaps Go language) but because it is not a standard behavior, I didn't find helpful information yet. (This post is quite similar while it is not very helpful.)
Please help me, thanks!
EDIT: Sorry for the ambiguity. Not only the possibility for such task, I'm also looking for a way, or even sample codes to achieve it.
As far as I understand (and as written in a comment by Jeff Bencteux in another answer), TCP Fast Open addresses this for TCP.
See this LWN article:
Eliminating a round trip
Theoretically, the initial SYN segment could contain data sent by the initiator of the connection: RFC 793, the specification for TCP, does permit data to be included in a SYN segment. However, TCP is prohibited from delivering that data to the application until the three-way handshake completes.
...
The aim of TFO is to eliminate one round trip time from a TCP conversation by allowing data to be included as part of the SYN segment that initiates the connection.
Obviously if you write your own software on both sides, it is possible to make it work however you want. But if you are relying on standard software on either end (such as, for example, a standard linux or Windows kernel), then no, it isn't possible, because according to TCP, you cannot send data until the session is established, and the session isn't established until you get an acknowledgment to your SYN from the other peer.
So, for example, if you send a SYN packet that also includes additional payload to a linux kernel (caveat: this is speculation to some extent since I haven't actually tried it), it will simply ignore the payload and proceed to acknowledge (SYN/ACK) or reject (with RST) the SYN depending on whether there's a listener.
In any case, you could try this, but since you're going "off the reservation" so to speak, you would need to craft your own raw packets; you won't be able to convince your local OS to create them for you.
The python scapy package could construct it:
#!/usr/bin/env python2
from scapy.all import *
sport = 3377
dport = 2222
src = "192.168.40.2"
dst = "192.168.40.135"
ether = Ether(type=0x800, dst="00:0c:29:60:57:04", src="00:0c:29:78:b0:ff")
ip = IP(src=src, dst=dst)
SYN = TCP(sport=sport, dport=dport, flags='S', seq=1000)
xsyn = ether / ip / SYN / "Some Data"
packet = xsyn.build()
print(repr(packet))
TCP Fast open do that. But both ends should speak TCP fast open. QUIC a new protocol is based to solve this problem AKA 0-RTT.
I had previously stated it was not possible. In the general sense, I stand by that assessment.
However, for the client, it is actually just not possible using the connect() API. There is an alternative connect API when using TCP Fast Open. Example:
sfd = socket(AF_INET, SOCK_STREAM, 0);
sendto(sfd, data, data_len, MSG_FASTOPEN,
(struct sockaddr *) &server_addr, addr_len);
// Replaces connect() + send()/write()
// read and write further data on connected socket sfd
close(sfd);
There is no API to allow the server to attach data to the SYN-ACK sent to the client.
Even so, enabling TCP Fast Open on both the client and server may allow you to achieve your desired result, if you only mean data from the client, but it has its own issues.
If you want the same reliability and data stream semantics of TCP, you will need a new reliable protocol that has the initial data segment in addition to the rest of what TCP provides, such as congestion control and window scaling.
Luckily, you don't have to implement it from scratch. The UDP protocol is a good starting point, and can serve as your L3 for your new L4.
Other projects have done similar things, so it may be possible to use those instead of implementing your own. Consider QUIC or UDT. These protocols were implemented over the existing UDP protocol, and thus avoid the issues faced with deploying TCP Fast Open.
I am aware this has been asked before. However I am unclear on how to construct the command in order to accomplish my task and the previous question was never marked as answered. I need to send multiple packets with scapy and my use case is to send DNS queries to a remote server server using UDP. This is the command that I need to use:
sr1(IP(dst="192.168.155.128")/UDP()/DNS(rd=1,qd=DNSQR(qname="www.oreilly.com")))
In the above example sr1 means send a packet at layer 3 but there are more function definitions to send packets. See Here. The remaining parts in between the braces is how to assemble a DNS query packet with scapy for an A record.
But what I want to do is send more than one packet in a single command. The previous question had a suggested answer of this:
sendp(p, iface=eth0, inter=1 , count=x )
Where p is your packet or a list of packets and count is the number of times to repeat the send operation.
This is where I am lost. If this is the correct answer, how would I integrate that into my command and what would it look like?
Thanks in advance!
Working solution: In the above example you will need to use a different function definition to send the packets.
Replace sendp with send, (sendp sends at layer2, send uses layer 3, and sr1 is designed to send only one packet) and place " , count=x" in between the last two closing braces. Where x = the number of packets you want to send. Running from the scapy prompt the command and output should look like this:
>>> send(IP(dst="192.168.155.128")/UDP()/DNS(rd=1,qd=DNSQR(qname="www.oreilly.com")), count=100 )
....................................................................... .............................
Sent 100 packets.
>>>
A simple ICMP packet can also be sent can also be constructed. In this example we are sending 100 ICMP packets.
>>> send(IP(dst="192.168.155.128")/ICMP()/"testICMPpacket", count=100 )
....................................................................... .............................
Sent 100 packets.
>>>
I use Scapy to create an initial OpenVPN packet and send it to OpenVPN server (acting as a client). OpenVPN part of the packet I'm just reusing from old captured connection, but its irrelevant here.
Thing is, I add a payload of 42bytes but for some reason when I capture packet with Wireshark, I can see 84bytes of OpenVPN stuff. Last half of that is correct payload I sent, but I can't figure out what is the first half. All other layers (Ethernet, IP, UDP) have correct size.
#!/usr/bin/env python
import socket
from scapy.all import *
mysocket=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
mysocket.connect(('192.168.138.129', 1194))
mystream=StreamSocket(mysocket)
ascapypacket=Ether()/IP(dst="192.168.138.129")/UDP(dport=1194, len=50)/Raw(load="\x38\x81\x38\x14\x62\x1d\x67\x46\x2d\xde\x86\x73\x4d\x2c\xbf\xf1\x51\xb2\xb1\x23\x1b\x61\xe4\x23\x08\xa2\x72\x81\x8e\x00\x00\x00\x01\x50\xff\x26\x2c\x00\x00\x00\x00\x00")
etherLoad = len(ascapypacket.getlayer(Ether)) # display size
print etherLoad
ipLoad = len(ascapypacket.getlayer(IP)) # display size
print ipLoad
udpLoad = len(ascapypacket.getlayer(UDP)) # display size
print udpLoad
rawLoad = len(ascapypacket.getlayer(Raw)) # display size
print rawLoad
mystream.send(ascapypacket)
I made an image. Here you can see green stuff is correct - first part is IP and UDP layers, and 2nd green part is my OpenVPN payload, but I don't understand what is the red part.
Edit: If I don't send that Raw payload I still get those 42 bytes for some reason.
You've created an ordinary UDP datagram socket:
mysocket=socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
This socket manages the ethernet, IP & UDP layers by itself, with guidance from the user through various auxiliary methods and arguments, such as the connect method you've in fact used in your code snippet:
mysocket.connect(('192.168.138.129', 1194))
Its various send methods, even when encapsulated as part of a scapy's StreamSocket object, are expecting to receive as their "data-to-send" argument just the application payload layered above the UDP layer.
However, you're passing on to it the entire protocol stack payload, i.e. the ethernet, IP & UDP headers, which is misinterpreted to be part of the payload data that you wish to send to the other side:
ascapypacket=Ether()/IP(dst="192.168.138.129")/UDP(dport=1194, len=50)/Raw(load="\x38\x81\x38\x14\x62\x1d\x67\x46\x2d\xde\x86\x73\x4d\x2c\xbf\xf1\x51\xb2\xb1\x23\x1b\x61\xe4\x23\x08\xa2\x72\x81\x8e\x00\x00\x00\x01\x50\xff\x26\x2c\x00\x00\x00\x00\x00")
Thus, the data you've marked in red is actually the payload data you've yourself set, before it is followed by the OpenVPN part:
Ether()/IP(dst="192.168.138.129")/UDP(dport=1194, len=50)
The first part marked in green, which you've mistakenly identified as created by yourself, is actually generated by the socket object (the kernel, the appropriate driver and the underlying hardware, to be more accurate).
Depending on your needs, you should either instantiate your socket as a raw one:
mysocket = socket(socket.AF_PACKET, socket.SOCK_RAW)
or set the payload accordingly as just the OpenVPN data:
ascapypacket=Raw(load="\x38\x81\x38\x14\x62\x1d\x67\x46\x2d\xde\x86\x73\x4d\x2c\xbf\xf1\x51\xb2\xb1\x23\x1b\x61\xe4\x23\x08\xa2\x72\x81\x8e\x00\x00\x00\x01\x50\xff\x26\x2c\x00\x00\x00\x00\x00")
I am developing a program that sniffs network packets using a raw socket (AF_PACKET, SOCK_RAW) and processes them in some way.
I am not sure whether my program runs fast enough and succeeds to capture all packets on the socket. I am worried that the recieve buffer for this socket occainally gets full (due to traffic bursts) and some packets are dropped.
How do I know if packets were dropped due to lack of space in the
socket's receive buffer?
I have tried running ss -f link -nlp.
This outputs the number of bytes that are currently stored in the revice buffer for that socket, but I can not tell if any packets were dropped.
I am using Ubuntu 14.04.2 LTS (GNU/Linux 3.13.0-52-generic x86_64).
Thanks.
I was having a similar problem as you. I knew that tcpdump was able to to generate statistics about packet drops, so I tried to figure out how it did that. By looking at the code of tcpdump, I noticed that it is not generating those statistic by itself, but that it is using the libpcap library to get those statistics. The libpcap is on the other hand getting those statistics by accessing the if_packet.h header and calling the PACKET_STATISTICS socket option (at least I think so, but I'm no C expert).
Therefore, I saw only two solutions to the problem:
I had to interact somehow with the linux header files from my Pyhton script to get the packet statistics, which seemed a bit complicated.
Use the Python version of libpcap which is pypcap to get those information.
Since I had no clue how to do the first thing, I implemented the second option. Here is an example how to get packet statistics using pypcap and how to get the packet data using dpkg:
import pcap
import dpkt
import socket
pc=pcap.pcap(name="eth0", timeout_ms=10000, immediate=True)
def packet_handler(ts,pkt):
#printing packet statistic (packets received, packets dropped, packets dropped by interface
print pc.stats()
#example packet parsing using dpkt
eth=dpkt.ethernet.Ethernet(pkt)
if eth.type != dpkt.ethernet.ETH_TYPE_IP:
return
ip =eth.data
layer4=ip.data
ipsrc=socket.inet_ntoa(ip.src)
ipdst=socket.inet_ntoa(ip.dst)
pc.loop(0,packet_handler)
tpacket_stats structure is defined in linux/packet.h header file
Create variable using the tpacket_stats structre and pass it to getSockOpt with PACKET_STATISTICS SOL_SOCKET options will give packets received and dropped count.
-- some times drop can be due to buffer size
-- so if you want to decrease the drop count check increasing the buffersize using setsockopt function
First off, switch your operating system.
You need a reliable, network oriented operating system. Not some pink fluffy "ease of use" with "security" functionality enabled. NetBSD or Gentoo/ArchLinux (the bare installations, not the GUI kitted ones).
Start a simultaneous tcpdump on a network tap and capture the traffic you're supposed to receive along side of your program and compare the results.
There's no efficient way to check if you've received all the packets you intended to on the receiving end since the packets might be dropped on a lower level than you anticipate.
Also this is a question for Unix # StackOverflow, there's no programming here what I can see, at least there's no code.
The only certain way to verify packet drops is to have a much more beefy sender (perhaps a farm of machines that send packets) to a single client, record every packet sent to your reciever. Have the statistical data analyzed and compared against your senders and see how much you dropped.
The cheaper way is to buy a network tap or even more ad-hoc enable port mirroring in your switch if possible. This enables you to dump as much traffic as possible into a second machine.
This will give you a more accurate result because your application machine will be busy as it is taking care of incoming traffic and processing it.
Further more, this is why network taps are effective because they split the communication up into two channels, the receiving and sending directions of your traffic if you will. This enables you to capture traffic on two separate machines (also using tcpdump, but instead of a mirrored port, you get a more accurate traffic mirroring).
So either use port mirroring
Or you buy one of these: