Here is my issue:
A) I'm a scripting newbie
B) I have a file where I need the data separated into a CSV style table, I problem is there are three areas of data (see below):
(Area 1, not relevant) Total IPv4 packets captured: 2245686
# L4 Protocol # Packets Relative Frequency[%] Protocol description
1 5602 0.249456 Internet Control Message Protocol
.... (more data here)
(Area 2, relevant) Total TCP packets: 2238186
# Port # Packets Relative Frequency[%] Protocol description
22 2138555 95.548583 The Secure Shell (SSH) Protocol
.... (more data here)
(Area 3, relevant) Total UDP packets: 1623
# Port # Packets Relative Frequency[%] Protocol description
.... (more data here)
(Area 4, relevant) Total SCP packets: 0
# Port # Packets Relative Frequency[%] Protocol description
.... (more data here)
(This is a Tranalyzer _protocols output)
So what I need to do is make the output look like:
# Port,# Packets,Relative Frequency[%],Protocol description
22,2138555,95.548583,The Secure Shell (SSH) Protocol,(more data...)
but I also need to take the data from each area and put it in a separate CSV file (TCP, UDP, SCP) so that for all flows I put in the table (each in a different _protocols file in different subdirectories) the data can go into one of these 3 files and build what will be a memory-crushing spreadsheet (hence why a CSV.)
I'm also completely open to any other way to represent this anyone can suggest.
Much appreciated!
Following command will extract and convert the data file to a csv.
Change x to TCP, UDP or SCP to extract particular set of data.
Change <analyzer_output.txt> to proper file name before executing.
x="SCP"; \
cat <analyzer_output.txt> | sed "1,/^Total $x/d; /^Total /,\$d; /^\s*$/d" | \
sed 's/\([0-9]\)\s\+\([0-9]\)/\1,\2/g; s/\([0-9]\)\s\+\([A-Za-z]\)/\1,\2/g' \
> $x.csv
Resulting SCP.csv file for below sample data
41,2138555,95.548583,The Secure Shell (SSH) Protocol
42,2138555,95.548583,The Secure Shell (SSH) Protocol
Sample data file
Total IPv4 packets captured: 2245686
1 5602 0.249456 Internet Control Message Protocol
2 5602 0.249456 Internet Control Message Protocol
Total TCP packets: 2238186
21 2138555 95.548583 The Secure Shell (SSH) Protocol
22 2138555 95.548583 The Secure Shell (SSH) Protocol
Total UDP packets: 1623
31 2138555 95.548583 The Secure Shell (SSH) Protocol
32 2138555 95.548583 The Secure Shell (SSH) Protocol
Total SCP packets: 0
41 2138555 95.548583 The Secure Shell (SSH) Protocol
42 2138555 95.548583 The Secure Shell (SSH) Protocol
Related
I am learning about codecs,and I get this question that I didnt understood the answers.
Assuming CODEC G.711 where each datagram carries 20ms of voice, indicate:
3) [E] How many bytes of G.711 encoding does each datagram carry?
A- 20ms/8*0,02=160
4) What is the byte size of each frame carrying G.711 on an Ethernet network?
Note: The dimensions (in bytes) of the base headers of some of the protocols that might be involved in the communication: Ethernet = 18, IP = 20, TCP = 20, UDP = 8, ICMP = 8, RTP = 12
A-18+20+8+12+160=218
I didnt get this math..
g711 codecs pure bandwidth(codec only) is 64kbit, exact
g711 packet length can be 10,20(default),30.. upto 150ms.
So for default settings you have 20ms packet(50 packet/sec) at 64kbit = 160Bytes without udp packet header
Full length of g711(default 20ms) packet is
TPS = 18 bytes+20 bytes+8 bytes+12 bytes+160 bytes
You have 160 bytes of raw data, first you make it rtp packet(timestamp mostly), see packet size at https://en.wikipedia.org/wiki/Real-time_Transport_Protocol
RTP required fore reorder when you get two packet in different order(sometimes happens).
Now you have rtp, BUT it not suitable for send, need know where to send, need address and port. For port part you use UDP packet https://en.wikipedia.org/wiki/User_Datagram_Protocol
For address you use IP packet header, without address it not go destination machine
https://en.wikipedia.org/wiki/Internet_Protocol
Okay, now you have packet. But you still need actually send it. For send you use some hardware level protocol, in this case it is ETHERNET. Ethernet have mac address, which allow fast switching without parse of IP. That is last 18 bytes
https://scialert.net/fulltext/?doi=ajsr.2017.110.115
In some cases you can prefer TCP(when you have packet loss or complex networking), in this case you not use UDP, use TCP instead of it. So you swap 8bytes UDP for 20 bytes TCP.
I am using datastax opscenter 6.0 for DSE Cassandra Monitoring. Configuration is done to send SNMP trap but Trap receiver ( HP Openview in this case ) is not receiving this alert.
I don't see any SNMP realted errors in opscenter logfile. How do I trace the exact error?
Here is my snmp.conf file:
[snmp]
# set to 1 to enable SNMP trap sending
enabled=1
# Levels can be a comma-delimited list of any of the following:
# DEBUG,INFO,WARN,ERROR,CRITICAL,ALERT
# If the left is empty, OpsCenter will listen for all levels.
levels=ALERT
# Comma-delimited list of cluster names for which
# this alert config will be eligible to run.
# If left empty, this alert will be called for events on all clusters.
clusters=
# SNMP engine ID, specified by rfc3411 and rfc5343.
# See http://tools.ietf.org/html/rfc3411#section-5
# SnmpEngineID definition for more information.
#
# 32 octet (max length) unique hex engine ID. Must not be all zeroes or all
# 255's. The first four octets specify the enterprise ID, left filled
# with zeroes and starting with an 8. The fifth octet specifies a format scheme
# that specifies the nature of the remaining octets. The remaining octets
# are given in accordance with the specified format.
#
# Format Schemes:
# 1 -- IPv4 Address scheme
# 2 -- IPv6 Address scheme
# 3 -- MAC Address scheme
# 4 -- Text Address scheme
# 5 -- Octets scheme
#
# Default scheme is octets scheme; if nothing else, you should change
# 01020304 to a unique octet string.
#engine_id=80:00:00:00:05:01:02:03:04
# IPv4 address of the SNMP target.
target_ip=*.*.*.* ( commented due to security urpose )
# Port to direct traps to on the SNMP target.
target_port=162
# Set to 1 to use SNMPv3 and the user/privacy key/auth key model. Set to 0 to
# use SNMPv1/community model.
use_snmpv3=0
# SNMPv1/2 community name (for community security model)
community_name=public
# SNMPv3 username
#user=opscusername
# SNMPv3 authentication protocol
# Options:
# MD5 -- MD5-based authentication protocol
# SHA -- SHA-based authentication protocol
# NoAuth -- no authentication to use
#auth_protocol=SHA
# SNMPv3 authentication key
#auth_key=authkey1
# SNMPv3 privacy protocol
# Options:
# DES -- DES-based encryption protocol
# AES -- AES128-based encryption protocol (RFC3826)
# 3DES -- triple DES-based encryption protocol (Extended Security Options)
# AES192 -- AES192-based encryption protocol (Extended Security Options)
# AES256 -- AES256-based encryption protocol (Extended Security Options)
# NoPriv-- no encryption to use
#privacy_protocol=AES
# SNMPv3 privacy key
#privacy_key=privkey1
Try setting levels=ALERT to levels= and make sure its not just that your filtering the events your looking for first (can turn it back once have it working like you want, just easier to see more things).
Can use wireshark or tcpdump to check if the trap is being sent with something like:
tcpdump -i eth1 -T snmp "(port 161 or 162)"
(note: eth1 may need to be replaced with your interface name). SNMP clients can be a little bit of a pita in being setup correctly as well, so good to check if they are being sent and not handled vs not sent.
I have apache server running on Ubuntu. Client connects and downloads an image. I need to extract RTT estimations for the underlying TCP connection. Is there a way to do this? Maybe something like running my tcp stack in debug mode to have it log this info somewhere?
Note that I don't want to run tcpdump and extract RTTs from the recorded trace! I need the TCP stack's RTT estimations (apparently this is part of the info you can get with TCP_INFO socket option). Basically need something like tcpprob (kprobe) to insert a hook and record the estimated RTT of the TCP connection on every incoming packet (or on every change).
UPDATE:
I found a solution. rtt, congestion window and more can be logged using tcpprobe. I posted an answer below.
This can be done without the need for any additional kernel modules using the ss command (part of the iproute package), which can provide detailed info on open sockets. It won't show it for every packet but most of this info is calculated over a number of packets. E.g. To list the currently open TCP (t option) sockets and associated internal TCP info (i) information - including congestion control algorithm, rtt, cwnd etc:
ss -ti
Here's some example output:
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 0 192.168.56.102:ssh 192.168.56.1:46327
cubic wscale:6,7 rto:201 rtt:0.242/0.055 ato:40 mss:1448 rcvmss:1392
advmss:1448 cwnd:10 bytes_acked:33169 bytes_received:6069 segs_out:134
segs_in:214 send 478.7Mbps lastsnd:5 lastrcv:6 lastack:5
pacing_rate 955.4Mbps rcv_rtt:3 rcv_space:28960
This can be done using tcpprobe, which is a module that inserts a hook into the tcp_recv processing path using kprobe records the state of a TCP connection in response to incoming packets.
Let's say you want to probe tcp connection on port 443, you need to do the following:
sudo modprobe tcp_probe port=443 full=1
sudo chmod 444 /proc/net/tcpprobe
cat /proc/net/tcpprobe > /tmp/output.out &
pid=$!
full=1: log on every ack packet received
full=0: log on only condo changes (if you use this your output might be empty)
Now pid is the process which is logging the probe. To stop, simply kill this process:
kill $pid
The format of output.out (according to the source at line 198):
[time][src][dst][length][snd_nxt][snd_una][snd_cwnd][ssthresh][snd_wnd][srtt][rcv_wnd]
I'm looking for a detailed documentation about content of files /proc/net/nf_conntrack and/or /proc/net/ip_contrack on Linux systems.
Yes, I know, there are many utilities which can show me the content of these files in human readable format, but... I'd like to do it on a SOHO router, with Tomato USB firmware (by Shibby).
The optware AFAIK deprecated and the entware doesn't contain any of these utilities, so I'd like to write a script instead of them, but I didn't find a detailed description of these files :(
The format of a line from /proc/net/ip_conntrack is the same as for /proc/net/nf_conntrack, except the first two columns are missing.
I'll try to summarize the format of the latter file, as I understand it from the net/netfilter/nf_conntrack_standalone.c, net/netfilter/nf_conntrack_acct.c and the net/netfilter/nf_conntrack_proto_*.c kernel source files. The term layer refers to the OSI protocol layer model.
First column: The network layer protocol name (eg. ipv4).
Second column: The network layer protocol number.
Third column: The transmission layer protocol name (eg. tcp).
Fourth column: The transmission layer protocol number.
Fifth column: The seconds until the entry is invalidated.
Sixth column (Not all protocols): The connection state.
All other columns are named (key=value) or represent flags ([UNREPLIED], [ASSURED], ...). A line can contain up to two columns having the same name (eg. src and dst). Then, the first occurrence relates to the request direction and the second occurrence relates to the response direction.
Meaning of the flags:
[ASSURED]: Traffic has been seen in both (ie. request and response) direction.
[UNREPLIED]: Traffic has not been seen in response direction yet. In case the connection tracking cache overflows, these connections are dropped first.
Please note that some column names appear only for specific protocols (eg. sport and dport for TCP and UDP, type and code for ICMP). Other column names (eg. mark) appear only if the kernel was built with specific options.
Examples:
ipv4 2 tcp 6 300 ESTABLISHED src=1.1.1.2 dst=2.2.2.2 sport=2000 dport=80 src=2.2.2.2 dst=1.1.1.1 sport=80 dport=12000 [ASSURED] mark=0 use=2 belongs to an established TCP connection from host 1.1.1.2, port 2000, to host 2.2.2.2, port 80, from which responses are sent to host 1.1.1.1, port 12000, timing out in five minutes. For this connection, packets have been seen in both directions.
ipv4 2 icmp 1 3 src=1.1.1.2 dst=1.1.1.1 type=8 code=0 id=32354 src=1.1.1.1 dst=1.1.1.2 type=0 code=0 id=32354 mark=0 use=2 belongs to an ICMP echo request packet from host 1.1.1.2 to host 1.1.1.1 with an expected echo reply packet from host 1.1.1.1 to host 1.1.1.2, timing out in three seconds.
The response destination host is not necessarily the same as the request source host, as the request source address may have been masqueraded by the response destination host.
Please note that the following information might not be up-to-date!
Fields available for all entries:
bytes (if accounting is enabled, request and response)
delta-time (if CONFIG_NF_CONNTRACK_TIMESTAMP is enabled)
dst (request and response)
mark (if CONFIG_NF_CONNTRACK_MARK is enabled)
packets (if accounting is enabled, request and response)
secctx (if CONFIG_NF_CONNTRACK_SECMARK is enabled)
src (request and response)
use
zone (if CONFIG_NF_CONNTRACK_ZONES is enabled)
Fields available for dccp, sctp, tcp, udp and udplite transmission layer protocols:
dport (request and response)
sport (request and response)
Fields available for icmp transmission layer protocol:
code (request and response)
id (request and response)
type (request and response)
Fields available for gre transmission layer protocol:
dstkey (request and response)
srckey (request and response)
stream_timeout
timeout
Allowed values for the sixth field:
dccp transmission layer protocol
CLOSEREQ
CLOSING
IGNORE
INVALID
NONE
OPEN
PARTOPEN
REQUEST
RESPOND
TIME_WAIT
sctp transmission layer protocol
CLOSED
COOKIE_ECHOED
COOKIE_WAIT
ESTABLISHED
NONE
SHUTDOWN_ACK_SENT
SHUTDOWN_RECD
SHUTDOWN_SENT
tcp transmission layer protocol
CLOSE
CLOSE_WAIT
ESTABLISHED
FIN_WAIT
LAST_ACK
NONE
SYN_RECV
SYN_SENT
SYN_SENT2
TIME_WAIT
The file ip_conntrack contains only ipv4 specific conntrack entries whereas nf_conntrack includes both ipv4 and ipv6 protocol conntrack entries.
nf_conntrack file is registered with proc file system using code in
net/netfilter/nf_conntrack_standalone.c
whereas ip_conntrack file is registered with proc file system through the code in
net/netfilter/nf_conntrack_l3proto_ipv4_compat.c
I have an application that has a TCP client and a server. I set up the client and server on separate machines. Now I want to measure how much bandwidth is being consumed ( bytes sent and received during a single run of the application). I have discovered that wireshark is one such tool that can help me get this statistic. However, wireshark seems to be GUI dependent. What I wanted was a way to automate the measuring and reporting of this statistic. I dont care about the information about individual packets captured by wireshark. I dont need that information. Is there some way to run wireshark so that all it does is write to a file, the total bytes sent and received between two hosts while the application was running on both ends?
Also, is there a better way to capture this statistic ? Through netstat or /proc/dev/net or any other tool ?
Both my machines have ubuntu 10.04 or later running on them.
Bro is an appropriate tool to measure connection-oriented statistics. You can either record a trace of your application communication or analyze it in realtime:
bro -r <trace>
bro -i <interface>
Thereafter, have a look at the connection log (conn.log) in the same directory for the amount of bytes sent and received by the application. Specifically, you're interested in the TCP payload size, which conn.log exposes via the columns orig_bytes and resp_bytes. Here is an example:
bro-cut id.orig_h id.resp_h conn_state orig_bytes resp_bytes < conn.log | head
which yields the following output:
192.168.1.102 192.168.1.1 SF 301 300
192.168.1.103 192.168.1.255 S0 350 0
192.168.1.102 192.168.1.255 S0 350 0
192.168.1.103 192.168.1.255 S0 560 0
192.168.1.102 192.168.1.255 S0 348 0
192.168.1.104 192.168.1.255 S0 350 0
192.168.1.104 192.168.1.255 S0 549 0
192.168.1.103 192.168.1.1 SF 303 300
192.168.1.102 192.168.1.255 S0 - -
192.168.1.104 192.168.1.1 SF 311 300
Each row represents a single connection, transport-layer ports omitted. The last two columns represent the bytes sent by the originator (first column) and responder (second column). The column conn_state represents the connection status. Please refer to the documentation for all possible field values. Some important values are:
S0: Connection attempt seen, no reply.
S1: Connection established, not terminated.
SF: Normal establishment and termination. Note that this is the same symbol as for state S1. You can tell the two apart because for S1 there will not be any byte counts in the summary, while for SF there will be.
REJ: Connection attempt rejected.