I am using tc to do QOS on an asterisk server. I want to prioritize voice and sip traffic but also cap all the rest to a fixed limit.
Here is my script:
#!/bin/bash
IFACE=eth1
UPSPEED=1.5mbit
tc qdisc del dev $IFACE root
tc qdisc add dev $IFACE root handle 1: prio priomap 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 0
tc qdisc add dev $IFACE parent 1:1 handle 10: sfq perturb 10
tc qdisc add dev $IFACE parent 1:2 handle 20: sfq perturb 10
tc qdisc add dev $IFACE parent 1:3 handle 30: tbf rate $UPSPEED burst 4kb mtu 1500 latency 100ms
tc filter add dev $IFACE protocol ip parent 1: prio 1 u32 match ip tos 0xb8 0xff flowid 1:1
tc filter add dev $IFACE protocol ip parent 1: prio 1 u32 match ip tos 0x60 0xff flowid 1:2
RTP and SIP traffic is well managed, being sent to the first and the second band. All other traffic is also well managed, being sent to the third band. However, for some reasons, If I download from the server, it is always at 10-16k/sec instead of the 185-190k (1.5mbit) specified in my script.
Worst, it seems that no matter how I change the tbf variables, the speed stays the same.
Using qdisc -s ls, I managed to find out that packets were dropped :
qdisc prio 1: bands 3 priomap 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 0
Sent 12610974 bytes 45683 pkt (dropped 1147, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 10: parent 1:1 limit 126p quantum 1514b perturb 10sec
Sent 7802180 bytes 36590 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc sfq 20: parent 1:2 limit 126p quantum 1514b perturb 10sec
Sent 181620 bytes 283 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc tbf 30: parent 1:3 rate 1500Kbit burst 4Kb lat 100.0ms
Sent 4627174 bytes 8810 pkt (dropped 1147, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
But I don't know why. Again, changed the tbf variables doesn't change anything, packets keep being dropped.
Please note that the mtu of eth1 is 1500.
Anyone?
Add mtu 100000 to your tc qdisc creation command. See this post for more info.
Quoting the post, in case it disappears:
Basically if your interface has TSO/GSO enabled (check using ethtool -k ethX), or you're using the loopback interface - then you'll probably hit a problem. It turns out that the loopback interface has GSO/TSO enabled as default, plus since it is a software interface its default mtu is 16384 (as compared to 1500 for normal Ethernet interface). This matters as the tbf queue checks the size of the incoming 'packets' - which in the case of GSO/TSO are much larger than a normal on-the-wire packet - instead they're up to 9 x iface's mtu. So for normal interfaces it's about 12K, but for loopback it is about 100k!
This appears to apply to Xen interfaces, too:
# ethtool -k eth0
Features for eth0:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: on
tx-checksum-unneeded: off [fixed]
tx-checksum-ip-generic: off [fixed]
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on # THIS IS TSO
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: off [fixed]
tx-tcp6-segmentation: off [fixed]
udp-fragmentation-offload: off [fixed]
generic-segmentation-offload: on # THIS IS GSO
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: off [fixed]
rx-vlan-filter: off [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: on
loopback: off [fixed]
I had the same issue. Turning off segmentation offloading will solve the problem (and you won't need to pass mtu to tc anymore).
sudo ethtool -K eth1 tso off
sudo ethtool -K eth1 gso off
sudo ethtool -K eth1 gro off
sudo ethtool --offload eth1 rx off tx off
Related
I have two Redhat VMs as below, want to access VM1's private network "192.168.122.X" from VM2.
May i know is this possible ?
VM1 with below network
route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 0 0 0 eth1
10.78.103.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.122.33 netmask 255.255.255.0 broadcast 192.168.122.255
inet6 fe80::5054:ff:feb9:de20 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:b9:de:20 txqueuelen 1000 (Ethernet)
RX packets 24179913 bytes 28549369321 (26.5 GiB)
RX errors 0 dropped 2119 overruns 0 frame 0
TX packets 5673508 bytes 769017163 (733.3 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.78.103.33 netmask 255.255.255.0 broadcast 10.78.103.255
inet6 fe80::5054:ff:feb3:4fc2 prefixlen 64 scopeid 0x20<link>
ether 52:54:00:b3:4f:c2 txqueuelen 1000 (Ethernet)
RX packets 37582534 bytes 25923821763 (24.1 GiB)
RX errors 0 dropped 4185 overruns 0 frame 0
TX packets 28245404 bytes 10327246086 (9.6 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
VM2 with below network
route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default gateway 0.0.0.0 UG 0 0 0 eth0
10.64.88.128 0.0.0.0 255.255.255.128 U 0 0 0 eth0
ifconfig
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.64.88.203 netmask 255.255.255.128 broadcast 10.64.88.255
inet6 fe80::250:56ff:feb9:a955 prefixlen 64 scopeid 0x20<link>
ether 00:50:56:b9:a9:55 txqueuelen 1000 (Ethernet)
RX packets 31613507 bytes 3779816165 (3.5 GiB)
RX errors 0 dropped 872 overruns 0 frame 0
TX packets 4874585 bytes 3778397609 (3.5 GiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
I have a Beaglebone Black (running on Debian) which I'm connecting to my laptop (running on Arch Linux) through usb. The laptop is able to read Beaglebone as a storage device. However, I am unable to connect to Beaglebone through the browser/ssh by accessing the following ip - 192.168.7.2.
ifconfig -a gives the following output-
enp0s20u2: flags=4098<BROADCAST,MULTICAST> mtu 1500
ether 54:4a:16:c1:f1:53 txqueuelen 1000 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 192.168.3.204 netmask 255.255.255.0 broadcast 192.168.3.255
inet6 fe80::2ad2:44ff:fe8e:bd61 prefixlen 64 scopeid 0x20<link>
ether 28:d2:44:8e:bd:61 txqueuelen 1000 (Ethernet)
RX packets 40024 bytes 35440467 (33.7 MiB)
RX errors 0 dropped 30 overruns 0 frame 0
TX packets 17546 bytes 2112866 (2.0 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
lo: flags=73<UP,LOOPBACK,RUNNING> mtu 65536
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10<host>
loop txqueuelen 0 (Local Loopback)
RX packets 2137 bytes 479657 (468.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 2137 bytes 479657 (468.4 KiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
wlp2s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 10.10.5.86 netmask 255.255.252.0 broadcast 10.10.7.255
ether b8:ee:65:ac:7c:65 txqueuelen 1000 (Ethernet)
RX packets 19264 bytes 18721516 (17.8 MiB)
RX errors 0 dropped 4 overruns 0 frame 0
TX packets 17238 bytes 2273813 (2.1 MiB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
enp0s20u2 is the interface of Beaglebone and has no ip allotted to it (as per the documentation, its ip should be 192.168.7.2) and thus I am unable to ssh or browse into it.
Finally succeeded by running the following command on the laptop:
sudo dhclient <interface>
Now, I'm able to ssh/browse Beaglebone Black.
I would like to retrieve the DNS / gateway of the wwan0 interface into a variable. Is there an other way (more robust) than store the result of the route command within a tab an get the value by using this ID ?
root#sn :~# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
default 176.28.206.105 0.0.0.0 UG 0 0 0 eth0
176.24.196.164 * 255.255.255.224 U 0 0 0 wwan0
176.26.210.210 176.28.211.211 255.255.255.255 UGH 0 0 0 eth0
176.28.4.4 * 255.255.0.0 U 0 0 0 eth0
172.24.201.100 * 255.255.255.255 UH 0 0 0 eth0
I'm not quite sure, what you mean by "retrieve DNS/gateway". But if you intent to get the first two columns of the output of route for the wwan0 Interface, then this could be helpful:
dest=$(route | awk '{if($8=="wwan0") print $1}')
gw=$(route | awk '{if($8=="wwan0") print $2}')
echo "data for wwan0: $dest / $gw"
prints
data for wwan0: 176.24.196.164 / *
in the case of your routing table.
In Solaris, the output of 'netstat -i' gives something like the following:
root# netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 136799 0 136799 0 0 0
igb0 1500 vulture vulture 1272272 0 347277 0 0 0
Note that there is a Queue field on the end.
In Linux, 'netstat -i' gives output with no Queue field:
[root#roseate ~]# netstat -i
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0 1500 0 2806170 0 0 0 791768 0 0 0 BMRU
eth1 1500 0 0 0 0 0 0 0 0 0 BMU
eth2 1500 0 0 0 0 0 0 0 0 0 BMU
eth3 1500 0 0 0 0 0 0 0 0 0 BMU
lo 16436 0 1405318 0 0 0 1405318 0 0 0 LRU
I've figured out how to get collisions in Linux by adding the -e option, but is there a way to get the Queue in Linux?
The only reference to queue I ever saw in netstat on Linux was when using -s, but that's probably too garrulous for your use-case?
$ netstat -na | awk 'BEGIN { RecvQ=0; SendQ=0; } { RecvQ+=$2; SendQ+=$3; } END { print "RecvQ " RecvQ/1024; print "SendQ " SendQ/1024; }'
RecvQ 255.882
SendQ 0.0507812
For per interface, I have dirty way
[spatel#us04 ~]$ for qw in `/sbin/ifconfig | grep 'inet addr:' | cut -d: -f2 | awk '{print $1}'`; do echo `/sbin/ip addr | grep $qw | awk '{print $7}'` : ; echo `netstat -na | grep $qw | awk 'BEGIN { RecvQ=0; SendQ=0; } { RecvQ+=$2; SendQ+=$3; } END { print "RecvQ " RecvQ/1024; print "SendQ " SendQ/1024; }'`; done
eth0 :
RecvQ 0 SendQ 0
eth2 :
RecvQ 0.0703125 SendQ 1.56738
:
RecvQ 0 SendQ 0
I ended up using
tc -s -d qdisc
[root#roseate ~]# tc -s -d qdisc
qdisc mq 0: dev eth2 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc mq 0: dev eth3 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
qdisc mq 0: dev eth0 root
Sent 218041403 bytes 1358829 pkt (dropped 0, overlimits 0 requeues 1)
rate 0bit 0pps backlog 0b 0p requeues 1
qdisc mq 0: dev eth1 root
Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0)
rate 0bit 0pps backlog 0b 0p requeues 0
which gives backlog bytes and packets.
Source
I would like to know the possible values of st column in /proc/net/tcp. I think the st column equates to STATE column from netstat(8) or ss(8).
I have managed to identify three codes:
sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 0100007F:08A0 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7321 1 ffff81002f449980 3000 0 0 2 -1
1: 00000000:006F 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 6656 1 ffff81003a30c080 3000 0 0 2 -1
2: 00000000:0272 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 6733 1 ffff81003a30c6c0 3000 0 0 2 -1
3: 0100007F:0277 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7411 1 ffff81002f448d00 3000 0 0 2 -1
4: 0100007F:0019 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7520 1 ffff81002f4486c0 3000 0 0 2 -1
5: 0100007F:089F 00000000:0000 0A 00000000:00000000 00:00000000 00000000 0 0 7339 1 ffff81002f449340 3000 0 0 2 -1
6: 0100007F:E753 0100007F:0016 01 00000000:00000000 02:000AFA92 00000000 500 0 18198 2 ffff81002f448080 204 40 20 2 -1
7: 0100007F:E752 0100007F:0016 06 00000000:00000000 03:000005EC 00000000 0 0 0 2 ffff81000805dc00
The above shows:
On line sl 0: a listening port on tcp/2208. st = 0A = LISTEN
On line sl 6: An established session on tcp/22. st = 01 = ESTABLISHED
On line sl 7: An socket in TIME_WAIT state after ssh logout. No inode. st = 06 = TIME_WAIT
Can anyone expand on this list? The proc(5) manpage is quite terse on the subject stating:
/proc/net/tcp
Holds a dump of the TCP socket table. Much of the information is not of use apart from debugging. The "sl" value is the kernel hash slot for the socket, the "local address" is the local address and
port number pair. The "remote address" is the remote address and port number pair (if connected). ’St’ is the internal status of the socket. The ’tx_queue’ and ’rx_queue’ are the outgoing and incom-
ing data queue in terms of kernel memory usage. The "tr", "tm->when", and "rexmits" fields hold internal information of the kernel socket state and are only useful for debugging. The "uid" field
holds the effective UID of the creator of the socket.
And on a related note, the above /proc/net/tcp output is showing a few listening processes (2208, 62, 111 etc). However, I cannot see a listening tcp connection on tcp/22, althought the established and time_wait states are shown. Yes, I can see them in /proc/net/tcp6 but should they not be present in /proc/net/tcp also? Netstat output shows it differently to applications bound only to ipv4. E.g.
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 4231/portmap
tcp 0 0 :::22 :::* LISTEN 4556/sshd
Many thanks,
-Andrew
They should match to the enum in ./include/net/tcp_states.h in the linux kernel sources:
enum {
TCP_ESTABLISHED = 1,
TCP_SYN_SENT,
TCP_SYN_RECV,
TCP_FIN_WAIT1,
TCP_FIN_WAIT2,
TCP_TIME_WAIT,
TCP_CLOSE,
TCP_CLOSE_WAIT,
TCP_LAST_ACK,
TCP_LISTEN,
TCP_CLOSING, /* Now a valid state */
TCP_MAX_STATES /* Leave at the end! */
};
As for your 2. question, are you really sure there's not an sshd listening on e.g. 0.0.0.0:22 ? If not, I suspect what you're seeing is related to v4-mapped-on-v6 sockets, see e.g. man 7 ipv6