strace can't parse my netlink message, but it appears valid - linux

Question

I'm taking my first stab at using the NetLink API in linux. I'm using Rust because it hasn't bitten me in the ass enough for me to go back to C yet. I figured a good place to start would be to enumerate the netlink devices, since there's already a utility that does that (ip link). When I run the Rust code it returns 3 devices out of the 6 devices that ip link returns. So I'm trying to inspect the request that I'm sending vs. what ip link is sending.
# Mine
$ sudo strace -ff -v -e trace=%network mine 2>&1 |grep sendto
sendto(3,
{
{nlmsg_len=40, nlmsg_type=0x12 /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x300, nlmsg_seq=1620766291, nlmsg_pid=0},
"\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x1d\x00\x01\x00\x00\x00"
},
40, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, 12) = 40
$ sudo strace -ff --const-print-style=raw -v -e trace=%network mine 2>&1 |grep sendto
sendto(3,
{
{nlmsg_len=40, nlmsg_type=0x12, nlmsg_flags=0x301, nlmsg_seq=1620766293, nlmsg_pid=0},
"\x11\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x1d\x00\x01\x00\x00\x00"
},
40, 0, {sa_family=0x10, nl_pid=0, nl_groups=00000000}, 12) = 40
# ip link
$ sudo strace -ff -v -e trace=%network ip link 2>&1 |grep sendto
sendto(3,
{
{nlmsg_len=40, nlmsg_type=RTM_GETLINK, nlmsg_flags=NLM_F_REQUEST|NLM_F_DUMP, nlmsg_seq=1620765818, nlmsg_pid=0},
{ifi_family=AF_PACKET, ifi_type=ARPHRD_NETROM, ifi_index=0, ifi_flags=0, ifi_change=0},
{
{nla_len=8, nla_type=IFLA_EXT_MASK},
1
}
},
40, 0, NULL, 0) = 40
$ sudo strace -ff --const-print-style=raw -v -e trace=%network ip link 2>&1 |grep sendto
sendto(3,
{
{nlmsg_len=40, nlmsg_type=0x12, nlmsg_flags=0x301, nlmsg_seq=1620765854, nlmsg_pid=0},
{ifi_family=0x11, ifi_type=0, ifi_index=0, ifi_flags=0, ifi_change=0},
{
{nla_len=8, nla_type=0x1d},
1
}
},
40, 0, NULL, 0) = 40
The two differences that I can spot are that (A) ip link binds the socket whereas the Rust library provides a dest_addr argument to sendto. I doubt this is relevant. (B) strace can parse the structure sent by ip link but can't seem to fully parse the structure sent by my Rust program. strace says both programs agree on the struct nlmsghdr header. However, strace doesn't parse the struct ifinfomsg of my program. Looking at the bytes, however, it appears to match.
The rust library I'm using netlink-packet-route doesn't seem to have an obvious equivalent to struct rtattr. Adjacent to the ifinfomsg (called LinkHeader in the rust library) there's a list of what it calls Nla. The enum values line up with their C equivilents and as shown above the constant values line up too.
man rtnetlink doesn't mention anything about IFLA_EXT_MASK as a possible rtattr for RTM_GETLINK and I don't get many other hits in documentation for it.
I guess the next step is to pop both into gdb and see if there's any other observable difference between the two calls.
The super-ugly demo-quality Rust code that produces the above message:
use std::io::{Result, Error, ErrorKind};
use netlink_sys::{Socket, protocols::NETLINK_ROUTE, SocketAddr};
use netlink_packet_core::{NetlinkMessage, NetlinkHeader, NLM_F_DUMP, NLM_F_REQUEST};
use netlink_packet_route::{RtnlMessage, LinkMessage, LinkHeader, AF_PACKET, ARPHRD_NETROM};
use netlink_packet_core::NetlinkPayload::InnerMessage;
use netlink_packet_route::RtnlMessage::NewLink;
use netlink_packet_route::link::nlas::Nla;
use std::time::{UNIX_EPOCH, SystemTime};
fn main() -> Result<()> {
println!("Hello, world!");
let socket = Socket::new(NETLINK_ROUTE)?;
let kernel_addr = SocketAddr::new(0, 0);
let msgid = SystemTime::now().duration_since(UNIX_EPOCH).map_err(|_|{Error::from(ErrorKind::Other)})?.as_secs();
let nlas: Vec<Nla> = vec![Nla::ExtMask(1)];
let mut packet = NetlinkMessage {
header: NetlinkHeader {
sequence_number: msgid as u32,
flags: NLM_F_DUMP | NLM_F_REQUEST,
..Default::default()
},
payload: RtnlMessage::GetLink(LinkMessage {header: LinkHeader {
interface_family: AF_PACKET as u8,
link_layer_type: ARPHRD_NETROM,
..LinkHeader::default()}, nlas, ..LinkMessage::default()}).into(),
};
packet.finalize();
let mut buf = vec![0; packet.header.length as usize];
packet.serialize(&mut buf[..]);
let n_sent = socket.send_to(&buf[..], &kernel_addr, 0).unwrap();
assert_eq!(n_sent, buf.len());
let mut buf = vec![0; 4096];
loop {
let (n_received, sender_addr) = socket.recv_from(&mut buf[..], 0).unwrap();
assert_eq!(sender_addr, kernel_addr);
for i in &mut buf[n_received..] { *i = 0 };
if n_received == 4096 { return Err(Error::from(ErrorKind::OutOfMemory))}
if buf[4] == 2 && buf[5] == 0 {
println!("the kernel responded with an error");
return Err(Error::from(ErrorKind::ConnectionReset));
}
if buf[4] == 3 && buf[5] == 0 {
println!("Done");
return Ok(());
}
let resp = NetlinkMessage::<RtnlMessage>::deserialize(&buf).expect("Failed to deserialize message");
match resp.payload {
InnerMessage(i) => match i {
NewLink(j) => {
let name = j.nlas.iter().find(|nla| { matches!(nla, Nla::IfName(_))});
println!("index {:?}: {:?}", j.header.index, name);
},
_ => println!("Some other message type")
},
_ => println!("Some other message type")
}
}
}

Answer 1

So it turns out that netlink will send more than one response structure in a single response packet. I discovered this by setting up a monitor interface for netlink and tcpdump'ing it. The total number of bytes sent back in response to ip link and to my program were similar, but they were distributed over differing numbers of packets. The size of the buffer used on the recvmsg call affects the way netlink packs the results.
sudo ip link add nlmon0 type nlmon
sudo ip link set dev nlmon0 up
sudo tcpdump -i nlmon0 -w /tmp/dump &
ip link
mine
fg
CTRL+C
tshark -r /tmp/dump
1 0.000000 → Netlink route 56
2 0.000073 → Netlink route 2660
3 0.000127 → Netlink route 2688
4 0.000205 → Netlink route 4060
5 0.000258 → Netlink 36
6 3.622386 → Netlink route 56
7 3.622449 → Netlink route 2660
8 3.622512 → Netlink route 2688
9 3.623179 → Netlink route 2740
10 3.623748 → Netlink route 1336
11 3.624273 → Netlink 36
I will need to figure out how this Rust library is setup to deal with this.