sock_sendmsg(): implicit function calls by Kernel - linux

I tried to dump all data sent by a specific process on Linux by hooking a handler to the Kernel's function sock_sendmsg() defined in linux/socket.c.
I could do that by writing a systemtap probe handler for probe kernel.function("sock_sendmsg#net/socket.c") that dumps all data blocks passed with the 2nd argument struct msghdr *msg.
Here's the excerpt from net/socket.c:
int sock_sendmsg(struct socket *sock, struct msghdr *msg, size_t size)
{
struct kiocb iocb;
struct sock_iocb siocb;
int ret;
init_sync_kiocb(&iocb, NULL);
iocb.private = &siocb;
ret = __sock_sendmsg(&iocb, sock, msg, size);
if (-EIOCBQUEUED == ret)
ret = wait_on_sync_kiocb(&iocb);
return ret;
}
I tested my systemtap script hook_sendmsg.stp. First I ran hook_sendmsg.stp in one terminal. Then I opened another terminal and invoked telnet command to connect to stackoverflow.com and typed HEAD / HTTP/1.0<Enter twice> in the terminal. I got the following output from hook_sendmsg.stp:
root#debian:~# stap -g hook_sendmsg.stp
message block [0]; type=3(raw); state=1(unconnected)
14 00 00 00 16 00 01 03 ec 95 f4 52 00 00 00 00 |...........R....|
00 00 00 00 |................|
message block [0]; type=3(raw); state=1(unconnected)
14 00 00 00 16 00 01 03 ec 95 f4 52 00 00 00 00 |...........R....|
00 00 00 00 |................|
message block [0]; type=2(udp); state=1(unconnected)
4d 0d 01 00 00 01 00 00 00 00 00 00 0d 73 74 61 |M............sta|
63 6b 6f 76 65 72 66 6c 6f 77 03 63 6f 6d 00 00 |ckoverflow.com..|
01 00 01 |................|
message block [0]; type=2(udp); state=1(unconnected)
0f 1e 01 00 00 01 00 00 00 00 00 00 0d 73 74 61 |.............sta|
63 6b 6f 76 65 72 66 6c 6f 77 03 63 6f 6d 00 00 |ckoverflow.com..|
1c 00 01 |................|
message block [0]; type=1(tcp); state=3(connected)
48 45 41 44 20 2f 20 48 54 54 50 2f 31 2e 30 0d |HEAD / HTTP/1.0.|
0a |................|
message block [0]; type=1(tcp); state=3(connected)
0d 0a |................|
This shows that totally sock_sendmsg() were called 6 times in the context of telnet. Obviously the 3rd and 4th are DNS queries to Google's public DNS servers 8.8.8.8 and 8.8.4.4. The 5th and 6th are the two lines of HTTP request sent from telnet. But what were the 1st and 2nd called for? Did they called internally by the Kernel?
Thanks in advance.

They are NETLINK messages, as you can see if you monitor the telnet command from userspace with strace (check the socket family):
sendto(3, "\24\0\0\0\26\0\1\3?\254\364R\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
sendto(3, "\24\0\0\0\26\0\1\3?\254\364R\0\0\0\0\0\0\0\0", 20, 0, {sa_family=AF_NETLINK, pid=0, groups=00000000}, 12) = 20
send(3, "\212\355\1\0\0\1\0\0\0\0\0\0\rstackoverflow\3com\0\0"..., 35, MSG_NOSIGNAL) = 35
send(3, "\241\353\1\0\0\1\0\0\0\0\0\0\rstackoverflow\3com\10i"..., 50, MSG_NOSIGNAL) = 50
send(3, "|\341\1\0\0\1\0\0\0\0\0\0\rstackoverflow\3com\0\0"..., 35, MSG_NOSIGNAL) = 35
send(3, "HEAD / HTTP/1.0\r\n", 17, 0) = 17
send(3, "\r\n", 2, 0) = 2
Netlink is an interface used by the userspace processes to communicate with kernel (so, they are send by the telnet), usually for routing, firewalling... pourposes [1]. Some usual network C functions (i.e. getsockname) works internally with this kind of NETLINK messages.
If you want to investigate a bit more you should translate the payload of the send_message to the netlink header structure and check what type of messages are.:
struct nlmsghdr {
__u32 nlmsg_len; /* Length of message including header. */
__u16 nlmsg_type; /* Type of message content. */
__u16 nlmsg_flags; /* Additional flags. */
__u32 nlmsg_seq; /* Sequence number. */
__u32 nlmsg_pid; /* Sender port ID. */
};
[1]http://man7.org/linux/man-pages/man7/netlink.7.html

Related

Wrong inputTranscript in the lex response

I am trying to test amazon web node api sdk for lex for audio input and output and successfully configured it, but when I test it, it sends me garbage inputTranscript(like "yeah", "oh no", "um yeah", etc.) in response.
The request parameters are as below,
var params = {
botAlias: 'Test', /* required */
botName: 'Revert', /* required */
contentType: 'audio/l16; rate=16000; channels=1', /* required */
inputStream: <Buffer 52 49 46 46 80 55 02 00 57 41 56 45 66 6d 74 20 10 00 00 00 01 00 02 00 80 3e 00 00 00 7d 00 00 04 00 10 00 64 61 74 61 54 55 02 00 00 00 00 00 00 00 ... >, /* required */
userId: '12321', /* required */
accept: 'audio/*',
sessionAttributes: {"firstName": "Joe"}
};
The response I am getting as below,
{ contentType: 'audio/mpeg',
sessionAttributes: { firstName: 'Joe' },
message: 'Sorry No Match',
dialogState: 'ElicitIntent',
inputTranscript: 'yeah',
audioStream: <Buffer 49 44 33 04 00 00 00 00 00 23 54 53 53 45 00 00 00 0f 00 00 03 4c 61 76 66 35 37 2e 35 36 2e 31 30 31 00 00 00 00 00 00 00 00 00 00 00 ff f3 60 c4 00 ... > }
For voice recording I am using my laptop inbuilt mic and RecordRTC api as,
recorder = RecordRTC(microphone, {
type: 'audio',
recorderType: StereoAudioRecorder,
desiredSampRate: 16000
});
The recorded voice is encoded into base64 and sent to node server where it is decoded back and sent to lex api in buffer format.

Internet Shortcut IDList value decoding

I have been trying to find any information on a value that is stored in an url shortcut that is generated from MS-Office for SharePoint Libraries. We are moving away from Office 2007 to Office 365 and the feature of 2007 that creates these links doesn't appear to be available in Office 365.
If a .url file doesn't have the IDList value when you click on it in a save-as dialog it changes the file name to the shortcut. I tried changing the .url to .lnk but non-office applications can't handle them.
Contents of .url file
[InternetShortcut]
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA8gPuuBAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAAAAAAQAQZk12cAAgNAgAAEAw7+CAAAAAAAAAAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
URL=file://blart/DavWWWRoot/teams/edms
IconFile=C:\ProgramData\Microsoft\Office\SharePointTeamSite.ico
IconIndex=0
[{000214A0-0000-0000-C000-000000000046}]
Prop3=19,9
Same user regenerated file
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA0baRNAAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAAAAAAQAQZk12cAAgNAgAAEAw7+CAAAAAAAAAAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
Value from a file from another user
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA4BZpCAAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAdHBlfQAQZk12cAAgNAgAAEAw7+qXQJdJdHBlfqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
I have not been able to figure out how to decode this value or have I been able to figure out what data is being stored in it. I have read through a bunch of different ways to create shortcuts programmatically with Windows scripting and shell 32 in Powershell and C# wrappers, C++ looking for any info. The only glimmer of hope I have is that shell links also have an item id list of pidl's, but I can't confirm if this is related.
What I do know is that the IDList can be moved to another .url and it will take you to the original path. There are some sections of the IDList that change depending on the user who created them and or when the url files are recreated.
Internet shortcuts
https://msdn.microsoft.com/en-us/library/bb776784(v=vs.85).aspx
Shell link
https://msdn.microsoft.com/en-us/library/dd891312.aspx
The .URL file format is not officially documented and only the first 2 bytes of the IDList format is documented.
I believe the IDList string used to be stored by calling WritePrivateProfileStruct and at some point changed to the format you posted. A function called TextToBinary decodes the string into a IStream instance but I did not bother trying to figure out the format. The stream is stuffed into a PROPVARIANT and the documented IUniformResourceLocator interface knows how to deal with all of this for you but unfortunately the idlist entry is not documented as a property you can read/write.
The PIDL (IDList) after .INI string decoding and some basic formating looks like this:
:1: 20 bytes
{208D2C60-3AEA-1069-A2D7-08002B30309D}:Network
:2: 78 bytes
4E 00 C3 01 E5 5C 5C 63 6F 66 6B 2E 6B 65 6C 6F N....\\c ofk.kelo
77 6E 61 2E 63 61 5C 44 61 76 57 57 57 52 6F 6F wna.ca\D avWWWRoo
74 00 57 65 62 20 43 6C 69 65 6E 74 20 4E 65 74 t.Web Cl ient Net
77 6F 72 6B 00 68 74 74 70 3A 2F 2F 63 6F 66 6B work.htt p://cofk
2E 6B 65 6C 6F 77 6E 61 2E 63 61 00 2E 00 __ __ .kelowna .ca...
:3: 76 bytes
4C 00 31 00 00 00 00 00 00 00 00 00 10 00 74 65 L.1..... ......te
61 6D 73 00 38 00 08 00 04 00 EF BE 00 00 00 00 ams.8... ........
00 00 00 00 2A 00 00 00 00 00 00 00 00 00 00 00 ....*... ........
00 00 00 00 3C F8 B8 6E 00 00 00 00 00 00 74 00 ....<..n ......t.
65 00 61 00 6D 00 73 00 00 00 14 00 __ __ __ __ e.a.m.s. ....
:4: 74 bytes
4A 00 31 00 00 00 00 00 00 00 00 00 10 00 65 64 J.1..... ......ed
6D 73 00 00 36 00 08 00 04 00 EF BE 00 00 00 00 ms..6... ........
00 00 00 00 2A 00 00 00 00 00 00 00 00 00 00 00 ....*... ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 65 00 ........ ......e.
64 00 6D 00 73 00 00 00 14 00 __ __ __ __ __ __ d.m.s... ..
:5: 0 bytes
and the result after asking the shell to convert this PIDL is \\cofk.kelowna.ca\DavWWWRoot\teams\edms.
The difference between the 3 examples you posted is just in the path part of the PIDL (teams\edms), probably the modified date of the folder or uninitialized data.

How to get the offset in a block device of an inode in a deleted partition

During a fresh installation, I accidentally formatted a disk containing datas. I have tried using some tools: testdisk, foremost, but I did not get good results. (see my unsuccessful post on superuser).
So I have decided to read some docs about ext2 filesystem structure, and I could get some results:
The deleted partition have a directory tree like that:
dev
|-scripts
|-projects
|-services
|-...
Medias
|-downloads
|-Musique
|-...
backup
...
So, based on the ext2 directory entry format:
Directory Entry
Starting_Byte Ending_Byte Size_in_Bytes Field_Description
0 3 4 Inode
4 5 2 Total size of this entry (Including all subfields)
6 6 1 Name Length least-significant 8 bits
7 7 1 Type indicator (only if the feature bit for "directory entries have file type byte" is set, else this is the most-significant 8 bits of the Name Length)
8 8+N-1 N Name characters
I tried to find some datas matching this structure.
I used this script:
var bindexOf = require('buffer-indexof');
var currentOffset=0;
var deviceReadStream = fs.createReadStream("/dev/sdb");
deviceReadStream.on('error',function(err){
console.log(err);
});
deviceReadStream.on('data',function(data){
var dirs = ["dev","scripts","services","projects","Medias","downloads","Musique","backup"];
dirs.forEach(function(dir){
dirOctetFormat = new Buffer(2);
dirOctetFormat.writeUInt8(dir.length,0);
dirOctetFormat.writeUInt8(2,1);// type is directory
dirOctetFormat= Buffer.concat( [dirOctetFormat, new Buffer(dir)]);
var offset = bindexOf( data, dirOctetFormat );
if( offset >= 0 ){
console.log( dir + " entry found at offset " + (currentOffset + offset) );
}
});
currentOffset += data.length;
});
}
I found data which seems to be the directory entry of the dev directory:
===== Current offset: 233590226944 - 217.5478515625Gio ======
scripts entry found at offset 233590227030
services entry found at offset 233590227014
projects entry found at offset 233590228106
If it is the case, I got the inode numbers of its children directories: scripts, projects, services,...
But I do not know what to do with that!
I tried to deduce the location of these inodes, based on this guide,
but as I was unable to find a superblock of the deleted filesystem, I just have to make guesses about the block size, the number of blocks, ...
and that seems a little bit fuzzy to me to hope obtaining a result.
So could you have some intervals for all values needed to obtain the offset of an inode, and a more formal formula to get this offset?
If you have only erased the partition table (or modified it) you can still get your data, if data has not been reused for something else.
ext2 filesystems have a MAGIC number in superblock, so to recover your partition you have only to search for it. I did this on one machine and was able to recover not one, but seven partitions in one disk. You have some chances to get invalid numbers, but just search for that magic. Magic number is defined in include/uapi/linux/magic.h and value is #define EXT2_SUPER_MAGIC 0xEF53 (it's found at offset #define EXT2_SB_MAGIC_OFFSET 0x38 ---from file include/linux/ext2_fs.h)
To search for the superblock, just try to find 0xef53 at offset 0x38 in one sector of the disk, it will mark the first block of the partition. Be careful, that superblock is replicated several times in one partition, so you'll find all the copies of it.
Good luck! (I had when it happened to me)
Edit (To illustrate with an example)
Just see the magic number in one of my own partitions:
# hd /dev/sda3 | head -20
00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 40 62 08 00 00 87 21 00 26 ad 01 00 f6 30 15 00 |#b....!.&....0..|
00000410 1d 31 08 00 00 00 00 00 02 00 00 00 02 00 00 00 |.1..............|
00000420 00 80 00 00 00 80 00 00 90 1f 00 00 cf 60 af 55 |.............`.U|
00000430 fc 8a af 55 2d 00 ff ff 53 ef 01 00 01 00 00 00 |...U-...S.......|<- HERE!!!
00000440 36 38 9d 55 00 00 00 00 00 00 00 00 01 00 00 00 |68.U............|
00000450 00 00 00 00 0b 00 00 00 00 01 00 00 3c 00 00 00 |............<...|
00000460 46 02 00 00 7b 00 00 00 5a bf 87 15 12 8f 44 3b |F...{...Z.....D;|
00000470 97 e7 f3 74 4d 75 69 12 72 6f 6f 74 00 00 00 00 |...tMui.root....|
00000480 00 00 00 00 00 00 00 00 2f 00 61 72 67 65 74 00 |......../.arget.|
00000490 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000004c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 02 |................|
000004d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000004e0 08 00 00 00 00 00 00 00 00 00 00 00 93 54 99 ab |.............T..|
000004f0 aa 64 46 b3 a6 73 94 34 a3 79 46 28 01 01 00 00 |.dF..s.4.yF(....|
00000500 0c 00 00 00 00 00 00 00 e5 61 92 55 0a f3 02 00 |.........a.U....|
00000510 04 00 00 00 00 00 00 00 00 00 00 00 ff 7f 00 00 |................|
00000520 00 80 10 00 ff 7f 00 00 01 00 00 00 ff ff 10 00 |................|
Remember it is on offset 0x38 counted from the block origin, and assume the super block is the second block (block 0 reserved for bootcode, so it will be block 1, with two sectors per block, to make 1k blocksize) in the partition, so you'll have to rewind 0x438 bytes from the beginning of the magic number to get the partition origin.
I have run the command on my whole disk, getting the following result:
# hd /dev/sda | grep " [0-9a-f][0-9a-f] 53 ef" | sed -e 's/^/ /' | head
006f05f0 ee 00 00 11 66 0a 00 00 53 ef 00 00 11 66 2d 00 |....f...S....f-.|
007c21d0 55 2a aa 7d f4 aa 89 55 53 ef a4 91 70 40 c1 00 |U*.}...US...p#..|
20100430 fc 8a af 55 2d 00 ff ff 53 ef 01 00 01 00 00 00 |...U-...S.......|
2289a910 0f 8f 4f 03 00 00 81 fe 53 ef 00 00 0f 84 ce 04 |..O.....S.......|
230d4c70 0a 00 00 00 1c 00 00 00 53 ef 01 00 00 00 00 00 |........S.......|
231b7e50 a0 73 07 00 00 00 00 00 53 ef 0d 00 00 00 00 00 |.s......S.......|
23dbd230 d5 08 ad 2b ee 71 07 8a 53 ef c2 89 d4 bb 09 1f |...+.q..S.......|
25c0c9e0 06 00 00 00 00 4f 59 c0 53 ef 32 c0 0e 00 00 00 |.....OY.S.2.....|
25d72ca0 b0 b4 7b 3d a4 f7 84 3b 53 ef ba 3c 1f 32 b9 3c |..{=...;S..<.2.<|
25f0eab0 f1 fd 02 be 28 59 67 3c 53 ef 9c bd 04 30 72 bd |....(Yg<S....0r.|
Clearly, there are much more uninteresting lines in this listing than the ones we need. To locate the one interesting here, we have to do some computing with the numbers. We have seen that sectors are 512 bytes long (this is 0x200 in hex) and we can have the superblock magic at offset 0x438, so we expect valid offsets to be at 0xXXXXXX[02468ace]38 only. Just select the lines with offsets ending in that expression, and you'll get the first superblock valid (in the third line) at offset 0x20100430.
Substract 0x430 to give the byte offset of the partition (0x20100000, and then, divide the result by 0x200, giving 0x100800, or 1050624)
# fdisk -l /dev/sda | sed -e 's/^/ /'
Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DF97DAD4-727D-4BB3-BD7B-3C5A584A2747
Device Start End Sectors Size Type
/dev/sda1 2048 526335 524288 256M EFI System
/dev/sda2 526336 1050623 524288 256M BIOS boot
/dev/sda3 1050624 18628607 17577984 8.4G Linux filesystem <-- HERE!!!
/dev/sda4 18628608 77221887 58593280 28G Linux filesystem
/dev/sda5 77221888 85035007 7813120 3.7G Linux filesystem
/dev/sda6 85035008 104566783 19531776 9.3G Linux filesystem
/dev/sda7 104566784 135817215 31250432 14.9G Linux swap
/dev/sda8 135817216 155348991 19531776 9.3G Linux filesystem
/dev/sda9 155348992 1953523711 1798174720 857.4G Linux filesystem

DNS txt record invalid packet, FORMERR

I'm having trouble with my home made "for fun" nameserver. It's been a couple of months since I updated it so I'm a bit rusty and thought I'd ask here and see if someone else sees what's wrong. I'm getting a FORMERR when asking for a TXT record, and the same problem occur on different domains, so there's probably something wrong in the packet formatting. Anyone?
dig txt ffffff.com #ns1.ffffff.com
;; Got bad packet: FORMERR
1024 bytes
ce bf 84 00 00 01 00 01 00 02 00 00 06 66 66 66 .............fff
66 66 66 03 63 6f 6d 00 00 10 00 01 c0 0c 00 10 fff.com.........
00 01 00 00 02 58 00 13 12 57 65 6c 63 6f 6d 65 .....X...Welcome
20 74 6f 20 66 66 66 66 66 66 66 00 c0 0c 00 02 .to.fffffff.....
00 01 00 00 02 58 00 10 03 6e 73 31 06 66 66 66 .....X...ns1.fff
66 66 66 03 63 6f 6d 00 c0 0c 00 02 00 01 00 00 fff.com.........
02 58 00 10 03 6e 73 32 06 66 66 66 66 66 66 03 .X...ns2.ffffff.
63 6f 6d 00 00 00 00 00 00 00 00 00 00 00 00 00 com.............
In the example above supplied, I added an incorrect 00 (null terminator) at the end of the TXT-string. After removing the null terminator from the TXT records, the txt records now work on my nameserver.

nodejs add null terminated string to buffer

I am trying to replicate a packet.
This packet:
2C 00 65 00 03 00 00 00 00 00 00 00 42 4C 41 5A
45 00 00 00 00 00 00 00 00 42 4C 41 5A 45......
2c 00 is the size of the packet...
65 00 is the packet id 101...
03 00 is the number of elements in the array...
Now here comes my problem, 42 4C 41 5A 45 is a string... There are exactly 3 Instances of that string in that packet if it is complete... But my problem is it is not just null terminated it has 00 00 00 00 spaces between those instances.
My code:
function channel_list(channels) {
var packet = new SmartBuffer();
packet.writeUInt16LE(101); // response packet for list of channels
packet.writeUInt16LE(channels.length)
channels.forEach(function (key){
console.log(key);
packet.writeStringNT(key);
});
packet.writeUInt16LE(packet.length + 2, 0);
console.log(packet.toBuffer());
}
But how do I add the padding?
I am using this package, https://github.com/JoshGlazebrook/smart-buffer/
Smart-Buffer keeps track of its position for you so you do not need to specify an offset to know where to insert the data to pad your string. You could do something like this with your existing code:
channels.forEach(function (key){
console.log(key);
packet.writeString(key); // This is the string with no padding added.
packet.writeUInt32BE(0); // Four 0x00's are added after the string itself.
});
I'm assuming you want: 42 4C 41 5A 45 00 00 00 00 42 4C 41 5A 45 00 00 00 00 etc.
Editing based on comments:
There is no built in way to do what you want, but you could do something like this:
channels.forEach(function (key){
console.log(key);
packet.writeString(key);
for(var i = 0; i <= (9 - key.length); i++)
packet.writeInt8(0);
});

Resources