I have a need to test if a program that I'm writing is parsing the gzip header correctly, and that includes reading the FEXTRA, FNAME, and FCOMMENT fields. Yet it seems that gzip doesn't support creating archives with the FEXTRA and FCOMMENT fields -- only FNAME. Are there any existing tools which can do all three of these?
The Perl module IO::Compress::Gzip optionally lets you set the three fields you are intrested in. (Fair disclosure: I am the author of the module)
Here is some sample code that sets FNAME to "filename", FCOMMENT to "This is a comment" and creates an FEXTRA field with a single subfield with ID "ab" and value "cde".
use IO::Compress::Gzip qw(gzip $GzipError);
gzip \"payload" => "/tmp/test.gz",
Name => "filename",
Comment => "This is a comment",
ExtraField => [ "ab" => "cde"]
or die "Cannot create gzip file: $GzipError" ;
And here is a hexdump of the file it created.
00000000 1f 8b 08 1c cb 3b 3a 5a 00 03 07 00 61 62 03 00 |.....;:Z....ab..|
00000010 63 64 65 66 69 6c 65 6e 61 6d 65 00 54 68 69 73 |cdefilename.This|
00000020 20 69 73 20 61 20 63 6f 6d 6d 65 6e 74 00 2b 48 | is a comment.+H|
00000030 ac cc c9 4f 4c 01 00 15 6a 2c 42 07 00 00 00 |...OL...j,B....|
0000003f
Related
Assume I visit the following link somerandomwebsite.com/a.pdf and download the file a.pdf. Now assume that the host replaces a.pdf with a new version of the same file under the same name so now the previous link would lead me to download a different file.
Is there a way for me to prove that the file I downloaded was indeed downloaded from that link at a given time?
File Attribute
This is by no means a proof you can use to convince someone else, but if your browser, platform, and file system support it, you may find an xattr on the downloaded file that tells you the URL.
On macOS:
$ xattr -l -p com.apple.metadata:kMDItemWhereFroms Downloads/logo-stackoverflow.svg
com.apple.metadata:kMDItemWhereFroms:
00000000 62 70 6C 69 73 74 30 30 A1 01 5F 10 47 68 74 74 |bplist00.._.Ghtt|
00000010 70 73 3A 2F 2F 73 74 61 63 6B 6F 76 65 72 66 6C |ps://stackoverfl|
00000020 6F 77 2E 64 65 73 69 67 6E 2F 61 73 73 65 74 73 |ow.design/assets|
00000030 2F 69 6D 67 2F 6C 6F 67 6F 73 2F 73 6F 2F 6C 6F |/img/logos/so/lo|
00000040 67 6F 2D 73 74 61 63 6B 6F 76 65 72 66 6C 6F 77 |go-stackoverflow|
00000050 2E 73 76 67 08 0A 00 00 00 00 00 00 01 01 00 00 |.svg............|
00000060 00 00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 |................|
00000070 00 00 00 00 00 54 |.....T|
00000076
On Linux:
$ getfattr -d logo-stackoverflow.svg
# file: logo-stackoverflow.svg
user.xdg.origin.url="https://stackoverflow.design/assets/img/logos/so/logo-stackoverflow.svg"
Wayback Machine
You might find the URL was archived by a service, such as Internet Archive Wayback Machine. For example: https://web.archive.org/web/20201101014003/https://stackoverflow.design/assets/img/logos/so/logo-stackoverflow.svg
Timestamping Authority (TSA)
For a convincing proof, you might rely on a third-party to access the URL and provide a cryptographic signature with the contents, including a timestamp. For example: freetsa.org provides a "URL screenshot online" service you can use to get a signed PDF showing the accessed website.
My question is how I get python to read a file where the text is in 16bit characters. The rest of the post describes the situation.
I have a text file which is a playlist export from iTunes.
Here is a short section including the header
Name Artist Composer Album Grouping Work Movement Number Movement Count Movement Name Genre Size Time Disc Number Disc Count Track Number Track Count Year Date Modified Date Added Bit Rate Sample Rate Volume Adjustment Kind Equalizer Comments Plays Last Played Skips Last Skipped My Rating
Keyboard Works of the Masters Randolph Hokanson Pan125b 2054816 64 03/11/2017, 18:00 03/11/2017, 17:01 256 44100 MPEG audio file 1 03/11/2017, 17:02 4 08/03/2018, 16:07
08 Traccia 08 11159905 464 03/11/2017, 17:39 03/11/2017, 16:59 192 48000 MPEG audio file 1 03/11/2017, 16:59
09 Traccia 09 17787361 741 03/11/2017, 17:39 03/11/2017, 16:58 192 48000 MPEG audio file 5 08/03/2018, 10:58
10 Traccia 10 10128290 421 03/11/2017, 17:39 03/11/2017, 16:58 192 48000 MPEG audio file 1 03/11/2017, 16:58
When I use this code to read it, the program hangs. (The i holds the number of lines in the file). The hex dumps which follow seem to show the export from iTunes is in 16bit characters.
The complete code for reading the text file is
file_name="full path to file goes here"
f = open(file_name, "r")
i=227
for x in range(0, i):
line = f.readline()
When I read the code into text wrangler, selected all the text, and pasted it into a new document. The code worked fine.
A text dump of part of the original file looks like this to start with the new file following
00000000: FF FE 4E 00 61 00 6D 00 65 00 09 00 41 00 72 00 ..N.a.m.e...A.r.
00000010: 74 00 69 00 73 00 74 00 09 00 43 00 6F 00 6D 00 t.i.s.t...C.o.m.
00000020: 70 00 6F 00 73 00 65 00 72 00 09 00 41 00 6C 00 p.o.s.e.r...A.l.
00000030: 62 00 75 00 6D 00 09 00 47 00 72 00 6F 00 75 00 b.u.m...G.r.o.u.
00000040: 70 00 69 00 6E 00 67 00 09 00 57 00 6F 00 72 00 p.i.n.g...W.o.r.
00000050: 6B 00 09 00 4D 00 6F 00 76 00 65 00 6D 00 65 00 k...M.o.v.e.m.e.
00000060: 6E 00 74 00 20 00 4E 00 75 00 6D 00 62 00 65 00 n.t. .N.u.m.b.e.
00000070: 72 00 09 00 4D 00 6F 00 76 00 65 00 6D 00 65 00 r...M.o.v.e.m.e.
00000080: 6E 00 74 00 20 00 43 00 6F 00 75 00 6E 00 74 00 n.t. .C.o.u.n.t.
00000090: 09 00 4D 00 6F 00 76 00 65 00 6D 00 65 00 6E 00 ..M.o.v.e.m.e.n.
000000A0: 74 00 20 00 4E 00 61 00 6D 00 65 00 09 00 47 00 t. .N.a.m.e...G.
000000B0: 65 00 6E 00 72 00 65 00 09 00 53 00 69 00 7A 00 e.n.r.e...S.i.z.
000000C0: 65 00 09 00 54 00 69 00 6D 00 65 00 09 00 44 00 e...T.i.m.e...D.
000000D0: 69 00 73 00 63 00 20 00 4E 00 75 00 6D 00 62 00 i.s.c. .N.u.m.b.
000000E0: 65 00 72 00 09 00 44 00 69 00 73 00 63 00 20 00 e.r...D.i.s.c. .
000000F0: 43 00 6F 00 75 00 6E 00 74 00 09 00 54 00 72 00 C.o.u.n.t...T.r.
New file
0000: 4E 61 6D 65 09 41 72 74 69 73 74 09 43 6F 6D 70 Name.Artist.Comp
0010: 6F 73 65 72 09 41 6C 62 75 6D 09 47 72 6F 75 70 oser.Album.Group
0020: 69 6E 67 09 57 6F 72 6B 09 4D 6F 76 65 6D 65 6E ing.Work.Movemen
0030: 74 20 4E 75 6D 62 65 72 09 4D 6F 76 65 6D 65 6E t Number.Movemen
0040: 74 20 43 6F 75 6E 74 09 4D 6F 76 65 6D 65 6E 74 t Count.Movement
0050: 20 4E 61 6D 65 09 47 65 6E 72 65 09 53 69 7A 65 Name.Genre.Size
Your file beginning looks like UTF-16 - see Byte order marks - Wikipedia
Use
file_name="full path to file goes here"
with io.open(file_name,'r', encoding='utf-16-le') as f:
for line in f:
# do something with line
when opening it.
There is no need to use range() or readlines() when reading line by line. If you really need the line-numbers use:
for lineNr,line in enumerate(f):
This will generate an alert:
alert tcp any any <> any any (msg:"Test_A"; sid:3000001; rev:1;)
This will not:
alert tcp any any <> any any (msg:"Test_B"; content:"badurl.com"; http_header; sid:3000002; rev:1;)
I have tried: fast_pattern:only; metadata:service http; nocase; http_header; and others. I cannot get it to work at this generic level. Any ideas why the content attribute does not work? The packet has a URL.
Updated from the comments
0000 9c d2 4b 7d 96 60 3c 15 c2 dc 48 fa 08 00 45 00 ..K}.<. ..H...E.
0010 01 5c ac 2c 40 00 40 06 cf f5 c0 a8 c8 1e 41 fe .\.,#.#. ......A.
0020 f2 b4 dc 41 00 50 d0 e7 97 d0 ae b8 f9 ba 80 18 ...A.P.. ........
0030 ff ff da 1f 00 00 01 01 08 0a 34 03 84 d8 b7 cc ........ ..4.....
0040 3f 04 47 45 54 20 2f 20 48 54 54 50 2f 31 2e 31 ?.GET / HTTP/1.1
0050 0d 0a 48 6f 73 74 3a 20 6d 79 64 6f 6d 61 69 6e ..Host: mydomain
0060 2e 63 6f 6d 0d 0a 55 73 65 72 2d 41 67 65 6e 74 .com..Us er-Agent
The rule that you have provided will never fire with the example packet that you have provided. You have used a content:"POST"; with a http_method modifier but you are attempting to match a packet that is a GET request.
I think that the right content modifier should be http_uri, not http_header. Unless you are trying to capture the Host POST parameter.
I am unable to unzip file in linux centos. Getting following error
End-of-central-directory signature not found. Either this file is not
a zipfile, or it constitutes one disk of a multi-part archive. In the
latter case the central directory and zipfile comment will be found on
the last disk(s) of this archive.
As you are mentioning jar in your comments we can consider this a programming question ;-)
First of all you should try to validate your file. If available you can even compare the checksum provided for this file and / or the filesize with the location you downloaded it from.
To verify the zip file on a low level you can use this command:
hexdump -C -n 100 file.zip
This will show you the first 100 bytes of the zips structure which will look similar to this:
00000000 50 4b 03 04 0a 00 00 00 00 00 88 43 65 47 11 7a |PK.........CeG.z|
00000010 39 1e 15 00 00 00 15 00 00 00 0e 00 1c 00 66 69 |9.............fi|
00000020 6c 65 31 69 6e 7a 69 70 2e 74 78 74 55 54 09 00 |le1inzip.txtUT..|
00000030 03 0f 05 3b 56 2f 05 3b 56 75 78 0b 00 01 04 e8 |...;V/.;Vux.....|
00000040 03 00 00 04 e8 03 00 00 54 68 69 73 20 69 73 20 |........This is |
00000050 61 20 66 69 6c 65 0a 1b 5b 31 37 7e 0a 50 4b 03 |a file..[17~.PK.|
00000060 04 0a 00 00 |....|
The first two byte of the file have to be PK, if not the file is invalid. Some bytes later you will find the name of the first file stored. In this example it is file1inzip.txt.
I'm developing a BitTorrent client and I'm having trouble getting answers to my piece requests.
To debug, I followed a conversation between uTorrent and transmission using Wireshark and tried to imitate same conversation in my client. But it still doesn't work.
Below is an example conversation happening between my client and transmission. (my client also using -TR--- prefixed peer id, this is only for testing purposes and I'll change this)
Indented messages are coming from transmission, others are messages my client send.
Note that this conversation is not exactly same as how uTorrent and transmission would talk, because my client does not support fast extension yet. (BEP 6)
(Output is taken from Wireshark, lines starting with -- are my comments)
00000000 13 42 69 74 54 6f 72 72 65 6e 74 20 70 72 6f 74 .BitTorr ent prot
00000010 6f 63 6f 6c 00 00 00 00 00 10 00 00 f8 9e 0d fd ocol.... ........
00000020 9c fc a8 52 d9 7a d6 af a4 4d 8f 73 ce 70 b6 36 ...R.z.. .M.s.p.6
00000030 2d 54 52 32 38 34 30 2d 36 68 61 67 76 30 73 70 -TR2840- 6hagv0sp
00000040 34 67 37 6b 4g7k
-- ^ my handshake to transmission
00000000 13 42 69 74 54 6f 72 72 65 6e 74 20 70 72 6f 74 .BitTorr ent prot
00000010 6f 63 6f 6c 00 00 00 00 00 10 00 04 f8 9e 0d fd ocol.... ........
00000020 9c fc a8 52 d9 7a d6 af a4 4d 8f 73 ce 70 b6 36 ...R.z.. .M.s.p.6
00000030 2d 54 52 32 38 34 30 2d 72 73 35 68 71 67 32 68 -TR2840- rs5hqg2h
00000040 6e 70 68 64 nphd
-- ^ transmission answers to my handshake
00000044 00 00 00 1a 14 00 64 31 3a 6d 64 31 31 3a 75 74 ......d1 :md11:ut
00000054 5f 6d 65 74 61 64 61 74 61 69 33 65 65 65 _metadat ai3eee
-- ^ my extended handshake to transmission
00000044 00 00 00 72 14 00 64 31 3a 65 69 31 65 31 3a 6d ...r..d1 :ei1e1:m
00000054 64 31 31 3a 75 74 5f 6d 65 74 61 64 61 74 61 69 d11:ut_m etadatai
00000064 33 65 65 31 33 3a 6d 65 74 61 64 61 74 61 5f 73 3ee13:me tadata_s
00000074 69 7a 65 69 31 34 37 65 31 3a 70 69 35 31 34 31 izei147e 1:pi5141
00000084 33 65 34 3a 72 65 71 71 69 35 31 32 65 31 31 3a 3e4:reqq i512e11:
00000094 75 70 6c 6f 61 64 5f 6f 6e 6c 79 69 31 65 31 3a upload_o nlyi1e1:
000000A4 76 31 37 3a 54 72 61 6e 73 6d 69 73 73 69 6f 6e v17:Tran smission
000000B4 20 32 2e 38 34 65 00 00 00 02 05 80 2.84e.. ....
-- ^ transmission's extended handshake and bitfield
000000C0 00 00 00 01 01 .....
-- ^ transmission unchokes me
00000062 00 00 00 01 02 .....
-- ^ my interested message
00000067 00 00 00 0d 06 00 00 00 00 00 00 00 00 00 00 40 ........ .......#
00000077 00 .
-- ^ piece request
-- no answers ...
00000078 00 00 00 0d 06 00 00 00 00 00 00 00 00 00 00 40 ........ .......#
00000088 00 .
-- ^ piece request again, with 10 seconds interval
-- again no answers...
00000089 00 00 00 0d 06 00 00 00 00 00 00 00 00 00 00 40 ........ .......#
00000099 00 .
-- ^ piece request again, with 10 seconds interval
-- no answers...
Any ideas what am I doing wrong?
Thanks.
EDIT: I updated my client to send unchoke just after sending interested, but I'm still having same problem...
The problem was that I was requesting a piece bigger than the total size of the torrent.
The torrent I was using has 2 files, in total of 12KB. However the piece size of the torrent is 16KB and I was requesting 16KB piece even though the torrent file has only one piece and it's 12 KB in total.
After requesting 12KB instead of 16KB, the problem was solved.