I would like to know to recover deleted file from FAT. I created fat.img as below.
cd /tmp
dd if=/dev/zero of=fat.img bs=1024 count=100
mkfs.msdos fat.img
mkdir -p /tmp/fs
sudo mount -t msdos fat.img /tmp/fs -o umask=000,loop
Now i am creating file with some text.
cd/tmp/fs
echo "hello world"> name
Using hexdump to see how it was saved
cd ..
hexdump -C fat.img
00000000 eb 3c 90 6d 6b 66 73 2e 66 61 74 00 02 04 01 00 |.<.mkfs.fat.....|
00000010 02 00 02 c8 00 f8 01 00 20 00 40 00 00 00 00 00 |........ .#.....|
00000020 00 00 00 00 80 01 29 3c 69 e6 fb 4e 4f 20 4e 41 |......)<i..NO NA|
00000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 0e 1f |ME FAT12 ..|
00000040 be 5b 7c ac 22 c0 74 0b 56 b4 0e bb 07 00 cd 10 |.[|.".t.V.......|
00000050 5e eb f0 32 e4 cd 16 cd 19 eb fe 54 68 69 73 20 |^..2.......This |
00000060 69 73 20 6e 6f 74 20 61 20 62 6f 6f 74 61 62 6c |is not a bootabl|
00000070 65 20 64 69 73 6b 2e 20 20 50 6c 65 61 73 65 20 |e disk. Please |
00000080 69 6e 73 65 72 74 20 61 20 62 6f 6f 74 61 62 6c |insert a bootabl|
00000090 65 20 66 6c 6f 70 70 79 20 61 6e 64 0d 0a 70 72 |e floppy and..pr|
000000a0 65 73 73 20 61 6e 79 20 6b 65 79 20 74 6f 20 74 |ess any key to t|
000000b0 72 79 20 61 67 61 69 6e 20 2e 2e 2e 20 0d 0a 00 |ry again ... ...|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 f8 ff ff 00 f0 ff 00 00 00 00 00 00 00 00 00 00 |................|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000400 f8 ff ff 00 f0 ff 00 00 00 00 00 00 00 00 00 00 |................|
00000410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000600 4e 41 4d 45 20 20 20 20 20 20 20 20 00 00 00 00 |NAME ....|
00000610 00 00 00 00 00 00 21 86 91 4b 03 00 0c 00 00 00 |......!..K......|
00000620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00004e00 68 65 6c 6c 6f 20 77 6f 72 6c 64 0a 00 00 00 00 |hello world.....|
00004e10 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00019000
After deleting file name we can see change in hexdump
00000600 4e 41 4d 45 20 20 20 20 20 20 20 20 00 00 00 00 |.AME ....|
00000610 00 00 00 00 00 00 21 86 91 4b 03 00 0c 00 00 00 |......!..K......|
00000620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
And here is my question do you have any suggestion how can i change fat.img to .AME to NAME to recovery my file?
how can i change fat.img to .AME to NAME to recovery my file?
The short answer is with dd, an example with the necessary caveats follows below.
Adding to Martin's answer, while manipulating the bytes to restore the file within the floppy image is a relatively straight-forward proposition with dd, computing where and what within the file allocation table(s) needs to be restored is the challenge. Walking through the use of dd to restore the file itself, knowing what bytes need attention is illustrated by the following example.
Creating a floppy image to work with saves you from having to experiment on your actual image. Simply duplicate your image you wish to work with, or create a new one within a file on your hard drive. You can do that easily with mkfs.msdos (adjust the filesystem type as needed), and then mount the file within your filesystem as follows, e.g.
$ mkfs.msdos -C /home/david/tmp/tt/floppy_144.img 1440
$ sudo mount /home/david/tmp/tt/floppy_144.img /mnt/fd
Now let's add the NAME file:
$ echo "hello world" > NAME
$ sudo cp -a NAME /mnt/fd
$ ls -l /mnt/fd
total 1
-rwxr-xr-x 1 root root 12 Dec 17 13:55 NAME
$ cat /mnt/fd/NAME
hello world
Before deleting the file from your image, hexdump the contents so you can see exactly what needs to be restored. (this is what you must compute in order to know where and what to restore with your original image, you will need to consult a reference for the precise filesystem at issue)
$ hexdump -C floppy_144.img >flpwname.txt
Now delete the file from your image and again save a hexdump showing the changes.
$ sudo rm /mnt/fd/NAME
$ hexdump -C floppy_144.img >flpwoname.txt
Now you can examing the difference with diff. What you find is you must restore more than the first name of the file that was deleted, you will need to restore the file allocation table entries so that the restored file can again be located within the filesystem (both copies of the FAT), e.g.
$ diff flpwname.txt flpwoname.txt
16c16
< 00000200 f0 ff ff 00 f0 ff 00 00 00 00 00 00 00 00 00 00 |................|
---
> 00000200 f0 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
19c19
< 00001400 f0 ff ff 00 f0 ff 00 00 00 00 00 00 00 00 00 00 |................|
---
> 00001400 f0 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
22c22
< 00002600 4e 41 4d 45 20 20 20 20 20 20 20 20 00 00 fa 9e |NAME ....|
---
> 00002600 e5 41 4d 45 20 20 20 20 20 20 20 20 00 00 fa 9e |.AME ....|
Note above the entries for the file allocation table(s) at 0x204 and 0x1404 were zeroed when the file was removed. Restoring the bytes to the original can be easily done with dd but pay attention to your options. Specifically your block size (bs), output block size (obs), count and seek must all be in bytes (specified by appending c) to the number and you must set the notrunc conversion option to prevent truncating your image following the changes you make. Lastly, all sizes must be specified in decimal not hexadecimal.
Further, if you are using bash, you can use a process redirection to specify the bytes to replace (e.g. if=<(printf "\xf0\xff") to write the hex bytes f0 and ff), otherwise, you will have to prepare input files containing your replacement strings. The dd commands to restore the FAT and the first character of the filename are fairly simple (consult man 1 dd for option explanation).
Below we restore the first copy of the FAT, then the second, and finally restore the first character of the filename. The seek (offset) values are just those provided by hexdump converted to decimal. (you should unmount your filesystem before making changes. you can make changes while your floppy image is mounted, but they won't be reflected until you remount)
$ sudo umount /mnt/fd
$ dd if=<(printf "\xf0\xff") of=floppy_144.img \
bs=1c obs=1c count=2c seek=516c conv=notrunc
$ dd if=<(printf "\xf0\xff") of=floppy_144.img \
bs=1c obs=1c count=2c seek=5124c conv=notrunc
$ dd if=<(printf "N") of=floppy_144.img \
bs=1c obs=1c count=1c seek=9728c conv=notrunc
Now you can create a hexdump of the repaired floppy image and compare that to the original. If all has gone as it should, there will be no difference.
$ hexdump -C floppy_144.img >flprepair.txt
$ diff flpwname.txt flprepair.txt
Finally, just remount your filesystem and confirm the file has been restored.
$ sudo mount /home/david/tmp/tt/floppy_144.img /mnt/fd
$ ls -l /mnt/fd
total 1
-rwxr-xr-x 1 root root 12 Dec 17 13:55 NAME
$ cat /mnt/fd/NAME
hello world
That's it. I hope this is what you were looking for. There are a number of tools that automate this process for you, but dd and a pencil and paper can get you by.
The full hexdumps follow for completeness:
Original/Restored
$ cat flpwname.txt
00000000 eb 3c 90 6d 6b 66 73 2e 66 61 74 00 02 01 01 00 |.<.mkfs.fat.....|
00000010 02 e0 00 40 0b f0 09 00 12 00 02 00 00 00 00 00 |...#............|
00000020 00 00 00 00 00 01 29 2c 72 18 ba 4e 4f 20 4e 41 |......),r..NO NA|
00000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 0e 1f |ME FAT12 ..|
00000040 be 5b 7c ac 22 c0 74 0b 56 b4 0e bb 07 00 cd 10 |.[|.".t.V.......|
00000050 5e eb f0 32 e4 cd 16 cd 19 eb fe 54 68 69 73 20 |^..2.......This |
00000060 69 73 20 6e 6f 74 20 61 20 62 6f 6f 74 61 62 6c |is not a bootabl|
00000070 65 20 64 69 73 6b 2e 20 20 50 6c 65 61 73 65 20 |e disk. Please |
00000080 69 6e 73 65 72 74 20 61 20 62 6f 6f 74 61 62 6c |insert a bootabl|
00000090 65 20 66 6c 6f 70 70 79 20 61 6e 64 0d 0a 70 72 |e floppy and..pr|
000000a0 65 73 73 20 61 6e 79 20 6b 65 79 20 74 6f 20 74 |ess any key to t|
000000b0 72 79 20 61 67 61 69 6e 20 2e 2e 2e 20 0d 0a 00 |ry again ... ...|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 f0 ff ff 00 f0 ff 00 00 00 00 00 00 00 00 00 00 |................|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001400 f0 ff ff 00 f0 ff 00 00 00 00 00 00 00 00 00 00 |................|
00001410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002600 4e 41 4d 45 20 20 20 20 20 20 20 20 00 00 fa 9e |NAME ....|
00002610 91 4b 91 4b 00 00 f5 9e 91 4b 03 00 0c 00 00 00 |.K.K.....K......|
00002620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00004400 68 65 6c 6c 6f 20 77 6f 72 6c 64 0a 00 00 00 00 |hello world.....|
00004410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00168000
After NAME Deletion
$ cat flpwoname.txt
00000000 eb 3c 90 6d 6b 66 73 2e 66 61 74 00 02 01 01 00 |.<.mkfs.fat.....|
00000010 02 e0 00 40 0b f0 09 00 12 00 02 00 00 00 00 00 |...#............|
00000020 00 00 00 00 00 01 29 2c 72 18 ba 4e 4f 20 4e 41 |......),r..NO NA|
00000030 4d 45 20 20 20 20 46 41 54 31 32 20 20 20 0e 1f |ME FAT12 ..|
00000040 be 5b 7c ac 22 c0 74 0b 56 b4 0e bb 07 00 cd 10 |.[|.".t.V.......|
00000050 5e eb f0 32 e4 cd 16 cd 19 eb fe 54 68 69 73 20 |^..2.......This |
00000060 69 73 20 6e 6f 74 20 61 20 62 6f 6f 74 61 62 6c |is not a bootabl|
00000070 65 20 64 69 73 6b 2e 20 20 50 6c 65 61 73 65 20 |e disk. Please |
00000080 69 6e 73 65 72 74 20 61 20 62 6f 6f 74 61 62 6c |insert a bootabl|
00000090 65 20 66 6c 6f 70 70 79 20 61 6e 64 0d 0a 70 72 |e floppy and..pr|
000000a0 65 73 73 20 61 6e 79 20 6b 65 79 20 74 6f 20 74 |ess any key to t|
000000b0 72 79 20 61 67 61 69 6e 20 2e 2e 2e 20 0d 0a 00 |ry again ... ...|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
000001f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa |..............U.|
00000200 f0 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000210 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00001400 f0 ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00001410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00002600 e5 41 4d 45 20 20 20 20 20 20 20 20 00 00 fa 9e |.AME ....|
00002610 91 4b 91 4b 00 00 f5 9e 91 4b 03 00 0c 00 00 00 |.K.K.....K......|
00002620 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00004400 68 65 6c 6c 6f 20 77 6f 72 6c 64 0a 00 00 00 00 |hello world.....|
00004410 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00168000
Related
Suppose I create a simple PNG with:
convert -size 1x1 canvas:red red.png
Here is a similar image (bigger size) for reference:
Then run the command identify on it. It tells me the ColorSpace of the image is sRGB but there seems to be NO indication of this inside the file. In fact running
$ hexdump -C red.png
00000000 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 |.PNG........IHDR|
00000010 00 00 00 01 00 00 00 01 01 03 00 00 00 25 db 56 |.............%.V|
00000020 ca 00 00 00 04 67 41 4d 41 00 00 b1 8f 0b fc 61 |.....gAMA......a|
00000030 05 00 00 00 20 63 48 52 4d 00 00 7a 26 00 00 80 |.... cHRM..z&...|
00000040 84 00 00 fa 00 00 00 80 e8 00 00 75 30 00 00 ea |...........u0...|
00000050 60 00 00 3a 98 00 00 17 70 9c ba 51 3c 00 00 00 |`..:....p..Q<...|
00000060 06 50 4c 54 45 ff 00 00 ff ff ff 41 1d 34 11 00 |.PLTE......A.4..|
00000070 00 00 01 62 4b 47 44 01 ff 02 2d de 00 00 00 07 |...bKGD...-.....|
00000080 74 49 4d 45 07 e5 01 0d 17 04 37 80 ef 04 02 00 |tIME......7.....|
00000090 00 00 0a 49 44 41 54 08 d7 63 60 00 00 00 02 00 |...IDAT..c`.....|
000000a0 01 e2 21 bc 33 00 00 00 25 74 45 58 74 64 61 74 |..!.3...%tEXtdat|
000000b0 65 3a 63 72 65 61 74 65 00 32 30 32 31 2d 30 31 |e:create.2021-01|
000000c0 2d 31 33 54 32 33 3a 30 34 3a 35 35 2b 30 30 3a |-13T23:04:55+00:|
000000d0 30 30 2d af d4 01 00 00 00 25 74 45 58 74 64 61 |00-......%tEXtda|
000000e0 74 65 3a 6d 6f 64 69 66 79 00 32 30 32 31 2d 30 |te:modify.2021-0|
000000f0 31 2d 31 33 54 32 33 3a 30 34 3a 35 35 2b 30 30 |1-13T23:04:55+00|
00000100 3a 30 30 5c f2 6c bd 00 00 00 00 49 45 4e 44 ae |:00\.l.....IEND.|
00000110 42 60 82 |B`.|
00000113
does not provide a clue, that I know of.
I understand that identifying the ColorSpace of an image, that does not contain that information, is a very hard problem -- see one proposed solution looking at the histogram of colors here.
So how identify, from the ImageMagick suite, determines the ColorSpace of this image?
It is common, but not standardized to assume that an image without an embedded or sidecar ICC profile or without an explicit encoding description is encoded according to IEC 61966-2-1:1999, i.e. sRGB specification.
This is just a bug in ImageMagick. You can use exiftool to check whether sRGB + intent chunk is present. In this case, no.
Gamma 2.2 is not sRGB. Thus ImageMagic is wrong here. That is a common problem on Wikipedia, all SVG images when converted to PNG have this and it destroys the colours. See: https://phabricator.wikimedia.org/T26768
We will have to reencode all images on Wikipedia, since we use ImageMagick. Sigh.
Background is that I have a log file that contains hex dumps that I want to convert with xxd to get that nice ASCII column that shows possible strings in the binary data.
The log file format looks like this:
My interesting hex dump:
00 53 00 6f 00 6d 00 65 00 20 00 74 00 65 00 78
00 74 00 20 00 65 00 78 00 61 00 6d 00 70 00 6c
00 65 00 20 00 75 00 73 00 69 00 6e 00 67 00 20
00 55 00 54 00 46 00 2d 00 31 00 36 00 20 00 69
00 6e 00 20 00 6f 00 72 00 64 00 65 00 72 00 20
00 74 00 6f 00 20 00 67 00 65 00 74 00 20 00 30
00 78 00 30 00 30 00 20 00 62 00 79 00 74 00 65
00 73 00 2e
Visually selecting the hex dump and do xxd -r -p followed by a xxd -g1 on the result does exactly what I'm aiming for.
However, since the number of dumps I want to convert are quite a few I would rather automate the process.
So I'm using the following substitute command to do the conversion:
:%s/\(\x\{2\} \?\)\{16\}\_.*/\=system('xxd -g1',system('xxd -r -p',submatch(0)))
The expression matches the entire hex dump in the log file. The match is sent to xxd -r -p as stdin and its output is used as stdin for xxd -g1.
Well, that's the idea at least.
The thing is that the above almost works. It produces the following result:
My interesting hex dump:
00000000: 01 53 01 6f 01 6d 01 65 01 20 01 74 01 65 01 78 .S.o.m.e. .t.e.x
00000010: 01 74 01 20 01 65 01 78 01 61 01 6d 01 70 01 6c .t. .e.x.a.m.p.l
00000020: 01 65 01 20 01 75 01 73 01 69 01 6e 01 67 01 20 .e. .u.s.i.n.g.
00000030: 01 55 01 54 01 46 01 2d 01 31 01 36 01 20 01 69 .U.T.F.-.1.6. .i
00000040: 01 6e 01 20 01 6f 01 72 01 64 01 65 01 72 01 20 .n. .o.r.d.e.r.
00000050: 01 74 01 6f 01 20 01 67 01 65 01 74 01 20 01 30 .t.o. .g.e.t. .0
00000060: 01 78 01 30 01 30 01 20 01 62 01 79 01 74 01 65 .x.0.0. .b.y.t.e
00000070: 01 73 01 2e .s..
All 00 bytes have mysteriously transformed into 01.
It should have produced the following:
My interesting hex dump:
00000000: 00 53 00 6f 00 6d 00 65 00 20 00 74 00 65 00 78 .S.o.m.e. .t.e.x
00000010: 00 74 00 20 00 65 00 78 00 61 00 6d 00 70 00 6c .t. .e.x.a.m.p.l
00000020: 00 65 00 20 00 75 00 73 00 69 00 6e 00 67 00 20 .e. .u.s.i.n.g.
00000030: 00 55 00 54 00 46 00 2d 00 31 00 36 00 20 00 69 .U.T.F.-.1.6. .i
00000040: 00 6e 00 20 00 6f 00 72 00 64 00 65 00 72 00 20 .n. .o.r.d.e.r.
00000050: 00 74 00 6f 00 20 00 67 00 65 00 74 00 20 00 30 .t.o. .g.e.t. .0
00000060: 00 78 00 30 00 30 00 20 00 62 00 79 00 74 00 65 .x.0.0. .b.y.t.e
00000070: 00 73 00 2e .s..
What am I not getting here?
Of course I can use macros and other ways of doing this, but I want to understand why my substitution command doesn't do what I expect.
Edit:
For anyone that want to achieve the same thing I provide the substitution expression that works on an entire file. The expression above was only for testing purposes using the log file example also from above.
The one below is the one that performs a correct conversion, modified based on the information Kent provided in his answer.
:%s/\(\(\x\{2\} \)\{16\}\_.\)\+/\=system('xxd -p -r | xxd -g1',submatch(0))
very likely, the problem is string conversion in the system() The input will be converted into a string by vim, so does the output of your first xxd command.
You can try to extract that hex parts into a file. then:
xxd -r -p theFile|vim -
And then calling the system('xxd -g1', alltext), you are gonna get something else than 00 too.
This doesn't work in the same way of a pipe (xxd ...|xxd...). But unfortunately, the system() function doesn't accept pipes.
If you want to fix your :s command, you need to call systemlist() on your first xxd call to get the data in binary format, then pass it to the 2nd xxd:
:%s/\(\x\{2\} \?\)\{16\}\_.*/\=system('xxd -g1',systemlist('xxd -r -p',submatch(0)))
The cmd above will generate the 00s. since there is no string conversion.
However, when working with some data format other than plain string, perhaps we can use filters instead of calling system(). It would be a lot eaiser. For your example:
2,$!xxd -r -p|xxd -g1
My question is how I get python to read a file where the text is in 16bit characters. The rest of the post describes the situation.
I have a text file which is a playlist export from iTunes.
Here is a short section including the header
Name Artist Composer Album Grouping Work Movement Number Movement Count Movement Name Genre Size Time Disc Number Disc Count Track Number Track Count Year Date Modified Date Added Bit Rate Sample Rate Volume Adjustment Kind Equalizer Comments Plays Last Played Skips Last Skipped My Rating
Keyboard Works of the Masters Randolph Hokanson Pan125b 2054816 64 03/11/2017, 18:00 03/11/2017, 17:01 256 44100 MPEG audio file 1 03/11/2017, 17:02 4 08/03/2018, 16:07
08 Traccia 08 11159905 464 03/11/2017, 17:39 03/11/2017, 16:59 192 48000 MPEG audio file 1 03/11/2017, 16:59
09 Traccia 09 17787361 741 03/11/2017, 17:39 03/11/2017, 16:58 192 48000 MPEG audio file 5 08/03/2018, 10:58
10 Traccia 10 10128290 421 03/11/2017, 17:39 03/11/2017, 16:58 192 48000 MPEG audio file 1 03/11/2017, 16:58
When I use this code to read it, the program hangs. (The i holds the number of lines in the file). The hex dumps which follow seem to show the export from iTunes is in 16bit characters.
The complete code for reading the text file is
file_name="full path to file goes here"
f = open(file_name, "r")
i=227
for x in range(0, i):
line = f.readline()
When I read the code into text wrangler, selected all the text, and pasted it into a new document. The code worked fine.
A text dump of part of the original file looks like this to start with the new file following
00000000: FF FE 4E 00 61 00 6D 00 65 00 09 00 41 00 72 00 ..N.a.m.e...A.r.
00000010: 74 00 69 00 73 00 74 00 09 00 43 00 6F 00 6D 00 t.i.s.t...C.o.m.
00000020: 70 00 6F 00 73 00 65 00 72 00 09 00 41 00 6C 00 p.o.s.e.r...A.l.
00000030: 62 00 75 00 6D 00 09 00 47 00 72 00 6F 00 75 00 b.u.m...G.r.o.u.
00000040: 70 00 69 00 6E 00 67 00 09 00 57 00 6F 00 72 00 p.i.n.g...W.o.r.
00000050: 6B 00 09 00 4D 00 6F 00 76 00 65 00 6D 00 65 00 k...M.o.v.e.m.e.
00000060: 6E 00 74 00 20 00 4E 00 75 00 6D 00 62 00 65 00 n.t. .N.u.m.b.e.
00000070: 72 00 09 00 4D 00 6F 00 76 00 65 00 6D 00 65 00 r...M.o.v.e.m.e.
00000080: 6E 00 74 00 20 00 43 00 6F 00 75 00 6E 00 74 00 n.t. .C.o.u.n.t.
00000090: 09 00 4D 00 6F 00 76 00 65 00 6D 00 65 00 6E 00 ..M.o.v.e.m.e.n.
000000A0: 74 00 20 00 4E 00 61 00 6D 00 65 00 09 00 47 00 t. .N.a.m.e...G.
000000B0: 65 00 6E 00 72 00 65 00 09 00 53 00 69 00 7A 00 e.n.r.e...S.i.z.
000000C0: 65 00 09 00 54 00 69 00 6D 00 65 00 09 00 44 00 e...T.i.m.e...D.
000000D0: 69 00 73 00 63 00 20 00 4E 00 75 00 6D 00 62 00 i.s.c. .N.u.m.b.
000000E0: 65 00 72 00 09 00 44 00 69 00 73 00 63 00 20 00 e.r...D.i.s.c. .
000000F0: 43 00 6F 00 75 00 6E 00 74 00 09 00 54 00 72 00 C.o.u.n.t...T.r.
New file
0000: 4E 61 6D 65 09 41 72 74 69 73 74 09 43 6F 6D 70 Name.Artist.Comp
0010: 6F 73 65 72 09 41 6C 62 75 6D 09 47 72 6F 75 70 oser.Album.Group
0020: 69 6E 67 09 57 6F 72 6B 09 4D 6F 76 65 6D 65 6E ing.Work.Movemen
0030: 74 20 4E 75 6D 62 65 72 09 4D 6F 76 65 6D 65 6E t Number.Movemen
0040: 74 20 43 6F 75 6E 74 09 4D 6F 76 65 6D 65 6E 74 t Count.Movement
0050: 20 4E 61 6D 65 09 47 65 6E 72 65 09 53 69 7A 65 Name.Genre.Size
Your file beginning looks like UTF-16 - see Byte order marks - Wikipedia
Use
file_name="full path to file goes here"
with io.open(file_name,'r', encoding='utf-16-le') as f:
for line in f:
# do something with line
when opening it.
There is no need to use range() or readlines() when reading line by line. If you really need the line-numbers use:
for lineNr,line in enumerate(f):
I have been trying to find any information on a value that is stored in an url shortcut that is generated from MS-Office for SharePoint Libraries. We are moving away from Office 2007 to Office 365 and the feature of 2007 that creates these links doesn't appear to be available in Office 365.
If a .url file doesn't have the IDList value when you click on it in a save-as dialog it changes the file name to the shortcut. I tried changing the .url to .lnk but non-office applications can't handle them.
Contents of .url file
[InternetShortcut]
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA8gPuuBAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAAAAAAQAQZk12cAAgNAgAAEAw7+CAAAAAAAAAAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
URL=file://blart/DavWWWRoot/teams/edms
IconFile=C:\ProgramData\Microsoft\Office\SharePointTeamSite.ico
IconIndex=0
[{000214A0-0000-0000-C000-000000000046}]
Prop3=19,9
Same user regenerated file
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA0baRNAAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAAAAAAQAQZk12cAAgNAgAAEAw7+CAAAAAAAAAAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
Value from a file from another user
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA4BZpCAAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAdHBlfQAQZk12cAAgNAgAAEAw7+qXQJdJdHBlfqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
I have not been able to figure out how to decode this value or have I been able to figure out what data is being stored in it. I have read through a bunch of different ways to create shortcuts programmatically with Windows scripting and shell 32 in Powershell and C# wrappers, C++ looking for any info. The only glimmer of hope I have is that shell links also have an item id list of pidl's, but I can't confirm if this is related.
What I do know is that the IDList can be moved to another .url and it will take you to the original path. There are some sections of the IDList that change depending on the user who created them and or when the url files are recreated.
Internet shortcuts
https://msdn.microsoft.com/en-us/library/bb776784(v=vs.85).aspx
Shell link
https://msdn.microsoft.com/en-us/library/dd891312.aspx
The .URL file format is not officially documented and only the first 2 bytes of the IDList format is documented.
I believe the IDList string used to be stored by calling WritePrivateProfileStruct and at some point changed to the format you posted. A function called TextToBinary decodes the string into a IStream instance but I did not bother trying to figure out the format. The stream is stuffed into a PROPVARIANT and the documented IUniformResourceLocator interface knows how to deal with all of this for you but unfortunately the idlist entry is not documented as a property you can read/write.
The PIDL (IDList) after .INI string decoding and some basic formating looks like this:
:1: 20 bytes
{208D2C60-3AEA-1069-A2D7-08002B30309D}:Network
:2: 78 bytes
4E 00 C3 01 E5 5C 5C 63 6F 66 6B 2E 6B 65 6C 6F N....\\c ofk.kelo
77 6E 61 2E 63 61 5C 44 61 76 57 57 57 52 6F 6F wna.ca\D avWWWRoo
74 00 57 65 62 20 43 6C 69 65 6E 74 20 4E 65 74 t.Web Cl ient Net
77 6F 72 6B 00 68 74 74 70 3A 2F 2F 63 6F 66 6B work.htt p://cofk
2E 6B 65 6C 6F 77 6E 61 2E 63 61 00 2E 00 __ __ .kelowna .ca...
:3: 76 bytes
4C 00 31 00 00 00 00 00 00 00 00 00 10 00 74 65 L.1..... ......te
61 6D 73 00 38 00 08 00 04 00 EF BE 00 00 00 00 ams.8... ........
00 00 00 00 2A 00 00 00 00 00 00 00 00 00 00 00 ....*... ........
00 00 00 00 3C F8 B8 6E 00 00 00 00 00 00 74 00 ....<..n ......t.
65 00 61 00 6D 00 73 00 00 00 14 00 __ __ __ __ e.a.m.s. ....
:4: 74 bytes
4A 00 31 00 00 00 00 00 00 00 00 00 10 00 65 64 J.1..... ......ed
6D 73 00 00 36 00 08 00 04 00 EF BE 00 00 00 00 ms..6... ........
00 00 00 00 2A 00 00 00 00 00 00 00 00 00 00 00 ....*... ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 65 00 ........ ......e.
64 00 6D 00 73 00 00 00 14 00 __ __ __ __ __ __ d.m.s... ..
:5: 0 bytes
and the result after asking the shell to convert this PIDL is \\cofk.kelowna.ca\DavWWWRoot\teams\edms.
The difference between the 3 examples you posted is just in the path part of the PIDL (teams\edms), probably the modified date of the folder or uninitialized data.
I'm having trouble with my home made "for fun" nameserver. It's been a couple of months since I updated it so I'm a bit rusty and thought I'd ask here and see if someone else sees what's wrong. I'm getting a FORMERR when asking for a TXT record, and the same problem occur on different domains, so there's probably something wrong in the packet formatting. Anyone?
dig txt ffffff.com #ns1.ffffff.com
;; Got bad packet: FORMERR
1024 bytes
ce bf 84 00 00 01 00 01 00 02 00 00 06 66 66 66 .............fff
66 66 66 03 63 6f 6d 00 00 10 00 01 c0 0c 00 10 fff.com.........
00 01 00 00 02 58 00 13 12 57 65 6c 63 6f 6d 65 .....X...Welcome
20 74 6f 20 66 66 66 66 66 66 66 00 c0 0c 00 02 .to.fffffff.....
00 01 00 00 02 58 00 10 03 6e 73 31 06 66 66 66 .....X...ns1.fff
66 66 66 03 63 6f 6d 00 c0 0c 00 02 00 01 00 00 fff.com.........
02 58 00 10 03 6e 73 32 06 66 66 66 66 66 66 03 .X...ns2.ffffff.
63 6f 6d 00 00 00 00 00 00 00 00 00 00 00 00 00 com.............
In the example above supplied, I added an incorrect 00 (null terminator) at the end of the TXT-string. After removing the null terminator from the TXT records, the txt records now work on my nameserver.