I'm attempting to copy/paste ASCII characters from a Hex editor into a Sublime Text 3 Plain Text document, although NUL characters do not show/display and the string is truncated:
Hexadecimal:
48 65 6C 6C 6F 2C 20 57 6F 72 6C 64 21 00 66 6F
6F 62 61 72 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00
ASCII:
Hello, World!�foobar�������������������������
Sublime Text: Truncates copied string and does not show NUL characters
TextMate: Shows NUL via "Show Invisibles"
I've tried the suggestion mentioned here by adding "draw_white_space": "all" to my preferences — still no luck! Is this possible with Sublime Text 3?
You're not alone in having this problem - others have posted bug reports about this behaviour: https://github.com/SublimeTextIssues/Core/issues/393
However it's not consistent:
Behaviour seems dependent on the file and where the NUL chars exist;
Similar issue here, with the console: https://github.com/SublimeTextIssues/Core/issues/1939
Related
I'm trying to store a Buffer into a bytea type column. I'm using a Postgres database and I have successfully connected to this database with node-postgres. I am able to update any other field, but I just can't find out what the syntax is to properly store a Buffer.
At the moment, there are already images in that database, that were written with a different system and language. I am not able to to re-use this system to achieve what we need.
The output of those existing images is also a Buffer:
<Buffer 89 50 4e 47 0d 0a 1a 0a 00 00 00 0d 49 48 44 52 00 00 04 38 00 00 04 38 08 06 00 00 00 ec 10 6c 8f 00 00 00 04 73 42 49 54 08 08 08 08 7c 08 64 88 00 ... 13315 more bytes>
And I have prepared the an image that should overwrite this value:
<Buffer 75 ab 5a 8a 66 a0 7b fa 67 81 b6 ac 7b ae 22 54 13 91 c3 42 86 82 80 00 00 03 52 52 11 14 80 00 00 2a 00 00 00 2a 02 00 00 00 00 14 48 3e 9a 00 00 00 ... 3153 more bytes>.
All good, so far.
I now need to use the proper SQL UPDATE statement, but I have not been able to figure that out. I have found some answers suggesting converting it using .toString('hex') and prepending it with \\x, but this does not result in the same value format.
My update statement now looks something like this (where imageData is the second Buffer example above):
await pool.query(
`UPDATE image
SET data = '${imageData}'::bytea
WHERE id = '00413567-fdd7-4765-be30-7f80c2d8ce57'`
)
Some requirements:
I can not use an external file
I can not use a different value format
I can not use a different tech stack
Background is that I have a log file that contains hex dumps that I want to convert with xxd to get that nice ASCII column that shows possible strings in the binary data.
The log file format looks like this:
My interesting hex dump:
00 53 00 6f 00 6d 00 65 00 20 00 74 00 65 00 78
00 74 00 20 00 65 00 78 00 61 00 6d 00 70 00 6c
00 65 00 20 00 75 00 73 00 69 00 6e 00 67 00 20
00 55 00 54 00 46 00 2d 00 31 00 36 00 20 00 69
00 6e 00 20 00 6f 00 72 00 64 00 65 00 72 00 20
00 74 00 6f 00 20 00 67 00 65 00 74 00 20 00 30
00 78 00 30 00 30 00 20 00 62 00 79 00 74 00 65
00 73 00 2e
Visually selecting the hex dump and do xxd -r -p followed by a xxd -g1 on the result does exactly what I'm aiming for.
However, since the number of dumps I want to convert are quite a few I would rather automate the process.
So I'm using the following substitute command to do the conversion:
:%s/\(\x\{2\} \?\)\{16\}\_.*/\=system('xxd -g1',system('xxd -r -p',submatch(0)))
The expression matches the entire hex dump in the log file. The match is sent to xxd -r -p as stdin and its output is used as stdin for xxd -g1.
Well, that's the idea at least.
The thing is that the above almost works. It produces the following result:
My interesting hex dump:
00000000: 01 53 01 6f 01 6d 01 65 01 20 01 74 01 65 01 78 .S.o.m.e. .t.e.x
00000010: 01 74 01 20 01 65 01 78 01 61 01 6d 01 70 01 6c .t. .e.x.a.m.p.l
00000020: 01 65 01 20 01 75 01 73 01 69 01 6e 01 67 01 20 .e. .u.s.i.n.g.
00000030: 01 55 01 54 01 46 01 2d 01 31 01 36 01 20 01 69 .U.T.F.-.1.6. .i
00000040: 01 6e 01 20 01 6f 01 72 01 64 01 65 01 72 01 20 .n. .o.r.d.e.r.
00000050: 01 74 01 6f 01 20 01 67 01 65 01 74 01 20 01 30 .t.o. .g.e.t. .0
00000060: 01 78 01 30 01 30 01 20 01 62 01 79 01 74 01 65 .x.0.0. .b.y.t.e
00000070: 01 73 01 2e .s..
All 00 bytes have mysteriously transformed into 01.
It should have produced the following:
My interesting hex dump:
00000000: 00 53 00 6f 00 6d 00 65 00 20 00 74 00 65 00 78 .S.o.m.e. .t.e.x
00000010: 00 74 00 20 00 65 00 78 00 61 00 6d 00 70 00 6c .t. .e.x.a.m.p.l
00000020: 00 65 00 20 00 75 00 73 00 69 00 6e 00 67 00 20 .e. .u.s.i.n.g.
00000030: 00 55 00 54 00 46 00 2d 00 31 00 36 00 20 00 69 .U.T.F.-.1.6. .i
00000040: 00 6e 00 20 00 6f 00 72 00 64 00 65 00 72 00 20 .n. .o.r.d.e.r.
00000050: 00 74 00 6f 00 20 00 67 00 65 00 74 00 20 00 30 .t.o. .g.e.t. .0
00000060: 00 78 00 30 00 30 00 20 00 62 00 79 00 74 00 65 .x.0.0. .b.y.t.e
00000070: 00 73 00 2e .s..
What am I not getting here?
Of course I can use macros and other ways of doing this, but I want to understand why my substitution command doesn't do what I expect.
Edit:
For anyone that want to achieve the same thing I provide the substitution expression that works on an entire file. The expression above was only for testing purposes using the log file example also from above.
The one below is the one that performs a correct conversion, modified based on the information Kent provided in his answer.
:%s/\(\(\x\{2\} \)\{16\}\_.\)\+/\=system('xxd -p -r | xxd -g1',submatch(0))
very likely, the problem is string conversion in the system() The input will be converted into a string by vim, so does the output of your first xxd command.
You can try to extract that hex parts into a file. then:
xxd -r -p theFile|vim -
And then calling the system('xxd -g1', alltext), you are gonna get something else than 00 too.
This doesn't work in the same way of a pipe (xxd ...|xxd...). But unfortunately, the system() function doesn't accept pipes.
If you want to fix your :s command, you need to call systemlist() on your first xxd call to get the data in binary format, then pass it to the 2nd xxd:
:%s/\(\x\{2\} \?\)\{16\}\_.*/\=system('xxd -g1',systemlist('xxd -r -p',submatch(0)))
The cmd above will generate the 00s. since there is no string conversion.
However, when working with some data format other than plain string, perhaps we can use filters instead of calling system(). It would be a lot eaiser. For your example:
2,$!xxd -r -p|xxd -g1
I copied a piece of text from a website. This piece of text contains a space. I later try to manipulate this string in C#, but my code doesn't recognize the space.
I started digging deeper, so I tried the following Powershell command to convert the string to hexadecimal to see what's going on:
"2+1 53" | Format-Hex
(see screenshot here: Powershell code)
As you can see in the image, it shows that the result is:
32 2B 31 3F 35 33
which converted back to normal text is
2+1?53
Notice that the question mark wasn't present in my original string. What is going on? How can a question mark be present but not show up? Or where did it come from if it was not present in my original string?
Update:
Perhaps I should stress that I need to figure out what that "space" character is, so that I can later get rid of it using "replace" method.
Most likely, there's another character in that text, which is not a space. You can check it by putting the text into a file and then using
Get-Content C:\temp\file.txt | Format-Hex
To reproduce, I used that text:
Get-Service –Name BITS
# ^ it's not a normal dash, check it at http://asciivalue.com/index.php
This happens if I paste into console window:
31.88 ms | C:\> "Get-Service -Name BITS" | Format-Hex
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 47 65 74 2D 53 65 72 76 69 63 65 20 3F 4E 61 6D Get-Service ?Nam
00000010 65 20 42 49 54 53 e BITS
And that when I get it from the script:
60.02 ms | C:\> Get-Content C:\temp\script.ps1 | Format-Hex
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 47 65 74 2D 53 65 72 76 69 63 65 20 3F 3F 3F 4E Get-Service ???N
00000010 61 6D 65 20 42 49 54 53 ame BITS
As you can see, that character is being converted to question mark (3F in hex output) or triple question mark (3F 3F 3F) while getting content from file.
It depends on the version of PowerShell. I have 5.1, there the default encoding of Format-Hex is ASCII, which will replace every non-ASCII character (like your space) with a question mark.
Specify a different encoding to prevent non-ASCII characters from being replaced. Example:
PS> "⇆" | Format-Hex -Encoding Unicode
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 C6 21 Æ!
Here, the code point is U+21C6. Google that and you will find out what it represents.
A regular space is 0x20. There are many unicode spaces. http://jkorpela.fi/chars/spaces.html How did you make the string? Here's an example of EN SPACE (nut), U+2002. You should be able to copy and paste this yourself. Hmm, in powershell 7 for windows, the special space doesn't paste.
[int[]][char[]]'foo bar' | % tostring x
66
6f
6f
2002
62
61
72
'foo bar' | Format-Hex -Encoding BigEndianUnicode
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
00000000 00 66 00 6F 00 6F 20 02 00 62 00 61 00 72 .f.o.o ..b.a.r
Format-hex will translate it to ascii by default in powershell 5. Unicode characters will be replaced by 3F or ?.
'foo bar' | format-hex -Encoding ascii
Label: String (System.String) <72F012A4>
Offset Bytes Ascii
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
------ ----------------------------------------------- -----
0000000000000 66 6F 6F 3F 62 61 72 foo?bar
UTF8 will encode in one to three bytes depending on how high the unicode number is (how many bits to encode). Three bytes in this case (U+2002), always starting with 'E' and then the first number, in this case '2'.
' ' | format-hex -Encoding utf8
# "`u{2002}" | format-hex # powershell 7
Label: String (System.String) <03ACFE1C>
Offset Bytes Ascii
00 01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F
------ ----------------------------------------------- -----
0000000000000 E2 80 82 �
I found the answer to my question here: String Comparison, .NET and non breaking space
The "space" that was present in the string was the "non-breaking space" and getting rid of it in C# was as easy as:
string cellText = "String with non breaking spaces.";
cellText = Regex.Replace(cellText, #"\u00A0", " ");
I am trying to recover some BMP image header, and the all information I got is this:
the bytes of the header that have been deleted are 54
the start offset is 1130 (decimal)
the width is 350 (decimal).
bits per pixel is 8
no compression
So I read little bit (mostly from here) and I saw which bytes present what, and I got the next first 54 bytes of the BMP header:
42 4D 6E 7F 0C 00 00 00 00 00 36 00 00 00 28 00 00 00 F4 01 00 00 22 02 00 00 01 00 18 00 00 00 00 00 38 7F 0C 00 13 0B 00 00 13 0B 00 00 00 00 00 00 00 00 00 00
But for some reason, when I am trying to open my all BMP images and my header, I get just a black image. Did I do something wrong? Do I build the
wrong header from the information above?
I have been trying to find any information on a value that is stored in an url shortcut that is generated from MS-Office for SharePoint Libraries. We are moving away from Office 2007 to Office 365 and the feature of 2007 that creates these links doesn't appear to be available in Office 365.
If a .url file doesn't have the IDList value when you click on it in a save-as dialog it changes the file name to the shortcut. I tried changing the .url to .lnk but non-office applications can't handle them.
Contents of .url file
[InternetShortcut]
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA8gPuuBAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAAAAAAQAQZk12cAAgNAgAAEAw7+CAAAAAAAAAAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
URL=file://blart/DavWWWRoot/teams/edms
IconFile=C:\ProgramData\Microsoft\Office\SharePointTeamSite.ico
IconIndex=0
[{000214A0-0000-0000-C000-000000000046}]
Prop3=19,9
Same user regenerated file
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA0baRNAAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAAAAAAQAQZk12cAAgNAgAAEAw7+CAAAAAAAAAAqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
Value from a file from another user
IDList=UAwHYBGLNCi66kGEieNCAsCMw0pTAMcAlzFXj9mZr5yalx2b35WYuMWYcRUY2d1VXJ1bvRHAXVmYgMEbpVmb0BiTlR3dvJ3aAgGd0BnOv8yYvZ2ausWZs92duFmLjFGAuAATAEDAAAAAAAAAAAAEAQXZh12cAgDAIAABA8uvAAAAAAAAAAgKAAAAAAAAAAAAAAAAAAAA4BZpCAAAAAAAAQHAlBQYA0GAzBAAAQBAKBQMAAAAAAAdHBlfQAQZk12cAAgNAgAAEAw7+qXQJdJdHBlfqAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAQZAQGAtBwcAAAAUAAAAA
I have not been able to figure out how to decode this value or have I been able to figure out what data is being stored in it. I have read through a bunch of different ways to create shortcuts programmatically with Windows scripting and shell 32 in Powershell and C# wrappers, C++ looking for any info. The only glimmer of hope I have is that shell links also have an item id list of pidl's, but I can't confirm if this is related.
What I do know is that the IDList can be moved to another .url and it will take you to the original path. There are some sections of the IDList that change depending on the user who created them and or when the url files are recreated.
Internet shortcuts
https://msdn.microsoft.com/en-us/library/bb776784(v=vs.85).aspx
Shell link
https://msdn.microsoft.com/en-us/library/dd891312.aspx
The .URL file format is not officially documented and only the first 2 bytes of the IDList format is documented.
I believe the IDList string used to be stored by calling WritePrivateProfileStruct and at some point changed to the format you posted. A function called TextToBinary decodes the string into a IStream instance but I did not bother trying to figure out the format. The stream is stuffed into a PROPVARIANT and the documented IUniformResourceLocator interface knows how to deal with all of this for you but unfortunately the idlist entry is not documented as a property you can read/write.
The PIDL (IDList) after .INI string decoding and some basic formating looks like this:
:1: 20 bytes
{208D2C60-3AEA-1069-A2D7-08002B30309D}:Network
:2: 78 bytes
4E 00 C3 01 E5 5C 5C 63 6F 66 6B 2E 6B 65 6C 6F N....\\c ofk.kelo
77 6E 61 2E 63 61 5C 44 61 76 57 57 57 52 6F 6F wna.ca\D avWWWRoo
74 00 57 65 62 20 43 6C 69 65 6E 74 20 4E 65 74 t.Web Cl ient Net
77 6F 72 6B 00 68 74 74 70 3A 2F 2F 63 6F 66 6B work.htt p://cofk
2E 6B 65 6C 6F 77 6E 61 2E 63 61 00 2E 00 __ __ .kelowna .ca...
:3: 76 bytes
4C 00 31 00 00 00 00 00 00 00 00 00 10 00 74 65 L.1..... ......te
61 6D 73 00 38 00 08 00 04 00 EF BE 00 00 00 00 ams.8... ........
00 00 00 00 2A 00 00 00 00 00 00 00 00 00 00 00 ....*... ........
00 00 00 00 3C F8 B8 6E 00 00 00 00 00 00 74 00 ....<..n ......t.
65 00 61 00 6D 00 73 00 00 00 14 00 __ __ __ __ e.a.m.s. ....
:4: 74 bytes
4A 00 31 00 00 00 00 00 00 00 00 00 10 00 65 64 J.1..... ......ed
6D 73 00 00 36 00 08 00 04 00 EF BE 00 00 00 00 ms..6... ........
00 00 00 00 2A 00 00 00 00 00 00 00 00 00 00 00 ....*... ........
00 00 00 00 00 00 00 00 00 00 00 00 00 00 65 00 ........ ......e.
64 00 6D 00 73 00 00 00 14 00 __ __ __ __ __ __ d.m.s... ..
:5: 0 bytes
and the result after asking the shell to convert this PIDL is \\cofk.kelowna.ca\DavWWWRoot\teams\edms.
The difference between the 3 examples you posted is just in the path part of the PIDL (teams\edms), probably the modified date of the folder or uninitialized data.