I'm trying to convert from a HEX to ASCII and I'm getting this message. I would like to understand how to interpret it in the correct way.
0x2b6162630704fe17
Using the npm module hex2ascii it returns this:
"+abc\u0007\u0004þ\u0017"
if I convert from an online converter, it returns :
+abcþ
Could someone help me to interpret this? I am using node.
Am I doing something wrong?
Appreciate the help!
If you look at the string in the console, you will notice that the two strings you're posted are actually the same.
The gist is, the string contains nonprintable unicode characters, which get escaped by the hex2ascii module.
The online converter you are using tries to display those characters. Since they are not printable you simply cannot see them.
Let's convert the hex string
var conv = "2b6162630704fe17".match (/(..)/g).reduce ((a,c) => a + String.fromCharCode(parseInt (c,16)), "")
conv //"+abcþ"
It looks just like the String from the converter! Let's compare it to the other string
conv === "+abc\u0007\u0004þ\u0017" // true
Are you sure its 8 digit ASCII?
If it is, each 2 hex characters represents a given ASCII number.
So:
2b6162630704fe17
First 2b, which is 2 * 16 + 11 = 43 - which is a plus sign
61, which is 6 * 16 + 1 = 97 = lowercase a
62, which is 6 * 16 + 2 = 98 = lowercase b
63, which is 6 * 16 + 3 = 99 = lowercase c
07, which is 0 * 16 + 7 = 7 = that's a special unprintable character.
references to convert the numbers to characters - asciitable.com
Based on the 07, I wonder if your data is truly ascii, or a different encoding.
Related
Is there a built in way, or reasonably standard package that allows you to convert a standard UUID into a short string that would enable shorter URL's?
I.e. taking advantage of using a larger range of characters such as [A-Za-z0-9] to output a shorter string.
I know we can use base64 to encode the bytes, as follows, but I'm after something that creates a string that looks like a "word", i.e. no + and /:
id = base64.StdEncoding.EncodeToString(myUuid.Bytes())
A universally unique identifier (UUID) is a 128-bit value, which is 16 bytes. For human-readable display, many systems use a canonical format using hexadecimal text with inserted hyphen characters, for example:
123e4567-e89b-12d3-a456-426655440000
This has length 16*2 + 4 = 36. You may choose to omit the hypens which gives you:
fmt.Printf("%x\n", uuid)
fmt.Println(hex.EncodeToString(uuid))
// Output: 32 chars
123e4567e89b12d3a456426655440000
123e4567e89b12d3a456426655440000
You may choose to use base32 encoding (which encodes 5 bits with 1 symbol in contrast to hex encoding which encodes 4 bits with 1 symbol):
fmt.Println(base32.StdEncoding.EncodeToString(uuid))
// Output: 26 chars
CI7EKZ7ITMJNHJCWIJTFKRAAAA======
Trim the trailing = signs when transmitting, so this will always be 26 chars. Note that you have to append "======" prior to decode the string using base32.StdEncoding.DecodeString().
If this is still too long for you, you may use base64 encoding (which encodes 6 bits with 1 symbol):
fmt.Println(base64.RawURLEncoding.EncodeToString(uuid))
// Output: 22 chars
Ej5FZ-ibEtOkVkJmVUQAAA
Note that base64.RawURLEncoding produces a base64 string (without padding) which is safe for URL inclusion, because the 2 extra chars in the symbol table (beyond [0-9a-zA-Z]) are - and _, both which are safe to be included in URLs.
Unfortunately for you, the base64 string may contain 2 extra chars beyond [0-9a-zA-Z]. So read on.
Interpreted, escaped string
If you are alien to these 2 extra characters, you may choose to turn your base64 string into an interpreted, escaped string similar to the interpreted string literals in Go. For example if you want to insert a backslash in an interpreted string literal, you have to double it because backslash is a special character indicating a sequence, e.g.:
fmt.Println("One backspace: \\") // Output: "One backspace: \"
We may choose to do something similar to this. We have to designate a special character: be it 9.
Reasoning: base64.RawURLEncoding uses the charset: A..Za..z0..9-_, so 9 represents the highest code with alphanumeric character (61 decimal = 111101b). See advantage below.
So whenever the base64 string contains a 9, replace it with 99. And whenever the base64 string contains the extra characters, use a sequence instead of them:
9 => 99
- => 90
_ => 91
This is a simple replacement table which can be captured by a value of strings.Replacer:
var escaper = strings.NewReplacer("9", "99", "-", "90", "_", "91")
And using it:
fmt.Println(escaper.Replace(base64.RawURLEncoding.EncodeToString(uuid)))
// Output:
Ej5FZ90ibEtOkVkJmVUQAAA
This will slightly increase the length as sometimes a sequence of 2 chars will be used instead of 1 char, but the gain will be that only [0-9a-zA-Z] chars will be used, as you wanted. The average length will be less than 1 additional character: 23 chars. Fair trade.
Logic: For simplicity let's assume all possible uuids have equal probability (uuid is not completely random, so this is not the case, but let's set this aside as this is just an estimation). Last base64 symbol will never be a replaceable char (that's why we chose the special char to be 9 instead of like A), 21 chars may turn into a replaceable sequence. The chance for one being replaceable: 3 / 64 = 0.047, so on average this means 21*3/64 = 0.98 sequences which turn 1 char into a 2-char sequence, so this is equal to the number of extra characters.
To decode, use an inverse decoding table captured by the following strings.Replacer:
var unescaper = strings.NewReplacer("99", "9", "90", "-", "91", "_")
Example code to decode an escaped base64 string:
fmt.Println("Verify decoding:")
s := escaper.Replace(base64.RawURLEncoding.EncodeToString(uuid))
dec, err := base64.RawURLEncoding.DecodeString(unescaper.Replace(s))
fmt.Printf("%x, %v\n", dec, err)
Output:
123e4567e89b12d3a456426655440000, <nil>
Try all the examples on the Go Playground.
As suggested here, If you want just a fairly random string to use as slug, better to not bother with UUID at all.
You can simply use go's native math/rand library to make random strings of desired length:
import (
"math/rand"
"encoding/hex"
)
b := make([]byte, 4) //equals 8 characters
rand.Read(b)
s := hex.EncodeToString(b)
Another option is math/big. While base64 has a constant output of 22
characters, math/big can get down to 2 characters, depending on the input:
package main
import (
"encoding/base64"
"fmt"
"math/big"
)
type uuid [16]byte
func (id uuid) encode() string {
return new(big.Int).SetBytes(id[:]).Text(62)
}
func main() {
var id uuid
for n := len(id); n > 0; n-- {
id[n - 1] = 0xFF
s := base64.RawURLEncoding.EncodeToString(id[:])
t := id.encode()
fmt.Printf("%v %v\n", s, t)
}
}
Result:
AAAAAAAAAAAAAAAAAAAA_w 47
AAAAAAAAAAAAAAAAAAD__w h31
AAAAAAAAAAAAAAAAAP___w 18owf
AAAAAAAAAAAAAAAA_____w 4GFfc3
AAAAAAAAAAAAAAD______w jmaiJOv
AAAAAAAAAAAAAP_______w 1hVwxnaA7
AAAAAAAAAAAA_________w 5k1wlNFHb1
AAAAAAAAAAD__________w lYGhA16ahyf
AAAAAAAAAP___________w 1sKyAAIxssts3
AAAAAAAA_____________w 62IeP5BU9vzBSv
AAAAAAD______________w oXcFcXavRgn2p67
AAAAAP_______________w 1F2si9ujpxVB7VDj1
AAAA_________________w 6Rs8OXba9u5PiJYiAf
AAD__________________w skIcqom5Vag3PnOYJI3
AP___________________w 1SZwviYzes2mjOamuMJWv
_____________________w 7N42dgm5tFLK9N8MT7fHC7
https://golang.org/pkg/math/big
I've made a Webservice which returns a security Token after a successful authentification.
However when debugging I noticed that every hash the webservice returned finishes with "=" such as:
"tINH0JxmryvB6pRkEii1iBYP7FRedDqIEs0Ppbw83oc="
"INv7q72C1HvIixY1qmt5tNASFBEc0PnXRSb780Y5aeI="
"QkM8Kog8TtCczysDmKu6ZOjwwYlcR2biiUzxkb3uBio="
"6eNuCU6RBkwKMmVV6Mhm0Q0ehJ8Qo5SqcGm3LIl62uQ="
"dAPKN8aHl5tgKpmx9vNoYvXfAdF+76G4S+L+ep+TzU="
"O5qQNLEjmmgCIB0TOsNOPCHiquq8ALbHHLcWvWhMuI="
"N9ERYp+i7yhEblAjaKaS3qf9uvMja0odC7ERYllHCI="
"wsBTpxyNLVLbJEbMttFdSfOwv6W9rXba4GGodVVxgo="
"sr+nF83THUjYcjzRVQbnDFUQVTkuZOZYe3D3bmF1D8="
"9EosvgyYOG5a136S54HVmmebwiBJJ8a3qGVWD878j5k="
"8ORZmAXZ4dlWeaMOsyxAFphwKh9SeimwBzf8eYqTis="
"gVepn2Up5rjVplJUvDHtgIeaBL+X6TPzm2j9O2JTDFI="
Why such a behavior ?
This is because you don't see the raw bytes of the hash but rather the Base64 encoding.
Base64-encoding converts a block of 3 bytes to a block of four characters. This works well if the number of bytes is divisible by 3. If it is not, then you use a padding-character so the number of resulting characters is still divisible by 4.
So:
(no of bytes)%3 = 0 => no padding needed
(no of bytes)%3 = 1 => pad with ==
(no of bytes)%3 = 2 => pad with =
A SHA256-hash is 256 bit, that's 32 bytes. So you will get 40 characters for the first 30 bytes, 3 characters for the last 2 bytes and the padding will always be one =.
These strings are encoded using base64, = characters are used as paddings, to make the last block of a base64 string contains four characters.
The following Ruby code could be used to get base64 decoded string:
require 'base64'
s = "tINH0JxmryvB6pRkEii1iBYP7FRedDqIEs0Ppbw83oc="
puts Base64.decode64(s).bytes.map{|e| '%02x' % e}.join
Output: b48347d09c66af2bc1ea94641228b588160fec545e743a8812cd0fa5bc3cde87
I've been trying different ways to read this hex string, and I cannot figure out how. Each method only converts part of it. The online converters don't do it, and this is the method I tried:
function string.fromhex(str)
return (str:gsub('..', function (cc)
return string.char(tonumber(cc, 16))
end))
end
packedStr = "1b4c756151000104040408001900000040746d702f66756e632e416767616d656e696f6e2e6c756100160000002b0000000000000203000000240000001e0000011e008000000000000100000000000000170000002900000000000008410000000500000006404000068040001a400000160000801e0080000a00800041c0000022408000450001004640c1005a40000016800c80458001008500000086404001868040018600420149808083458001008580020086c04201c5000000c640c001c680c001c600c301014103009c80800149808084450001004980c38245c003004600c4005c408000458002004640c40085800400860040018640400186c0440186004201c14003005c00810116000280450105004641c50280010000c00100025c8180015a01000016400080420180005e010001614000001600fd7f4580050081c005005c400001430080004700060043008000478001004300800047c003001e008000190000000405000000676d63700004050000004368617200040700000053746174757300040b000000416767616d656e696f6e000404000000746d7000040f000000636865636b65645f7374617475730004030000007477000409000000636861724e616d6500040900000066756c6c6e616d6500040a0000006775696c644e616d65000407000000737472696e670004060000006d617463680004060000006775696c6400040400000025772b0001010403000000756900040d00000064726177456c656d656e7473000407000000676d617463680004030000005f470004050000004e616d650004060000007461626c6500040900000069734d656d6265720004060000006572726f7200044e0000005468697320786d6c206973206e6f742070726f7065726c7920636f6e6669677572656420666f722074686973206368617261637465722e20506c6561736520636f6e74616374204b616575732100040900000071756575654163740000000000410000001800000018000000180000001800000018000000180000001900000019000000190000001a0000001a0000001a0000001a0000001b0000001b0000001b0000001b0000001b0000001b0000001c0000001c0000001c0000001c0000001c0000001c0000001c0000001c0000001c0000001c0000001d0000001d0000001e0000001e0000001e0000001f0000001f0000001f0000001f0000001f0000001f0000001f0000001f0000001f0000001f0000002000000020000000200000002000000020000000200000002000000021000000210000001f000000220000002400000024000000240000002500000025000000260000002600000027000000270000002900000005000000020000006e0009000000400000001000000028666f722067656e657261746f7229002b000000370000000c00000028666f7220737461746529002b000000370000000e00000028666f7220636f6e74726f6c29002b000000370000000200000074002c000000350000000000000003000000290000002a0000002b000000010000000800000072657466756e6300010000000200000000000000"
local f = assert(io.open("unsquished.lua", "w+"));
f:write(packedStr:fromhex());
f:close()
This simply gives me a bunch of gibberish surrounded by a few readable strings.
Could someone please tell me how to convert the entirety of this string into readable format? Thank you!
Break your packedStr in parts of 2
1b = 27
4c = 76
75 = 117
61 = 97
and so forth. When you use string.char() with the resulting decimal output, it converts them to equivalent ASCII values. Of the total possible 256 ASCII values in extended ASCII table, only 95 are printable characters.
Thus, you'll always receive the gibberish text. Here's what you'd receive when trying to print each of the character separately: http://codepad.org/orM7pmAb and that is the only possible "readable" output.
I have a program that reads in hex strings, and returns text based on the parameters of the string. A hex string goes as follows:
A : B : C
A - Length of B
B - Name of interface
C - Unimportant
so for example;
0465746830010000 =
04 : 65746830 : 010000 =
4 : eth0 : __________
Now, I want to process the hex strings so that if there is a character that isn't in the alphabet, 0-9, or '-' , it lets me know somehow.
Such as here :
0266010000
02 : 6601 : _______
2 : f[unreadable] : _______
Any ideas on how would I process this so that it lets me know if any of the characters outside of these parameters arise?
A quick regex can do that, for instance in java:
myString.matches([0-9a-fA-F]*)
will return true if the entire string consists of hex digits, and will return false if there are other characters.
Also, why would you want to accept the - character? These strings should (if properly designed) contain the unsigned representations of the numbers. If they are out of your control, then I guess you just gotta deal with it, but still.
Suppose that I is the size of a T integer.
What is the maximum size of a string S that contains the digits of T arranged into characters?
For example:
T = 12345
S = '12345'
log10(T) + 1 will give you the size (in characters) of the string S
Actually, the basic equation only works for ASCII characters in ASCII or UTF-8 encoding; one byte per character. For UTF-16, these same characters would be encoded as 2 bytes each, and in UTF-32, 4 bytes each. This matters, depending on the programming language and runtime; .NET strings are stored and encoded in UTF-16.
So, it's actually (log(N) + 1)*sizeof(char)
The answer is the log (base10) of I + 1
Thus an int I of 1000 - log I would give you 3 + 1 = 4