I am writing a JPEG parser/modifier/unparser. First, to make sure we are using the same terninology I include
a definition adapted from a very useful source
DHT( Define Huffman Table) marker:
Field Size Description
Marker Identifier 2 bytes 0xff, 0xc4 to identify DHT marker
Length 2 bytes This specify length of Huffman table
HT information 1 byte bit 0..3 : number of HT (0..3, otherwise error)
bit 4 : type of HT, 0 = DC table, 1 = AC table
bit 5..7 : not used, must be 0
Number of Symbols 16 bytes Number of symbols with codes of length 1..16,
the sum(n) of these bytes is the total number of
codes, which must be <= 256
Symbols n bytes Table containing the symbols in order of
increasing code length ( n = total number of
codes ).
My Question:
The Symbols must be ordered by increasing code length. But within each code length do they have to be ordered (eg by increasing value).
The reason I ask is that my table generated from frequencies collected from the AC scan data yields the right bit lengths, but they are not in increasing value order within the tree (when read in depth-first traversal order). I need to order these and regenerate the tree to get the same bit pattern as when I write and read my table.
I suspect that this is because in my write routine I specifically order the Symbols by bit length then value. If ordering by value is unnecessary I can remove the overhead (in code and at runtime).
The codes are implicitly ordered by increasing value. The DHT marker only specifies the number of codes of each length. The procedure in the JPEG standard describes how to generate the actual Huffman code for each value.
Related
For practice I've implemented the qoi specification in rust. In it there is a small hash function to store recently used pixels:
index_position = (r * 3 + g * 5 + b * 7 + a * 11) % 64
where r, g, b, and a are the red, green, blue and alpha channels respectively.
I assume this works as a hash because it creates a unique prime factorization for the numbers with the mod to limit the number of bytes. Anyways I implemented it naively in my code.
While looking at other implementations I came across this bit hack to optimize the hash calculation:
fn hash(rgba:[u8:4]) -> u8 {
let v = u32::from_ne_bytes(rgba);
let s = (((v as u64) << 32) | (v as u64)) & 0xFF00FF0000FF00FF;
s.wrapping_mul(0x030007000005000Bu64.to_le()).swap_bytes() as u8 & 63
}
I think I understand most of what's going on but I'm confused about the magic number (the multiplicand). To my understanding it should be flipped. As a step by step example:
let rgba = [0x12, 0x34, 0x56, 0x78].
On my machine (little endian) this gives v the value 0x78563412.
The bit shifting spreads the values, giving s = 0x7800340000560012.
Now here's where I get confused. The magic number has the values that should be multiplied aligned in a 64 bit field (3, 5, 7, 11), spaced the same way that the original values are. However they seem to be in reverse order from the values:
0x7800340000560012
0x030007000005000B
When multiplying it would seem that the highest value, the alpha channel (0x78), is being multiplied by 3, while the lowest value, the red channel (0x12), is being multiplied by 11. I'm also not entirely sure why this multiplication works anyway, after multiplying the values by various powers of 2.
I understand that the bytes are then swapped to big endian and trimmed, but that's not until after the multiplication step which loses me.
I know that the code produces the correct hash, but I don't understand why that's the case. Can anyone explain to me what I'm missing?
If you think about the way the math works, you want this flipped order, because it means all the results from each of the "logical" multiplications cluster in the same byte. The highest byte in the first value multiplied by the lowest byte in the second produces a result in the highest byte. The lowest byte in the first value's product with the highest byte in the second value produces a result in the same highest byte, and the same goes for the intermediate bytes.
Yes, the 0x78... and 0x03... are also multiplied by each other, but they overflow way past the top of the value and are lost. Having the order "backwards" means the result of the multiplications we care about all ends up summed in the uppermost byte (the total shift of the results we want is always 56 bits, because the 56th bit offset value is multiplied by the 0th, the 40th by the 16th, the 16th by the 40th, and the 0th by the 56th), with the rest of the multiplications we don't want having their results either overflow (and being lost) or appearing in lower bytes (which we ignore). If you flipped the bytes in the second value, the 0x78 * 0x0B (alpha value & multiplier) component would be lost to overflow, while the 0x12 * 0x03 (red value & multiplier) component wouldn't reach the target byte (every component we cared about would end up somewhat that wasn't the uppermost byte).
For a possibly more intuitive example, imagine doing the same work, but where all the bytes of one input except a single component are zero. If you multiply:
0x7800000000000000 * 0x030007000005000B
the logical result is:
0x1680348000258052800000000000000
but removing the overflow reduces that to:
0x2800000000000000
//^^ result we care about (actual product of 0x78 and 0x0B is 0x528, but only keeping low byte)
Similarly,
0x0000340000000000 * 0x030007000005000B
produces:
0x9c016c000104023c0000000000
overflowing to:
0x04023c0000000000
//^^ result we care about (actual product of 0x34 and 0x5 was 0x104, but only 04 kept)
In that case, the other multiplications did leave data in result (not all overflowed), but since we only look at the high byte, the rest gets ignored.
If you keep doing this math step by step and adding the results, you'll find that the high byte ends up the correct answer to the four individual multiplications you expected (mod 256); flip the order, and it won't work out that way.
The advantage to putting all the results in that high byte is that it allows you to use swap_bytes to move it cheaply to the low byte, and read the value directly (no need to even mask it on many architectures).
I would like to count '01' sequence in 5760 binary bits.
First, I would like to combine several binary numbers then count # of '01' occurrences.
For example, I have 64 bits integer. Say, 6291456. Then I convert it into binary. Most significant 4 bits are not used. So I'll get 60 bits binary 000...000011000000000000000000000
Then I need to combine(just put bits together since I only need to count '01') first 60 bits + second 60 bits + ...so 96 of 60 bits are stitched together.
Finally, I want to count how many '01' appears.
s = binToString(5760 binary bits)
cnt = s.count('01');
num = 6291226
binary = format(num, 'b')
print(binary)
print(binary.count('01'))
If I use number given by you i.e 6291456 it's binary representation is 11000000000000000000000 which gives 0 occurrences of '01'.
If you always want your number to be 60 bits in length you can use
binary = format(num,'060b')
It will add leading 0 to make it of given length
Say that nums is your list of 96 numbers, each of which can be stored in 64 bits. Since you want to throw away the most 4 significant bits, you are really taking the number modulo 2**60. Thus, to count the number of 01 in the resulting string, using the idea of #ShrikantShete to use the format function, you can do it all in one line:
''.join(format(n%2**60,'060b') for n in nums).count('01')
Was trying to solve this popular interview question - http://www.careercup.com/question?id=3406682
There are 2 approaches to this that i was able to grasp -
Brian Kernighan's algo -
Bits counting algorithm (Brian Kernighan) in an integer time complexity
Lookup table.
I assume when people say use a lookup table, they mean a Hashmap with the Integer as key, and the count of number of set bits as value.
How does one construct this lookup table? Do we use Brian's algo to to count the number of bits the first time we encounter an integer, put it in hashtable, and next time we encounter that integer, retrieve value from hashtable?
PS: I am aware of the hardware and software api's available to perform popcount (Integer.bitCount()), but in context of this interview question, we are not allowed to use those methods.
I was looking for Answer everywhere but could not get the satisfactory explanation.
Let's start by understanding the concept of left shifting. When we shift a number left we multiply the number by 2 and shifting right will divide it by 2.
For example, if we want to generate number 20(binary 10100) from number 10(01010) then we have to shift number 10 to the left by one. we can see number of set bit in 10 and 20 is same except for the fact that bits in 20 is shifted one position to the left in comparison to number 10. so from here we can conclude that number of set bits in the number n is same as that of number of set bit in n/2(if n is even).
In case of odd numbers, like 21(10101) all bits will be same as number 20 except for the last bit, which will be set to 1 in case of 21 resulting in extra one set bit for odd number.
let's generalize this formual
number of set bits in n is number of set bits in n/2 if n is even
number of set bits in n is number of set bit in n/2 + 1 if n is odd (as in case of odd number last bit is set.
More generic Formula would be:
BitsSetTable256[i] = (i & 1) + BitsSetTable256[i / 2];
where BitsetTable256 is table we are building for bit count. For base case we can set BitsetTable256[0] = 0; rest of the table can be computed using above formula in bottom up approach.
Integers can directly be used to index arrays;
e.g. so you have just a simple array of unsigned 8bit integers containing the set-bit-count for 0x0001, 0x0002, 0x0003... and do a look up by array[number_to_test].
You don't need to implement a hash function to map an 16 bit integer to something that you can order so you can have a look up function!
To answer your question about how to compute this table:
int table[256]; /* For 8 bit lookup */
for (int i=0; i<256; i++) {
table[i] = table[i/2] + (i&1);
}
Lookup this table on every byte of the given integer and sum the values obtained.
I have a database table where every row has its unique ID (RowID).
Is there a good way to convert this RowID to a unique key that is always 6 characters in length. Unique key characters can be {A-Za-z0-9}. One example of unique key would be: a5Fg3A.
Of course I do realize there's only certain number of keys I can generate using this method but that doesn't matter for my case.
I've thought much about this but I can't come up with an algorithm that would be able to do this properly.
One idea I had was:
Unique key = RowID
If RowID is lower than 100000 then append 0 in front of it, for example:
123 becomes 000123
1 becomes 000001
Then for numbers in the range of 100000 to 900000 I would replace first number to a string, e.g. 0 = a, 1 = b, 2 = c, ..., 9 = j.
Then I could do the same with capital letter, etc.
My problem is that my algorithm is very limited and generates low number of keys because it wouldn't utilize all possible characters.
So basically I should be able to generate 56800235584 unique keys assuming every key is of length 6 and utilizes these characters: {A-Za-z0-9}.
A-Z = 26 characters
a-z = 26 characters
0-9 = 10 characters
So it's 62^6 unique keys.
Any feedback would be appreciated on how this could be done properly (or even optimal) :-)
Thanks!
You can sort your IDs, and then attach an increasing lexicographical string to each.
Simple example where your alphabet is only {a,b} (for simplicity only), and Ids= [20,1,7,90]:
sort: Ids = [1,7,20,90]
Attach increasing strings:
1 = aaaaaa
7 = aaaaab
20 = aaaaba
90 = 0000bb
If you want it as a hash function of some sort, and not data dependent - you can just use the same binary encoding that is used to the number, and convert it similary (i.e. 1 = aaaaaa, 2 = aaaaab, 3 = aaaaac...)
[Edit: basically the same as base-62 suggested by #HighPerformanceMark in comments]
The advantages of the first approach: allows you to deal with up to 62^6 numbers, regardless of that their size is, while the second approach does not allow it.
The second approach however, allows you a consistent conversion from number to string, regardless on the specific data.
If you want to make A-Z a-z 0-9 to be the alphabet as you noticed you have base 62 number system. So encode the unique rowid in base 62, there is a standard algorithm to do so. If your application allows (needs) it you can add a few more printable characters like '+', '/', '!', '#'.. so you get more uniques. The ready made answer is base64 encoding, widely used.
There are many ways to do this - the challenge is picking the one that's "best" for whatever your criteria are. Some examples, but far from exhaustive (some already suggested elsewhere):
pad with an increasing sequence
base-62 representation (note: base-64 is in common use and might even already have code available for it in whatever libraries you have at hand)
truncated cryptographic hash (slow, but has some other properties that might be useful, depending on exactly why you need to do this; if you only have to do it once, the performance hit may be worth it)
other not-necessarily-cryptographic hash functions that might be considerably faster
......
I was wondering how I would hash a string value (ex: "myObjectName") to int values between 0-19
I'm guaranteed to have no more than 20 unique string values.
Thanks
Adding my comment as an answer as suggested:
I would suggest that hashing isn't the exact path you should follow here.
One method would be using a dictionary (like the built in data structure in Python) that has a key-value pair of your string and a number from 1-20 (or 0 - 19)
As you read or see each string, you could check to see if a dictionary entry exists, if so, do whatever needs to be done, if not, create a new dictionary entry with the next available number (generated by looking at the number of existing entries in the dictionary).
You could use any sort of hashing you like, but in this case, you could do with adding up the ASCII values (or unicode code point, if you like) of the characters, and apply modulo 20 to the result. It will give you a number from 0 to 19.
But this is nog guaranteed to result in a number that uniquely identifies your 20 strings. No hashing algorithm will guarantee that hashing a collection of 20 random strings will result in a unique code for each string..
Do md5 sum, convert to number and do modulo 20. E.g. in PHP:
hexdec(substr(md5("hello"), 1, 8)) % 20
The substr() is needed so that the number can be converted to integer.