Using Sybase ASE 12.5 I have the following situation.
2 values stored in float cols when multiplied give a value.
Converting that value to a varchar (or retrieving it with Java) gives the underlying precise value which the floats approximated to.
My issue is that the value as represented by the floats is correct, but the precise value is causing issues (due to strict rounding rules).
For example
declare #a float,#b float
select #a = 4.047000, #b = 1033000.000000
select #a*#b as correct , str(#a*#b,40,20) as wrong
gives:
correct: 4180551.000000,
wrong: 4180550.9999999995343387
Similarly when
#a = 4.047000, #b = 1
...you get
correct: 4.047000,
wrong: 4.0469999999999997
(same thing happens using convert(varchar(30), #a*#b) and cast(#a*#b, varchar(30) )
I appreciate it would be easy to just round the first example in java but for various business reasons that cannot be done and in any case it wouldn't work for the second.
I also cannot change the float table column datatype.
Is there anyway to get the float representation of the multiplication product either as a string or the actual 'correct' value above?
Thanks
Chris
Related
I am trying to encrypt with rsa using the formula c = m^p mod q.
The Problem is if the number is too large, python3 convert it to float when doing modulo.
I tried to stop converting by converting into int
c = int(int((pow(n,p)) % q))
the problem is when p is too big it automatically has decimals and python thinks , that i am trying integer to float. Which leads to this:
OverflowError: int too large to convert to float
Is there a way to solve this ?
This may not solve your problem, but it does address the specific concerns you put forth in your question and suggests possible causes based on what you've told us.
The problem you're having isn't with %. As per the documentation,
The floor division and modulo operators are connected by the following identity: x == (x//y)*y + (x%y).
Given integer x and y, (x//y)*y is always an exact integer, so x - (x//y)*y == x%y must also be an integer.
Since you said you are using the built-in pow function, I suspect that your problem is that your inputs are floats instead of ints. In that case, both pow and ** will try to convert the other of the argument to float, which could be the source of your error. If this is the case, wrapping each argument in int will make the error go away, but your RSA implementation will be incorrect.
a = 28.85
b = 2000
print(a*b)
Result 57700.0
select name from fake limit 57700.0 ,10
This sentence is incorrect.
Multiplying a float with an int naturally gives you a float as an answer.
So, as #Rakesh suggested, truncate it with int(a*b).
Beware that you will lose everything after the dot...
You can keep the result as float if you use format in your print statement and cut all numbers behind the dot.
print("{:0.0f}".format(a * b))
I knew that a range of float color value in a shader [0..1] is mapped into range of [0..255] in UCHAR buffer.
According to this, I was expecting for steps of size of 1/255 in shader color values for each change in UCHAR buffer.
But the results were surprisingly different. Here is for the first two steps:
Red float value in Shader -> UCHAR value in a read Buffer
0.000000 -> 0
0.002197 -> 0
0.002198 -> 1
0.006102 -> 1
0.006105 -> 2
The first two steps are around 0.002197 and 0.006102 which are different than the expected steps: 0.00392 and 0.00784.
So what is the mapping formula ?
Unsigned integer normalization is based on the formula f = i/INT_MAX, where f is the floating point value (after clamping to [0, 1]), i is the integer value, and INT_MAX is the maximum integer value for the integer's bitdepth (255) in this case.
So if you have a float, and want the unsigned, normalized integer value of it, you use i = f * INT_MAX. Of course... integers do not have the same precision as floats. So if the result of f * INT_MAX is 0.5, what is the integer value of that? It could be 0, or it could be 1, depending on how things are rounded.
Implementations are permitted to round integer values in any way they prefer. They are encouraged to use nearest rounding (the post-conversion 0.49 would become 0, and 0.5 would become 1), but that is not a requirement. The only requirements are that it must pick one of the two nearest values (it can't turn 0.5 into 3) and that the exact floating-point values of 0.0 and 1.0 (which includes any values clamped to them) must be exactly represented as integer 0 and INT_MAX.
If you have an explicit need to have direct rounding, you can always do the normalization yourself. In fact, GLSL has specific functions to help you. The following assumes that you are trying to write to a texture with the Vulkan format R8G8B8A8_UNORM, and we're assuming you're writing to a storage image, not via outputs from the fragment shader (you can do that too, but you lose blending).
So, step 1 is to change your layout format to be r32ui. That is, you are now writing an unsigned 32-bit value, rather than 4 unsigned 8-bit normalized values. That's perfectly valid.
Step 2 is to employ the packUNorm4x8 function. This function does float-to-integer normalization, but the specification explicitly performs rounding correctly. Use the return value of that function in your imageStore function, and you're fine.
If you want to write to a fragment shader output, that's a bit more complex. There, you will need to use a different image view, one that uses the R32_UINT format. So you're creating a 32-bit unsigned integer view of a 4x8-bit normalized texture. That has to become a render target, so you're going to have to do subpass surgery. From there, just write the result of packUNorm4x8.
Of course, you immediately lose blending and similar operations, since you're writing integers values. And since you had to do that subpass surgery, it's likely that any shader writing to it will need to do this too.
Also, note that in both cases, you will likely need to adjust the order of the components of the value you write. packUNorm4x8 is explicitly defined to be little endian, whereas (I believe?) R8G8B8A8 is specified to be in that order, most-significant to least. So you'll probably need to essentially do endian swapping with packUNorm4x8(value.abgr).
Why does the math module return the wrong result?
First test
A = 12345678917
print 'A =',A
B = sqrt(A**2)
print 'B =',int(B)
Result
A = 12345678917
B = 12345678917
Here, the result is correct.
Second test
A = 123456758365483459347856
print 'A =',A
B = sqrt(A**2)
print 'B =',int(B)
Result
A = 123456758365483459347856
B = 123456758365483467538432
Here the result is incorrect.
Why is that the case?
Because math.sqrt(..) first casts the number to a floating point and floating points have a limited mantissa: it can only represent part of the number correctly. So float(A**2) is not equal to A**2. Next it calculates the math.sqrt which is also approximately correct.
Most functions working with floating points will never be fully correct to their integer counterparts. Floating point calculations are almost inherently approximative.
If one calculates A**2 one gets:
>>> 12345678917**2
152415787921658292889L
Now if one converts it to a float(..), one gets:
>>> float(12345678917**2)
1.5241578792165828e+20
But if you now ask whether the two are equal:
>>> float(12345678917**2) == 12345678917**2
False
So information has been lost while converting it to a float.
You can read more about how floats work and why these are approximative in the Wikipedia article about IEEE-754, the formal definition on how floating points work.
The documentation for the math module states "It provides access to the mathematical functions defined by the C standard." It also states "Except when explicitly noted otherwise, all return values are floats."
Those together mean that the parameter to the square root function is a float value. In most systems that means a floating point value that fits into 8 bytes, which is called "double" in the C language. Your code converts your integer value into such a value before calculating the square root, then returns such a value.
However, the 8-byte floating point value can store at most 15 to 17 significant decimal digits. That is what you are getting in your results.
If you want better precision in your square roots, use a function that is guaranteed to give full precision for an integer argument. Just do a web search and you will find several. Those usually do a variation of the Newton-Raphson method to iterate and eventually end at the correct answer. Be aware that this is significantly slower that the math module's sqrt function.
Here is a routine that I modified from the internet. I can't cite the source right now. This version also works for non-integer arguments but just returns the integer part of the square root.
def isqrt(x):
"""Return the integer part of the square root of x, even for very
large values."""
if x < 0:
raise ValueError('square root not defined for negative numbers')
n = int(x)
if n == 0:
return 0
a, b = divmod(n.bit_length(), 2)
x = (1 << (a+b)) - 1
while True:
y = (x + n//x) // 2
if y >= x:
return x
x = y
If you want to calculate sqrt of really large numbers and you need exact results, you can use sympy:
import sympy
num = sympy.Integer(123456758365483459347856)
print(int(num) == int(sympy.sqrt(num**2)))
The way floating-point numbers are stored in memory makes calculations with them prone to slight errors that can nevertheless be significant when exact results are needed. As mentioned in one of the comments, the decimal library can help you here:
>>> A = Decimal(12345678917)
>>> A
Decimal('123456758365483459347856')
>>> B = A.sqrt()**2
>>> B
Decimal('123456758365483459347856.0000')
>>> A == B
True
>>> int(B)
123456758365483459347856
I use version 3.6, which has no hardcoded limit on the size of integers. I don't know if, in 2.7, casting B as an int would cause overflow, but decimal is incredibly useful regardless.
In an IOS program i am trying to divide some float value but the result is incorrect
float a = 179.891891;
float b = 8.994595;
NSLog(#"Result %f",a/b);
On dividing the two (a/b) the output i get is 20.0000 instead of 19.9999989993991 . I have tried using double instead of float but still the same issue . The value of "b" keeps on varying as i obtain it from some calculations . I need the result to be exactly precise as it further is required for some calculations and 20.0000 instead of 19.9999989993991 make a lot of difference in the final output i get .
Any help on this would be really great :).
I dont know in Objective C, but in C, you should do casting: (float)(a/b) .Otherwise it is integer division.