This question already has answers here:
How to get a float result by dividing two integer values using T-SQL?
(10 answers)
Closed 7 years ago.
I am trying to calculate some performance metrics as [RATE] in SQL but no matter what I do I am only getting integer values in return.
In my dataset I have a Numerator and a Denominator both stored as integers that are broken down into groups and subsets. The rate is calculated slightly differently depending on the group and subset. For group 1 the calculation is simply N/D. For group 2 the calculation is (N*1000)/D with the exception of 1 subset which is calculated (N*10000)/D.
I wrote the query as:
SELECT [Group]
,[SubSet]
,[Numerator] N
,[Denominator] D
,CASE WHEN D=0 Then NULL
WHEN [Group]='G1' THEN [N]/[D]
WHEN [SubSet]='S2' THEN ([N]*10000)/[D]
WHEN [SubSet] NOT LIKE 'S2%' AND [G] NOT LIKE 'G1%' THEN ([N]*1000)/[D] as [RATE]
No matter what I do the outcome variables are integers. I tried formatting RATE as varchar, decimal, and float with no success. I tried changing N and D's format to varchar, decimal, and float as well. I tried changing the equations from (N*1000)/D to (N/D)*1000 but still no effect.
What am I missing/doing wrong?
The problem you are having is because SQL is doing integer division, which will only return whole numbers. To get a decimal return value you must have at least one value as a decimal.
Try this:
(CAST([N] as decimal(12,6))/[D]) * 1000
Adjust decimal(12,6) based on the precision you are expecting. 12,6 will return a decimal with 6 digits after the decimal point. If you wanted only 2 decimal places use 16,2.
If you then want to round the calculated value you will need to make use of the ROUND function in SQL.
Round to the second decimal place:
ROUND((CAST([N] as decimal(12,6))/[D]) * 1000, 2)
You need to use CAST:
CAST ((N*1000) AS FLOAT) / D
Hope this helps.
SELECT (n * 1000.0) will do it.
Related
This question already has answers here:
Python 3.x rounding behavior
(13 answers)
Closed 3 years ago.
I want to round some int numbers but I came across with the strange feature of round() for example
round(2.1) = 2
round(2.5) = 2 #it rounds to ceil
round(2.7) = 3
it rounds differently with the odd number as follow
round(5.1) = 5
round(5.5) = 6 #it rounds to floor
round(5.7) = 6
it rounds the X.5 to the floor with the x = even numbers but with the X = odd numbers it rounds to the ceil
I want to ask what is the advantage of this round? and where can I use it in our examples ? or what is its usage?
Looks like if it's close it goes to the even option.
From the documentation https://docs.python.org/3/library/functions.html#round
Return number rounded to ndigits precision after the decimal point. If ndigits is omitted or is None, it returns the nearest integer to its input.
For the built-in types supporting round(), values are rounded to the closest multiple of 10 to the power minus ndigits; if two multiples are equally close, rounding is done toward the even choice (so, for example, both round(0.5) and round(-0.5) are 0, and round(1.5) is 2). Any integer value is valid for ndigits (positive, zero, or negative). The return value is an integer if ndigits is omitted or None. Otherwise the return value has the same type as number.
For a general Python object number, round delegates to number.round.
Note The behavior of round() for floats can be surprising: for example, round(2.675, 2) gives 2.67 instead of the expected 2.68. This is not a bug: it’s a result of the fact that most decimal fractions can’t be represented exactly as a float. See Floating Point Arithmetic: Issues and Limitations for more information.
I have some time data that, without VBA, I need to round to the nearest 10 milliseconds. For example:
input: 01:02:03.017 output: 01:02:03.020
input: 03:12:44.123 output: 03:12:44.120
Current approach is to convert to an integer number of milliseconds; round that to the nearest 10; finally convert back to time:
=ROUND(A1*86400000,10)/86400000
I must be making a really stupid error, just don't see it.
EDIT:
The formula is returning the same value as the input?!?
Change the 10 to 0
By multiplying the value you want the round to the nearest integer, then divide again. By using 10 you are rounding to the 10th decimal place after creating a integer time.
=ROUND(A1*86400000,0)/86400000
Assuming your time data is in column A:
=TEXT(A1,"hh:mm:ss.00")+0
Change the number format with TEXT, then add the ending 0 with +0
I am stuck into a very weird problem in currency calculation. I need to keep digits after point but somehow spark is showing flooring or ceiling value
as example
My expected output is 871.25
spark output is 871.00
i am getting double type value from intermediate table for this i have cast double value to decimal value with fixed 2 digit precision. My test code is
spark.sql("select cast(SUM(TRANAMT) as DECIMAL(20,2)) as Expr1 from CMSDLG").show()
I am not getting which part i need to focus here. Kindly Help me . I am using pyspark 2.0
you should sum after cast...
spark.sql("select SUM(cast(TRANAMT as DECIMAL(20,2)) ) as Expr1 from CMSDLG").show()
Use this query instead, to cast varchar into decimal and then round the sum to 2 decimal places.
spark.sql("select round(sum(cast(TRANAMT as decimal(65,30))),2) as Expr1 from CMSDLG").show()
I have user input in two cells, named "UpperRangeHigh" and "UpperRangeLow". I have the following code:
dRangeUpper = [UpperRangeHigh] - [UpperRangeLow]
lLines = Int(dRangeUpper * 100 / lInterval)
The user inputs 120.3 and 120 into the input cells respectively. lInterval has the value 10. VBA produces the result of 2 for lLines, instead of 3.
I can overcome this problem by adding 0.000000001 to dRangeUpper, but I'm wondering if there is a known reason for this behaviour?
This appears to be a problem with Excel's calculation and significant digits. If you do:
=120.3 - 120 and format the cell to display 15 decimal places, the result appears as:
0.2999999999999970
Here is a brief overview which explains how Excel uses binary arithmetic and that this may result in results divergent from what you would expect:
http://excel.tips.net/T008143_Avoiding_Rounding_Errors_in_Formula_Results.html
You can overcome this by forcing a rounded precision, e.g., to 10 decimal places:
lLines = Int(Round(dRangeUpper, 10) * 100 / lInterval
Kindly use single or double when working with decimals to get more accurate results.
Sub sample()
Dim dRangeUpper As Double
dRangeUpper = CDbl("120.3") - CDbl("120")
lLines = Round(CDbl(dRangeUpper * 100 / 10), 4)
End Sub
output = 3
This is a known Floating point issue within Excel
http://support.microsoft.com/kb/78113
From MSDN:
To minimize any effects of floating point arithmetic storage
inaccuracy, use the Round() function to round numbers to the number of
decimal places that is required by your calculation. For example, if
you are working with currency, you would likely round to 2 decimal
places:
=ROUND(1*(0.5-0.4-0.1),2)
In your case, using round() instead of INT should do the trick using 0 rather than 2
I have this obscure rounding problem in VBA.
a = 61048.4599674847
b = 154553063.208822
c = a + b
debug.print c
Result:
154614111.66879
Here is the question, why did VBA rounded off variable c? I didn't issued any rounding off function. The value I was expecting was 154614111.6687894847. Even if I round off or format variable c to 15 decimal places I still don't get my expected result.
Any explanation would be appreciated.
Edit:
Got the expected results using cDec. I have read this in Jonathan Allen's reply in Why does CLng produce different results?
Here is the result to the test:
a = cDec(61048.4599674847)
b = cDec(154553063.208822)
c = a + b
?c
154614111.6687894847
The reason is the limited precission that can be stored in a floating point variable.
For a complete explanation you shoud read the paper What Every Computer Scientist Should Know About Floating-Point Arithmetic, by David Goldberg, published in the March, 1991 issue of Computing Surveys.
Link to paper
In VBA the default floating point type is Double which is a IEEE 64-bit (8-byte) floating-point number.
There is another type available: Decimal which is a 96-bit (12-byte) signed integers scaled by a variable power of 10
Put simply, this provides floating point numbers to 28 digit precission.
To use in your example:
a = CDec(61048.4599674847)
b = CDec(154553063.208822)
c = a + b
debug.print c
Result:
154614111.6687894847
Its not obscure, but its not necessarily obvious.
I think you've sort of answered it - but the basic problem is one of the "size" of the values that is how much data can be stored in a variable of a given type.
If (and this is very crude) you count the number of digits in each of the numbers in your first example you will see that you have 15 so whilst the range of values that a float (the default type) can represent is huge the precision is limited to 15 digits (I'm sure someone will be along to correct this, I'll tick the wiki box...)
So when you add the two numbers together it loses the least significant values in order to remain within the allowable precision for a flow.
By doing a cDec you're converting to a different type of variable (decimal) that is capable of greater precision