I'm trying to divide two decimal(15,2) numbers and return a high precision decimal of 15 places, but I keep getting the [2616] Numeric overflow error on anything beyond 10 decimal places, is there another way to write this to achieve a decimal result of 15 places right of the period?
select (cast(19983.74 as decimal(15,2))*1.0000000000000)/(cast(15897.40 as decimal(15,2))*1.0000000000000)
For more complex calculations I prefer casting the 1st operand to NUMBER, this results in the maximum possible precision:
SELECT Cast(19983.74 AS NUMBER)/15897.40
1.25704454816510875992300627775611106219
Finally cast back to whatever you need:
SELECT Cast(Cast(19983.74 AS NUMBER)/15897.40 AS DECIMAL(38,15))
1.257044548165109
See the Teradata Documentation. Defaults for result precision depend on MaxDecimal setting, but a CAST to something larger than MaxDecimal setting will override.
SELECT CAST(19983.74 AS DECIMAL(38,15))/15897.40
Related
The value is the result of dividing the sum of 2.01 and 2.52 by 2 (2.01 + 2.52) / 2. Excel displays this value as 2.265 and when formatted to 2 digits, it's 2.27. However, the value stored in the file is 2.2649999999999997. When I recreate this is C#, I also get that value in my variable, not 2.265. I understand this is due to floating point precision issues with the division of 4.53 by 2.
double result = (2.01 + 2.52) / 2;
Console.WriteLine(result);
The Console displays 2.265 but the value shown in the QuickWatch debugger shows 2.264999999999997. I suspect the conversion of the value to a string on the in the WriteLine method is correcting for the floating point precision error.
When I apply Math.Round(result, 2, MidpointRounding.AwayFromZero), the result comes back as 2.26 not 2.27 as I expected. It seems like it looks at the first number to the right of the digit I want rounded, sees it's a 4 and ignores everything else to the right of it. The problem is that those 9's are only there because of the precision problem and need to be included, or better yet, the value should be 2.265.
What I have done in my code, is to read the text value from the Excel spreadsheet "2.2649999999999997", convert that to a double 2.264999999999999 and then to a string, which gives me "2.265". I then convert that back to a double 2.265 so that I can apply the Math.Round to it and get the expected result of 2.27. Here's the full code:
double result = Convert.ToDouble(((2.01 + 2.52) / 2).ToString());
Console.WriteLine(Math.Round(result, 2, MidpointRounding.AwayFromZero));
Is my approach for floating point precision and rounding issues, relying on ToString to clean it up, the correct one? If not, how should I have done it?
First: The problem is hard. Because 4.53/2 = 2.265. This rounds to 2.27. However the tiniest rounding error in the calculation resulting in a smaller result (2.264999999....) will lead to a rounding to 2.26. This is what is happening here.
To solve this problem you need to have a floating point arithmetic which has the same internal rounding errors as Excel does.
From this document https://en.wikipedia.org/wiki/Numeric_precision_in_Microsoft_Excel it appears as if Excel uses a modified version of IEEE 754, while C# uses IEEE 754. I do not know where the differences are, but it appears as if internally Excel generates different rounding errors.
This document describes the differences: https://support.microsoft.com/en-us/kb/78113/en-us
(For example, Excel does not use denormalized numbers. This implies a different behavior for rounding errors for numbers < 2).
So I assume you cannot solve this using "double"
Update
However, now that I understand that the issue is not the arithmetic, but the way Excel displays the number, maybe this is a solution
Math.Round(Math.Round(result, 3, MidpointRounding.AwayFromZero), 2, MidpointRounding.AwayFromZero)
First round to 3, then to 2 digits. It appears to me that Excel is doing this.
Excel is rounding numbers inconsistently that is causing me issues. When using ROUND(), sometimes it rounds a specific number up, while at other times it rounds the same value down.
I've tried setting Excel to show exact values in settings, but it doesn't change anything.
This is an example of what is happening.
This is the simple formula ROUND((A1-B1)/2,4)
For one record I have the values (.3159 - .3152) which evaluate to .0007 then divide by 2 to get .00035.
For the next record I have the values (.3554 - .3547) which also evaluates to .0007 and divided by 2 results in .00035
So, even though both values are .00035 when I round off to 4 decimal places I am getting .0003 for one and .0004 for another. Same number, rounding to the same number of places, two different results. How can I fix this?
This is an issue with floating point numbers that is inherent and cannot be solved, only avoided.
Try these tests in Excel:
=(0,3159-0,3152)=(0,3554-0,3547) gives you FALSE.
=(0,3159-0,3152)-(0,3554-0,3547) gives you something like 5.55112E-17.
If you cannot accept the differences, you should round already in the middle of the calculation, not only at the end:
=ROUND(0.3159-0.3152,4)=ROUND(0.3554-0.3547,4) is TRUE
=ROUND(0.3159-0.3152,4)-ROUND(0.3554-0.3547,4) is 0
further reading: Is floating point arithmetic stable? and Binary floating point and .NET, by highly regarded Jon Skeet.
A friend of mine discovered a really weird thing in MS Excel. Excel rounds down some specific numbers the wrong way, actually it rounds down a number that shouldn't need rounding.
As far as I have tested, it happens in most versions of MS Excel 2007+
Eg. the number 10358.165790 will be rounded down to 10358.1657899999.
Apparently it only happens in this interval: 8192.165790 - 65535.165790.
It is really weird - it doesn't happen with eg. .165890 or .165690, only with .165790.
Do any of you know why this happens and why it only accounts to certain numbers?
Excel uses an IEEE754 64 bit double precision floating point type to represent numeric data; with some clever formatting and roundup tricks to get sums like 1/3 + 1/3 + 1/3 correct.
What you are observing is a natural consequence of that numeric scheme only being accurate to 15 significant figures. Unless the number happens to be a dyadic rational, in which case it can be stored exactly, the closest representable number is chosen to the one you actually want. This may be below or above a rounding cutoff.
It will occur in other ranges other than the one you cite too.
I have put two date time values in cells A1 and B1. The date time is precisely 41703.0416666667. One is ouput from an SQL database the other manually written.
The result of =if(A1=B1,1,0) is 1.
The result of =MATCH(B1,A1,0) is #N/A.
Does anyone have any theories as to why this may be happening?
Probably an issue with converting decimal to binary. See related answer:
Simple HLOOKUP Failing with Excel 2010
This appears to be a limitation of storing floating point numbers in binary - as described here: http://support.microsoft.com/kb/214118
Many combinations of arithmetic operations on floating-point numbers
in Microsoft Excel and Microsoft Works may produce results that appear
to be incorrect by very small amounts. For example, the equation
=1*(.5-.4-.1) may be evaluated to the quantity (-2.78E-17), or -0.0000000000000000278 instead of 0.
This problem is not unique to excel either but rather a result of:
IEEE 754 specifies that numbers be stored in binary format to reduce
storage requirements and allow the built-in binary arithmetic
instructions that are available on all microprocessors to process the
data in a relatively rapid fashion. However, some numbers that are
simple, nonrepeating decimal numbers are converted into repeating
binary numbers that cannot be stored with perfect accuracy.
The issue is with the floating-point calculation (see http://support.microsoft.com/kb/78113), one possible workaround is to work with the round() function. In your case, rounding to 10 or 12 decimals would probably be enough to address the issue.
I have a database which houses scaled integers, the longest being 10 digits long. I am attempting to convert these to decimal values in Excel, moving the decimal point left by 4 digits, i.e. dividing by 10000.
Given that these integers are currently under the 15-digit significant figure limit, and will remain so, is there a possibility that I can encounter rounding errors?
is there a possibility that I can encounter rounding errors?
Strictly speaking I think yes. For example:
but what may be significant is that the discrepancy as shown (all formatted the same, the smaller black ones created by formula, the red ones by difference of those immediately above) is in the tenth decimal place, so hopefully not a problem.