Results 1 to 6 of 6
  1. #1
    Star Lounger
    Join Date
    Jan 2001
    Location
    Brandon, Manitoba, Canada
    Posts
    63
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Incorrect calculation!

    Why is it that when I get access to perform a calculation in a query such as Expres1:=[value1]/100
    I get an answer such as 0.850000023841858 instead of just .85? This is having an effect down the line and resulting in mathematical errors!

  2. #2
    Plutonium Lounger
    Join Date
    Dec 2000
    Location
    Sacramento, California, USA
    Posts
    16,775
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Incorrect calculation!

    You haven't told Access you want to round the number, so it is returning the full value as a double. Look up Round in on-line help.
    Charlotte

  3. #3
    Uranium Lounger
    Join Date
    Jan 2001
    Location
    South Carolina, USA
    Posts
    7,295
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Incorrect calculation!

    For the same reason that you don't get .33 when you divide 1 by 3 in decimal arithmetic. When you convert [value1] and 100 to binary floating point numbers and thendivide them, the answer does not come out to excactly .85 because of rounding errors, it comes out to 0.850000023841858. You can fix this by changing your formula to:

    <pre>Expres1:=Round([value1]/100,2)
    </pre>


    In the above, the 2 is the number of decimal place to which you want to round, in this case 2 to give you .85.
    Legare Coleman

  4. #4
    Star Lounger
    Join Date
    Jan 2001
    Location
    Brandon, Manitoba, Canada
    Posts
    63
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Incorrect calculation!

    Does this mean that if I want 14 point precision and I divide 85 by 100 that I must settle for a wrong value?
    i.e. 0.850000023841858 becomes 0.85000002384186 instead of the actual value of .85000000000000

  5. #5
    Uranium Lounger
    Join Date
    Jan 2001
    Location
    South Carolina, USA
    Posts
    7,295
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Incorrect calculation!

    The value is not wrong, it is correct in the number system used by computers. In decimal, 2/3 to 14 decimals is: .66666666666667. Is that incorrect because the 6 in the 15th place caused the last digit to round up to 7? But, yes, if you want 14 digits, you will have to settle for the rounding error you get when you do deicmal arithmetic in binary floating point.
    Legare Coleman

  6. #6
    3 Star Lounger
    Join Date
    Jan 2001
    Location
    Edmonton, Alberta, Canada
    Posts
    326
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Incorrect calculation!

    Define "wrong"! <img src=/S/smile.gif border=0 width=15 height=15>

    Computers "like" to do arithmetic using the base 2 (i.e. binary) system. That's no big deal for integers; what is an integer in one number system is an integer in any other number system. It DOES matter for floating point (i.e. fractions).

    What is a repeating decimal in one base (e.g. 1/3 = 0.333333.... in base 10) is not necessarily a repeating decimal in another base. 1/3 in base 10 is 1/10 in base 3, which is 0.1 in base 3. 1/10 (or .1) in base 10 is 1/1010 in base 2, which is the repeating binary fraction .00110011....

    That's a long-winded way of saying that you'll get rounding errors in floating point arithmetic on computers even where you might think that a particular fraction can be represented exactly.

    There are often ways around that at a cost of efficiency or number range. For instance, if all you care about is 2 decimal places, you can multiply everything by 100 and work in integers (e.g. store cents rather than dollars). That's basically how the currency type works except that it scales by 10000 rather than 100 to allow 4 decimal places.

    PS Hey! I found somebody else with my initials!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •