Results 1 to 3 of 3

Thread: Rounding Error?

  1. #1
    Lounger
    Join Date
    May 2001
    Location
    Engerland
    Posts
    34
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Rounding Error?

    Hi

    I'm trying to do a fairly simple calculation in Access 97 (SR-2) but am getting a strange result. I am converting text into numbers using the CDbl function, then doing basic addition/subtraction.

    For example I have the following text values that I am converting to cash: "63.81", "11.17" and "74.90". I am then adding the 2 former numbers and subtracting the latter. This should give me 0.08 as a result, however I am getting 0.0799999999999965 - I can format it on screen so that it appears as 0.08 but need to use the underlying figure for calculations so would prefer it to be accurate. Can anyone explain this or am I just being thick? <img src=/S/bash.gif border=0 alt=bash width=35 height=39>

  2. #2
    WS Lounge VIP rory's Avatar
    Join Date
    Dec 2000
    Location
    Burwash, East Sussex, United Kingdom
    Posts
    6,280
    Thanks
    3
    Thanked 191 Times in 177 Posts

    Re: Rounding Error?

    Hi Ade,
    No you're not being thick - VBA is! It's not very good at maths really - if you enter ?63.81+11.17-74.90 in an immediate window you'll get exactly the same result so it's nothing to do with using the CDbl function to convert the strings. MS admits it in <A target="_blank" HREF=http://support.microsoft.com/support/kb/articles/Q165/3/73.ASP>Q165373</A> about Office 97 and as far as I can see, it hasn't been fixed in O2K. They do suggest a workaround though for what it's worth! <img src=/S/grin.gif border=0 alt=grin width=15 height=15>
    Hope that helps.
    Regards,
    Rory

    Microsoft MVP - Excel

  3. #3
    Plutonium Lounger
    Join Date
    Dec 2000
    Location
    Sacramento, California, USA
    Posts
    16,775
    Thanks
    0
    Thanked 1 Time in 1 Post

    Re: Rounding Error?

    The underlying figure is *not* necessarily more accurate. Floating pooint numbers, which is what doubles are, are imprecise. You can get wierd number creep with them using simple addition. Use the CCur conversion instead of CDbl and it will save you a lot of grief. CCur holds 4 decimal places, but the default display is 2.
    Charlotte

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •