Results 1 to 2 of 2
  1. #1
    New Lounger
    Join Date
    Apr 2003
    Location
    Dover, New Jersey, USA
    Posts
    6
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Visor calculator error

    Anybody else find an error like this?

    Using the advanced calculator on my Visor (2 Mb), in both float mode and fixed mode with 13 decimal places or higher, if I do a calculation like 3250-3242.82 I get 7.1799999999998 instead of 7.18. Is this a known error in the CPU? Any fixes? I've also reproduced this on a Prism.

  2. #2
    Lounger
    Join Date
    Jan 2001
    Location
    Minneapolis, MN
    Posts
    38
    Thanks
    0
    Thanked 0 Times in 0 Posts

    Re: Visor calculator error

    (Yes, I know this is ancient.)

    This may be a display problem, but it isn't an error.

    The binary equivalent of one-tenth is an infinite repeating number - unless you code around it, you're going to end up with an answer such as you described. There are a variety of ways to code around it - but I doubt that anyone has bothered for PalmOS, it takes a chunk of code.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •