amit punekar

Ranch Hand

Posts: 544

posted 7 years ago

Hi,

In the test code below the output precision varies based on the value of the double used for calculation.

What I really want is basically to shift the decimal places to the left as I am dividing by 100. ( converting % value to a regular value by dividing 100).

In some cases it displays too much precise value and in some cases does the expected.

Regards,

Amit

In the test code below the output precision varies based on the value of the double used for calculation.

What I really want is basically to shift the decimal places to the left as I am dividing by 100. ( converting % value to a regular value by dividing 100).

In some cases it displays too much precise value and in some cases does the expected.

Regards,

Amit

posted 7 years ago

Read number 20 of this.

"Any fool can write code that a computer can understand. Good programmers write code that humans can understand." --- Martin Fowler

Please correct my English.

amit punekar

Ranch Hand

Posts: 544

posted 7 years ago

Thanks for your replies.

What I do not understand why for 96.74 the result has so many decimal places and for 96.78 there are only 4 as expected.

That's precisely what I did not understand?

Yeah,I had read #20 and used BigDecimal to resolve this as below but was trying to find out the issue explained above.

Regards.

Amit

What I do not understand why for 96.74 the result has so many decimal places and for 96.78 there are only 4 as expected.

That's precisely what I did not understand?

Yeah,I had read #20 and used BigDecimal to resolve this as below but was trying to find out the issue explained above.

Regards.

Amit

amit punekar

Ranch Hand

Posts: 544

posted 7 years ago

Thanks once again for replies,

I went through the beginner's article but then could not understand why it is not applied for all double values. why the result varies for one double value(96.74) than the other (96.78) ?

I might be missing some link due to lack of knowledge on this topic, between the article and the actual scenario which I posted earlier?

If someone can point me to the missing link it would be great.

Regards,

Amit

I went through the beginner's article but then could not understand why it is not applied for all double values. why the result varies for one double value(96.74) than the other (96.78) ?

I might be missing some link due to lack of knowledge on this topic, between the article and the actual scenario which I posted earlier?

If someone can point me to the missing link it would be great.

Regards,

Amit

W. Joe Smith

Ranch Hand

Posts: 710

posted 7 years ago

You aren't missing a link or anything. It is just the nature of using base 2 as opposed to base 10. I'm not a math genius so I can't explain the nitty gritty technical details, but basically it boils down to the fact that some numbers can be calculated more precisly in base 2 than others.

I believe (and any super math smart people will probably say I am wrong) that 1/4 vs. 1/3 is a good example. Put in decimal form, 1/4 is .25. That is much easier to translate into base 2 than 1/3, which translates into .33333333333333333...

I believe (and any super math smart people will probably say I am wrong) that 1/4 vs. 1/3 is a good example. Put in decimal form, 1/4 is .25. That is much easier to translate into base 2 than 1/3, which translates into .33333333333333333...

SCJA

When I die, I want people to look at me and say "Yeah, he might have been crazy, but that was one zarkin frood that knew where his towel was."

posted 7 years ago

Yes, 1/3 in base 10 is a good example.

In representing 1/3 as a base-10 decimal, we might start with 0.3, which is 3/10. This is an

Now suppose we had a

Potential loss of precision in a computer is similar, using a base 2 system subject to the bit limits of floats or doubles.

In representing 1/3 as a base-10 decimal, we might start with 0.3, which is 3/10. This is an

**approximation**of 1/3. If we wanted to be more precise, we could add another digit, making it 0.33, which is 33/100. This is a better approximation (more precise), but still not exact. So we continue adding digits, getting closer and closer to 1/3 (adding precision), but never quite reaching it an exact representation.Now suppose we had a

**limit**on the number of digits we can use. Say, for example, we could not use any more than 4 digits to the right of the decimal. So when we try to represent 1/3, the best precision we can get is 3333/10000, which is not equal to 1/3. On the other hand, we have no problem representing 1/4 exactly as 0.25 because it fits within that limit.Potential loss of precision in a computer is similar, using a base 2 system subject to the bit limits of floats or doubles.

"We're kind of on the level of crossword puzzle writers... And no one ever goes to them and gives them an award." *~Joe Strummer*

sscce.org