• Post Reply Bookmark Topic Watch Topic
  • New Topic

1f/10 != 1d/10  RSS feed

 
Arnold Reuser
Ranch Hand
Posts: 196
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
I don't understand the following :
1f/10 == 1d/10 results in false
1d/10 == 0.1 results in true
I tried to read IEEE754, but that doesn't help me a lot.
Can someone explain me why this behaviour is correct?
 
Jim Yingst
Wanderer
Sheriff
Posts: 18671
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
Basically, 1/10 in binary ends up as an infinitely repeating series of digits. Like 1/3 = 0.333333... in decimal. If you had two decimal formats, one with 4 digits and one with 4, you couldn't expect 1/3 to be the same in both formats, because one is 0.3333 and the other is 0.33333333. It's the same thing with 1/10 in decimal. Basically when you use floatng point, you can't really trust the == operator as the numbers involved may be slightly rounded up or down by various effects.
 
Arnold Reuser
Ranch Hand
Posts: 196
  • Mark post as helpful
  • send pies
  • Quote
  • Report post to moderator
thnx.
 
Consider Paul's rocket mass heater.
  • Post Reply Bookmark Topic Watch Topic
  • New Topic
Boost this thread!