This is a rootsum square computation: the program takes 1024 signed integers, converts each integer to float, computes the sum of the squares, takes the square root of the result and inverts it. That value is used to scale all of the original data, which is then converted back to signed integers. The data is in .wav file PCM format and is interpreted as being between 1 and 1, so there is a scale factor applied when converting to and from floating point. The java program is using the double data type (64 bit floating point format).
The boundary case is simply setting all of the original integers equal to the smallest positive value, 0x00000001.
Here is the key fragment of code from the java app:
Thanks for your help!
 1
SCJP 1.4  SCJP 6  SCWCD 5  OCEEJBD 6  OCEJPAD 6
How To Ask Questions How To Answer Questions
Lynn Oliver wrote:
This is a rootsum square computation: the program takes 1024 signed integers, converts each integer to float, computes the sum of the squares, takes the square root of the result and inverts it. That value is used to scale all of the original data, which is then converted back to signed integers.
Since I don't have the full context in which you are performing this calculation I could be wrong but I'm not sure this makes sense. I would expect one to scale using the RMS (root mean square) and not the RSS.
Retired horse trader.
Note: doubleunderline links may be advertisements automatically added by this site and are probably not endorsed by me.
In your case, what is your expected result and why do you think the results you're getting are wrong? If I'm scaling a minimum value, in general I'd expect something like the minimum value back, however much I'm scaling it by. If it's out by 1, I wouldn't be worried because we're dealing with minimum values, so the actual difference is insignificant. If you're talking about a sound recording, think about the inaccuracy for minimum values that would have been introduced when it was encoded originally: you're trying to be accurate about some data that wasn't accurate to start with. So using BigDecimals to try to increase accuracy would be pointless.
Python might be using something like a BigDecimal automatically, but it will be, in comparison with Java, very slow.
To put it another way, obviously you'd expect some inaccuracy to enter when you convert from a double to an integer. If you have a value of 2.6, what integer are you expecting? Whether it's 3 or 2, you're going to be "inaccurate" by a large percentage. But this "inaccuracy" will only be a large percentage when dealing with values close to the minimum (i.e. 1).
Incidentally, if you want to round to the nearest integer, rather than rounding down, all you have to do is to add 0.5 before converting to an integer. (Or subtract 0.5 for negative numbers.) This would probably be more appropriate for a sound recording than always rounding down.
Instead of multiplying each integer with the inverse of the RSS, you can also divide each integer by the RSS.
In general, when working with floating points, avoid unnecessary operations to avoid loss of information.
The mind is a strange and wonderful thing. I'm not sure that it will ever be able to figure itself out, everything else, maybe. From the atom to the universe, everything, except itself.
The java app came up with 0x02D413CD or 47453133. With the test data set equal to 1, the exact result is the value of factor, or 0x40000000.
Unfortunately, I do not have the ability to modify the java app as I do not have the source for it. So, no way to improve the algorithm.
The exercise was to determine if the python version is at least as accurate as the java app. The observed differences using real data (average difference 0.05%, max difference 0.10%) were small enough to be insignificant in terms of the audio data, but I wanted to get a feel for where they are coming from. Using the decimal library in python with 90 significant digits is massive overkill, but I plan to use those results as a baseline for comparison with both the java app and the python app.
I know that people are more familiar with RMS computations, but in this application root sum square is correct.
Retired horse trader.
Note: doubleunderline links may be advertisements automatically added by this site and are probably not endorsed by me.
Oh the stink of it! Smell my tiny ad!
The WEB SERVICES and JAXRS Course
https://coderanch.com/t/690789/WEBSERVICESJAXRS
