What the ~ operator does is to return the one's complement of the number. It only works for integers. Two's complement is defined by subtraction, but there are other formulae which give the same results.
If you work out the two's complement of
i in n bits, it is 2ⁿ −
i: for 10 as an
int that is
0b1_0000_0000_0000_0000_0000_0000_0000_0000 −
 
0b0000_0000_0000_0000_0000_0000_0000_1010
    0b1111_1111_1111_1111_1111_1111_1111_0110
The one's complement is obtained by inverting every bit. It can be calculated from 2ⁿ − 1 −
i: for 10 as an
int that is
0b1111_1111_1111_1111_1111_1111_1111_1111 −
0b0000_0000_0000_0000_0000_0000_0000_1010
0b1111_1111_1111_1111_1111_1111_1111_0101 which means −11 QED
You can try it for any number: you will see there is an extra − 1 in the formula: so when you take the one's complement of 10 you get −11.
Notice that the formulae for complement arithmetic give different results depending how many bits you have, so
you should always say, “Two's complement of 12345 in 32 bits” or similar. There is also ten's complement but you don't use that in computing.