According to the api the addPropertyNode method “Adds a property node to the path the ConstraintViolation will be associated to.
name describes a single property. In particular, dot (.) is not allowed.”
But whatever this value is, it still reads the my.custom.message from ViolationMessages.properties. So, what does this do? When should I use it?
In your example, you specify the message template, which leads to the message being loaded from ViolationMessages.properties. The property node takes the current property path (for the root object that's empty), and appends something to it. Those are two different attributes of the constraint violation.
You can use it to report where in a request a constraint violation occurred. If you have one class with four String properties all annotated with @NotEmpty, and one of them is null, it's the property path that you can use which of the four properties is the actual empty one.