> 1. Adding custom control to palette?
No.
A custom control palette is not supported in SceneBuilder 1.x, maybe in SceneBuilder 2.x or later such a feature might be implemented.
DTL-5160 extension or plugin support DTL-5348 Using third party components on Scene Builder
> 2. Reading custom file format
No.
SceneBuilder only supports the FXML file format.
If you wanted to work with a different file format, you would have to create an external tool which converted that format to FXML so that SceneBuilder can work with it, then, after the FXML is updated and save in SceneBuilder, run the external tool again to convert the FXML file updated by SceneBuilder back to your required format.