The term Singleton usually means a static object that has global scope. Or one that is used statically, with code similar to
Calendar cal = Calendar.getInstance();
Using a singleton this way makes it essentially a global wad of static data and methods, and it makes proper
unit testing impossible. Every class that uses any part of the data in the singleton is closely coupled to the detailed implementation. It created implicit dependencies that break the whole concept of unit testing. All of this is separate and distinct from your performance bottleneck concerns.
A much better design is to have a factory return an instance, so that for testing, you can return a mock instance that just addresses the needs of the specific unit under test.