"I don't mind if my unit test suite takes a few minutes to run." But do the other people on your team mind? Does your team lead mind? Your manager? If it take a few minutes instead of a few seconds, do you still run the full suite a dozen times a day? At what point do people stop running the tests at all?
We've never gotten to the point where the unit test suite takes too long for us all to run frequently. It's 1-2 minutes for a few thousand tests.
With the "short running" integration tests, after 6-10 minutes we all stopped running the full suite of tests. That was a sign that it needed refactoring and we got down to a more manageable amount.
I learned that a key here is that people don't always complain the tests are slow. Just slowly stop running them.