Yes. I used to think no but changed my mind. There are two reasons for changing my mind
1) It is an unused resource most of the time. Although it is rare and the only option available to someone who needs it, I don’t see a big problem if someone used it when it is not needed.
2) Some people have hidden disabilities and need those bathrooms. There is no way for me to tell if someone really needs it or not.