I would say you should always try them on and see what works for you. The same principles apply to choosing neutral climbing shoes as they do to choosing aggressive climbing shoes - try them on and feel what feels best for you.

I would keep in mind the intended usage of the shoes though, since if you are getting neutral shoes, you may be climbing longer routes such as in sport and trad climbing, in which case, downsizing to something very tight may feel very painful and uncomfortable due to wearing the shoes for so long.