Rock climbers wear smaller shoes because it gives them better support for their feet while climbing. Wearing smaller shoes allows you to be more precise with your feet, stand on smaller foot holds, and exert much more force from your toes compared to wearing larger shoes.
Climbing shoes are also designed to be very minimalistic in terms of materials and "fluff". Meaning, they have very little rubber around them in comparison to everyday sport shoes. This allows your toes to actually feel what you are standing on, and, in a way, to grip the holds with your toes. This minimalistic approach also makes the shoes seem much smaller than they are when comparing them with street shoes or general sport shoes.
It is important to note, though, that beginner climbers should probably not try and downsize their climbing shoes too much, or at all. Wearing smaller climbing shoes, which is known as downsizing, provides extra benefits in performance, but, these benefits are not really noticeable for beginners, since the routes beginners climb do not actually require or utilize these performance boosts.
