I would wear gloves not only because of germs but because of the chemicals you may come into contact with too. For just an average person cleaning their house here and there, they can probably come into the contact with cleaning chemicals and remain relatively healthy. Though, if you are cleaning a couple hours a day and you come into contact with those cleaning chemicals frequently, then they can effect your health.
If the labels on your chemicals say to wash immediately after contact with skin, then as a cleaner you should probably be wearing gloves while using it. Be sure to read the labels.
As for viruses and other things, some germs can remain alive on non-porous surfaces for up to a day or two. You are going to come into contact with those things before you even start cleaning. How many times do you rub your eyes, wipe your nose, touch your mouth, or touch any other mucous membrane? Quite frequenty I imagine. That is all it really takes to get infected with a virus that may have found its way onto your hand. Gloves are good in this respect as you normally don't want to touch your face while wearing gloves.
You may want to bring some tissue paper or a dry towel around with you if you ever need to wipe your face or anything.