I have finally admitted to myself that I can't travel with friends/family while being a tourist AND do any kind of serious photography. So many trips I've taken and carried a whole kit full of gear and ended up cursing it and (depending on the locale) worrying about it. Once I admitted that to myself, it took a lot of pressure off. I also try to avoid making elaborate excuses to myself that I need to buy a really good compact camera since it inevitably leads to an expensive decision that rarely lives up to my imagination. So while it sounds a little feeble, I think I could get by with my iPhone (under the travel-with-family description).
Maybe I'm strange (no maybe about it) but I find photography to be a solitary endeavor. When with family and friends, there are too many other distractions, like food, drink, and just enjoying their company. It always feels odd to tell the group "you go on ahead, I'm going to hang out here and take some photographs. I'll catch up with you." I've done it, but rarely with any success.
On the other hand, I nearly always head out with only a body and one lens. I find myself a lot more comfortable adapting to the lens than carrying a lot of stuff and not using it. But then, most of my excursions are for personal use, a professional has to be prepared and is more likely to always take the full kit.