When to take and when not to take barbs is an important element of strategy in TW2. I've taken every barb in my home province to prevent anyone from getting a foothold there: if they cant take a barb to build a church, they can only send 1 full strength attack at me at a time. If they try to take a abandoned village, they have no defense against counterattacks. And when attacking, its a good idea to take a barb in an enemy province, defend it, and build a church before attacking. Instead of complaining that people are taking barbs near you, crush them and continue farming. Or just shift to building cavalry and farm further out if you can't kill them for some reason. Or just don't let them noble "your" barbs: station defenders in what you consider your property, remove them right before your farmers hit, then replace them afterwards.