There are different speculations as to what actually makes them grow. What is important is the world you are on because it could potentially have different settings altogether.
The general consensus on Alnwick is that the more you farm them, the more they grow.
The general consensus on Bastille is that the more you farm them, the less they grow.
The other two worlds for US I am unsure what they think makes them grow over there because I've never played those worlds.
I know that in TW1 it was more or less just whatever the server decided the barbs would be. The barbs in TW1 would grow to about 2500 then shrink down to 600 and then go back up to 2500. So if we assume they borrowed from TW1, it's just an algorithm that we can't see and can barely influence. This is where the concept of "barb shaping" comes in. When you spy the buildings of a barb and you know their point cap, you can kind of manipulate the sequence in which it builds. If you cat down the buildings that it doesn't need for resources, there is a chance that the algorithm will decide to build more resource buildings. Then let's say you want to noble the barb, you can then bring the barb down from the max barb cap and try to get it to build the HQ, barracks, and other buildings for you. Thus, you are shaping the barb how you need it to be in your area.
I hope this helps. I kind of rambled a bit and I am sorry for that.