Unless you have a goal to ask you to upgrade, then it's really just personal preference whether you upgrade or not. I have issues with how much I can have on my board, so for me, upgrading buildings to get additional RP make it easier to expand as I just put on my board what I need to unlock a plot, then put it back in storage once the plot is open. So, upgrading at least 1 of every royal building is really helpful for me. For others, upgrading is a personal goal and given collection time may also decrease, that can make a difference as well in upgrading. The mines and sawmills drop ready made stone blocks and beams, and yes some town classification building drop crops. Anything that has multi drops is beneficial. At the end of the day though, it's optional in most cases to upgrade, so just let it be a part of your game play style.