I have not been able to upgrade my chateau yet. I have completed the requirements twice now and it still keeps just dropping a single hat and will not give me the screen to upgrade.
Are you on Goal 5 yet? This is when upgrade option should show up. 2 of each item needed to upgrade.
Along with 4 fencing swords.