Exact same problem for me.
I did notice that the Castle 'jumps' from the right to the left of the new squares when you complete the bridge.
I tried storing my vineyard to see if moving the castle onto another island helped but now can't move it back so have temporarily lost my vineyard and it's 240 people.