It’s because your game hasn’t synced yet.
Whenever you play you need to wait a little before you start collecting - otherwise it’ll roll back to the sync point.
You can visit a neighbour or go into the Baron’s Tent first and this will sync the game if you’re unsure if you’ve synced yet.
Alternatively if you hard close the game when you leave, every time you start up you’ll be synced.
Hope this helps.