Looks like when the pending invites are removed from the games, they will also be removed from the list you have not accepted yet.
Going forward, these now will only remain in your list for 7 days so make sure if you would like someone to be your neighbor that you accept their request within 7 days.