Turned out there's a limit of 4 for the storybook gnomes. I'm quite upset since I've wasted my effort without even knowing there's a limit. I checked everywhere and I'm pretty sure they didn't mention this at all anywhere. I only found out after getting my 5th gnome and was forced to sell.