You can potentially earn gems from Piper's Treehouse. Ranging 2-3 gems if you get lucky and pop a balloon that has them.
As well as "Achievement" blue house.
For each level you complete, you get 1 for level one completion, 2 for level two completion, and 3 for level three completion, I believe
As well as clicking on a Mole, Butterly, or Otter that pops up on your farm, though that might be on rare occasion. I've gotten 1 gem from clicking on them on some occasions.
Also, you can sometimes get 2 gems from mining an "Ore Vein"