When you suddenly have less gems it likely means you have bought something you did not want by clicking too fast. I have been there...done that.
Here are some very good suggestions from Rynetory2

Quote Originally Posted by Rynetory2 View Post
Some tips for those who don't know and falling into traps. Your farm layout is very important to avoid gem traps.

1. Never put fruit trees and growing ash trees next to machines or places where you frequently click. Put these trees to a safe area.
2. Always use bucket to collect from trees and animals.
3. During truffles hunt and mining, always be conscious of your remaining tries to avoid clicking on extending with 15 gems.
4. Place herb garden, bee hive away to places where you are unlikely to click. Same thing for show pony pens, it's 50 gems for speeding up.
5. When cooking or making items, don't do speed clicks in case you don't have the ingredients and end up wasting gems.
6. Always use single click, never double click in FS. It doesn't work like the double click we often do on our mouse.
7. Never play when you are tired or in a hurry. You tend to click super fast.

Hope these will help some of you. I had greatly reduced my gems wastage. So far so good, crossing my fingers lol