Here's my 2 cents on spending gems (probably not even worth 1 cent though). I won't buy any because of the gem traps, I've personally have lost at least 100. So gems are very valuable to me. I would never use them to speed things up (on purpose anyways). In order of importance: 1 wizard school 2 second wizard school 3 fruit trees 4 save for future updates that are gem only and/or purchasing gem only items such as green arches and some pink animals i missed once I've accomplished all goals and expansions.

Since you already have two schools I would go for the fruit trees, very helpful with several goals and the royal exchange. I'm working on getting two of each as I never have enough, especially apples.