Here's what I use to do. Every 6 months when I see a new thread on deleting inactive neighbors I run to my neighbors list and starting with A-C then D-F (etc), I visit and check for new items, current wall postings and if buildings are at work. If it looks suspiciously inactive I post a nice message on their wall that I'm cleaning up my nbrs list and if they could please let me know if they are still playing. Then after a few days I get distracted and forget what I was doing. Then 6 months later start the whole thing again, notice that the last message on some of my neighbors' walls are 6 months old from me asking if they are still active and know those are the ones to delete.
Now I wait for a goal that requires asking my neighbors for help say 50 requests. I then send out about 30 a day for several days (solely to see who will respond) and make a list of everyone who responds. I give it a week after the goal is over (because responses will still come in after the goal is fulfill) and anyone who did not respond gets deleted. If it's someone I use to be in contact with, etc, I'll visit on them and ask them if they're still playing just to be sure.
Personally, I don't care if someone doesn't visits me regularly or taps hearts or has no social rating. I only care if they respond to requests. So if they aren't responding to those then I just let them go. (That has nothing to do with those neighbors I love and adore and regularly visit me. In those cases their visits are very important :-))