If they always have buildings with hearts (sure sign they're active), but don't visit/help/gift anyone (always 0 SR hearts), EVER, I remove them after a month.

I practice what I preach: I visit/help all my neighbors, almost daily, and rotate gifts (A-Z) to all with 3/4 SR hearts. I don't require proof (posting on my wall or notices on my news feed). Their SR rating is proof enough that they're active AND good neighbors