On inactive players, I wait until a new quest line opens up, and then I delete any neighbors that don't respond to requests. I make sure that I send out less than 20 requests a day, and I'll wait a few days after the goal involving requests has completed incase some more trickle in after. I keep my neighbor list to around 20 so it's pretty easy to keep track.
By the way, I don't think my RP has changed in a long time. And I often forget to check my wall. And I haven't rearranged my kingdom in a very long time. And I don't write on my neighbors' walls when I'm only dropping by to pop hearts. I must have given a neighbor or two horrible grief over whether I was active or not, lol! But I actually do visit my neighbors almost every day and I'm an active player most days.