The way you said it, it sounded like you were talking about adding every new meme to the filter list, which would get real old, real fast.
The additions to the spoiler list are easy to miss for one reason or another. I’ve had to go and take things off at least twice.
What would be more likely: people leaving because all they see is meme, or people leaving because they don’t see any meme?
The memes are a feature, not a bug. Memes are a part of the fandom. A site that includes pony images would have to include the memes.
Leaving because they don’t see memes is unlikely; leaving because they see memes is nonsensical. You don’t leave a grocery store because it has milk. The memes are a part of the product list. Some people are lactose intolerant, sure, but they don’t turn around and walk out if they walk into a milk display first thing through the door.
Not unless they have been having a really bad day, anyway.