What would make you leave a fandom, or prevent you from getting into it in the first place?
Too much toxicity and negativity.
I've left some fandoms for this exact reason. Sometimes I can still enjoy something even if I distance myself from its fanbase and more likely than not I can remain blissfully unaware of whatever fandom drama is happening, but other times such negativity can unfortunately be inescapable. All fandoms have a toxic side to them, and some can be louder than others, and it is more or less subjective depending on the platform you use and who you follow, and sometimes you need to reevaluate your own involvement in said fandom. In general, fandom is supposed to be fun, if you're not longer having fun, no matter what the cause, then you need to take a step back. Taking a breather can be good, and it can allow yourself to gain some perspective, regroup, and rethink if/when you jump back into the fandom again.
(Also, as something I've learned overtime, you can still be fannish about something without ever being involved with its fandom. It doesn't make you less of a fan or less enthusiastic about fanfic/fanart/meta/etc if you're not deeply involved or hardly interact with the fandom community.)
When it comes to the piece of media itself, if I'm no longer enjoying it I just stop consuming it altogether. I just don't have the time or the patience for something that I'm unhappy with.