If a game isn’t fun I don’t generally keep at it. It depends on why it isn’t fun–a lot can change as you learn the landscape of a game but it has to have something else going for it for that to be worthwhile for me.
But there are always people interested in trying a game or buying it used or I can give it away. There are plenty of games I enjoy or find interesting right from the start so I try not to over-invest. I’m not immune to investment bias, by any means, but I do try to think of it the other way around–I already shelled out the money. It’s only getting more expensive if I also shell out my time, so it better be worth it.
The trouble for me is when I want to like the game not because of hype or because I bought it but because of some subset of the experiences the game provides. There’s this one mechanic or this one feeling or this one bit of narrative that worms its way into my psyche and I just want to enjoy that without everything else getting in the way … and sometimes that’s just not going to happen. That’s when I tend to spend too much time with something be it board game, video game, book, TV show, whatever. Though it tends to happen least with board games as they require other people to be right there with me.