Should I even date him if he thinks I’m unhappy

When I’m with my boyfriend we usually have a lot of fun but sometimes he all sad and weird and whenever I ask him why he says he feels like I don’t have fun when I’m with him. Literally the only reason I don’t have fun every once in a while is because I look over at him and he’s being all mopy and quiet.