I disagree about the hobby being silly. TTRPGs are grounded in the story-telling tradition that is as old as Mankind.
However, like other forms of literary expression, individual components and concepts can be, and are, childish.
But as you have noted, the choice at the table level, which is...