It's an issue of branding. Certain phrases, expressions, or terms are used to reference some other specific thing often enough that they become shorthand for that thing in the public eye. Somehow "4:20" became code for smoking marijuana, despite the fact that it's just a time on the clock. So if you hear someone say "Let's have a meeting at 4:20," it carries the implication that the "meeting" is to go light up.
This doesn't necessarily have to be the case, of course. There might very well be an actual meeting going on at 4:20 that has nothing to do with weed, the same way that there can be an issue of states' rights that has nothing to do with institutional racism, or issues of men's rights that have nothing to do with misogyny, but the use of those phrases as shorthand for the other thing is now widely understood, and so using that term will invariably invoke that thing in the minds of other people, rightly or wrongly.