Again, it depends on what you're analyzing and how you've got your data. If you have widely skewed data, a mean value may be significantly different from a median value. So you'd figure out which gives you a more useful value and whether or not you could toss out the outliers, none of which you have to handle symmetrically. For example, if you were looking at the average income of a cohort of graduates from a college english department, you might consider omitting the outlier who won the lottery or became a professional basketball player since they're definitely an oddball and would skew the usefulness of the data (to the prospective student - though not, perhaps, to the recruiter - and thus we have the potential to lie with true statistics).
In the case above, the outliers at 1-3 (notice how few of them there are and that there's a gap between them and the much larger cluster of responses) have relatively little impact on any analysis of mean or median. But if there was a big spike at 1, you might suspect review bombing and ignore the whole bunch.