Um, be careful there. You started with simulated violence, but then use *real* physical conflicts (minor ones, as toddlers) as your example. Apples and oranges.
De-sensitization to violence *does* happen. If you are 11 years old, and you regularly see real violence in your home, in your school, and in your community, yes, you get de-sensitized (meaning - you have a decreased emotional response to it). And yes, it seems that de-sensitization correlates with violent behavior as a young adult. But this isn't about kids pulling hair when they are two. This is about seeing people getting threatened, beaten, or shot.
What is much less clear is whether realistically simulated violence in a specific context (like videogames, or movies) has anything like the the same impact on a person's behavior as the multi-context exposure to real people getting hurt around you I described above. And, to be clear - the current rollback on the idea of video games having an impact is based mostly on noting how early studies were seriously flawed, not on further studies that show the effect isn't present. In effect, while lots of folks have opinions, the science-jury is still out on that one.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4539292/