Major downsides?
None
Minor downsides?
You can't run Windows updates or download anything from Micro$oft in Firefox without jumping through a bunch of hoops that M$ calls "Security features". So keep IE on your PC only to download updates.
Occasionally, you will run across a website that is optimized for IE, and it won't format quite correctly on the screen. This is actually caused because IE uses a few non-standard codes which Firefox and other browsers don't conform to, because the IE formatting is not correct. It's not a problem with Firefox, but a problem with IE. But since IE is the most common browser, some sites have chosen to use poor code to work with the popular browser, becasue M$ won't fix the problem on their end.
You need to install a bunch of plugins (Java and some others) which come preinstalled on IE. I actually like this, as I can choose to ignore things I don't wnat, instead of having M$ tell me what must be installed for my browser to work.