I lived in a house for 9 years which was frequently habitated by rats and mice. Our neighborhood had a rat-sized underground cavern system which spanned for blocks. It was futile trying to get rid of them.
However, you can keep them out of your house. Wherever you find an opening, before you re-cover and re-paint, plug the hole with steel wool. If your house is made out of dry-wall, there's no way a mouse is "stuck" there. They can eat their way through it. They can't eat their way through steel wool.
Also, judging by the amount of rodent bodies we found when we renovated and tore down walls and ceilings, dead rodents don't really stink up a place. However, they are flammible and a fire hazard. Live ones too. Sometimes they bite through electrical wires, they get shocked with voltage, and catch on fire. That does smell really bad, and also has the unfortunate side effect of catching your house on fire. And if you live in a row-house, it may also set your entire block on fire.
What you need to do is set traps on the outside of your house, and plug the holes inside your house with steel wool. Make sure you check EVERYWHERE for holes. If you have an unfinished basement, that's a key point of entry. Also, check all of your kitchen and bathroom cabinets. Mice can chew through those too and get easy access to your kitchen.
If you have children, do NOT use rat poison. It's too easy for something to happen to your little one.
That's all I can think of now...
Oh, you'll know for sure you have a mouse if you find pellets in your house. Otherwise it could be the pipes, like Zander said.