An Avengers sequel needs to learn from the X-men movies mistakes and avoid "insert new character here" syndrome. The movie established the team. Let that be the team.
The more characters that are introduced, the harder it gets to provide adequate story/screen time for each character (see introduction of non-essential new characters in Star Trek films for examples). The actors' contracts are formed around # of movie appearances. So if, for example, Thor gets short-shrift in Avengers 2 to make way for Vision, the actor is done after Thor 3 & Avengers 3 (barring a new contract). Also, unlike in comics, the actors will age and the longer the span of time between movies, the less likely you'll see them. If Ruffalo appears in Iron Man 3, that counts against his movie appearances so while he'll be in Avengers 2 & 3, it hedges against him having standalone Hulk movies.
If Avengers 2 or Avengers 3 has a completely different roster, they're going to derail the franchise. X3 had a weak-ass plot but it sure had a lot of throwaway appearances, didn't it?
I think it's far more likely you'll see characters like Wasp, Ultron, & Vision in Ant-Man (or its potential sequels) than in the Avengers. Ant Man hasn't really been an A-Lister that can carry his own series in the comics and if you're going to do an Ant Man movie, you're going to need to have some classic Pym storylines from the Avengers to do it well.
They could use Ant Man as the lead-in to Avengers 2 (or 3) if they wanted to go the Ultron route. However, I still believe additional heroes on the roster = diminshed story/screen time for the other heroes and/or the villain, which is big negative, IMO.
I put even odds that a Thanos conflict might be a build-up storyline that would culminate in Avengers 3 as the capstone "6th movie" for Iron Man, Thor, & Cap. Since Whedon thinks an Avengers 2 would be better served as smaller scale & more character focused, it's hard to see Thanos as an Avengers 2 BBEG, but we'll have to see who's writing/directing it...
Finally, can we take the Whedon/Buffy/WB vs UPN discussion to a new/forked thread, already? Sheesh.