Most players I know collect 2-3 Factions pretty casually and one faction “seriously” (ie: owning everything in that one Faction).
That stated, a good game of Infinity “requires” about 10 models (realistically closer to 14-16 for most Factions), and about 20-30 models in a Faction with low-to-no proxying will allow you to field a staggering array of teams that are viable for good games. 50-75 models will usually get you about everything for most Factions, and even the largest usually have less than 150 total (which, admittedly, is a lot, but compared to something like WarMachine, where I have… gosh… 400-500 models per Faction?).
Anyway, moral of the story is that several small factions is absolutely a common way to play! Just keep in mind you will learn a little slower if you don’t stick with one Faction for your first 5-6 months (a lot of the curve is figuring out what your one team can do, and then slowly modifying it to do other things).
For the record, for low model count, smaller Factions your best bets are (in no particular order):
Pan Oceania (Military Orders specifically, or Neoterra)
ALEPH (Steel Phalanx specifically)
Combined Army (Onyx or Morat)
Yu Jing (Invincible Army)
Nomads (whichever sectorial can take the Riot Grrls)
You will struggle a little more with small model count armies if you choose:
(Which isn’t to say you can’t have a kick-ass 10 model army in JSA or whatever, but it is to say that it is more difficult!)