How many players can do boss challenges?

Four is the ceiling everywhere in Aniimo's co-op, not just here. Cooperative parties cap at four players, and open-world co-op runs at the same size whether you are exploring, working through puzzles, or fighting bosses. So there is no larger raid-style group to graduate into once you know the fight. The group you use for everything else is the group you bring to a boss challenge.

Matchmaking is the part that matters day to day. You do not need three people lined up in advance to attempt an event boss. Queue for it and the game fills the seats. That applies to the weekly boss damage challenge too, which is built around four-player teams and has matchmaking attached, so the weekly is reachable even if nobody you know is online.

The duo option is worth keeping in mind when matchmaking is not what you want. Two players can take a boss challenge event on directly. That is fewer bodies on the field, but each player still fields an active combat team of up to four Aniimo and can swap between them mid-fight, so a pair is not as thin as it sounds. The coverage you would normally spread across four players has to come out of your own team instead, which means going in as a duo is a question of how well your four Aniimo cover each other rather than how many friends you have.

Not settled: How a boss challenge adjusts when you enter as two rather than four, including whether the weekly damage target moves with the group size, is not settled.