Listening to the Dark Interlopers episode of Book of Mudora, the episode where they try to figure out which race/faction became the Twili, and Idk why that even has to be the case. They were a bunch of mages we never heard of before that adapted to the Twilight realm.

Like, I think they're just descendants of Hylian magic users who pissed off the gods and got banished. I don't understand why they have to be a distinct people in their entirety that got wholesale banished.

Honestly, I got the impression the banishment happened a Really long time ago, it explicitly mentions this happens in Hyrule, so most likely after Skyward Sword and before Ocarina of time.

Heck, right there in the cutscene, the Interlopers are represented by Link, a Hylian.

The Spirit warns Link of the corrupting potential of the Magic, because Hylians, his people, were the interlopers.

Heck I guess consider this thread a Book Of Mudora question, @ArcaneCrystal

Another thing, Hylians become Twilight beasts and gain physical form in the twilight realm. We hear about it happen, the Interlopers did the same thing and became the Twili, they just didnt become monsters cause they were good at magic