So the Super Mario 3D All Stars is running Mario 64 in an emulator, right, but it's also got this system where it's doing patches at runtime.
This seems to work in two parts:
the 02_UNSMJ3.002.cfg file lists which instructions need patching (complete with comments on why!)

Then there's 06_UNSMJ3.002.lua which contains a HookFunc_AfterInst which is calling into some unseen internal lua code of the emulator, but the function names are very clear:

So it's definitely hacking the font, in multiple places. They can't just swap in a new texture, because there's code in the game that does the texture layout, and it'd be wrong with a bigger font swapped in.

BTW if you wanted to un-invert the cannon aiming? this is where you'd do it.

so now obviously the question is "where are these new textures?" and that seems to be obvious.
There's a subfolder called, uh, Textures, containing "texture_pack.cpio".
CPIO is a super-common format so it's easy to extract.

however the result is a bunch of .rpt files, all of which seem surprisingly small, and I don't know how to interpret these as images yet. Maybe they're not even images? maybe they're pointers to elsewhere?

OK so these files seem to be zlib compressed (not .gz: the header seems to be wrong) 32bit image files, with a small header.

ahh, apparently it's a known format: https://twitter.com/crashoveride953/status/1307139644337905667

so now the question is: where the heck is the font?

we've got the code calling things like "n64HackGetFontS88Address" but that code isn't available in plaintext. it's presumably implemented in one of the precompiled binaries here.

there are a series of .bin files under Message/EUen, but they're too small to be fonts. I think they're pointers to other fonts, but I have no idea how to decode them

so probably the quickest way to find the font would be to just start decoding RPT files until I find it

that is NOT a font

also not a font. I need to dump this tool, actually. It doesn't really support this version of RPT, so it's really just working as a raw image displayer, and I can pull the size from the header. doing that manually will take forever.
doing it automatically will take... let's see

fun fact: there's more than one RPT subformat here.
The first one is only used for 5 images.

that was format 8.
here's adding in format 16.

added in 0x01 and now we've got a lot of font glyphs! huge ones.
god, I hope this isn't the only font...

here's a closeup on one of those.

added in 04 and now we've got the colorful font

added 02 (these types are all very similar or identical so it's easy to add)
and we've got this greenish font

added 0x20 and it's one image and not even one that decoded properly!

ok so it seems my "type" thing is wrong, because the thing I was choosing on seems to be 8 for two different formats.
But here's 08, which has a lot of textures.

including this important one, for the "I baked you a cake" screen.

so here's 20. it seems to be only used for the ending screen! just broken up in to a bunch of tiny pieces.

still haven't fixed that last corrupt one, but it's definitely not a font, so I don't super-care.

which mean this one-texture-per-glyph thing is the only font left... ugh.

which does seem to match: https://twitter.com/AkBKukU/status/1306964769451122688

yep, that's the font.

So there problem now is that I need a clean picture of the background to use as a base.
The one I'm using in the generator now is from the N64, so it's lower res... at least in the textures & pixels of Peach and the word "Peach"

I got THAT one by hacking the N64 rom to not show any text on that screen.
I don't currently have a way to run arbitrary hacked ROMs on a switch... (or a switch) so I can't just do that.

I guess I could cheat: if I get a lossless screenshot of this screen from the game, I can then overlay the original N64 font of the letter, then manually overlay the upscaled "Peach" signature, then do the glyphs on top of that.

ANYWAY in the meantime if you want a Super Mario 64 generator, here's my existing one, which uses the lower-res N64 textures:
http://deathgenerator.com/#m64 

fixed the glitched texture!

it turns out I'm an idiot and my decoding of image sizes was only working because all the images were <=256 pixels wide

I put the code in a gist in case anyone else is trying to DECODE NINTENDO SUPER MARIO 3D ALL-STARS MARIO 64 RPT TEXTURES: https://gist.github.com/foone/469e3c0912a405011bee1e146bf07a55

I don't think it actually helps google find it if you shout, but it can't hurt

Thanks to @AkBKukU I'm much closer to having a good blank

Here's a thread with more info on the emulation from @LuigiBlood: https://twitter.com/LuigiBlood/status/1307359681237651456