Using some of the number/math features under Type Settings

For extra context: I found this help page useful, but it doesn’t explain the Calculator setting at all.

Most of the features in the Details panel (basically everything below Stylistic Sets) are defined by the font, not by Figma, so we don’t have any kind of documentation on these because they’re totally arbitrary. (We’ve seen typos in these from time to time which people have flagged — these seem like typos in the Figma UI but actually we can’t control these as we read directly from the font file!) The “Calculator” stylistic set is defined by SF Pro.

I can’t find any documentation on SF Pro’s stylistic sets (some fonts provide documentation on stylistic sets and other OpenType features — see for example this for Inter or Hoefler & Co provides really nice documentation for their fonts). But I dug into the font file for SF Pro to see which glyphs this feature is defined for and it looks like this feature is pretty limited. As far as I can tell, (at this moment/as of the current SF Pro version I have), here are all the sequences you can use in this feature (sequences that ligate into a different character when you turn on this set):

√x
x√y
y√x
1/x
+/−
∛x

I’m not sure if there are any fonts available that do what you want, although there are several Figma plugins that render LaTeX to insert into a Figma document. Not sure if you’re familiar with LaTeX (or how much complexity you’re looking for), but you should be able do use something like Latex Editor to insert \sqrt{1+x} to achieve what you’re talking about.