How to get rid of variable references in the document?

I am working on a plugin that swaps remote variables with local ones. After running few tests I got this

Bunch of duplicates that do not affect anything. How do I get rid of them?

Here is the code I made - source-foundation/src/utils/swap-variables.ts at main · namad/source-foundation · GitHub

It does the following

  1. Collect all layers within the selection with Object.keys(n.boundVariables).length > 0
  2. Then I iterate through these layers and check which variables are bound to which props
  3. I retrieve bound variable details with figma.variables.getVariableById(varId) and figma.variables.getVariableCollectionById(collectionId)
  4. I check against local collections if the collection with the same name exists and has a variable with the same name in it, here is the code I use source-foundation/src/utils/figma-variables.ts at main · namad/source-foundation · GitHub
  5. If there is a full match I use figma.variables.setBoundVariableForPaint(), figma.variables.setBoundVariableForEffect() and node.setBoundVariable() to bind local variables instead of what is bound to the layer originally

It works just fine but after each run, I can see these ghost collections added to the page modes

Video of what’s going on