Actually found a thread with a good solution in the link.
https://forum.figma.com/t/why-are-there-ghost-components-in-the-used-in-this-file-tray/3038/13
I would add that to aid this process; if you have a component in the “used” section that is named the same as the local version, change the name of the local master, publish the change to the library,and then it makes it much easier to find and replace the bad assets.