Hey @Tushti_Sachdeva, sorry for the trouble and confusion!
One thing to keep in mind is that Figma loads layers on pages that you open, so navigating across pages requires loading any remaining content on those pages, which can result in memory and number of layers increasing in the memory panel.
There are a number of factors that contribute to memory, including hidden layers, large component or variant libraries, as well as multiple large images, pages, and assets.
More info and tips on reducing your memory usage is available here:
Figma Learn - Help Center
I have the same problem, i have a file with large sets of vectors and layers. Flatenning layers seemed to work well at first, then it just did not matter. Now the memory keeps going up no matter what i do. Text layers taking 2%, simpled vectors 1%