Hi,
I think you are confusing the developer and the end user.
In your logic you'll have to cleanup a reference you estimate not being used anymore, and that will have no penalty like garbagecollection for Unity Objects. So this is fine, it's like many many aspects of pur c# coding, if you are not clean, it will get dirty...
delegates not being removed, leaked objects here and there, etc etc.
I have extensive experience with XmMaker on many projects which uses soft references in static hashtables, and the problem of memory isn't there, it's not like you are going to constantly create new references, once you have your set, you'll reuse them, assign new values to it.
I never had the following for example: dozens of ennemies each requiring an xml ref that would lead to a memoryleak as they get destroyed if reference not cleaned up. But still within XmlMaker it would not make sense, your enemies could be categorized and you would only need common xml data for certain ennemies, I don't see any use case for a unique xml set of data per instance, it doesn't make sense, xml is for easily storing data for configuration and saving data. not for live data as the game is being executed.
in the best scenario for the worse case mentionned above, you should simply have a "ClearReference" and "ClearAllReferences" action for your framework and use that when the user is going back to the main menu, or switching from one level to another, if indeed you create uniquely references per level ( which I doubt is making sense, you should try organize your data so that it's generic for all levels).
Bye,
Jean