So what is this problem called?
I refer to them as the problems of "transparent persistence" and "normalization".
What theory describes the possible solutions?
I'm not sure what theories, but database implementations address these issues in practice. Your were describing Magma exactly up until the "loosen our definition" paragraph, however, there's no reason the other program could not be coded to access the Magma DB or files directly -- the complete meta-definitions are included as part of the connection process.
Are there classes of solutions that have similar trade-offs? Where can I learn more?
But, obviously you know about ODBMS's, so probably this isn't the answer you're looking for.