That and I thought there was a rumor that they had some sort of agreement in place when they donated for the current building
Not a rumor and not uncommon at all. You have no idea how many parks, endowments, etc would revert back to descendants if they ceased to be used as designed.
There was a 10 year (or death of Mary, whichever came first) minimum was placed on the building.