There are three of them, each dedicated respectively to the three principles of Truth, Love and Courage. Each altar room has four entrances, from the four dungeons representing the anti-forces of the virtues that the principle creates. For example, since Honor comes from Truth and Courage, the dungeon Shame has access to these two altar rooms. The dungeon Hythloth, opposing Spirituality, has access to all three of them. The connections are more clear in the diagram below.
In Ultima IV the Stranger had to enter the three altar rooms and use the fitting Virtue Stones on them in order to get all pieces of the Key of Three Parts. After that time, the altar rooms weren't seen again.