In ancient times, the island was struck by meteorites destroying its primary city, and later forgotten. It was rediscovered in Ultima III when, on the advice of a scant few citizens making outrageous claims, the Stranger's party steered their vessel into a great whirlpool, washing up on the shores of the lost island. Among other things, the Four Cards that were ultimately used to destroy Exodus were discovered on Ambrosia at the shrines. The island vanished again when the land was reformed in the great cataclysm following Exodus' death.
Ambrosia was again found by the Avatar in Ultima VII, while searching for the caddellite meteor. The island was then barren, but used by outlaws to hide from the authorities. Apart from the meteor, and the hideouts of the outlaws, there wasn't much to see on the island.
Although I, myself, failed in my search for the Lost Continent, I bring the story oft told by the minstrels who roam the land, in hope that it will aid in thy quest.
"Many, many years ago, there lived a strong and sensitive people who inhabited the island-continent of Ambrosia. There people developed great powers over the forced of nature, and 'tis rumoured that even the power to change one's physical being rested within the realms of their knowledge. From these stories grew the legends of the Magic Shrines of Ambrosia.
"The Island prospered as the years passed, and the strange power grew in strength.
"Then, without warning, there was a great and violent upheavel, and the Earth sank suddenly into the oceans. A great whirlpool pulled the Island beneath the depths of the sea. The whirlpool exists yet, and many claim it to be the path to lost Ambrosia."
--Sentri the Seeker