dontwatchtv

Just speculating. The Pyramids of Egypt were once perfectly smooth. But, nearby villagers would go and chip away stones from the Pyramids to take and build their own homes.

Or like the Talibans and the Wahhabis(Saudi) destroy any building that are considered buildings that honor false idols.

In Greece, the stone temples had wooden roofs that simply rotted away over time.

As far as Peru, I'm guessing that the stones from ancient sites were easy pickings for building materials for local villagers.