According to news.com.au, a sheet of scaffolding in the building caught alight and collapsed onto Pitt Street and Alfred Street near Circular Quay at about 8.40am (local time).
Everyone in the building, which was under construction, have been evacuated. There are seven crews at present on scene to fight the fire.
"I saw black smoke everywhere when I got off my train at Circular Quay. It was a really strange thing to see first thing in the morning," news.com.au quoted an eye-witness as saying.