Yemeni pro-government forces withdrew from Perim (also known as Mayun) on Friday.
Eyewitnesses and news agencies reported that Houthi forces had moved toward the island.
The Houthis later issued a statement saying their military operation to oust the Saudi-backed forces was a “success” and that “maritime shipping is safe for all companies except Saudi vessels.”
Houthi military spokesman Yahya Sarea said the “large-scale” military operation was launched on September 3 due to Saudi Arabia’s “blatant aggression” against “our dear people.”
Sarea also said the Houthis had expelled “Saudi enemy” forces from six districts in the Taiz and Hodeidah regions, saying hundreds of troops had been killed, wounded or captured.
However, the Houthi statement does not mention Perim Island.
Saudi Arabia has not publicly commented on the latest developments.
Perim Island is located at the narrowest point of the Bab al-Mandab Strait, which has become increasingly important following the virtual closure of the Strait of Hormuz, another important waterway, in the conflict between the United States and Iran.
Saudi Arabia relies on the Red Sea and the Bab al-Mandab Strait to deliver oil to customers in Asia.
The escalating conflict between the Houthis, the Yemeni government and Saudi Arabia has contributed to rising oil prices.
Hundreds of people have been reported killed and thousands displaced in Yemen since the Houthis launched an offensive in the country’s southwest a week ago.
The Saudi-led coalition has also carried out air strikes on Houthi-controlled areas in western Yemen in support of pro-government forces, and the Houthis have launched missiles and drones at cities and oil facilities in southern Saudi Arabia.
At least 46,000 people have been forced to flee their homes in Yemen since fighting escalated, according to the International Organization for Migration (IOM).
The IOM said Friday that the number is “growing alarmingly by the hour.”
Amy Pope, IOM director-general, said: “Families have been forced to flee for the second or third time in this conflict and have little left.”
The Houthis overthrew the internationally recognized government in 2015, prompting Saudi Arabia’s military intervention in the civil war.
The group currently controls most of Yemen, including the capital Sanaa. The truce, which lasted more than four years, effectively collapsed in July.