"We strongly condemn his comments. This is the land of Periyar, MGR, and Amma, we will not accept his views," Jayakumar said.
The minister asserted that his party would have taken action against Raja, had he not withdrawn his comments.
Periyar's bust was vandalised in Tamil Nadu's Vellore city on Tuesday evening after Raja in his now-deleted Facebook Post wrote, "Who is Lenin and what is the connection between Lenin and India? What connection has India with Communists? Lenin's statue has been removed in Tripura. Today it is Lenin's statue in Tripura, tomorrow it will be the statue of caste fanatic EVR Ramasamy."
Following the criticism, the BJP leader has now apologised for the post and blamed his social media team for the comments on Periyar.
A statue of 'Lenin' was razed in south Tripura, the first such incident surfacing in the country, after the BJP swept the Left front government out of power in Tripura assembly election.
Prime Minister Narendra Modi and the Ministry of Home Affairs have condemned in strong words the incidents in Tamil Nadu and Tripura.
A statue of Bharatiya Jana Sangh founder Syama Prasad Mukherjee was also vandalised at Kalighat area of south Kolkata on Wednesday, a third such incident within two days.(ANI)