The 36-year-old has played the superhero since 2011's 'Captain America: The First Avenger'. He has starred in two sequels - 2014's 'Captain America: The Winter Soldier' and 2016's 'Captain America: Civil War' - along with two 'Avengers' movies.
"You want to get off the train before they push you off," he told the publication.
The former 'Human Torch' will be returning for reshoots for the fourth 'Avengers' movie later this year, but it is expected that they will mark the end of the road for him in the MCU.
Reports are doing the round that Downey Jr., too, may be ready to leave the Marvel universe.
'Avengers: Infinity War' hits the big screens on April 27. (ANI)