He has pleaded not guilty.
Manafort, who performed lobbying work for a pro-Russian former Ukrainian president before serving as Mr Trump's campaign chairman in the 2016 U.S. election, is facing an array of charges in two indictments brought by Mr Mueller in federal courts in Washington and Alexandria, Virginia. Trump has also denied any collusion between himself and the Kremlin and has called Mueller's investigation a "witch hunt".
U.S. District Judge Amy Berman questioned Manafort's attorney on the legal reasoning behind the former Trump aide's argument that Mueller's investigation has overstepped and should be shut down. Manafort has sought to have the case dismissed, arguing that the charges against him are outside of Mueller's authority.
The Justice Department, which was represented by career attorneys (though members of Mueller's team were present in the courtroom), argued that a civil lawsuit is not the appropriate place to bring issues with Mueller's investigation that instead could be brought in the criminal cases against Manafort.
Manafort filed a civil lawsuit on Jan 3 against Mr Mueller and Deputy Attorney General Rod Rosenstein, the Justice Department official who appointed him, in a key legal test of how far the special counsel's mandate extends.
The government will tell the judge that Manafort has no right to challenge how the Justice Department's rules governing special prosecutors are enforced.
Manafort faces two trials in Washington and Virginia courts, the first of which is scheduled for July.
Jackson asked why the Russia-specific part of Mueller's appointment order wouldn't apply to Manafort anyway, since it referred to "any links and/or coordination between the Russian government and individuals associated with the campaign of President Donald Trump".
Also in Monday night's filing, Mueller's prosecutors explain how they might try to connect Manafort's pre-campaign lobbying work to possible crimes during the 2016 campaign.
But Special Counsel Robert Mueller on Tuesday filed a court document that showed that Rosenstein had given him broad authority to investigate Manafort for a variety of potential crimes, including his ties to the former government of Ukraine, including former President Viktor Yanukovych. The current charges include money laundering, conspiring to hide his work for Ukraine from USA officials, filing false tax returns, and bank fraud.
Berman Jackson didn't decide on the lawsuit on Wednesday, and will issue an opinion later. Downing turned around to look at Manafort, then said forcefully, "Never". Later, Downing clarified that the civil lawsuit was not seeking the dismissal of Manafort's charges but rather "prospective relief", an injunction preventing future action by Mueller's team.
Downing said it didn't undercut their case, pointing out that they were no longer pursuing the claim that Mueller exceeded his authority under the appointment order.
The Justice Department is handling the defense against Manafort's civil suit, but at least a half dozen lawyers from the special counsel's office attended Wednesday's hearing, including Manafort prosecutors Andrew Weissmann, Greg Andres, and Brian Richardson.