The case before the court involved Seila Law, a firm that received a Civil Investigative Demand from the agency.

Under President Obama, the agency defended its current structure, but current CFPB Director Kathy Kraninger has joined the Trump Administration in saying that the agency is illegally constituted.

As a result, the court has asked former Solicitor General Paul Clement to present an argument defending the agency.

Credit union trade groups have said they want Congress to make changes to the agency's structure by amending Dodd-Frank to have the CFPB governed by a multi-member commission.

Justice Ruth Bader Ginsburg pointed out that it is uncommon for the Justice Department not to defend the constitutionality of a statute.

Representing Seila, Kannon Shanmugam said that Dodd-Frank gave the CFPB unprecedented power.

"Never before in American history has Congress given so much executive power to a single individual who does not answer to the President," Shanmugam said. "By significantly limiting the President's ability to remove the CFPB's director, Congress violated the core presidential prerogatives to exercise the executive power and to take care that the laws be faithfully executed."

Justice Sonia Sotomayor said there are two other agencies that are somewhat insulated by presidential control, the Office of Special Counsel and the Social Security Administration.

"And I see the Social Security Administration being as powerful, if not more powerful, than this agency, because the Social Security Administration affects virtually every American," she said. "This agency is limited to the financial market."

In passing Dodd-Frank, Congress was attempting to protect consumers who had not been protected by other federal agencies, Ginsburg said.

"What about the people that Congress was concerned about, that is, the consumers who were not well protected by the array of agencies that were handling these problems?" she asked.

Justice Brett Kavanaugh pointed out that a new president would be confronted with a director appointed by a previous president.

"The next President in 2021 or 2025, or whenever, will have to deal with a CFPB director appointed by the prior President potentially for his or her whole term without being able…to do anything about that difference in policy," he said.

Consumer groups had asked Kavanaugh to recuse himself from this case since as a lower court judge, he ruled that the agency's structure was unconstitutional.

And Justice Neil Gorsuch said that if the structure was found to be constitutional, Congress could go to great lengths in protecting other executive agency officials.

"What would be the stopping point for Congress's ability to place high levels, serious impediments to presidential removal powers of …members of the cabinet and other executives — what we think of traditionally as executive agencies?" he asked.

The court is expected to decide the case by this summer.