President Barack Obama said on Monday that he will not tolerate political bias at the Internal Revenue Service.
The President promised to get to the bottom of the agency's admitted targeting of conservative groups.
The IRS acknowledged on Friday that the agency targeted the Tea Party and other groups during the 2012 elections.
Republicans have challenged the agency's assertion that low-level employees were responsible.
A draft of a watchdog report says IRS officials knew agents targeted Tea Party groups as early as 2011.
Debbie Dooley is a national coordinator and board member of the Tea Party Patriots.
"If they are allowed to go after us and get away with just a slap on the wrist or an apology, my message to groups on the left is, it's just a matter of time before they come after you as well," Dooley said.
President Obama addressed the revelations for the first time after welcoming British Prime Minister David Cameron to the White House.
"I've got no patience with it. I will not tolerate it and we will make sure that we find out exactly what happened on this," President Obama said.
Georgia Sen. Johnny Isakson said in a statement,"It is unconscionable that in the United States of America, groups were unfairly targeted by a government agency for exercising their First Amendment rights. Americans from every political affiliation should be outraged by this discrimination."
The IRS says no group had its tax exempt status revoked, although some withdrew their applications.
The White House earlier said it would address any misconduct.
The penalty for that type of misconduct is to be fired, but a Republican congressman may introduce legislation to increase the penalty to a five year prison term and a $5,000 fine.
An IRS official says about 300 groups were singled out for review of their tax-exempt status and about a fourth of those came under scrutiny because they had "Tea Party" or "patriot" somewhere in their applications.