He goofed, he admitted it, and now he's being investigated for it.

The office of the Commissioner of Official Languages confirmed Wednesday that three complaints had been lodged about Justin Trudeau's actions at a town hall meeting in Sherbrooke.

By Thursday, that number jumped to nine complaints.

The prime minister came under fire when he was asked questions in English at Tuesday's meeting and he answered in French, saying he would do so "because he was in Quebec."

Commissioner spokesperson Nelson Kalil said the complaints were filed under two sections related to getting minority language services from federal workers.

Kalil would not disclose who made the complaints, since they are allowed to be anonymous.

Eastern Townships resident Judy Ross had asked Trudeau about the lack of health services available in English in the Eastern Townships.

Her question: "Of particular concern in this area are services to minority populations. Most of the services are available in French only. It is only in the last 10 years that we have had a group dedicated to trying to help the English population in this area. I would really appreciate your comments on this subject."

She was quite upset when Trudeau answered her in French.

"I was so disappointed that by the time he got through that bit of fantasy land (explaining why he was speaking in French), I really didn't take in the rest," Ross said in an interview with The Canadian Press. "I was too miffed."

"It (mental health) is a topic that's very difficult to explain and express in your own language, let alone a second language. Even people who are bilingual prefer to have their services in their mother tongue. And I thought, with his life experience, he would be sensitive to that."

The PM's response generated outrage and angered many anglophone community groups, including the Townshippers' Association.

"For the prime minister to declare in Quebec 'we speak French' and then to answer each and every question that's posed in English in French left the members of the community wondering exactly what is happening. Is this a change of policy for the federal government?" said Gerald Cutting.

The Quebec Community Groups Network had the same reaction.

"The event of the prime minister taking a formal position: we're in Quebec, we're going to speak French, was very surprising and very wrong," said Geoffrey Chambers.

"He can't have meant to do something that is as hurtful and harmful as this would have been, and I think he should say that."

Chambers and the QCGN are now demanding Trudeau apologize for his actions.

Trudeau admitted on Wednesday that he made an error.

"I should maybe have answered part of that question in English," said Trudeau, while pointing out that he recently answered a questioned posed in French in Peterborough, Ont., in English.

He added that from now on he will work to ensure his responses are more bilingual -- no matter where he is speaking.

Kalil says the investigation could take three to six months.