Dominance of mortgage lending by a few large banks is preventing consumers from getting lower rates and preventing a recovery, argues William Dudley, president of the Federal Reserve Bank of New York.

The Fed's latest quantitative easing program is supposed to push down mortgage rates by having the Fed buy $40 billion of mortgage-backed securities (MBS) per month.

But the lower rates are not being passed on to consumers, and a lack of competition in a mortgage-lending industry dominated by a few key players is at least partly responsible, Dudley said in a speech at the National Association for Business Economics in New York.

Editor's Note: Economist Unapologetically Calls Out Bernanke, Obama for Mishandling Economy. See What They Did

For instance, Wells Fargo now originates about a third of new home loans, according to the Financial Times.

"Federal Reserve MBS purchases have succeeded in driving down mortgage rates to historically low levels," he said. "But these purchases would have had still more effect on the economy if pass-through rates from the secondary market to the primary market had been higher."

Officials should work to increase competition so those secondary market rates can pass through to homebuyers and homeowners, Dudley said.

Other factors are keeping mortgage rates from falling even lower and hampering mortgage lending, he noted. Warranties that Fannie Mae and Freddie Mac place on their loan purchases can force banks to buy back loans that go bad, which discourages banks from lending.

Also, Fannie Mae and Freddie Mac, which purchase or guarantee most home loans, increased their guarantee fees they charge lenders, who then pass those fees along to consumers.

"Factors limiting pass through warrant ongoing attention from policymakers," Dudley said.

Large banks will dominate mortgage lending even more, as community banks find competing against them difficult, a community banker told CNCB. Many smaller banks will not be able to survive the low interest rate environment and meet increased regulatory costs, Valley National CEO Gerald Lipkin told CNBC.

Banks' net interest margins, the difference between what banks charge clients and what they pay depositors, are being squeezed.

“It’s very difficult to make money in this environment on a net interest margin basis,” he told CNBC.

Editor's Note: Economist Unapologetically Calls Out Bernanke, Obama for Mishandling Economy. See What They Did