The broker commission system at the heart of the U.S. residential housing market is facing unprecedented antitrust scrutiny from the Department of Justice and two private class-action lawsuits that risk weakening the National Association of Realtors, the industry's powerful lobbying group.