That was my fear, too, and the truth is you WILL lose clients. But those are not the clients you want, anyway. Figure out your ideal client and target them in your marketing.
Fewer clients, but more work and profit per client.