What book am I thinking of: It's about a group of highly capable people who withhold their abilities from those with nothing to contribute to their society. They convince skilled individuals to move to a concealed town, and although many are reluctant, most ultimately abandon their former lives and the rest of the world. "Blimey, everyone'd be wantin' magic solutions to their problems. Nah, we're best left alone."