Smith is mainly talking about class and the division of labor. This was a general way of talking about class for the time. That is, there’s the real citizens of the country, the educated elite, whose minds engage with many diverse things. Then there are people who work. They are narrow and limited. They are incapable of discussing things such as government and policy. So we don’t ask them to participate.
Here’s the whole original quotation:
In the progress of the division of labor, the employment of the far greater part of those who live by labor, that is, of the great body of the people, comes to be confined to a few very simple operations; frequently to one or two. But the understandings of the greater part of men are necessarily formed by their ordinary employments. The man whose whole life is spent in performing a few simple operations, of which the effects, too, are perhaps always the same, or very nearly the same, has no occasion to exert his understanding, or to exercise his invention, in finding out expedients for removing difficulties which never occur. He naturally loses, therefore, the habit of such exertion, and generally becomes as stupid and ignorant as it is possible for a human creature to become.