Some scholars of political philosophy--the Straussians, for example--would explain this by saying that Smith was defending the commercial society as "low but solid." To secure the solid benefits of a commercial way of life--liberty and opulence--he was willing to accept the lower aims of modernity, even when this meant depriving most human beings of genuine happiness and genuine excellence. And yet he could not do this without lamenting the loss of the human greatness promoted by pre-modern thought. This is what the Straussians like to call "the problem of the bourgeois." This is Joseph Cropsey's argument in Polity and Economy.
But this interpretation of Smith seems hardly plausible if one notices that Smith defends the "wise and virtuous man" as the standard of moral and intellectual perfection as manifested in the life of philosophic friends (TMS VI.i.14, 216; VI.ii.I.18, 224-25; VI.iii.23-25, 247-48). He observes that the division of labor in a commercial society allows for the intellectual commerce of philosophers "whose trade it is, not to do anything, but to observe everything" (WN I.i.9, 21). And he presents the life of David Hume as showing how a commercial society provides the conditions for the philosophic life--a life that in Hume's case approached "as nearly to the idea of a perfectly wise and virtuous man, as perhaps the nature of human frailty will permit" (Letter to Strahan, Nov. 9, 1776, CAS, 221).
Moreover, Smith saw his account of virtue as compatible with Aristotle's teaching in the Nicomachean Ethics (TMS VI.iii, 237-62; VII.ii.I.12-14, 270-72). This has led Ryan Hanley and other Smith scholars to interpret Smith as an Aristotelian virtue ethicist (Calkins and Werhane 1998; Hanley 2009).
My explanation of this is that Smith's commercial liberalism coincides most closely with Aristotle's teaching about friendship and philosophy in Books 8 and 9 of the Nicomachean Ethics, which also happens to be one of the sections of Aristotle's moral and political writing that shows a propensity to liberalism, while also showing many references to his biology.
As I have argued in some previous posts, how one reads Aristotle's Ethics turns crucially on whether one sees the peak of his teaching in the books on friendship (books 8-9) or in the final book on philosophy as a purely contemplative life beyond the moral virtues (book 10). Philosophy as presented in book 10 is a purely theoretical life of solitary contemplation without friendship, and it's a life striving for an abstraction of the mind from the body. But philosophy as presented in books 8-9 is a life of philosophic conversation and thought with one's friends, which combines all the moral and intellectual virtues as perfecting the psychosomatic unity of human biological nature. Smith's moral and political philosophy follows the approach that Aristotle took in books 8-9.
In The Liberal Temper in Greek Politics, Eric Havelock argued that in ancient Greece there was a tradition of liberal thought supporting Periclean democracy. While most of the writing in this tradition--including Democritus, Antiphon, Protagoras, and Lycophron--survived only as fragments, Havelock thought the ideas of this liberal tradition could be found scattered in the texts of Greek tragedy and poetry as well as the texts of Plato and Aristotle. In particular, he thought that books 8-9 of the Nicomachean Ethics incorporates ideas from the biological anthropology of the liberals--especially "friendship as a biological and social fact" that
"becomes that spontaneous feeling of sympathy or goodwill which all members of a species are supposed by definition to feel for each other, and which expresses their recognition that they have common traits. Beginning perhaps as a herd instinct, it becomes the basis for that cooperation which creates and supports human society" (298).The biological character of the anthropology in this part of the Ethics becomes evident as soon as one notices how often Aristotle here invokes the biological propensities to pleasure, sexual mating, parental care, and other social instincts (see, for example, 1153b7-1154a1, 1154b1-22, 1155a17-22, 1159a27-33, 1161b15-1162a30). The liberal character of this anthropology becomes clear when one notices how Aristotle presents social order as arising spontaneously in the natural and voluntary associations of society (see, for example, 1159b25-1160a30).
Smith follows in this tradition of Aristotelian liberalism by arguing for a biological emergence of social order from the natural instincts of human beings as social animals (see, for example, TMS I.ii.2-3, 28; 77-78; II.ii.3-5, 86-87; III.3.13, 142; VI.ii.I-II, 219-34; LJ, 141-43, 163-67). While legal coercion is required to enforce the negative rights of justice to be free from unfair injury, the other moral duties are enforced through social praise and blame and the spontaneous order of civil society (TMS II.ii.1-4, 85-86; LJ, 7-9).
Smith also follows Aristotle in looking to philosophical friendship as the peak of human happiness that embraces all of the moral and intellectual virtues. The life of a Platonic or Aristotelian philosopher "necessarily supposes the utmost perfection of all the intellectual and of all the moral virtues. It is the best head joined to the best heart. It is the most perfect wisdom combined with the most perfect virtue" (TMS, VI.i.15, 216). The friendship of such philosophers is the highest form of friendship that is possible only among men of the highest virtue (TMS, VI.ii.I.18, 224-25).
Smith's clearest portrait of such philosophic friendship was in his letter to Strahan describing the magnanimity and cheerfulness of Hume in facing his own death while conversing with his friends. As argued by Eric Schliesser, this letter considered in the circumstances surrounding it shows how philosophy and philosophic friendship is possible in a modern commercial society.
Both Hume and Smith defended commercial civilization as superior to savage and premodern societies, and they suggested that in a commercial society, philosophy could prosper as a professional trade in the division of labor. The opulence and liberty of a commercial society would provide philosophers with the intellectual commerce and the leisured independence necessary for living a philosophic life with their friends. (Does this confirm the claim in Book 8 of Plato's Republic that democracy is the only regime that leaves people the freedom to live the philosophic life?)
Smith indicates that philosophers can benefit their fellow citizens through mechanical inventions, through the propagation of general knowledge, or through promoting "general ideas concerning the great subjects of religion, morals, and government" as bearing upon human happiness (LJ, 574). So, for example, Smith's writing on morality, economics, and politics could be seen as an exercise in philosophic beneficence.
But many of Hume's critics doubted his beneficence in his philosophic attacks on religious beliefs in miracles, providence, immortality of the soul, and the afterlife. And while Smith often invoked conceptions of divine providence over the order of nature, many of this readers suspected that he was as much of an infidel as his friend Hume.
Smith showed his sensitivity to this public suspicion in refusing Hume's request for publishing The Dialogues Concerning Natural Religion after his death. And yet his letter on Hume's cheerful facing of death without any belief in the afterlife provoked a public outcry against Smith.
The Dialogues provide incisive philosophical arguments against natural theology--against the idea that the order of nature manifests the intelligent design of the Creator. The Dialogues also provide some of the clearest anticipations of Darwin's argument for the emergence of natural order through the unintended order of evolution. So we are left wondering whether Smith agreed with Hume's proto-Darwinian thinking, or whether he thought that philosophers should show their beneficence by suppressing such teaching.
Calkins, Martin J., and Patricia H. Werhane, "Adam Smith, Aristotle, and the Virtues of Commerce," The Journal of Value Inquiry, 32 (1998): 43-60.
Den Uyl, Douglas, and Charles Griswold, "Adam Smith on Friendship and Love," Review of Metaphysics, 49 (March 1996): 609-37.
Hanley, Ryan Patrick, Adam Smith and the Character of Virtue (New York: Cambridge University Press, 2009).
Havelock, Eric A., The Liberal Temper in Greek Politics (New Haven, CN: Yale University Press, 1957).
Schliesser, Eric, "The Obituary of a Vain Philosopher: Adam Smith's Reflections on Hume's Life," Hume Studies, 29 (November 2003): 327-62.
Some posts on related topics can be found here, here, and here.