We study the deterioration of employment in middle-wage, routine occupations in the United States in the last 35 years. The decline is primarily driven by changes in the propensity to work in routine jobs for individuals from a small set of demographic groups. These same groups account for a substantial fraction of both the increase in non-employment and employment in low-wage, non-routine manual occupations observed during the same time period. We analyze a general neoclassical model of the labor market featuring endogenous participation and occupation choice. We show that in response to an increase in automation technology, the model embodies an important tradeoff between reallocating employment across occupations and reallocation of workers towards non-employment. Quantitatively, we find that advances in automation technology on their own account for a relatively small portion of the joint decline in routine employment and associated rise in non-routine manual employment and non-employment.