Struggling to get resettled, Withers moved his family to Lexington, KY and purchased Fairlawn Stock Farm. Knowing little about trotting bloodlines, Withers studied the breed carefully and eventually populated his stock farm with over seventy respectable broodmares.

In 1875, he purchased Almont for a reported $15,000 and Immortal Happy Medium for a price in excess of $25,000 – an equivalent of about $500,000 in today's currency.

Other stallions that stood at Fairlawn through the years included Aberdeen, Ethan Allen Jr. and several other home bred sons of Almont, including Alecto, Almont Wilkes and Maximus.

Withers was reportedly the first to take the risk of listing prices in his sale catalogs. As Fairlawn's fame grew, both Ulysses S. Grant and King Kalakahua of Hawaii visited the farm and horsemen from around the world began to request Fairlawn's stock.

Withers' efforts disseminated the American trotting breed throughout Australia, Bessarabia, Canada, Denmark, England, France, Germany, Hawaii, Italy, Japan, New Zealand, Russia, and Sweden.

According to the January 1886 issue of Wallace's Monthly, in less than two decades Fairlawn had become arguably the most well known stock farm in the world.
Mark Field of Wallace's Monthly writes of him: "…he has not only made his name illustrious throughout the domain of civilization, but has perhaps more palpably than any other living man demonstrated by actual experiment the general principles upon which the breeding of diagonal steppers may be reduced to a science."

Withers died at the age of sixty-six on June 16, 1889. Speculation at the time valued Fairlawn Stock Farm, his life accomplishment, at $500,000 – an equivalent of $11,650,000 in today's currency.   He is one of the 2009 inductees into Harness Racing's Hall of Fame in Goshen, New York.