The distillery was built six miles south-west of Elgin, in the grounds of Pluscarden Abbey, on account of the quality of the water source. Not surprisingly, many illicit distillers used the vicinity in the late eighteenth century. Miltonduff went legal in 1824 and by the end of the century was one of the largest producers in Scotland, making in excess of one million litres a year and using triple distillation an unusual technique for Highland/Speyside distilleries.
After debuting to a successful start in 1866, the distillery was purchased by William Stuart. He was joined by Thomas Yool in 1890 and the distillery was extended in the min 1890s, at which time it was producing over one million litres of alcohol. In 1936 Yool sold to Hiram Walker, who had bought George Ballantine & Company the year before.