The romantic notion of a road through the scenic southern Appalachian Mountains existed long before the Parkway's inception in the 1930s. Col. J. H. Pratt, who worked for the North Carolina Geologic and Economic Survey, charted such a route prior to World War I. His proposed mountain road extended from Roanoke, Virginia, through North Carolina and on to Greenville, South Carolina. He was actually able to build a short section of his road in North Carolina before the war began. The current Parkway follows sections of the colonel's old road near Altapass, North Carolina (mileposts 323 to 326).

The idea that finally reached fruition occurred to several men nearly simultaneously. When President Franklin Delano Roosevelt toured the Civilian Conservation Corps (CCC) camps of Shenandoah National Park on August 11, 1933, he responded with enthusiasm to Virginia senator Harry F. Byrd's recommendation that Skyline Drive be connected with a second scenic drive that would extend to the Great Smoky Mountains National Park. In September, Byrd met with Virginia governor G. J. Pollard and Theodore E. Straus of the Public Works Administration to discuss the possibility of a parkway. The governor approved the idea of the project and appointed Byrd chairman of a Virginia committee that would expedite the project.