I found this note recorded in (North Carolina and its Resources. State Board of Agriculture, xv, 413 p., ill., Winston, M. I. & J. C. Stewart, Public Printers and Binders, 1896, Call number C917 N87a (North Carolina Collection, University of North Carolina at Chapel Hill [electronic copy_http://docsouth.unc.edu/nc/state/state.html#p73]) "In Davie county are the Butler, Callahan Mountain, Isaac Allen and other mines of minor importance."
I wasn't searching for anything remotely related to gold mining and stumbled onto the above electronic copy. Are you familiar with Issac Allen? The only Allen farm that I am aware of is off of Hwy 64 near the Center VFD. This mining would probably have occurred between 1860 and 1900.