hmmm I don't know if this is what you were thinking of, but I know that they can do certain types of genetic testing to determine your ancestry. Usually, its through mitochondrial DNA (which is passed by the mother), and Y-chromasome DNA (passed father to son). I don't know how expensive it is, but I know you can find out some things about your ancestry. For instance, they have traced Jewish populations from all over the world back to the middle east in this way. I don't know if this is what you were thinking of, but it came to mind.
Or, of course, there is genealogical research. I don't know how exactly its done, but I beleive the basics are going to the place where the furthest back ancestors you know (your DH's father in this case), and trying to find birth records, and from those, learning your DH's grandparents' names, and then trying to find death records or marriage records, which might give you more information to find birth records, and so on. I know that going back into history, in Europe and America before birth certificates, church baptismal records are the main source, I don't know much else. People really serious about it also look at everything, I think, correspondence, military records, court records, all sorts of things. Seems a bit too labor intensive for me. You could look into it though.