How many witnesses are there who physically saw and/or touched the golden plates from which the Book of Mormon was reportedly translated? I understand the Book of Mormon has a list of witnesses.  However, I have heard that these witnesses may not have actually ever seen the gold plates that contained the original manuscripts for the Book of Mormon, but were witnesses in a different sort of way.

So, just how many people physically saw and/or touched the gold plates from which Joseph Smith is said to have translated the Book of Mormon?

Note: Martin Harris, one of the witnesses, later explicitly denied seeing the plates with his physical eyes.
See article..
The testimonies of these "lists of witnesses" is included in the introduction to the Book of Mormon, in which they explain what they saw and experienced.

From the testimony of the Three Witnesses:


  And we also testify that we have seen the engravings which are upon
  the plates; and they have been shown unto us by the power of God, and
  not of man. And we declare with words of soberness, that an angel of
  God came down from heaven, and he brought and laid before our eyes,
  that we beheld and saw the plates, and the engravings thereon; and we
  know that it is by the grace of God the Father, and our Lord Jesus
  Christ, that we beheld and bear record that these things are true. And
  it is marvelous in our eyes. Nevertheless, the voice of the Lord
  commanded us that we should bear record of it; wherefore, to be
  obedient unto the commandments of God, we bear testimony of these
  things.


And from the testimony of the Eight Witnesses:


  Joseph Smith, Jun., the translator of this work, has shown unto us the
  plates of which hath been spoken, which have the appearance of gold;
  and as many of the leaves as the said Smith has translated we did
  handle with our hands; and we also saw the engravings thereon, all of
  which has the appearance of ancient work, and of curious workmanship.
  And this we bear record with words of soberness, that the said Smith
  has shown unto us, for we have seen and hefted, and know of a surety
  that the said Smith has got the plates of which we have spoken.


Both groups include a solemn testimony that they did actually see the golden plates that the record was inscribed upon.  The first group (three men) was shown them by an angel, but they do not claim to have touched them, while the second (eight men) was shown them by Joseph Smith, and was allowed to handle them.  So those eight and three, plus Joseph Smith himself, makes a total of twelve who physically witnessed the original record.

Because of the strength of these testimonies, it was common, back when the men in question were still alive, to make claims that they had denied or recanted their testimonies in some way, as a way to attempt to undermine the authenticity of the Book of Mormon.  The linked Wikipedia article contains a few such claims attributed to Martin Harris, one of the Three Witnesses, but it also gives his claim that "No man heard me in any way deny the truth of the Book of Mormon [or] the administration of the angel that showed me the plates."

A discussion of this apparent discrepancy can be found at the FAIR wiki, which assumes that Harris did actually say what was attributed to him.  A simpler explanation, of course, would be deliberate incorrect attribution. Fact checking is hard enough even today with the Internet at our fingertips; in the mid 19th century, such a task was exponentially more difficult, which made it much easier for dishonest people to employ flat-out falsehoods in many different causes.

In any case, the last word on the subject should be Martin Harris's literal last words.  On his deathbed, he once again reaffirmed that:


  The Book of Mormon is no fake. I know what I know. I have seen what I
  have seen and I have heard what I have heard. I have seen the gold
  plates from which the Book of Mormon is written. An angel appeared to
  me and others and testified to the truthfulness of the record, and had
  I been willing to have perjured myself and sworn falsely to the
  testimony I now bear I could have been a rich man, but I could not
  have testified other than I have done and am now doing for these
  things are true.


The other two of the Three Witnesses both gave similar reaffirmations on their own deathbeds.