Where does the belief the Bible is written by the Holy Spirit come from? My pastor mentioned the Bible is written by the Holy Spirit, through man. 

Where does this concept come from (rather than just written by wise or experienced men)?

Other than direct references to Jesus, how do we know the rest is not just man's ideas and are indeed God inspired or "written"?

Is there a chapter in the Bible that states this or is it tradition?
Great question!
The short answer is that Paul made a general statement to this effect:

All Scripture is inspired by God -2 Timothy 3:16

The phrase &quot;inspired by God&quot; is literally translated as &quot;God-breathed&quot;.  So the Bible is not merely a collection of wise men's writings, but is actually authored by God, through men, by the Spirit.
Now, on to more specific examples.
Prophecy

no prophecy of Scripture is a matter of one’s own interpretation, for no prophecy was ever made by an act of human will, but men moved by the Holy Spirit spoke from God. -2 Peter 1:20-21

Clearly prophecy is the work of the Holy Spirit.  (Also see Jeremiah 1:1-9, 2 Samuel 23:2, Acts 3:18, and Hebrews 10:15-17.)
However, many people teach that &quot;not all Scripture is prophecy&quot;, and they use the division of &quot;Law, Psalms, and Prophets&quot; to signify the three chunks of the Old Testament.
Psalms

“Brethren, the Scripture had to be fulfilled, which the Holy Spirit foretold by the mouth of David...&quot; -Acts 1:16
“For it is written in the book of Psalms, ‘Let his homestead be made desolate, and let no one dwell in it’; and, ‘Let another man take his office.’ -Acts 1:20

The two passages being quoted in verse 20 are Psalms 69:25 and Psalms 109:8.  Although these were &quot;Psalms&quot;, clearly they were prophetic utterances by the Holy Spirit, which &quot;had to be fulfilled.&quot;
The Psalms were the work of the Holy Spirit, and could also be considered &quot;prophecy.&quot;  (See also Matthew 22:41-44.)
Law
We all know that the 10 Commandments were written by the very finger of God, but what about the rest of the Law?
Acts 3:22 and Acts 7:37 show that Moses, the writer of the Law, was also a Prophet!
We have a nice exposition of some &quot;Law&quot; stuff in Hebrews which shows clearly that the things written in the Law also carried the intent of the Holy Spirit:

Now when these things have been so prepared, the priests are continually entering the outer tabernacle performing the divine worship, but into the second, only the high priest enters once a year, not without taking blood, which he offers for himself and for the sins of the people committed in ignorance. The Holy Spirit is signifying this, that the way into the holy place has not yet been disclosed while the outer tabernacle is still standing, which is a symbol for the present time. Accordingly both gifts and sacrifices are offered which cannot make the worshiper perfect in conscience, since they relate only to food and drink and various washings, regulations for the body imposed until a time of reformation. -Hebrews 9:6-10

Gospels
All of Jesus words and His actions were authored by God, and Spirit-led.  But what about the records of them?

“These things I have spoken to you while abiding with you. But the Helper, the Holy Spirit, whom the Father will send in My name, He will teach you all things, and bring to your remembrance all that I said to you.  -John 14:25-26

So yes, the Gospels are also Spirit-written.
Epistles
2 Peter 3:14-16 shows that the epistles are also part of Scripture.
Clearly given the event of Acts 2, explanations of the Spirit's ministry through us (1 Corinthians 12, Romans 12, Ephesians 4, etc.), and constant claims such as the following, it can be safely assumed that the New Testament Scriptures were also Spirit-written.

If anyone thinks he is a prophet or spiritual, let him recognize that the things which I write to you are the Lord’s commandment. -1 Corinthians 14:37

Summary
All Scripture is authored by God via the Holy Spirit.  A case can also be made that all Scripture is prophecy, which makes it even more clear that the Spirit was the instrument of God for authoring Scripture.  Beyond the generic statements, we have a number of specific examples which support this doctrine.