Question about nativity and timelines I can't quite wrap my head round this one. How do we know so much about the birth of Christ, but almost nothing about his growing up?

As I see it, there were only a handful of people at the birth, Mary, Joseph, some shepherds and possibly the owner of the stable. That's been documented and detailed in the gospels in reasonable detail, so someone there must have provided details. 

If they realised that His birth was significant then I can understand the sharing of the facts of his birth, but if that was the case then why not more detail about His growing up?

If they didn't realise until He started his ministry that he was the same Jesus, and then connected the stories together, then there wouldn't have been any real oral traditional stories of his birth. How was there such a detailed narrative about it?
It doesn't make sense to me to document only the birth and then nothing until the ministry. 

I understand the way stories were handed down, and I understand that the gospels were written later, but I just can't reconcile the above.
This answer is from the perspective of history and the Bible (whether divinely inspired or merely a message that humans wanted us to hear).

There are four specific ages that are significant in Judaism:


Birth
1 week old: male circumcision Brit Milah.  This symbolizes the physical covenant between God and God's People.
12-13 years old: morally responsible for one's actions Bar Mitzvah or Bat
Mitzvah.  Past this age, people are no longer to be treated as children, but as adults.
30 years old: qualified for leadership.  Behaviour and attitudes can change significantly before 30, but are relatively stable after that.  E.g. Priests had to be at least 30 years old (Numbers 4:3), Joseph began running Egypt (Genesis 41:41,46), and David was anointed King of Israel (2Samuel 5:3).


The Gospels explicitly mention Jesus's birth (Luke 2:7), his circumcision (Luke 2:21), his unaccompanied visit to the Temple (Luke 2:46), and the beginning of his ministry.  Jesus was both Priest and King, so 30 years old is a doubly appropriate age.

What happened between those key events was not considered relevant, so there was no need for it to be recorded.  Luke 2:51-52 provides all the details we need to know about his life between 12 and 30:


  Then he went down to Nazareth with [his parents] and was obedient to them. But
  his mother treasured all these things in her heart.  And Jesus grew in
  wisdom and stature, and in favor with God and man.