Was the virgin conception of Jesus known to the people at the time? In the Bible, it seems that Mary consulted with Elizabeth after the Annunciation but from the people's reaction to Jesus at ministry, it occurs to me that they are not aware of the surrounding story about the birth of Jesus at all.


  Isn’t this the carpenter’s son? Isn’t his mother’s name Mary, and aren’t his brothers James, Joseph, Simon and Judas? [Matthew 13:55, NIV] 


So, is it that Elizabeth (maybe Zacharias as well), Mary, Joseph concealed the Annunciation to protect Jesus until the Passion?

Is there any biblical lines that indicate any of Apostles or Disciples or people aware of the Annunciation or Virgin Conception of Jesus before the Scripture telling so?
If Mary withheld this mystery from Joseph her spouse, it is more than reasonable to believe that those who knew of it [Joseph, Elizabeth, Mary's parents] left it to God to reveal it at his appointed time.

To the General Public, no! as the following passages reveal:


  Mt 13:55
  (RSVCE)
  55 Is not this the carpenter’s son? Is not his mother called Mary? And
  are not his brethren[a] James and Joseph and Simon and Judas?
  
  Lk 4:22
  (RSVCE)
  22 And all spoke well of him, and wondered at the gracious words which
  proceeded out of his mouth; and they said, “Is not this Joseph’s son?”
  
  Jn 1:45
  (RSVCE)
  45 Philip found Nathan′a-el, and said to him, “We have found him of
  whom Moses in the law and also the prophets wrote, Jesus of Nazareth,
  the son of Joseph.”


It required divine revelation for Elizabeth and Joseph to know.



Why wasn't it scandalous for Mary to be pregnant while betrothed to Joseph? 

This is answered by the two-stage process in marriage among the Jews. The betrothed were considered husband and wife and could have marital relations [cf. Tb 7:9 (RSVCE) - Tb 8 and Mt 1:20 (RSVCE): Before God, Mary is considered Joseph's wife].



This is another example of the interplay in human [salvation] history between God's will and free human actions.