Did the first Christian believers keep the Sabbath and if so on which day? According to Colossians, there appears to be some disagreement amongst the first century church on whether to keep the Sabbath:

Colossians 2:16
Therefore do not let anyone judge you by what you eat or drink, or with regard to a religious festival, a New Moon celebration or a Sabbath day.

Did the first century church keep the Sabbath?
And if so which day was it on? The Jewish Saturday or the new Christian Sunday?
The writings of the early Christians make it clear that Christians abandoned the Jewish custom of worship on the Sabbath (seventh day) and instead held the first day, which they called the Lord's Day, to be the sacred day of worship.  One of the simplest explanations on the subject comes from Tertullian, around 200 AD:


  But why is it, you ask, that we gather on the Lord's Day to celebrate
  our solemnities?  Because that was the way the Apostles also did.
  
  -- De Fuga in Persecutione, XIV: 11


This was not an innovation of the second century AD, as a full century earlier, (101 AD,) we find Ignatius saying:


  Those who were brought up in the ancient order of things have come to
  the possession of a new hope, no longer observing the Sabbath, but
  living in the observance of the Lord's day on which our life was
  sprung by him and his death.
  
  -- Epistle to the Magnesians, Chapter 9


And just in case there is any confusion as to the identity of the Lord's Day, we can turn to Justin Martyr. (140 AD)


  And on the day which is called Sunday there is an assembly in the same
  place of all who live in cities or in country districts; and the
  records of the apostles, or the writings of the prophets, are read as
  long as we have time.
  
  ...
  
  Sunday is the day on which we all hold our common assembly,
  because it is the day on which God, when he changed the darkness and
  matter, made the world; and Jesus Christ our Savior on the same day
  rose from the dead.
  
  -- Apologies 1:67


And again from Justin Martyr, we see that Christians did not keep the Jewish holy days:


  We neither accord with the Jews in their peculiarities in regard to
  food nor in their sacred days.
  
  -- Apologies Sec. 21


So yes, it's clear from the historical record that the first-century Christians, such as Ignatius, had organized worship meetings on the Lord's Day (Sunday), and that they did not consider observance of the Jewish Sabbath important, it having been fulfilled by Christ with the rest of the Law of Moses.