Does it matter which day I celebrate Sabbath / Lord's day on? Inspired by this question: What qualifies as work on Sunday?

I know some churches are very specific that we should commit one weekday to the Lord (Saturday or Sunday). Much like Jews celebrate the Sabbath.

The need to rest one day a week is very real for me. I really find it necessary to take one day off work, studies and all that. I usually do this on Sunday, as it's usually practical. However, if I need to do something on Sunday, I have no problem with instead taking the following Tuesday off, resting, reading and praying. This seems to work well for me.

I've never found a reason to specify the weekday I should spend like Lord's day. Are there any?
In Genesis 2:2-3 (NIV) it says,


  2  By the seventh day God had finished the work he had been doing; so on the seventh day he rested from all his work. 3  Then God blessed the seventh day and made it holy, because on it he rested from all the work of creating that he had done. 


It would seem to indicate only the existence of 'a day of rest', but there is no reference to a certain day of the week being more deserving of said title. I only see evidence that we should observe 'a day of rest'; though whatever day of the week that might happen to be is of little to no consequence really.