How do decentralized churches/denominations practically deal with "brothers" from another denomination in light of Matthew 18:17 "If your brother sins against you, go and rebuke him in private. If he
  listens to you, you have won your brother.
  
  But if they will not listen, take one or two others along, so that
  'every matter may be established by the testimony of two or three
  witnesses. 
  
  But if he won't listen, take one or two more with you, so that by the
  testimony of two or three witnesses every fact may be established.
  
  If they still refuse to listen, tell it to the church; and if they
  refuse to listen even to the church, treat them as you would a pagan
  or a tax collector."


It must be noted that the word Church is used in the singular form - not a church, and not one of the churches.

In light of this verse, what happens in a situation where there are two "brothers" from two different denominations, (baptist, presbyterian, pentacostal, etc.) and one of the brothers commits a very scandalous, heinous sin against the other.

Who's church would the sinful brother be taken to?

Either church would have no jurisdictional rights over the other. 

As a Catholic I know that there is a canonical process for one to follow...but the RCC is the most centralized religious body in the world. 

Have any Protestant pastors here ever had to deal with a situation like this?
If appealing to authority is necessary, then you would have to appeal to authority that the brother in the wrong is submitting himself to. In this case, it would need to be someone in his congregation (or its hierarchy of authority).

The entire goal of the church discipline described in Mt 18 is to win him over to the truth. He ought to have close ties to the people in the congregation he's a part of, and so if those people must cut him off, the hope is that he will care about those lost/damaged relationships enough to reconsider what he's doing and repent.