video_output: decklink: split video mode selection
video_output: decklink: split video mode selection

Uses modeid in native byte order
Avoids setting values before selecting mode

xkfz007/vlc,xkfz007/vlc,xkfz007/vlc,xkfz007/vlc,xkfz007/vlc,xkfz007/vlc,xkfz007/vlc