Restart source_bin whenever input video signal is lost
Restart source_bin whenever input video signal is lost

Without this, the pipeline never recovers from a loss of input signal,
even after the input signal to the capture device is re-established.

The symptoms, in chronological order, were:

* The Hauppauge HDPVR capture device lost its input signal
* The video on screen turned completely blue
* The pipeline continued to receive and process a few frames (15 or so)
  of blue screen.
* The Hauppauge's light turned off and no further frames received by
  the pipeline. The X Window with the blue screen stays up on screen.
* After the video input to the Hauppauge is re-established, the
  Hauppauge's light never switched back on. The X Window with the blue
  screen is still up on screen and the pipeline is stuck. `stbt run`
  never terminates.

A momentary loss of input signal can be caused by:

* A stbt script that restarts the set-top box.
* A stbt script that causes an HDMI/HDCP renegotiation (e.g. by changing
  between SD and HD channels on a set-top box with a buggy HDMI/HDCP
  implementation).

SOME NOTES ON THE IMPLEMENTATION:

`restart_source_bin` is run in the main application thread (the glib
thread) because it is scheduled from a glib timer; so it can safely shut
down the gstreamer pipeline elements. On the other hand, `on_underrun`
is run from the queue element's streaming thread; and you cannot change
the state of an element from its streaming thread.

`threads_init` is to avoid a segfault in `on_underrun` with the error
message "Fatal Python error: GC object already tracked". This solution
is recommended by a StackOverflow answer.[1]

After receiving an underrun signal from the queue, we wait for one
second before calling `restart_source_bin`, and if the queue receives
further buffers in that time, we cancel `restart_source_bin`. This is
because we always get one `underrun` signal when starting the pipeline
for the first time and after restarting the source bin.

The signal from the queue telling us that it is receiving further
buffers is "running"[2]. We chose "running" (has received more than some
min-threshold of frames) rather than "pushing" (has received a single
frame) to add some extra hysteresis to the system. We might want to
revisit this decision now that we are only receiving a single underrun
after restarting the source bin (during development of this commit, a
different implementation of `restart_source_bin` was causing a severely
reduced frame rate after restarting, and the queue was continuously
emitting underrun signals).

Note that glib and gtk modules are both provided by the pygobject
package. At some point, several functions were moved from gtk to glib,
because they are not UI-specific. For example, gtk also has add_timeout
but it is deprecated; we use glib.add_timeout instead. gtk.threads_init
is deprecated in favour of gtk.gdk.threads_init; but the latter also
initialises gdk threads that we don't use, so we really should use
glib.threads_init instead, but glib.threads_init isn't available on
ScientificLinux6's version of pygobject2 (2.20.0).

SOME OTHER THINGS WE TRIED:

Setting sink_bin to PAUSED instead of READY[3] caused a really slow
framerate (2-3 fps) after re-gaining video signal.

Just setting `pipeline` to PLAYING without first setting `source_bin`
PLAYING caused the video to freeze as soon as the source_bin was
restarted. The pipeline continued to receive frames (as evidenced by the
debug output from `on_match`), but the image was frozen at the first
frame received after restarting the pipeline.

Just setting the source bin to null without completely removing and
re-creating it caused a severe latency problem: The pipeline would
restart, but the image on screen would remain blue until the pipeline
had run (after re-starting) for the same amount of time that it had been
running before the video loss. Presumably the source bin was somehow
reporting that the new video was part of the same original stream, and
the sink bin was waiting for stream timing information that came after
the frames it had already seen.

Just setting the entire pipeline to null and then playing: The X window
disappears, the pipeline seems to start running but the X window never
reappears.

[1] http://stackoverflow.com/questions/7496629/gstreamer-appsrc-causes-random-crashes/7498503#7498503

[2] http://gstreamer.freedesktop.org/data/doc/gstreamer/stable/gstreamer-plugins/html/gstreamer-plugins-queue.html#GstQueue-running

[3] http://gstreamer.freedesktop.org/data/doc/gstreamer/stable/manual/html/section-elements-states.html

LewisHaley/stb-tester,wmanley/stb-tester,LewisHaley/stb-tester,stb-tester/stb-tester,LewisHaley/stb-tester,wmanley/stb-tester,LewisHaley/stb-tester,martynjarvis/stb-tester,martynjarvis/stb-tester,stb-tester/stb-tester,wmanley/stb-tester,wmanley/stb-tester,martynjarvis/stb-tester,martynjarvis/stb-tester,LewisHaley/stb-tester,stb-tester/stb-tester,wmanley/stb-tester,LewisHaley/stb-tester,martynjarvis/stb-tester,martynjarvis/stb-tester,stb-tester/stb-tester,martynjarvis/stb-tester,LewisHaley/stb-tester