If It Can Fail, It Will Fail
When I’m hired to webcast an event for a client, I like to believe that I’ve planned adequate levels of redundancy for nearly every aspect of the production.
Our kits have backups for everything from cameras, lenses, and microphones to video switchers and encoders.
For the encoding pipeline, we standardize on
having dual outbound internet connections, using one per encoder. For the government work
we do, we usually have access to two dedicated
connections: an internal government backbone
with more-than-sufficient bandwidth for 720p
broadcasts and a local carrier connection such
as a cable modem. Each H.264 (or H.265/HEVC)
encoder uses a dedicated connection, and the
Wowza Streaming Engine tech we’ve built allows an immediate switchover to the secondary
encoder stream should the primary stream fail.
Having two connections is likely enough redundancy for just about any live-streaming work
we do, and I fully appreciate that not all scenarios will allow for having one, let alone two, hard-wired internet connections. In the past couple
of weeks, though, we’ve had issues with internet stability at venues where previously there
were none—and because we were prepared,
we had minimal downtime for the webcasts at
hand. I thought it would be useful to readers to
put some guidelines together for redundancy
throughout the entire live-streaming pipeline:
1. Prepare multiple networking options in
advance of the live webcast. If you’re working
with a hotel or conference venue, ask just how
dedicated your internet connection will be. Do
they have a backup internet provider you could
utilize? If not, look into the cost of hardware and
data plans that can utilize multiple USB 3G/4G/
LTE modems such as those offered by Mushroom
Networks or Teradek. I also like to prepare a single hotspot device such as a ZTE Turbo Hub as
a third backup. Essentially, any networking appliance that can provide a wired Ethernet connection to an encoder can play a backup role.
2. Prepare more than one origin (or ingest)
streaming media server. If you’re using a CDN
or hosted streaming environment, make sure
you have primary and backup publishing points,
one for each encoder. If you like to build your
own streaming server instances on the cloud,
have two origin servers ready to receive the incoming streams from the server. You may want
to have one origin on standby should the primary origin not be reachable (e.g., an Amazon
Web Services region becomes unavailable due
to a DDoS attack). Your edge servers need to be
configured to locate the availability of streams
on each of your origin servers.
3. Thoroughly test the throughput of each internet connection prior to the webcast. Run your
encoders for at least 30 minutes to ensure that
bandwidth is stable. If possible, monitor the incoming stream on your streaming server to see
if bandwidth fluctuates wildly during the test.
Be sure to run some “active” content during
the test—encoders will not necessarily utilize
the full bitrate you’ve specified if you’re simply pushing color bars and test tone. Run live
camera feeds of the venue, preferably during a
dress rehearsal of the event.
4. Thoroughly test the failover system prior
to the webcast. Take each encoder offline to see
how effectively your system works. If you have a
solid setup, you can bring each component back
online without your live-stream player displaying an error.
5. Have a disaster recovery plan in place. Write
up a specification that anyone monitoring the
encoding gear can follow in the event of a network outage. Backups can fail too, and your clients should know in advance how you and your
team will respond in the event of one (or more)
network outages. You can extend this plan to
include anything related to your production set-up, including camera or mic failures.
If you have redundancies built into your live-streaming checklist, you’ll know exactly how to
handle any failure before it turns into a disaster.
Robert Reinhardt ( firstname.lastname@example.org) is founder of VideoRx
and is internationally regarded as an expert on multimedia
application development and online video, particularly in HTML5,
iOS, Flash, AVC/H.264, and HEVC/H.265.
Comments? Email us at email@example.com, or check
the masthead for other ways to contact us.