Home / Blog / How to Stream Live Video From Your Drone with voxl-streamer
How to Stream Live Video From Your Drone with voxl-streamer

How to Stream Live Video From Your Drone with voxl-streamer

Jan 12, 2021 Posted by: Lauren Young

Written By Eric Katzfey

Drones are marvels of engineering and a lot of fun to fly, but what is there to do with a drone beyond just flying around? There are a lot of answers to that question and more are being discovered every day, but most of them revolve around cameras, images, and video. In fact, drones are often referred to as flying cameras. Many use cases include storing images and video for later viewing or processing the video on board in a computer vision application such as Visual Inertial Odometry (VIO), Visual Obstacle Avoidance (VOA), or object recognition using AI. Often, however, you will also want to see the video from your camera in real time while your drone is in flight. In this blog I will go into detail on the topic of real-time video streaming, how ModalAI supports it, and provide some pointers to more information about this subject.

Choose From Hundreds of Image Sensors

In order to stream video from your drone you will need a few different components. The most obvious component is the camera or image sensor to produce the video frames. There are a huge number of choices out there and it can be tough to decide what to use on your drone. In many cases, you will want multiple cameras for different situations and use cases. At ModalAI, we provide both the hardware and software to use a wide variety of cameras and image sensors with our VOXL® platform. For example, there are small low-cost black and white image sensors like the OmniVision ov7251 that connect via MIPI. These sensors are excellent choices for computer vision applications such as VIO and VOA. There is a wide variety of high resolution image sensors that also connect via MIPI and produce high quality full color video suitable for viewing. In addition to MIPI, some cameras connect using USB. There are a variety of protocols that can be used by USB connected cameras. A very popular option is USB Video Class (UVC), which is what most webcams support, and the Picture Transfer Protocol (PTP) which a lot of high end DSLR cameras support. Some cameras even have HDMI out capability and VOXL supports HDMI input as well. Thermal sensors, such as those from FLIR, are popular in search and rescue and government applications and will often connect via either USB or HDMI.

Stay Connected with a Data Link

The other important drone component for real-time video streaming is the data link. The data link is what allows you to transfer your video from the drone to your viewing application. In almost all cases this will be a wireless connection unless your drone is tethered somehow. There are a variety of choices in what wireless technology to use. WiFi is a popular choice, 4G / 5G cellular is great for both commercial use and for Beyond Visual Line Of Sight (BVLOS) flights, and Microhard radios are great for outdoor use. A cellular link also has the benefit of direct Internet connectivity. There are many more options to choose from and each one has its pros and cons. Ultimately, for video streaming, you need to understand the bandwidth you can expect from the link in your typical use case to decide how much video you can stream over it and at what quality. These wireless technologies all advertise certain peak data rates but the actual rate you see will vary depending on the distance between the drone and the receiving station, obstructions in the path, interference, etc. ModalAI provides support for all of the most popular wireless network solutions either directly or through add on peripherals.

Set Up a Ground Station

The streamed video will need to be viewed by a software application at a receiving ground station. In some cases this will be integrated directly into the drone controller used by the pilot, but in other cases it could be a laptop computer or, in the case of a cellular modem with direct Internet access, it could be a computer anywhere in the world. This discussion will assume that the popular drone ground control station (GCS) software QGroundControl is being used to view the real time video stream.

voxl-streamer Enables Real-Time Video Streaming

Okay, so now you have a video source, a wireless link to transfer the video, and a receiving station where you can view it. But without something to tie all of these components together you will not be able to view your video. The video streaming solution that ties all of this together and makes real time video streaming possible has a number of different functions to take care of. First, it must be able to accept the digital format of the video images coming into it and, often, convert it into other formats. The video frames coming in are usually in some sort of raw format and are typically very large. This stream of video frames needs to be compressed so that it will not require more bandwidth than what is available on the wireless link. And finally, it needs to be able to support a network protocol to communicate with the viewing application and transfer the data. The voxl-streamer application from ModalAI can handle these core functions and other typical needs such as image rotation. The application is based on the GStreamer Open Source media framework which contains a set of software libraries that can handle all of these individual functions.


Compressing the video stream is one of the most important tasks for the streaming software. There are a variety of ways to lower the data rate of a video stream produced by a camera connected to the drone. The most basic ways involve lowering the frame rate and decreasing the image resolution. If the camera is producing 60 frames per second and you choose to stream only 30 frames per second then you have effectively cut your data rate in half. Likewise, if the camera is producing 720p high resolution images (1280 x 720) and you resize those images to VGA (640 x 360) then you cut the data rate by a factor of 4. Combining frame rate decreases and resolution decreases can result in a significant savings of bandwidth. Beyond that there are some techniques to compress the data even farther.


There are a variety of industry standard techniques available for video compression including Motion JPEG (MJPEG), Advanced Video Coding (AVC), aka H.264, and High Efficiency Video Coding (HEVC), aka H.265. Voxl-streamer supports H.264. This compression method can be tuned in a variety of ways in order to achieve a desired bitrate on the data link. However, care must be taken not to over compress the video or else the resulting stream might be of such low quality that it is not suitable for viewing. The process of compressing the video stream is called video encoding. The encoding can be done by a software algorithm or a hardware accelerator. The VOXL platform includes a hardware accelerator for video encoding and voxl-streamer makes use of this. Since the encoding algorithm is compute intensive it helps to move it into a separate hardware accelerator. For those technical folk that are curious about this, it is done via the OpenMAX API using the GStreamer plugin omxh264enc.


The final part of all this is the protocol to connect the drone with the viewing application. Once again there are a multitude of options available for this! The one that seems to be used frequently in the drone space and is supported by QGroundControl, VLC, and many other viewing applications is called the Real Time Streaming Protocol (RTSP). Using this protocol, the voxl-streamer application configures a unique URI (e.g. rtsp://192.168.1.1:8900/live) that a viewing application can connect to over the data link in order to access the video stream.


In many cases there will be multiple cameras attached to the drone. You can run multiple instances of voxl-streamer to produce a unique RTSP URI for each one. Using the correct URI you can then stream from any camera and dynamically switch between them. You can even have multiple streams active at the same time. Our default viewing application, QGroundControl, can be configured to view the desired stream by manually updating the URI on the configuration screen. However, there is also a way to associate camera names to URIs such that you can choose the camera name from a drop down menu and it will automatically connect to the correct RTSP URI. This is all accomplished via the camera protocol in Mavlink, a well known open source messaging protocol. ModalAI provides a useful application called mavlink-camera-manager to help set this up.

Available Soon! Stay in the Loop

Voxl-streamer will be available in our next software release. Be sure to sign up for our newsletter to be notified about new product and software releases. Of course, our software is open source so if this doesn’t quite fit your particular application you can modify it as needed. Please reach out and let us know what you are doing with video streaming as we are always excited to hear what cool things other people are doing and how we can improve our feature set. For a exclusive access to the beta version of voxl-streamer, emailcontact@modalai.com

Reference:

 

Share article on LinkedIn