We introduced Drone movement in the last post, we are going to try the video capture. It is also very simple with the help of DJITELLOPY and CV2 API. With the basic movement control and video capture, we can start face detection and face tracking very soon. Let’s show you how to capture video from the Drone Tello.

CV2

CV2, stands for OpenCV (Open Source Computer Vision Library), which is a popular open-source computer vision and machine learning software library. It provides a wide range of functions and tools for various computer vision tasks, image and video processing, machine learning, and more. OpenCV is widely used in the fields of computer vision, robotics, image processing, and artificial intelligence.

As we mentioned in the previous post, CV2 already installed when we installing the DJITELLOPY 2.50 package. We just need to import the CV2 library when start the program.

Threading

As we need to capture and display the video from drone by the same time it is flying, we need parallel progressing. THREADING is basic and common used Python.

How’s the program working

# Before you run this program, ensure to connect Tello with the WIFI

# Import Tello class from djitellopy library
from djitellopy import Tello

# Import additional library CV2 - OpenCV for image processing, threading for multi-tasking
import cv2
import threading

# Assign tello to the Tello class
tello = Tello()

# def a video capture and display function
def capturing_video(tello):
    while True:
        frame = tello.get_frame_read().frame
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        cv2.imshow('Tello Video Stream', frame)
        cv2.moveWindow('Tello Video Stream',0,0)
        cv2.waitKey(1)

# Connect to the drone via WIFI
tello.connect()

# Instrust Tello to start video stream and ensure first frame read
tello.streamon()
while True:
            frame = tello.get_frame_read().frame
            if frame is not None:
                break

# Start the video capture thread when the drone is flying
video_thread = threading.Thread(target=capturing_video, args=(tello,), daemon=True)
video_thread.start()

# Take off the drone
tello.takeoff()

# Do combo action such as move up & down and rotating
tello.move_up(30)
tello.move_down(30)
tello.move_up(30)
tello.move_down(30)

tello.rotate_counter_clockwise(30)
tello.rotate_clockwise(60)
tello.rotate_counter_clockwise(30)

# Landing the drone
tello.land()

# Stop the video stream
tello.streamoff()

# Show the battery level before ending the program
print("Battery :", tello.get_battery())

# Stop the connection with the drone
tello.end()

Video Stream from Tello

# Instrust Tello to start video stream and ensure first frame read
tello.streamon()
while True:
            frame = tello.get_frame_read().frame
            if frame is not None:
                break

With DJITELLOPY to achieve the video stream is very simple, we use streamon() to instruct Tello to enable the video stream and get_frame_read() to read the existing video frame.

However, there may be a delay after calling streamon() in the DJITELLOPY library. When we call streamon(), we are initiating the video streaming from the Tello drone’s camera. The drone needs some time to establish the streaming connection and start sending video frames.

So, we setup a while loop to ensure the camera is ready and the first frame is being read before we proceed to the next step.

The get_frame_read() method in the djitellopy library returns a VideoFrame object that provides access to the current video frame from the Tello drone’s camera. Apart from the frame attribute, which contains the video frame data as a NumPy array, the VideoFrame object has other attributes that provide information about the frame. These attributes include:

• frame: The actual video frame data as a NumPy array.
• time: The timestamp of the frame in milliseconds.
• frame_number: The sequential number of the frame.
• h: The height of the video frame.
• w: The width of the video frame.
• channel: The number of color channels in the frame (usually 3 for RGB).

Video Capture and display

# def a video capture and display function
def capturing_video(tello):
    while True:
        frame = tello.get_frame_read().frame
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        cv2.imshow('Tello Video Stream', frame)
        cv2.moveWindow('Tello Video Stream',0,0)
        cv2.waitKey(1)

We define a function to read the frame from the drone and show as image in a separated window by CV2. There are two highlights,

1. The frame we capture from Tello is in RGB colors but cv2 processing image as ‘BGR’ order. If we show the image directly, it will result as something bluish (Smurfs?). We need to convert this from RGB to BGR before we can show this properly.
2. cv2.waitKey() must be excuted after the cv2.imshow(), otherwise, the image will not be displayed.

Parallel Processing

Since we need to capture the frame and display during the drone flying, we need parallel processing for capture_video().

# Start the video capture thread when the drone is flying
video_thread = threading.Thread(target=capturing_video, args=(tello,), daemon=True)
video_thread.start()

We started the video capture thread just before the drone takeoff, so that we can see the video window when the drone take off and flying.

Again, that’s cool and easy, we just add few more lines to the last program and make it fly with video capturing. We believe that we can start doing face detection and tracking now.

We are starting a new page for Drone programming by Python, Ryze Tello is a very good price drone. With their SDK, which makes it very easy to program by different platforms such as Scratch and Python. We picked this to start our journey of Drone Programming, and hope that we can achieve Face recognition and Gesture-based control as our ultimate target.  Then, Swarm with Tello Edu – multiple drone control and performance.

SDK & DJITELLOPY API

Ryze Tello already provide SDK to connect the drone via a WIFI UDP port, allows users to control the drone with text command. You can refer to this link for their SDK 2.0 document and below is the example how’s the SDK working.

# Before you run this program, ensure to connect Tello with the WIFI

import socket
import time

# Tello IP and port
Tello_IP = '192.168.10.1'
Tello_PORT = 8889

# Create a UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', Tello_PORT))

# Function to send commands to the drone
def send_command(command):
    sock.sendto(command.encode(), (Tello_IP, Tello_PORT))
    
def receive_response():
    response, _ = sock.recvfrom(256)
    return response.decode()


# Connect to the Tello drone via WIFI
send_command('command')
receive_response()

# Take off the drone
send_command('takeoff')
receive_response()

# Do combo action such as move up & down and rotating
send_command('up 30')
receive_response()
send_command('down 30')
receive_response()
send_command('up 30')
receive_response()
send_command('down 30')
receive_response()

send_command('cw 30')
receive_response()
send_command('ccw 60')
receive_response()
send_command('cw 30')
receive_response()

# Landing the drone
send_command('land')
receive_response()

# Show the battery level before ending the program
send_command('battery?')
print("Battery :", receive_response())

# Stop the connection with the drone
send_command('End')

To make life easier and the program easy to read, there are few third parties created library to support the SDK, such as EASYTELLO, TELLOPY, and DJITELLOPY. We picked DJITELLOPY from Murtaza’s Workshop – Robotics and AI (Thank you!) for our project.

First of all, go ahead to install DJITELLOPY into your Python by using command ‘PIP install djitellopy‘ in your terminal. The latest version I can download right now is 2.50 including the following libraries, DJITELLOPY, NUMPY, OPENCV-PYTHON, AV and PILLOW.

‘Hello World’ from Tello

OK, let start our first drone programming try with the code below.

# Before you run this program, ensure to connect Tello with the WIFI

# Import Tello class from djitellopy library
from djitellopy import Tello

# Assign tello to the Tello class
tello = Tello()

# Connect to the Tello drone via WIFI
tello.connect()

# Take off the drone
tello.takeoff()

# Do combo action such as move up & down, rotating and flipping
tello.move_up(30)
tello.move_down(30)
tello.move_up(30)
tello.move_down(30)

tello.rotate_counter_clockwise(30)
tello.rotate_clockwise(60)
tello.rotate_counter_clockwise(30)

# Landing the drone
tello.land()

# Show the battery level before ending the program
print("Battery :", tello.get_battery())

# Stop the connection with the drone
tello.end()

It is pretty simple, right? Just import the DJITELLOPY API and you can use those command control the drone Tello. The first thing you must do is to ‘tello.connect()’ to the Tello, so that it is asked to be in SDK mode and accept different commands. Then, I just have the drone takeoff and do some combo as shown in the code. Finally, I get the battery information and display before complete the process.

Next step, we will learn how to capture video from the drone.