All posts by Adam Chan

Drone Programming – Video Capturing

Alex's Post, Drone Control

We introduced Drone movement in the last post, we are going to try the video capture. It is also very simple with the help of DJITELLOPY and CV2 API. With the basic movement control and video capture, we can start face detection and face tracking very soon. Let’s show you how to capture video from the Drone Tello.

CV2

CV2, stands for OpenCV (Open Source Computer Vision Library), which is a popular open-source computer vision and machine learning software library. It provides a wide range of functions and tools for various computer vision tasks, image and video processing, machine learning, and more. OpenCV is widely used in the fields of computer vision, robotics, image processing, and artificial intelligence.

As we mentioned in the previous post, CV2 already installed when we installing the DJITELLOPY 2.50 package. We just need to import the CV2 library when start the program.

Threading

As we need to capture and display the video from drone by the same time it is flying, we need parallel progressing. THREADING is basic and common used Python.

How’s the program working

# Before you run this program, ensure to connect Tello with the WIFI

# Import Tello class from djitellopy library
from djitellopy import Tello

# Import additional library CV2 - OpenCV for image processing, threading for multi-tasking
import cv2
import threading

# Assign tello to the Tello class
tello = Tello()

# def a video capture and display function
def capturing_video(tello):
    while True:
        frame = tello.get_frame_read().frame
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        cv2.imshow('Tello Video Stream', frame)
        cv2.moveWindow('Tello Video Stream',0,0)
        cv2.waitKey(1)

# Connect to the drone via WIFI
tello.connect()

# Instrust Tello to start video stream and ensure first frame read
tello.streamon()
while True:
            frame = tello.get_frame_read().frame
            if frame is not None:
                break

# Start the video capture thread when the drone is flying
video_thread = threading.Thread(target=capturing_video, args=(tello,), daemon=True)
video_thread.start()

# Take off the drone
tello.takeoff()

# Do combo action such as move up & down and rotating
tello.move_up(30)
tello.move_down(30)
tello.move_up(30)
tello.move_down(30)

tello.rotate_counter_clockwise(30)
tello.rotate_clockwise(60)
tello.rotate_counter_clockwise(30)

# Landing the drone
tello.land()

# Stop the video stream
tello.streamoff()

# Show the battery level before ending the program
print("Battery :", tello.get_battery())

# Stop the connection with the drone
tello.end()
Python

Video Stream from Tello

# Instrust Tello to start video stream and ensure first frame read
tello.streamon()
while True:
            frame = tello.get_frame_read().frame
            if frame is not None:
                break
Python

With DJITELLOPY to achieve the video stream is very simple, we use streamon() to instruct Tello to enable the video stream and get_frame_read() to read the existing video frame.

However, there may be a delay after calling streamon() in the DJITELLOPY library. When we call streamon(), we are initiating the video streaming from the Tello drone’s camera. The drone needs some time to establish the streaming connection and start sending video frames.

So, we setup a while loop to ensure the camera is ready and the first frame is being read before we proceed to the next step.

The get_frame_read() method in the djitellopy library returns a VideoFrame object that provides access to the current video frame from the Tello drone’s camera. Apart from the frame attribute, which contains the video frame data as a NumPy array, the VideoFrame object has other attributes that provide information about the frame. These attributes include:

  • frame: The actual video frame data as a NumPy array.
  • time: The timestamp of the frame in milliseconds.
  • frame_number: The sequential number of the frame.
  • h: The height of the video frame.
  • w: The width of the video frame.
  • channel: The number of color channels in the frame (usually 3 for RGB).

Video Capture and display

# def a video capture and display function
def capturing_video(tello):
    while True:
        frame = tello.get_frame_read().frame
        frame = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        cv2.imshow('Tello Video Stream', frame)
        cv2.moveWindow('Tello Video Stream',0,0)
        cv2.waitKey(1)
Python

We define a function to read the frame from the drone and show as image in a separated window by CV2. There are two highlights,

  1. The frame we capture from Tello is in RGB colors but cv2 processing image as ‘BGR’ order. If we show the image directly, it will result as something bluish (Smurfs?). We need to convert this from RGB to BGR before we can show this properly.
  2. cv2.waitKey() must be excuted after the cv2.imshow(), otherwise, the image will not be displayed.

Parallel Processing

Since we need to capture the frame and display during the drone flying, we need parallel processing for capture_video().

# Start the video capture thread when the drone is flying
video_thread = threading.Thread(target=capturing_video, args=(tello,), daemon=True)
video_thread.start()
Python

We started the video capture thread just before the drone takeoff, so that we can see the video window when the drone take off and flying.

Again, that’s cool and easy, we just add few more lines to the last program and make it fly with video capturing. We believe that we can start doing face detection and tracking now.

# of view 10,393

Drone Programming – Ryze Tello Introduction

Drone Control

We are starting a new page for Drone programming by Python, Ryze Tello is a very good price drone. With their SDK, which makes it very easy to program by different platforms such as Scratch and Python. We picked this to start our journey of Drone Programming, and hope that we can achieve Face recognition and Gesture-based control as our ultimate target.  Then, Swarm with Tello Edu – multiple drone control and performance.

SDK & DJITELLOPY API

Ryze Tello already provide SDK to connect the drone via a WIFI UDP port, allows users to control the drone with text command. You can refer to this link for their SDK 2.0 document and below is the example how’s the SDK working.

# Before you run this program, ensure to connect Tello with the WIFI

import socket
import time

# Tello IP and port
Tello_IP = '192.168.10.1'
Tello_PORT = 8889

# Create a UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind(('', Tello_PORT))

# Function to send commands to the drone
def send_command(command):
    sock.sendto(command.encode(), (Tello_IP, Tello_PORT))
    
def receive_response():
    response, _ = sock.recvfrom(256)
    return response.decode()


# Connect to the Tello drone via WIFI
send_command('command')
receive_response()

# Take off the drone
send_command('takeoff')
receive_response()

# Do combo action such as move up & down and rotating
send_command('up 30')
receive_response()
send_command('down 30')
receive_response()
send_command('up 30')
receive_response()
send_command('down 30')
receive_response()

send_command('cw 30')
receive_response()
send_command('ccw 60')
receive_response()
send_command('cw 30')
receive_response()

# Landing the drone
send_command('land')
receive_response()

# Show the battery level before ending the program
send_command('battery?')
print("Battery :", receive_response())

# Stop the connection with the drone
send_command('End')
Python

To make life easier and the program easy to read, there are few third parties created library to support the SDK, such as EASYTELLO, TELLOPY, and DJITELLOPY. We picked DJITELLOPY from Murtaza’s Workshop – Robotics and AI (Thank you!) for our project.

First of all, go ahead to install DJITELLOPY into your Python by using command ‘PIP install djitellopy‘ in your terminal. The latest version I can download right now is 2.50 including the following libraries, DJITELLOPY, NUMPY, OPENCV-PYTHON, AV and PILLOW.

‘Hello World’ from Tello

OK, let start our first drone programming try with the code below.

# Before you run this program, ensure to connect Tello with the WIFI

# Import Tello class from djitellopy library
from djitellopy import Tello

# Assign tello to the Tello class
tello = Tello()

# Connect to the Tello drone via WIFI
tello.connect()

# Take off the drone
tello.takeoff()

# Do combo action such as move up & down, rotating and flipping
tello.move_up(30)
tello.move_down(30)
tello.move_up(30)
tello.move_down(30)

tello.rotate_counter_clockwise(30)
tello.rotate_clockwise(60)
tello.rotate_counter_clockwise(30)

# Landing the drone
tello.land()

# Show the battery level before ending the program
print("Battery :", tello.get_battery())

# Stop the connection with the drone
tello.end()
Python

It is pretty simple, right? Just import the DJITELLOPY API and you can use those command control the drone Tello. The first thing you must do is to ‘tello.connect()’ to the Tello, so that it is asked to be in SDK mode and accept different commands. Then, I just have the drone takeoff and do some combo as shown in the code. Finally, I get the battery information and display before complete the process.

Next step, we will learn how to capture video from the drone.

# of view 16,849

Mice Maze

Features Projects, Makecode & Scratch

My Second dream was ‘mice maze’, it was very popular when I was childhood, program a ‘mice’ to solve the maze automatically. Actually, my 25 years old final year project was ‘smart vacuum cleaner’, it is now very popular. But, it was very new concept at that time and I defined some ‘mice’ behavior. And now, I proof that it work based on Scratch.

Scratch, thanks a again!!

P.S. I like to stare the screen to see the mice to run the path. If you want to run it faster, hold shift when press the green flag.. 🙂

Basic idea for the mice’s logic

That’s a very basic and simple logic for the mice…. XD

  1. We make the mice in a square with dimension same as the maze path, we are using 48pt x 48pt.
  2. The mice is moving in a unit of it’s size, i.e. 48pt
  3. We define the priority of the movement for the mice to follow,
    • Always go forward
    • Go right if cannot go forward
    • Go left if both forward and right are blocked
    • Move backward if no more movement.
  4. Then, we need to record each movement and possible (not explore) directions.
  5. When the mice move backward, it will read the record and go to un-explore directions.

How does the program work

Prior I created the mice maze, I actually made a program by using WASD key to move the mice, solve the maze manually. So, keep in mind, W – up, D – right, S – down, A – left, it is too hard for you if you are FPS game player.. LOL

Sensor – detecting wall

Before the ‘mice’ do any movement, it will detect any wall in it’s four directions. In this program, I move the ‘mice’ to four directions in a certain distance. See if it will hit a wall (black in color) or not. If not hit, W A S & D will be put into the corresponding variable Wall U, D, R & L, tell the ‘mice’ that the direction is OK to move. Otherwise, keep the variable as X, means there is a wall.

Movement

Path Tracking

I am using a LIST called ‘Path’ to record each movement of the ‘mice’,

# of view 10,489

Frog vs Spider

Makecode & Scratch

Scratch made Adam’s dream came, create a game… Since my childhood, I was expert in Assembly, i.e. 6502, x86, 8051, modifying OS and even created a mini Chinese display system. But, never created a single game.. I tried to make game with Assembly, it was painful even make a character moving.. Thanks Scratch!!

# of view 2,915