[Part 2] Home Appliance Automation with Hand Gestures

In our recent blog we have talked about software installation guide to setup and begin automation of home appliances that can be used to detect human hand gesture to control home appliances.

We are now in the 2^nd part of the blog where we intend to develop an application that is built on RASBERRY PI using java and computer vision

n APIs. It will detect the hand gesture and consequently pass the signals to IR device which will then perform appropriate home automation.

This blogs guides you with the complete procedure for detecting human hand gesture and interacting with hardware/home appliances.

Let us first have a look at the role of the application in implementing hand gesture recognition for home automation.

Process

• Application and IR device must be part of same network.
• Application capture camera input continuously.
• From captured image frames, Hand gesture is detected using OpenCV data structure/algorithms.
• Appropriate signals are found and consequently passed on to IR device (HTTP call) with proper command (Ex ON/OFF) to external device.

Below are the core functionalities and modules that an application contains:

• Gesture detection – The application contains open computer vision APIs. OpenCV and JavaCV process the camera input and apply algorithms to process their frames.
• Smart Config – It is the heart of application that can talk with IR device to control external device.This module has two modes: Learning and Execution. During Learning mode, It configures IR device using TI Smart config protocol. it monitors and records data sequence for different infrared sequence for home appliances operations. During Execution mode, it can send the recorded sequences to IR device, which are converted by IR device into infrared sequence to control different devices.
• Signal Processing – It is a remote processing part of the application, in which application stores the signals and passes it to IR device for further operation.

Let us have a look at the technologies used in implementing hand gesture recognition.

OpenCV

In the project we use OpenCV contours for recognizing the hand image.

Contours can be explained simply as a curve joining all the continuous points (along the boundary), having same color or intensity. The contours are a useful tool for shape analysis and object detection and recognition. For better accuracy, use binary images. So before finding contours, apply threshold or canny edge detection. findContours function modifies the source image. So if you want source image even after finding contours, already store it to some other variables.

In OpenCV, finding contours is like finding white object from black background. So remember, object to be found should be white and background should be black.

Method / Algorithms to detect hand from video frame:

Briefly let us look at the several ways to process a frame to detect the hand and their gestures:

Background Subtraction– A simple method to start with is we can subtract the pixel values. However this will result in negative values and values greater than 255, which is the maximum value used to store an integer. And what if we have a black background? Nothing gets subtracted in that case. Instead we use an inbuilt background sub-tractor based on a Gaussian Mixture-based Background/Foreground Segmentation Algorithm. Background subtraction involves calculating a reference image, subtracting each new frame from this image and threshold the result which results is a binary segmentation of the image which highlights regions of non-stationary objects .We then use erosion and dilation to enhance the changes to make it more prominent.

Color detection – The skin color detection is one of the most popular methods. This method is simple and depends on skin color that can be white, black, or other colors, and the environment light conditions, as well as the background.

Edge detection– It is an edge detection operator that uses a multi-stage algorithm to detect  wide range of edges in images, It’s aim is to satisfy the below criteria:

• Lower Error RateGood
• Localization
• Minimal Response

Note – For our solution we have used color detection method to detect hand from frame with user skin color. Application itself has GUI to detect user hand skin color and persist for further application uses.

After analyzing the application side requirements let’s move on to middle level device for further communication called IR device.

IR Device

IR device works as IR transreceiver. It is used to record IR command for different appliances and that recorded sequence is used to control appliances based on different gestures detected in previous phase. IR device can be connected to TV, set-top boxes, stereos, air conditioners and other appliances and then can pass infrared remote control signal and start the home appliance.

After understanding the all the devices, lets now understand the procedure to develop the application.

Guide to develop the Application

Home automation is the residential extension of building automation and involves the control and automation of lighting, air conditioning, and security, as well as home appliances such as washer/dryers, television, ovens or refrigerators/freezers use Wi-Fi for remote monitoring.

Modern systems generally consist of switches and sensors connected to a central hub sometimes called a “gateway” from which the system is controlled with a user interface that is interacted either with a wall-mounted terminal, mobile phone software, tablet computer or a web interface, often but not always via internet cloud services.

The application we have developed has a user interface which is able to manage devices and communication path. It has a robust feature to make device as habituate by their Learning Mode and Execution Mode.

SmartConfig

It’s a Wi-Fi setup process that allows multiple in-home devices to connect to Wi-Fi networks quickly and efficiently. It is specifically for applications that typically do not have a display or keyboard to enter network name and password.

SmartConfig interface gives end users the ability to easily connect IR devices to an access point. Through a simple SmartConfig method/interface, consumers can use any programming concept to create own application using SmartConfig Interface.

Connect IR device / hardware with application, is a first step to connect IR device to application by sending Broadcast Datagram packets with network (Wi-Fi) details (SSID and Password).

This process is able to send and receive broadcast datagram packets and make connection with application and IR hardware device.  Application has a GUI (Graphical User Interface) that allow user to connect device.

Application first check whether any IR device is already connected, if not than it will prompt a window to connect device with some validations as.

Application creates a socket connection with host (Router) and sends broadcast datagram packets with SSID and password to get connected with IR device and wait for acknowledgment from IR smart device.

Device learning – After IR device is connected with the application, the user is able to setup commands/signal for home appliances based on their hand gesture, and it starts with a learning step.

In learning phase user is able to setup one or multiple instructions or command for multiple devices using GUI of application.

Our current implementation has basic gestures like 0-5 count, up-down, left-right. Based on user configuration, application will operate their attached devices.

For e.g.:  Increase TV volume on Up-Down movement of hand.

After learning phase, you are ready to operate your home appliances with your hand gestures. Now user can Switch On, Off the TV and also manage volume in our application implementation

We have configured 6-7 command to operate TV as: ON, OFF, Volume Up- Down, Mute, Open menu bar, Exit menu bar and their operations.

Signal Processing (Remote service)- Our application is able to detect hand gesture and pass the IR signal to configured home appliances according to human hand gesture.

Application is capable of managing User interaction, IR device communication, signal and skin color persistence.

Application gateway is incorporated to connect smart home appliances, and an IOT connection management platform is required to enable various applications.

These applicable solutions include: home automation, which idyllically promotes happier, and more comfortable and convenient lifestyle.

The solution will help operators to transform from traditional home broadband services to provide intelligent home services.