MariaDB Galera cluster

For some time now I have been trying to get a MySql NDB cluster up and running. I have tried installing it from repositories as well as binaries… and I have failed every single time at one point or another. I usually got the management node up and running, but the data nodes and API (SQL) nodes were a completely different matter.

In the end I was forced to face the fact that I was unable to install it successfully, but it is such a shame to surrender the cluster entirely. There are a number of clustering options out there. I won’t even start to get into the different version, but the cluster I ended up installing successfully was the MariaDB Galera cluster.

I still had to restart the installation multiple times because I made a mistake here and there. You have just got to LOVE virtual computers and their ability to revert to different snapshots.

Setup

I am running the ESXi vSphere hypervisor on a Dell T3500 workstation with 12Gb of ram.

In the ESXi host I had prevously installed 5 machines to install the NDB cluster. Each of these machines was given 2 cores and 2Gb of RAM. Instead of reinstalling the machines from scratch I choose to go back to the first, more or less clean. snapshot of the machine and start the installation process from there.

From my regular computer I am using the MobaXterm (https://mobaxterm.mobatek.net/) client to connect to the different virtual computers through a SSH prompt.

In my Galera cluster I started out using three nodes. Unlike the NDB cluster where you have Management nodes, API nodes and data nodes the Galera cluster sinply has a number of nodes that replicate synchronously.

There are a number of steps where you need to do things to diverse configuration files. Personally I hate the VI editor that comes with Centos7 as a default and choose to install the nano editor instead. You should be able to install it by writing

yum install nano

Initial issues

One of the major issues I have faced while trying to install the cluster was the simple fact that a lot of tutorials online was old and outdated. After trying to use those tutorials I ended up more than once having to revert back to a clean image to start all over again.

In the end I managed to find a very good video tutorial in youtube that was fairly recent. What made it even better was the fact that the guy did EVERY step in the installation so that you could really follow everything from scratch. A lot of other tutorials tend to skip over “easy” stuff that ends up being the thing that comes back and bites you in the a** later on. The tutorial was a series in 7 parts going from initial setup to complete install.

Comparing to my attemt at installing the NDB cluster however the installation of the MariaDB Galera cluster was very straight forward and relativly easy… even for someone like me who is no Linux wiz.

Disclaimer

This tutorial is more in the way of making me remember what I have done as well as getting a quick way to get a MariaDB Galera Cluster up and running in a short amount of time. It may or may not work for you, and it will require a clean install.

This is only a test cluster. There is probably a ton of setting you need to do to get it production worthy, so don’t expect to be able to have a perfect HA cluster after doing the steps in this tutorial, but something that may become a HA cluster with a bit of tweking.

The first part of the tutorial will contains a bit more text with commands in italic and configuration files and printouts as blockquotes. At the end I will redo the commands in basicly a copy paste fassion.

Initial setup

There are a couple of things that needs to be done prior to the installation.

In the tutorials I have read/seen they usually tell you to disable the firewall and selinux. To me that sounds like a bad way of doing things. So far I have been able to get the firewall rules up and running, but there is still work that needs to be done to get the selinux working.

For this tutorial I will set the selinux to be permissive instead of disables, because that way it will keep on logging the things it would normally block if it was enforcing.

Most of this tutorial require that you do each of the steps on each of the nodes in the cluster, but some steps should be done only to one of the cluster nodes. If nothing is written then assume that whatever you do to one you do to the others.

Setup SELinux

First we need to edit the selinux configuration file

nano /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing – SELinux security policy is enforced.
# permissive – SELinux prints warnings instead of enforcing.
# disabled – No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
# targeted – Targeted processes are protected,
# minimum – Modification of targeted policy. Only selected processes are protected.
# mls – Multi Level Security protection.
SELINUXTYPE=targeted

After updating the configuration file we need to restart the computer for it to reload the configuration.

shutdown -r now

After rebooting the computer check the se status to see if the change was successful

sestatus

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

Configuring SELinux

This is where I want to do more to be able to set SELinux to be enforcing to increase the security but so far I have only come a short way on the subject so I may or may not post an additional note on how I did this part.

Swap space

For this test system the system swap space will be more than enough, but in a real world application the amount of RAM and also swap space is very important to the Galera cluster. The cluster stores a lot of information in order to provide the synchronization. It will start out using RAM, but if there is not enough RAM the cluster will start to save the information to the swap space. I will not be going through how to set up more swap space but here is a nice link to how it is done.

https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-centos-7

Hosts file

While setting the cluster I will be using host names for the computers in the network. I am currently not sure if this is required, but there are certain parameters in the server configuration file that wants you to enter the node name.

nano /etc/hosts

If this is a fresh install then this file should contain only two lines

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6

We need to add our cluster computers to the file

192.168.0.10    gcluster01
192.168.0.11    gcluster02
192.168.0.12    gcluster03

Test the connectivity from each of the nodes using the ping command.

ping gcluster01
ping gcluster02
ping gcluster03

OK, so now we have done the first part of the tutorial to prepare the different nodes for the coming installation and configuration

Installation

Now we need to install MariaDB. From MariaDB version 10.1 the Galera cluster is part of the main installation. Earlier versions had specific Galera packages. The MariaDB installation packages was not part of the yum repository be default so it was nessessary to add the repositories manually.

Adding the MariaDB repository

First we need to add a link to the repository

nano /etc/yum.repos.d/MariaDB.repo

This will open a blank file, copy the blockquote below to the new file.

# MariaDB 10.2 CentOS repository list – created 2018-09-21 05:45 UTC
# http://downloads.mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.2/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

A very good resource to find out exactly what this should be is the MariaDB foundation. This is where I got the above repository information. If you use a different linux distro then this may be different. This is the link I used

https://downloads.mariadb.org/mariadb/repositories/#mirror=lund&distro=CentOS&distro_release=centos7-amd64–centos7&version=10.2

Installing MariaDB

Time to do the actual install.

yum install MariaDB-server MariaDB-client MariaDB-common

After the installation completes you can check to see which components that was installed

rpm -qa | grep -i mariadb

MariaDB-compat-10.2.17-1.el7.centos.x86_64
MariaDB-client-10.2.17-1.el7.centos.x86_64
MariaDB-common-10.2.17-1.el7.centos.x86_64
MariaDB-server-10.2.17-1.el7.centos.x86_64

Setting up firewall rules

In most tutorials people just tell you to disable the firewall to make things work. You can do this by entring

systemctl disable firewalld

Personally I think this approach is kind of scary. Sure it may work for a test system, but shutting off the firewall seem like a really bad idea if you ask me.

The Galera cluster use 4 ports for replication. The first one is the standard connection port (default is 3306) the rest of the ports used are

4567 TCP and UDP
4568 TCP
4444 TCP

The Galera cluster also require that you set the service up in the firewall. The commands I used was

firewall-cmd –zone=public –add-service=mysql –permanent
firewall-cmd –zone=public –add-port=3306/tcp –permanent
firewall-cmd –zone=public –add-port=4567/tcp –permanent
firewall-cmd –zone=public –add-port=4568/tcp –permanent
firewall-cmd –zone=public –add-port=4444/tcp –permanent
firewall-cmd –zone=public –add-port=4567/udp –permanent
firewall-cmd –reload

The last line will reload the firewall using the new ports. This can be archived by restarting the computer as well I guess, but why do it when there is no need for it.

Basic Configuration

If everything has went according to plan it is now time to do a basic configuration. We need to set MariaDB up as well as configure the cluster.

DO THE REST OF THIS SECTION ON ONE NODE ONLY!

systemctl start mariadb

Run the secure installation of MariaDB.

If this is a fresh install then the root password will be blank. While the command is running choose to remove anonymous access, test database etc. Basicly answer yes to all questions that the secure install asks you.

mysql_secure_installation

Basic Galera Configuration

When you first open the configuration all galera settings will be commented out. In the config file below I have marked the lines that has changed or been added in bold text.

nano /etc/my.cnf.d/server.cnf

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]

# this is only for the mysqld standalone daemon
[mysqld]

#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=”gcomm://192.168.0.10,192.168.0.11,192.168.0.12″
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
bind-address=0.0.0.0

#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0

wsrep_cluster_name=”MariaDBCluster”
wsrep_sst_method=rsync
wsrep_node_address=”192.168.0.10″
wsrep_node_name=”gcluster01″

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.2 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don’t understand
[mariadb-10.2]

The setting wsrep_cluster_address should contain all the IPs of the nodes of the cluster. gcomm implies that it is a group communication.

The settings wsrep_node_address and wsrep_node_name is node specific and must be changed to the IP and hostname of each node.

The setting bind-address can be set to listen to a specific IP. With the setting set as 0.0.0.0 it will listen to any IP address on the node.

In order for the replication to work you must make sure that the replication provider in setting wsrep_provider exists.

ls -al /usr/lib64/galera/libgalera_smm.so

-rwxr-xr-x. 1 root root 34691616 13 feb 2018 /usr/lib64/galera/libgalera_smm.so

Starting the cluster

Installation and basic configuration is now done. It is time to start the cluster.

First node

This is the node where you started MariaDB at an earlier stage of the tutorial. After configuring the cluster we now need to stop MariaDB and start it as a cluster instead.

systemctl stop mariadb

Create a new cluster and start MariaDB again by entring

galera_new_cluster

If everything goes as planned and MariaDB starts without any issues then it is time to see if our new cluster is up and running.

First we need to connect to MariaDB

mysql -uroot -p

Enter the password you created during the secure install.

We then wish to see the size of our cluster. At this point in time it should be 1.

show status like ‘wsrep_cluster_size’;

+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 1     |
+--------------------+-------+
1 row in set (0.00 sec)

Second and third node

If everything is set up the way it should be all you have to do to join the node to the cluster is to start MariaDB

systemctl start mariadb

Connect to MariaDB to check the cluster

mysql -uroot -p

Enter the same password as you used on the first node.

show status like ‘wsrep_cluster_size’;

+--------------------+-------+
| Variable_name      | Value |
+--------------------+-------+
| wsrep_cluster_size | 2     |
+--------------------+-------+
1 row in set (0.00 sec)

This time the cluster size should be 2. Go back to the first node and rerun the query. The result should show you that the cluster size is 2 on the first node as well.

For the third node simply replicate the same steps as in node 2.

Checking replication

The first sign that replication is up and running and working is the fact that you could log into all nodes even if you only ran the secure installation on the first node. The user and password has replicated to the other nodes, but to be able to check the actual table replication you could do something simple like this

Log into mysql on one of the nodes

mysql -uroot -p

CREATE DATABASE db1;
USE db1;
CREATE TABLE testtable (c1 INT);
INSERT INTO testtable (c1) VALUES (1);

Now log into the server on the other nodes and check to see if the new database and table has been replicated successfully.

mysql -uroot -p

USE db1;
SELECT * FROM testtable;

+------+
| c1   |
+------+
| 1    |
+------+

That is it. The cluster should now be up and running!

The quick copy paste way

This is more a copy paste of the above tutorial. This will not do specific checks to see if the cluster is up and running nor do any replication checks. It is only there to speed the installation up.

Initial setup

nano /etc/selinux/config

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing – SELinux security policy is enforced.
# permissive – SELinux prints warnings instead of enforcing.
# disabled – No SELinux policy is loaded.
SELINUX=permissive
# SELINUXTYPE= can take one of three two values:
# targeted – Targeted processes are protected,
# minimum – Modification of targeted policy. Only selected processes are protected.
# mls – Multi Level Security protection.
SELINUXTYPE=targeted

shutdown -r now

sestatus

SELinux status: enabled
SELinuxfs mount: /sys/fs/selinux
SELinux root directory: /etc/selinux
Loaded policy name: targeted
Current mode: permissive
Mode from config file: permissive
Policy MLS status: enabled
Policy deny_unknown status: allowed
Max kernel policy version: 31

nano /etc/hosts

127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.0.10    gcluster01
192.168.0.11    gcluster02
192.168.0.12    gcluster03

ping gcluster01
ping gcluster02
ping gcluster03

Installing MariaDB

nano /etc/yum.repos.d/MariaDB.repo

# MariaDB 10.2 CentOS repository list – created 2018-09-21 05:45 UTC
# http://downloads.mariadb.org/mariadb/repositories/
[mariadb]
name = MariaDB
baseurl = http://yum.mariadb.org/10.2/centos7-amd64
gpgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1

yum install MariaDB-server MariaDB-client MariaDB-common

rpm -qa | grep -i mariadb

MariaDB-compat-10.2.17-1.el7.centos.x86_64
MariaDB-client-10.2.17-1.el7.centos.x86_64
MariaDB-common-10.2.17-1.el7.centos.x86_64
MariaDB-server-10.2.17-1.el7.centos.x86_64

Firewall setup

firewall-cmd –zone=public –add-service=mysql –permanent
firewall-cmd –zone=public –add-port=3306/tcp –permanent
firewall-cmd –zone=public –add-port=4567/tcp –permanent
firewall-cmd –zone=public –add-port=4568/tcp –permanent
firewall-cmd –zone=public –add-port=4444/tcp –permanent
firewall-cmd –zone=public –add-port=4567/udp –permanent
firewall-cmd –reload

Basic configuration

ONLY DO THIS TO ONE OF THE NODES!

systemctl start mariadb
mysql_secure_installation

Galera configuration

nano /etc/my.cnf.d/server.cnf

#
# These groups are read by MariaDB server.
# Use it for options that only the server (but not clients) should see
#
# See the examples of server my.cnf files in /usr/share/mysql/
#

# this is read by the standalone daemon and embedded servers
[server]

# this is only for the mysqld standalone daemon
[mysqld]

#
# * Galera-related settings
#
[galera]
# Mandatory settings
wsrep_on=ON
wsrep_provider=/usr/lib64/galera/libgalera_smm.so
wsrep_cluster_address=”gcomm://192.168.0.10,192.168.0.11,192.168.0.12″
binlog_format=row
default_storage_engine=InnoDB
innodb_autoinc_lock_mode=2
#
# Allow server to accept connections on all interfaces.
#
bind-address=0.0.0.0

#
# Optional setting
#wsrep_slave_threads=1
#innodb_flush_log_at_trx_commit=0

wsrep_cluster_name=”MariaDBCluster”
wsrep_sst_method=rsync
wsrep_node_address=”192.168.0.10″
wsrep_node_name=”gcluster01″

# this is only for embedded server
[embedded]

# This group is only read by MariaDB servers, not by MySQL.
# If you use the same .cnf file for MySQL and MariaDB,
# you can put MariaDB-only options here
[mariadb]

# This group is only read by MariaDB-10.2 servers.
# If you use the same .cnf file for MariaDB of different versions,
# use this group for options that older servers don’t understand
[mariadb-10.2]

Check that the galera provider file exists
ls -al /usr/lib64/galera/libgalera_smm.so

-rwxr-xr-x. 1 root root 34691616 13 feb 2018 /usr/lib64/galera/libgalera_smm.so

Starting the cluster

First node (Same node as the one in Basic configuration)

systemctl stop mariadb
galera_new_cluster

Second node

systemctl start mariadb

Third node

systemctl start mariadb

Done!

Raspberry PI as a webcam

There are a couple of pages out there that tells you how to turn your Raspberry PI into streaming web cam and/or surveilance camera. Most of them focus on getting the software up and running, but so far I have not seen anyone with a full rundown on how to turn the PI into the streaming camera. There will be a few comments, but mostly it will be as straight forward as I can possibly make it.

In this tutorial I will be using a Raspberry PI 3 Model B, a 16gb SDHC card and a Logitech C270 webcam. The RPI will be connected to a screen/mouse/keyboard.

Unfortunatly nothing is ever easy, so I will be bringing up a couple of issues I have had to overcome during my deployment.

Operating system

Here you can go two ways. First is to download Raspbian (Jessie at the time of this writing) or you can go with a dedicated Motion detection software called MotionEyeOS.

Raspbian is a fully fletched operating system with all the benefits and drawbacks that comes with it.

MotionEyeOS on the other hand have a much smaller footprint and is specifically made for surveilance. The reason I choose not to go with MotionEyeOS is that after installing it it seemed to require a hardwire connection to even boot and since I did not have one at the time I choose the Raspbian route instead.

In order to install either of them to a SD card you need a SD card reader/writer. Then you need software to write the image to the SD card. I choose Etcher for this, but there are other programs out there that can do the same thing.

Basic Raspbian setup

The out of the box version of Raspbian comes with UK keyboard and timezone. Since I live in Sweden I had to change these settings to match my own. These settings can be found in the menu Preferences/Raspberry PI Configuration and then selecting the Localisation tab.

I also opened up the SSH (command line) and VNC (GUI) interfaces so that I can access the Raspberry without having it hooked up to a screen/mouse/keyboard. This can also be found in the Raspberry PI Configuration under the tab Interfaces. These interfaces will require other software to interact with them, I use PuTTY for SSH and VNC Viewer for VNC.

 

Network

We need to connect to the local network. If you have the RPI connected by Ethernet cable then the connection should be automatic. If you use WIFI however you need to connect to your router manually. This can be done by clicking the network icon in the top right corner of the desktop and selecting your network and then entring the credentials to access it.

Update Raspbian

First of all update Raspbian to the latest version this can be done by opening a terminal window and typing

sudo apt-get update
sudo apt-get upgrade

Let the update/upgrade run its course before continuing.

Static MAC address

In some of the tutorials I have seen online on how to set RPI up as a web cam there have been mentions of finding out what IP address the RPI has. This is all well and good, however it is also a fact that RPI Jessie out of the box will give the on board Ethernet and Wifi network cards random MAC addresses.

What this means in practice is that every time you restart your RPI your IP address will change. I also had a WIFI extender which complicated things immensly so I bashed my head over this part for quite some time before I got it working.

First of all we need to give the RPI a static MAC address. A MAC address is a series of HEX numbers, six to be precise in the following pattern 00:00:00:00:00:00.

To accomplish this I created a shell script that is run after the RPI reboots.

sudo nano /etc/init.d/mac.sh

This will open the nano editor window with a new empty file.

Add this to the file but change the MAC address to something you like. In my case I took a piece of text and converted it to HEX and put that as the MAC address. For instance using the word ‘RaspPI’ as a MAC address would make it 52:61:73:70:50:49

Add this to the mac.sh file

#!/bin/sh

ifconfig eth0 down hw ether 52:61:73:70:50:49
ifconfig eth0 up

ifconfig wlan0 down hw ether 52:61:73:70:50:50
ifconfig eth up

Save the file by pressing Ctrl+x and answer yes to saving the file.

Now we need to give the file permission to be executed

sudo chmod +x /etc/init.d/mac.sh

Finally we need to make this file run when the RPI is rebooted

sudo update-rc.d mac.sh defaults

Done! Now we need to reboot the RPI before we continue.

Installing motion

Motion is a nice piece of software that turns your RPI into a surveilance unit. Basic installation is very simple but there are a lot of settings that can be made. I will bring up the changes I made to make things work but they can be set pretty much however you choose. There are a couple of them that are more important to make things work however, I will highlight these

sudo apt-get install motion

This will install the motion software. Take note that during the installation it will mention that the user motion has been added, and that the user motion has been added to the group video and the group motion. It also tells you about a default folder /var/lib/motion. We will have use of these later on in the tutorial.

Backup the config file

During my trials I made a stupid mistake which made the config file corrupt, which in turn disabled the camera software entirely. I never managed to find the error and fix it so I ended up having to reinstalling Raspbian from scratch and starting all over again… not fun, but it gave me the opportunity to make this detailed tutorial so something good came out of it anyway.

cp /etc/motion/motion.conf /etc/motion/motion.bak

Edit the settings file

Now we need to make som setting changes

sudo nano /etc/motion/motion.conf

The settings file is pretty big, but it is well commented. There are a few default values that have been set to other values than the default value say it should be. These are the settings I changed to get things running.

Daemon

This say that default is off, but it is set to on anyway. This need to be on for the daemon to run.

daemon on

Capture device options

These settings can pretty much be set to taste. Default image size is 320×240 which is different from what is said to be the default value. In my case I made the settings like this

width 640
height 480
framerate 100

Image file output

output_pictures off (on by default which outputs a LOT of images to the output folder)
quality 100

Global network options

stream_quality 100
stream_motion on (I like the 1 image/s unless motion is detected since it conserves bandwidth otherwise it will stream at the given rate or as close to it as possible)
stream_maxrate 10 (Frames per second to be streamed)
stream_localhost off (IMPORTANT if you wish to use the camera outside localhost)

Save the file by pressing Ctrl+x and select yes when asked to save and overwrite the old config file.

Make motion start after boot

We wish to start the motion service after the RPI boots, to do this we need to make a small change in another file.

sudo nano /etc/default/motion

start_motion_daemon=yes (Default value is no)

Save the file by pressing Ctrl+x and answer yes to save and overwrite the old file.

Motion permissions

It is possible to start the server now and get the camera to stream images… however, as soon as a motion is detected the camera daemon will crash and shut down. The reason for this is that the folder /var/lib/motion is created and owned by root, but the user running the motion daemon is the user ‘motion’. The user ‘motion’ does not have permission to write in the default folder which causes the daemon to crash.

What we need to do is to allow the motion user to modify the default folder.

sudo chgrp motion /var/lib/motion
sudo g+rwx /var/lib/motion

Reboot the RPI

Now the server should be up and running. Try by opening the Chromium web browser and navigate to 127.0.0.1:8081

If everything is OK you should get a stream of what your web cam is currently pointing at.

Other issues

One very interresting issue I had during the course of getting my RPI up and running was that the SD card went completely bonkers. It suddenly showed up as being 40Mb instead of 16Gb. Reformating it in windows did not bring the lost space back. What I had to do to fix it was to put the SD card into my Android phone and format it there. The phone also said that it was 40Mb but after formating the original 16Gb was “visible” again.

Good luck!

Camera web server

Ok, so now I have set my webserver up to allow me to access my cameras from outside my LAN without exposing the cameras themselves.

Aprelium Abyss Web Server

I am using a very nice and free web server called Aprelium Abyss Web Server. What I really like about this server is the ease of configuration as well as the posibility to host native .Net apps.

I found this server a number of years ago when I was looking for a way to host a .Net web page and I needed the web server to run on Windows XP. Sure XP had a personal version of IIS, but that was SEVERELY limited and basically did not work the way I wanted it to. So I started looking for option and at least back then they were equally severely limited. Then I happened on this nice web server and presto!

You can run the server on both Windows, Mac and Linux. I have tried to install the server on Windows and Linux and both worked like a charm. The linux version can not run .Net apps however.

The basic setup is really easy. Not only installation wise but configuration wise as well.

Basic configuration

The basic configuration has the server as http on port 80. Coming from the IIS world I kind of like my default document folder to be named inetroot. The default folder in Abyss is to htdocs so I changed this as a personal preference. This step is in no way requird to make things work.

I created a folder “cam” inside the inetroot folder and added a Index.html file to it.

Reverse proxy

Reverse proxy allows the web server to access resources inside the firewall and pass the information on to the web page. This was always one of the major hickups I encountered before. As I have mentioned before I absolutely refuse to expose the actual cameras to the web due to security issues. This way I can access the cameras without actually exposing their built in web server to the world.

In the Abyss consol I did the following

  1. Press Configure on the host
  2. Press the Reverse Proxy option
  3. Add the different cameras as different paths. For instance “/cam/camera1”, “/cam/camera2” etc. and point the proxy to their respective image URLs. Note: For some reason the DCS-935 generates a 500 server error every other time the reverse proxy is called, I have no clue why.
  4. Restart the server (with comes up as a bar in the console gui)

Password protection

As mentioned before I do not wish to expose the cameras to the world. Sure they currently only show my garden ouside but still. So I needed to find a way to password protect the cameras.

Here Abyss comes with another nice feature. Access control. You can add users and/or groups and then set certain virtual paths to only allow the specific users/groups. This was spot on what I wanted. In my case I created a user and set the /cam/ folder to only be accessable to that user… which in turn made the different cameras password protected at the same time. Perfect!

Http/Https

Passwords are good… but if you expose the passwords by sending them in clear text then just about anyone can sniff them out and start using them themselves. I obviously needed to make my side Https.

I have no real need for a CA certificate since I will pretty much be the only one using my site. So a self signed certificate is good enough for my uses.

Yet again Abyss stepped up to the plate. Changing the standard Http to Https was quick and painless… or so I thought… The configuration worked flawlessly and it seemed to work as intended… but my web page refused to show.

It did not take me long to realize my mistake. Http runs standard on port 80. Https runs standard on port 443. My router was set to allow port 80 but not port 443. I simply changed my router settings from port 80 to port 443 and then the page was back up and running again.

You can read about how to change from http to https here:

Abyss – Self signed certificate

Now I just needed to update the Index.html file to allow me to watch the cameras. In my previous post I added the javascript I used to update my images when they were clicked.

I consider this part of the project done.

Next will be to look closer att Google Firebase Cloud Messaging and making a very basic Android app that allows the server to send a notification to my cellphone. So far I have only installed Android Studio but have not even opened it yet 🙂

New way of looking at things

Ok, so in my last post I told you that I was ditching my attempts at making ESP-8266 due to the fact that I have been unable to crack the nut of making it battery powered. I am sure there is a way to make it work, but I have not managed to find it.

A new approach

So what to do next?

Well, this time I thought I was going to do it a bit differently by using more standard components to make things work. I will also ditch the battery powered part and run it while hooked up to the electric grid.

New setup

One of the things I wanted to make out of this project was the ability to scare an intruder off BEFORE they manage to get inside the house.

This project will require a number of parts to work the way I want it to.

  • A PIR sensor that detects movement that allows actions to be taken
  • An outdoor alarm siren (or more than one)
  • Cameras placed in strategic positions
  • A web page that allows me to see what happens from the cameras.
  • Some kind of notification being sent to my mobile phone
  • Some kind of app or web page that allows me to take action against the possible intruders (sounding the siren(s)). Preferrably before the enter the house.

Home automation

There are a number of different home automation systems out there. Nexa and Z-wave to name a few. They all have pros and cons. One of the biggest issues seem to be security wise. Especially Nexa seem to be more or less oblivious to security. Z-wave have the abilty to communicate through a 128bit AES crypto… unfortunatly the crypto is not required and there are a number of Z-wave components that may or may not support encrypted communication.

 

Flow of information

How do I go about making this happen? Well, this is my initial thought (which I don’t know if it is going to work yet.

  1. A PIR operated flood light placed in a strategic position
  2. A Z-wave smart plug with power consumption feedback
  3. A Z-wave controller (RaZpberry2)
  4. A push notification to my mobile alerting me to the situation, sending pictures from the camera covering the PIR area
  5. I check the images for intruders
  6. If false alarm, do nothing. If intruder is detected, sound the alarm

PIR operated flood light

This is just a common PIR activated flood light that lights up when someone comes close. Nothing special at all.

Z-wave smart plug

There are a number of different smart plugs that allows you to activate or deactivate the plug. A couple of them comes with the ability to report back the power consumption. My thought here is that the PIR sensor will draw more power when the flood light is activated than when the sensor is in overwatch mode. If that is the case then it should be possible to use the flood light PIR sensor to detect the movement instead of getting a (much more expensive) separate Z-wave PIR detector.

Z-wave controller

There are any number of controllers out there. Everything from standalone units, to USB sticks to the RaZpberry2 controller that you connect to the GPIOs of a common Raspberry PI 3. My thought here is that the RaZpberry will allow more custom automation than a USB stick or standalone unit. So far my searches on the matter have been inconclusive.

Push notification app

Cool, I get to do an Android app that allows me to recive a push notification that the flood light have been triggered… as well as allow me to push a button to activate the siren… the only question left is… How do I even start? I have never developed any Android app, not even Hello world… Oh well, learning is part of the fun anyway 🙂

Cameras

I have had a couple of IP cameras of different brands. They have a few things in commong… the saddest one is the fact that their motion detection algorithms suck a**. Either they trigger on a leaf blowing in the wind or they won’t trigger even if you are jumping up and down right in front of the camera.

One of the cameras didn’t even have a image adress, the only way to get images was to install their app… and I BLOODY HATE “Cloud cameras” due to the security issues connected to them. Suffice it to say that that camera is no longer in use.

The cameras I currently use is two Dlink DCS-932 (640×480) and one Dlink DCS-935 (1280×720). The DCS-932 cameras resolution s a wee bit to low to my taste, but they will work for their intended purpouse. The DCS-935 is pretty decent, but the recording function does not work satisfactory for sequences longer than 10 seconds.

Unfortunatly I don’t know which camera to use. Cameras are pretty expensive and one thought I have been playing around with is to use Raspberrys and their camera modules to facilitate full HD cameras at a lower price than worse commercial cameras.

Camera web page

I need a web page that gives me an overview of what the different cameras see.

One issue here is the fact that I will NOT be doing any port forwarding to the actual cameras because that would make my network wide open to intrusions and I would not want that now would I ?

I have been using the Aprellium Abyss web server X1 for quite some time. These days they have what is called a reverse poxy. This allows me to get the images from the camera by pointing to a specific URL and I  have made this work pretty neatly. Abyss also allows me to set access rights to a path so that the actual camera page is password protected.

The webpage in itself is actually done. It contains a simple table with images of my different cameras as well as a simple javascript that reloads the images when you press the image of the camera. So far it has worked out pretty good.

This is the javascript I use. The reason for the ignoreCache parameter is the fact that the DCS-932 cameras seem to be caching the images even if I use pretty much every disable cache meta header available. The new Date().getTime() solves that problem by adding the current time to each image url.

function refreshImage(id, url, ignoreCache) {
var image = document.getElementById(id);
if (image) {
if (ignoreCache) {
image.src = url + ‘?’ + new Date().getTime();
}
else {
image.src = url;
}
}
else
console.log(‘Could not find ID: ‘ + id); }

I then call the javascript this way.

<img id=”<imageId>” src=”<cameraUrl>” alt=”Could not load camera xxx” onclick=”refreshImage(‘<imageId>’, ‘<cameraUrl>’, true)” />

Simple and straight forward, I just replace <imageId> with whatever Id I wish the camera to have and <cameraUrl> to the cameras image url I am golden.

Alarm siren

There are a couple of Z-wave sirens available. One is actually solar powered but quite expensive… too expensive for this trial. My thought was actually to use a regular 12V siren with a smart plug that I turn on or off with the app.

Current status

I currently have a Raspberry PI 3 running RetroPie. I just got a different sd card that I will install raspbian on. Other than that I have nothing of the materials above except the cameras

Initially I will get the RaZpberry Z-wave controller and a single smart plug to testing purpouses after that I will have to see.

Project termination

I have not found any reliable way to make this work and lately I have been far to busy to try so I am calling it quits… for now.

I am starting to look into another solution entirely with a mix of Z-Wave and regular sirens and stuff. I might blog a bit about that as I go along.

ZzzZzzZzz

Deep sleep

I have read about people putting their ESPs into a deep power saving sleep which allows batteries to run for many months if not years which is exactly what I am looking for.

Since I am running NodeMCU firmware on my ESP you can initiate deep sleep by running the code

node.dsleep(timeoutinmicroseconds);

Not that it is microseconds and not milliseconds, so to make the chip sleep for 60 seconds you would have to enter the value 60 000 000. To someone like me who is used to think in timing events in milliseconds this is a rather important difference 🙂

Setting the timeout to 0 makes the chip sleep indefinitly until the chip is reset.

You can read more about it in the NodeMCU documentation at:

https://nodemcu.readthedocs.io/en/master/en/modules/node/#nodedsleep

Deep sleep interrupt

So now you have your chip fast asleep. How do get that thing to wake up?

Well, there is the timeout alarmbell… BUT… in order to make that work we need to connect GPIO16 to the reset pin… Wait, what?!? GPIO16?

The ESP8266-01 have GPIO00 and GPIO02, but no GPIO16.

As I read up I actually managed to find a blog about someone that hacked the ESP-01 to allow deep sleep, but that involves some fairly funky soldering. The chip itself is not exactly on the larger side, and having to solder a wire from one of the miniscule pins of the main chip to the reset pin seems a bit… interresting…

You can read more about it here:

http://tim.jagenberg.info/2015/01/18/low-power-esp8266/

Another problem is the fact that apparently the GPIO16 is set to high, and when the timeout occurs it is temporarely set to low to reset the chip… the PIR sensor however is set to low unless it detects a movement where it goes high… which is the opposite.

A different way to power

In a thread, which I unfortunatly can not find the link to again I found someone saying that if you only have VCC connected but no CH_PD then the chip is essentially in power saving mode, with networking and everything else turned off… hmm, now this IS interresting.

Experiment

I hooked the battery up to the ESP and to the PIR sensor… but instead of having the ESP chip connected to the VCC and CH_PD pins at the same time I simply connected the PIR output to the CH_PD pin directly.

I started doing some twisting and turning on the PIR potentiometers that adjust sensitivity and delay (as in how long the power should be on if a detection is noted)… lets just say that it was a royal pain in the ***. Turning the nob 1mm to far in either direction and the timing went haywire. I currently have it set up so that it sends power approximatly 20 seconds when it detects a movement. The chip takes about 4-6 seconds to wake up and connect to the WIFI network.

So far the sensitivity seem to be all over the place. Most of the time it detects movement within 3m, some times in low lighting conditions however it can detect a movement 9-10m away, but not consistently. I have not yet figured out the sensitivity setting.

What I did then was to open up a command console and start pinging the chip IP continuously with this command.

ping 192.168.0.XX /t

Initially all I got was timeouts, then I sent one of my lab rats (aka our oldest kid) down to walk into the room and presto! The pings hit their mark perfectly!

So far I have had this contraption running for a week and so far the batteries are still in the game.

What this opens up is the possiblity to allow the chip to send data to some remote site… OR it allows you to build a simple pinging solution that sends out pings to your ESPs and whenever that ping hits home you know you have a detection and then sends that data to the remote site.

I am currently not sure which of the two solutions to persue. The second solution seems a bit like crossing the stream to get some water… BUT it should be more battery saving than having the chip connect to an external site over the network.

Success, and yet not

So apparently my latest experiment was a success… if you have a room or place where you seldom go to. If it is a room that frequently have people in it, say your kitchen or living room however the PIR would trigger pretty much all the time running your batteries dry…

When I think about it then sure, during evening hours the batteries would be taxed pretty hard, but during night and during day hours when everyone is at work the chip should be in constant low power mode. So the 3-4 days of continuous WIFI connection should be able to be extended to at least a couple of weeks… not enough, but better.

I guess I still a fair amount of walking before I get to the close….

Experiments

Battery life

Ok, so now the move is over and I have finally had some time that allowed me to conduct a couple of experiments.

The first experiment was fairly straight forward. I hooked up 3x 2450mAh AA batteries to my ESP and simply left them there until they ran out.  At the time I hooked them up they had been used for short periods of time to see if they could run the ESP. The ESP connected to my network but did nothing else.

Conclusion: The battery life was 3-4 days… and that was running the ESP without any PIR Sensor at all… and that is simply not even close enough to make a solution viable. In order to make it an actual option the batterylife should be 6+ months or preferrably 1-2 years… not days.

Running the ESP on PIR output

So we need to power the ESP chip only when we have a detection. OK, the ESP runs on 3.3V… the power output of the PIR sensor is 3.3V… Hmm… match made in heaven?

So, what I did was to hook the PIR sensor output to the VCC and CH_PD pins directly. This should make the ESP power up and do its thing right?

Apparently not… What happened was that as the PIR detected a movement the ESP LED flashed at about 1/10 of its ordinary brightness for about half a second and then nothing. I think that the voltage is there, but not the amps. The current is simply not enough to run the ESP chip… which is fairly power hungry…

Is there a way to fassion a kind of circuit breaker that connects the batteries directly to the ESP and only using the PIR to trigger the breaker? So far I have not found anything about it. If you have any clue or link to someone who does then I would appreciate it greatly if you could give it to me.

Moving

I have been away for some time now, and it will be a while longer. I am in the middle of moving between places and things are kind of hectic right now. But I’ll be back as soon as possible

Hooking the ESP8266 up to the Arduino

As I mentioned in my post Battle of the Voltages there are a bunch of things that you need to consider while connecting your ESP8266 to your Arduino.

Pinout

First lets show the pinout of the ESP8266.

esp8266_pinout

The inside row

RX – Recieve. When connecting to a Serial interface this should be connected to the TX pin. When connecting to an Arduino however the connection should be made to the Arduino RX pin.
GPIO 0 – General Input Output 0. When flashing the chip this pin need to be connected to ground (GND) while the chip boots. Otherwise you can use this to send or recieve data from sensors and/or light LEDs etc with it.
GPIO 2 – General Input Output 2 (what happened to GPIO 1?). You can use this pin to send or recieve data from sensors and or light LEDs etc.
GND – Ground. Connect this to ground or in case of using batteries to the negative terminal of the battery.

The outside row

VCC – Power in. Connect this to the VCC of your power source. In case you are using battries the connect this to the positive terminal of the battery. This requires 3.3V . Using more than 3.6V may fry the chip. I have used 3X AA at 3.9V successfully.
RESET – Put this high to reset the chip
CH_PD – Chip Power-Down. This is required to be pulled high while the chip is in use.
TX – Transmit. When connectiing to a Serial interface this should be connected to the RX pin. When connection to an Arduino however the connection should be made to the Arduino TX pin.

One of the first things you may notice is that the placement of the differen pins probably could not be any worse. VCC and GND are placed as far away from each other as humanly possible and on different sides. Same with RX (recieve) and TX (transmit). Making it very hard to connect things to it in an easy manner.

Doing it all wrong

So how did I connect my ESP8266? Well, of course I did everything wrong… or rather. I did a lot of messing around. I tried making a voltage divider by using 3x 220Ohm resistors. It worked great when I checked the voltage with my multimeter… but when I connected VCC and CH_PD the voltage halved and it failed miserably. I think it may have to do with the resistors being to low and that using 3x 10kOhm resistors may work out better. I have not tried that yet thought.

CH_PD up

In order to make the chip run at all you apparently need to send 3.3V to the CH_PD pin. I have read that some people simply solder the two pins together, others use a 10k resistor between the VCC and the CH_PD pin. In my attempts I connected the 3.3V power supply to both VCC and CH_PD and it worked.

Speaking is not shouting

As I mentioned in the War of the Voltages the ESP require 3.3V with a max current of 250mA. The Arduino however only supply 50mA on its 3.3V pin… BUT it seems as if you are “only” talking to the ESP through your serial interface then the 3.3V pin on your arduio is good enought to get things rolling. I suspect that the network part won’t work very well however.

TX to RX and RX to TX or TX to TX and RX to RX?

There is a title for you! Makes your eyes pop just trying to read it… but it is a valid question. I have read on multiple posts that you need to connect TX (Transmit) to RX (Recieve) on your serial interface, and vise versa… but apparently this is not the way when you use an Arduino as a intermediat. When using the Arduino as the serial interface the only way I got it to work was to connect TX to TX and RX to RX… Honestly it seems like different people have different experiences here so it may be up for grabs what works for you if you ever try it.

Arduino as a Serial interface

The arduino can act as a serial interface (FTDI) and it is possible that I may have ruined my Arduino in my trials to flash the ESP to NodeMCU. At first I could not get the flashing procedure to work, but then I read a post where they mentioned that you could get it working by removing the ATMEGA328P chip from the Arduino, basicly turning it from a “smart” Micro Controller to a “stupid” serial interface. And that actually made it work… The strange thing is that after I pulled the ATMEGA chip from the Arduino and replaced it the connection to my chip still works like when it was removed… so as I said, I don’t know if my Arduino is messed up or not, but it seem to be working now.

 

Battle of the Voltages

One of the first things that I found out when I started reading up on the ESP8266 was that it used AT commands, something I have never used before. I also (luckily enough) found out before I tried to hook it up that it works at 3.3V with a maximum current of 250ma.

The UNO have a 5V and a 3.3V pin. The problem is that the 3.3V pin on the UNO only allows 50ma, so the UNO can not handle the ESP8266 at full blast. The ESP8266 is also said to have a maximum voltage of 3.6V so using the 5V pin on the Arduino may just end up frying the ESP8266 chip.

When I bought these things there were NOTHING that gave me warnings about this. No manuals, no connection schemes, nothing from the retailer.

To make things even more interresting I have found out that the PIR sensor need 5V input, but have a 3.3V output… way to make things complicated to hook up.

This acutally made me kind of dispair and drop the project for quite some time before I decided to take it up again. Problems are meant to be conquerd and overcome… and besides, whats the fun if everything goes your way right from the start?