How to collect wireless sensors readings using rtl_433 and display them using OpenWrt

Goal: to use an OpenWrt capable router to receive wireless sensors reading and log them using OpenWrt luci-app-statistics module.
Why: because my router is always on anyway, so why not to use it to also log wireless sensors readings?
What is needed?

• A router (or any other device) with USB port(s) that is supported by OpenWrt
• A RTL-SDR dongle
• Wireless temperature/humidity sensors supported by the rtl_433 application

Note: I’m referring to the old OpenWrt project name, which is currently called LEDE, which in turn will be called OpenWrt again. It’s because of the nonsense they started some time ago by forking the project and calling it LEDE because they couldn’t agree on which side they should start counting bits. Some of them there little-endians and the others big-endians. Finally they have agreed to use both approaches and now they are busy merging everything back together and use the good old name [sic!]. So the next release is supposed to be called OpenWrt as their branch name suggest. At the moment this branch is not usable but maybe it will be released at the end of the year. It is still not known which year should it be…

Hardware

 

The rtl_sdr dongle

 

rtl_433 works perfectly with a normal not modified DVB-T RTL2832U/R820T(2) dongles. There’s absolutely no need to buy so called improved dongles like NooElec, Blog v3 nonsense or any other modification that are based on the same chipset.

 

Wireless sensors

I’m using cheap wireless temperature/humidity sensors like depicted in the very beginning of the article. These sensors are released under different names - ‘TFA Dostmann Funk-Thermo-Hygrometer’, ‘Froggit Funk Thermometer FT0075’, ‘WS-10 Wireless Indoor/Outdoor Thermo-Hygrometer‘ to name a few. Currently one could buy the so called ‘Bresser Thermometer Temeo Hygro Quadro‘ on Amazon for 30 Euros  which would make 10 Euro per sensor. rtl_433 recognizes these sensors as ‘Ambient Weather Temperature Sensor’, device number 20.

Cross compiling and installing the rtl_433 application

This is the most easy step. Provided that the OpenWrt build system is installed use description given at https://github.com/flux242/rtl-433-openwrt to compile and install the rtl_433 daemon.

By default the following rtl_433 options are set:

• device: 0 (I only have one RTL-SDR dongle attached to the router)
• ppm error: 46 (corresponds to my RTL-SDR dongle ppm error)
• demodulators: 20 (corresponds to my wireless sensors)
• units: si
• utc: 0 (i.e. local time)

Adapting only these options should be enough but other options could also be checked in the /etc/init.d/rtl_433 script. To change them edit /etc/config/rtl_433 file or use uci commands.

Note 1: Do not change the output option which is set to json.

Note 2: If rtl_433 compiled binary, its config and init script aren’t installed from an ipk but just moved to the router then the init script should be enabled manually:

/etc/init.d/rtl_433 enable
/etc/init.d/rtl_433 start

Note 3: crosscompillation of the application is not needed any longer because it can be installed from the repository of the OpenWrt 18.06 branch. Still the init script and the config file should be copied over to the router. Check the Makefile 'install' section at the end.

Router load

The question is: does rtl_433 application work stable on my router and how does it load its CPU? Well on my relatively old Buffalo WZR-HP-AG300H which is 680MHz 24Kc MIPS router I haven’t noticed any stability issues or router reboots in a year or so. And the router load is around 20%:

This is true when only one decoder is activated. If I would activate all decoders available then rtl_433 would load router CPU up to 40%.

The only problem I had was when the mpd daemon was running at the same time the rtl_433 would stop working. Maybe it is becase the mpd itself is a poorly written pile of crap that loads CPU too much or maybe the USB2.0 was the bottle neck because mpd sends its output over the same USB port to a sound card. And additionally I have a USB screen 320x200 attached which is updated at least 5 times per second. Probably it was already too much so I had to deactivate the mpd daemon on my router.

Edit (10Nov2018): Actually it turned out that the default 'nice' parameter in the mpd's init script was set to -10 for some strange reason. After I changed this parameter to 10 both mpd and rtl_433 do coexist together without any problem. There are still CPU usage spikes from mpd:

but they do not disturb rtl_433 like before.

 

The rtl_433 is working, now what?

Here starts the interesting part. The rtl_433 can only output to its stdout or to a file and there’s nothing I can do with a file. A normal file will grow in its size and I’d need to handle this somehow. I could use a FIFO named pipe but read/write operations on it are blocking. What I need is to be able to broadcast program’s output on the local network using UDP sockets. I could patch the rtl_433 so that it would broadcast using UDP sockets or as an alternative I could (theoretically) write a daemon that would attach itself to the rtl_433’s stdout and rebroadcast. And neither did I want to patch the rtl_433 nor I wanted to write, cross compile and maintain an additional daemon!

The dead end as it seems but there’s another option - a tricky one. One could redirect the rtl_433 output to the system log and then system log could also be sent to a remote log server using UDP or TCP sockets! And the good part is - one can simply use the socat as the remote log server because there’s no handshaking or any other communication protocol defined for such a server. So the socat would just listen on a specific UDP port and then rebroadcast to a different UDP port on the local network. Using this scenario one could use as many consumers as needed.

But what sounds easy in the theory isn’t that easy in practice. The log server using only one instance of socat is not enough because the input should be filtered too. First of all only rtl_433 messages should be allowed. Then everything except the json part should be removed from each message. Then UDP has no EOF marker so there’s no easy way to separate incoming messages from each other. And then I need a way to reliably find all spawn children in my socat server script because otherwise if I restart it not all spawn processes will be killed by the procd. The last wouldn’t be necessary if I could use a single socat instance but alas, I’m using a complex pipe of two socat instances and an awk filter in between.

 

Redirecting rtl_433 output to a local log server over UDP

This step is very easy. First of all in the rtl_433 daemon init script ( /etc/init.d/rtl_433 ) the procd should be instructed to redirect daemon’s output to the syslog:

  procd_set_param stdout 1

Additionally the following options should be added into the system section of the /etc/init.d/system

config system
  option log_port '6665'
  option log_ip '127.0.0.1'
  option log_proto 'udp'
  option log_prefix '
'
  option log_remote '1'

The log_port can be changed to a desired one. The most important part is the log_prefix option here. The way it is specified is not a typo. The new line serves as the messages separator.

Now all syslog messages will be additionally sent to a UDP server listening locally on the port 6665.

 

The socat UDP local remote log server

The UDP log server should listen for incoming messages on one UDP port, filter them and retransmit on another UDP port. This can be done with a very simple shell script:

#!/bin/sh
# /mnt/sb/bin/sensorsbroadcast.sh

cleanup()
{
  while [ -n "$1" ]; do
    kill "$1"
    shift
  done
}

find_children()
{
  local ppid
  local fname
  local i
  for i in $(ls -d /proc/[0-9]*); do
    fname="$i/task/$(basename $i)/status"
    [ -e "$fname" ] && {
      ppid="$(grep PPid $i/task/$(basename $i)/status|cut -f2)"
      [ "$ppid" -eq "$1" ] && {
        printf "$(basename $i) "
      }
    }
  done
}

broadcast_ip=$(uci get network.lan.ipaddr|sed -nr 's/(.*)./\1255/p')
port_listen=$(uci -q get 'system.@system[-1].log_port' || echo 6665)
port_broadcast=6666

( socat -T10 -u udp-listen:"$port_listen",fork,reuseaddr stdout | \
  awk '/rtl_433/{a=gensub(/[^{]*([^}]*).*/,"\\1",G);if(length(a)>0){print a"}";fflush();}}' |\
  socat -u - udp-datagram:"$broadcast_ip":"$port_broadcast",broadcast ) &
childpid=$!

sleep 1 # need to give it some time to create subprocesses
children=$(find_children "$childpid")

trap "cleanup $children" EXIT INT TERM

wait

The tricky part here is to find all subprocess PIDs to clean up properly. The only way I’ve found to do this is to iterate over all running processes. Yes, I could install a fully fledged ps command from core utils instead of the crippled busybox’s variant to find children PIDs but I do not want additional dependencies (socat dependency is enough).

Additionally I have written an init script /etc/init.d/sensorbroadcast to start my socat log server script automatically upon system start:

#!/bin/sh /etc/rc.common

START=99
USE_PROCD=1

start_service() {
  procd_open_instance
  procd_set_param command /mnt/sd/bin/sensorsbroadcast.sh
  procd_set_param stdout 0 # forward stdout of the command to logd
  procd_set_param stderr 0 # same for stderr
  procd_set_param user nobody # run service as user nobody
  procd_close_instance
}

The init script should be enabled first:

/etc/init.d/sensorsbroadcast enable

Now after rebooting the router it should be possible to receive wireless sensors readings broadcasted on my local network:

me@desktop:~$ nc -luk 6666
{"time" : "2017-11-03 11:02:09", "model" : "Ambient Weather F007TH Thermo-Hygrometer", "device" : 60, "channel" : 3, "battery" : "Ok", "temperature_C" : 19.500, "humidity" : 58}
{"time" : "2017-11-03 11:02:54", "model" : "Ambient Weather F007TH Thermo-Hygrometer", "device" : 143, "channel" : 1, "battery" : "Ok", "temperature_C" : 11.444, "humidity" : 83}
...

One could say that I could start the rtl_433 in my sensorbroadcast.sh script to avoid all complications with the UDP log server. Yes I could but then such solution wouldn’t be as robust as mine. With my solution I have a separation of concerns where the rtl_433 and sensorbroadcast.sh are started and do work independently. If rtl_433 crashes (and it does crash) for example I don’t have to care about it - it’ll be restarted automatically by the procd. Additionally both daemons don’t even have to be started on the same machine.

There’s one negative side effect of this approach though - the system log is filled with sensors readings that pop other messages out (OpenWrt’s syslog is a ring buffer located in RAM). Additionally there’s one restriction - maximum string length that can be logged is 1024 bytes. If rtl_433 produces longer json strings for a specific sensor then this approach may not work.

 

Sensors readings are broadcasted, now what?

Now it’s time to log the data using luci-app-statistics module.

This is also easier to say than to implement. The problem here is that wireless sensors are sending messages asynchronously but the collectd should be fed with data with specified time interval, i.e. synchronously. I came up with the following collecting script:

#!/bin/sh
# /mnt/sd/bin/collectsensor.sh

logger "*** collectsensor script is started for channel - $1 ***"

CHANNEL=${1:-1} # by default channel 1
MYFIFO=/tmp/sensor_channel_$CHANNEL.fifo
LISTENPORT=6666

HOST=$COLLECTD_HOSTNAME
INTERVAL=$COLLECTD_INTERVAL

[ -z "$INTERVAL" ] && INTERVAL=5
[ -z "$HOST" ] && HOST=Buffalo
INTERVAL=$(printf "%.0f" $INTERVAL) # float to int

cleanup()
{
  kill $1 2>/dev/null
  kill $2 2>/dev/null
  rm $3 2>/dev/null
}

[ -e "$MYFIFO" ] && rm "$MYFIFO"
mkfifo "$MYFIFO"

socat -T10 -u udp-listen:$LISTENPORT,reuseaddr,fork stdout >"$MYFIFO" &
pid1=$!

( while sleep $INTERVAL; do echo 'PUTVALNOW' >"$MYFIFO"; done ) &
pid2=$!

trap "cleanup $pid1 $pid2 $MYFIFO" EXIT INT TERM

while read LINE <"$MYFIFO" ; do
  if [ "PUTVALNOW" = "$LINE" ]; then
    [ -n "$temperature_line" ] && [ -n "$humidity_line" ] && { 
      echo "PUTVAL \"$HOST/exec-rtl-sensor-channel-$CHANNEL/temperature\" interval=$INTERVAL N:${temperature_line}"
      echo "PUTVAL \"$HOST/exec-rtl-sensor-channel-$CHANNEL/humidity\" interval=$INTERVAL N:${humidity_line}"
    }
  else
    channel_line=$(echo $LINE|jsonfilter -e '$.channel')
    [ "$CHANNEL" -eq "$channel_line" ] && {
      temperature_line=$(echo $LINE|jsonfilter -e '$.temperature_C')
      humidity_line=$(echo $LINE|jsonfilter -e '$.humidity')
      # current values are also stored in a file to be used with lcd4linux for example 
      echo "$temperature_line" > /tmp/sensor_channel_${CHANNEL}_temperature.txt
      echo "$humidity_line" > /tmp/sensor_channel_${CHANNEL}_humidity.txt
    }
  fi
done

Inside there are two additional processes created to communicate with the main loop over a FIFO named pipe. The first process is the socat listening to the broadcasted messages and the second one sends trigger messages with specified time interval. The main loop waits for the next message and checks if it’s a trigger message. If it is then it feeds collectd with stored data otherwise it parses the json message and stores new values.

The socat instance that listens to the broadcast messages should fork otherwise only the first message is processed. But as it get forked its child process will never be killed because there won’t be an EOF for it. As the result for each broadcasted message there will be a child process created filling up memory until it is full and then - game over. To solve this problem I’ve added a timeout of 10 seconds (-T10). So for each incoming UDP message socat forks itself, process the message and then dies after 10 seconds.

Another thing worth to note is that I’m creating a database per channel. Beside channel number my sensors are broadcasting their device id’s which I could use to create the database for. The problem is that the sensor generates a new device id when it is powered on. So it would be enough just to change sensor’s batteries and the logging would stop. The channel on the other hand is persistent because it is defined by a DIP switch.

Now on the collectd exec plugin (collectd-mod-exec module) configuration page sensors that are broadcasting on channels 3 and 8 (my outside and inside sensors) should be registered:

 

The last missing part

There’s only one part that is still missing - the /usr/lib/lua/luci/statistics/rrdtool/definitions/exec.lua script should be updated to make everything work:

module("luci.statistics.rrdtool.definitions.exec", package.seeall)

-- local util = require "luci.util"

function rrdargs(graph, plugin, plugin_instance)

    -- within the exec.lua's rrdargs() function you can decide
    -- for which rrd you've been called.
    -- util.exec("logger " .. "plugin: " .. plugin .. "plugin_inst: " .. plugin_instance)
    --
    --
    if "rtl-sensor-channel-8" == plugin_instance then
        return {
            {
                title = "Room Temperature",
                vlabel = "Celsius",
                data = {
                    types = { "temperature" },
                    options = {
                        temperature = {
                            title  = "Room temperature",
                            color  = "ff0000"
                        }
                    }
                }
            },

            {
                title = "Room relative humidity",
                vlabel = "%",
                data = {
                    types = { "humidity" },
                    options = {
                        humidity = {
                            title  = "Room relative humidity",
                            color  = "0000ff"
                        }
                    }
                }
            }
        }
    end

    if "rtl-sensor-channel-3" == plugin_instance then
        return {
            {
                title = "Outside Temperature",
                vlabel = "Celsius",
                data = {
                    types = { "temperature" },
                    options = {
                        temperature = {
                            title  = "Outside temperature",
                            color  = "ff0000"
                        }
                    }
                }
            },

            {
                title = "Outside relative humidity",
                vlabel = "%",
                data = {
                    types = { "humidity" },
                    options = {
                        humidity = {
                            title  = "Outside relative humidity",
                            color  = "0000ff"
                        }
                    }
                }
            }
        }
    end

end

If everything is set up correctly then temperature and humidity graphs should be generated: 

 

Additional settings worth to mention

By default ‘Rows per RRA’ is set to 100 which is too small. As the rule of thumb this value should not be smaller as number of pixels in generated graphs along X axis to avoid choppy looking graphs. As far as the width of generated graphs is 600 the value should be at least 600: 

Unfortunately if the database is already generated it will be necessary to stop collectd and luci_statistics and manually remove the old database entries for the new value to take effect. Database is created there where the ‘Storage directory‘ points to.