How to receive and decode multiple weather sondes with only one RTL-SDR receiver (Part 2)

As I finished the first article I got a tip from Zilog80 that he has finished a preliminary version of his iq_server concept. iq_server is a kind of a channelizer that allows clients to connect over TCP and request decimated baseband IQ stream of floats for a specific frequency offset. I was really excited to learn about his work because it would allow to allocate/dispose decoding processes upon request dynamically!

The main problem with the script I've written before was that the slots were preallocated statically upon the script's start. It's because I cannot attach a newly created process to the IQ data stream I get from rtl_sdr. This is really inefficient because it consumes lot of CPU resources for nothing. As far as all the decoders were started at the same time for the requested frequency the CPU was immediately overloaded. I solved this problem by patching the csdr framework which allowed me to receive and decode at least 5 sondes at the same time. Additionally I had to patch the rs92mod decoder code to make it reread ephemeris data upon receiving SIGUSR1 signal. This all became obsolete as I switched to the iq_server/iq_client concept! The changes to the script are minimal as I kept the scanning and slot management logic unchanged:

#!/bin/bash

. ./defaults.conf

SCAN_BINS=4096
SCAN_OUTPUT_STEP=10000  # in Hz
SCAN_AVERAGE_TIMES=100
SCAN_UPDATE_RATE=1
SCAN_UPDATE_RATE_DIV=5 # 5 seconds
SCAN_POWER_THRESHOLD=-69

SCANNER_OUT_PORT=5676
SCANNER_COM_PORT=5677
DECODER_PORT=5678

SLOT_TIMEOUT=10 # i.e 30 seconds *10 = 5 minutes
SLOT_ACTIVATE_TIME=4 # 4 * 5 seconds = 20 seconds min activity

OPTIND=1 #reset index
while getopts "ha:p:f:s:g:p:P:t:" opt; do
  case $opt in
     h)  show_usage $(basename $0); exit 0; ;;
     a)  address="$OPTARG" ;;  # not used atm
     p)  port="$OPTARG" ;;     # not used atm
     f)  TUNER_FREQ="$OPTARG" ;;
     s)  TUNER_SAMPLE_RATE="$OPTARG" ;;
     g)  TUNER_GAIN="$OPTARG" ;;
     P)  DONGLE_PPM="$OPTARG" ;;
     t)  SCAN_POWER_THRESHOLD="$OPTARG" ;;
     \?) exit 1 ;;
     :)  echo "Option -$OPTARG requires an argument" >&2;exit 1 ;;
  esac
done
shift "$((OPTIND-1))"
 
[ ! "$TUNER_FREQ" -eq 0 ] || show_error_exit "Wrong frequency"
[ ! "$TUNER_SAMPLE_RATE" -eq 0 ] || show_error_exit "Wrong sample rate"

DECIMATE=$((TUNER_SAMPLE_RATE/DEMODULATOR_OUTPUT_FREQ))
[ "$((DECIMATE*DEMODULATOR_OUTPUT_FREQ))" -ne "$TUNER_SAMPLE_RATE" ] && show_error_exit "Sample rate should be multiple of $DEMODULATOR_OUTPUT_FREQ"


cleanup()
{
  local children child
  children="$1 $2 $(get_children_pids $1) $(get_children_pids $2)"
  kill $children &>/dev/null;wait $children &>/dev/null
}

scan_power()
{
  ./csdr convert_u8_f | \
  ./csdr fft_cc $SCAN_BINS $((TUNER_SAMPLE_RATE/(SCAN_UPDATE_RATE*SCAN_AVERAGE_TIMES/SCAN_UPDATE_RATE_DIV))) | \
  ./csdr logaveragepower_cf -70 $SCAN_BINS $SCAN_AVERAGE_TIMES | \
  ./csdr fft_exchange_sides_ff $SCAN_BINS | \
  ./csdr dump_f | tr ' ' '\n' | \
   tee >(
    awk -v bins=$SCAN_BINS '{printf("%.1f ",$0);if(0==(NR%bins)){printf("\n")};fflush()}' |
    awk -v f=$TUNER_FREQ -v bins="$SCAN_BINS" -v sr="$TUNER_SAMPLE_RATE" '
      {printf("{\"response_type\":\"log_power\",\"samplerate\":%d,\"tuner_freq\":%d,\"result\":\"%s\"}\n", sr, f, $0);
      fflush()}' |
    socat -u - UDP4-DATAGRAM:127.255.255.255:$SCANNER_OUT_PORT,broadcast,reuseaddr
   ) |
   awk -v f=$TUNER_FREQ -v sr=$TUNER_SAMPLE_RATE -v bins=$SCAN_BINS '
     BEGIN{fstep=sr/bins;fstart=f-sr/2;print fstep;print fstart}
     {printf("%d %.1f\n",fstart+fstep*((NR-1)%bins),$0);
      if(0==(NR%bins)){printf("\n")};fflush()}' | \
   awk -v outstep="$SCAN_OUTPUT_STEP" -v step=$((TUNER_SAMPLE_RATE/SCAN_BINS)) '
     function abs(x){return (x<0)?-x:x}
     {if(length($1)!=0){if(abs($1-outstep*int($1/outstep)<step)){print $0}}
      else{print};
      fflush();}' | \
   awk -v outstep="$SCAN_OUTPUT_STEP" -v thr=$SCAN_POWER_THRESHOLD '{if (length($2)!=0){if(int($2)>thr){print outstep*int(int($1)/outstep)" "$2;fflush()}}}'
}

# the line below should come before the m10mod if needed.
#      tee >(c50dft -d1 --ptu --json /dev/stdin > /dev/stderr) | \
decode_sonde()
{
  local bpf3=$(calc_bandpass_param 5000 48000)
  local bpf9=$(calc_bandpass_param 9600 48000)

  (
    ./iq_client --freq $(calc_bandpass_param "$(($1-TUNER_FREQ))" "$TUNER_SAMPLE_RATE") |
    tee >(
      ./csdr bandpass_fir_fft_cc -$bpf9 $bpf9 0.02 |
      ./csdr fmdemod_quadri_cf | ./csdr limit_ff | ./csdr convert_f_s16 |
      sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 |
      ./m10mod --ptu --json > /dev/stderr
    ) |
    ./csdr bandpass_fir_fft_cc -$bpf3 $bpf3 0.02 |
    ./csdr fmdemod_quadri_cf | ./csdr limit_ff | ./csdr convert_f_s16 |
    sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 |
    tee >(./dfm09mod --ptu --ecc --json -vv /dev/stdin > /dev/stderr) \
        >(./dfm09mod --ptu --ecc --json -i /dev/stdin > /dev/stderr) \
        >(./rs41mod --ptu --ecc --crc --json -vv /dev/stdin > /dev/stderr) \
        >(./rs92mod -e "$EPHEM_FILE" --crc --ecc --json /dev/stdin > /dev/stderr) | \
    aplay -r 48000 -f S8 -t wav -c 1 -B 500000 &> /dev/null
  ) &>/dev/stdout | while read LINE; do
      echo "$LINE" | grep --line-buffered -E '^{' | jq --unbuffered -rcM '. + {"freq":"'"$1"'"}' | \
      (flock 200; socat -u - UDP4-DATAGRAM:127.255.255.255:$DECODER_PORT,broadcast,reuseaddr) 200>$MUTEX_LOCK_FILE
    done
}

declare -A actfreq # active frequencies
declare -A slots   # active slots

(socat -u UDP-RECVFROM:$SCANNER_COM_PORT,fork,reuseaddr - | while read LINE; do
  case "$LINE" in
       for freq in "${!actfreq[@]}"; do 
         actfreq[$freq]=$((actfreq[$freq]-1))
         [ "${actfreq[$freq]}" -gt "$SLOT_TIMEOUT" ] && actfreq[$freq]=$SLOT_TIMEOUT
         if [ "${actfreq[$freq]}" -eq 0 ]; then
           # deactivate slot
           [ -z "${slots[$freq]}" ] || {
             cleanup "${slots[$freq]}"
             unset slots[$freq]
           }
           unset actfreq[$freq]
         elif [ "${actfreq[$freq]}" -ge $SLOT_ACTIVATE_TIME ]; then
           # activate slot
           [ -z "${slots[$freq]}" ] && {
             decode_sonde "$freq" &
             slots[$freq]=$!
           }
         fi
       done
       ;;
    *) freq="${LINE% *}"
       [ -z "${_FREQ_BLACK_LIST[$freq]}" ] && {
         [ -n "$freq" ] && actfreq[$freq]=$((actfreq[$freq]+1))           
       }
       ;;
  esac
done) &
pid1=$!

(while sleep 30; do echo "TIMER30" | socat -u - UDP4-DATAGRAM:127.255.255.255:$SCANNER_COM_PORT,broadcast,reuseaddr; done) &
pid2=$!

trap "cleanup $pid1 $pid2" EXIT INT TERM

rtl_sdr -p $DONGLE_PPM -f $TUNER_FREQ -g $TUNER_GAIN -s $TUNER_SAMPLE_RATE - |
tee >(scan_power | socat -u - UDP4-DATAGRAM:127.255.255.255:$SCANNER_COM_PORT,broadcast,reuseaddr) |
./iq_server --fft /tmp/fft.out --bo 32 - 2400000 8

Now the decode_sonde processes are allocated and disposed dynamically upon request from the power scanning logic! The shift_addition_switchable_cc block I added to the csdr is not needed anymore and there's no need to patch the rs92mod! Still there should be another cron job to update the ephemeris data regularly (at least once in two hours).

Results are pretty impressive - I can receive 4 sondes at the same time and the CPU load is about 50-55% which is about 30% less than before!

I have also tried to get power measurements from the iq_server but current implementation in the iq_server is not very optimal ant it loads CPU 30% more than my csdr based solution in the scan_power function.

 
Update 24 Feb 2021:  I have implemented an automatic noise level detection in the scan_power function. Problem with the previous solution was that the threshold had to be adjusted if gain was changed. Additionally some electronic devices could increase the noise level for 2-3 dB and then all slots would be filled with noise signals. Now the noise level is dynamically calculated and the threshold script parameter defines not the absolute value like before but the signal power in dB. So a signal is considered active if its power level is above current noise level plus the threshold. I'm using 6dB as the signal threshold. This change is not reflected here but only on the github.

 

Now only the auto_rx needs to be updated upstream to work without any hardware by only listening to the UDP! The required changes would be minimal.

The older script from the previous article is renamed and kept here for historical reasons.


Update 12 Aug 2020:  Zilog80 has added an FM demodulator called iq_fm. Using his demodulator the decode_sonde can be rewritten the following way:

decode_sonde()
{
  (
    ./iq_client --freq $(calc_bandpass_param "$(($1-TUNER_FREQ))" "$TUNER_SAMPLE_RATE") |
    tee >(
      ./iq_fm --lpbw 19.2 - 48000 32 --bo 16 |
      sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 |
      ./m10mod --ptu --json > /dev/stderr
    ) |
    ./iq_fm --lpbw 10.0 - 48000 32 --bo 16 |
    sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 |
    tee >(./dfm09mod --ptu --ecc --json -vv /dev/stdin > /dev/stderr) \
        >(./dfm09mod --ptu --ecc --json -i /dev/stdin > /dev/stderr) \
        >(./rs41mod --ptu --ecc --crc --json -vv /dev/stdin > /dev/stderr) \
        >(./rs92mod -e "$EPHEM_FILE" --crc --ecc --json /dev/stdin > /dev/stderr) | \
    aplay -r 48000 -f S8 -t wav -c 1 -B 500000 &> /dev/null
  ) &>/dev/stdout | while read LINE; do
      echo "$LINE" | grep --line-buffered -E '^{' | jq --unbuffered -rcM '. + {"freq":"'"$1"'"}' | \
      (flock 200; socat -u - UDP4-DATAGRAM:127.255.255.255:$DECODER_PORT,broadcast,reuseaddr) 200>$MUTEX_LOCK_FILE
    done
}

Despite the fact that all decoders can demodulate the decimated IQ stream I'm using the iq_fm because I start all decoders at the same time and then it is much more efficient to do demodulation only once for all of them. Well I do it twice with different filter parameters for different sonde groups.

As an alternative of starting all decoders at the same time one could use the dft_detect utility. Here I defined a new function decode_sonde_with_type_detect and it is called by default currently:

start_decoder()
{
  local decoder bw

  case "$1" in
    RS41) decoder="./rs41mod --ptu --ecc --crc --json -vv /dev/stdin > /dev/stderr";bw=10 ;;
    RS92) decoder="./rs92mod -e "$EPHEM_FILE" --crc --ecc --json /dev/stdin > /dev/stderr";bw=10 ;;
    DFM9) decoder="tee >(./dfm09mod --ptu --ecc --json /dev/stdin > /dev/stderr) | ./dfm09mod --ptu --ecc --json -i /dev/stdin > /dev/stderr";bw=10 ;;
     M10) decoder="./m10mod --ptu --json > /dev/stderr";bw=19.2 ;;
       *) ;;
  esac

  ./iq_fm --lpbw $bw - 48000 32 --bo 16 |
  sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 |
  tee >(aplay -r 48000 -f S8 -t wav -c 1 -B 500000 &> /dev/null) |
  eval "$decoder"
}

decode_sonde_with_type_detect()
{
    ./iq_client --freq $(calc_bandpass_param "$(($1-TUNER_FREQ))" "$TUNER_SAMPLE_RATE") |
    (type=$(./dft_detect --iq - 48000 32 | awk -F':' '{print $1}'); start_decoder "$type") &>/dev/stdout |
    while read LINE; do
      echo "$LINE" | grep --line-buffered -E '^{' | jq --unbuffered -rcM '. + {"freq":"'"$1"'"}' | \
      (flock 200; socat -u - UDP4-DATAGRAM:127.255.255.255:$DECODER_PORT,broadcast,reuseaddr) 200>$MUTEX_LOCK_FILE
    done
}

I haven't yet played with the dft_detect much but it should do

Patching auto-rx to let it work without SDR hardware by listening to UDP messages

I've patched current auto-rx trunk (2020-08-16) to let it work without and SDR hardware by only listening to the UDP messages my script broadcasts on port 5678

The trick is to add 'TCP' in front of the device_idx number (like I have in the patch above)


Then I simply just start it using 'python3 ./auto_rx.py -t 0 -m UDP -f 402'


Additionally I've added couple of scripts to parse some statistics from the auto-rx log files or even plot sondes' flight path like in the following pictures:

Full flight path of a nearby sonde

Partial flight path of a faraway sonde