The main problem with the script I've written before was that the slots were preallocated statically upon the script's start. It's because I cannot attach a newly created process to the IQ data stream I get from rtl_sdr. This is really inefficient because it consumes lot of CPU resources for nothing. As far as all the decoders were started at the same time for the requested frequency the CPU was immediately overloaded. I solved this problem by patching the csdr framework which allowed me to receive and decode at least 5 sondes at the same time. Additionally I had to patch the rs92mod decoder code to make it reread ephemeris data upon receiving SIGUSR1 signal.
This all became obsolete as I switched to the iq_server/iq_client concept! The changes to the script are minimal as I kept the scanning and slot management logic unchanged:
Now the decode_sonde processes are allocated and disposed dynamically upon request from the power scanning logic! The shift_addition_switchable_cc block I added to the csdr is not needed anymore and there's no need to patch the rs92mod! Still there should be another cron job to update the ephemeris data regularly (at least once in two hours).
Results are pretty impressive - I can receive 4 sondes at the same time and the CPU load is about 50-55% which is about 30% less than before!
I have also tried to get power measurements from the iq_server but current implementation in the iq_server is not very optimal ant it loads CPU 30% more than my csdr based solution in the scan_power function.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 | #!/bin/bash. ./defaults.confSCAN_BINS=4096SCAN_OUTPUT_STEP=10000 # in HzSCAN_AVERAGE_TIMES=100SCAN_UPDATE_RATE=1SCAN_UPDATE_RATE_DIV=5 # 5 secondsSCAN_POWER_THRESHOLD=-69SCANNER_OUT_PORT=5676SCANNER_COM_PORT=5677DECODER_PORT=5678SLOT_TIMEOUT=10 # i.e 30 seconds *10 = 5 minutesSLOT_ACTIVATE_TIME=4 # 4 * 5 seconds = 20 seconds min activityOPTIND=1 #reset indexwhile getopts "ha:p:f:s:g:p:P:t:" opt; do case $opt in h) show_usage $(basename $0); exit 0; ;; a) address="$OPTARG" ;; # not used atm p) port="$OPTARG" ;; # not used atm f) TUNER_FREQ="$OPTARG" ;; s) TUNER_SAMPLE_RATE="$OPTARG" ;; g) TUNER_GAIN="$OPTARG" ;; P) DONGLE_PPM="$OPTARG" ;; t) SCAN_POWER_THRESHOLD="$OPTARG" ;; \?) exit 1 ;; :) echo "Option -$OPTARG requires an argument" >&2;exit 1 ;; esacdoneshift "$((OPTIND-1))" [ ! "$TUNER_FREQ" -eq 0 ] || show_error_exit "Wrong frequency"[ ! "$TUNER_SAMPLE_RATE" -eq 0 ] || show_error_exit "Wrong sample rate"DECIMATE=$((TUNER_SAMPLE_RATE/DEMODULATOR_OUTPUT_FREQ))[ "$((DECIMATE*DEMODULATOR_OUTPUT_FREQ))" -ne "$TUNER_SAMPLE_RATE" ] && show_error_exit "Sample rate should be multiple of $DEMODULATOR_OUTPUT_FREQ"cleanup(){ local children child children="$1 $2 $(get_children_pids $1) $(get_children_pids $2)" kill $children &>/dev/null;wait $children &>/dev/null}scan_power(){ ./csdr convert_u8_f | \ ./csdr fft_cc $SCAN_BINS $((TUNER_SAMPLE_RATE/(SCAN_UPDATE_RATE*SCAN_AVERAGE_TIMES/SCAN_UPDATE_RATE_DIV))) | \ ./csdr logaveragepower_cf -70 $SCAN_BINS $SCAN_AVERAGE_TIMES | \ ./csdr fft_exchange_sides_ff $SCAN_BINS | \ ./csdr dump_f | tr ' ' '\n' | \ tee >( awk -v bins=$SCAN_BINS '{printf("%.1f ",$0);if(0==(NR%bins)){printf("\n")};fflush()}' | awk -v f=$TUNER_FREQ -v bins="$SCAN_BINS" -v sr="$TUNER_SAMPLE_RATE" ' {printf("{\"response_type\":\"log_power\",\"samplerate\":%d,\"tuner_freq\":%d,\"result\":\"%s\"}\n", sr, f, $0); fflush()}' | socat -u - UDP4-DATAGRAM:127.255.255.255:$SCANNER_OUT_PORT,broadcast,reuseaddr ) | awk -v f=$TUNER_FREQ -v sr=$TUNER_SAMPLE_RATE -v bins=$SCAN_BINS ' BEGIN{fstep=sr/bins;fstart=f-sr/2;print fstep;print fstart} {printf("%d %.1f\n",fstart+fstep*((NR-1)%bins),$0); if(0==(NR%bins)){printf("\n")};fflush()}' | \ awk -v outstep="$SCAN_OUTPUT_STEP" -v step=$((TUNER_SAMPLE_RATE/SCAN_BINS)) ' function abs(x){return (x<0)?-x:x} {if(length($1)!=0){if(abs($1-outstep*int($1/outstep)<step)){print $0}} else{print}; fflush();}' | \ awk -v outstep="$SCAN_OUTPUT_STEP" -v thr=$SCAN_POWER_THRESHOLD '{if (length($2)!=0){if(int($2)>thr){print outstep*int(int($1)/outstep)" "$2;fflush()}}}'}# the line below should come before the m10mod if needed.# tee >(c50dft -d1 --ptu --json /dev/stdin > /dev/stderr) | \decode_sonde(){ local bpf3=$(calc_bandpass_param 5000 48000) local bpf9=$(calc_bandpass_param 9600 48000) ( ./iq_client --freq $(calc_bandpass_param "$(($1-TUNER_FREQ))" "$TUNER_SAMPLE_RATE") | tee >( ./csdr bandpass_fir_fft_cc -$bpf9 $bpf9 0.02 | ./csdr fmdemod_quadri_cf | ./csdr limit_ff | ./csdr convert_f_s16 | sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 | ./m10mod --ptu --json > /dev/stderr ) | ./csdr bandpass_fir_fft_cc -$bpf3 $bpf3 0.02 | ./csdr fmdemod_quadri_cf | ./csdr limit_ff | ./csdr convert_f_s16 | sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 | tee >(./dfm09mod --ptu --ecc --json -vv /dev/stdin > /dev/stderr) \ >(./dfm09mod --ptu --ecc --json -i /dev/stdin > /dev/stderr) \ >(./rs41mod --ptu --ecc --crc --json -vv /dev/stdin > /dev/stderr) \ >(./rs92mod -e "$EPHEM_FILE" --crc --ecc --json /dev/stdin > /dev/stderr) | \ aplay -r 48000 -f S8 -t wav -c 1 -B 500000 &> /dev/null ) &>/dev/stdout | while read LINE; do echo "$LINE" | grep --line-buffered -E '^{' | jq --unbuffered -rcM '. + {"freq":"'"$1"'"}' | \ (flock 200; socat -u - UDP4-DATAGRAM:127.255.255.255:$DECODER_PORT,broadcast,reuseaddr) 200>$MUTEX_LOCK_FILE done}declare -A actfreq # active frequenciesdeclare -A slots # active slots(socat -u UDP-RECVFROM:$SCANNER_COM_PORT,fork,reuseaddr - | while read LINE; do case "$LINE" in for freq in "${!actfreq[@]}"; do actfreq[$freq]=$((actfreq[$freq]-1)) [ "${actfreq[$freq]}" -gt "$SLOT_TIMEOUT" ] && actfreq[$freq]=$SLOT_TIMEOUT if [ "${actfreq[$freq]}" -eq 0 ]; then # deactivate slot [ -z "${slots[$freq]}" ] || { cleanup "${slots[$freq]}" unset slots[$freq] } unset actfreq[$freq] elif [ "${actfreq[$freq]}" -ge $SLOT_ACTIVATE_TIME ]; then # activate slot [ -z "${slots[$freq]}" ] && { decode_sonde "$freq" & slots[$freq]=$! } fi done ;; *) freq="${LINE% *}" [ -z "${_FREQ_BLACK_LIST[$freq]}" ] && { [ -n "$freq" ] && actfreq[$freq]=$((actfreq[$freq]+1)) } ;; esacdone) &pid1=$!(while sleep 30; do echo "TIMER30" | socat -u - UDP4-DATAGRAM:127.255.255.255:$SCANNER_COM_PORT,broadcast,reuseaddr; done) &pid2=$!trap "cleanup $pid1 $pid2" EXIT INT TERMrtl_sdr -p $DONGLE_PPM -f $TUNER_FREQ -g $TUNER_GAIN -s $TUNER_SAMPLE_RATE - |tee >(scan_power | socat -u - UDP4-DATAGRAM:127.255.255.255:$SCANNER_COM_PORT,broadcast,reuseaddr) |./iq_server --fft /tmp/fft.out --bo 32 - 2400000 8 |
Results are pretty impressive - I can receive 4 sondes at the same time and the CPU load is about 50-55% which is about 30% less than before!
I have also tried to get power measurements from the iq_server but current implementation in the iq_server is not very optimal ant it loads CPU 30% more than my csdr based solution in the scan_power function.
Update 24 Feb 2021: I have implemented an automatic noise level detection in the scan_power function. Problem with the previous solution was that the threshold had to be adjusted if gain was changed. Additionally some electronic devices could increase the noise level for 2-3 dB and then all slots would be filled with noise signals. Now the noise level is dynamically calculated and the threshold script parameter defines not the absolute value like before but the signal power in dB. So a signal is considered active if its power level is above current noise level plus the threshold. I'm using 6dB as the signal threshold. This change is not reflected here but only on the github.
Now only the auto_rx needs to be updated upstream to work without any hardware by only listening to the UDP! The required changes would be minimal.
The older script from the previous article is renamed and kept here for historical reasons.
Update 12 Aug 2020: Zilog80 has added an FM demodulator called iq_fm. Using his demodulator the decode_sonde can be rewritten the following way:
Despite the fact that all decoders can demodulate the decimated IQ stream I'm using the iq_fm because I start all decoders at the same time and then it is much more efficient to do demodulation only once for all of them. Well I do it twice with different filter parameters for different sonde groups.
As an alternative of starting all decoders at the same time one could use the dft_detect utility. Here I defined a new function decode_sonde_with_type_detect and it is called by default currently:
I haven't yet played with the dft_detect much but it should do
The trick is to add 'TCP' in front of the device_idx number (like I have in the patch above)
Then I simply just start it using 'python3 ./auto_rx.py -t 0 -m UDP -f 402'
Additionally I've added couple of scripts to parse some statistics from the auto-rx log files or even plot sondes' flight path like in the following pictures:
The older script from the previous article is renamed and kept here for historical reasons.
Update 12 Aug 2020: Zilog80 has added an FM demodulator called iq_fm. Using his demodulator the decode_sonde can be rewritten the following way:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 | decode_sonde(){ ( ./iq_client --freq $(calc_bandpass_param "$(($1-TUNER_FREQ))" "$TUNER_SAMPLE_RATE") | tee >( ./iq_fm --lpbw 19.2 - 48000 32 --bo 16 | sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 | ./m10mod --ptu --json > /dev/stderr ) | ./iq_fm --lpbw 10.0 - 48000 32 --bo 16 | sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 | tee >(./dfm09mod --ptu --ecc --json -vv /dev/stdin > /dev/stderr) \ >(./dfm09mod --ptu --ecc --json -i /dev/stdin > /dev/stderr) \ >(./rs41mod --ptu --ecc --crc --json -vv /dev/stdin > /dev/stderr) \ >(./rs92mod -e "$EPHEM_FILE" --crc --ecc --json /dev/stdin > /dev/stderr) | \ aplay -r 48000 -f S8 -t wav -c 1 -B 500000 &> /dev/null ) &>/dev/stdout | while read LINE; do echo "$LINE" | grep --line-buffered -E '^{' | jq --unbuffered -rcM '. + {"freq":"'"$1"'"}' | \ (flock 200; socat -u - UDP4-DATAGRAM:127.255.255.255:$DECODER_PORT,broadcast,reuseaddr) 200>$MUTEX_LOCK_FILE done} |
Despite the fact that all decoders can demodulate the decimated IQ stream I'm using the iq_fm because I start all decoders at the same time and then it is much more efficient to do demodulation only once for all of them. Well I do it twice with different filter parameters for different sonde groups.
As an alternative of starting all decoders at the same time one could use the dft_detect utility. Here I defined a new function decode_sonde_with_type_detect and it is called by default currently:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | start_decoder(){ local decoder bw case "$1" in RS41) decoder="./rs41mod --ptu --ecc --crc --json -vv /dev/stdin > /dev/stderr";bw=10 ;; RS92) decoder="./rs92mod -e "$EPHEM_FILE" --crc --ecc --json /dev/stdin > /dev/stderr";bw=10 ;; DFM9) decoder="tee >(./dfm09mod --ptu --ecc --json /dev/stdin > /dev/stderr) | ./dfm09mod --ptu --ecc --json -i /dev/stdin > /dev/stderr";bw=10 ;; M10) decoder="./m10mod --ptu --json > /dev/stderr";bw=19.2 ;; *) ;; esac ./iq_fm --lpbw $bw - 48000 32 --bo 16 | sox -t raw -esigned-integer -b 16 -r 48000 - -b 8 -c 1 -t wav - highpass 10 gain +5 | tee >(aplay -r 48000 -f S8 -t wav -c 1 -B 500000 &> /dev/null) | eval "$decoder"}decode_sonde_with_type_detect(){ ./iq_client --freq $(calc_bandpass_param "$(($1-TUNER_FREQ))" "$TUNER_SAMPLE_RATE") | (type=$(./dft_detect --iq - 48000 32 | awk -F':' '{print $1}'); start_decoder "$type") &>/dev/stdout | while read LINE; do echo "$LINE" | grep --line-buffered -E '^{' | jq --unbuffered -rcM '. + {"freq":"'"$1"'"}' | \ (flock 200; socat -u - UDP4-DATAGRAM:127.255.255.255:$DECODER_PORT,broadcast,reuseaddr) 200>$MUTEX_LOCK_FILE done} |
Patching auto-rx to let it work without SDR hardware by listening to UDP messages
I've patched current auto-rx trunk (2020-08-16) to let it work without and SDR hardware by only listening to the UDP messages my script broadcasts on port 5678
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| diff --git a/auto_rx.py b/auto_rx.py | |
| index b2f941f..359edad 100644 | |
| --- a/auto_rx.py | |
| +++ b/auto_rx.py | |
| @@ -595,8 +595,8 @@ def main(): | |
| autorx.sdr_list = config['sdr_settings'] | |
| # Check all the RS utilities exist. | |
| - if not check_rs_utils(): | |
| - sys.exit(1) | |
| +# if not check_rs_utils(): | |
| +# sys.exit(1) | |
| # Start up the flask server. | |
| # This needs to occur AFTER logging is setup, else logging breaks horribly for some reason. | |
| diff --git a/autorx/decode.py b/autorx/decode.py | |
| index fe37d12..0d7c93b 100644 | |
| --- a/autorx/decode.py | |
| +++ b/autorx/decode.py | |
| @@ -421,7 +421,7 @@ class SondeDecoder(object): | |
| elif self.sonde_type == "UDP": | |
| # UDP Input Mode. | |
| # Used only for testing of new decoders, prior to them being integrated into auto_rx. | |
| - decode_cmd = "python -m autorx.udplistener" | |
| + decode_cmd = "python3 -m autorx.udplistener" | |
| else: | |
| return None | |
| @@ -822,8 +822,12 @@ class SondeDecoder(object): | |
| # If no subtype field provided, we use the identified sonde type. | |
| _telemetry['type'] = self.sonde_type | |
| - _telemetry['freq_float'] = self.sonde_freq/1e6 | |
| - _telemetry['freq'] = "%.3f MHz" % (self.sonde_freq/1e6) | |
| + if 'freq' in _telemetry: | |
| + _telemetry['freq_float'] = float(_telemetry['freq'])/1e6 | |
| + _telemetry['freq'] = "%.3f MHz" % _telemetry['freq_float'] | |
| + else: | |
| + _telemetry['freq_float'] = self.sonde_freq/1e6 | |
| + _telemetry['freq'] = "%.3f MHz" % (self.sonde_freq/1e6) | |
| # Add in information about the SDR used. | |
| _telemetry['sdr_device_idx'] = self.device_idx | |
| diff --git a/autorx/templates/index.html b/autorx/templates/index.html | |
| index 53352f9..f01c2ac 100644 | |
| --- a/autorx/templates/index.html | |
| +++ b/autorx/templates/index.html | |
| @@ -134,7 +134,14 @@ | |
| attribution: '© '+esrimapLink+', '+esriwholink, | |
| maxZoom: 18, | |
| }); | |
| - sondemap.addControl(new L.Control.Layers({'OSM':osm_map, 'ESRI Satellite':esri_sat_map})); | |
| + var open_topo_map = L.tileLayer( | |
| + 'https://{s}.tile.opentopomap.org/{z}/{x}/{y}.png', | |
| + { | |
| + attribution: '© <a href="https://opentopomap.org/credits">OpenTopoMap</a> contributors', | |
| + maxZoom: 17, | |
| + }); | |
| + | |
| + sondemap.addControl(new L.Control.Layers({'OSM':osm_map, 'ESRI Satellite':esri_sat_map, 'OpenTopoMap':open_topo_map})); | |
| // Home Icon. | |
| var homeIcon = L.icon({ | |
| @@ -504,7 +511,7 @@ | |
| <div class="row"> | |
| <div class='col-12'> | |
| Auto-Follow Latest Sonde: <input type="checkbox" id="sondeAutoFollow" checked> Hide Map: <input type="checkbox" id="hideMap"> | |
| - <div id="sonde_map" style="height:400px;width:100%"></div> | |
| + <div id="sonde_map" style="height:600px;width:100%"></div> | |
| <br> | |
| </div> | |
| </div> | |
| diff --git a/autorx/udplistener.py b/autorx/udplistener.py | |
| index 96722bf..954186d 100644 | |
| --- a/autorx/udplistener.py | |
| +++ b/autorx/udplistener.py | |
| @@ -10,7 +10,7 @@ import traceback | |
| import socket | |
| import sys | |
| -def udp_rx_loop(hostname='localhost', port=50000): | |
| +def udp_rx_loop(hostname='', port=50000): | |
| """ | |
| Listen for incoming UDP packets, and emit them via stdout. | |
| """ | |
| @@ -53,6 +53,6 @@ if __name__ == "__main__": | |
| if len(sys.argv) > 1: | |
| _port = int(sys.argv[1]) | |
| else: | |
| - _port = 50000 | |
| + _port = 5678 | |
| udp_rx_loop(port=_port) | |
| diff --git a/station.cfg b/station.cfg | |
| index 0eed214..b6a781f 100644 | |
| --- a/station.cfg | |
| +++ b/station.cfg | |
| @@ -20,7 +20,7 @@ sdr_quantity = 1 | |
| # If using multiple SDRs, you MUST allocate each SDR a unique serial number using rtl_eeprom | |
| # i.e. to set the serial number of a (single) connected RTLSDR: rtl_eeprom -s 00000002 | |
| # Then set the device_idx below to 00000002, and repeat for the other [sdr_n] sections below | |
| -device_idx = 0 | |
| +device_idx = TCP0 | |
| # Frequency Correction (ppm offset) | |
| # Refer here for a method of determining this correction: https://gist.github.com/darksidelemm/b517e6a9b821c50c170f1b9b7d65b824 | |
Then I simply just start it using 'python3 ./auto_rx.py -t 0 -m UDP -f 402'
Additionally I've added couple of scripts to parse some statistics from the auto-rx log files or even plot sondes' flight path like in the following pictures:
 |
| Full flight path of a nearby sonde |
 |
| Partial flight path of a faraway sonde |
1 comment:
Alex, thanks for your code...
please it is possible to update the patch for auto_rx because the actually version is 1.53?
Lutz
Post a Comment