In my previous blog post I described a new Tensorflow based Machine Learning model that learns Morse code from annotated audio .WAV files with 8 kHz sample rate.
In order to evaluate the performance characteristic of the decoding accuracy from noisy audio source files I created a set of training & validation materials with Signal-to-Noise Ratio from -22dB to +30 dB. Target SNR_dB was created using the following Python code:
# Desired linear SNR
SNR_linear = 10.0**(SNR_dB/10.0)
# Measure power of signal - assume zero mean
power = morsecode.var()
# Calculate required noise power for desired SNR
noise_power = power/SNR_linear
# Generate noise with calculated power (mu=0, sigma=1)
noise = np.sqrt(noise_power)*np.random.normal(0,1,len(morsecode))
# Add noise to signal
morsecode = noise + morsecode
These audio .WAV files contain random words with maximum 5 characters - 5000 samples at each SNR level with 95% used for training and 5% for validation. The Morse speed in each audio sample was randomly selected from 30 WPM or 25 WPM.
The training was performed until 5 consecutive epochs did not improve the character error rate. The duration of these training sessions varied from 15 - 45 minutes on Macbook Pro with2.2 GHz Intel Core i7 CPU.
I captured and plotted the Character Error Rate (CER) and Signal-to-Noise Ratio (SNR) of each completed training and validation session. The following graph shows that the Morse decoder performs quite well until about -12 dB SNR level and below that the decoding accuracy drops fairly dramatically.
|CER vs. SNR graph
To view how noisy these files are here are some random samples - first 4 seconds of 8KHz audio file is demodulated, filtered using 25Hz 3rd order Butterworth filter and decimated by 125 to fit into a (128,32) vector. These vectors is shown as grayscale images below:
|-6 db SNR
|-11 dB SNR
|-13 dB SNR
|-16 dB SNR
The Tensorflow model appears to perform quite well on decoding noisy audio files, at least when the training set and validation set have the same SNR level.
The next experiments could include more variability with a much bigger training dataset that has a combination of different SNR, Morse speed and other variables. The training duration depends on the amount of training data so it can take a while to perform these larger scale experiments on a home computer.