TensorFlow: a new LSTM RNN based Morse decoder

INTRODUCTION

In my previous post I created an experiment to train a LSTM Recurrent neural network (RNN) to detect symbols from noisy Morse code. I continued experiments, but this time I used the new TensorFlow open source library for machine intelligence. The flexible architecture of TensorFlow allows to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API.

TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well.

EXPERIMENT 

I started with the TensorFlow MNIST example authored by Aymeric Damien. MNIST is a large database of handwritten digits that is commonly used for machine learning experiments and algorithm development. Instead of training a LSTM RNN model using handwritten characters I created a Python script to generate a lot of Morse code training material. I downloaded ARRL Morse training text files and created a large text file. From this text file the Python script generates properly formatted training vectors, over 155,000 of them.  The software is available as Python inotebook format in Github.

The LSTM RNN model has the following parameters:

# Parameters
learning_rate = 0.001 
training_iters = 114000 
batch_size = 126

# Network Parameters
n_input = 1 # each Morse element is normalized to dit length 1 
n_steps = 32 # timesteps (training material padded to 32 dit length)
n_hidden = 128 # hidden layer num of features 
n_classes = 60 # Morse character set 

The training takes approximately 15 minutes on my Thinkpad X301 laptop. The progress of loss function and accuracy % over the training is depicted in Figure 1 below. The final accuracy was 93.6% after 114,000 training samples.

Figure 1.  Training progress over time

I was testing the model with generated data while adding noise gradually to signals using the "sigma" parameter on the Python scripts.  The results are below:

Test case:     QUICK BROWN FOX JUMPED OVER THE LAZY FOX 0123456789
Results:
Noise 0.0:  QUICK BROWN VOC YUMPED OVER THE LACY VOC ,12P45WOQ.
Noise 0.02: QUICK BROWN VOC YUMPED OVER THE LACY FOC 012P45WOQ.
Noise 0.05: QUICK BROWN VOC YUMPED OVER THE LACQ VOC ,,2P45WO2.
Noise 0.1:  Q5IOK BROWN FOX YUMPED O4ER THE LACY FOC 012P4FWO2,
Noise 0.2: .4IOK WDOPD VOO 2FBPIM QFEF TRE WAC2 4OX 0,.PF52Q91
As can be seen above at "sigma" level 0.2 the decoder starts to make a lot of errors.

CONCLUSIONS

The software learns the Morse code by going through the training vectors multiple times. By going through 114,000 characters in training the model achieves 96.3% accuracy. I did not try to optimize anything and I just used the reference material that came with TensorFlow library. This experiment shows that it is possible to build an intelligent Morse decoder that learns the patterns from the data and also allows to scale up more complex models with better accuracy and better tolerance for QSB and noisy signals.

TensorFlow proved to be a very powerful new machine learning library that was relatively easy to use. The biggest challenge was to figure out what data formats to use with various API calls. Due to the complexity and richness of the TensorFlow library I am fairly sure that much can be done to improve the efficiency of this software. As TensorFlow has been designed so that it works on a desktop, server, tablet or even on a mobile phone this open new possibilities to build an intelligent, learning Morse decoder for different platforms.

 73 Mauri AG1LE