TensorFlow revisited: a new LSTM Dynamic RNN based Morse decoder

TensorFlow revisited: a new LSTM Dynamic RNN based Morse decoder

It has been almost two years since I was playing with TensorFlow based Morse decoder.  This is a long time in the rapidly moving Machine Learning field.

I created a new version of the LSTM Dynamic RNN based Morse decoder using TensorFlow package and Aymeric Damien's example.  This version is much faster and has also ability to train/decode on variable length sequences.  The training and testing sets are generated from sample text files on the fly, I included the Python library and the new TensorFlow code in my Github page. 

The demo has ability to train and test using datasets with noise embedded.    Fig 1. shows the 50 first test vectors with gaussian noise added. Each vector is padded to 32 values.  Unlike the previous version of LSTM network this new version has ability to train variable length sequences.  The Morse class handles the generation of training vectors based on input text file that contains randomized text. 

Fig 1. "NOW 20 WPM TEXT IS FROM JANUARY 2015 QST PAGE 56 " 

Below are the TensorFlow model and network parameters I used for this experiment: 

# MODEL Parameters

learning_rate = 0.01

training_steps = 5000

batch_size = 512

display_step = 100

n_samples = 10000 

# NETWORK  Parameters

seq_max_len = 32 # Sequence max length

n_hidden = 64    # Hidden layer num of features  

n_classes = 60   # Each morse character is a separate class

Fig 2. shows the training loss and accuracy by minibatch. This training took 446.9 seconds and final testing accuracy reached was 0.9988.  This training session was done without any noise in the training dataset. 

Fig 2. Training Loss and Accuracy plot.

Sample session to use the trained model is below: 

# ================================================================

#   Use saved model to predict characters from Morse sequence data

# ================================================================

# 

NOISE = False

saver = tf.train.Saver()

testset = Morse(n_samples=10000, max_seq_len=seq_max_len,filename='arrl2.txt')

test_data = testset.data

if (NOISE): 

    test_data = test_data +  normal(0.,0.1, 32*10000).reshape(10000,32,1)

test_label = testset.labels

test_seqlen = testset.seqlen

# Later, launch the model, use the saver to restore variables from disk, and

# do some work with the model.

with tf.Session() as sess:

    # Restore variables from disk.

    saver.restore(sess, "/tmp/morse_model.ckpt")

    print("Model restored.")

    y_hat = tf.argmax(pred,1)

    ch = sess.run(y_hat, feed_dict={x: test_data, y: test_label,seqlen: test_seqlen})

    s = ''

    for c in ch:

        s += testset.decode(c)

    print( s)

Here is the output from the decoder (this is using arrl2.txt file as input): 

INFO:tensorflow:Restoring parameters from /tmp/morse_model.ckpt
Model restored.
NOW 20 WPM TEXT IS FROM JANUARY 2015 QST  PAGE 56 SITUATIONS WHERE I COULD HAVE BROUGHT A DIRECTIONAL ANTENNA WITH ME, SUCHAS A SMALL YAGI FOR HF OR VHF.  IF ITS LIGHT ENOUGH, ROTATING A YAGI CAN BEDONE WITH THE ARMSTRONG METHOD, BUT IT IS OFTEN VERY INCONVENIENT TO DO SO.PERHAPS YOU DONT WANT TO LEAVE THE RIG BEHIND WHILE YOU GO OUTSIDE TO ADJUST THE ANTENNA TOWARD THAT WEAK STATION, OR PERHAPS YOU'RE IN A TENT AND ITS DARK OUT THERE.  A BATTERY POWERED ROTATOR PORTABLE ROTATION HAS DEVELOPED A SOLUTION TO THESE PROBLEMS.  THE 12PR1A IS AN ANTENNA ROTATOR FIGURE 6 THAT FUNCTIONS ON 9 TO 14 V DC.  AT 12 V, THE UNIT IS SPECIFIED TO DRAW 40 MA IDLE CURRENT AND 200 MA OR LESS WHILE THE ANTENNA IS TURNING. IT CAN BE POWERED FROM THE BATTERY USED TO RUN A TYPICAL PORTABLE STATION.WHILE THE CONTROL HEAD FIGURE 7 WILL FUNCTION WITH AS LITTLE AS 6 V, A END OF 20 WPM TEXT QST DE AG1LE  NOW 20 WPM     TEXT IS FROM JANUARY 2014 QST  PAGE 46 TRANSMITTER MANUALS SPECIFI

As the reader can observe the LSTM network has learned near perfectly to translate incoming Morse sequences to  text. 

Next I did set the NOISE variable to True.  Here is the decoded message with noise: 

NOW J0 O~M TEXT IS LRZM JANUSRQ 2015 QST  PAGE 56 SITRATIONS WHEUE I XOULD HAVE BRYUGHT A DIRECTIZNAF ANTENNS WITH ME{ SUYHSS A SMALL YAGI FYR HF OU VHV'  IV ITS LIGHT ENOUGH, UOTSTING A YAGI CAN BEDONE FITH THE ARMSTRONG METHOD8 LUT IT IS OFTEN VERQ INOGN5ENIENT TC DG SC.~ERHAPS YOR DZNT WINT TO LEAVE THE RIK DEHIND WHILE YOU KO OUTSIME TO ADJUST THE AATENNA TYOARD THNT WEAK STTTION0 OU ~ERHAPS COU'UE IN A TENT AND ITS MARK OUT THERE.  S BATTERC JYWERED RCTATOR ~ORTALLE ROTATION HAS DEVELOOED A SKLUTION TO THESE ~UOBLEMS.  THE 1.JU.A IS AN ANTENNA RYTATCR FIGURE 6 THAT FRACTIZNS ZN ) TO 14 V DC1  AT 12 W{ THE UNIT IS SPECIFIED TO DRSW }8 MA IDLE CURRENT AND 20' MA OR LESS WHILE THE ANTENNA IS TURNING. IT ZAN BE POOERED FROM THE BATTEUY USED TO RRN A T}~IXAL CQMTUBLE STATION_WHILE IHE }ZNTROA HEAD FIGURE 7 WILA WUNXTION WITH AS FITTLE AA 6 F8 N END ZF 2, WPM TEXT OST ME AG1LE  NOW 20 W~M     TEXT IS LROM JTNUARJ 201} QST  ~AGE 45 TRANSMITTER MANUALS S~ECILI

Interestingly this text is still quite readable despite noisy signals. The model seems to mis-decode some dits and dahs but the word structure is still visible. 

As a next step I re-trained the network using the same amount of noise in the training dataset.  I expected the loss and accuracy to be worse.   Fig 3. shows that training accuracy to 0.89338 took much longer and maximum testing accuracy was only 0.9837.

Fig. 3  Training Loss and Accuracy with noisy dataset

With the new model trained using noisy data I did re-run the testing phase. Here is the decoded message with noise:

NOW 20 WPM TEXT IS FROM JANUARY 2015 QST  PAGE 56 SITUATIONS WHERE I COULD HAWE BROUGHT A DIRECTIONAL ANTENNA WITH ME0 SUCHAS A SMALL YAGI FOR HF OR VHF1  IF ITS LIGHT ENOUGH0 ROTATING A YAGI CAN BEDONE WITH THE ARMSTRONG METHOD0 BUT IT IS OFTEN VERY INCONVENIENT TO DO SO1PERHAPS YOU DONT WANT TO LEAVE THE RIG BEHIND WHILE YOU GO OUTSIDE TO ADJUST THE ANTENNA TOWARD THAT WEAK STATION0 OR PERHAPS YOU1RE IN A TENT AND ITS DARK OUT THERE1  A BATTERY POWERED ROTATOR PORTABLE ROTATION HAS DEVELOPED A SOLUTION TO THESE PROBLEMS1  THE 12PR1A IS AN ANTENNA ROTATOR FIGURE 6 THAT FUNCTIONS ON 9 TO 14 V DC1  AT 12 V0 THE UNIT IS SPECIFIED TO DRAW 40 MA IDLE CURRENT AND 200 MA OR LESS WHILE THE ANTENNA IS TURNING1 IT CAN BE POWERED FROM THE BATTERY USED TO RUN A TYPICAL PORTABLE STATION1WHILE THE CONTROL HEAD FIGURE Q WILL FUNCTION WITH AS LITTLE AS X V0 A END OF 20 WPM TEXT QST DE AG1LE  NOW 20 WPM     TEXT IS FROM JANUARY 2014 QST  PAGE 46 TRANSMITTER MANUALS SPECIFI

As reader can observe now we have nearly perfect copy from noisy testing data.  The LSTM network has gained ability to pick-up the signals from noise.  Note that training data and testing data are two completely separate datasets.

CONCLUSIONS

Recurrent Neural Networks have gained a lot of momentum over the last 2 years. LSTM type networks are used in machine learning systems, like Google Translate,  that can translate one sequence of characters to another language efficiently and accurately.  

This experiment shows that a relatively small TensorFlow based  neural network  can learn  Morse code sequences and translate them to text.   This experiment shows also that  adding noise to the training data  will slow down the learning rate and will impact overall training accuracy achieved.  However,  applying similar noise level in the testing phase will significantly improve  the testing accuracy when using a model trained under noisy training signals. The network has learned the signal distribution and is able to decode more accurately. 

So what are the practical implications of this work?   With some signal pre-processing LSTM RNN could provide a self learning Morse decoder that only needs a set of labeled audio files to learn a particular set of sequences.  With large enough training dataset the model could achieve over 95% accuracy.

73  de AG1LE 

Mauri