Noise Reduction Method for Morse Signals

MATCHING PURSUIT METHOD FOR NOISE REDUCTION 

In my previous post I shared a new method for reducing noise from Morse signals based on MPTK toolkit.  I contacted Dr. Remi Gribonval who is one of the authors of the MPTK toolkit and asked for his advice in how to optimize the atoms for Morse code signals.   To my surprise he responded the same day and gave me some great ideas. There was also a fruitful discussion in eham.net  that helped me to figure out how to optimize the dictionary suitable for processing audio signals containing Morse code.

FREQUENCY DOMAIN VIEW 

I used the same pileup.wav file than in the previous post and used the new, revised dictionary by issuing the following commands

mpd -D dic_mdct_morse2.xml -s 10 pileup.wav pileup.bin
mpr pileup.bin pileup_reco.wav

Runtime of mpd command was 73 seconds on Thinkpad X301 laptop. The length of pileup.wav is 71.5 seconds, so mpd can decompose audio almost in real time to selected 10 dB SNR.

To illustrate the noise reduction capability I zoomed into a four second long audio segment starting at 1:00.0.  The result is shown in figure 1 spectrum plot below.  Figure 2 shows the original noisy signal. It is easy to observe that there has been quite significant noise reduction and Morse signals are much more visible from background noise.  Unlike a normal bandpass filter that passes only the frequency of a single CW signal within range, this new method automatically detects and passes all CW signals in a broad frequency range while reducing surrounding noise. You may want to download the filtered audio file to listen and compare with the original audio file.

Fig 1. Filtered pileup signal

Fig 2. Original pileup signal

Here is the content of the dictionary file  dic_mdct_morse2.xml.  Sampling rate used in the original audio file was 8000 Hz.

<?xml version="1.0" encoding="ISO-8859-1"?>
<dict>
<libVersion>0.2</libVersion>
        <block>
        <param name="blockOffset" value="0" />
        <param name="fftSize" value="256" />
        <param name="type" value="mdst" />
        <param name="windowLen" value="256" />
        <param name="windowShift" value="32" />
        <param name="windowopt" value="0" />
        <param name="f0Max" value="800" />
        <param name="f0Min" value="400" />     
        <param name="windowtype" value="rectangle" />
   </block>

        <block>
        <param name="blockOffset" value="0" />
        <param name="fftSize" value="256" />
        <param name="type" value="mdst" />
        <param name="windowLen" value="256" />
        <param name="windowShift" value="64" />
        <param name="windowopt" value="0" />
        <param name="f0Max" value="800" />
        <param name="f0Min" value="400" />     
        <param name="windowtype" value="rectangle" />
   </block>

</dict>

TIME DOMAIN VIEW 

I wanted to see how this new noise reduction method impacts the amplitude envelope of the CW signals. I created a sample Morse audio test file with 20 dB SNR and used a dictionary file dic_mdct_morse.xml below for processing.   Figure 3. below shows letter Q  - as you can observe there is visible noise riding on signal and also between "dahs" and "dit".

Fig 3.  Letter Q  with 20 dB SNR

Figure 4. below shows filtered signal.  The waveform shows no noise between "dahs" and "dit".  Also, there is very little noise on signal but selected atoms are visible as 3 "bumps" riding on top of "dah".

Fig 4. Letter Q filtered

I did another experiment with much worse signal-to-noise ratio.  Figure 6. shows letters QUI  with 6 dB SNR  (this was normalized to -3 dB before processing to prevent overflows).  Figure 5 shows the filtered version of this signal. Noise reduction is visible but there is still some noise left.

Fig 5.  Filtered QUI 

Fig 6.  Letters QUI with 6 dB SNR

Here is the dictionary file  dic_mdct_morse.xml used in the time domain examples above.  The sampling rate in these artificially generated files is 4000 Hz therefore the atom sizes are scaled down.


<?xml version="1.0" encoding="ISO-8859-1"?>
<dict>
<libVersion>0.2</libVersion>
        <block>
        <param name="blockOffset" value="0" />
        <param name="fftSize" value="128" />
        <param name="type" value="mdst" />
        <param name="windowLen" value="128" />
        <param name="windowShift" value="16" />
        <param name="windowopt" value="0" />
        <param name="f0Max" value="800" />
        <param name="f0Min" value="400" />     
        <param name="windowtype" value="rectangle" />
   </block>

        <block>
        <param name="blockOffset" value="0" />
        <param name="fftSize" value="128" />
        <param name="type" value="mdst" />
        <param name="windowLen" value="128" />
        <param name="windowShift" value="32" />
        <param name="windowopt" value="0" />
        <param name="f0Max" value="800" />
        <param name="f0Min" value="400" />     
        <param name="windowtype" value="rectangle" />
   </block>

</dict>

CONCLUSIONS

Matching Pursuit is a powerful new digital audio processing method for reducing noise in Morse signals.  The experiments above demonstrate that it is possible to significantly improve signal to noise ratio even in cases where there are multiple CW stations sending simultaneously at nearby frequencies.
MPTK toolkit provides rich set of tools to further optimize the digital signal processing. Examples above were the result of some 3 hours of work testing various atoms and related parameters.

All software used is available as open source making it feasible to integrate this noise reduction method to various other projects.  If you find this article valuable please provide your comments below.


73
Mauri AG1LE

Sparse representations of noisy Morse signals - matching pursuit algorithm in Morse decoders

MATCHING PURSUIT ALGORITHM IN MORSE DECODING

Recently I started doing some research on sparse representations of audio signals. There is a lot of research papers available in how to use sparse coding for machine learning purposes, such as computer vision and audio classification. There is also growing body of evidence that sparseness constitutes a general principle of sensory coding in the human nervous system.

There are also many algorithms designed to transform audio signals to sparse codes.  One such algorithm is Matching Pursuit  that decomposes any signal into linear expansion of waveforms that are selected from a redundant dictionary of functions. A matching pursuit isolates the signal structures that are coherent with respect to given dictionary.  A classic paper that explains this approach was written by Stephane G. Mallat and Zhifeng Zhang in 1993.

Matching Pursuit Tool Kit (MPTK) provides a fast implementation of the Matching Pursuit algorithm for the sparse decomposition of multichannel signals. It comprises a C++ library, standalone command line utilities, and some scripts for running it and plotting the results through Matlab. This article explains the concept and the implementation details of the toolkit.

To experiment with sparse coding I selected a noisy WAV audio file that contains 1:15 long CW pileup example.   I used Audacity to review the spectrum  plot of the original audio file as shown on Fig 1. below. The audio sample contains some 13 ... 15 CW stations, some hardly audible under noise.

Fig 1.  Spectrum display - original audio file

The MPTK toolkit comes with a library of dictionaries. I tested all of them and also created also my own variations to come up with a set of suitable "atoms"  that would be able to extract CW signals from the audio file above.  I also reconstructed the audio file from book of decomposed "atoms" to listen the outcome.  

The best initial results were obtained by using "dic_test.xml"  dictionary that is composed of a combination of  harmonic and gabor atoms. Figure 2. below shows the spectrum plot of reconstructed audio file and noise reduction is quite visible. The CW signals are highlighted with red and white colors whereas background noise has almost disappeared.

Fig 2.  Spectrum display - reconstructed audio file 

I am also enclosing the reconstructed audio file.  It sounds quite different from the original, noisy audio. The higher frequency signals sound like echo (perhaps too much harmonics) and at 900 Hz  IN3NJB is clear but dits and dahs could have more sharp edges (perhaps too long gabor wavelet?).  

This experiments shows that Matching Pursuit is a very powerful signal processing tool that could be used to improve Morse decoder capabilities, especially with pileups and noisy signals.  However, more work is needed to build an optimized dictionary of "atoms".  For example the modified Morlet wavelet discussed in this article  might improve the reconstruction accuracy.  

If you find this article interesting please provide your comments and feedback below. 

73 

Mauri AG1LE