Tuesday, December 24, 2013

New Morse Decoder - Part 2

New Bayesian Morse Decoder 


In my previous post I shared initial results of the new Bayesian Morse Decoder.  I have worked a bit more on this topic to integrate the decoder software to FLDIGI v3.21.75.   Dave  W1HKJ provided me some space on his web server so I posted full source code including all changes and new source files related to Bayesian Morse decoder in here.

To demonstrate the Bayesian decoding capability I tuned my KX3  on W1AW frequency 3.58105 Mhz  at 8:00 PM EST today and let two identical FLDIGI copies of the same v3.21.75 software running from the same audio source.  I have a SignaLink USB connected between KX3  and my ThinkPad laptop running Linux Mint OS.

On Fig 1. below  I have the legacy SOM decoder on the left side and the new Bayesian decoder on the right .  The legacy decoder makes a lot of errors and CW bulletin is not really readable.  On the right side I have enabled Bayesian decoder and you can actually read the bulletin, though the lack of proper word spaces make this still a bit difficult to read. In any case Bayesian decoder provides significant accuracy improvement with real life noisy CW signals.

Figure 1.  FLDIGI running legacy and Bayesian Morse decoder





If you choose to  compile FLDIGI from the sources yourself  please see Figure 2. below.  When you go to menu item Configure / Modems / CW / General  and click the tick box "Bayes decoding"  that will enable the new Bayesian decoder (and disables legacy decoder).  You will also notice that CW RX speed indicator  (Fig 1. bottom left corner of FLDIGI window)  will change more rapidly - this is a feature of the new decoder.  It evaluates the instantaneous speed continuously and uses this information as part of the decoding process.

Figure 2.  Enabling Bayesian decoder. 



I would be interested in getting your feedback and suggestions how to improve this software.  Current version is buggy  and it does crash occasionally.  Also, I still have not found the root cause of the problem that causes approximately 3% base error rate.

There are number of improvement areas that need work. For example noise estimator ("noise.c") does not seem to be able to keep track of the real noise level. This noise power information is later used by Kalman filters  ("kalfil.c") in estimating likelihood of keystate given the observed signals. Bayesian inference is then used in updating posterior conditional probabilities for each new path in ("probp.c"). I am getting intermittent overflows in this section of the code. Any help in this area would be very welcome.

However,  given the visible improvement in the decoding accuracy at least during certain conditions  (when you have the signal level correct, not too much noise etc.) I am now posting this for alpha testing with hopes that I could get some more eyes to look at the code and find those bugs.

Happy Holidays!

73
Mauri AG1LE  




11 comments:

  1. Hi,

    I've built the software. I have no experience in amateur radio but do have some in C and signal processing. I'm interested in seeing if I can help with the decoder. Could you please provide some sample wav files and instructions for how to run the them through the decoder via the UI?

    Cheers,
    Roy

    ReplyDelete
    Replies
    1. Hi Roy
      Here is an example WAV file https://www.dropbox.com/sh/2j9chxsizxjeqgp/4C36_4qby6/rand16db20wpm.wav that you can use. If you are able to compile FLDIGI from sources (they are here http://www.w1hkj.com/ag1le/) you can install the software and open the WAV file by "File / Audio / Playback" menu. Once you have the audio running enable tickbox "Bayesian decoder" in Fig2. above. This menu is in "Configure / Modems / CW / General" section.

      Delete
    2. Hi, I can't find the "File / Audio / Playback" menu. Under "File" I only have "Folders", "Macros", "Text Capture" and "Exit".

      Delete
    3. Hi Anonymous
      have you installed sndfile library on your system? In many systems you can do this by "sudo apt-get install libsndfile" or "sudo apt-get install libsndfile-dev" if you want sources. You need to run ./configure again and compile sources.
      73
      Mauri

      Delete
  2. IS there a language model here? Language models are typically used in speech recognition, and I bet could provide a huge improvement to the readout. If you don't have one, a model would substantially change your inference procedures, but it would provide some "top down" knowledge about likely sequences of characters that I bet would help you a lot!

    ReplyDelete
    Replies
    1. This software does not include a language model as ham radio operators use many different kinds of "languages". In contest situation they use short acronyms, while in "rag chewing" more normal language but peppered with shorthands etc. I have thought about language models but would like to get this layer working properly before starting another project.

      Delete
  3. Merry Christmas and a Hapy New Year to you and yours.
    I am using the Windows version of Fldigi (3.21.77) is there any method to include Bayes decoding? Maybe in a futuer version?
    73

    ReplyDelete
    Replies
    1. Once I get this working in reliable manner then Dave W1HKJ can make a decision to include it in mainline FLDIGI releases. However, I need more help to finish alpha testing and fix the bugs in the code.
      73
      Mauri

      Delete
  4. Thank you, unfortunately I can n't help you in this testings. Good luck.
    73

    ReplyDelete
  5. Your Bayesian decoder does not detect WORD SPACE of 7 bits and makes text almost unreadable to humans. Compare FLDIGI with good MRP40 CW decoder.
    I am glad you continued your research on this noncommercial subject. I stoped mine as I enjoy ionospheric peak for few more years.

    73 de Mario, S56A, N1YU - HF CW DSP PhD in 1990s and again in 2010

    ReplyDelete
    Replies
    1. Hi Marijan

      In the initial alpha release there was a bug in transl.cxx file that caused word space to be decoded but not printed on scree. This bug was fixed in the subsequent release. The latest release is in here: http://www.w1hkj.com/ag1le/fldigi-3.21.77.bayes.tar.gz

      73
      Mauri AG1LE

      Delete