Multichannel CW decoder for FLDIGI
I have been working on the Bayesian Morse decoder for a while. The latest effort was focused on making it possible to automatically detect all CW signals in the audio band and spin up a new instance of the Bayesian decoder for each detected signal.Figure 1. shows a running version implemented on top of FLDIGI version 3.21.77. The waterfall shows 9 CW signals from 400Hz to 1200 Hz. The software utilizes the FLDIGI Signal Browser user interface and you can set the signal threshold using the slider bar below the signal browser window. The user interface works very much like the PSK or RTTY browser.
 |
| Figure 1. Multichannel CW decoder for FLDIGI |
Fs = 8000; % Fs is sampling frequency - 8 Khz
Ts = 10*Fs; % Total sample time is 10 seconds
% create 9 different parallel morse sessions - 10 seconds each at 20-35 WPM speed
% TEXT audio file noiselevel Hz speed WPM
x1=morse('CQ TEST DE AG1LE','cw1.wav', 20,1200,Fs,20, Ts);
x2=morse('TEST DE SP3RQ CQ','cw2.wav', 15, 1100,Fs,35, Ts);
x3=morse('DE W3RQS CQ TEST','cw3.wav', 20, 1000,Fs,30, Ts);
x4=morse('SM0LXW CQ TEST DE','cw4.wav',15, 900,Fs, 25, Ts);
x5=morse('CQ TEST DE HS1DX','cw5.wav', 20, 800,Fs, 20, Ts);
x6=morse('TEST DE JA1DX CQ','cw6.wav', 10, 700,Fs, 20, Ts);
x7=morse('DE JA2ATA CQ TEST','cw7.wav',20, 600,Fs, 20, Ts);
x8=morse('UA2HH CQ TEST DE','cw8.wav', 15, 500,Fs, 20, Ts);
x9=morse('CQ TEST DE CT1CX','cw9.wav', 20, 400,Fs, 20, Ts);
% weighted sum - merge all the audio streams together
% 9 signals arranged in frequency order 1200Hz ... 400Hz
y = 0.1*x1 + 0.1*x2 + 0.1*x3 + 0.1*x4 + 0.1*x5 + 0.1*x6 + 0.1*x7 + 0.1*x8 + 0.1*x9;
% write to cwcombo.wav file
wavwrite(y,Fs,'cwcombo.wav');
I have saved the full sources of this experimental FLDIGI version in Github: FLDIGI source
UPDATE July 20, 2014: I rebased this using Source Forge latest branch - new version is here: fldigi-3.22.0CN.tar.gz
Let me know if you are interested in testing this software. I would be interested in getting feedback on scalability, any performance problems as well as how well the CW decoder works with real life signals.
73
Mauri AG1LE
Hello Mauri, I would like to test the software. I'm not a CW user at all and not able to decode morse by myself. So I totally depend on software to decode CW. I use DM780 from HRD till now but it has difficulties decoding when there are to many signals around. Besides that the signal has to be over 20wpm and strong for a reliable decode. I'm curious how your software will behave? 73, Bas
ReplyDeleteDo you have ability to compile FLDIGI from the sources? I am using Linux Mint distribution for development and testing. The tools are standard linux development tools. Instructions how to compile from source can be found from http://www.w1hkj.com/FldigiHelp-3.21/html/install_from_source_page.html
DeleteI have also used DM780 from HRD on my Windows machine and not happy with CW decoder. Is your main workstation Windows based?
73
Mauri AG1LE
Hello Mauri, I'll try to find some time to install Linux Mint and compile the source. My main computer is windows based. I guess this is all not working in Windows. 73, Bas
DeleteHi Bas
DeleteI am working to get a Windows release out. I don't have capability to cross-compile so I need some help there. Stay tuned - I will make a new post when experimental Windows release is available for testing.
73
Mauri
Hi,
ReplyDeleteGreat to see this work progressing so quickly.
There seems to be a missing file from the source code tarball. I tried to compile this on Linux Mint 16 and got the following error:
cw_rtty/cw.cxx:40:21: fatal error: view_cw.h: No such file or directory
#include "view_cw.h"
How can I get this file?
Hi
DeleteSorry for this - I added missing file to src/Makefile.am and rebuilt the distribution file. Try downloading again and rebuilding it. I did this and now the missing view_cw.h was in the dist file.
73
Mauri AG1LE
Mauri,
ReplyDeleteI use both Fldigi and CWSkimmer. Glad to see someone working on a decoder similer to the one CWSkimmer uses.
I would like to test this, but not interested in compiling the source code. Is the compiled program available?
Larry-W8LIG
Hi Larry
DeletePre-compiled Windows version is not yet available. I haven't configured Windows compilation but I can check with Dave W1HKJ if he knows anybody who could do it and post it.
73
Mauri AG1LE
Larry - not sure if you have seen the follow up post. Pre-compiled Windows version is now available at http://w1hkj.com/ag1le/
DeleteHi,
ReplyDeleteI've got it running now and am testing it on real signals. Very neat! One question - I notice some signals high in the passband (say, above 1500 Hz), that I can identify by ear as CW, are not detected and decoded by the Bayesian decoder (in the signal browser window). If I double-click the signal, it is decoded in the main RX window (but still not the signal browser window). Can you think of some reason for this?
Still need to do more testing... it seems many of the signals I am receiving on 40m now have a poor fist. :-)
I got it. changed fupper to 3000 in view_cw.cxx and now I can decode signals up to 3000 Hz in the passband.
DeleteI used flower and fupper to set the limits in method view_cw::findsignals(struct PEAKS *p) in the view_cw.cxx. The maximum is IMAGE_WIDTH which is 4000 Hz in my version. This corresponds with Nyquist frequency (1/2 of sampling frequency 8000 Hz used for FFT for Waterfall display).
Delete>Still need to do more testing... it seems many of the signals I am receiving on 40m now have a poor fist. :-)
DeleteIt looks like speed estimator initialization has a bug. I am getting "T T O E ..." sequences and speed indicator "CW Rx 50" under waterfall is stuck on 50 WPM. Let me check the code in initl.cxx. The ilrate[] array contains the initialized values.
qrp-gaijin
DeleteSee the updated version - speed estimation bug is fixed in this one.
Updated code available in W1HKJ website - link updated in the article.
At last, someone working on a skimmer alternative.
ReplyDeleteI am willing to give it some testing, willl it complile on an Ubuntu system do you think?
I'm fairly OK with C, C++ but not so brilliant with linux OS, but I can normally compile stuff ok.
Iain
Hi Iain
DeleteI am using Linux Mint 15 for development. You can test compilation fairly quickly.
- dowload the tar.gz package to a directory
- tar xvf fldigi-3.21.77.multich.tar.gz
- cd fldigi-3.21.77
- ./configure
- make
if you get an error your system may miss some required libraries. See http://www.w1hkj.com/FldigiHelp-3.21/html/install_from_source_page.html for help how to install those libraries.
Hi,
DeleteInstalled the latest fltk and it nearly completes the compilation but errors with:
fileselector/fileselect.cxx:109:16: error: ‘namespace FSEL { }’ redeclared as different kind of symbol
./include/fileselect.h:9:7: error: previous declaration of ‘class FSEL’
Any ideas?
Hi Iain
DeleteI did some search but could not find this bug. I did rebase the software on Dave W1HKJ's request using branch in Source Forge. Updated distribution is in here: http://www.w1hkj.com/ag1le/fldigi-3.22.0CN.tar.gz
Eveybody - see update: latest version fldigi-3.22.0CN.tar.gz is http://www.w1hkj.com/ag1le/ - this has some bug fixes on speed estimation side, it works better now.
ReplyDeleteI am working with Dave, W1HKJ to get Windows binaries as there seems to be demand for those.
A new version with several bugfixes posted in here: http://www.w1hkj.com/ag1le/fldigi-3.21.83cw-a1.tar.gz
ReplyDeleteThanks. I got the 3.22CN version compiled ok, but the link above is broken?
ReplyDeleteWindows version is now available for testing - see http://ag1le.blogspot.com/2014/07/new-morse-decoder-part-6.html
ReplyDeleteHi Mauri,
ReplyDeleteglad to see your work on the CW decoding for fldigi. Are there plans to integrate it permanently to the official fldigi versions? Are there plans for a connection to the reverse beacon network, so fldigi with your extension becomes a real alternative to CWSkimmer? I think this would be a significant development, thinking of using a Beagle Bone Black as standalone CW Skimmer...
If you are in need of support for development (especially targeting this Telnet/ReverseBeaconNetwork topic), just tell me! :)
Keep it up!
Hi Chris
DeleteI am working with Dave W1HKJ to integrate this functionality to an official FLDIGI version. Dave has posted the latest alpha build in http://www.w1hkj.com/ag1le/ (see fldigi-3.21.83cw-a5_setup.exe for Windows version, sources are in tarball).
> Are there plans for a connection to the reverse beacon network, so fldigi with your extension becomes a real alternative to CWSkimmer?
I didn't have any plans for this but this sounds very interesting. Would you be able to help integrating this functionality to FLDIGI? I don't have much knowledge about how ReverseBeaconNetwork works and what kind of APIs would be available. Are there any technical specification what kind of data is required for RBN?
73
Mauri AG1LE
Hi Mauri,
Deletean integration of your development into the official FLDIGI version sounds very promising.
With or without RBN support, it will be very useful to the Ops.
To enable CW Skimming functionality we would have to parse the received and decoded CW signals for the typical CW grammar (CQ CQ TEST CALLSIGN and so on) to detect valid transmissions automatically. The gained information (Callsign of the Sender, receive SNR, time, frequency) is either provided via a TELNET interface (classical operation of the original CW Skimmer software), and/or uploaded to the database of the reverse beacon network. Done that, the received stations show up on http://www.reversebeacon.net.
I don't know the exact data format required to send entries to the RBN, but I am pretty sure that the guys of the RBN would fully support such an development, as it is the key to operation on (lightweight) (standalone) Linux based operation systems.
mny 73,
Chris DL1COM
Hi Mauri,
Deletetoday I compiled fldigi 3.21.83 with your developments. Works like a charm!
While looking through the source code, I stumbled upon the fact that fldigi is already supporting autospotting in PSK, reporting to pskreporter.info. So, the algorithms to detect CQ calls and callsigns are already existing! :D
As you are in contact with W1HKJ, perhaps you may propose him to add the possibility to report spots to the RBN in CW, PSK and RTTY mode. Basically, it is exactly the same thing which he already does when reporting to pskreport.info - only the message format to send spots to the RBN has to be asked from the RBN team :)
I can't believe I recognized the autospotting functionality before ;)
73,
DL1COM
Hi Mauri,
ReplyDeleteI don't understand if your devs about bayesian method morse decoder are included in last versions of fldigi ?
Could you tell me ?
Jean-Pierre - F1RUM
No, Bayesian method was not included in the last versions of FLDigi. I didn't get the performance to a satisfactory level.
DeleteThanks Mauri,
DeleteI understand, however, actual morse decoder in fldigi is not efficient at all. Android Morse Expert (and certainly CW Skimmer that I can't install on my Linux machine) is really awesome compare to fldigi.
Is there a mean to help you to try to improve fldigi cw decoder ? fldigi is certainly the best ham software I use, and thanks again for the good work with Dave W1HKJ.
Hi Jean-Pierre, the morse decoder in FLDIGI requires manual tuning of parameters to work better. I did work on a few features like matched filter and SOM decoding to improve the accuracy that were accepted to FLDIGI CW decoder.
DeleteHowever, the Bayesian decoder had quite a lot of new code and I got it to alpha level with a few testers helping to find issues. We never got the code to be accurate enough for real world signals, so eventually I abandoned this approach and started looking alternatives, such as experimenting with Machine Learning algorithms to train a model with a lot of noisy CW.
You can check this http://ag1le.blogspot.com/2020/04/new-real-time-deep-learning-morse.html where I did document one such experiment. If I would have more time available, this might be the way forward but integrating to FLDIGI would be quite a lot of work.
Dave W1HKL has done a monumental work pulling FLDIGI together, and there are some really talented people who have contributed their skills to create FLDIGI open source software. My hat off the the FLDIGI team who have spent so much of their time to give this package for free to the ham community.
What comes to CW Skimmer and Android Morse Expert, Alex VE3NEA has the "secret sauce" that he has spent many years to make perfect. They are the golden standard at the moment what comes to accuracy, for sure. Even if Alex would provide his algorithms as an open source it would be still a lot of work to integrate those to FLDIGI and make it seamless CW decoder experience for the users.
I hope this gives you the perspective and some ideas how to help. If you really want to build a better CW decoder for FLDIGI you need to invest a lot of time and energy to make it happen. I spent probably hundreds of hours reading research papers, learning digital signal processing, bayesian probability and related math, read multiple Ph.D. thesis written by very smart people, and re-learned how to write C and C++ code again.
For me the learning is the journey, and this blog is an attempt to document some of my experiments along the way.
73
Mauri AG1LE
Mauri, you're right, making fldigi is an awesome effort, and I'm aware of needed knowledge just for this cw decoder. Your new approach, using machine learning, seems promising, and it's very cool you don't call it "artificial intelligence" as this kind of methods is so often called ;-)
DeleteUnfortunately, I realize that I'm very far from being able to provide any help for this dev.
I will comment on your post about amchine learning...
tnx