tag:blogger.com,1999:blog-33267732143291832842024-03-16T03:08:03.767-04:00Ham Radio Blog by AG1LEHam radio projects and experiments by AG1LEag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.comBlogger56125tag:blogger.com,1999:blog-3326773214329183284.post-35646159163061384682021-07-04T19:56:00.010-04:002021-07-06T19:44:14.193-04:00Cloudberry Live - listen your rig from everywhere with Web UI using Raspberry Pi<h2 style="text-align: left;">Overview </h2><div>I wanted to have a fully open source solution to listen my radio(s) from my mobile phone and laptop over the web using a low cost Raspberry Pi as the rig control server. While there are many different remote station solutions out there I could not find one that would just work with a normal web browser (Chrome, Safari, etc) and without doing complicated network configurations exposing your internal WiFi network via a router. Also, I wanted to have the solution that is really easy to install to Raspberry Pi and update new versions as new features get added to the software. </div><div><br /></div><div>I revisited the <a href="http://ag1le.blogspot.com/2016/02/kx3-remote-control-and-audio-streaming.html" target="_blank">KX3 remote control project</a> I did in Feb 2016 and started a new <b>Cloudberry Live </b>project. Cloudberry Live has several new improvements, such as no need to install Mumble client on your devices - you can just listen your radio(s) using a regular web browser. I did also upgrade my <a href="https://ag1le.blogspot.com/2017/01/amazon-echo-alexa-skills-for-ham-radio.html" target="_blank">Amazon Alexa skill</a> to leverage the ability to stream audio to Amazon Echo devices and control the frequency using voice commands. </div><div><br /></div><div>Here is a short demo video how Cloudberry.live works: </div><div><br /></div><div><br /></div><div class="separator" style="clear: both; text-align: center;"><iframe allowfullscreen="" class="BLOG_video_class" height="413" src="https://www.youtube.com/embed/cj3abdFF-xA" width="496" youtube-src-id="cj3abdFF-xA"></iframe></div><br /><div><br /></div><div><br /></div><div><br /></div><h2 style="text-align: left;">Features </h2><div style="text-align: left;"><ul style="text-align: left;"><li>Listen your radio using web streaming from anywhere.</li><li>Web UI that works with mobile, tablet and laptop browsers (Chrome and Safari tested) </li><li>View top 10 DX cluster spots, switch the radio to the frequency with one click. </li></ul></div><div style="text-align: left;">The software is currently at alpha stage - all the parts are working as shown in the demo above but need refactoring and general clean-up. The cloudberry.live proxy service is currently using a 3rd party open source proxy provider <a href="https://github.com/azimjohn/jprq" target="_blank">jprq</a>. My plan is to host a reverse proxy myself in order to simplify the installation process. </div><div style="text-align: left;"><br /></div><div style="text-align: left;">The software is written using Python Flask framework and bash scripts. The deployment to Raspberry Pi is done using Ansible playbook that configures the environment correctly. I am using NGINX webserver to serve the web application. </div><div style="text-align: left;"><br /></div><div style="text-align: left;">The audio streaming portion is using <a href="https://en.wikipedia.org/wiki/HTTP_Live_Streaming" target="_blank">HTTP Live Streaming (HLS)</a> protocol and ffmpeg is used to stream audio from ALSA port and encode it using AAC format. There is a python http.server on port 8000 serving HLS traffic. I have tested Safari and Chrome browsers to be able to stream HLS audio. Chrome requires Play HLS M3u8 extension to be installed.</div><div style="text-align: left;"><br /></div><div style="text-align: left;">The home screen is shown below. This gives you the top 10 spots and a link to open audio streaming window. By clicking the frequency link on the freq column the server sends hamlib commands to the radio to set the frequency and mode. Only USB and LSB modes are supported in the current software version.</div><div style="text-align: left;"><br /></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJIasQnaY01Z8QkH0g1P6sgiiNgxs-az_eEU9v0SuaPSp_0CpQ99fx6-tStU_JSqbRHwnGx7JJcjkwV-zuojEZTcxxCU0QpFfWBmBXekOOwpg61Cdli3a9f4K1Sbg7WDQVYTwSTiqMhWc/s1189/Screen+Shot+2021-07-04+at+7.09.29+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="695" data-original-width="1189" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJIasQnaY01Z8QkH0g1P6sgiiNgxs-az_eEU9v0SuaPSp_0CpQ99fx6-tStU_JSqbRHwnGx7JJcjkwV-zuojEZTcxxCU0QpFfWBmBXekOOwpg61Cdli3a9f4K1Sbg7WDQVYTwSTiqMhWc/w640-h374/Screen+Shot+2021-07-04+at+7.09.29+PM.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div>The Tune screen is shown below. This is still works-in-progress and needs some polishing. The Select Frequency allows to enter the frequency using numbers. The VFO range bar allows to change the radio frequency by dragging the green selection bar. The band selection buttons don't do anything at the moment. </div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA9avFXfgmGy1-sIfULVg2W0ro3xOiCfwppRw9RmisNku4q43FSTThzTYNpVnymyQMTLGT0GpdBk9mS0H6xlk9Y-J89FVrOI8GvcG7jMqNSH-FaMTW49XIFMOMw0KkjHQ7wzalAHs1mIU/s1189/Screen+Shot+2021-07-04+at+7.11.35+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="695" data-original-width="1189" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhA9avFXfgmGy1-sIfULVg2W0ro3xOiCfwppRw9RmisNku4q43FSTThzTYNpVnymyQMTLGT0GpdBk9mS0H6xlk9Y-J89FVrOI8GvcG7jMqNSH-FaMTW49XIFMOMw0KkjHQ7wzalAHs1mIU/w640-h374/Screen+Shot+2021-07-04+at+7.11.35+PM.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div><br /></div>The Configure Rig screen allows you to select your rig from the list of hamblib supported radios. I am using ICOM IC-7300 that is currently the default setting. <div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIwYdM7pecpjrs81xM_Q8sLULu6hyphenhyphen5V_ii2TJMu5EWdDQNmctt4qGAgAfZTBtj2A0lFjzWJZYi32eW1ee7H8f5QmLD24ldo92F1FjtiNVCNTlspMl-zAgS48D7JQRgtKUjUwe61iJ-GsE/s1189/Screen+Shot+2021-07-04+at+7.15.38+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="695" data-original-width="1189" height="374" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIwYdM7pecpjrs81xM_Q8sLULu6hyphenhyphen5V_ii2TJMu5EWdDQNmctt4qGAgAfZTBtj2A0lFjzWJZYi32eW1ee7H8f5QmLD24ldo92F1FjtiNVCNTlspMl-zAgS48D7JQRgtKUjUwe61iJ-GsE/w640-h374/Screen+Shot+2021-07-04+at+7.15.38+PM.png" width="640" /></a></div><br /><div style="text-align: left;">The Search button on the menu bar allows to check call sign from hamdb.org database. A pop-up window will show the station details:</div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpe1G910VTcIdYPc_0HzGlj9FCD4zIhKpyIiIJpVByO_Gm09wsmXmS19blzV3H5zWE3QZ6y_Yp7IzUF7BvjEpiptWrXZIEXD_6vW8YJ_MbGHaQUDSlWQqd5KeQ0DIpVJgJCRvtvemk6Zc/s582/Screen+Shot+2021-07-04+at+7.18.39+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="77" data-original-width="582" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpe1G910VTcIdYPc_0HzGlj9FCD4zIhKpyIiIJpVByO_Gm09wsmXmS19blzV3H5zWE3QZ6y_Yp7IzUF7BvjEpiptWrXZIEXD_6vW8YJ_MbGHaQUDSlWQqd5KeQ0DIpVJgJCRvtvemk6Zc/s320/Screen+Shot+2021-07-04+at+7.18.39+PM.png" width="320" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8uottihOzcEkXxNxSKbOn_AnkWhUPVT1dvh7yZMrIr2VSe-cCwuQLorIJ8fMFIroY623FoZG91mDaqc8B6Pyy7jVBmla3gmVy-pcx91fkK8Puw9tBjmTBuzNovr9X0uaHkwEvIwrztAg/s1192/Screen+Shot+2021-07-04+at+7.20.38+PM.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="539" data-original-width="1192" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8uottihOzcEkXxNxSKbOn_AnkWhUPVT1dvh7yZMrIr2VSe-cCwuQLorIJ8fMFIroY623FoZG91mDaqc8B6Pyy7jVBmla3gmVy-pcx91fkK8Puw9tBjmTBuzNovr9X0uaHkwEvIwrztAg/s320/Screen+Shot+2021-07-04+at+7.20.38+PM.png" width="320" /></a></div><br /><div style="text-align: left;"><br /></div><div style="text-align: left;"><br /></div><h2 style="text-align: left;">Amazon Alexa Skill</h2><div style="text-align: left;">I created a new Alexa Skill Cloudberry Live (not published yet) that uses the web API interface for selecting the frequency based on DX cluster spots and the HLS streaming to listen your radio. While the skill is currently using only my station, my goal would be to implement some sort of registration process so that Alexa users would have more choice to listen ham radio traffic from DX stations around the world using Cloudberry.live software. </div><div style="text-align: left;"><br /></div><div style="text-align: left;">This would give an opportunity also for people with disabilities to enjoy listening HF bands using voice controlled, low cost ($20 - $35) smart speakers. By keeping your radio (Raspberry Pi server) online you could help to grow the ham community. </div><div style="text-align: left;"><br /></div><h2>Installation</h2>I have posted the software to Github in a private repo. The software will have the following key features<br /><ul style="text-align: left;"><li>One step software installation to Raspberry Pi using <a href="https://www.ansible.com/" target="_blank">Ansible</a> playbooks.</li></ul><ul style="text-align: left;"><li>Configure your radio using Hamlib </li></ul><ul style="text-align: left;"><li>Get your personalized Cloudberry.live weblink</li></ul>I have been developing cloudberry.live on my Macbook Pro and pushing new versions to RaspBerry Pi server downstairs where my IC-7300 is located. Typical Ansible playbook update takes about 32 seconds (this includes restarting the services). I can see the access and error logs on the server using SSH consoles - this makes debugging quite easy. <p></p><h2 style="text-align: left;">Questions? </h2><div>I am looking for collaborators to work with me on this project. If you are interested in open source web development using Python Flask framework let me know by posting a comment below. </div><div><br /></div><div><br /></div><div>73 de </div><div>Mauri AG1LE</div></div><div><br /></div><br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com4tag:blogger.com,1999:blog-3326773214329183284.post-38905571144503215422021-03-31T17:07:00.005-04:002023-06-03T15:34:45.460-04:00New exciting Digital Mode CT8 for ham radio communications <p style="text-align: right;">April 1, 2021 </p><h3 style="text-align: left;">Overview </h3><p>CT8 is a new exciting digital mode designed for interactive ham radio communication where signals may be weak and fading, and openings may be short. </p><p>A beta release of CT8 offers sensitivity down to –48 dB on the AWGN channel, and DX contacts with 4 times longer distance than FT8. An auto-sequencing feature offers the option to respond automatically to the first decoded reply to your CQ. </p><p>The best part of this new mode is that it is easy to learn how to decode in your head, thus no decoder software is needed. Alpha users of CT8 mode report that learning to decode CT8 is ten times easier than Morse code. For those who rather use a computer, an open source Tensorflow based Machine Learning decoder software is included in this beta release. </p><p>CT8 is based on novel avian vocalization encoding scheme. The character combinations were designed to be very easily recognizable to leverage existing QSO practices in the communication modes like CW. </p><p>Below is an example audio clip on how to establish a CT8 contact - the message format should be familiar to anybody who have listened Morse code in ham radio bands before. </p><p>
</p><figure>
<figcaption>Listen to the "<a href="https://zappa-deepmorse.s3.amazonaws.com/cqag1le.mp3" target="_blank">CQ CQ DE AG1LE K</a>" - the audio has rich syllabic tonal and harmonic features that are very easy to recognize even under noisy band conditions. </figcaption></figure><div><br /></div>Fig 1. below shows the corresponding spectrogram. Notice the harmonic spectral features that ensure accurate symbol decoding and provide high sensitivity and tolerance against rapid fading, flutter and QRM.<br /><p></p><table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzJL7q6bvD0k-p2IUyeMohs47qqoHKUfshDsrkUgKumNZ-5RCRFQgNUMSy-f-ulqxXzKaSsT-QSwrOm82tL_kGGt-hw-n_C6i4u8TkAhi39IWGCKHinNySnJaCdHukIknQcoysOsXdqUg/s1586/Screen+Shot+2021-01-17+at+11.50.44+AM.png" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="486" data-original-width="1586" height="195" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzJL7q6bvD0k-p2IUyeMohs47qqoHKUfshDsrkUgKumNZ-5RCRFQgNUMSy-f-ulqxXzKaSsT-QSwrOm82tL_kGGt-hw-n_C6i4u8TkAhi39IWGCKHinNySnJaCdHukIknQcoysOsXdqUg/w640-h195/Screen+Shot+2021-01-17+at+11.50.44+AM.png" title="Fig 1. CT8 spectrogram" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Fig 1. CT8 spectrogram - CQ CQ CQ DE AG1LE K</td></tr></tbody></table><br /><p><br /></p><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;"><br /></h3><p style="text-align: left;"><br /><br /></p><p style="text-align: left;">The audio clip sample may sound a bit like a chicken. This is actually a key feature of avian vocalization encoding. </p><h3 style="text-align: left;">Scientific Background </h3><p style="text-align: left;">The idea behind CT8 mode is not new. There is a lot of research done on avian vocalizations over the past hundred years. From late 1990s digital signal processing software has become widely available and vocal signals can be analyzed using sonograms and spectrograms with a personal computer.</p><p style="text-align: left;">In research article [1] Dr. Nicholas Collias described sound spectrograms of 21 of the 26 vocal signals in the extensive vocal repertoire of the African Village Weaver (Ploceus cucullatus). A spectrographic key to vocal signals helps make these signals comparable for different investigators. Short-distance contact calls are given in favorable situations and are generally characterized by low amplitude and great brevity of notes. Alarm cries are longer, louder, and often strident calls with much energy at high frequencies, whereas threat notes, also relatively long and harsh, emphasize lower frequencies. </p><p style="text-align: left;">In a very interesting research article [2] by Kevin G. McCracken and Frederick H. Sheldon conclude that the characters most subject to ecological convergence, and thus of least phylogenetic value, are first peak-energy frequency and frequency range, because sound penetration through vegetation depends largely on frequency. The most phylogenetically informative characters are number of syllables, syllable structure, and fundamental frequency, because these are more reflective of behavior and syringeal structure. In the figure below give details about Heron phylogeny, corresponding spectrograms, vocal characters, and habitat distributions. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://www.pnas.org/cms/10.1073/pnas.94.8.3833/asset/76e00413-4ff6-4e39-bc06-538147d0c20a/assets/graphic/pq0870336001.jpeg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="578" data-original-width="800" height="578" src="https://www.pnas.org/cms/10.1073/pnas.94.8.3833/asset/76e00413-4ff6-4e39-bc06-538147d0c20a/assets/graphic/pq0870336001.jpeg" width="800" /></a></div><br /><div class="separator" style="clear: both; text-align: center;"><br /></div><br /><p style="text-align: left;">Habitat distributions suggest that avian species that inhabit open areas such as savannas, grasslands, and open marshes have higher peak-energy (J) frequencies (kHz) and broader frequency ranges (kHz) than do taxa inhabiting closed habitats such as forests. Number of syllables is the number most frequently produced. </p><p style="text-align: left;">Ibises, tiger-herons, and boat-billed herons emit a rapid series of similar syllables; other heron vocalizations generally consist of singlets, doublets, or triplets. Syllabic structure may be tonal (i.e., pure whistled notes) or harmonic (i.e., possessing overtones; integral multiples of the base frequency). Fundamental frequency (kHz) is the base frequency of a syllable and is a function of syringeal morphology. </p><p style="text-align: left;">These vocalization features can be used for training modern machine learning algorithms. In fact, in a series of studies published [3] between 2014 and 2016, Georgia Tech research engineer Wayne Daley and his colleagues exposed groups of six to 12 broiler chickens to moderately stressful situations—such as high temperatures, increased ammonia levels in the air and mild viral infections—and recorded their vocalizations with standard USB microphones. They then fed the audio into a machine learning program, training it to recognize the difference between the sounds of contented and distressed birds. According the Scientific American article [4] Carolynn “K-lynn” Smith, a biologist at Macquarie University in Australia and a leading expert on chicken vocalizations, says that although the studies published so far are small and preliminary, they are “a neat proof of concept” and “a really fascinating approach.”</p><h3 style="text-align: left;">What does CT8 stand for? </h3><p>Building on this solid scientific foundation it is easy to imagine very effective communication protocols that are based on millions of years of evolution of various avian species. After all, birds are social animals and have very expressive and effective communication protocols, whether to warn others about approaching predator or to invite flock members to join feasting on a corn field. </p><p>Humans have domesticated several avian species and have been living with species like <a href="https://en.wikipedia.org/wiki/Chicken" target="_blank">chicken (Gallus gallus domesticus) for over 8000 years</a>. Therefore CT8 mode sounds inherently natural to humans and it is much easier to learn to decode than Morse code based on extensive alpha testing performed by the development team. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://upload.wikimedia.org/wikipedia/commons/thumb/f/fd/Icelandic_Settler_Chicken.jpg/1280px-Icelandic_Settler_Chicken.jpg" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="769" data-original-width="800" height="384" src="https://upload.wikimedia.org/wikipedia/commons/thumb/f/fd/Icelandic_Settler_Chicken.jpg/1280px-Icelandic_Settler_Chicken.jpg" width="400" /></a></div><p><br /></p><p>CT8 stands for "Chicken Talk" version 8 -- over a year of development effort and seven previous encoding versions tested over difficult band conditions, and with hundreds of Machine Learning models trained, the software development team has finally been able to release CT8 digital mode. </p><h3 style="text-align: left;">Encoding Scheme </h3><div>From ham radio perspective the frequency range of these avian vocalizations is below 4 kHz in most cases. This makes it possible to use existing SSB or FM transceivers without any modifications, other than perhaps adjustment of the filter bandwidth available in modern rigs. The audio sampling rate used in this project was 8 kHz, so the original audio source files were re-sampled using a Linux command line tool: <br /><br /></div><div><p class="p1" style="font-family: Menlo; font-size: 11px; font-stretch: normal; font-variant-east-asian: normal; font-variant-numeric: normal; line-height: normal; margin: 0px;"><span class="s1" style="font-variant-ligatures: no-common-ligatures;">sox<span class="Apple-converted-space"> </span>-b16 -c 1 input.wav output.wav<span class="Apple-converted-space"> </span>rate 8000</span></p></div><p>The encoding scheme for the CT8 mode was done by collecting various free audio sources of chicken sounds and carefully assembling vowels, plosives, fricatives and nasals using <a href="https://home.cc.umanitoba.ca/~krussll/phonetics/acoustic/spectrogram-sounds.html" target="_blank">this resource</a> as the model. Free open source cross-platform audio software <a href="https://www.audacityteam.org/download/" target="_blank">Audacity</a> was used to extract vocalizations using the spectrogram view and also creating labeled audio files.</p><p>Figure 3. below shows a sample audio file with assigned character labels. </p><table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left;"><tbody><tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc_gOwv60VEq0jWIojeQ4FNKE7I9g9RZzShxepkWCqWOOpZwu0QsaFTvhf18L-ocmH9ESosC7fQ4hCEk7Z_YhCIYur8PrUzXeMAlIgmNeSs3nnPEwiD28n8dxc1h4rMxn9BQnXBXwCfME/s1732/Screen+Shot+2021-01-17+at+12.29.39+PM.png" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="994" data-original-width="1732" height="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgc_gOwv60VEq0jWIojeQ4FNKE7I9g9RZzShxepkWCqWOOpZwu0QsaFTvhf18L-ocmH9ESosC7fQ4hCEk7Z_YhCIYur8PrUzXeMAlIgmNeSs3nnPEwiD28n8dxc1h4rMxn9BQnXBXwCfME/w640-h368/Screen+Shot+2021-01-17+at+12.29.39+PM.png" width="640" /></a></td></tr><tr><td class="tr-caption" style="text-align: center;">Fig 3. Labeled vocalizations using Audacity software</td></tr></tbody></table><br /><p><br /></p><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3><br /></h3><h3>CT8 Software</h3><p>The encoder software is written in C++ and Python and runs on Windows, OSX, and Linux. The sample decoder is made available from Github as open source software, if there is enough interest on this novel communication mode from the ham radio community. </p><p>For the CT8 decoder a <a href="https://ag1le.blogspot.com/2019/02/training-computer-to-listen-and-decode.html" target="_blank">Machine Learning based decoder</a> software was built on top of open source Tensorflow framework. The decoder was trained on short 4 second audio clips and in the experiments character error rate 0.1% and word accuracy of 99.5% was achieved. With more real-world training material the ML model is expected to achieve even better decoding accuracy. </p><h3>Future Enhancements</h3><p>CT8 opens a new era for ham radio communication protocol development using <a href="https://en.wikipedia.org/wiki/Biomimetics" target="_blank">biomimetics</a> principles. Adding new phonemes using the principles of ecological signals as described in article [2] can open up things like "DX mode" for long distance communication. For example the vocalizations of Cetaceans (whales) could be also used to build a new phoneme map for DX contacts - some of the lowest frequency whale sounds can travel <a href="https://journeynorth.org/tm/hwhale/SingingHumpback.html" target="_blank">through the ocean as far as 10,000 miles </a>without losing their energy. </p><p><br /></p><p>73 de AG1LE </p><p><br /></p><p>PS. If you made it down here, I hope that you enjoyed this figment of my imagination and I wish you a very happy April 1st.</p><p><br /></p><h3 style="text-align: left;">References</h3><p>[1] Nicholas E. Collias, <a href="https://www.researchgate.net/publication/232677880_Vocal_Signals_of_the_Village_Weaver_A_Spectrographic_Key_and_the_Communication_Code" target="_blank">Vocal Signals of the Village Weaver: A Spectrographic Key and the Communication Code</a></p><p>[2] Kevin G. McCracken and Frederick H. Sheldon, <a href="https://www.pnas.org/content/94/8/3833" target="_blank">Avian vocalizations and phylogenetic signal</a></p><p>[3] Wayne Daley, et al <a href="https://www.researchgate.net/publication/316903091_Identifying_rale_sounds_in_chickens_using_audio_signals_for_early_disease_detection_in_poultry" target="_blank">Identifying rale sounds in chickens using audio signals for early disease detection in poultry</a></p><p>[4] Scientific American, Ferris Jabr, <a href="https://www.scientificamerican.com/article/fowl-language-ai-decodes-the-nuances-of-chicken-ldquo-speech-rdquo/" target="_blank">Fowl Language: AI Decodes the Nuances of Chicken “Speech”</a></p><div><br /></div>ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com3tag:blogger.com,1999:blog-3326773214329183284.post-66771601327662610622020-04-12T09:20:00.002-04:002020-04-12T14:09:44.335-04:00New real-time deep learning Morse decoder<br />
<h3>
Introduction</h3>
I have done some experiments with deep learning models previously. <a href="http://ag1le.blogspot.com/2019/02/training-computer-to-listen-and-decode.html" target="_blank">This previous blog post </a> covers the new approach of building Morse decoder by training a CNN-LSTM-CTC model using audio that is converted to small image frames.<br />
<br />
In this latest experiment I trained a new Tensorflow based CNN-LSTM-CTC model using 27.8 hours of Morse audio training set (25,000 WAV files - each clip 4 seconds) and achieved character error rate of 1.5% and word accuracy of 97.2% after 2:29:19 training time. The training data corpus was created from <a href="http://www.arrl.org/code-practice-files" target="_blank">ARRL Morse code practice files</a> (text files).<br />
<br />
<h3>
New real-time deep learning Morse decoder</h3>
I wanted to see if this new model is capable of decoding audio in real-time so I wrote a simple Python script to listen microphone, create a spectrogram, detect the CW frequency automatically, and feed 128 x 32 images to the model to perform the decoding inference.<br />
<br />
With some tuning of the various components and parameters I was able to put together a working prototype using standard Python libraries and the Tensorflow Morse decoder that is available as <a href="https://github.com/ag1le/LSTM_morse/blob/master/MorseDecoder.py" target="_blank">open source in Github</a>.<br />
<br />
I recorded this sample YouTube video below in order to document this experiment.<br />
<br />
Starting from the top left I have <a href="http://www.w1hkj.com/files/fldigi/" target="_blank">FLDIGI</a> window open decoding CW at 30 WPM speed. On the top middle I have console window open printing the frame number, CW tone frequency followed by "infer_image:" and decoded text as well as the probability that the model assigns to this result.<br />
<br />
On the top right I have the <a href="https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.specgram.html" target="_blank">Spectrogram window</a> that plots 4 seconds of the audio on a frequency scale. The morse code is quite readable on this graph.<br />
<br />
On the bottom left I have <a href="https://www.audacityteam.org/download/" target="_blank">Audacity</a> playing a sample <a href="http://www.arrl.org/files/file/Morse/Archive/30%20WPM/170103_30WPM.mp3" target="_blank">30 WPM practice file</a> <a href="https://www.blogger.com/"><span id="goog_869187283"></span>from ARRL<span id="goog_869187284"></span></a>. Finally, on the bottom right I have the 128x32 image frame that I am feeding to the model.<br />
<br />
<br />
<iframe allow="accelerometer; autoplay; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/rUqpT7Q-mSs" width="560"></iframe>
<br />
<br />
<br />
<h3>
Analysis</h3>
The full text at 30 WPM is here - I have highlighted the text section that is playing in the above video clip.<br />
<br />
<pre style="overflow-wrap: break-word; white-space: pre-wrap;">� NOW 30 WPM � TEXT IS FROM JULY 2015 QST PAGE 99 �
AGREEMENT WITH SOUTHCOM GRANTED ATLAS ACCESS TO THE SC 130S TECHNOLOGY.
THE ATLAS 180 ADAPTED THE MAN PACK RADIOS DESIGN FOR AMATEUR USE. AN
ANALOG VFO FOR THE 160, 80, 40, AND 20 METER BANDS REPLACED THE SC 130S
STEP TUNED 2 12 MHZ SYNTHESIZER. OUTPUT POWER INCREASED FROM 20 W TO 100
W. AMONG THE 180S CHARMS <span style="background-color: yellow;">WAS ITS SIZE. IT MEASURED 9R5 X 9R5 X 3 INCHES.</span></pre>
<pre style="overflow-wrap: break-word; white-space: pre-wrap;"><span style="background-color: yellow;">THATS NOTHING SPECIAL TODAY, BUT IT WAS A TINY RIG IN 1974. THE FULLY
SOLID STATE TRANSCEIVER FEATURED NO TUNE OPERATION. THE VFOS 350 KHZ RANGE
REQUIRED TWO BAND</span> SWITCH SEGMENTS TO COVER 75/80 METERS, BUT WAS AMPLE FOR
THE OTHER BANDS. IN ORDER TO IMPROVE IMMUNITY TO OVERLOAD AND CROSS
</pre>
<pre style="overflow-wrap: break-word; white-space: pre-wrap;">MODULATION, THE 180S RECEIVER HAD NO RF AMPLIFIER STAGE THE ANTENNA INPUT
CIRCUIT FED THE RADIOS MIXER DIRECTLY. A PAIR OF SUCCESSORS EARLY IN 1975,
ATLAS INTRODUCED THE 180S SUCCESSOR IN REALITY, A PAIR OF THEM. THE NEW
210 COVERED 80 10 METERS, WHILE THE OTHERWISE IDENTICAL 215 COVERED 160 15
METERS HEREAFTER, WHEN THE 210 SERIES IS MENTIONED, THE 215 IS ALSO
IMPLIED. BECAUSE THE 210 USED THE SAME VFO AND BAND SWITCH AS THE 180,
SQUEEZING IN FIVE BANDS SACRIFICED PART OF 80 METERS. THAT BAND STARTED AT
� END OF 30 WPM TEXT � QST DE W1AW �</pre>
<br />
As can be seen from the YouTube video FLDIGI is able to copy this CW quite well. The new deep learning Morse decoder is also able to decode the audio with probabilities ranging from 4% to over 90% during this period.<br />
<br />
It has visible problems when the current image frame cuts the Morse character into parts. The scrolling 128x32 image that is produced from the spectrogram graph does not have any smarts - it is just copied at every update cycle and fed into the infer_image() function. This means that a single Morse character is moving out of the frame but some part of the character can be still visible, causing incorrect decodes.<br />
<br />
The decoder has also problems with some numbers even when fully visible in the 128x32 image frame. The ARRL training material that I used to build the corpus for training has about 8.6% words that are numbers (such as bands, frequencies and years). I believe that the current model doesn't have enough examples to decode all the numbers correctly.<br />
<br />
The final problem is the lack of spaces between the words. The current model doesn't know about the "Space" character so it is just decoding what it has been trained on.<br />
<br />
<br />
<h3>
Software</h3>
The python script running the model is quite simple and listed below. I adapted the main Spectogram loop from <a href="https://github.com/ayared/Live-Specgram" target="_blank">this Github repo</a>. I used the following constants in mic_read.py.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">RATE = 8000</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">FORMAT = pyaudio.paInt16 #conversion format for PyAudio stream</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">CHANNELS = 1 #microphone audio channels</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">CHUNK_SIZE = 8192 #number of samples to take per read</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SAMPLE_LENGTH = int(CHUNK_SIZE*1000/RATE) #length of each sample in ms</span><br />
<br />
<br />
specgram.py<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Created by Mauri Niininen (AG1LE)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Real time Morse decoder using CNN-LSTM-CTC Tensorflow model</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">adapted from https://github.com/ayared/Live-Specgram</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">############### Import Libraries ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">from matplotlib.mlab import specgram</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">import matplotlib.pyplot as plt</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">import matplotlib.animation as animation</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">import numpy as np</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">import cv2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">############### Import Modules ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">import mic_read</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">from morse.MorseDecoder import Config, Model, Batch, DecoderType</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">############### Constants ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">SAMPLES_PER_FRAME = 4 #Number of mic reads concatenated within a single window</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nfft = 256 # NFFT value for spectrogram</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">overlap = nfft-56 # overlap value for spectrogram</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">rate = mic_read.RATE #sampling rate</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">############### Call Morse decoder ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">def infer_image(model, img):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> if img.shape == (128, 32):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> batch = Batch(None, [img])</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> (recognized, probability) = model.inferBatch(batch, True)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> return img, recognized, probability</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> else:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> print(f"ERROR: img shape:{img.shape}")</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"># Load the Tensorlow model </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">config = Config('model.yaml')</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">model = Model(open("morseCharList.txt").read(), config, decoderType = DecoderType.BestPath, mustRestore=True)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">stream,pa = mic_read.open_mic()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">############### Functions ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">get_sample:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">gets the audio data from the microphone</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">inputs: audio stream and PyAudio object</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">outputs: int16 array</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">def get_sample(stream,pa):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> data = mic_read.get_data(stream,pa)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> return data</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">get_specgram:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">takes the FFT to create a spectrogram of the given audio signal</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">input: audio signal, sampling rate</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">output: 2D Spectrogram Array, Frequency Array, Bin Array</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">see matplotlib.mlab.specgram documentation for help</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">def get_specgram(signal,rate):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> arr2D,freqs,bins = specgram(signal,window=np.blackman(nfft), </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Fs=rate, NFFT=nfft, noverlap=overlap,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pad_to=32*nfft )</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> return arr2D,freqs,bins</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">update_fig:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">updates the image, just adds on samples at the start until the maximum size is</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">reached, at which point it 'scrolls' horizontally by determining how much of the</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">data needs to stay, shifting it left, and appending the new data. </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">inputs: iteration number</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">outputs: updated image</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">"""</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">def update_fig(n):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> data = get_sample(stream,pa)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> arr2D,freqs,bins = get_specgram(data,rate)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im_data = im.get_array()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> if n < SAMPLES_PER_FRAME:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im_data = np.hstack((im_data,arr2D))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im.set_array(im_data)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> else:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> keep_block = arr2D.shape[1]*(SAMPLES_PER_FRAME - 1)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im_data = np.delete(im_data,np.s_[:-keep_block],1)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im_data = np.hstack((im_data,arr2D))</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im.set_array(im_data)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> # Get the image data array shape (Freq bins, Time Steps)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> shape = im_data.shape</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> # Find the CW spectrum peak - look across all time steps</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> f = int(np.argmax(im_data[:])/shape[1])</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> # Create a 32x128 array centered to spectrum peak </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> if f > 16: </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> print(f"n:{n} f:{f}")</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> img = cv2.resize(im_data[f-16:f+16][0:128], (128,32)) </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> if img.shape == (32,128):</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> cv2.imwrite("dummy.png",img)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> img = cv2.transpose(img)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> img, recognized, probability = infer_image(model, img)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> if probability > 0.0000001:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> print(f"infer_image:{recognized} prob:{probability}")</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> return im,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">def main():</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> global im</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ############### Initialize Plot ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> fig = plt.figure()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> """</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Launch the stream and the original spectrogram</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> """</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> stream,pa = mic_read.open_mic()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> data = get_sample(stream,pa)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> arr2D,freqs,bins = get_specgram(data,rate)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> """</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> Setup the plot paramters</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> """</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> extent = (bins[0],bins[-1]*SAMPLES_PER_FRAME,freqs[-1],freqs[0])</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> im = plt.imshow(arr2D,aspect='auto',extent = extent,interpolation="none",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> cmap = 'Greys',norm = None) </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> plt.xlabel('Time (s)')</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> plt.ylabel('Frequency (Hz)')</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> plt.title('Real Time Spectogram')</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> plt.gca().invert_yaxis()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> #plt.colorbar() #enable if you want to display a color bar</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> ############### Animate ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> anim = animation.FuncAnimation(fig,update_fig,blit = True,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> interval=mic_read.CHUNK_SIZE/1000)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> try:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> plt.show()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> except:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> print("Plot Closed")</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> ############### Terminate ###############</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> stream.stop_stream()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> stream.close()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pa.terminate()</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> print("Program Terminated")</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">if __name__ == "__main__":</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> main()</span><br />
<br />
I did run this experiment on Macbook Pro (2.2 GHz Quad-Core Intel Core i7) and MacOS Catalina 10.15.3. The Python version used was Python 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 05:52:31) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.57)] on darwin<br />
<h3>
</h3>
<h3>
</h3>
<h3>
Conclusions</h3>
This experiment demonstrates that it is possible to build a working real time Morse decoder based on deep learning Tensorflow model using a slow interpreted language like Python. The approach taken here is quite simplistic and lacks some key functionality, such as alignment of decoded text to audio timeline.<br />
<br />
It also shows that there are still more work to do in order to build a fully functioning, open source and high performance Morse decoder. A better event driven software architecture would allow building a proper user interface with some controls, like audio filtering. Such an architecture would enable also building server side decoders running based on audio feeds from WebSDR receivers etc.<br />
<br />
Finally, the Tensorflow model in this experiment has a very small training set, only 27.8 hours of audio. If you compare to commercial ASR (automatic speech recognition) engines they have been trained using over 1000X more labeled audio training material. To get better performance from deep learning models you need to have a lot of high quality labeled training material that matches with the typical sound environment the model will be used on.<br />
<br />
<br />
73<br />
Mauri AG1LE<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com9tag:blogger.com,1999:blog-3326773214329183284.post-58369309255927054012019-07-13T17:37:00.002-04:002020-04-04T10:06:49.254-04:00DeepMorse - Web based tool for CW Puzzles and Training<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjabq9ZWsJF9bCPcZy-1pGSRoEtuOCmE1vygUrvM0cOxOSakKjsTro70SiKgQdw1mTuY97JCLWpB_mmanDkQRGgPZIldkT6IBZ0UV8oasqxS6M8MeL_vneQDemmDYkbtj1ZE2ptP8na5Qs/s1600/Screen+Shot+2019-07-13+at+5.20.49+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjabq9ZWsJF9bCPcZy-1pGSRoEtuOCmE1vygUrvM0cOxOSakKjsTro70SiKgQdw1mTuY97JCLWpB_mmanDkQRGgPZIldkT6IBZ0UV8oasqxS6M8MeL_vneQDemmDYkbtj1ZE2ptP8na5Qs/s1600/Screen+Shot+2019-07-13+at+5.20.49+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjabq9ZWsJF9bCPcZy-1pGSRoEtuOCmE1vygUrvM0cOxOSakKjsTro70SiKgQdw1mTuY97JCLWpB_mmanDkQRGgPZIldkT6IBZ0UV8oasqxS6M8MeL_vneQDemmDYkbtj1ZE2ptP8na5Qs/s1600/Screen+Shot+2019-07-13+at+5.20.49+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a></div>
<br />
<h3>
Introduction </h3>
<br />
I started working on a new project recently. The idea behind this "DeepMorse" project is to create a web site that contains curated Morse code audio clips. The website would allow subscribers to upload annotated CW audio clips (MP3, WAV, etc) and associated metadata.<br />
<br />
As a subscriber you would be able to provide the story behind the clip as well as some commentary or even photos. After uploading the site would show the graphical view of the audio clip much like the modern Software Defined Radios (SDRs) and users would be able to play back the audio and see the metadata.<br />
<br />
Since this site would contain "real world" recordings and some really difficult to copy audio clips, this would also provide ultimate test of your CW copying skills. The system would save a score on your copying accuracy before it gives you the "ground truth" of annotated audio. You could compete for the top scores with all the other CW aficionados.<br />
<br />
The site could also be used to share historical records of curated Morse code audio materials with the ham radio community. For CW newbies the site would have a treasure trove of different kinds of training materials when you get tired of listening ARRL morse practice MP3 files. For experienced CW operators you could share some of your best moments when working using your favorite operating mode, teaching newbies how to catch the "big fish".<br />
<br />
<h3>
User Interface</h3>
I wanted to experiment combining audio and graphical waveform view of the audio together, giving the user ability to listen, scroll back and re-listen as well as zoom into the waveform.<br />
<br />
Part of the user interface is also the free text form where user can enter the text they heard in the audio clip. By pressing "Check" button the system will calculate the accuracy compared to the "ground truth" text. System is using <a href="https://www.blogger.com/blogger.g?blogID=3326773214329183284#%20https://github.com/luozhouyang/python-string-similarity#normalized-levenshtein" target="_blank">normalized Levenshtein method</a> to calculate the accuracy in percentage (0...100%) where 100% is perfect copy.<br />
<br />
Figure 1. below shows the main listening view.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2dqXvqdnFD1MBIpC4-UiahLTEiyJMl14j_LugtN-OCE3IgqjO3-DkLwzrRd8edZyElkCQSMx7dN_X3PWk8SYrPJurV9IuXRFu2wog36XItXpU2ne1GJsK97yR7ZM3LnARK5teNQhPeEA/s1600/Screen+Shot+2019-07-13+at+5.21.08+PM.png" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-left: auto; margin-right: auto; text-align: center;"><img border="0" data-original-height="495" data-original-width="1185" height="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2dqXvqdnFD1MBIpC4-UiahLTEiyJMl14j_LugtN-OCE3IgqjO3-DkLwzrRd8edZyElkCQSMx7dN_X3PWk8SYrPJurV9IuXRFu2wog36XItXpU2ne1GJsK97yR7ZM3LnARK5teNQhPeEA/s640/Screen+Shot+2019-07-13+at+5.21.08+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. DeepMorse User Interface</td></tr>
</tbody></table>
<br />
<br />
<h3>
Architecture</h3>
<div>
I wrote this web application using Python Django web framework and it took only a few nights to get the basic structure together. The website is running in AWS using serverless Lambda functions and serverless Aurora RDS MySQL database. The audio files are stored into an S3 bucket.<br />
<br />
Using serverless database backend sounds like oxymoron, since there is a database server managed by AWS. It also brings some challenges such as slow "cold start" that will be visible for end users. When you click the "Puzzles" menu you normally will get this view (see Figure 2. below).<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinRxzDiYD0AA1iOfAXmEMDtoinIlHOvnwuoRXpILrjpW8d1PjwbDYh4GrmMCqrDHKmTU09mjWXLYffandIyFsrSNLpeb9fI2K0X3uePjag9eDF4T2-fP7qUPvBa8WpmM5SY_ID_mTifNA/s1600/Screen+Shot+2019-07-14+at+8.59.11+AM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="757" data-original-width="1600" height="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinRxzDiYD0AA1iOfAXmEMDtoinIlHOvnwuoRXpILrjpW8d1PjwbDYh4GrmMCqrDHKmTU09mjWXLYffandIyFsrSNLpeb9fI2K0X3uePjag9eDF4T2-fP7qUPvBa8WpmM5SY_ID_mTifNA/s640/Screen+Shot+2019-07-14+at+8.59.11+AM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. Puzzles View </td></tr>
</tbody></table>
<br /></div>
<div>
<br /></div>
<h3>
</h3>
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
However, if the serverless database server has timed out due to no activity, it will take more than 30 seconds to come up. By this time the front end webserver has also timed out and the user will see this below instead (see Figure 3.). A simple refresh of the browser will fix the situation and both the front end and the backend will be then available. </div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJP9ri8hlxKZBEHZrYO_G-HCpTNZYBI0HaxFqbqJ2RWX2Hy0vAlxQAhMT4BboE1gQ8DNHIr6SEpeYJq3ciNicK4TWLF1mFdMhvK0yYjBjCctiLV-GmFDql2MZ0Z2lU_nsDjNtuCptPKX4/s1600/Screen+Shot+2019-07-14+at+9.08.04+AM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="374" data-original-width="1066" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJP9ri8hlxKZBEHZrYO_G-HCpTNZYBI0HaxFqbqJ2RWX2Hy0vAlxQAhMT4BboE1gQ8DNHIr6SEpeYJq3ciNicK4TWLF1mFdMhvK0yYjBjCctiLV-GmFDql2MZ0Z2lU_nsDjNtuCptPKX4/s640/Screen+Shot+2019-07-14+at+9.08.04+AM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3. Serverless "Time Out" error message</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
So what is then the benefit of using AWS serverless technology? The benefit is that you get billed only for usage and if the application is not used 24x7 this means significant cost savings. For a hobby project like DeepMorse I am able to run the service very cost efficiently. </div>
<div>
<br /></div>
<div>
The other benefit of serverless technologies is automatic scaling - if the service becomes suddenly hugely popular the system is able to scale up rapidly. </div>
<div>
<br /></div>
<h3>
Next Steps</h3>
<div>
I am looking for some feedback from early users trying to figure out what features might be interesting for Morse code aficionados. </div>
<div>
<br /></div>
<div>
73 de Mauri </div>
<div>
AG1LE<br />
<br />
<br />
<h4>
SCREEN SHOTS</h4>
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifGhJvAa7sFThuEufmbNosXGhG0iaKkqbeq2ipmSjf7BLVVmRMsblfakRTHQSeE8NpeLRa9Gh3o4RhEC2kaVOMW-d0gxGigXwilmxQmaXfiI3cyMQ98Xm-WRjEz1MKVgMjCs4lRyR-Oy4/s1600/Screen+Shot+2019-07-15+at+9.15.25+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="595" data-original-width="617" height="385" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifGhJvAa7sFThuEufmbNosXGhG0iaKkqbeq2ipmSjf7BLVVmRMsblfakRTHQSeE8NpeLRa9Gh3o4RhEC2kaVOMW-d0gxGigXwilmxQmaXfiI3cyMQ98Xm-WRjEz1MKVgMjCs4lRyR-Oy4/s400/Screen+Shot+2019-07-15+at+9.15.25+PM.png" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO4zZp2Z2wjTioYICT21IbrcBQ7X2CocGa5j245wGXL3dzAXXb8YeF2JfVO71-lLn3klpBECxS6aulkhDcLylLvKb5Xvgfvxe3CwTzE729smLObeAKjUBOwiO1hbFtyucfLiZYoac63Zc/s1600/Screen+Shot+2019-07-15+at+9.30.56+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="598" data-original-width="804" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO4zZp2Z2wjTioYICT21IbrcBQ7X2CocGa5j245wGXL3dzAXXb8YeF2JfVO71-lLn3klpBECxS6aulkhDcLylLvKb5Xvgfvxe3CwTzE729smLObeAKjUBOwiO1hbFtyucfLiZYoac63Zc/s400/Screen+Shot+2019-07-15+at+9.30.56+PM.png" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2Fl8DmVr_3yxzAKPhsE6g8OWCf60XymCXYj5V6yHcMTfWEQeYGvYmu0yAYbgzh_3OVygg_IQEXm-RnDATqI-2-d1KuH7UOU1HXI6yxI0uJhBKcZn77v25O09DiT42Cx6cI5jX3Kwd3Js/s1600/Screen+Shot+2019-07-19+at+8.14.41+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="616" data-original-width="553" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh2Fl8DmVr_3yxzAKPhsE6g8OWCf60XymCXYj5V6yHcMTfWEQeYGvYmu0yAYbgzh_3OVygg_IQEXm-RnDATqI-2-d1KuH7UOU1HXI6yxI0uJhBKcZn77v25O09DiT42Cx6cI5jX3Kwd3Js/s400/Screen+Shot+2019-07-19+at+8.14.41+PM.png" width="358" /></a><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzPBX1O49UO-QjLK4i0bsKdf_NnDwFDm9VSHXfFV7Mt7BpAQ9FPeWWviAkFSsy8OIoz3WCoNFFXiGXtLE6GjbOMFyGVn-MGbdKAxuVGdC9ksl1TQlOpOcSWWRYZe8NI3C3ONrB6NGRvOQ/s1600/Screen+Shot+2019-08-18+at+2.44.23+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="539" data-original-width="769" height="280" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhzPBX1O49UO-QjLK4i0bsKdf_NnDwFDm9VSHXfFV7Mt7BpAQ9FPeWWviAkFSsy8OIoz3WCoNFFXiGXtLE6GjbOMFyGVn-MGbdKAxuVGdC9ksl1TQlOpOcSWWRYZe8NI3C3ONrB6NGRvOQ/s400/Screen+Shot+2019-08-18+at+2.44.23+PM.png" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvw9dRhJ7V5E2vUjSG_rg-4qMUPpjyoLUj4azrAkgZUI0iXW5uUTcKshD4o11gihcD_fQt13pPBJxaS-NPRFzom0HWWVsi18onJH5uzo1vrQ52Edbapd58-svKmac3xYHQKez__rdr2eQ/s1600/Screen+Shot+2019-08-19+at+8.48.48+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="623" data-original-width="1092" height="227" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvw9dRhJ7V5E2vUjSG_rg-4qMUPpjyoLUj4azrAkgZUI0iXW5uUTcKshD4o11gihcD_fQt13pPBJxaS-NPRFzom0HWWVsi18onJH5uzo1vrQ52Edbapd58-svKmac3xYHQKez__rdr2eQ/s400/Screen+Shot+2019-08-19+at+8.48.48+PM.png" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdPFI2ANo21cQ_WPwlWNI3OZvWbH9fk4MdfWygQ_25dMR2jC0eoeMgQ9QkU-9rqMzH3DqCkkj6FGoWqOaBVE2lnlQPw6hIadma4qO86cqgFZPUNZl7kVmUAuw8AsT46yUR35QRsk5NYJk/s1600/Screen+Shot+2019-08-24+at+11.27.24+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="728" data-original-width="1294" height="225" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdPFI2ANo21cQ_WPwlWNI3OZvWbH9fk4MdfWygQ_25dMR2jC0eoeMgQ9QkU-9rqMzH3DqCkkj6FGoWqOaBVE2lnlQPw6hIadma4qO86cqgFZPUNZl7kVmUAuw8AsT46yUR35QRsk5NYJk/s400/Screen+Shot+2019-08-24+at+11.27.24+PM.png" width="400" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhApCGO-WKkGFPdmf72XExink0Gs-8ATLev09WfQrlTuNR1Kvb5gIhEC3BIh4DsOhBlmX4K6Yozu_oyVunzNTZdBtr_NHZqcM86cDw_RScSElwNcZDLOwQtOU0E80up9FZ-v__LragKrmc/s1600/Screen+Shot+2019-08-25+at+12.53.09+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="738" data-original-width="1071" height="220" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhApCGO-WKkGFPdmf72XExink0Gs-8ATLev09WfQrlTuNR1Kvb5gIhEC3BIh4DsOhBlmX4K6Yozu_oyVunzNTZdBtr_NHZqcM86cDw_RScSElwNcZDLOwQtOU0E80up9FZ-v__LragKrmc/s320/Screen+Shot+2019-08-25+at+12.53.09+PM.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbp70KQiTEUixLQM0QXD7h4OwVIWbjm8x1T4pHM_AbrN54djl759KwdXo6yX1jdC3w4Ee3v151v4rBZtb0uG_0zfddzk4tOMPNrHu0dm_bHU8BDBeQIjZAjMpC3yPxIIay5Fjsypk-vQc/s1600/Screen+Shot+2019-08-25+at+12.53.35+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="737" data-original-width="1082" height="217" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbp70KQiTEUixLQM0QXD7h4OwVIWbjm8x1T4pHM_AbrN54djl759KwdXo6yX1jdC3w4Ee3v151v4rBZtb0uG_0zfddzk4tOMPNrHu0dm_bHU8BDBeQIjZAjMpC3yPxIIay5Fjsypk-vQc/s320/Screen+Shot+2019-08-25+at+12.53.35+PM.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvt6sM7nSa5_USP-G9wEgrLxP8Y0ldbF96eetF7g33pKYq04X8gi8LhUSB8rMO6WxZpPE2fdTH1o_ljbkmn4tG5XqCTITnC0Zb1t6o93oEn6i-ZmhJ-1oyeI3Xe2ObKsDuY6L3mOBgTxk/s1600/Screen+Shot+2019-08-27+at+7.28.36+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="915" data-original-width="1600" height="182" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvt6sM7nSa5_USP-G9wEgrLxP8Y0ldbF96eetF7g33pKYq04X8gi8LhUSB8rMO6WxZpPE2fdTH1o_ljbkmn4tG5XqCTITnC0Zb1t6o93oEn6i-ZmhJ-1oyeI3Xe2ObKsDuY6L3mOBgTxk/s320/Screen+Shot+2019-08-27+at+7.28.36+PM.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRjuRkdjgAU1rlC5ZQ_8IrPrAcBmUrnxNmkyhLW2zIWQurOV6dg5zuO4SVi1lMoWhY47pMFqeX5O5FkT5NJULgLaifF7Txyb8C02hR4GrYEO4fGjhBUc9ApF9w_1ixKxwNxOY1TRM2FvU/s1600/Screen+Shot+2019-08-30+at+9.44.53+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="660" data-original-width="889" height="237" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRjuRkdjgAU1rlC5ZQ_8IrPrAcBmUrnxNmkyhLW2zIWQurOV6dg5zuO4SVi1lMoWhY47pMFqeX5O5FkT5NJULgLaifF7Txyb8C02hR4GrYEO4fGjhBUc9ApF9w_1ixKxwNxOY1TRM2FvU/s320/Screen+Shot+2019-08-30+at+9.44.53+PM.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkjlWfBXLGtBQol2zrwWJiYvpfMAyqN7HgYi7VOZ7Wcob6X40C73yWlPyh9kkT0BFssjZYsZ2sJVTxOmzo88OIEEV87ZDHNzE2t4IDNACsHwUFFZYDyW2kR6QPZeHWIoVxYvqVG9ObiWs/s1600/Screen+Shot+2019-09-22+at+7.18.26+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="656" data-original-width="1189" height="176" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkjlWfBXLGtBQol2zrwWJiYvpfMAyqN7HgYi7VOZ7Wcob6X40C73yWlPyh9kkT0BFssjZYsZ2sJVTxOmzo88OIEEV87ZDHNzE2t4IDNACsHwUFFZYDyW2kR6QPZeHWIoVxYvqVG9ObiWs/s320/Screen+Shot+2019-09-22+at+7.18.26+PM.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtno4NkNbeKeTEPAbAk-HO52dM2jvltZBr3k9nPkGjZA9-xCPN9EmOW-YF_ICB2khW36SY4OWLaW9Zpj3Hbhup9PC3Xtd44c-K3IjCJW6yOyJkBcrpKQlzAU8AFuP7MNtXKMqZhDeRt5g/s1600/Screen+Shot+2019-09-30+at+8.10.11+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="791" data-original-width="1491" height="169" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtno4NkNbeKeTEPAbAk-HO52dM2jvltZBr3k9nPkGjZA9-xCPN9EmOW-YF_ICB2khW36SY4OWLaW9Zpj3Hbhup9PC3Xtd44c-K3IjCJW6yOyJkBcrpKQlzAU8AFuP7MNtXKMqZhDeRt5g/s320/Screen+Shot+2019-09-30+at+8.10.11+PM.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-83733167940691399722019-02-10T14:35:00.000-05:002019-02-10T14:35:07.937-05:00Performance characteristics of the ML Morse Decoder<br />
<br />
<br />
<br />
In <a href="http://ag1le.blogspot.com/2019/02/training-computer-to-listen-and-decode.html" target="_blank">my previous blog post</a> I described a new Tensorflow based Machine Learning model that learns Morse code from annotated audio .WAV files with 8 kHz sample rate.<br />
<br />
In order to evaluate the performance characteristic of the decoding accuracy from noisy audio source files I created a set of training & validation materials with Signal-to-Noise Ratio from -22dB to +30 dB. Target SNR_dB was created using the following Python code:<br />
<br />
<span style="font-family: "Courier New", Courier, monospace; font-size: xx-small;"> </span><b><span style="font-family: "Courier New", Courier, monospace; font-size: xx-small;"> </span><span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Desired linear SNR</span></b><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> SNR_linear = 10.0**(SNR_dB/10.0)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <b># Measure power of signal - assume zero mean </b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> power = morsecode.var()</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <b># Calculate required noise power for desired SNR</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> noise_power = power/SNR_linear</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <b> # Generate noise with calculated power (mu=0, sigma=1)</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> noise = np.sqrt(noise_power)*np.random.normal(0,1,len(morsecode))</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <b># Add noise to signal</b></span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> morsecode = noise + morsecode</span><br />
<br />
These audio .WAV files contain random words with maximum 5 characters - 5000 samples at each SNR level with 95% used for training and 5% for validation. The Morse speed in each audio sample was randomly selected from 30 WPM or 25 WPM.<br />
<br />
The training was performed until 5 consecutive epochs did not improve the character error rate. The duration of these training sessions varied from 15 - 45 minutes on Macbook Pro with2.2 GHz Intel Core i7 CPU. <br />
<br />
I captured and plotted the Character Error Rate (CER) and Signal-to-Noise Ratio (SNR) of each completed training and validation session. The following graph shows that the Morse decoder performs quite well until about -12 dB SNR level and below that the decoding accuracy drops fairly dramatically.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEiQ8PXNH08YPqu-Y-6ZzUt_ybZFxQ5IlCM5mDUvbzH8FkVv_u1lCB6Tu8P0kbcjkgpdUJeKscyy4pVSXThqCWec5KKFeDH8l0Qgfe9K_UEYjWjPzKFwy0QbIqdcH_olj4_h7BHrzigic/s1600/Screen+Shot+2019-02-10+at+1.50.29+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="891" data-original-width="1274" height="446" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEiQ8PXNH08YPqu-Y-6ZzUt_ybZFxQ5IlCM5mDUvbzH8FkVv_u1lCB6Tu8P0kbcjkgpdUJeKscyy4pVSXThqCWec5KKFeDH8l0Qgfe9K_UEYjWjPzKFwy0QbIqdcH_olj4_h7BHrzigic/s640/Screen+Shot+2019-02-10+at+1.50.29+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">CER vs. SNR graph</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
To view how noisy these files are here are some random samples - first 4 seconds of 8KHz audio file is demodulated, filtered using 25Hz 3rd order Butterworth filter and decimated by 125 to fit into a (128,32) vector. These vectors is shown as grayscale images below:<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGMvJtcWTzyO80HbJJqIvuK6JWMyTXjbKEGoGWRRhJ2vN62ZLvdOkbDmvAFcvLhrEEGS4s6j1r7-VmarNep0fHD-ybIrimDSWJjjDvn5hKHypm6Em7cBkRqGVH5TlFKevVQxKTxKS2vaU/s1600/SNR-6WPM305c6a4dad0a364f23acc612d221cd9f61.wav.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhGMvJtcWTzyO80HbJJqIvuK6JWMyTXjbKEGoGWRRhJ2vN62ZLvdOkbDmvAFcvLhrEEGS4s6j1r7-VmarNep0fHD-ybIrimDSWJjjDvn5hKHypm6Em7cBkRqGVH5TlFKevVQxKTxKS2vaU/s320/SNR-6WPM305c6a4dad0a364f23acc612d221cd9f61.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">-6 db SNR</td></tr>
</tbody></table>
<br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgop-Cj2_OaM0IN6Cj10AV5O3rQnA_TYfKtPdqIq_fwqa4rLggu6mD3S7e_3eIXFRwXhHqNJvm8LYUI8J04ZyVQloova8QgslZG0qDrUZctURk0MfVX_XZ8xlJPQasdvPQfGXX3btGe8Gw/s1600/SNR-11WPM30ef70f1df28a7473c974a5a1b0cb564c6.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgop-Cj2_OaM0IN6Cj10AV5O3rQnA_TYfKtPdqIq_fwqa4rLggu6mD3S7e_3eIXFRwXhHqNJvm8LYUI8J04ZyVQloova8QgslZG0qDrUZctURk0MfVX_XZ8xlJPQasdvPQfGXX3btGe8Gw/s320/SNR-11WPM30ef70f1df28a7473c974a5a1b0cb564c6.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">-11 dB SNR</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkR31cn7Z-N5wYaj5ARKbzf2-NRuYQopb8m6OYzmjjuJOdmbw__651YoT-ocyaXw2JhdRrSJE-uO2yLct7VlaxA3rMOROoV2wfmGuH3RRB4i2x6tBhay9u7APWrea69Bp5fUZYAXQ9xoU/s1600/SNR-13WPM2540959a06a9cb4f8db8684bd1f9678e15.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkR31cn7Z-N5wYaj5ARKbzf2-NRuYQopb8m6OYzmjjuJOdmbw__651YoT-ocyaXw2JhdRrSJE-uO2yLct7VlaxA3rMOROoV2wfmGuH3RRB4i2x6tBhay9u7APWrea69Bp5fUZYAXQ9xoU/s320/SNR-13WPM2540959a06a9cb4f8db8684bd1f9678e15.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">-13 dB SNR</td></tr>
</tbody></table>
<div>
<br /></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_rMuvXSZhLhc8LXpkeC3gYNWewP2j2khcyHjtfrSFgSkJUl8Ce4DxUVepgx8tbeQHsyEKjiA-yue3l8V-rpEl6SpTZluh8M-0aUS6c3AfzTKOIWAVZX3FTZwxHdstJp4v1ciR2rUMKIY/s1600/SNR-16WPM302c58361ac3744fa5b9877ca9e700f93a.wav.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh_rMuvXSZhLhc8LXpkeC3gYNWewP2j2khcyHjtfrSFgSkJUl8Ce4DxUVepgx8tbeQHsyEKjiA-yue3l8V-rpEl6SpTZluh8M-0aUS6c3AfzTKOIWAVZX3FTZwxHdstJp4v1ciR2rUMKIY/s320/SNR-16WPM302c58361ac3744fa5b9877ca9e700f93a.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">-16 dB SNR</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
Conclusions</h3>
<div>
The Tensorflow model appears to perform quite well on decoding noisy audio files, at least when the training set and validation set have the same SNR level. </div>
<div>
<br /></div>
<div>
The next experiments could include more variability with a much bigger training dataset that has a combination of different SNR, Morse speed and other variables. The training duration depends on the amount of training data so it can take a while to perform these larger scale experiments on a home computer. </div>
<div>
<br /></div>
<div>
73 de</div>
<div>
Mauri AG1LE </div>
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com2tag:blogger.com,1999:blog-3326773214329183284.post-90704304516405169372019-02-02T23:51:00.002-05:002019-02-03T17:58:57.573-05:00Training a Computer to Listen and Decode Morse Code<h3>
Abstract</h3>
<div>
<div>
I trained a Tensorflow based CNN-LSTM-CTC model with 5.2 hours of Morse audio training set (5000 files) and achieved character error rate of 0.1% and word accuracy of 99.5% I tested the model with audio files containing various levels of noise and found the model to decode relatively accurately down to -3 dB SNR level. </div>
</div>
<h3>
Introduction</h3>
Decoding Morse code from audio signals is not a novel idea. The author has written many different software decoder implementations that use simplistic models to convert a sequence of "Dits" and "Dahs" to corresponding text. When the audio signal is noise free and there is no interference, these simplistic methods work fairly well and produce nearly error free decoding. Figure 1. below shows "Hello World" with 35 dB signal-to-noise ratio that most conventional decoders don't have any problems decoding.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguBaC1PrEw-HFRzKqf1tnhl4diuuGvuoPH9UtI-yXbY7U0DfEyV6P7PcfL_DrMgqOZSP6vFR7w4Sl6G8iFSaNlO34vLtdp_UEERjSW3CLXpyXie3xWbEaoDPvaf-vS6LpkM_qcCMLMYsc/s1600/morse_recognition.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="471" data-original-width="1170" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEguBaC1PrEw-HFRzKqf1tnhl4diuuGvuoPH9UtI-yXbY7U0DfEyV6P7PcfL_DrMgqOZSP6vFR7w4Sl6G8iFSaNlO34vLtdp_UEERjSW3CLXpyXie3xWbEaoDPvaf-vS6LpkM_qcCMLMYsc/s640/morse_recognition.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">"Hello World" with 30 dB SNR </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
Figure 2 below shows the same "Hello World" but with -12 dB signal-to-noise ratio using exactly same process as above to extract the demodulated envelope. Humans can still hear and even recognize the Morse code faintly in the noise. Computers equipped with these simplistic models have great difficulties decoding anything meaningful out of this signal. In ham radio terms the difference of 47 dB corresponds roughly eight S units - human ears & brain can still decode S2 level signals whereas conventional software based Morse decoders produce mostly gibberish.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZTk-7qZzhxtHVoVLWM2n60Q5KDhmusz-vWn_mVvkgM4fcQuOFAj9TWoIcBBxG7xQlR_7mW4rZkxLBDD6SUry7HAQXJrMBNaV1PDGujLtNJ_QfkOjW0WZ-RbRH_z4trXZFoZbCsXzHi6c/s1600/morse_recognition2.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="471" data-original-width="1170" height="256" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgZTk-7qZzhxtHVoVLWM2n60Q5KDhmusz-vWn_mVvkgM4fcQuOFAj9TWoIcBBxG7xQlR_7mW4rZkxLBDD6SUry7HAQXJrMBNaV1PDGujLtNJ_QfkOjW0WZ-RbRH_z4trXZFoZbCsXzHi6c/s640/morse_recognition2.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">"Hello World" with -12 dB SNR </td></tr>
</tbody></table>
<br />
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<br />
<br />
<h3>
New Approach - Machine Learning</h3>
I have been quite interested in Machine Learning (ML) technologies for a while. From software development perspective ML is changing the paradigm how we are processing data.<br />
<br />
In traditional programming we look at the input data and try to write a program that uses some processing steps to come up with the output data. Depending on the complexity of the problem software developer may need to spend quite a long time coming up with the correct algorithms to produce the right output data. From Morse decoder perspective this is how most decoders work: they take input audio data that contains the Morse signals and after many complex operations the correct decoded text appears on the screen. <br />
<br />
Machine Learning changes this paradigm. As a ML engineer you need to curate a dataset that has a representative selection of input data with corresponding output data (also known as label data). The computer then applies a training algorithm to this dataset that eventually discovers the correct "program" - the ML model that provides the best matching function that can infer the correct output, given the input data.<br />
<br />
See Figure 3. that tries to depict this difference between traditional programming and the new approach with Machine Learning.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAljwlpV3mHd3-HwZ7tK0LH0q7jO5TG2SwOLUttufEnF1fU_Uy0ezE67DyHvQ6YmvXGvXQe67RQFdbCqP28wxIykdy3Ynas0fYPihENFJUSrV-8oQQTy0SILehffQOPaGTuvKybbjeiRU/s1600/Programming_vs_MachineLearning.png.001.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="768" data-original-width="1024" height="480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiAljwlpV3mHd3-HwZ7tK0LH0q7jO5TG2SwOLUttufEnF1fU_Uy0ezE67DyHvQ6YmvXGvXQe67RQFdbCqP28wxIykdy3Ynas0fYPihENFJUSrV-8oQQTy0SILehffQOPaGTuvKybbjeiRU/s640/Programming_vs_MachineLearning.png.001.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Programming vs. Machine Learning</td></tr>
</tbody></table>
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<br />
So what does this new approach mean in practice? Instead of trying to figure out ever more complex software algorithms to improve your data processing and accuracy of decoding, you can select from some standard machine learning algorithms that are available in open source packages like Tensorflow and focus on building a neural network model and curating a large dataset to train this model. The trained model can then be used to make the decoding from the input audio data. This is exactly what I did in the following experiment.<br />
<br />
I took a Tensorflow implementation of <a href="https://github.com/ag1le/SimpleHTR" target="_blank">Handwritten Text Recognition</a> created by Harald Scheidl [3] that he has posted in Github as an open source project. He has provided excellent documentation on how the model works as well as references to <a href="http://www.fki.inf.unibe.ch/databases/iam-handwriting-database" target="_blank">the IAM dataset</a> that he is using for training the handwritten text recognition.<br />
<br />
Why would a model created for handwritten text recognition work for Morse code recognition?<br />
<br />
It turns out that the Tensorflow standard learning algorithms used for handwriting recognition are very similar to ones used for speech recognition.<br />
<br />
The figures below are from <a href="https://distill.pub/2017/ctc/" target="_blank">Hannun, "Sequence Modeling with CTC", Distill, 2017</a>. In the article Hannun [2] shows that the (x,y) coordinates of a pen stroke or pixels in image can be recognized as text, like the spectrogram of speech audio signals. Morse code has similar properties as speech - the speed can vary a lot and hand-keyed code can have unique rhythm patterns that make it difficult to align signals to decoded text. The common theme is that we have some variable length input data that need to be aligned with variable length output data. The algorithm that comes with Tensorflow is called Connectionist Temporal Classification (CTC) [1].<br />
<div>
<br /></div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjABIP5Jg4K2q7_GJpmqb29BseWyUBqwFcTCOiE6XMe2INy8evuAKguMRF-TVX3obMF0lRP5uJFH1TXYfinKIW3jTd2Mq8Ccb_3Fp9QzGvnmutb0s9ylRsOTkYCzwXpETop9PPqsY6sLe4/s1600/Screen+Shot+2019-02-02+at+8.12.33+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="480" data-original-width="726" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjABIP5Jg4K2q7_GJpmqb29BseWyUBqwFcTCOiE6XMe2INy8evuAKguMRF-TVX3obMF0lRP5uJFH1TXYfinKIW3jTd2Mq8Ccb_3Fp9QzGvnmutb0s9ylRsOTkYCzwXpETop9PPqsY6sLe4/s200/Screen+Shot+2019-02-02+at+8.12.33+PM.png" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;"></td></tr>
</tbody></table>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9nzQzM5a3b46bKe2dogA6qQNTaaPGHfeWJE6aYXdSESJumY5Q__zZUPqLJ9MfFJNmex3lOYHiTQ-KnNkiVKyVB0SYItPwiePeWQEbaOwTBqV6z9wk2SOFOB6mEyj4vkIMm8B84fe0cIU/s1600/Screen+Shot+2019-02-02+at+8.12.51+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em; text-align: center;"><img border="0" data-original-height="480" data-original-width="726" height="130" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi9nzQzM5a3b46bKe2dogA6qQNTaaPGHfeWJE6aYXdSESJumY5Q__zZUPqLJ9MfFJNmex3lOYHiTQ-KnNkiVKyVB0SYItPwiePeWQEbaOwTBqV6z9wk2SOFOB6mEyj4vkIMm8B84fe0cIU/s200/Screen+Shot+2019-02-02+at+8.12.51+PM.png" width="200" /></a><br />
<br />
<h3>
Morse Dataset</h3>
The Morse code audio file can be easily converted to a representation that is suitable as input data for these neural networks. I am using single track (mono) WAV files with 8 kHz sampling frequency.<br />
<br />
The following few lines of Python code takes 4 seconds sample from an existing WAV audio file, finds the signal peak frequency, de-modulates and decimates the data so that we get a (1,256) vector that we re-shape to (128, 32) and write into a PNG file.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">def find_peak(fname):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> # Find the signal frequency and maximum value</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> Fs, x = wavfile.read(fname)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> f,s = periodogram(x, Fs,'blackman',8192,'linear', False, scaling='spectrum')</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> threshold = max(s)*0.9 # only 0.4 ... 1.0 of max value freq peaks included</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> maxtab, mintab = peakdet(abs(s[0:int(len(s)/2-1)]), threshold,f[0:int(len(f)/2-1)] )</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> return maxtab[0,0]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">def demodulate(x, Fs, freq):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> # demodulate audio signal with known CW frequency </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> t = np.arange(len(x))/ float(Fs)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> mixed = x*((1 + np.sin(2*np.pi*freq*t))/2 )</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> #calculate envelope and low pass filter this demodulated signal</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> #filter bandwidth impacts decoding accuracy significantly </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> #for high SNR signals 40 Hz is better, for low SNR 20Hz is better</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> # 25Hz is a compromise - could this be made an adaptive value?</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> low_cutoff = 25. # 25 Hz cut-off for lowpass</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> wn = low_cutoff/ (Fs/2.) </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> b, a = butter(3, wn) # 3rd order butterworth filter</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> z = filtfilt(b, a, abs(mixed))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> # decimate and normalize</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> decimate = int(Fs/64) # 8000 Hz / 64 = 125 Hz => 8 msec / sample </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> o = z[0::decimate]/max(z)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> return o</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">def process_audio_file(fname, x, y, tone):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> Fs, signal = wavfile.read(fname)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> dur = len(signal)/Fs</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> o = demodulate(signal[(Fs*(x)):Fs*(x+y)], Fs, tone)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> return o, dur</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">filename = "error.wav"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">tone = find_peak(filename)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">o,dur = process_audio_file(filename,0,4, tone)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">im = o[0::1].reshape(1,256)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">im = im*256.</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">img = cv2.resize(im, (128, 32), interpolation = cv2.INTER_AREA)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">cv2.imwrite("error.png",img)</span><br />
<br />
Here is the resulting PNG image - it contains "ERROR M". The labels are kept in a file that contains also the corresponding audio file name.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHb50QsyylxbpOm3W2jBXFv4tFH-40I_bV8k8dtjq-ew7W6hVbGjCww_be55jT5XfTqb-Kpv_rOfI-hC-qBl91ghZrncSR9-VeQCz9tXDM6wmQs9gyi-qQY1RA6rCeq00fLzbkk0l6vws/s1600/f5f0c1d7aa804ef3b1c1d4626a51eec2.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHb50QsyylxbpOm3W2jBXFv4tFH-40I_bV8k8dtjq-ew7W6hVbGjCww_be55jT5XfTqb-Kpv_rOfI-hC-qBl91ghZrncSR9-VeQCz9tXDM6wmQs9gyi-qQY1RA6rCeq00fLzbkk0l6vws/s320/f5f0c1d7aa804ef3b1c1d4626a51eec2.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">4 second audio sample converted to a (128,32) PNG file</td></tr>
</tbody></table>
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<br />
<br />
<br />
<br />
<br />
<br />
It is very easy to produce a lot of training and validation data with this method. The important part is that each audio file must have accurate "labels" - this is the textual representation of the Morse audio file.<br />
<br />
I created a small Python script to produce this kind of Morse training and validation dataset. With a few parameters you can generate as much data as you want with different speed and noise levels.<br />
<br />
<h3>
Model</h3>
<div>
I used Harald's model to start the Morse decoding experiments. </div>
<div>
<br /></div>
<div>
The model consists of 5 CNN layers, 2 RNN (LSTM) layers and the CTC loss and decoding layer. The illustration below gives an overview of the NN (green: operations, pink: data flowing through NN) and here follows a short description:</div>
<div>
<div>
<ul>
<li>The input image is a gray-value image and has a size of 128x32</li>
<li>5 CNN layers map the input image to a feature sequence of size 32x256</li>
<li>2 LSTM layers with 256 units propagate information through the sequence and map the sequence to a matrix of size 32x80. Each matrix-element represents a score for one of the 80 characters at one of the 32 time-steps</li>
<li>The CTC layer either calculates the loss value given the matrix and the ground-truth text (when training), or it decodes the matrix to the final text with best path decoding or beam search decoding (when inferring)</li>
<li>Batch size is set to 50</li>
</ul>
</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://github.com/ag1le/SimpleHTR/raw/master/doc/nn_overview.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" data-original-height="416" data-original-width="640" height="260" src="https://github.com/ag1le/SimpleHTR/raw/master/doc/nn_overview.png" width="400" /></a></div>
<div>
<br /></div>
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
It is not hard to imagine making some changes to the model to allow for longer audio clips to be decoded. Right now the limit is about 4 seconds audio converted to (128x32) input image. Harald is actually providing <a href="https://towardsdatascience.com/faq-build-a-handwritten-text-recognition-system-using-tensorflow-27648fb18519" target="_blank">details of a model</a> that can handle larger input image (800x64) and output up to 100 characters strings.<br />
<h3>
Experiment</h3>
Here are parameters I used for this experiment:<br />
<br />
<ul>
<li>5000 samples, split into training and validation set: 95% training - 5% validation</li>
<li>Each sample has 2 random words, max word length is 5 characters</li>
<li>Morse speed randomly selected from [20, 25, 30] words-per-minute </li>
<li>Morse audio SNR: 40 dB </li>
<li>batchSize: 100 </li>
<li>imgSize: [128,32] </li>
<li>maxTextLen: 32</li>
<li>earlyStopping: 20 </li>
</ul>
<br />
Training time was 1hr 51mins on a Macbook Pro 2.2 GHz Intel Core i7<br />
Training curves of character error rate, word accuracy and loss after 50 epochs were the following:<br />
<div style="text-align: left;">
</div>
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1gxx4NyPNdCsAcLJ41-vlhw7THXWVv0nyxmMDGF0R7-0VoF96GsS0m8MRTadYfL4GZkQrx8Dvc3PrJOEZgXnv7dQTXDhZcxQ1SKwHh5dgsU8BbfC6rs4Elv7g24yB7AEeD12kCHHmppY/s1600/Screen+Shot+2019-02-03+at+4.27.47+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="950" data-original-width="1600" height="377" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1gxx4NyPNdCsAcLJ41-vlhw7THXWVv0nyxmMDGF0R7-0VoF96GsS0m8MRTadYfL4GZkQrx8Dvc3PrJOEZgXnv7dQTXDhZcxQ1SKwHh5dgsU8BbfC6rs4Elv7g24yB7AEeD12kCHHmppY/s640/Screen+Shot+2019-02-03+at+4.27.47+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Training over 50 epochs</td></tr>
</tbody></table>
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The best character error rate was 14.9% and word accuracy was 36.0%. These are not great numbers - the reason was that I had training data containing 2 words in each sample - in many cases this was too many characters to fit in the 4 second time window, therefore the training algorithm did not see the second word in the training material in many cases. <br />
<br />
I did re-run the experiment with 5000 samples, but with just one word in each sample. It took 54 mins 7 seconds to do this training. New parameters are below:<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">model:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> # model constants</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> batchSize: 100 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> imgSize: !!python/tuple [128,32] </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> maxTextLen: 32</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> earlyStopping: 5</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">morse:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnTrain: "morsewords.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnAudio: "audio/"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> count: 5000</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> SNR_dB: </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 20</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 30</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 40</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> f_code: 600</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Fs: 8000</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> code_speed: </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 30</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 25</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 20</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> length_N: 65000</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> play_sound: False</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> word_max_length: 5</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> words_in_sample: 1</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">experiment:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> modelDir: "model/"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnAccuracy: "model/accuracy.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnTrain: "model/morsewords.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnInfer: "model/test.png"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnCorpus: "model/corpus.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnCharList: "model/charList.txt"</span><br />
<br />
<br />
Here is the outcome of that second training session:<br />
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<br />
<div class="p1">
<span class="s1">Total training time was 0:54:07.857731</span></div>
<div class="p1">
<span class="s1">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
</span></div>
<div class="p1">
<span class="s1">Character error rate:<span class="Apple-converted-space"> </span>0.1%. Word accuracy: 99.5%.</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0IjvKkI6JYcawlJfzGeXmRaYxHS_HI0LiRwD-nbUva2_FfgcwMpfHC4Y6M2oVMufhOWxQSld8zdWy_E1qYgSfARcxopHXXSTiBIom90qiPaQONlpYOvBL0Dix5vbyuqJMTOOgLhNuU0g/s1600/Screen+Shot+2019-02-03+at+5.53.01+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="950" data-original-width="1600" height="377" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0IjvKkI6JYcawlJfzGeXmRaYxHS_HI0LiRwD-nbUva2_FfgcwMpfHC4Y6M2oVMufhOWxQSld8zdWy_E1qYgSfARcxopHXXSTiBIom90qiPaQONlpYOvBL0Dix5vbyuqJMTOOgLhNuU0g/s640/Screen+Shot+2019-02-03+at+5.53.01+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Training over 33 epochs</td></tr>
</tbody></table>
<div class="p1">
<span class="s1"><br /></span></div>
<br />
<br />
With a larger dataset the training will take longer. One possibility would be to use AWS cloud computing service to accelerate the training for a much larger dataset. </div>
<div>
<br /></div>
<div>
Note that the model did not know anything about Morse code at the start. It did learn the character set, the structure of the Morse code and the words just by "listening" through the provided sample files. This is approximately 5.3 hours of Morse code audio materials with random words. (5000 files * 95% * 4 sec/file = 19000 seconds). </div>
<div>
<br /></div>
<div>
It would be great to get some comparative data on how quickly humans will learn to produce similar character error rate. </div>
<h3>
Results</h3>
<div>
I created a small "helloword.wav" audio file with HELLO WORLD text at 25 WPM in different signal-to-noise ratios (-6, -3, +6, +50) dB to test the first model. </div>
<div>
<br /></div>
<div>
Attempting to decode the content of the audio file I got the following results. Given that the training was done with +40 dB samples I was quite surprised to see relatively good decoding accuracy. The model also provides probability how confident it is about the result. These values vary between 0.4% to 5.7%. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
File: -6 dB SNR </div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">python MorseDecoder.py -f audio/helloworld.wav<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Validation character error rate of saved model: 15.4</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Python: 2.7.10 (default, Aug 17 2018, 19:45:58)<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Tensorflow: 1.4.0</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">2019-02-02 22:40:51.970393: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Init with stored values from model/snapshot-22</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">inferBatch: probs:[ 0.00420194] texts:['HELL Q PE']<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Recognized: "HELL Q PE"</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Probability: 0.00420194</span></span></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<br />
<div class="p1">
<span class="s1"><span style="font-family: inherit;">['HELL Q PE']</span></span></div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj215m9-ncuvLIoaHnAsFUilNVgntPJw1gNkpx4wbt3F3zfAeUqq_MkZj8gPrXXjvbbzbUSaXP5PmEBFs8AR_a08DpTIu17At_n-1YKA-lmA_aAo5YaeejaAXY2V_pXWzJHNrNeQjgdHjo/s1600/helloworld.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj215m9-ncuvLIoaHnAsFUilNVgntPJw1gNkpx4wbt3F3zfAeUqq_MkZj8gPrXXjvbbzbUSaXP5PmEBFs8AR_a08DpTIu17At_n-1YKA-lmA_aAo5YaeejaAXY2V_pXWzJHNrNeQjgdHjo/s320/helloworld.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">-6 dB HELLO WORLD</td></tr>
</tbody></table>
<br />
<br />
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "times"; font-size: small; font-variant-ligatures: normal;">File: -3 dB SNR </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">python MorseDecoder.py -f audio/helloworld.wav<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Validation character error rate of saved model: 15.4</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Python: 2.7.10 (default, Aug 17 2018, 19:45:58)<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Tensorflow: 1.4.0</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">2019-02-02 22:36:32.838156: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Init with stored values from model/snapshot-22</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">inferBatch: probs:[ 0.05750186] texts:['HELLO WOE']<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Recognized: "HELLO WOE"</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Probability: 0.0575019</span></span></div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<br />
<div class="p1">
<span class="s1"><span style="font-family: "times" , "times new roman" , serif; font-size: xx-small;">['HELLO WOE']</span></span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6TOcx8cHEFSdZ6yEkJ8eY4utSwEAEk_ryG7D1zLez21xoZ5dw_nmzuv7vjLAITm3hYfqvRZCo-iK3-goKqxjWkI8Cp1QJyTuX8oZuBuxU3yV_DixYUTH9u5XYRmKScUEp7QEFBB8Yb5Y/s1600/helloworld.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6TOcx8cHEFSdZ6yEkJ8eY4utSwEAEk_ryG7D1zLez21xoZ5dw_nmzuv7vjLAITm3hYfqvRZCo-iK3-goKqxjWkI8Cp1QJyTuX8oZuBuxU3yV_DixYUTH9u5XYRmKScUEp7QEFBB8Yb5Y/s320/helloworld.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">-3 dB HELLO WORLD</td></tr>
</tbody></table>
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<div class="p1">
<span style="font-family: "times"; font-size: small; font-weight: normal;"><br /></span>
<span style="font-family: "times"; font-size: small; font-weight: normal;"><br /></span>
<span style="font-family: "times"; font-size: small; font-weight: normal;"><br /></span>
<span style="font-family: "times"; font-size: small; font-weight: normal;"><br /></span>
<span style="font-family: "times"; font-size: small; font-weight: normal;"><br /></span>
<span style="font-family: "times"; font-size: small; font-weight: normal;">File: +6 dB SNR </span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">python MorseDecoder.py -f audio/helloworld.wav<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">Validation character error rate of saved model: 15.4</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">Python: 2.7.10 (default, Aug 17 2018, 19:45:58)<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">Tensorflow: 1.4.0</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">2019-02-02 22:38:57.549928: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">Init with stored values from model/snapshot-22</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">inferBatch: probs:[ 0.03523131] texts:['HELLO WOT']<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">Recognized: "HELLO WOT"</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small; font-weight: normal;">Probability: 0.0352313</span></span></div>
<div class="p1">
<span class="s1" style="font-family: "times" , "times new roman" , serif; font-size: xx-small; font-weight: normal;">['HELLO WOT']</span></div>
<div class="p1">
<span class="s1"><br /></span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVGmRoCQ7aYItVd_wUGqIyWKr9hUsEfGs0eT9XfpTl3kTCmkHPk2qhZt_G6GwuqacxfBGXvEm_pe_nv7yDjbmfeDUL_GK4ttr7SDKymCV0F6D6e-6-dDTuN5CZ9PVF41lATIbPfY5apiY/s1600/helloworld.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVGmRoCQ7aYItVd_wUGqIyWKr9hUsEfGs0eT9XfpTl3kTCmkHPk2qhZt_G6GwuqacxfBGXvEm_pe_nv7yDjbmfeDUL_GK4ttr7SDKymCV0F6D6e-6-dDTuN5CZ9PVF41lATIbPfY5apiY/s320/helloworld.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">+6 dB HELLO WORLD</td></tr>
</tbody></table>
<div class="p1">
<span class="s1"><span style="font-family: inherit; font-size: x-small;"><br /></span></span></div>
</h3>
<h3>
<span style="font-family: inherit; font-size: x-small;"><br /></span></h3>
<h3>
<span style="font-family: inherit; font-size: x-small;"><br /></span></h3>
<div>
<span style="font-family: inherit; font-size: x-small;"><br /></span></div>
<div>
<div class="p1">
<span class="s1"><span style="font-family: inherit; font-size: x-small;"><br /></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: inherit; font-size: x-small;">File: +50 dB SNR </span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">python MorseDecoder.py -f audio/helloworld.wav<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Validation character error rate of saved model: 15.4</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Python: 2.7.10 (default, Aug 17 2018, 19:45:58)<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Tensorflow: 1.4.0</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">2019-02-02 22:42:55.403738: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">inferBatch: probs:[ 0.03296029] texts:['HELLO WOT']<span class="Apple-converted-space"> </span></span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Recognized: "HELLO WOT"</span></span></div>
<div class="p1">
<span class="s1"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Probability: 0.0329603</span></span></div>
<div class="p1">
<span class="s1" style="font-size: xx-small;">['HELLO WOT']</span></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUeYs33D6yUNMBLjxtVpzky1ZViYvANnt2HNcA2H3ndmgC5aP3e6hmcijOrMI0Ggtd-2SIK_b8Nx-FY4iDE_xkU-a8dA7ZUxhiIPiPMUwEGtYo_BsGs7ZwjFjfDwkJyflfyOrO3ZW6wx0/s1600/helloworld.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUeYs33D6yUNMBLjxtVpzky1ZViYvANnt2HNcA2H3ndmgC5aP3e6hmcijOrMI0Ggtd-2SIK_b8Nx-FY4iDE_xkU-a8dA7ZUxhiIPiPMUwEGtYo_BsGs7ZwjFjfDwkJyflfyOrO3ZW6wx0/s320/helloworld.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">+50 dB HELLO WORLD</td></tr>
</tbody></table>
<div class="p1">
<span class="s1"><br /></span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
</h3>
<h3>
</h3>
<div>
<br />
<br />
<br />
<br />
<br />
In comparison, I took one file that was used in the training process. This file contains "HELLO HERO" text at +40 dB SNR. Here is what the decoder was able to decode - with much higher probability 51.8% </div>
<div>
<br /></div>
<div>
<span style="font-family: "menlo"; font-size: x-small; font-variant-ligatures: no-common-ligatures;">File: +40 dB SNR </span></div>
<div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
<br />
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">python MorseDecoder.py -f audio/6e753ac57d4849ef87d5146e158610f0.wav</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Validation character error rate of saved model: 15.4</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Python: 2.7.10 (default, Aug 17 2018, 19:45:58)<span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)]</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Tensorflow: 1.4.0</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">2019-02-02 22:53:27.029448: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Init with stored values from model/snapshot-22</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">inferBatch: probs:[ 0.51824665] texts:['HELLO HERO']<span class="Apple-converted-space"> </span></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Recognized: "HELLO HERO"</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Probability: 0.518247</span></div>
<div class="p1">
<span class="s1" style="font-size: xx-small;">['HELLO HERO']</span></div>
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiucWSJojYCZDbU3cvUSfzxXjOHckmgDDy4-uF7DvHk6oW72XlvxqBb3XP9B1yVNn6NlB4SQZEqKfKzZPq1rFRf_iRDdDOpV96SP2PUzhwGivtJj1p7X-2Ah1dO36FGxfp8TGQRRmoC1Do/s1600/6e753ac57d4849ef87d5146e158610f0.wav.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="32" data-original-width="128" height="80" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiucWSJojYCZDbU3cvUSfzxXjOHckmgDDy4-uF7DvHk6oW72XlvxqBb3XP9B1yVNn6NlB4SQZEqKfKzZPq1rFRf_iRDdDOpV96SP2PUzhwGivtJj1p7X-2Ah1dO36FGxfp8TGQRRmoC1Do/s320/6e753ac57d4849ef87d5146e158610f0.wav.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">+40 dB HELLO HERO</td></tr>
</tbody></table>
<div>
<br /></div>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
Conclusions</h3>
This is my first machine learning experiment where I used Morse audio files for both training and validation of the model. The current model limitation is that only 4 second audio clips can be used. However, it is very feasible to build a larger model that can decode longer audio clip with a single inference operation. Also, it would be possible to feed a longer audio file in 4 second pieces to get decoding happening across the whole file.<br />
<br />
This Morse decoder doesn't have a single line of code that would explicitly spell out the Morse codebook. The model literally learned from the training data what Morse code is and how to decode it. It represents a new paradigm in building decoders, and is using similar technology what companies like Google, Microsoft, Amazon and Apple are using for their speech recognition products.<br />
<br />
I hope that this experiment demonstrates to the ham radio community how to build high quality, open source Morse decoders using a simple, standards based ML architecture. With more computing capacity and larger training / validation datasets that contain accurate annotated (labeled) audio files it is now feasible to build a decoder that will surpass the accuracy of conventional decoders (like the one in FLDIGI software).<br />
<br />
73 de Mauri<br />
AG1LE<br />
<br />
<h3>
Software and Instructions</h3>
The initial version of the software is available in Github - see <a href="https://github.com/ag1le/LSTM_morse/blob/master/MorseDecoder.py" target="_blank">here</a><br />
<br />
Using from the command line:<br />
<br />
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">python MorseDecoder.py -h</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">usage: MorseDecoder.py [-h] [--train] [--validate] [--generate] [-f FILE]</span></div>
<div class="p2">
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="s1"></span><br /></span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">optional arguments:</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>-h, --help<span class="Apple-converted-space"> </span>show this help message and exit</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>--train <span class="Apple-converted-space"> </span>train the NN</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>--validate<span class="Apple-converted-space"> </span>validate the NN</span></div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>--generate<span class="Apple-converted-space"> </span>generate a Morse dataset of random words</span></div>
<div class="p1">
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>
</div>
<div class="p1">
<span class="s1" style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><span class="Apple-converted-space"> </span>-f FILE <span class="Apple-converted-space"> </span>input audio file</span></div>
<br />
<br />
To get started you need to generate audio training material. The count variable in model.yaml config file tells how many samples will get generated. Default is 5000.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: 11px;">python MorseDecoder.py </span><span style="font-size: 11px;">--generate</span></span><br />
<span style="font-family: "menlo"; font-size: 11px;"><br /></span>
<br />
Next you need to perform the training. You need to have "audio/", "image/" and "model/" subdirectories on the folder you are running the program.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: 11px;">python MorseDecoder.py </span><span style="font-size: 11px;">--</span><span style="font-size: 11px;">train</span></span><br />
<span style="font-family: "menlo"; font-size: 11px;"><br /></span>
<br />
Last this to do is to validate the model:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: 11px;">python MorseDecoder.py </span><span style="font-size: 11px;">--validate</span></span><br />
<br />
To have the model decode a file you should use:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: 11px;">python MorseDecoder.py -f audio/myfilename.wav </span><br />
<br />
<br />
<br />
<br />
Config file model.yaml (first training session):<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">model:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> # model constants</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> batchSize: 100 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> imgSize: !!python/tuple [128,32] </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> maxTextLen: 32</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> earlyStopping: 20 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">morse:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnTrain: "morsewords.txt"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnAudio: "audio/"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> count: 5000</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> SNR_dB: 20</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> f_code: 600</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> Fs: 8000</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> code_speed: 30</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> length_N: 65000</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> play_sound: False</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> word_max_length: 5</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> words_in_sample: 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">experiment:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> modelDir: "model/"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnAccuracy: "model/accuracy.txt"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnTrain: "model/morsewords.txt"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnInfer: "model/test.png"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnCorpus: "model/corpus.txt"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> fnCharList: "model/charList.txt"</span><br />
<h3>
</h3>
Config file model.yaml (second training session):<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">model:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> # model constants</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> batchSize: 100 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> imgSize: !!python/tuple [128,32] </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> maxTextLen: 32</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> earlyStopping: 5</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">morse:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnTrain: "morsewords.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnAudio: "audio/"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> count: 5000</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> SNR_dB: </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 20</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 30</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 40</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> f_code: 600</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Fs: 8000</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> code_speed: </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 30</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 25</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> - 20</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> length_N: 65000</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> play_sound: False</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> word_max_length: 5</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> words_in_sample: 1</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">experiment:</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> modelDir: "model/"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnAccuracy: "model/accuracy.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnTrain: "model/morsewords.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnInfer: "model/test.png"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnCorpus: "model/corpus.txt"</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> fnCharList: "model/charList.txt"</span><br />
<div>
<br /></div>
<h3>
References</h3>
<div>
[1] A. Graves, S. Fernandez, F. Gomez, and J. Schmidhuber, “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks,” in Proceedings of
the 23rd international conference on Machine learning. ACM,
2006, pp. 369–376. https://www.cs.toronto.edu/~graves/icml_2006.pdf</div>
<div>
[2] Hannun, "Sequence Modeling with CTC", Distill, 2017. https://distill.pub/2017/ctc/</div>
<div>
[3] Harald Scheidl "Handwritten Text Recognition with TensorFlow", https://github.com/githubharald/SimpleHTR</div>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000}
p.p2 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; min-height: 13.0px}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com9tag:blogger.com,1999:blog-3326773214329183284.post-71607758080580819232017-11-25T22:07:00.003-05:002017-11-25T22:26:23.418-05:00MORSE: DENOISING AUTO-ENCODER<h3>
Introduction</h3>
<a href="https://en.wikipedia.org/wiki/Autoencoder#Denoising_autoencoder" target="_blank">Denoising auto-encoder</a> (DAE) is an artificial neural network used for unsupervised learning of efficient codings. DAE takes a partially corrupted input whilst training to recover the original undistorted input.<br />
<br />
For ham radio amateurs there are many potential use cases for de-noising auto-encoders. In this blogpost I share an experiment where I trained a neural network to decode morse code from very noisy signal.<br />
<br />
Can you see the Morse character in the figure 1. below? This looks like a bad waterfall display with a lot of background noise.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiE49JamcFN-lmVMhsrd0mwKj4XQHZzu8FIXLRlLeJIWx9b4sfrCsuRWNPh_wnqNIz26GoN3FxbaxyHBNdcOz36aBkyhnindQT46iSLr11doNe9iVbY4qqASaOlHkVoP6_GiNxU9UpIJeA/s1600/Screen+Shot+2017-11-25+at+8.34.50+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="496" data-original-width="558" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiE49JamcFN-lmVMhsrd0mwKj4XQHZzu8FIXLRlLeJIWx9b4sfrCsuRWNPh_wnqNIz26GoN3FxbaxyHBNdcOz36aBkyhnindQT46iSLr11doNe9iVbY4qqASaOlHkVoP6_GiNxU9UpIJeA/s400/Screen+Shot+2017-11-25+at+8.34.50+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. Noisy Input Image</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: left;">
To my big surprise this trained DAE was able to decode letter 'Y' on the top row of the image. The reconstructed image is shown below in Figure 2. To put this in perspective, how often can you totally eliminate the noise just by turning a knob in your radio? This reconstruction is very clear with a small exception that timing of last 'dah' in letter 'Y' is a bit shorter than in the original training image. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9yWq4pE-eDnLwr3_ld5KNO-cmmVlxHJQVa0oxlX52bxGRA48BlRPYFwnO6HbodZ0X2TrvUvtItUOZMhTgS29fUp2Q7lA2HQPa4fxcF0VAu7J-C5IXYfJUnUxyWzmvMGUFwwbq3mGccRc/s1600/Screen+Shot+2017-11-25+at+8.35.02+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="510" data-original-width="574" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg9yWq4pE-eDnLwr3_ld5KNO-cmmVlxHJQVa0oxlX52bxGRA48BlRPYFwnO6HbodZ0X2TrvUvtItUOZMhTgS29fUp2Q7lA2HQPa4fxcF0VAu7J-C5IXYfJUnUxyWzmvMGUFwwbq3mGccRc/s400/Screen+Shot+2017-11-25+at+8.35.02+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 2. Reconstructed Out Image </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
For reference, below is original image of letter 'Y' that was used in the training phase. </div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgohKBNv4W1-6zPimEtLbDPJU_54gPmR7phJJTFagcUT8hk1t4Q2w_ElOz0qinXVL9o1MunIgNTDAep-CRT0ucmRa4KdLGYFIE4SMQa_DAxGtHQVnCn9-DPNY1beP67x9RA_t3IszX-t5c/s1600/Screen+Shot+2017-11-25+at+8.34.33+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="505" data-original-width="546" height="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgohKBNv4W1-6zPimEtLbDPJU_54gPmR7phJJTFagcUT8hk1t4Q2w_ElOz0qinXVL9o1MunIgNTDAep-CRT0ucmRa4KdLGYFIE4SMQa_DAxGtHQVnCn9-DPNY1beP67x9RA_t3IszX-t5c/s400/Screen+Shot+2017-11-25+at+8.34.33+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 3. Original image used for training </td></tr>
</tbody></table>
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<div>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
Experiment Details</h3>
As a starting point I used Tensorflow tutorials using Jupyter Notebooks, in particular <a href="https://github.com/sjchoi86/tensorflow-101/blob/master/notebooks/dae_mnist_dropout.ipynb" target="_blank">this excellent de-noising autoencoder</a> example that uses MNIST database as the data source. The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning. The MNIST database contains 60,000 training images and 10,000 testing images. Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrUgy6vUwplxt8Z_DxFMiAMPWqjV_qXkY9qQM6M0ZxdxppkynU0453ZltCZoBciYiWaL6wKtk4nFx6cgxJf_ozeTPb03KQShWmT_0QpICyzOw58tltt5TuQ0aApLlt7sQ2ykgvseGP7n8/s1600/training.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1600" data-original-width="198" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrUgy6vUwplxt8Z_DxFMiAMPWqjV_qXkY9qQM6M0ZxdxppkynU0453ZltCZoBciYiWaL6wKtk4nFx6cgxJf_ozeTPb03KQShWmT_0QpICyzOw58tltt5TuQ0aApLlt7sQ2ykgvseGP7n8/s640/training.png" width="76" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 4. Morse images</td></tr>
</tbody></table>
I created a simple Python script that generates a Morse code dataset in MNIST format using a text file as the input data. To keep things simple I kept the MNIST image size (28 x 28 pixels) and just 'painted' morse code as white pixels on the canvas. These images look a bit like waterfall display in modern SDR receivers or software like CW skimmer. I created all together 55,000 training images, 5000 validation images and 10,000 testing images.<br />
<br />
To validate that these images look OK I plotted first ten characters "BB 2BQA}VA" from the random text file I used for training. Each image is 28x28 pixels in size so even the longest Morse character will easily fit on this image. Right now all Morse characters start from top left corner but it would be easy to generate more randomness in the starting point and even length (or speed) of these characters. <br />
<br />
In fact the original MNIST images have a lot of variability in the handwritten digits and some are difficult even for humans to classify correctly. In MNIST case you have only ten classes to choose from (numbers 0,1,2,3,4,5,6,7,8,9) but in Morse code I had 60 classes as I wanted to include also special characters in the training material.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqCKVIsQa1zLis8ItnwFhLV2b6CXOGLLvv_RmXIgZZUXQPBdYs1C9Vyyr_Q50YGwPQVb6LSXI6my0rfpc0ljAjWRqq35Qr67WacRj_9qyBcGy_Vbpt-Qm4_Zv_-Lmd-nhtBWCewjNUslU/s1600/mnist.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="252" data-original-width="87" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgqCKVIsQa1zLis8ItnwFhLV2b6CXOGLLvv_RmXIgZZUXQPBdYs1C9Vyyr_Q50YGwPQVb6LSXI6my0rfpc0ljAjWRqq35Qr67WacRj_9qyBcGy_Vbpt-Qm4_Zv_-Lmd-nhtBWCewjNUslU/s1600/mnist.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 5. MNIST images</td></tr>
</tbody></table>
<br />
Figure 4. shows the Morse example images and Figure 5. shows the MNIST example handwritten images.<br />
<br />
When training DAE network I added modest amount of gaussian noise to these training images. See example on figure 6. It is quite surprising that the DAE network is still able to decode correct answers with three times more noise added on the test images.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKLVAr-hWzv8sFA0rj1ochvsX-6QhvV8oIVsnBAnVRhr01uduPt8PSvksBwNhCSv3t_Vj9q_H_4ZFUDLFg8LSAk9mHW8-Llpl4IhllTraoW3oeEG9M99OuGBDCw2uLHbt-2OhLN42WuI8/s1600/noisy.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="244" data-original-width="266" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKLVAr-hWzv8sFA0rj1ochvsX-6QhvV8oIVsnBAnVRhr01uduPt8PSvksBwNhCSv3t_Vj9q_H_4ZFUDLFg8LSAk9mHW8-Llpl4IhllTraoW3oeEG9M99OuGBDCw2uLHbt-2OhLN42WuI8/s1600/noisy.png" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 6. Noise added to training input image</td></tr>
</tbody></table>
<br />
<br />
<br /></div>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br /></div>
<h3>
Network model and functions</h3>
<div>
A typical feature in auto-encoders is to have hidden layers that have less features than the input or output layers. The network is forced to learn a ”compressed” representation of the input. If the input were completely random then this compression task would be very difficult. But if there is structure in the data, for example, if some of the input features are correlated, then this algorithm will be able to discover some of those correlations.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Network Parameters</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_input = 784 # MNIST data input (img shape: 28*28)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_hidden_1 = 256 # 1st layer num features</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_hidden_2 = 256 # 2nd layer num features</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_output = 784 # </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">with tf.device(device2use):</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # tf Graph input</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> x = tf.placeholder("float", [None, n_input])</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> y = tf.placeholder("float", [None, n_output])</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dropout_keep_prob = tf.placeholder("float")</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # Store layers weight & bias</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> weights = {</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'out': tf.Variable(tf.random_normal([n_hidden_2, n_output]))</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> biases = {</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'b1': tf.Variable(tf.random_normal([n_hidden_1])),</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'b2': tf.Variable(tf.random_normal([n_hidden_2])),</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'out': tf.Variable(tf.random_normal([n_output]))</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span></div>
</div>
<br />
The functions for this neural network are below. The cost function calculates the mean square of the difference of output and training images.<br />
<br />
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">with tf.device(device2use):</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # MODEL</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> out = denoising_autoencoder(x, weights, biases, dropout_keep_prob)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # DEFINE LOSS AND OPTIMIZER</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> cost = tf.reduce_mean(tf.pow(out-y, 2))</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> optimizer = tf.train.AdamOptimizer(learning_rate=0.01).minimize(cost) </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # INITIALIZE</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> init = tf.initialize_all_variables()</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # SAVER</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> savedir = "nets/"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><span style="font-size: x-small;"> saver = tf.train.Saver(max_to_keep=3)</span> </span></div>
</div>
<h3>
Model Training </h3>
<div>
I used the following parameters for training the model. Training took 1780 seconds on a Macbook Pro laptop. The cost curve of training process is shown in Figure 6. </div>
<div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">training_epochs = 300</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">batch_size = 1000</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">display_step = 5</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">plot_step = 10</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWiKIHAaMAjFexKN93-SGNuy_QpY-L4guZQkwfEUpHV2usosaIbIBzqiHODAn8kR1GwBFUrZZTyTyDr2hBHmWLI-QRBAtw2a5_VcOgIBcgQuY_C3mmwmtTrtCFENlBdiuCA47zpcp1wwY/s1600/Screen+Shot+2017-11-25+at+8.35.16+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="642" data-original-width="824" height="311" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgWiKIHAaMAjFexKN93-SGNuy_QpY-L4guZQkwfEUpHV2usosaIbIBzqiHODAn8kR1GwBFUrZZTyTyDr2hBHmWLI-QRBAtw2a5_VcOgIBcgQuY_C3mmwmtTrtCFENlBdiuCA47zpcp1wwY/s400/Screen+Shot+2017-11-25+at+8.35.16+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 6. Cost curve</td></tr>
</tbody></table>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><br />
<div style="text-align: left;">
It is interesting to observe what is happening to the weights. Figure 7 shows the first hidden layer "h1" weights after training is completed. Each of these blocks have learned some internal representation of the Morse characters. You can also see the noise that was present in the training data.<br />
<br /></div>
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWKFkHyY60HOo_f3U3UJR604iCwFoImxP-sLDVuWuGR_kDmub_VRKyETo5KXEsSYSdMbyBpa9k-AVS4DYNxm6tE_uMa2CJ_fW4oH9BzAUP8RTFYsZa3o9BLEtwqV7j1_jOCNwpKR65Lw8/s1600/Screen+Shot+2017-11-25+at+8.35.35+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="610" data-original-width="642" height="380" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWKFkHyY60HOo_f3U3UJR604iCwFoImxP-sLDVuWuGR_kDmub_VRKyETo5KXEsSYSdMbyBpa9k-AVS4DYNxm6tE_uMa2CJ_fW4oH9BzAUP8RTFYsZa3o9BLEtwqV7j1_jOCNwpKR65Lw8/s400/Screen+Shot+2017-11-25+at+8.35.35+PM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 7. Filter shape for "h1" weights</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWKFkHyY60HOo_f3U3UJR604iCwFoImxP-sLDVuWuGR_kDmub_VRKyETo5KXEsSYSdMbyBpa9k-AVS4DYNxm6tE_uMa2CJ_fW4oH9BzAUP8RTFYsZa3o9BLEtwqV7j1_jOCNwpKR65Lw8/s1600/Screen+Shot+2017-11-25+at+8.35.35+PM.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"></a><br /></div>
<h3>
Software</h3>
<a href="https://github.com/ag1le/LSTM_morse/blob/master/MNIST-MORSE-DE_NOISING_DECODER-ENCODER.ipynb" target="_blank">The Jupyter Notebook source code</a> of this experiment has been posted to Github. Many thanks to the original contributors of <a href="https://github.com/sjchoi86/tensorflow-101/blob/master/notebooks/dae_mnist_dropout.ipynb" target="_blank">this</a> and other Tensorflow tutorials. Without them this experiment would not have been possible.<br />
<div>
<br /></div>
<h3>
Conclusions
</h3>
<div>
This experiment demonstrates that de-noising auto-encoders could have many potential use cases for ham radio experiments. While I used MNIST format (28x28 pixel images) in this experiment, it is quite feasible to use other kinds of data, such as audio WAV files, SSTV images or some data from other digital modes commonly used by ham radio amateurs. </div>
<div>
<br /></div>
<div>
If your data has a clear structure that will have noise added and distorted during a radio transmission, it would be quite feasible to experiment implementing a de-noising auto-encoder to restore near original quality. It is just a matter of re-configuring the DAE network and re-training the neural network.</div>
<div>
<br /></div>
<div>
If this article sparked your interest in de-noising auto-encoders please let me know. Machine Learning algorithms are rapidly being deployed in many data intensive applications. I think it is time for ham radio amateurs to start experimenting with this technology as well. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
73 </div>
<div>
Mauri AG1LE </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-90884900715029273362017-11-05T21:18:00.002-05:002017-11-06T12:07:06.640-05:00<h3 class="post-title entry-title" itemprop="name" style="background-color: white; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 22px; font-stretch: normal; font-weight: normal; line-height: normal; margin: 0.75em 0px 0px; position: relative;">
TensorFlow revisited: a new LSTM Dynamic RNN based Morse decoder</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
It has been almost two years since I was playing with <a href="http://ag1le.blogspot.com/2015/12/tensorflow-new-lstm-rnn-based-morse.html" target="_blank"><span style="color: blue;">TensorFlow based Morse decoder</span></a>. This is a long time in the rapidly moving Machine Learning field.</div>
<div>
<br /></div>
<div>
I created a new version of the LSTM Dynamic RNN based Morse decoder using TensorFlow package and <a href="https://github.com/aymericdamien/TensorFlow-Examples/blob/master/notebooks/3_NeuralNetworks/dynamic_rnn.ipynb" target="_blank">Aymeric Damien's example</a>. This version is much faster and has also ability to train/decode on variable length sequences. The training and testing sets are generated from sample text files on the fly, I included the Python library and the new TensorFlow code in <a href="https://github.com/ag1le/LSTM_morse" target="_blank"><span style="color: blue;">my Github page</span></a>. </div>
<div>
<br /></div>
<div>
The demo has ability to train and test using datasets with noise embedded. Fig 1. shows the 50 first test vectors with gaussian noise added. Each vector is padded to 32 values. Unlike the previous version of LSTM network this new version has ability to train variable length sequences. The Morse class handles the generation of training vectors based on input text file that contains randomized text. </div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0WS1kSuMNgW8MaOOx4xr-KN038LPhpSgbCTcI1eEpkBL1ryYIiIwKxkHQpbQRj7BLIPnLQB8YDIwwwux84I-_TtHQT-xe_4lIEmK1ObT45jeVsioCNCW45nJtqCfYooQ18OwhJYza08w/s1600/morse_encoding.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="578" data-original-width="385" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0WS1kSuMNgW8MaOOx4xr-KN038LPhpSgbCTcI1eEpkBL1ryYIiIwKxkHQpbQRj7BLIPnLQB8YDIwwwux84I-_TtHQT-xe_4lIEmK1ObT45jeVsioCNCW45nJtqCfYooQ18OwhJYza08w/s400/morse_encoding.png" width="266" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. <span style="font-size: xx-small;">"NOW 20 WPM TEXT IS FROM JANUARY 2015 QST PAGE 56 " </span></td></tr>
</tbody></table>
<br />
<span style="font-size: xx-small;"><br /></span>
<br />
<div class="separator" style="clear: both; text-align: center;">
<span style="font-family: inherit;"><br /></span></div>
<div>
<div>
<span style="font-family: inherit;">Below are the TensorFlow model and network parameters I used for this experiment: </span></div>
<div>
<span style="font-family: inherit;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># MODEL Parameters</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">learning_rate = 0.01</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">training_steps = 5000</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">batch_size = 512</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">display_step = 100</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_samples = 10000 </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># NETWORK Parameters</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">seq_max_len = 32 # Sequence max length</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_hidden = 64 # Hidden layer num of features </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">n_classes = 60 # Each morse character is a separate class</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<span style="font-family: inherit;">Fig 2. shows the training loss and accuracy by minibatch. This training took <span style="background-color: white; white-space: pre-wrap;">446.9 seconds and final </span><span style="background-color: white; white-space: pre-wrap;">testing accuracy reached was 0.9988. </span> This training session was done without any noise in the training dataset. </span><br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmtLDgKrelKGJpr9EvTCOFpOPalrDCbE2rTvQJkpMva__WLrsahmXGtOmfQthFIQVF0PbYKyz1jJMD2uIS3M5QFYBGy-8ydvAuA8LS82pBku0H_DZDeul_0Asvi_82YF4NqI9enbOJju0/s1600/training.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="288" data-original-width="432" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjmtLDgKrelKGJpr9EvTCOFpOPalrDCbE2rTvQJkpMva__WLrsahmXGtOmfQthFIQVF0PbYKyz1jJMD2uIS3M5QFYBGy-8ydvAuA8LS82pBku0H_DZDeul_0Asvi_82YF4NqI9enbOJju0/s320/training.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 2. Training Loss and Accuracy plot.</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
Sample session to use the trained model is below: </div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"># ================================================================</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"># Use saved model to predict characters from Morse sequence data</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"># ================================================================</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"># </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">NOISE = False</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">saver = tf.train.Saver()</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">testset = Morse(n_samples=10000, max_seq_len=seq_max_len,filename='arrl2.txt')</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">test_data = testset.data</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">if (NOISE): </span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> test_data = test_data + normal(0.,0.1, 32*10000).reshape(10000,32,1)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">test_label = testset.labels</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">test_seqlen = testset.seqlen</span></div>
<div>
</div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"># Later, launch the model, use the saver to restore variables from disk, and</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"># do some work with the model.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">with tf.Session() as sess:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> # Restore variables from disk.</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> saver.restore(sess, "/tmp/morse_model.ckpt")</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> print("Model restored.")</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> y_hat = tf.argmax(pred,1)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> ch = sess.run(y_hat, feed_dict={x: test_data, y: test_label,seqlen: test_seqlen})</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> s = ''</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> for c in ch:</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> s += testset.decode(c)</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"> print( s)</span></div>
</div>
<div>
<br /></div>
<div>
Here is the output from the decoder (this is using arrl2.txt file as input): </div>
<div>
<br /></div>
<div>
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">INFO:tensorflow:Restoring parameters from /tmp/morse_model.ckpt
Model restored.
NOW 20 WPM TEXT IS FROM JANUARY 2015 QST PAGE 56 SITUATIONS WHERE I COULD HAVE BROUGHT A DIRECTIONAL ANTENNA WITH ME, SUCHAS A SMALL YAGI FOR HF OR VHF. IF ITS LIGHT ENOUGH, ROTATING A YAGI CAN BEDONE WITH THE ARMSTRONG METHOD, BUT IT IS OFTEN VERY INCONVENIENT TO DO SO.PERHAPS YOU DONT WANT TO LEAVE THE RIG BEHIND WHILE YOU GO OUTSIDE TO ADJUST THE ANTENNA TOWARD THAT WEAK STATION, OR PERHAPS YOU'RE IN A TENT AND ITS DARK OUT THERE. A BATTERY POWERED ROTATOR PORTABLE ROTATION HAS DEVELOPED A SOLUTION TO THESE PROBLEMS. THE 12PR1A IS AN ANTENNA ROTATOR FIGURE 6 THAT FUNCTIONS ON 9 TO 14 V DC. AT 12 V, THE UNIT IS SPECIFIED TO DRAW 40 MA IDLE CURRENT AND 200 MA OR LESS WHILE THE ANTENNA IS TURNING. IT CAN BE POWERED FROM THE BATTERY USED TO RUN A TYPICAL PORTABLE STATION.WHILE THE CONTROL HEAD FIGURE 7 WILL FUNCTION WITH AS LITTLE AS 6 V, A END OF 20 WPM TEXT QST DE AG1LE NOW 20 WPM TEXT IS FROM JANUARY 2014 QST PAGE 46 TRANSMITTER MANUALS SPECIFI</span></pre>
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">
</span></pre>
</div>
<div>
As the reader can observe the LSTM network has learned near perfectly to translate incoming Morse sequences to text. </div>
<div>
<br /></div>
<div>
Next I did set the NOISE variable to True. Here is the decoded message with noise: </div>
<div>
<br /></div>
<div>
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">NOW J0 O~M TEXT IS LRZM JANUSRQ 2015 QST PAGE 56 SITRATIONS WHEUE I XOULD HAVE BRYUGHT A DIRECTIZNAF ANTENNS WITH ME{ SUYHSS A SMALL YAGI FYR HF OU VHV' IV ITS LIGHT ENOUGH, UOTSTING A YAGI CAN BEDONE FITH THE ARMSTRONG METHOD8 LUT IT IS OFTEN VERQ INOGN5ENIENT TC DG SC.~ERHAPS YOR DZNT WINT TO LEAVE THE RIK DEHIND WHILE YOU KO OUTSIME TO ADJUST THE AATENNA TYOARD THNT WEAK STTTION0 OU ~ERHAPS COU'UE IN A TENT AND ITS MARK OUT THERE. S BATTERC JYWERED RCTATOR ~ORTALLE ROTATION HAS DEVELOOED A SKLUTION TO THESE ~UOBLEMS. THE 1.JU.A IS AN ANTENNA RYTATCR FIGURE 6 THAT FRACTIZNS ZN ) TO 14 V DC1 AT 12 W{ THE UNIT IS SPECIFIED TO DRSW }8 MA IDLE CURRENT AND 20' MA OR LESS WHILE THE ANTENNA IS TURNING. IT ZAN BE POOERED FROM THE BATTEUY USED TO RRN A T}~IXAL CQMTUBLE STATION_WHILE IHE }ZNTROA HEAD FIGURE 7 WILA WUNXTION WITH AS FITTLE AA 6 F8 N END ZF 2, WPM TEXT OST ME AG1LE NOW 20 W~M TEXT IS LROM JTNUARJ 201} QST ~AGE 45 TRANSMITTER MANUALS S~ECILI</span></pre>
</div>
<div>
<br /></div>
<span style="font-family: inherit;">Interestingly this text is still quite readable despite noisy signals. The model seems to mis-decode some dits and dahs but the word structure is still visible. </span><br />
<span style="font-family: inherit;"><br /></span>
As a next step I re-trained the network using the same amount of noise in the training dataset. I expected the loss and accuracy to be worse. Fig 3. shows that training accuracy to 0.89338 took much longer and maximum testing accuracy was only 0.9837.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUJLWAM1TtWSAt5e-pU7K8xHnJv7aXOhyphenhyphenqJlsBLjv9-2sJ8Df33xGqKxWWtatroutIYWevVdVH53e96fkMlLl0P3noXRwJG-5tqq5M42lYR1PZ2BsXaOIcyvwlAAgYzC5Gmwq4BHZgcGA/s1600/training.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" data-original-height="288" data-original-width="432" height="213" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUJLWAM1TtWSAt5e-pU7K8xHnJv7aXOhyphenhyphenqJlsBLjv9-2sJ8Df33xGqKxWWtatroutIYWevVdVH53e96fkMlLl0P3noXRwJG-5tqq5M42lYR1PZ2BsXaOIcyvwlAAgYzC5Gmwq4BHZgcGA/s320/training.png" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig. 3 Training Loss and Accuracy with noisy dataset</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
With the new model trained using noisy data I did re-run the testing phase. Here is the decoded message with noise:<br />
<br />
<pre style="background-color: white; border-radius: 0px; border: 0px; box-sizing: border-box; line-height: inherit; overflow: auto; padding: 0px; vertical-align: baseline; white-space: pre-wrap; word-break: break-all; word-wrap: break-word;"><span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">NOW 20 WPM TEXT IS FROM JANUARY 2015 QST PAGE 56 SITUATIONS WHERE I COULD HAWE BROUGHT A DIRECTIONAL ANTENNA WITH ME0 SUCHAS A SMALL YAGI FOR HF OR VHF1 IF ITS LIGHT ENOUGH0 ROTATING A YAGI CAN BEDONE WITH THE ARMSTRONG METHOD0 BUT IT IS OFTEN VERY INCONVENIENT TO DO SO1PERHAPS YOU DONT WANT TO LEAVE THE RIG BEHIND WHILE YOU GO OUTSIDE TO ADJUST THE ANTENNA TOWARD THAT WEAK STATION0 OR PERHAPS YOU1RE IN A TENT AND ITS DARK OUT THERE1 A BATTERY POWERED ROTATOR PORTABLE ROTATION HAS DEVELOPED A SOLUTION TO THESE PROBLEMS1 THE 12PR1A IS AN ANTENNA ROTATOR FIGURE 6 THAT FUNCTIONS ON 9 TO 14 V DC1 AT 12 V0 THE UNIT IS SPECIFIED TO DRAW 40 MA IDLE CURRENT AND 200 MA OR LESS WHILE THE ANTENNA IS TURNING1 IT CAN BE POWERED FROM THE BATTERY USED TO RUN A TYPICAL PORTABLE STATION1WHILE THE CONTROL HEAD FIGURE Q WILL FUNCTION WITH AS LITTLE AS X V0 A END OF 20 WPM TEXT QST DE AG1LE NOW 20 WPM TEXT IS FROM JANUARY 2014 QST PAGE 46 TRANSMITTER MANUALS SPECIFI</span></pre>
<br />
As reader can observe now we have nearly perfect copy from noisy testing data. The LSTM network has gained ability to pick-up the signals from noise. Note that training data and testing data are two completely separate datasets.<br />
<h3>
CONCLUSIONS</h3>
<div>
Recurrent Neural Networks have gained a lot of momentum over the last 2 years. LSTM type networks are used in machine learning systems, like Google Translate, that can translate one sequence of characters to another language efficiently and accurately. </div>
<div>
<br /></div>
<div>
This experiment shows that a relatively small TensorFlow based neural network can learn Morse code sequences and translate them to text. This experiment shows also that adding noise to the training data will slow down the learning rate and will impact overall training accuracy achieved. However, applying similar noise level in the testing phase will significantly improve the testing accuracy when using a model trained under noisy training signals. The network has learned the signal distribution and is able to decode more accurately. </div>
<div>
<br /></div>
<div>
So what are the practical implications of this work? With some signal pre-processing LSTM RNN could provide a self learning Morse decoder that only needs a set of labeled audio files to learn a particular set of sequences. With large enough training dataset the model could achieve over 95% accuracy.</div>
<div>
<br /></div>
<div>
73 de AG1LE </div>
<div>
Mauri </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com2tag:blogger.com,1999:blog-3326773214329183284.post-13357847936628504492017-04-01T10:52:00.000-04:002017-04-01T21:34:53.972-04:00President Trump's "America First Energy Plan" Secrets Leaked: Quake Field Generator<div>
April 1st, 2017 Lexington, Massachusetts
</div>
<div>
<br /></div>
<div>
As President Trump has stated publicly many times, a sound energy policy begins with the recognition that we have vast untapped domestic energy reserves right here in America. Unfortunately, the secret details behind the ambitious <a href="https://www.whitehouse.gov/america-first-energy">America First Energy Plan</a> were leaked late last night.
</div>
<div>
<br /></div>
<div>
To pre-empt any fake news by the Liberal Media I am making a full disclosure of the secret project I have been working on the last 18 months in propinquity of MIT Lincoln Laboratory, a federally funded research and development center chartered to apply advanced technology to problems of national security.
</div>
<div>
<br /></div>
<div>
I am unveiling a breakthrough technology that will lower energy costs for hardworking Americans and maximize the use of American resources, freeing us from dependence on foreign oil. This technology allows harvesting clean energy from around the world and making other nations to pay for it according to President Trump's master plan.
</div>
<div>
<br /></div>
<div>
The technology is based on quake fields and provides virtually unlimited free energy, while protecting clean air and clean water, conserving our natural habitats, and preserving our natural reserves and resources.
</div>
<div>
<br /></div>
<h3>
<b>What is Quake Field?</b></h3>
<div>
<br /></div>
<div>
Quake field theory is relatively unknown part of seismology. Seismology is the scientific study of earthquakes and the propagation of elastic waves through the Earth or through other planet-like bodies. The field also includes studies of earthquake environmental effects such as tsunamis as well as diverse seismic sources such as volcanic, tectonic, oceanic, atmospheric, and artificial processes such as explosions.
</div>
<div>
<br /></div>
<div>
Quake field theory was formulated by Dr. James von Hausen in 1945 as part of the Manhattan project during World War II. Quake field theory provides a mathematical model how energy propagates through elastic waves. During the development of the first nuclear weapons scientists faced a big problem: nobody was able to provide an accurate estimate of the energy yield of the first atom bomb. People were concerned possible side effects and there was speculation that fission reaction could ignite the Earth atmosphere.
</div>
<div>
<br /></div>
<div>
Quake field theory provides precise field formulas to calculate energy propagation in planet-like bodies. The theory has been proven in hundreds of nuclear weapon tests during the Cold War period. However, most of the empirical research and scientific papers have been classified by the U.S. Government and therefore you cannot really find details in Wikipedia or other public sources due to the sensitivity of the information.</div>
<div>
<br /></div>
<div>
In the recent years U.S. seismologists have started to use quake field theory to calculate the amount of energy released in earthquakes. This work was enabled by creation of global network of seismic sensors that is now available. These sensors provide real time information on earthquakes over the Internet.
</div>
<div>
<br /></div>
<div>
I have a <a href="http://www.raspberryshake.org/" target="_blank">Raspberry Shake</a> at home. This is a Raspberry Pi powered device to monitor quake field activity and part of a global seismic sensor network. Figure 1 show quake field activity on March 25, 2017. As you can see it was a very active day. This system gives me a prediction when the quake field is activated. </div>
<div>
<br /></div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG8sMSIRKiFZQU3Rrvo1IhzjZA-bODoGewnl5-kkzc7_ypkpsbUR1RPmo7O9os5n42_WzygGG0zo2ow9G4sr7HaagNvV2H0EEbQ_VriSb3l6BhnJZieEQHFtC5hn2Un2-ToMdkuDh6uWg/s1600/R8C5B_SHZ_AM_00.2017032512.gif" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgG8sMSIRKiFZQU3Rrvo1IhzjZA-bODoGewnl5-kkzc7_ypkpsbUR1RPmo7O9os5n42_WzygGG0zo2ow9G4sr7HaagNvV2H0EEbQ_VriSb3l6BhnJZieEQHFtC5hn2Un2-ToMdkuDh6uWg/s640/R8C5B_SHZ_AM_00.2017032512.gif" width="579" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. Quake Field activity in Lexington, MA</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<h3>
<b>How much energy is available from Quake Field?</b></h3>
<div>
<br /></div>
<div>
A single magnitude 9 earthquake releases approximately 3.9 e+22 Joules of seismic moment energy (Mo). Much of this energy gets dissipated at the epicenter but approximately 1.99 e+18 Joules is radiated as seismic waves through the planet. To put this in perspective you could power the whole United States for 7.1 days with this radiated energy. This radiated energy equals to 15,115 million gallons of gasoline - just from a single large earthquake.
</div>
<div>
<br /></div>
<div>
The radiated energy is released as waves from the epicenter of a major earthquake and propagate outward as surface waves (S waves). In the case of compressional waves (P waves), the energy radiates from the focus under the epicenter and travels all the way through the globe. Figure 2 illustrates these two primary energy transfer mechanisms. Note that we don’t need to build any transmission network to transfer this energy so the capital cost would be very small. </div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div>
<br /></div>
<div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://earth.rice.edu/mtpe/geo/geosphere/hot/earthquakes/propagation.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://earth.rice.edu/mtpe/geo/geosphere/hot/earthquakes/propagation.jpg" height="378" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. Energy Transfer by Radiated Waves</td></tr>
</tbody></table>
<br /></div>
<div>
<br /></div>
<div>
Magnitude 2 and smaller earthquakes occur several hundred times a day world wide. Major earthquakes, greater than magnitude 7, happen more than once per month. “Great earthquakes”, magnitude 8 and higher, occur about once a year. <br />
<br />
The real challenge has been that we don’t have a technology harvest this huge untapped energy - until today.
</div>
<div>
<br /></div>
<div>
<h3>
<b>Introducing Quake Field Generator</b></h3>
</div>
<div>
<br /></div>
<div>
The following introduction explains the operating principles of quake field generator (QFG) technology. <br />
<br /></div>
<div>
Using the quake field theory and the seismic sensor data it is now possible to predict accurately when the S and P waves arrive to any location on Earth. The big problem has been to find efficient method how to convert the energy of these waves to electricity.
</div>
<div>
<br /></div>
<a href="https://en.wikipedia.org/wiki/Nanogenerator" target="_blank">A triboelectric nanogenerator</a> (TENG) is an energy harvesting device that converts the external mechanical energy into electricity by a conjunction of triboelectric effect and electrostatic induction.<br />
<br />
Ever since the first report of the TENG in January 2012, the output power density of TENG has been improved for five orders of magnitude within 12 months. The area power density reaches 313 W/m2, volume density reaches 490 kW/m3, and a conversion efficiency of ~60% has been demonstrated. Besides the unprecedented output performance, this new energy technology also has a number of other advantages, such as low cost in manufacturing and fabrication, excellent robustness and reliability, environmental-friendly, and so on.<br />
<br />
<div>
The Liberal Media outlets have totally misunderstood the "clean coal technology” that is the cornerstone of President Trump's master plan for energy independence. Graphene is coal, just in different molecular configuration. Graphene is one of materials exhibiting strong triboelectric effect. With recent advances in 3D printing technology it is now feasible to mass produce low cost triboelectric nanogenerators<span style="font-size: 13px;">.</span> Graphene is now commercially available for most 3D printers.
</div>
<div>
<br /></div>
<div>
The geometry of Quake Field Generator is based on fractals, minimizing the size of resonant transducer. My prototype consists of 10,000 TENG elements organized into a fractal shape. In this prototype version that I have been working on the last 18 months I have also implemented an automated tuning circuit that uses flux capacitors to maximize the energy capture at the resonance frequency. This brings the efficiency of the QFG to 97.8% - I am quite pleased with this latest design.<br />
<br /></div>
<div>
Figure 3. show my current Quake Field Generator prototype - this is a 10 kW version. It has four stacks of TENG elements. Due to the high efficiency of these elements the ventilation need is quite minimal.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-m6aIzEKT0Xt-hMuNlMLuleLDVzJwT_QjnAgWRrTZG8wIiA4dQJwlkEUsYMbgWk585bSygw_KL0gsicC4J0Au3SgR75yGh8i4blDoWpTPjYWO7fAr3i4-k6IgR94ndXtHM4U0sf9lIJc/s1600/QFG.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi-m6aIzEKT0Xt-hMuNlMLuleLDVzJwT_QjnAgWRrTZG8wIiA4dQJwlkEUsYMbgWk585bSygw_KL0gsicC4J0Au3SgR75yGh8i4blDoWpTPjYWO7fAr3i4-k6IgR94ndXtHM4U0sf9lIJc/s400/QFG.png" width="358" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3. Quake Field Generator prototype - 10 kW version</td></tr>
</tbody></table>
<br /></div>
<div>
<h3>
<span style="font-weight: bold;">So what does this news mean to an average American?</span></h3>
</div>
<div>
Quake Field Generator will be fully open source technology that will create millions of new jobs in the U.S. energy market. It leverages our domestic coal sources to build TENG devices from graphene (aka “clean coal”). </div>
<div>
<br /></div>
<div>
A simple 10 kW generator can be 3D printed in one day and it can be mounted next to your power distribution panel at your home. The only requirements are that the unit must have connection to ground to harvest the quake field energy and you need to use a professional electrician to make a connection to your home circuit.
</div>
<div>
<br /></div>
<div>
I have been running such a DYI 10 kW generator for over a year. So far I have been very happy with the performance of this Quake Field Generator. Once I finalize the design my plan is to publish the software, circuit design, transducer STL files etc. on Github. <br />
<br />
Let me know if you are interested in QFG technology - happy April 1st. </div>
<div>
<br /></div>
<div>
73
</div>
<!--?xml version="1.0" encoding="UTF-8"?-->
<br />
<div>
Mauri </div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-19232937735393438502017-01-29T18:00:00.001-05:002017-01-29T18:18:15.911-05:00Amazon Echo - Alexa skills for ham radio <h2>
</h2>
<div>
Demo video showing a proof of concept Alexa DX Cluster skill with remote control of Elecraft KX3 radio. </div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/Xh6jVeaIQvM/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/Xh6jVeaIQvM?feature=player_embedded" width="320"></iframe></div>
<div class="separator" style="clear: both; text-align: left;">
<br /></div>
<div>
<br /></div>
<h2>
Introduction</h2>
According to a Wikipedia article Amazon Echo is a smart speaker developed by Amazon. The device consists of a 9.25-inch (23.5 cm) tall cylinder speaker with a seven-piece microphone array. The device connects to the voice-controlled intelligent personal assistant service Alexa, which responds to the name "Alexa". The device is capable of voice interaction, music playback, making to-do lists, setting alarms, streaming podcasts, playing audiobooks, and providing weather, traffic and other real time information. It can also control several smart devices using itself as a home automation hub.<br />
<br />
Echo also has access to skills built with the Alexa Skills Kit. These are 3rd-party developed voice experiences that add to the capabilities of any Alexa-enabled device (such as the Echo). Examples of skills include the ability to play music, answer general questions, set an alarm, order a pizza, get an Uber, and more. Skills are continuously being added to increase the capabilities available to the user.<br />
<br />
The Alexa Skills Kit is a collection of self-service APIs, tools, documentation and code samples that make it fast and easy for any developer to add skills to Alexa. Developers can also use the "Smart Home Skill API", a new addition to the Alexa Skills Kit, to easily teach Alexa how to control cloud-controlled lighting and thermostat devices. A developer can follow tutorials to learn how to quickly build voice experiences for their new and existing applications.<br />
<h2>
</h2>
<h2>
Ham Radio Use Cases </h2>
For ham radio purposes Amazon Echo and Alexa service creates a whole new set of opportunities to automate your station and build new audio experiences.<br />
<br />
Here is a list of ideas what you could use Amazon echo for:<br />
<br />
- listen ARRL Podcasts<br />
- practice Morse code or ham radio examination<br />
- check space weather and radio propagation forecasts<br />
- memorize Q codes (QSL, QTH, etc.)<br />
- check call sign details from QRZ.com<br />
- use APRS to locate a mobile ham radio station <br />
<br />
<br />
I started experimenting with Alexa Skills APIs using mostly Python to create programs. One of the ideas I had was to get Alexa to control my Elecraft KX3 radio remotely. To make the skill more useful I build some software to pull latest list of spots from DX Cluster and use those to set the radio on the spotted frequency to listen some new station or country on my bucket list.<br />
<h2>
<br />Alexa Skill Description</h2>
Imagine if you could use and listen your radio station anywhere just by saying the magic words "Alexa, ask DX Cluster to list spots."<br />
<br />
<br />
<br />
Alexa would then go to a DX Cluster, find the latest spots on SSB (or CW) and allows you to select the spot you want to follow. By just saying "Select seven" Alexa would set your radio to that frequency and start playing the audio.<br />
<br />
<div style="text-align: left;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAK38M5S2DpumrlJHNmyFXnGAYlDkOlq4sHyflB9n5M4Z4AP6ZVu8xaQITfDL9f_ztpGPEka_3LiBnvLqe1WihwuAKgfPbbfyBQztkMOSzIHjG4fsgjwQl4ubKnbln8Ci69s-hX0Utzrc/s1600/List_of_Cluster_spots.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="311" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgAK38M5S2DpumrlJHNmyFXnGAYlDkOlq4sHyflB9n5M4Z4AP6ZVu8xaQITfDL9f_ztpGPEka_3LiBnvLqe1WihwuAKgfPbbfyBQztkMOSzIHjG4fsgjwQl4ubKnbln8Ci69s-hX0Utzrc/s400/List_of_Cluster_spots.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. Alexa DX Cluster Skill output </td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<h2>
System Architecture </h2>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Figure 3. below shows all the main components of this solution. I have a Thinkpad X301 laptop connected to Elecraft KX3 radio with KXUSB serial port and using built-in audio interface. X301 is running several processes: one for recording the audio into MP3 files, hamlib rigctld to control the the radio and a web server that allows Alexa skill to control the frequency and retrieve the recorded MP3 files.<br />
<br />
I implemented the Alexa Skill "DX Cluster" using Amazon Web Services Cloud. Main services are AWS Gateway and AWS Lambda. <br />
<br />
The simplified sequence of events is shown in the figure below:<br />
<br />
1. User says "Alexa, ask DX Cluster to list spots". Amazon Echo device sends the voice file to Amazon Alexa service that does the voice recognition.<br />
<br />
2. Amazon Alexa determines that the skill is "DX Cluster" and sends JSON formatted request to configured endpoint in AWS Gateway.<br />
<br />
3. AWS Gateway sends the request to AWS Lambda that loads my Python software.<br />
<br />
4. My "DX Cluster" software parses the JSON request, calls "ListIntent" handler. If not already loaded, it will make a web API request to pull the latest DX cluster data from ham.qth.com. The software will the convert the text to SSML format for speech output and returns the list of spots to Amazon Echo device. <br />
<br />
5. If user says "Select One" (the top one on the list), then the frequency of the selected spot is sent to the webserver running on X301 laptop. It will change the radio frequency using rigctl command and then return the URL to the latest MP3 that is recorded. This URL is passed to Amazon Echo device to start the playback.<br />
<br />
6. Amazon Echo device will retrieve the MP3 file from the X301 web server and starts playing.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxls2gotW1TScKj9pPidBJnvUAZB-bOUuP1qkPSV8NRgQnX7wwaNXEUdzyFJA82wWC-xHGwDiTfMxMqixWtiVXyEwtrhdpwNCpvIK7ABWSfcZGX8d-uT3K9nqO8EhHrRnGD-d2MH7ckZo/s1600/Alexa_DX_Cluster_Skill.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxls2gotW1TScKj9pPidBJnvUAZB-bOUuP1qkPSV8NRgQnX7wwaNXEUdzyFJA82wWC-xHGwDiTfMxMqixWtiVXyEwtrhdpwNCpvIK7ABWSfcZGX8d-uT3K9nqO8EhHrRnGD-d2MH7ckZo/s400/Alexa_DX_Cluster_Skill.png" width="308" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3. System Architecture</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<h2>
Software </h2>
<div>
As this is just a proof of concept the software is still very fragile and not ready for publishing. The software is written in Python language and is heavily using open source components, such as </div>
<div>
<br /></div>
<div>
<ul>
<li>hamblib - for controlling the Elecraft KX3 radio</li>
<li>rotter - for recording MP3 files from the radio </li>
<li>Flask - Python web framework </li>
<li>Boto3 - AWS Python libraries</li>
<li>Zappa - serverless Python services</li>
</ul>
<div>
<br /></div>
</div>
<div>
Once the software is a bit more mature I could post it on Github if there is any interest from the ham radio community for this. </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
73</div>
<div>
Mauri AG1LE </div>
<div>
<br /></div>
<h2>
</h2>
<style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style><style type="text/css">
p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px Menlo; color: #000000; background-color: #ffffff}
span.s1 {font-variant-ligatures: no-common-ligatures}
</style>ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-2307242307382283172016-02-06T21:08:00.002-05:002016-02-16T19:10:46.450-05:00KX3 Remote Control and audio streaming with Raspberry Pi 2 <h3>
REMOTE CONTROL OF ELECRAFT K3</h3>
I wanted to control my Elecraft KX3 transceiver remotely using my Android Phone. A quick Internet search yielded<span style="background-color: white;"><span style="color: lime;"><span style="background-color: white;"> <a href="http://kx3companion.com/kx3remote/" target="_blank"><span style="color: #38761d;">this site</span></a></span><span style="color: #38761d;"> </span></span></span>by Andrea IU4APC. His KX3 companion application on Android allows remote control using Raspberry Pi 2 and he has also links to an audio streaming application called Mumble.<br />
<br />
I did a quick ham shack inventory of hardware and software and realized that I had already everything required for this project. <br />
<br />
<span style="font-family: serif;">A short video how this works is in </span><span style="font-family: serif;">YouTube:</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<iframe allowfullscreen="" class="YOUTUBE-iframe-video" data-thumbnail-src="https://i.ytimg.com/vi/QRoJdlhE3VY/0.jpg" frameborder="0" height="266" src="https://www.youtube.com/embed/QRoJdlhE3VY?feature=player_embedded" width="320"></iframe></div>
<br />
<br />
KX3, Raspberry Pi2 and Android Phone connected together over Wifi.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghQxr6c_5fqATfRr_4uK0mXuwaXlhK58_LxOrbY-r_om2MfPuRKZ8LplgCV7hhf5habu7KeeoydLqhroj5pepk7HhNYq37cjWab0aOuk7MUPPbp9S86hGW6R80HEMTVZdh_ym1lIPz8yI/s1600/WP_20160206_006.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghQxr6c_5fqATfRr_4uK0mXuwaXlhK58_LxOrbY-r_om2MfPuRKZ8LplgCV7hhf5habu7KeeoydLqhroj5pepk7HhNYq37cjWab0aOuk7MUPPbp9S86hGW6R80HEMTVZdh_ym1lIPz8yI/s400/WP_20160206_006.jpg" width="225" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
HARDWARE COMPONENTS</h3>
Elecraft KX3<br />
<a href="http://www.elecraft.com/elecraft_prod_list.htm" target="_blank">Elecraft KXUSB Serial Cable for KX3</a><br />
<a href="https://www.blogger.com/"><span id="goog_1250337096"></span>Raspberry Pi 2<span id="goog_1250337097"></span></a> with Raspbian Linux. I have 32 GB SD memory card, 8 GB should also work.<br />
<a href="http://www.amazon.com/gp/product/B000KW2YEI" target="_blank">Behringer UCA202 USB Audio Interface </a> and audio cables<br />
Android Phone (I have <a href="https://oneplus.net/one" target="_blank">OnePlus One</a>)<br />
<br />
<br />
<h3>
CONFIGURE RASPBERRY PI AND KX3 COMPANION APP</h3>
Following <a href="http://kx3companion.com/kx3remote/" target="_blank">the instructions</a> I plugged the KXUSB Serial cable to the KX3 ACC1 port and to one of the two Raspberry Pi USB ports. <br />
<br />
I installed ser2net with following commands on command line:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo apt-get update </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo apt-get install ser2net </span><br />
<br />
then I edited the /etc/ser2net.conf file:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo nano /etc/ser2net.conf </span><br />
<br />
and added the following line:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 7777:raw:0:/dev/ttyUSB0:38400 8DATABITS NONE 1STOPBIT</span><br />
<br />
and saved the file by pressing CTRL+X and then Y<br />
<br />
I executed the ser2net:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ser2net </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo /etc/init.d/ser2net restart </span><br />
<br />
Once done with the host I downloaded the KX3 Companion app (<a href="https://play.google.com/store/apps/details?id=com.iu4apc.kx3companion_free&hl=en" target="_blank">link here</a>) on my Android phone and opened the app.<br />
<br />
To enable the KX3 Remote functionality you have to edit 3 options (“Remote Settings” section). <span style="font-family: serif;">Check the “Use KX3Remote/Piglet/Pigremote” option</span><br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7YReAmVHCr3PnLPGsAhLjcKj_4JkXMs_KL-mSBTrNB72lhDJLDUYy0YV8pWS4tRYvs742Nd9-txjj-1I3Zkj2hUdTmDxef15YXKkbm7Z7UWl-hmi6bUOvf1Kh5bJBOZ4ikN8dY4663vM/s1600/Screenshot_2016-02-06-16-10-20.png" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7YReAmVHCr3PnLPGsAhLjcKj_4JkXMs_KL-mSBTrNB72lhDJLDUYy0YV8pWS4tRYvs742Nd9-txjj-1I3Zkj2hUdTmDxef15YXKkbm7Z7UWl-hmi6bUOvf1Kh5bJBOZ4ikN8dY4663vM/s320/Screenshot_2016-02-06-16-10-20.png" width="180" /></a><br />
<br />
Set your PC/Raspberry Pi IP address in the “KX3Remote/Piglet/Pigremote IP” option. This below assumes that your RPI and Android phone are connected to the same Wifi network.<br />
<br />
In my case RPI is using WLAN0 interface connected to WiFi router and IP address is 192.168.0.47. This address depends on your local network configuration and <span style="font-family: serif;">you can get the Raspberry Pi IP address using command</span><br />
<div style="font-family: serif;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div style="font-family: serif;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ip addr show </span></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2NDTFrnqX-udedQHJ4R9ZBfzyfJ9_oZG0qiIQdHetFUKV5BfeRem6V890_d1HGx7WbK0paAML5U5gZxWjb5RlcfQI7bV2v2ceIK9Z2qsGv0joRXkcilNuCEdnpfbc-sZ7zTGF-16rjXc/s1600/Screenshot_2016-02-06-16-10-35.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg2NDTFrnqX-udedQHJ4R9ZBfzyfJ9_oZG0qiIQdHetFUKV5BfeRem6V890_d1HGx7WbK0paAML5U5gZxWjb5RlcfQI7bV2v2ceIK9Z2qsGv0joRXkcilNuCEdnpfbc-sZ7zTGF-16rjXc/s320/Screenshot_2016-02-06-16-10-35.png" width="180" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Set the choosen Port number (7777) on the PC/Raspberry Pi IP address in the “KX3Remote/Piglet/Pigremote Port” option<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhriqyZITXtD0Hvzq1CoSynfpdqgWPREJd62rO7ICQx7Sw2hb7Nz9dTpNqZDWeQs4EitJR1wLJz0neCMyE0WsW5jZSDPRrBNhfv54TtSKMij_VmIfplqkmWSRpFqnGraoSAoDd0YkGzqzA/s1600/Screenshot_2016-02-06-16-10-42.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhriqyZITXtD0Hvzq1CoSynfpdqgWPREJd62rO7ICQx7Sw2hb7Nz9dTpNqZDWeQs4EitJR1wLJz0neCMyE0WsW5jZSDPRrBNhfv54TtSKMij_VmIfplqkmWSRpFqnGraoSAoDd0YkGzqzA/s320/Screenshot_2016-02-06-16-10-42.png" width="180" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Now you can test the connection. By tapping "ON" button on the left top corner you can see if the connection was successful. A message "Connected to Piglet/Pigremote" should show up at the bottom - see below:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6YoEYglxiEiYJxoBcgaZGB2oH2bDZuhat0CDc0Z81Lix7I5uaD4u3zC5Zx3jJOjfXMUuDMA3Lqe0TaF4qdVJhx7joF7erBVqSgik2-k_ysrv6cx-n7TG1J4gYmJB33dy7nV2rgU_Ztng/s1600/Screenshot_2016-02-06-16-02-05.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj6YoEYglxiEiYJxoBcgaZGB2oH2bDZuhat0CDc0Z81Lix7I5uaD4u3zC5Zx3jJOjfXMUuDMA3Lqe0TaF4qdVJhx7joF7erBVqSgik2-k_ysrv6cx-n7TG1J4gYmJB33dy7nV2rgU_Ztng/s320/Screenshot_2016-02-06-16-02-05.png" width="180" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
If you are having problems with this, here are some troubleshooting ideas<br />
<br />
<ul>
<li>check the <span style="font-family: serif;">Raspberry Pi</span> IP address again</li>
<li>check that Raspberry Pi and Android Phone are on the same Wifi network</li>
<li>check that your KX3 serial port is set to 38400 bauds (this is the default in KX3 Companion App) </li>
</ul>
If everything works, you should be able to change the frequency and the bands on KX3 by tapping Band+/Band- and Freq+/Freq- buttons on the app. Current KX3 frequency will be updated on FREQUENCY field between buttons as you turn the VFO on KX3.<br />
<br />
<br />
<h3 style="font-family: serif;">
CONFIGURE RASPBERRY PI 2 FOR AUDIO </h3>
Plug in USB Audio Interface to Raspberry Pi 2 USB port. In my case I used Behringer UCA202 but there are many other alternatives available.<br />
<br />
The audio server is called Mumble. This is a low latency Voice over IP (VoIP) server designed for gaming community but it works well for streaming audio from KX3 to Android Phone and back. There is <a href="http://pimylifeup.com/raspberry-pi-mumble-server/" target="_blank">a great page</a> that describes installation in more details.<br />
<br />
I used the following commands to install mumble VoIP server<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> sudo apt-get install mumble-server</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> sudo dpkg-reconfigure mumble-server</span><br />
<br />
<br />
This last command will present you with a few options, set these however you would like mumble to operate.<br />
<br />
<ul>
<li>Autostart: I selected Yes </li>
<li>High Priority: I selected Yes (This ensures Mumble will always be given top priority even when the Pi is under a lot of stress) </li>
<li>SuperUser: Set the password here. This account will have full control over the server.</li>
</ul>
<br />
You need to know your IP address on Raspberry Pi 2 when configuring the Mumble client. Write it down as you will need it shortly. In my case it was 192.168.0.47<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ip addr show</span><br />
<br />
You may want to edit the server configuration file. I didn't do any changes but the installation page recommends changing welcome text and server password. You can do it using this command:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo nano /etc/mumble-server.ini</span><br />
<br />
Finally, you need to restart the server:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo /etc/init.d/mumble-server restart</span><br />
<br />
Now that we have the mumble server running we need to install the Mumble client on Raspberry Pi 2. This can be done with this command:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo apt-get install mumble</span><br />
<br />
Next you start the client application by typing:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">mumble</span><br />
<br />
This starts the mumble client. First you need to go through some configuration windows.<br />
<br />
You need to have USB audio interface input connected to KX3 Phones output when going though the Mumble Audio Wizard. I turned the audio volume to approximately 30.<br />
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicJhgnMoFNSNYoFxwGy9ILml7lZC3zsig7icKPQJkhhLWpzJsGtkDggNHMPhEx8k2hoNnVLkfFoWtHNOdk5geCwOKeEEumHh0AmQhHw1wMMRZTuL7DxpdxK4pH4lhai3ilVmqAsgZZg0Q/s1600/mumble_audio1.png" imageanchor="1" style="clear: left; display: inline !important; margin-bottom: 1em; margin-right: 1em; text-align: center;"><img border="0" height="244" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicJhgnMoFNSNYoFxwGy9ILml7lZC3zsig7icKPQJkhhLWpzJsGtkDggNHMPhEx8k2hoNnVLkfFoWtHNOdk5geCwOKeEEumHh0AmQhHw1wMMRZTuL7DxpdxK4pH4lhai3ilVmqAsgZZg0Q/s320/mumble_audio1.png" width="320" /></a><br />
<br />
You need to select the USB Audio device as the input device. Default device is "Default ALSA device" that is onboard audio chip. When clicking Device drop down list select SysDefault card - USB Audio Codec as shown on picture below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQ5jjzzCt4lvwoxaKo0ZDswT7TFwN1pXqew7nlWlytRiXG2TY4RfReWKwPnu6zcn85ZL371Fm3rrUZ2It5dXD5hbAtXcEJf_zHyti-HNvmB_VPI9jlpiu4RGpUf000WexJOABcJnE2q1o/s1600/mumble_audio2.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQ5jjzzCt4lvwoxaKo0ZDswT7TFwN1pXqew7nlWlytRiXG2TY4RfReWKwPnu6zcn85ZL371Fm3rrUZ2It5dXD5hbAtXcEJf_zHyti-HNvmB_VPI9jlpiu4RGpUf000WexJOABcJnE2q1o/s320/mumble_audio2.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The drop down list might be different depending on your hardware configuration. Select the SysDefault USB device.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwjuCWPEJ-xTHDAyRaThi2dOl_zVc8dsxh2ccYKe5fHb2Ac-HuAl7Sz4pj2HKNAVpo8wihFx3BsLn_MwXdndko02mnYpB_Xa4guYLE8xkYMllJvE37r6zApO4mGuDLRs_esd9WXxbL3aA/s1600/mumble_audio3.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="124" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwjuCWPEJ-xTHDAyRaThi2dOl_zVc8dsxh2ccYKe5fHb2Ac-HuAl7Sz4pj2HKNAVpo8wihFx3BsLn_MwXdndko02mnYpB_Xa4guYLE8xkYMllJvE37r6zApO4mGuDLRs_esd9WXxbL3aA/s320/mumble_audio3.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Once the Input and Output devices have been selected you can move forward with Next.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizc-sfTmwwThzPRHXsSpT4YRkqB-7eOLj36mwbKChaETv5fCAUxNYBOvVSpQiRm64bfpl3qTB7ZyK1fqIaXCnXLh_cyB1H6654zr4eGpHUCpybZ5uE7lCTwWT-iIpLdiWZcKP3Rqn998w/s1600/mumble_audio4.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEizc-sfTmwwThzPRHXsSpT4YRkqB-7eOLj36mwbKChaETv5fCAUxNYBOvVSpQiRm64bfpl3qTB7ZyK1fqIaXCnXLh_cyB1H6654zr4eGpHUCpybZ5uE7lCTwWT-iIpLdiWZcKP3Rqn998w/s320/mumble_audio4.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Next comes device tuning. I selected the longest delay for best sound quality.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdIzxy0leJwIAX8xbCCwUpz7WuQmJhTIgW1eSsus9-YLoepIjdNToj5ahT3SEuwgcIMG4kXr3NzmK4PXUIiManLr_CPXFl68CHyOaK03IJ6ZlA4Iv9d08ceFbcG24nEVDwWWu6_IkTVCk/s1600/mumble_audio5.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgdIzxy0leJwIAX8xbCCwUpz7WuQmJhTIgW1eSsus9-YLoepIjdNToj5ahT3SEuwgcIMG4kXr3NzmK4PXUIiManLr_CPXFl68CHyOaK03IJ6ZlA4Iv9d08ceFbcG24nEVDwWWu6_IkTVCk/s320/mumble_audio5.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Next comes Volume tuning. Make sure that KX3 audio volume is at least 30. You should see blue bar moving in sync with KX3 audio. Follow instructions.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8V1k2Fg2FVOnx_F0QP_2U_za8x-EzlmVlQ8_yQyahkFtwM98EJ6msuSy-Flt7SkZgQY01yim1bIaiJx-BH8ZFF0-ongcR5VUJSPzDrDRb5t9Y1t3KqgkeKUz-cemcM7AVRZD_65XaX44/s1600/mumble_audio6.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi8V1k2Fg2FVOnx_F0QP_2U_za8x-EzlmVlQ8_yQyahkFtwM98EJ6msuSy-Flt7SkZgQY01yim1bIaiJx-BH8ZFF0-ongcR5VUJSPzDrDRb5t9Y1t3KqgkeKUz-cemcM7AVRZD_65XaX44/s320/mumble_audio6.png" width="320" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj_4-GwDMqs9XGkPuFLLhFdq_HBd9FzwKqsgkas175wqJjYuDg7JG9XJbjHy7aQrXT1cGa-3z4k0GI4u7VyWuA8z28enKHSuldhriOvqfgdqI-RNeuLCDhPOzvVSKJ7MyBewtzDWDPuQo/s1600/mumble_audio7.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj_4-GwDMqs9XGkPuFLLhFdq_HBd9FzwKqsgkas175wqJjYuDg7JG9XJbjHy7aQrXT1cGa-3z4k0GI4u7VyWuA8z28enKHSuldhriOvqfgdqI-RNeuLCDhPOzvVSKJ7MyBewtzDWDPuQo/s1600/mumble_audio7.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj_4-GwDMqs9XGkPuFLLhFdq_HBd9FzwKqsgkas175wqJjYuDg7JG9XJbjHy7aQrXT1cGa-3z4k0GI4u7VyWuA8z28enKHSuldhriOvqfgdqI-RNeuLCDhPOzvVSKJ7MyBewtzDWDPuQo/s1600/mumble_audio7.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj_4-GwDMqs9XGkPuFLLhFdq_HBd9FzwKqsgkas175wqJjYuDg7JG9XJbjHy7aQrXT1cGa-3z4k0GI4u7VyWuA8z28enKHSuldhriOvqfgdqI-RNeuLCDhPOzvVSKJ7MyBewtzDWDPuQo/s1600/mumble_audio7.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhj_4-GwDMqs9XGkPuFLLhFdq_HBd9FzwKqsgkas175wqJjYuDg7JG9XJbjHy7aQrXT1cGa-3z4k0GI4u7VyWuA8z28enKHSuldhriOvqfgdqI-RNeuLCDhPOzvVSKJ7MyBewtzDWDPuQo/s320/mumble_audio7.png" width="320" /></a></div>
Next comes voice activity detection setting. Follow instructions. <br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3BiFJ6g3NhJ0apDkcMw6HBfapQjgZUTv6OQutLHmPxGbAXRsE9FhX5BuiMlWAczf2O8yZuBmmg7cG6ybYGYEdzxxvtJGmX8PRX9YRgmwRDLfmEf_YDf7yCevEduIL73HRE01U0Yt1iq0/s1600/mumble_audio8.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3BiFJ6g3NhJ0apDkcMw6HBfapQjgZUTv6OQutLHmPxGbAXRsE9FhX5BuiMlWAczf2O8yZuBmmg7cG6ybYGYEdzxxvtJGmX8PRX9YRgmwRDLfmEf_YDf7yCevEduIL73HRE01U0Yt1iq0/s1600/mumble_audio8.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3BiFJ6g3NhJ0apDkcMw6HBfapQjgZUTv6OQutLHmPxGbAXRsE9FhX5BuiMlWAczf2O8yZuBmmg7cG6ybYGYEdzxxvtJGmX8PRX9YRgmwRDLfmEf_YDf7yCevEduIL73HRE01U0Yt1iq0/s1600/mumble_audio8.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj3BiFJ6g3NhJ0apDkcMw6HBfapQjgZUTv6OQutLHmPxGbAXRsE9FhX5BuiMlWAczf2O8yZuBmmg7cG6ybYGYEdzxxvtJGmX8PRX9YRgmwRDLfmEf_YDf7yCevEduIL73HRE01U0Yt1iq0/s320/mumble_audio8.png" width="320" /></a></div>
Next comes quality selection. I selected high as I am testing this in local LAN network.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq8AiwkajwFQkf48vwIATXvYsBWNrEgHMfy_Mk1UhkmEDJACCgplj4LiKpr89bV52ePnj2b3WXiTL_ZUeQsTs0qoHF9khDIx5Ll5-zWhhojJ0ZnLL5GnowjVnkGas3J6ggRjmUBAFCrdw/s1600/mumble_audio9.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq8AiwkajwFQkf48vwIATXvYsBWNrEgHMfy_Mk1UhkmEDJACCgplj4LiKpr89bV52ePnj2b3WXiTL_ZUeQsTs0qoHF9khDIx5Ll5-zWhhojJ0ZnLL5GnowjVnkGas3J6ggRjmUBAFCrdw/s1600/mumble_audio9.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq8AiwkajwFQkf48vwIATXvYsBWNrEgHMfy_Mk1UhkmEDJACCgplj4LiKpr89bV52ePnj2b3WXiTL_ZUeQsTs0qoHF9khDIx5Ll5-zWhhojJ0ZnLL5GnowjVnkGas3J6ggRjmUBAFCrdw/s1600/mumble_audio9.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="162" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjq8AiwkajwFQkf48vwIATXvYsBWNrEgHMfy_Mk1UhkmEDJACCgplj4LiKpr89bV52ePnj2b3WXiTL_ZUeQsTs0qoHF9khDIx5Ll5-zWhhojJ0ZnLL5GnowjVnkGas3J6ggRjmUBAFCrdw/s320/mumble_audio9.png" width="320" /></a></div>
Audio settings are now completed.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBiw61kjUaBDSfHu3fEfOju_-oP2_PuzMuvhsO3XS4QV2jhnk-nuEZAmuFYxsrE4ydyRNNwubY65jonf2BWr5epZhqc1qe1oBg830wa4XXBaEKKc3A2F7vta1OFOxG9lEJTgSkDUQg2Wc/s1600/mumble_audio10.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><br /></a><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBiw61kjUaBDSfHu3fEfOju_-oP2_PuzMuvhsO3XS4QV2jhnk-nuEZAmuFYxsrE4ydyRNNwubY65jonf2BWr5epZhqc1qe1oBg830wa4XXBaEKKc3A2F7vta1OFOxG9lEJTgSkDUQg2Wc/s1600/mumble_audio10.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="223" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBiw61kjUaBDSfHu3fEfOju_-oP2_PuzMuvhsO3XS4QV2jhnk-nuEZAmuFYxsrE4ydyRNNwubY65jonf2BWr5epZhqc1qe1oBg830wa4XXBaEKKc3A2F7vta1OFOxG9lEJTgSkDUQg2Wc/s320/mumble_audio10.png" width="320" /></a></div>
<div class="separator" style="clear: both; text-align: left;">
Next comes server connect. You can "Add New..." by giving the IP address that you wrote down earlier. I gave the server label "raspberrypi" and username "pi".You don't have the change the port.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0Msj9C0X-dWb7ilCGScyQF4RRj_rcCAVLK5-q8YOe8H83eW8ZEgU-h5Hb9luxMxBy2vNuAk3POgRkeUPlkGG85wzTowUI_9SLfSWIIvjaNKh8W4-By3_kI0o6syAzC5DFeKP33oJZ6lU/s1600/mumble_audio11.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0Msj9C0X-dWb7ilCGScyQF4RRj_rcCAVLK5-q8YOe8H83eW8ZEgU-h5Hb9luxMxBy2vNuAk3POgRkeUPlkGG85wzTowUI_9SLfSWIIvjaNKh8W4-By3_kI0o6syAzC5DFeKP33oJZ6lU/s1600/mumble_audio11.png" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
When you connect to the server you should have a view like this below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9trSqUQduNi2TxMVlwg3cU6mV0VeQCndpy5Qo_NMpXrwUmR4mxonUHg8GyBK-77vVCVLuloSD6JrvWu97NuaFG0EKkT1-PuEX67Bfdi03y7nedMLOUoS2bG5dlYLpRqobz2nxiloloU4/s1600/mumble_audio12.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj9trSqUQduNi2TxMVlwg3cU6mV0VeQCndpy5Qo_NMpXrwUmR4mxonUHg8GyBK-77vVCVLuloSD6JrvWu97NuaFG0EKkT1-PuEX67Bfdi03y7nedMLOUoS2bG5dlYLpRqobz2nxiloloU4/s320/mumble_audio12.png" width="320" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Next step is then download mumble client on the Android phone and configure it.<br />
<br />
<br />
<h3>
CONFIGURE ANDROID PHONE </h3>
I downloaded <a href="https://play.google.com/store/apps/details?id=com.morlunk.mumbleclient.free&hl=en" target="_blank">free mumble client called Plumble</a> on my Android phone. You need to configure the Mumble server running on Raspberry Pi 2 on the software. Once you open Plumble client tap the "+" sign on right top corner.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVGqXYyBbOtHBnCohBz0_kz6rRkjVeLpGtTKrIb3-asDaNIw2ZloJB0OKRQsjCEX45EUn_9DDxhRmBy_v-j_RIcZFOjf71FTOP35pFK2gi2Bmvdsf9iCYdJ7VzEvxvXlxKGaoRhuUVxf8/s1600/Screenshot_2016-02-06-20-50-14.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgVGqXYyBbOtHBnCohBz0_kz6rRkjVeLpGtTKrIb3-asDaNIw2ZloJB0OKRQsjCEX45EUn_9DDxhRmBy_v-j_RIcZFOjf71FTOP35pFK2gi2Bmvdsf9iCYdJ7VzEvxvXlxKGaoRhuUVxf8/s320/Screenshot_2016-02-06-20-50-14.png" width="180" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I gave the label "KX3" and IP address of the Mumble server running on Raspberry Pi 2 - in my case the IP address is 192.168.0.47. For username I selected my ham radio call sign.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD2KOAY9q3ud8tT2chBnqhq7iko15AmcrNBr_dj1rc0RHQIkNlKS3WfgPabApypBBsV-cnV1xBAUSm1WcvxEBmmQzC5uI4Lu2B3MI6Z5iKSjRoE1PijslmLC4WV2m0FNc9j7AWvQk3-Uw/s1600/Screenshot_2016-02-06-20-50-51.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhD2KOAY9q3ud8tT2chBnqhq7iko15AmcrNBr_dj1rc0RHQIkNlKS3WfgPabApypBBsV-cnV1xBAUSm1WcvxEBmmQzC5uI4Lu2B3MI6Z5iKSjRoE1PijslmLC4WV2m0FNc9j7AWvQk3-Uw/s320/Screenshot_2016-02-06-20-50-51.png" width="180" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Since I did not configure any passwords on my server I left that field empty. Once the server has been added, you can try to connect to it.<br />
<br />
<h3 style="font-family: serif;">
OPERATION</h3>
If everything has gone well you should be able to connect to the Mumble VoIP server and hear a sound from your mobile phone.<br />
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfXPNdSSnfkd3QPc9f8FbZxtATUCkTeCONBM0Xr1qHxOibff_MAxVVgfBQo9OqGJdOmxZOoBQ6KV-GoIBpj-m7LBtRjbhTqAXChg-GUy7oJgeQBSJYSJ0xyy4x0DzEwwCKcLAKl_rGjAw/s1600/Screenshot_2016-02-06-20-51-47.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfXPNdSSnfkd3QPc9f8FbZxtATUCkTeCONBM0Xr1qHxOibff_MAxVVgfBQo9OqGJdOmxZOoBQ6KV-GoIBpj-m7LBtRjbhTqAXChg-GUy7oJgeQBSJYSJ0xyy4x0DzEwwCKcLAKl_rGjAw/s320/Screenshot_2016-02-06-20-51-47.png" width="180" /></a></div>
<br />
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<div>
On Raspberry Pi 2 you should see that another client "AG1LE" has connected to the server. See example below: </div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDuOwK-O22dQnmi4Tm8EpbmjYNPznmngiqoP6m3dgy4zfr4lexMtx86XAdcrVjQRbST3vbezSxGmBpUL45SZl_d_bRUV0WifKW9EZrF-TrVrF5-tONIyNl-KKEeHWhKnrwpJdNvpUJDGI/s1600/mumble_audio13.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="224" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDuOwK-O22dQnmi4Tm8EpbmjYNPznmngiqoP6m3dgy4zfr4lexMtx86XAdcrVjQRbST3vbezSxGmBpUL45SZl_d_bRUV0WifKW9EZrF-TrVrF5-tONIyNl-KKEeHWhKnrwpJdNvpUJDGI/s320/mumble_audio13.png" width="320" /></a></div>
<br />
<br />
<div style="text-align: left;">
</div>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
NEXT STEPS </h3>
If you want to extend from just listening KX3 to actually working remotely you need to configure your Wifi router to enable connection remotely over the Internet. Also, the USB audio interface need to be connected to the microphone (MIC) input of KX3 radio. KX3 must have VOX turned on to enable audio transmit.<br />
<br />
Documenting these steps will take a bit more time, so I leave it for the next session.<br />
<br />
Did you find these instructions useful? <span style="font-family: serif;">Any comments or feedback?</span><span style="font-family: serif;"> </span><br />
<span style="font-family: serif;"><br /></span>
<span style="font-family: serif;">73 </span><br />
<span style="font-family: serif;">Mauri AG1LE</span><br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-52696408158944983092015-12-27T23:35:00.003-05:002015-12-28T00:29:28.178-05:00TensorFlow: a new LSTM RNN based Morse decoder<h3>
INTRODUCTION</h3>
In my <a href="http://ag1le.blogspot.com/2015/11/experiment-deep-learning-algorithm-for.html">previous post</a> I created an experiment to train a LSTM Recurrent neural network (RNN) to detect symbols from noisy Morse code. I continued experiments, but this time I used the new <a href="https://www.tensorflow.org/">TensorFlow</a> open source library for machine intelligence. The flexible architecture of TensorFlow allows to deploy computation to one or more CPUs or GPUs in a desktop, server, or mobile device with a single API.<br />
<br />
TensorFlow was originally developed by researchers and engineers working on the Google Brain Team within Google's Machine Intelligence research organization for the purposes of conducting machine learning and deep neural networks research, but the system is general enough to be applicable in a wide variety of other domains as well.<br />
<br />
<h3>
EXPERIMENT </h3>
I started with the TensorFlow <a href="https://github.com/aymericdamien/TensorFlow-Examples/">MNIST example</a> authored by Aymeric Damien. <a href="https://en.wikipedia.org/wiki/MNIST_database">MNIST</a> is a large database of handwritten digits that is commonly used for machine learning experiments and algorithm development. Instead of training a LSTM RNN model using handwritten characters I created a Python script to generate a lot of Morse code training material. I downloaded ARRL Morse training text files and created a <a href="https://raw.githubusercontent.com/ag1le/LSTM_morse/master/arrl2.txt">large text file</a>. From this text file the Python script generates properly formatted training vectors, over 155,000 of them. The software is available as <a href="https://github.com/ag1le/LSTM_morse/blob/master/LSTM%20RNN%20Example.ipynb">Python inotebook format in Github</a>.<br />
<br />
The LSTM RNN model has the following parameters:<br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"># Parameters</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">learning_rate = 0.001 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">training_iters = 114000 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">batch_size = 126</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;"># Network Parameters</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">n_input = 1 # each Morse element is normalized to dit length 1 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">n_steps = 32 # timesteps (training material padded to 32 dit length)</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">n_hidden = 128 # hidden layer num of features </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">n_classes = 60 # Morse character set </span><br />
<br />
The training takes approximately 15 minutes on my Thinkpad X301 laptop. The progress of loss function and accuracy % over the training is depicted in Figure 1 below. The final accuracy was 93.6% after 114,000 training samples.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGlOYYMJSHoUK2DbZAo2BocvKnoaAwk4gzWdX1FJHWH57pNN4YgpiHJUD510_fGQ8VG2Z-4HBSKwA-axtUkdOJ_YXKQRL5laiBtAdN6rR6jTXabK2tcJtzm-GJdr1XwlqqBzpJXQa31Ps/s1600/machinelearning1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="402" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGlOYYMJSHoUK2DbZAo2BocvKnoaAwk4gzWdX1FJHWH57pNN4YgpiHJUD510_fGQ8VG2Z-4HBSKwA-axtUkdOJ_YXKQRL5laiBtAdN6rR6jTXabK2tcJtzm-GJdr1XwlqqBzpJXQa31Ps/s640/machinelearning1.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. Training progress over time</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I was testing the model with generated data while adding noise gradually to signals using the "sigma" parameter on the Python scripts. The results are below:<br />
<br />
<b>Test case:</b> QUICK BROWN FOX JUMPED OVER THE LAZY FOX 0123456789<br />
<b>Results:</b><br />
Noise 0.0: QUICK BROWN VOC YUMPED OVER THE LACY VOC ,12P45WOQ.<br />
Noise 0.02: QUICK BROWN VOC YUMPED OVER THE LACY FOC 012P45WOQ.<br />
Noise 0.05: QUICK BROWN VOC YUMPED OVER THE LACQ VOC ,,2P45WO2.<br />
Noise 0.1: Q5IOK BROWN FOX YUMPED O4ER THE LACY FOC 012P4FWO2,<br />
Noise 0.2: .4IOK WDOPD VOO 2FBPIM QFEF TRE WAC2 4OX 0,.PF52Q91<br />
As can be seen above at "sigma" level 0.2 the decoder starts to make a lot of errors.
<br />
<h3>
CONCLUSIONS</h3>
<br />
The software learns the Morse code by going through the training vectors multiple times. By going through 114,000 characters in training the model achieves 96.3% accuracy. I did not try to optimize anything and I just used the reference material that came with TensorFlow library.
This experiment shows that it is possible to build an intelligent Morse decoder that learns the patterns from the data and also allows to scale up more complex models with better accuracy and better tolerance for QSB and noisy signals.<br />
<br />
TensorFlow proved to be a very powerful new machine learning library that was relatively easy to use. The biggest challenge was to figure out what data formats to use with various API calls. Due to the complexity and richness of the TensorFlow library I am fairly sure that much can be done to improve the efficiency of this software. As TensorFlow has been designed so that it works on a desktop, server, tablet or even on a mobile phone this open new possibilities to build an intelligent, learning Morse decoder for different platforms.<br />
<br />
73
Mauri AG1LEag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com2tag:blogger.com,1999:blog-3326773214329183284.post-13233770027797102402015-11-24T22:35:00.000-05:002015-11-29T22:56:04.522-05:00Experiment: Deep Learning algorithm for Morse decoder using LSTM RNN<h3>
INTRODUCTION</h3>
In my <a href="http://ag1le.blogspot.com/2015/11/creating-training-material-for.html">previous post</a> I created a Python script to generate training material for neural networks.<br />
The goal is to test how well the modern <a href="https://en.wikipedia.org/wiki/Deep_learning">Deep Learning algorithms</a> would work in decoding noisy Morse signals with heavy QSB fading.<br />
<br />
I did some research on various frameworks and found <span style="color: #38761d;"><a href="http://danielhnyk.cz/choosing-framework-building-neural-networks-mainly-rrn-lstm/">this article</a> </span>from Daniel Hnyk. My requirements were quite similar - full Python support, LSTM RNN built-in and a simple interface.<br />
He had selected <a href="https://github.com/fchollet/keras">Keras</a> that is available in Github. There is a <a href="https://groups.google.com/forum/#!forum/keras-users">mailing list for Keras users</a> that is fairly active and quite useful to find support from other users. I installed Keras on my Linux laptop and using <a href="http://jupyter.org/">Jupyter </a>interactive notebooks it was easy to start experimenting with various neural network configurations.<br />
<br />
<h3>
<br />SIMPLE RECURRENT NEURAL NETWORK EXPERIMENT</h3>
Using various sources and above mailing list I came up with the following experiment. I have uploaded the <a href="https://raw.githubusercontent.com/ag1le/RNN-Morse/master/RNN-Morse.ipyb">Jupyter notebook file</a> in Github in case the reader wants to replicate the experiment.<br />
<br />
The source code or printed output text is shown below with <span style="font-family: "courier new" , "courier" , monospace;">courier font</span> and I have added some commentary as well as the graphs as pictures.<br />
<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [12]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#!/usr/bin/env python</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># MorseEncoder.py - Morse Encoder to generate training material for neural networks</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Generates raw signal waveforms with Gaussian noise and QSB (signal fading) effects</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Provides also the training target variables in separate columns. Example usage:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># WPM= 40 # speed 40 words per minute</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Tq = 4. # QSB cycle time in seconds (typically 5..10 secs)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># sigma = 0.02 # add some Gaussian noise</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># P = signal('QUICK BROWN FOX JUMPED OVER THE LAZY FOX ',WPM,Tq,sigma)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># from matplotlib.pyplot import plot,show,figure,legend</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># from numpy.random import normal</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># figure(figsize=(12,3))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lb1,=plot(P.t,P.sig,'b',label="sig")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lb2,=plot(P.t,P.dit,'g',label="dit")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lb3,=plot(P.t,P.dah,'g',label="dah")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lb4,=plot(P.t,P.ele,'m',label="ele")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lb5,=plot(P.t,P.chr,'c',label="chr")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># lb6,=plot(P.t,P.wrd,'r*',label="wrd")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># legend([lb1,lb2,lb3,lb4,lb5,lb6])</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># show()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># P.to_csv("MorseTest.csv")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Copyright (C) 2015 Mauri Niininen, AG1LE</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># MorseEncoder.py is free software: you can redistribute it and/or modify</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># it under the terms of the GNU General Public License as published by</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># the Free Software Foundation, either version 3 of the License, or</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># (at your option) any later version.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># MorseEncoder.py is distributed in the hope that it will be useful,</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># but WITHOUT ANY WARRANTY; without even the implied warranty of</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># GNU General Public License for more details.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># You should have received a copy of the GNU General Public License</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># along with bmorse.py. If not, see <http://www.gnu.org/licenses/>.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import numpy as np</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import pandas as pd</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from numpy import sin,pi</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from numpy.random import normal</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">pd.options.mode.chained_assignment = None #to prevent warning messages</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Morsecode = {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '!': '-.-.--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '$': '...-..-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> "'": '.----.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '(': '-.--.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ')': '-.--.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ',': '--..--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '-': '-....-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '.': '.-.-.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '/': '-..-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '0': '-----',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '1': '.----',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '2': '..---',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '3': '...--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '4': '....-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '5': '.....',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '6': '-....',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '7': '--...',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '8': '---..',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '9': '----.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ':': '---...',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ';': '-.-.-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '<AR>': '.-.-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '<AS>': '.-...',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '<HM>': '....--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '<INT>': '..-.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '<SK>': '...-.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '<VE>': '...-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '=': '-...-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '?': '..--..',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '@': '.--.-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'A': '.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'B': '-...',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'C': '-.-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'D': '-..',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'E': '.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'F': '..-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'G': '--.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'H': '....',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'I': '..',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'J': '.---',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'K': '-.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'L': '.-..',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'M': '--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'N': '-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'O': '---',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'P': '.--.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'Q': '--.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'R': '.-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'S': '...',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'T': '-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'U': '..-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'V': '...-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'W': '.--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'X': '-..-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'Y': '-.--',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> 'Z': '--..',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '\\': '.-..-.',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '_': '..--.-',</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> '~': '.-.-'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">def encode_morse(cws):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> s=[]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> for chr in cws:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> try: # try to find CW sequence from Codebook</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> s += Morsecode[chr]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> s += ' '</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> except:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if chr == ' ':</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> s += '_'</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> continue</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> print "error: '%s' not in Codebook" % chr</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> return ''.join(s)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">def len_dits(cws):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # length of string in dit units, include spaces</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> val = 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> for ch in cws:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch == '.': # dit len + el space </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> val += 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch == '-': # dah len + el space</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> val += 4</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch==' ': # el space</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> val += 2</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch=='_': # el space</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> val += 7</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> return val</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">def signal(cw_str,WPM,Tq,sigma):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # for given CW string i.e. 'ABC ' </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # return a pandas dataframe with signals and symbol probabilities</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # WPM = Morse speed in Words Per Minute (typically 5...50)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # Tq = QSB cycle time (typically 3...10 seconds) </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # sigma = adds gaussian noise with standard deviation of sigma to signal</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> cws = encode_morse(cw_str)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> #print cws</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # calculate how many milliseconds this string will take at speed WPM</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ditlen = 1200/WPM # dit length in msec, given WPM</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> msec = ditlen*(len_dits(cws)+7) # reserve +7 for the last pause</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> t = np.arange(msec)/ 1000. # time array in seconds</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> ix = range(0,msec) # index for arrays</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # Create a DataFrame and initialize</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> col =["t","sig","dit","dah","ele","chr","wrd","spd"]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P = pd.DataFrame(index=ix,columns=col)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.t = t # keep time </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig=np.zeros(msec) # signal stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.dit=np.zeros(msec) # probability of 'dit' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.dah=np.zeros(msec) # probability of 'dah' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.ele=np.zeros(msec) # probability of 'element space' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.chr=np.zeros(msec) # probability of 'character space' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.wrd=np.zeros(msec) # probability of 'word space' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.spd=np.ones(msec)*WPM #speed stored here </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> #pre-made arrays with multiple(s) of ditlen</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> z = np.zeros(ditlen) </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> z2 = np.zeros(2*ditlen)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> z4 = np.zeros(4*ditlen)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dit = np.ones(ditlen)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dah = np.ones(3*ditlen)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> # For all dits/dahs in CW string generate the signal, update symbol probabilities</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i = 0</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> for ch in cws:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch == '.':</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dur = len(dit)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig[i:i+dur] = dit</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.dit[i:i+dur] = dit</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i += dur</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dur=len(z)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig[i:i+dur] = z</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.ele[i:i+dur] = np.ones(dur)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i += dur</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch == '-':</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dur = len(dah)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig[i:i+dur] = dah</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.dah[i:i+dur]= dah</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i += dur </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dur=len(z)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig[i:i+dur] = z</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.ele[i:i+dur] = np.ones(dur)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i += dur</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch == ' ':</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dur = len(z2)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig[i:i+dur] = z2</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.chr[i:i+dur]= np.ones(dur)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i += dur</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if ch == '_':</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> dur = len(z4)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig[i:i+dur] = z4</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.wrd[i:i+dur]= np.ones(dur)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> i += dur</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if Tq > 0.: # QSB cycle time impacts signal amplitude</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> qsb = 0.5 * sin((1./float(Tq))*t*2*pi) +0.55</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig = qsb*P.sig</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if sigma >0.:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig += normal(0,sigma,len(P.sig))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> return P</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [13]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">print ('MorseEncoder started')</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">%matplotlib inline</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from matplotlib.pyplot import plot,show,figure,legend, title</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from numpy.random import normal</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">WPM= 40</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Tq = 1.8 # QSB cycle time in seconds (typically 5..10 secs)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sigma = 0.01 # add some Gaussian noise</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">P = signal('QUICK',WPM,Tq,sigma)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">figure(figsize=(12,3))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">lb1,=plot(P.t,P.sig,'b',label="sig")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">title("QUICK in Morse code - (c) 2015 AG1LE")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">legend([lb1])</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">show()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">print ('MorseEncoder finished. %d datapoints created' % len(P.sig)) </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">MorseEncoder started</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">The Jupyter notebook will plot this graph that basically shows the text 'QUICK' converted to noisy signal with strong QSB fading. This signal goes down close to zero between letters C and K as you can see below. </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpLeu58DL1n9AQbLFQW2wV5hrP3a0DCZjv7aOno4i8sSjPOaKVG4WH0SDEgVkEXSfdPTRzoyGOS6fxtuErGfB-8VF6m2wvBiGCxUa4zR4JxRfQ_iQ9Nb3tXzPEfZnXzrcO96_P7iBtkmw/s1600/index2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjpLeu58DL1n9AQbLFQW2wV5hrP3a0DCZjv7aOno4i8sSjPOaKVG4WH0SDEgVkEXSfdPTRzoyGOS6fxtuErGfB-8VF6m2wvBiGCxUa4zR4JxRfQ_iQ9Nb3tXzPEfZnXzrcO96_P7iBtkmw/s640/index2.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. The training signal containing noise and QSB fading</td></tr>
</tbody></table>
<span style="font-family: inherit;">The next section of the code imports some libraries (including Keras) that is used for Neural Network experimentation. I am also preparing the data to the proper format that Keras requires. </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">MorseEncoder finished. 1950 datapoints created</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [14]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Time Series Testing - Morse case</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import keras.callbacks</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from keras.models import Sequential </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from keras.layers.core import Dense, Activation, Dense, Dropout</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">from keras.layers.recurrent import LSTM</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import random</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import numpy as np</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">import matplotlib.pyplot as plt</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">%matplotlib inline</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Data preparation </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># use 100 examples of data to predict nb_samples (850) in the future</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">samples = 1950</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">examples = 1000</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">y_examples = 100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">x = np.linspace(0,1950,samples)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">nb_samples = samples - examples - y_examples</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">data = P.sig</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># prepare input for RNN training - 1 feature</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">input_list = [np.expand_dims(np.atleast_2d(data[i:examples+i]), axis=0) for i in xrange(nb_samples)]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">input_mat = np.concatenate(input_list, axis=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">lb1,=plot(x,data,label="input")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">lb2,=plot(x,P.dit,label="target")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">legend([lb1,lb2])</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">title("training input and target data")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Out[14]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><matplotlib.text.Text at 0x10c119b50></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">This graph shows the training data (the noisy, fading signal) and the target data (I selected 'dits' in this example). This is just to verify that I have the right datasets selected. </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifPwjNP-Vzlnb_oDF_b4mllzGn_tRfU49BNUND4TUtJDAZcQOU5FoAjRpRQoZMBN1ewamqMqilpebkT1zYmPtN4KDmNekNQ9oXAFG9ihYDHXgeeWbbBR8E6sZGPvg8PNGgGuBhGRkhApQ/s1600/index3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="438" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifPwjNP-Vzlnb_oDF_b4mllzGn_tRfU49BNUND4TUtJDAZcQOU5FoAjRpRQoZMBN1ewamqMqilpebkT1zYmPtN4KDmNekNQ9oXAFG9ihYDHXgeeWbbBR8E6sZGPvg8PNGgGuBhGRkhApQ/s640/index3.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. Training and target <span style="font-size: 12.8px;">data </span></td></tr>
</tbody></table>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">In the following sections I prepare the training target ('dits') to proper format and setup the neural network model. I am using LSTM with Dropout and the model has 300 hidden neurons. I have also a callback function defined to capture the loss data during the training so that I can plot the loss curve to see the training progress. </span><span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [15]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># prepare target - the first column in merged dataframe</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">ydata = P.dit</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">target_list = [np.atleast_2d(ydata[i+examples:examples+i+y_examples]) for i in xrange(nb_samples)]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">target_mat = np.concatenate(target_list, axis=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># set up a model</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">trials = input_mat.shape[0]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">features = input_mat.shape[2]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">hidden = 300</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model = Sequential()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model.add(LSTM(input_dim=features, output_dim=hidden,return_sequences=False))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model.add(Dropout(.2))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model.add(Dense(input_dim=hidden, output_dim=y_examples))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model.add(Activation('linear'))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model.compile(loss='mse', optimizer='rmsprop')</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Call back to capture losses </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">class LossHistory(keras.callbacks.Callback):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> def on_train_begin(self, logs={}):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> self.losses = []</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> def on_batch_end(self, batch, logs={}):</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> self.losses.append(logs.get('loss'))</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Train the model</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">history = LossHistory()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">model.fit(input_mat, target_mat, nb_epoch=100,callbacks=[history])</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Plot the loss curve </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">plt.plot( history.losses)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">title("training loss")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">Here I have started the training. I selected 100 epochs - this means that the software will go through the training material for 100 times during the training. As you can see this goes very quickly - with larger model or larger datasets the training might take minutes to hours per epoch. We have a very small model and small dataset here. </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 1/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.1050 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 2/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0927 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 3/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0870 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 4/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0823 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 5/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0788 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 6/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0756 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 7/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0724 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 8/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0693 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 9/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0668 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 10/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0639 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 11/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0611 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 12/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0586 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 13/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0561 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 14/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0539 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 15/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0519 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 16/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0495 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 17/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0476 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 18/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0456 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 19/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0441 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 20/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0430 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 21/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0411 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 22/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0400 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 23/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0387 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 24/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0378 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 25/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0370 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 26/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0356 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 27/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0350 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 28/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0340 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 29/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0334 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 30/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0328 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 31/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0322 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 32/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0317 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 33/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0309 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 34/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0302 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 35/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0299 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 36/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0296 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 37/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0290 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 38/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0285 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 39/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0283 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 40/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0277 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 41/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0272 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 42/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0268 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 43/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0265 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 44/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0258 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 45/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0256 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 46/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0253 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 47/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0251 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 48/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0248 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 49/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0246 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 50/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0241 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 51/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0236 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 52/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0233 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 53/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0234 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 54/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0230 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 55/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0229 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 56/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0224 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 57/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0223 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 58/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0218 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 59/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0218 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 60/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0215 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 61/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0215 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 62/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0212 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 63/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0208 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 64/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0209 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 65/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0207 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 66/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0205 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 67/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0203 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 68/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0200 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 69/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0200 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 70/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0197 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 71/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0197 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 72/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0198 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 73/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0193 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 74/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0191 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 75/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0189 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 76/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0188 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 77/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0189 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 78/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0185 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 79/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0185 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 80/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0184 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 81/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0183 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 82/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0181 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 83/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0180 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 84/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0179 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 85/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0177 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 86/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0177 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 87/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0174 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 88/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0177 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 89/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0175 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 90/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0173 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 91/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0172 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 92/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0171 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 93/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0171 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 94/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0167 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 95/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0167 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 96/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0170 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 97/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0164 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 98/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0166 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 99/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0163 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Epoch 100/100</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">850/850 [==============================] - 0s - loss: 0.0164 </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Out[15]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><matplotlib.text.Text at 0x11e055350></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">The following graph shows the training loss during the training process. This gives you an idea whether the training is progressing well or if you have some problem with the model or the parameters. </span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGEdoPQ-3MggOZgRW9Mfbbh-lE37jStcDEmoQPCAslbrfYia-j5OVEwiacMYfYvqSikQlNYLEhBZ95kDd_snavoibz-o_nLvuqIP7Fkkwpxhx6o8jSq78798U3GO13SA4WaAxe7YmGih0/s1600/index4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="443" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGEdoPQ-3MggOZgRW9Mfbbh-lE37jStcDEmoQPCAslbrfYia-j5OVEwiacMYfYvqSikQlNYLEhBZ95kDd_snavoibz-o_nLvuqIP7Fkkwpxhx6o8jSq78798U3GO13SA4WaAxe7YmGih0/s640/index4.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3. Training loss curve</td></tr>
</tbody></table>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [16]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Use training data to check prediction</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">predicted = model.predict(input_mat)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [17]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Plot original data (green) and predicted data (red)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">lb1,=plot(data,'g',label="training")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#lb2,=plot(ydata,'b',label="target")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">lb3,=plot(xrange(examples,examples+nb_samples), predicted[:,1],'r',label="predicted")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">legend([lb1,lb3])</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">title("training vs. predicted")</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Out[17]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><matplotlib.text.Text at 0x11f164610></span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">In this section I am checking the model prediction. Since I am using the training material this is supposed to show a good result if the training was successful. As you can see from figure 4. below the predicted graph (<span style="color: red;">red color</span>) is aligned with 'dits' in the training signal (<span style="color: lime;">green color</span>) despite QSB fading and noise in the signal. </span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJEtTRM0ISIAzXQ1djl1s1h5dUgX4brLv4LtZo8nWf6aMh48isFuRzY2cDK2wainBkb3jhj6MlWAnEVZR8NDA9qgnbFUiLeR0P0VjMWdQs5En13YqOPBgdHpfDqQXqv-7zS-bmmjo4WVg/s1600/index5.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="438" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjJEtTRM0ISIAzXQ1djl1s1h5dUgX4brLv4LtZo8nWf6aMh48isFuRzY2cDK2wainBkb3jhj6MlWAnEVZR8NDA9qgnbFUiLeR0P0VjMWdQs5En13YqOPBgdHpfDqQXqv-7zS-bmmjo4WVg/s640/index5.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 4. Training vs. predicted graph</td></tr>
</tbody></table>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">In the following section I will create another Morse signal, this time with text 'KCIUQ' but using the same noise, QSB and speed parameters. I am planning to use this signal to validate how well the model has generalized the 'dit' concept. </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [18]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># Let's change the input signal, instead of QUICK we have KCIUQ in Morse code </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">P = signal('KCIUQ',WPM,Tq,sigma)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">data = P.sig</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"># prepare input - 1 feature</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">input_list = [np.expand_dims(np.atleast_2d(data[i:examples+i]), axis=0) for i in xrange(nb_samples)]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">input_mat = np.concatenate(input_list, axis=0)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">plt.plot(x,data)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Out[18]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[<matplotlib.lines.Line2D at 0x136050f90>]</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
Here is the generated validation Morse signal. It has the same letter as before but in reverse order. Can you read letters 'KCIUQ' from the graph below?<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEin6St2r_6yuWumS_Xnkn_o3PEh9b2nvwuM8a0XacP6VZ93hGKhm8r7noZoOqbSIdFmmBmZwJUusQ6mxut0HREh-RnRMDd7rbsO9ptS2w_c-klOEEr4BPin8wRDUyL9xfCOAxeO3fvJaq0/s1600/index6.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="422" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEin6St2r_6yuWumS_Xnkn_o3PEh9b2nvwuM8a0XacP6VZ93hGKhm8r7noZoOqbSIdFmmBmZwJUusQ6mxut0HREh-RnRMDd7rbsO9ptS2w_c-klOEEr4BPin8wRDUyL9xfCOAxeO3fvJaq0/s640/index6.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 5. Validation Morse signal</td></tr>
</tbody></table>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: inherit;">In this section I use the above validation signal to create a prediction and the plot the results. </span><br />
<span style="font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">In [19]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">predicted = model.predict(input_mat)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">plt.plot(data,'g')</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">plt.plot(xrange(examples,examples+nb_samples), predicted[:,1],'r')</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Out[19]:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">[<matplotlib.lines.Line2D at 0x1217be9d0>]</span><br />
<span style="font-family: inherit;"><br /></span>
<span style="font-family: inherit;">As you can see from the graph below the predicted 'dit' symbols (<span style="color: red;">red color</span>) don't really line up with actual 'dits' in the signal (<span style="color: lime;">green color</span>). This is not a surprise to me. To build a good model that can generalize the learning you need to have a lot of training material (typically millions of datapoints) and the model needs to have enough neural nodes to capture the details of the underlying signals. </span><br />
<span style="font-family: inherit;">In this simple experiment I had only 1950 datapoints and 300 hidden nodes. There are only 8 'dit' symbols in the training material - learning CW skill well requires a lot more material and many repetitions, as any human who has gone through the process can testify. Same principle applies for neural networks. </span><br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja-puTdrmAjcYOsHBKWBxznW42AfGZm95GAUlGhZJ1at6T1SNlaluXeLnyJVkfGebEXEsghFRqcD4A587vlyAXeCo9D_yG8ObH63zrI5l8SheFD6UjcJZQppODpLQUtH6rtuQU2jHD6Lg/s1600/index7.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="414" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEja-puTdrmAjcYOsHBKWBxznW42AfGZm95GAUlGhZJ1at6T1SNlaluXeLnyJVkfGebEXEsghFRqcD4A587vlyAXeCo9D_yG8ObH63zrI5l8SheFD6UjcJZQppODpLQUtH6rtuQU2jHD6Lg/s640/index7.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 6. Validation test </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
CONCLUSIONS </h3>
In this experiment I built a proof of concept to test whether Recurrent Neural Networks (especially LSTM variant) could be used to learn to detect symbols from noisy Morse code that has deep QSB fading. This experiment may contain errors and misunderstandings from my part as I have only had a few hours to play with this Keras Neural Network framework. Also, the concept itself needs still more validation as I may have used the framework incorrectly. <br />
<br />
I think that the results look quite promising. In only 100 epochs the RNN model learned 'dits' from the noisy signal and was able to separate them from 'dah' symbols. As the validation test shows I overfitted the model to this small sample of training material used in the experiment. It will take much more training data and larger, more complicated neural network to learn to generalize the symbols in Morse code. The training process may also need more computing capacity. It might be beneficial to have a graphics card with GPU to speed up the training process going forward.<br />
<br />
Any comments or feedback? <br />
<br />
73<br />
Mauri AG1LE<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com2tag:blogger.com,1999:blog-3326773214329183284.post-62952943951792730502015-11-22T12:15:00.001-05:002015-11-22T12:55:13.062-05:00Creating Training Material for Recurrent Neural Networks<h3>
INTRODUCTION</h3>
In my <a href="http://ag1le.blogspot.com/2015/11/your-next-qso-partner-artificial.html">previous post</a> I shared an experiment I did using Recurrent Neural Network (RNN) software. I started thinking that perhaps RNNs could learn not just the QSO language concepts but also learn how to decode Morse code from noisy signals. Since I was able to demonstrate learning of the syntax, structure and commonly used phrases in QSOs just in 50 epochs after going through the training material, wouldn't the same concept work for actual Morse signals?<br />
<br />
Well, I don't really have any suitable training materials to test this. For the Kaggle competitions (<a href="http://ag1le.blogspot.com/2015/01/morse-learning-machine-v1-challenge.html">MLMv1</a>, <a href="https://inclass.kaggle.com/c/morse-learning-machine-challenge-v2/forums/t/14822/and-the-winner-is/82501">MLMv2</a>) I created a lot of training materials but the focus of these materials was different. The audio files and corresponding transcript files were open ended as I didn't want to narrow down possible approaches that participants might take. The materials were designed for a Kaggle competition in mind to be able to score participants' solutions.<br />
<br />
In machine learning you typically have training & validation material that has many different dimensions and a target variable (or variables) you are trying to model. With neural networks you can train the network to look patterns in the input(s) and set outputs to target values when the input pattern is detected. With RNNs you can introduce memory function - this is necessary because you need to remember signal values from the past to properly decode the Morse characters.<br />
<br />
In Morse code you typically have just one signal variable and goal is to extract decoded message from that signal. This could be done by having for example 26 outputs for each alphabet character and train the network to set output 'A' to high when pattern '.-' is detected in the signal input. Alternatively you could have output lines for symbols like 'dit' and 'dah' and 'element space' that are set high when corresponding pattern is detected in the input signal.<br />
<br />
Since a well working Morse decoder has to deal with different speeds (typically 5 ... 50 WPM), signals containing noise and <a href="https://en.wikipedia.org/wiki/Fading">QSB fading</a> and other factors I decided to create a Morse Encoder software that creates artificial training signals, but also corresponding symbols, speed information etc. I chose to use this symbols approach because it easier to debug errors and problems when you can plot the inputs vs. outputs graphically. See this <a href="https://en.wikipedia.org/wiki/Morse_code#Representation.2C_timing_and_speeds">Wikipedia article for details</a> about representation, timing of symbols and speed.<br />
<br />
The Morse Encoder generates a set of time synchronized signals and has also capability to add QSB type fading effects and Gaussian noise. See example of 'QUICK BROWN FOX JUMPED OVER THE LAZY FOX ' plotted with deep QSB fading with 4 second cycle time and 0.01 sigma Gaussian noise added in Figure 1. below.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge6Dfsx_15zbCvonwtQDmCIcRrvYQD9zOLVvFoFGJD9IuMsrKbvKJRVziQCHQXsfSJRSN4pEi9kTJ2EUFqm76hSA24xseX5N2LIUE3V70dcNPTBY0pcMHNjbjbffNfIcAM_Z4k0TI7bmM/s1600/MorseEncoder.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="185" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEge6Dfsx_15zbCvonwtQDmCIcRrvYQD9zOLVvFoFGJD9IuMsrKbvKJRVziQCHQXsfSJRSN4pEi9kTJ2EUFqm76hSA24xseX5N2LIUE3V70dcNPTBY0pcMHNjbjbffNfIcAM_Z4k0TI7bmM/s640/MorseEncoder.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. Morse Encoder output signal with QSB and noise</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The QSB for real life signals doesn't always follow sin() curve like in Fig 1. but as you can see from example below this is close enough. The big challenge is how to continue decoding correctly when the signal goes down to noise level as shown between 12000 to 14000 time samples (horizontal axis) below.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9SzoAZIZD2Woxadmr55w5wcH5YwF_phnqf-99CxVxf31LZKY7fVxzXb2-2H_ZRXaMzxv-c51z2OJKREv7rZgH7FMKtFyUlY0R2SV_4Aa-Zjp4KMabPTfXCNQKyIMRv6zR2kVtalt-vHk/s1600/MorseEncoder5.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="246" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9SzoAZIZD2Woxadmr55w5wcH5YwF_phnqf-99CxVxf31LZKY7fVxzXb2-2H_ZRXaMzxv-c51z2OJKREv7rZgH7FMKtFyUlY0R2SV_4Aa-Zjp4KMabPTfXCNQKyIMRv6zR2kVtalt-vHk/s640/MorseEncoder5.png" width="640" /></a></div>
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
TRAINING MATERIALS</h3>
To provide proper target values for RNN training the Morse Encoder creates a Python DataFrame with the following columns defined<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.t # keep time </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.sig # signal stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.dit # probability of 'dit' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.dah # probability of 'dah' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.ele # probability of 'element space' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.chr # probability of 'character space' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.wrd # probability of 'word space' stored here</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> P.spd # WPM speed stored here </span><br />
<div>
<br /></div>
Using these columns Morse Encoder takes the given text and parameters and then generates values to these columns. For example when there is a 'dit' in the signal, on corresponding rows the P.dit has probability of 1.0. Likewise, if there is a 'dah' in the signal, on corresponding rows the P.dah has probability of 1.0. This is shown on the Figure 2. below - dits are <span style="color: red;">red</span> and dahs are <span style="color: #38761d;">green</span>, while the signal is shown in <span style="color: blue;">blue</span> color.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgpYgRZN_DUuLoWkZ1gOblspjz9QYQrHndNzNBM-P4lMKYJOiwTl_AomQABoNgTN_R0_Cj4vM4D4sHW89gSMmvLKSf-91fne-tT3_kV9E06FC58F8VkTOz3UA1HTwuO6HmxHANHTWVdW0/s1600/MorseEncoder2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgpYgRZN_DUuLoWkZ1gOblspjz9QYQrHndNzNBM-P4lMKYJOiwTl_AomQABoNgTN_R0_Cj4vM4D4sHW89gSMmvLKSf-91fne-tT3_kV9E06FC58F8VkTOz3UA1HTwuO6HmxHANHTWVdW0/s640/MorseEncoder2.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 2. Dit and Dah probabilities </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Zoomed section of letters 'QUI ' is shown on Fig 3. below.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVo3VhDG5sJUxbMl2cM8AqRTcibwO32NZanJ-A6f_gmAQFmU3o397lEDwzuKNgn351AzliAg9kGRYrhpvrJV5SXlmUtYqYQfeNIaMi-n5Z2_v45_nM6RbdZRA_T7N4DYtOLcsKlkXXpzY/s1600/MorseEncoder3.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVo3VhDG5sJUxbMl2cM8AqRTcibwO32NZanJ-A6f_gmAQFmU3o397lEDwzuKNgn351AzliAg9kGRYrhpvrJV5SXlmUtYqYQfeNIaMi-n5Z2_v45_nM6RbdZRA_T7N4DYtOLcsKlkXXpzY/s640/MorseEncoder3.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 3. Zoomed section</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
Likewise we create probabilities for the spaces. In Figure 4 below element space is shown with <span style="color: magenta;">magenta</span> and character space with <span style="color: cyan;">cyan</span> color. I decided to set character space to probability 1.0 only after element space has passed, as can be seen from the graph.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbwJVtXefEVo3ci6FrbkjsahDeTVF3wwAHe-5k2MLLX8LruAKHGnsx2lE42rG14AFf1h65Dl7yj7nntZXdRWbJdxI53pqqCySzApn2d0Vaf_8YDG12UYZ-KD5rdq-zqgcscHU6sV8kYPw/s1600/MorseEncoder4.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="186" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbwJVtXefEVo3ci6FrbkjsahDeTVF3wwAHe-5k2MLLX8LruAKHGnsx2lE42rG14AFf1h65Dl7yj7nntZXdRWbJdxI53pqqCySzApn2d0Vaf_8YDG12UYZ-KD5rdq-zqgcscHU6sV8kYPw/s640/MorseEncoder4.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 4. Element Space and Character Space </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The resulting DataFrame can be saved into a CSV file with a simple Python command and it is very easy to manipulate or plot graphs. Conceptually it is like an Excel spreadsheet - see below:<br />
<br />
<table border="1" class="dataframe" style="background-color: white; border-collapse: collapse; border-spacing: 0px; border: 1px solid black; box-sizing: border-box; color: black; font-family: 'Helvetica Neue', Helvetica, Arial, sans-serif; font-size: 14px; line-height: 20px; margin-left: 0px; margin-right: 0px;"><thead style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; text-align: right;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;"></th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">t</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">sig</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">dit</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">dah</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">ele</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">chr</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">wrd</th><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; text-align: left; vertical-align: middle;">spd</th></tr>
</thead><tbody style="box-sizing: border-box;">
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.000</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.573355</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.001</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.531865</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">2</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.002</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.554412</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">3</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.003</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.551539</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">4</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.004</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.536430</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">5</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.005</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.561438</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">6</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.006</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.561170</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">7</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.007</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.546326</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">8</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.008</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.562902</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
<tr style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em;"><th style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">9</th><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.009</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0.533140</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">1</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">0</td><td style="border-collapse: collapse; border: 1px solid black; box-sizing: border-box; margin: 1em 2em; padding: 4px; vertical-align: middle;">40</td></tr>
</tbody></table>
<br />
The Morse Encoder software is stored in Github <a href="https://github.com/Morse-Learning-Machine-Challenge/MLMv1/blob/master/ag1le/MorseEncoder.py">MorseEncoder.py</a> and it is open source.<br />
<br />
<h3>
NEXT STEPS</h3>
Now that I have the capability to create proper training material automatically with some parameters, like speed (WPM), fading (QSB) and noise level (sigma) it is a trivial exercise to produce large quantities of these training files.<br />
<br />
My next focus area is to learn more about Recurrent Neural Networks (especially LSTM variants) and experiment with different network configurations. The goal would be to find a RNN configuration that is able to learn how to model the symbols correctly, even in presence of noise and QSB or at different speeds.<br />
<br />
73<br />
AG1LE<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-50222218156097061232015-11-15T17:38:00.000-05:002015-11-15T20:37:06.924-05:00Your next QSO partner - Artificial Intelligence by Recurrent Neural Network? <h3>
INTRODUCTION</h3>
Few months ago <a href="http://cs.stanford.edu/people/karpathy/" target="_blank">Andrej Karpathy</a> wrote a <a href="http://karpathy.github.io/2015/05/21/rnn-effectiveness/" target="_blank">great blog post</a> about recurrent neural networks. He explained how these networks work and implemented a character-level RNN language model which learns to generate Paul Graham essays, Shakespeare works, Wikipedia articles, LaTeX articles and even C++ code of Linux kernel. He also released the code of this RNN network on <a href="https://github.com/karpathy/char-rnn" target="_blank">Github</a>.<br />
<br />
It has been a while since I have experimented with RNNs. At the time I found RNNs difficult to train and did not pursue any further. Well, all that has changed in the last year or so. I installed Andrej's <a href="https://github.com/karpathy/char-rnn" target="_blank">char-rnn package</a> from Github in less than 10 minutes on my Linux laptop using instructions on the Readme.md file. I tested the installation by training the RNN with the Shakespeare's collected texts provided as part of the package.<br />
<br />
If you have GPU graphics card (like NVIDIA Titan) the training goes much faster. I did not have this so I let the training run in the background for over 24 hours on my Lenovo X301 laptop . Looking the results the RNN indeed learned to output Shakespeare like language as Andrej explains in his blog post. It certainly took me more than 24 hours to learn English language and I never learned to write dialogue like Shakespeare. Please note that RNN was a "<a href="https://en.wikipedia.org/wiki/Tabula_rasa" target="_blank">tabula rasa</a>" so it had to learn everything one character at the time - <b>this was pretty amazing result</b>!<br />
<br />
I decided to do an experiment to find out if this RNN technology could be used to build a ham radio robot.<br />
<h3>
TRAINING A HAM RADIO ROBOT</h3>
The robot would have to learn how people make CW QSOs in real life. I collected some 10,000 lines of examples of ham radio CW QSOs from various sources. Some examples were complete QSOs, some were short contest style exchanges and some just calling CQ. The quality of the language model depends on the amount of examples in the training file.<br />
<br />
To do this properly I would need at least a few megabytes of examples, but I found only about 200 kBytes after a few minutes of Internet search. I copied this material into a single file named "input.txt" in data/QSOs directory and started the training with the following command:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">th train.lua -data_dir "data/QSOs" -gpuid -1 </span><br />
<br />
RNN reads the input data, builds a dictionary, configures the neural network and parameters and starts the training. During training the software outputs examples in the cv directory. This allows you to see how RNN learns from the provided material.<br />
<br />
After only 12 epochs the RNN had already learned some basic concepts like call signs and basic words, but it was not enough training rounds to learn English or proper syntax of CW QSO. <br />
<br />
<span style="background-color: #6fa8dc;">GG3QXP DE WA1AXP/M K</span><br />
<span style="background-color: #6fa8dc;"><br /></span>
<span style="background-color: #6fa8dc;">RARR WIS DE M8BPY RR NAMME FDO RR0S ANOEN CUX/I AO RNEL SNETSL CL ATRSO IANLOED UR RV SI MEE DO RU @ UR RSR IX NUOR IS QS IN WIEZ DE QEMUM/7 W</span><br />
<div>
<br /></div>
<div>
<br /></div>
<div>
After 25 epochs the RNN had reached to the contest section (November Sweepstakes) of the input file. Note that the call signs below (like VE7LP or K2XC) were not included in the <a href="http://k7og.net/download-my-cw-practice-source/" target="_blank">original training file</a>. RNN has learned the proper call sign syntax from given examples and is generating this text below:</div>
<div>
<br /></div>
<div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">VE7LP 314 Q 52 SK=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">K2XC 153 M 68 WNY=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">K8CSC 285 A 15 MI=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">AE5MX 341 Q 47 SB=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">AE7J 419 A 06 WWA=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">K5RMC 137 M 43 OK=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">KB2QUT 34 U 21 NNY=</span></span></div>
<div>
<span style="background-color: #6fa8dc;"><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">K9MIP 120 M 08 WI=</span></span></div>
</div>
<div>
<br /></div>
Wow...RNN has also learned <a href="http://lists.contesting.com/_cq-contest/2007-11/msg00349.html" target="_blank">November Sweepstakes exchange</a> from provided examples. Even the <a href="http://www.arrl.org/contest-sections-list" target="_blank">ARRL sections</a> are correct. Note that RNN is generating these based on learned rules, not just copying provided examples. All this by reading the input text file one character at the time.<br />
<br />
After 39 epochs RNN has learned many CW acronyms, RST report and some basic QSO structure, though there is still a lot of nonsense in the output. RNN talks about name, QTH, weather (WX) and even RIG and antennas:<br />
<br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">GUDXTEN/8 IN = NAME HR IS ED ED QTH IS CHAPOTON FNCI HOO OO DINED MIAD VALT W FO FON CUR DS MS ES TOT FER CL IW QSO OB ULLOO = QRHPO DE LOOFSD SORK/ISTO= = = RIG HR IS LACKORI WAZH PA WANFOO = WX WR = 2IT WINLE WOOD DES UP 55 FE HW? + MJ1GJO DE MJ3ASA K</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> GUD DX ES 73 G8XFO DE 2E3CUD/9RP @ </span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> KC6XQ DE M5WMM/M DE M1TGL/M K</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> W63ED DE M5YUE</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">VVV VVV</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">CQ CQ CQ DE WA1NX/WA50 WB4AJH/6 KC0AHH K1 WAJH K</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">WA3JRC DE W4DD/5MM DE KC3GJJ/8 K</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> GV8SDE DE 2I8APZ GD ENZ/QRP GD3BOB </span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> G1KHC DE G3ECQ/QCP M7Y</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">VVV VVVVV</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">CQ CQ CQ DE W3ARJ/0 W11IA DE M9TFO/P WA2MJH/4 K</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;">1BJ0 WA3BH DE MJ1GDS MJ6XW K</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> MW6PJP/M DE MW6QIC UR RST HR 459 W QSB M5YES DE 2E6AO QRS 6NT QSO D = RIG HR IS K70 WNTN = NAME HR IS JIM JEC QTH IN COLLA NCME HO FEL PU DS MS ES 2INTOOBA MCONTS = = UR RRTR MNNI W IS HOX WBOO ISTEL INYD = =</span><br />
<span style="background-color: #6fa8dc; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> GUD RIS HL W0 IN W WOME DEN UTD PO P =</span><br />
<div>
<br /></div>
At this point RNN has seen the complete training material only 39 times. It has figured out many rules (like ending its turn on "K") but it has still problems with vocabulary, how to use call signs properly, etc.<br />
<br />
After 50 epochs the RNN had already mastered idioms such as TNX FER NICE QSO CU AGN and structure resembles a real QSO:<br />
<br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">KC2RD/4 DE W1GJV K</span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> W1ARD DE KC2RD/3RO UR RST HR 459 W QSB GD3DIA DE GD9GEN @ CD GG2RD/9 DE W1GRV/4 DE GU5TCH/MM R TNX FER NICE QSO CU AGN </span><br />
<span style="background-color: #3d85c6; font-family: 'courier new', courier, monospace; font-size: x-small;">M2YXT DE GD5FM UR RST HR 529 W QSB W1GRD DE W1GRR RR K</span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">GG TI TE UR 33 </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">IWAMO DE WA6EN </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">KC2X DE W1YDH KE9NZE/0 OL TU </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">UR RST HR 309 W QSB = NAME HR IS AANNY WAVEL FNH COTE TNX QST </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">= UR 7S PR =</span><span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> UR RST HR 599 W QSB = HR VY NERVOUS D DE MYUE USD 1S = </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">NAME HR IS DI EESTY == RIG HR IS HEATH 71 INTO A NME HR IS VILL HW? </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">2E9AAT DE GW6QI UR TS TAX DEL NAME H5 UE EU 539 FE KHHJ RS 2E MES LANNY = </span><br />
<span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;">QRY = NAME HR IS ED ED </span><span style="background-color: #3d85c6; font-family: "courier new" , "courier" , monospace; font-size: x-small;"> QTH IS PARD VORETP</span><br />
<br />
You can also see that some parts (like NAME HR ) are repeating multiple times. This was also noted by Andrej in his experiments. Since the training is done one letter at the time, and not word by word the RNN doesn't really get the context of these phrases.<br />
<h3>
PRACTICAL APPLICATIONS</h3>
<div>
This kind of ability to provide predictive text based on language models is widely used in many Internet services. When you type letters into Google search bar it will provide you alternatives based on prediction that has been learned from many other search phrases. See figure 1 below. </div>
<div>
<br /></div>
<div>
<br /></div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjICWl9p-iLWEoKrvgZVjEGPF8ZNzU6wQgJhPcVL15LJ9YRezMeijSmkdjZwe4y67yWeGUvUYyIMavhZT1ZDWybFFN_BpKzuQDSrv1ety2vqlAutHQidh-B7yA9r8AO-73Wpgx_B4IS_U/s1600/predictive_text.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="115" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhjICWl9p-iLWEoKrvgZVjEGPF8ZNzU6wQgJhPcVL15LJ9YRezMeijSmkdjZwe4y67yWeGUvUYyIMavhZT1ZDWybFFN_BpKzuQDSrv1ety2vqlAutHQidh-B7yA9r8AO-73Wpgx_B4IS_U/s640/predictive_text.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. Predictive search bar</td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<span style="background-color: #eeeeee;"><br /></span>
<br />
<br />
<br />
<br />
<br />
<br />
In the same manner RNN could provide a prediction based on characters entered so far and what it has learned from previous materials. This would be a useful feature for example in a Morse decoder. Also, building a system that would be able to respond semi-intelligently for example in a contest situation seems also feasible based on this experiment.<br />
<br />
However, there is a paradigm shift when we start using Machine Learning algorithms. In traditional programming you write a program that uses input data to come up with output data. In Machine Learning you provide both input data and output data and computer creates a program (aka model) that is then used to make predictions. See figure 2. below to illustrate this.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirCUp9XTey7_kIMPX_jGwZPzJma4xhzT5_LB9c07hb_p_s0LQE7Q-O1NdURCr1d4xcfe9AnqYlfKVn5h9SlQoVyq2x4deGzz7LLhP0oLsHuWT8BztU0k68f5Y0b4QIovaCiVEUhlmpOg4/s1600/machinelearning.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirCUp9XTey7_kIMPX_jGwZPzJma4xhzT5_LB9c07hb_p_s0LQE7Q-O1NdURCr1d4xcfe9AnqYlfKVn5h9SlQoVyq2x4deGzz7LLhP0oLsHuWT8BztU0k68f5Y0b4QIovaCiVEUhlmpOg4/s400/machinelearning.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. Machine Learning paradigm shift</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
To build a ham radio robot we need to start by defining the input data and expected output data. Then we need to collect large amount of examples that will then be used to train the model. Once the model is able to accurately predict correct output you can then embed it into the overall system. Some systems will continuously learn and update the model on the fly.<br />
<br />
In the case of ham radio robot we could focus on automating contest QSOs since the structure and syntax is well defined. In the experiment above RNN learned the rules by seeing the examples only 25 times. So the system could be monitoring a frequency, perhaps sending CQ TEST DE <MYCALL> or something similar. Once it receives a response it would then generate the output using the learned rules and would wait for acknowledgement and log a QSO.<br />
<br />
If the training material covers enough "real life" cases, such as missed letters in call signs, out of sequence replies, non-standard responses etc. the ham radio robot would learn to act like human operator and quickly resolve the issue. No extra programming needed, just enough training material to cover these cases.<br />
<h3>
CONCLUSIONS</h3>
Recurrent Neural Network (RNN) is a powerful technology to learn sequences and to build complex language models. A simple 100+ line program is able to learn complex rules and syntax of ham radio QSOs in less than 50 epochs when presented only a small number of examples ( < 200 kBytes of text).<br />
<br />
Building a ham radio robot to operate a contest station seems to be within reach using normal computers. The missing piece is to have enough real world training material and to figure out an optimal neural network configuration to learn how to work with human CW operators. With the recent advances of deep learning and RNNs this seems an easier problem than for example trying to build an automatic speech recognition system.<br />
<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-20539116337798040412015-09-11T23:55:00.002-04:002018-08-15T11:00:32.602-04:00Happiness FormulaMany people have tried to express human happiness in a mathematical formula. One of my personal favorites is <a href="http://dilbertblog.typepad.com/the_dilbert_blog/2007/03/happiness_formu.html" target="_blank">created by Scott Adams</a> (Dilbert fame). However, after many deep thoughts and a few drinks with my buddies I have concluded that Scott did not get the formula quite correct.<br />
<br />
The correct and official Happiness Formula is shown in Figure 1. below<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinFWiif_60Qfxo4fq-GcH154nVzWgb4xFH8SmTA8hjvPHb4HRF7PLjxIYsfV1GK0wLbVsI-2PDQeaB4805KPJQcxknBeVq_9ZDdmm2x-v7Z2TnHiaRWZQeLd0agFB1zw-F_luHV7OakBM/s1600/Screen+Shot+2015-09-12+at+12.33.14+PM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="68" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinFWiif_60Qfxo4fq-GcH154nVzWgb4xFH8SmTA8hjvPHb4HRF7PLjxIYsfV1GK0wLbVsI-2PDQeaB4805KPJQcxknBeVq_9ZDdmm2x-v7Z2TnHiaRWZQeLd0agFB1zw-F_luHV7OakBM/s640/Screen+Shot+2015-09-12+at+12.33.14+PM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. Happiness Formula</td></tr>
</tbody></table>
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
While Scott tried to explain happiness as a linear combination of each component he missed a few important points.<br />
<br />
<b>Integral over time </b>- human happiness varies over time. Happiness is a fragile mental state that can easily go up or down. True happiness must be an integral over the observation time period. The time period could be one fantastic night out with good friends celebrating your promotion or over several months when you are fighting for your life in a cancer treatment center. It could also be over a lifetime when you are on your death bed thinking of your life and all the happy experiences. It can also be over the time period when you fell madly in love, got married and eventually divorced. When you select a different time horizon, you end up with a different happiness value.<br />
<br />
<b>Normalization</b> - to be able to measure happiness you need to normalize the value by dividing the sum of components with expectations. If you expect the world you might not be happy even with the greatest partner or having billion dollars in your pocket. Your happiness depends on your expectations; winning a million dollars in a lottery when you least expect will boost your happiness for a while. If you expect to win 2 millions but you only get 1 million you will be disappointed. A small kid visiting DisneyWorld for the first time is super happy about the experience; an adult visiting same place for the 5th time gets easily bored and is not very happy.<br />
<br />
<b>Individual Coefficient</b> - C<span style="font-size: x-small;">i </span>also known as "Ida's constant" by the Finnish waitress who validated Happiness Formula after our happy group had spent significant effort and had many drinks to formulate happiness. This coefficient scales the happiness value for each individual to comply with International Unit of Happiness, aka "Anand". <br />
<br />
<b>Standardized Unit</b> - Anand (<a href="http://dict.hinkhoj.com/words/meaning-of-%E0%A4%86%E0%A4%A8%E0%A4%A8%E0%A5%8D%E0%A4%A6-in-english.html"> आनन्द </a>) is a Hindi word for happiness. One Anand is the unit of happiness, much like Tesla is the unit of magnetic flux density.<br />
<br />
So there you have it, a mathematically rigorous formula of Happiness.<br />
<br />
In the next post we focus on measurement techniques of Happiness and how to calibrate your measurement system against the International Anand standard held in safety in our Boston based laboratory.<br />
<br />
Until next time.<br />
<br />
Mauri<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-44046664991118585622015-09-10T02:13:00.001-04:002016-06-30T08:45:49.539-04:00Internet of Things (IoT) - hype vs. reality Over the last few years the hype around "Internet of Things" (IoT) has been growing rapidly. According to <a href="https://www.gartner.com/doc/3098434" target="_blank">Gartner Hype Cycle 2015</a> IoT is peaking currently. Assuming IoT follows this cycle this would mean that we are at the peak of inflated expectations and heading towards the through of disillusionment. See figure 1. below to see how IoT concept is tracking on the hype cycle.<br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://media.licdn.com/mpr/mpr/shrinknp_800_800/AAEAAQAAAAAAAAIKAAAAJGZkZmVmZjA1LTBiYjktNGY1Ni1iN2U3LTM0ZTMwNTQwN2M2Mw.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://media.licdn.com/mpr/mpr/shrinknp_800_800/AAEAAQAAAAAAAAIKAAAAJGZkZmVmZjA1LTBiYjktNGY1Ni1iN2U3LTM0ZTMwNTQwN2M2Mw.jpg" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. "Peak of Expected Inflated Expectations"</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I wanted to learn more about the IoT technology and do some concrete experiments to better understand what IoT can offer. I found the <a href="https://store.particle.io/?product=particle-photon" target="_blank">Particle Photon</a> board that is a small $19 Arduino compatible U.S. quarter size board with WiFi enabled Internet connectivity very suitable for prototyping some IoT ideas. See fig 2. below to get a sense of the size of this tiny board. I ordered two of these just to play a bit and try to build something useful out of these.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://pbs.twimg.com/media/CM9ZBttUkAA2rtN.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="160" src="https://pbs.twimg.com/media/CM9ZBttUkAA2rtN.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 2. Particle Photon board with and without breadboard headers</td></tr>
</tbody></table>
<h3>
<br />HUMIDITY CONTROLLER EXPERIMENT</h3>
I already have some previous experience working with Arduino compatible boards such as <a href="https://www.sparkfun.com/products/11114" target="_blank">Arduino Pro Mini</a> that is physically almost the same size of Particle Photon. In fact I used that board to build a simple humidity controller in our bathroom. This was a quick weekend project where I prototyped on a breadboard a simple circuit with a humidity sensor, a LED indicator and a relay driver to turn the bathroom fan on and off. I assembled the prototype parts including the breadboard, sensor, relay unit and 12v/5V power supply inside an Apple mouse plastic enclosure. See Fig 3.<br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNdQTx_9_F3hq4CX1boC50X42N5TrVu_GntjB98t_WfRIepejFie8qHcgRAnuYxPL-V2McWcvNYeIlnvXovQQbe2jlCZk45Je6gokHMurGUoFGzVxn8s5Zcrs52S_yhzYO3jC4KQqZW4c/s1600/IMG_20150830_132731.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNdQTx_9_F3hq4CX1boC50X42N5TrVu_GntjB98t_WfRIepejFie8qHcgRAnuYxPL-V2McWcvNYeIlnvXovQQbe2jlCZk45Je6gokHMurGUoFGzVxn8s5Zcrs52S_yhzYO3jC4KQqZW4c/s400/IMG_20150830_132731.jpg" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 3. Arduino based humidity controller prototype.</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
With a few holes drilled on this plastic enclosure to allow air to flow over the sensor I was able to fit the whole controller inside an existing vent box, see Fig 4. below. <br />
<div>
<br /></div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyQc8kYFyBdWPxMssxj8sxWB6CvtNu4TCfAzaRddrz4TbsWDts_tSxnYWCHhLAz_ZodzuaXFKG-kGn41AV3q8x4ZjQZcyBAJbbZ_jOLcJupxIo1sTP7R4fmAFRv_6tHdWpp1traAo5LHs/s1600/IMG_20150830_180834.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyQc8kYFyBdWPxMssxj8sxWB6CvtNu4TCfAzaRddrz4TbsWDts_tSxnYWCHhLAz_ZodzuaXFKG-kGn41AV3q8x4ZjQZcyBAJbbZ_jOLcJupxIo1sTP7R4fmAFRv_6tHdWpp1traAo5LHs/s400/IMG_20150830_180834.jpg" width="300" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 4. Humidity controller installed inside the vent box.</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
However, I did have a problem with this simple controller. The few lines of software that I wrote in winter time when relative humidity is normally quite low worked very well for many months but during summer months when relative humidity is much higher the software didn't work that well. Debugging this kind of embedded software is not that easy. I disassembled the prototype for 3 times uploading yet another software version but the damn thing kept starting the vent fan in the middle of night or at some random time.<br />
<h3>
<br />INTERNET ENABLED HUMIDITY CONTROLLER </h3>
I had to find a solution to this problem so when I learned about the Particle Photon board I knew that this might just be the solution. After reading some of the documentation I was pretty sure that having Internet connection would not only help me to debug the problem but also saves me a lot of trouble, as Particle Photon allows you to install the new firmware over the air. So I wouldn't have the get the vent box open, remove all wires, flash the Arduino board with new software and assemble everything back together. <br />
<br />
After connecting the Particle Photon board to my Wifi and adding the device on Particle.io web IDE, I used Particle.io web based software development environment (see Fig 5. below) to edit and debug the humidity controller software. I could just simply edit, compile new code, press a button and install firmware almost instantly over the Internet using the WiFi connection on this Photon board.<br />
<br />
How cool is this? <br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkXNs_U7rB6E1n6GFwMXFg1yiO4a4vxLE8HvjbVnx5Hdw7iPbr7TrTyIx5l45Ov7-jyqUobmQ42bi5cn7B2MdGX2C5UozZV4I6j4eQwSnnSul54rpexQiJQ-sSwuuM1t5zJO0hkEiMiGE/s1600/Screen+Shot+2015-09-10+at+1.25.26+AM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="368" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkXNs_U7rB6E1n6GFwMXFg1yiO4a4vxLE8HvjbVnx5Hdw7iPbr7TrTyIx5l45Ov7-jyqUobmQ42bi5cn7B2MdGX2C5UozZV4I6j4eQwSnnSul54rpexQiJQ-sSwuuM1t5zJO0hkEiMiGE/s640/Screen+Shot+2015-09-10+at+1.25.26+AM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 5. Web based software development environment</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Particle.io provides also excellent API that have simple to use Internet enabled functions that you can incorporate in your own software. In my case I wanted to debug how the humidity sensor behaves when you have a transient increase in relative humidity when taking a shower. I used the <a href="https://www.sparkfun.com/products/12064" target="_blank">HTU21D </a>sensor from Sparkfun. Easy way to debug is to publish your sensor data to the Internet using a simple function like <span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">Spark.publish("RH_temp",str,60,PRIVATE); </span><br />
<br />
You can use the <a href="https://dashboard.particle.io/user/logs" target="_blank">Particle Dashboard</a> to view the sensor data in near real time. This was almost too easy to describe on a blog like this.<br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJtqmt9P3SMznAwqnSchmX4bq_g0rMk68pxLR_9S8xzQ6f9uDsv9tEZFcouGipGYyf21LQa6sIuvhXvHgdH5GXi8NMA8-PbQUJyo3DJpCekPgc-xDLllNK2ah8vNnajiOWaIKmnlTKmng/s1600/Screen+Shot+2015-09-10+at+1.39.32+AM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="264" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJtqmt9P3SMznAwqnSchmX4bq_g0rMk68pxLR_9S8xzQ6f9uDsv9tEZFcouGipGYyf21LQa6sIuvhXvHgdH5GXi8NMA8-PbQUJyo3DJpCekPgc-xDLllNK2ah8vNnajiOWaIKmnlTKmng/s640/Screen+Shot+2015-09-10+at+1.39.32+AM.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 6. Particle Dashboard</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
You can use of course use your own web or mobile applications to read
and write data as well as control the input/output pins on the Photon
board.<br />
<br />
I used a simple API command line call to capture the sensor data for plotting: <br />
<br />
<span style="font-size: x-small;"><span style="font-family: "courier new" , "courier" , monospace;">curl -k https://api.particle.io/v1/devices/<your device id>\/events/?access_token\=<your access token> >photon_data.txt</span></span><br />
<br />
Figure 7. below shows the relative humidity transient after taking a shower and then the decline as the vent fan is running. You can also see the small temperature increase when the hot water is running. When looking at the data I realized that my RH% threshold had been too small. When I increased the threshold value the controller started working much better. Being able to extract the sensor data and publish it over the Internet made a big difference in debugging the original problem. <br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk_wYj_ctY_oadFM3ScikVYwTBqDe1rob3Cil7Z3dySyK7WS-NRqPSIjn8oaOQ2es1YoGzdAaeywpD7wKnGpHJDPS1yTL0s1CW_J268OBIb172RoVo5vO6ukYTd-GkvgAElKCXOMmzB_Y/s1600/humidity_controller.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="297" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk_wYj_ctY_oadFM3ScikVYwTBqDe1rob3Cil7Z3dySyK7WS-NRqPSIjn8oaOQ2es1YoGzdAaeywpD7wKnGpHJDPS1yTL0s1CW_J268OBIb172RoVo5vO6ukYTd-GkvgAElKCXOMmzB_Y/s640/humidity_controller.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 7. RH% delta and Temperature over time</td></tr>
</tbody></table>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"></span>
<br />
<br />
<h3>
</h3>
<h3>
PLOTTING </h3>
In order to collect more data and have a dashboard to plot and review the measurements I signed up for a free account at <a href="https://thingspeak.com/channels/public">ThingSpeak.</a> You get an API key and channel number. With these you can plot the values with a simple API call: <br />
<span style="font-family: "Courier New",Courier,monospace;">ThingSpeak.writeFields(myChannelNumber, myWriteAPIKey);</span><br />
<br />
Fig 7 and 8 below show the sensor data plot. Relative humidity peaks at 100% when taking a shower but since the fan is turned on almost instantly the humidity starts to drop quickly back to normal. You can see also a small increase in temperature at the same time. The drop in temperature is due to A/C that turns on at 6:00 AM. <br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyNAYSjVl3qvjHEEHJ_6eHGIT_qo6BoY1U4q_3WFcfdsVlb-FZxQOQM23LK6rZCxHccPH54ZrxxjWkcbBlW6mMe_mXp7dOqdQkJAlba9LV0s8xvURm-A0XIj4JfRQmQo0WjlTf93OUp8M/s1600/Screen+Shot+2016-06-30+at+8.35.07+AM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="273" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhyNAYSjVl3qvjHEEHJ_6eHGIT_qo6BoY1U4q_3WFcfdsVlb-FZxQOQM23LK6rZCxHccPH54ZrxxjWkcbBlW6mMe_mXp7dOqdQkJAlba9LV0s8xvURm-A0XIj4JfRQmQo0WjlTf93OUp8M/s400/Screen+Shot+2016-06-30+at+8.35.07+AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 7. Relative Humidity plot showing a peak </td></tr>
</tbody></table>
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnsGVVbAazYkP7G116IeTrvmwJ1YRsqqGOuVrgM4Y9zKjP_11R-JLXEWas5P_JAxDPt9UeCf90uaodw6k_1YErlBonIIMtlE2bSR3XjunT2uWasIgK0SfpfZNxJOVFvBGXBOqYFC6rs-c/s1600/Screen+Shot+2016-06-30+at+8.35.14+AM.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" height="271" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnsGVVbAazYkP7G116IeTrvmwJ1YRsqqGOuVrgM4Y9zKjP_11R-JLXEWas5P_JAxDPt9UeCf90uaodw6k_1YErlBonIIMtlE2bSR3XjunT2uWasIgK0SfpfZNxJOVFvBGXBOqYFC6rs-c/s400/Screen+Shot+2016-06-30+at+8.35.14+AM.png" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 8. Temperature plot</td></tr>
</tbody></table>
<br />
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
</h3>
<h3>
CONCLUSIONS </h3>
My quick foray into the world of "Internet of Things" took me about 2 hours on a Sunday afternoon. Using the latest Particle.io Photon board and the web based IDE I was able to convert my existing Arduino based humidity controller to an Internet enabled controller that is publishing sensor data in near real time and allows me to update the software over the air. <br />
<br />
This whole project felt like too easy - I expected that building IoT prototypes would be much harder but at least for this simple use case it took a novice like myself only a short time to solve a real world problem. Now the bathroom vent works as expected and humidity is under control. <br />
<h3>
<br />APPENDIX - SOFTWARE </h3>
The current software version is listed below. As you can see this is not rocket science - a few lines of code and you have an Internet connected sensor / controller. <br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">// This #include statement was automatically added by the Particle IDE.</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#include "HTU21D/HTU21D.h"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#include "application.h"</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">/* </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> HTU21D Humidity Controller</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> By: Mauri Niininen </span><span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">(c) Innomore LLC</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Date: Aug 30, 2015</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Uses the HTU21D library to control humidity using a fan.</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Hardware Connections (Breakout board to Photon)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -VIN = 5.3 V</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -VCC = 3.3 V</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -GND = GND</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -SDA = D0 (use inline 330 ohm resistor if your board is 5V)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -SCL = D1 (use inline 330 ohm resistor if your board is 5V)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> -RLY = D3 relay board </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> */</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#define HOUR 3600/10 // 1 HOUR in seconds - divide by loop delay 10 secs</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#define HRS_24 24 // 24 hours of history </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">// define class for 24 hr relative humidity </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">class RH24 {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">private:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float rh24[HRS_24]; // Keep last 24 hours of humidity </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> int counter;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> int index; </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">public:</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> void init(float RH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> void update_h(float RH); </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float avg(float RH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">};</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">//Create an instance of the objects</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">HTU21D mh;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">RH24 rh; </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">// for time sync</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">#define ONE_DAY_MILLIS (24 * 60 * 60 * 1000)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">unsigned long lastSync = millis();</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">void setup()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> pinMode(D7, OUTPUT);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> pinMode(D3, OUTPUT);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> while (! mh.begin()){</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D7,HIGH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> delay(200);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D7,LOW);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> delay(200);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // turn on the fan</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D3,HIGH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // initialize sensor </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> rh.init(mh.readHumidity()); </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> delay(5000);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // turn off the fan</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D3,LOW);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">// MAIN PROGRAM LOOP </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">void loop()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">{</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // read sensor humidity and temperature </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float humd = mh.readHumidity();</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float temp = mh.readTemperature();</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float avg = rh.avg(humd);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float delta = humd - avg;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // convert data to string</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> String h_str = String(humd,2);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> String t_str = String(temp,2);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> String d_str = String(delta,2);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> String avg_str = String(avg,2);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> String tm_str = String(millis());</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> String str = String(h_str+":"+t_str+":"+d_str+":"+avg_str+":"+tm_str);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // if relative humidity increases over 12% vs. 24 hour average, turn on the fan </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if (humd - avg > 12.0) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D7,HIGH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D3,HIGH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Spark.publish("RH_temp","ON",60,PRIVATE);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> else {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D7,LOW);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> digitalWrite(D3,LOW);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // time sync over Internet once a day</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if (millis() - lastSync > ONE_DAY_MILLIS) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // Request time synchronization from the Particle Cloud</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Spark.syncTime();</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> lastSync = millis();</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> // Send a published string to your devices...</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> Spark.publish("RH_temp",str,60,PRIVATE);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> delay(10000);</span><br />
<br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">void RH24::init(float RH){</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> counter = 0;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> index = 0;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> for (int i = 0; i < HRS_24; i++)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> rh24[i] = RH;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">void RH24::update_h(float RH) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> counter += 1;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if (counter > HOUR) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> counter = 0; </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> rh24[index] = RH; </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> index += 1;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> if (index >=HRS_24) </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> index = 0;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> }</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">} </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">float RH24::avg(float RH) {</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> update_h(RH);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> float sum = 0.0;</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> for (int i = 0; i < HRS_24; i++)</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> sum += rh24[i];</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> return (sum/HRS_24);</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">}</span><br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-72747033791300921992015-04-01T11:20:00.000-04:002015-04-01T11:45:57.851-04:00El-bug: a novel Morse decoder based on cockroach neural circuitsApril 1, 2015<br />
<br />
I have been working on a project to harness the power of biological neural circuits into a practical novel solution for digital communications. I decided to call this project "El-bug" as my focus was to find out how fast biological neural circuits can learn to decode Morse code.<br />
<br />
Biological neural circuits have some amazing properties compared to computer based artificial neural networks. State of the art deep learning algorithms require millions of data points and hours or days of repetitions to learn patterns in the data, where as biological neural circuits can often learn new patterns using only a few examples and in time scale of tens of milliseconds. Based on literature biological neural circuits are also adaptive and work well under noisy real world signals. <br />
<br />
Computer based learning algorithms require expensive hardware to store gigabytes of training data, GPUs to accelerate the learning process and complicated electronics to convert real world signals into digital pictures, audio or other representations. In comparison biological neural circuits are very small, typically come pre-integrated with sensory organs and require very little power in form of cheap organic energy sources such as glucose.<br />
<br />
The hardware used in this project is based on Arduino and the components are available at less than $50 from multiple sources online. The biological neural circuits of American cockroach (Periplaneta americana) is used to power the computation. These fascinating insects are readily available from many sources at low cost, or sometimes even free of charge.<br />
<br />
<h3>
SYSTEM ARCHITECTURE </h3>
The overall system architecture is shown in Figure 1 below. Arduino Pro Mini (3.3 V/8 MHz) has analog and digital interfaces and it is connected to a RFDuino Bluetooth module. Interface to cockroach neural circuitry is done using analog amplifier with frequency response designed for capturing bio-electrical neural spike signals. Digital output lines are used to provide electrical stimulation of the nerves.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDKVZZJqInzir9mMZGUXPqEllJ26iNBO50lpp8X8zo8qcp_9jp5zHbry6yvnFp7Z4GbUZSDjW2aCctL_APLvT9ZPwUGldHuyhz6YihKOL3KvlnoK7li3iACloXu32VkuqMmcjH4hiOaoM/s1600/Slide1.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDKVZZJqInzir9mMZGUXPqEllJ26iNBO50lpp8X8zo8qcp_9jp5zHbry6yvnFp7Z4GbUZSDjW2aCctL_APLvT9ZPwUGldHuyhz6YihKOL3KvlnoK7li3iACloXu32VkuqMmcjH4hiOaoM/s1600/Slide1.jpg" height="480" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. El-bug Morse decoder system architecture</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
COCKROACH ANATOMY AND NEURAL CIRCUITRY</h3>
There is a surprising amount of research available on the neural circuitry of <i>Periplaneta</i><i> </i><i>americana</i>. For example t<a href="https://backyardbrains.com/experiments/teachersguide/spikerbox" target="_blank">his source</a> explains:<br />
<i>"The anatomy of the cockroach is exceptionally accessible to electrophysiological experimentation for a variety of reasons. First, from the dorsal, or top, view the cockroach has a distinctive prothorax (the section directly behind, and shielding the head) and wings that give the cockroach its distinctive armored look. When flipped on its back, the ventral aspect of the cockroach reveals the basic segmented body sections distinctive of insects: the head, thorax, abdomen, and legs." </i><br />
<br />
See Figure 2 for details of cockroach anatomical features.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcRYIqzSQ9R7aP_C4Ns34VC9bjSyWjq8zl-l-7CbS3o3K1fMpXHX" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="300" src="https://encrypted-tbn1.gstatic.com/images?q=tbn:ANd9GcRYIqzSQ9R7aP_C4Ns34VC9bjSyWjq8zl-l-7CbS3o3K1fMpXHX" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. Cockroach Anatomy</td></tr>
</tbody></table>
<br />
<br />
After studying possible circuits to utilize I decided to focus on "escaping behavior" that the common cockroach (Periplaneta americana) exhibits. This is a robust behavior of turning away from wind puffs (Camhi et al. 1978). This behavior is termed “escaping behavior” since it is the initial movement when escaping from predators. This<a href="http://ediacaran.mech.northwestern.edu/neuromech/index.php?title=Roach_Lab_Spring_2010-_Dickinson&oldid=6002" target="_blank"> source</a> explains the detailed mechanism and neural circuitry in use:<br />
<i>"Understanding the anatomy of the cockroach nervous system is helpful when examining this escape behavior. The ventral nerve cord (VNC) of the cockroach is along its underbelly, rather than the dorsal side where the nerve cord of most vertebrates is located. The VNC is composed of several giant interneurons (GIs) and at the terminal ganglion afferents project to the dendrites of these GIs.</i><br />
<i>To detect wind directions, the cockroach has two cerci that are covered by numerous filiform hairs located at the rear of its body (Figure 1). Mechanoreceptors are attached at the base of the filiform hairs and are sensitive to wind puffs. Afferents send the neural signal from the mechanoreceptors to the terminal ganglion and thus provide input to the GIs. Due to its specific location and orientation on the cerci, each mechanoreceptor is sensitive to wind puffs from a specific direction relative to the cockroach. Afferents that are sensitive to similar wind directions are located close to each other within the abdominal ganglion." </i>Figure 3. from the same source shows typical measurement results as explained in the experiment.<br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://ediacaran.mech.northwestern.edu/neuromech/img_auth.php/thumb/6/6c/Lindsay_figure.png/800px-Lindsay_figure.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://ediacaran.mech.northwestern.edu/neuromech/img_auth.php/thumb/6/6c/Lindsay_figure.png/800px-Lindsay_figure.png" height="393" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3. Typical afferent response </td></tr>
</tbody></table>
<br />
The Arduino Pro Mini provides a low cost circuitry to measure the neural responses and it has 4 analog to digital channels readily available. An analog pre-amplifier such as in <a href="https://www.backyardbrains.com/products/spikerBox" target="_blank">Spikerbox </a>can easily produce voltage levels required by Arduino ADC. This is a 10 bit ADC and provides 3.2 mV resolution. According to this <a href="http://arduino.cc/en/Reference/analogRead" target="_blank">source</a> a single ADC read takes about 100 microseconds that is adequate speed for this purpose. The goal here is to try to establish a clear differentiation between responses to two types of electrical stimulation, similar to [<a href="http://ediacaran.mech.northwestern.edu/neuromech/index.php?title=Roach_Lab_Spring_2010-_Dickinson&oldid=6002" target="_blank">Yu-Wei 2010</a>] - see Figure 4 as example. <br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://ediacaran.mech.northwestern.edu/neuromech/img_auth.php/thumb/d/da/Trial_1%263_Spike%26Amp.png/850px-Trial_1%263_Spike%26Amp.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://ediacaran.mech.northwestern.edu/neuromech/img_auth.php/thumb/d/da/Trial_1%263_Spike%26Amp.png/850px-Trial_1%263_Spike%26Amp.png" height="375" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 4. Neural responses to stimulation</td></tr>
</tbody></table>
<a href="https://www.blogger.com/"></a><span id="goog_1001557894"></span><span id="goog_1001557895"></span><br />
<br />
<h3>
MORSE DECODING PROBLEM </h3>
So given above how could we build a functioning Morse code decoder using these key components? The schema is shown in Figure 5. Using the Bluetooth module we are sending audio containing noisy Morse code audio as streamed data stream to Arduino. The software in Arduino does very basic signal processing, calculates the envelope of the audio signal and after low pass filtering generates stimulus signals that are sent using digital output lines to cockroaches that are organized in a 4 level hierarchy corresponding to the alphabet. If numerals would be included we would need a fifth layer. <br />
<br />
At each level of the hierarchy the corresponding cockroach responds to electrical stimulus and based on a learned reaction will emit either "dit" or "dah" response. These "dit" and "dah" reactions are collected using the 10 bit ADC from 4 analog channels in Arduino and are organized as a sequence. Once the complete character has been received a simple "best matching unit" lookup is performed by Arduino and corresponding matched letter is sent using the serial interface over Bluetooth. <br />
<br />
Implementing this scheme in Arduino Pro Mini did take some effort as available RAM memory is only 2 kilobytes. After about 2 weeks of coding effort I managed to squeeze all the functionality in and have still some 343 bytes of RAM free.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5-rt4MFBqv1fNpV5lyPujg2slm7HsMRL3dwmVxphs6zneRhldJjXk4FkGnz8UwEYaHua0n6ySAFcFIJ8QWJj39wNykvLoGi7T-Jaw9rjnGuXjXqxbAuoSEKdNGUXJZQ8zQaP5Tmo23Ps/s1600/Slide2.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5-rt4MFBqv1fNpV5lyPujg2slm7HsMRL3dwmVxphs6zneRhldJjXk4FkGnz8UwEYaHua0n6ySAFcFIJ8QWJj39wNykvLoGi7T-Jaw9rjnGuXjXqxbAuoSEKdNGUXJZQ8zQaP5Tmo23Ps/s1600/Slide2.jpg" height="481" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 5. Morse decoding schema using cockroach hierarchy</td></tr>
</tbody></table>
<br />
<h3>
EXPERIMENTAL RESULTS </h3>
<br />
I did run a 20 hours of tests using El-bug Morse decoder. I compared the character error rate (CER) to signal to noise ratio (SNR) of the audio files with <a href="http://ag1le.blogspot.com/2014/06/new-morse-decoder-part-4.html" target="_blank">the previous results</a> achieved using Bayesian Morse decoder. The results are shown in Figure 6 below. I had to stop the experiment after 20 hours as 2 of the 4 cockroaches got tired of constant stimulation. They seem to have maximum decoding rate of 100 words per minute. <br />
<br />
Surprisingly the decoding accuracy of El-bug system appears to to be quite a lot of better compared to my previous records. With decent signal to noise ratio (> 12 dB <complete id="goog_1557236985">@500Hz) </complete> the decoding accuracy approaches 99%. Even at lower SNR values El-bug outperforms any known machine learning algorithms that I have tested so far. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEK4HynUMcSs0dqs2OG0mYw7ffZLYk8l7wBl3zjphBeFp5EPhOqDCirVHzkDsv8XOZFD7pzoRWqyWlVuhmAoQD82dtLHGdK9khUZPUm2jJazTTxyaxnk7MpMmUD5OQYq1jkHElU1vnSeY/s1600/Slide3.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEK4HynUMcSs0dqs2OG0mYw7ffZLYk8l7wBl3zjphBeFp5EPhOqDCirVHzkDsv8XOZFD7pzoRWqyWlVuhmAoQD82dtLHGdK9khUZPUm2jJazTTxyaxnk7MpMmUD5OQYq1jkHElU1vnSeY/s1600/Slide3.jpg" height="481" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 6. Experimental CER vs. SNR results </td></tr>
</tbody></table>
<br />
For the next version I am looking into integrating the electronic circuitry into a smaller form factor, something similar to Figure 7. below. This <a href="http://www.engr.ncsu.edu/magazine/spring-summer2013/cockroaches.php" target="_blank">source</a> provides additional inspiration to pursue this project further. <br />
<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://www.engr.ncsu.edu/magazine/media/images/cockroach.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="http://www.engr.ncsu.edu/magazine/media/images/cockroach.png" height="411" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 7. Portable El-bug Morse Decoder </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<h3>
CONCLUSIONS </h3>
If the reader has had patience to follow this story this far I must congratulate you. You have amazing neural circuitry in your brain that is able to absorb this amount of information and form an opinion about what is being presented to you. You may have already realized that this story may be just pure imagination and has no connection to reality whatsoever. <br />
<br />
<br />
Happy April 1, 2015! <br />
<br />
Mauri AG1LE <br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-69783940537802627282015-01-02T21:40:00.003-05:002015-01-02T21:44:29.748-05:00Morse Learning Machine v1 Challenge<h2>
Morse Learning Machine v1 Challenge Results</h2>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbZJKHLN_VXW-UlwWWzFiH52m7iAKvi76sEv0t5uUbp8EA2jny7VG6Y5UHfKS6OQhe-AInMDE1Shy-bSnuLYwdvY9k2GSwwT57_iR3Lj1wX_Urg7kkiq_9PNOtWeN2412rXoswvNJf5-o/s1600/morse76_76.png" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img alt="MLMv1" border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbZJKHLN_VXW-UlwWWzFiH52m7iAKvi76sEv0t5uUbp8EA2jny7VG6Y5UHfKS6OQhe-AInMDE1Shy-bSnuLYwdvY9k2GSwwT57_iR3Lj1wX_Urg7kkiq_9PNOtWeN2412rXoswvNJf5-o/s1600/morse76_76.png" height="200" title="Morse Learning Machine Challenge" width="200" /></a></div>
<a href="http://ag1le.blogspot.com/2014/09/morse-learning-machine-challenge.html" target="_blank">Morse Learning Machine v1 Challenge</a> (MLMv1) is officially finished. <a href="https://inclass.kaggle.com/c/morse-challenge" target="_blank">This challenge</a> was hosted by Kaggle and it created much more interest than I expected. There was <a href="http://www.eham.net/ehamforum/smf/index.php?topic=98713.0" target="_blank">active discussion in eham.net</a> in CW forum, as well as in Reddit <a href="http://www.reddit.com/r/morse/comments/2fey6z/morse_learning_machine_challenge/" target="_blank">here</a> and <a href="http://www.reddit.com/r/MachineLearning/comments/2fi18j/morse_learning_machine_challenge/" target="_blank">here</a>.<br />
The challenge made it to the <a href="http://www.arrl.org/news/morse-learning-machine-challenge-catching-on-with-hams" target="_blank">ARRL headline news</a> in September. <a href="https://www.google.com/search?q=%22morse+learning+machine%22" target="_blank">Google search</a> gives 1030 hits as different sites and bloggers worldwide picked up the news.<br />
<br />
The goal of this competition was to build a machine that learns how to decode audio files containing Morse code. To make it easier to get started I provided sample Python morse decoder and sample submission files.<br />
<br />
For humans it takes many months effort to learn Morse code and after years of practice the most proficient operators can decode Morse code up to 60 words per minute or even beyond. Humans have also extraordinary ability to quickly adapt to varying conditions, speed and rhythm. We wanted to find out if it is possible to create a machine learning algorithm that exceeds human performance and adaptability in Morse decoding.<br />
<br />
Total of 11 teams and 13 participants were competing almost 4 months for the perfect score 0.0 (this means no errors in decoding <a href="https://inclass.kaggle.com/c/morse-challenge/data" target="_blank">sample audio files</a>). During the competition there was active discussions in the <a href="https://inclass.kaggle.com/c/morse-challenge/forums" target="_blank">Kaggle forum</a> where participants shared their ideas, asked questions and got also some help from the organizer (<a href="https://www.kaggle.com/users/104160/ag1le" target="_blank">ag1le</a> aka myself).<br />
<br />
The <a href="https://inclass.kaggle.com/c/morse-challenge/details/evaluation" target="_blank">evaluation</a> was done by Kaggle platform based on submissions that the participants uploaded. Levenshtein distance was used as the evaluation metric to compare predicted results to the corresponding lines in the truth value file that was hidden from the participants.<br />
<br />
<h2>
Announcing the winners</h2>
According to the <a href="https://inclass.kaggle.com/c/morse-challenge/rules" target="_blank">challenge rules</a> I asked participants to make their submissions available as <a href="https://www.gnu.org/licenses/quick-guide-gplv3.html" target="_blank">open source with GPL v3</a> license or later to enable further development of machine learning algorithms. Resulting new Morse decoding algorithms, source code and supporting files are uploaded in <a href="https://github.com/Morse-Learning-Machine-Challenge/MLMv1" target="_blank">Github repository</a> by the MLMv1 participants.<br />
<div>
<br /></div>
I also asked the winners to provide a brief description about themselves, methods & tools used and any insights and takeaways from this competition.<br />
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;"><br /></strong>
<br />
<h3>
<span style="background-color: white; color: #222222;"><span style="font-family: inherit; font-size: large;">BrightMinds team: <a href="https://www.kaggle.com/users/238287/composman" target="_blank">Evgeniy Tyurin</a> and <a href="https://www.kaggle.com/users/238283/kdrobnyh" target="_blank">Klim Drobnyh</a></span></span></h3>
<a href="https://inclass.kaggle.com/c/morse-challenge/leaderboard/public" target="_blank">Public leaderboard score</a>: 0.0<br />
<a href="https://inclass.kaggle.com/c/morse-challenge/leaderboard/private" target="_blank">Private leaderboard score</a>: 0.0<br />
Source code & supporting files (waiting for posting by team)<br />
<div>
<br /></div>
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">What was your background prior to entering this challenge?</strong><br />
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">We have been studying machine learning for 3 years. Our interests has been different until now, but there are several areas we share experience in, such as image processing, computer vision and applied statistics.</span></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;"><br /></strong></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">What made you decide to enter?</strong></div>
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">Audio processing is a new and exciting field of computer science for us. We wanted to consolidate our expertise in our first joint machine learning project.</span></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;"><br /></strong></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">What preprocessing and supervised learning methods did you use?</strong></div>
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">At first we tried to use Fourier transform to get robust features and train supervised machine learning algorithm. But then we would have had extremely large train dataset to work with. That was the reason to change the approach in favour of simple statistical tests.</span></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;"><br /></strong></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">What was your most important insight into the data?</strong></div>
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">Our solution relies on the way the data was generated. So, observing the regularity in the data was this very insight that influenced the most.</span></div>
<div>
</div>
<div>
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">Were you surprised by any of your insights?</strong></div>
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">Actually, we expected that the data would be real. For example, recorded live from radio.</span></div>
<div>
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;"><br /></strong></div>
<div>
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">Which tools did you use?</strong></div>
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;">Our code was written in Python. We used numpy and scipy to calculate values of normal c</span><span style="color: #333333; font-family: 'Open Sans', sans-serif; font-size: 13px; line-height: 19px;">umulative density function.</span></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;"><br /></strong></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<strong style="background-color: transparent; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 21.9999923706055px;">What have you taken away from this competition?</strong></div>
<div>
<span style="background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px;"> We gained great experience in audio signal processing and the applicability of machine learning approach.</span></div>
<div style="-webkit-text-stroke-width: 0px; background-color: white; color: #222222; font-family: arial, sans-serif; font-size: 13px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px;">
<br /></div>
<br />
<h3>
<b><a href="https://www.kaggle.com/users/184612/dr-acid" target="_blank"><span style="font-size: large;">Tobias Lampert</span></a></b></h3>
<a href="https://inclass.kaggle.com/c/morse-challenge/leaderboard/public" target="_blank">Public leaderboard score</a>: 0.02<br />
<a href="https://inclass.kaggle.com/c/morse-challenge/leaderboard/private" target="_blank">Private leaderboard score</a>: 0.12<br />
<a href="https://github.com/Morse-Learning-Machine-Challenge/MLMv1/tree/master/acidtobi" target="_blank">Source code & supporting files from Tobias</a><br />
<br />
<div>
<div>
<span class="im" style="color: #500050;"><strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">What was your background prior to entering this challenge?</strong></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
I do not have a college degree, but I have 16 years of professional experience developing software in a variety of languages, mainly in the financial sector. I have been interested in machine learning for quite some time but seriously started getting into the topic after completing Andrew Ng's machine learning Coursera class about half a year ago.</div>
<span class="im" style="color: #500050;"></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span class="im" style="color: #500050;"><br clear="none" /></span></div>
<span class="im" style="color: #500050;">
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">What made you decide to enter?</strong></span></div>
<div>
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;"></strong></div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
Unlike most other Kaggle competitions, the raw data is very comprehensible and not just "raw numbers" - after all morse code audio can easily be decoded by humans with some experience. So to me it looked like an easy and fun exercise to write a program for the task. In the beginning this proved to be true and as expected I achieved decent results with relatively little work. However the chance of finding the perfect solution kept me trying hard until the very end!</div>
<span class="im" style="color: #500050;"></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span class="im" style="color: #500050;"><br clear="none" /></span></div>
<span class="im" style="color: #500050;">
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">What preprocessing and supervised learning methods did you use?</strong></span></div>
<div>
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;"></strong><br />
<div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
Preprocessing:<br />
- Butterworth filter for initial computation of dit length</div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
- FFT to transform data from time to frequency domain</div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
- PCA to reduce dimensionality to just one dimension<br />
Unsupervised learning:</div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
- K-Means to generate clusters of peaks and troughs<br />
Supervised learning:<br />
<div style="border: 0px none; margin: 0px; padding: 0px;">
- Neural network for morse code denoising</div>
</div>
<span class="im" style="color: #500050;"></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span class="im" style="color: #500050;"><br clear="none" /></span></div>
<span class="im" style="color: #500050;">
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">What was your most important insight into the data?</strong></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
The most important insight was probably the fact that all files have a structure which can be heavily exploited - due to the fact the pauses at the beginning and end have the same length in all files and wpm is constant the exact length of one dit can be computed. Using this information, the files can be cut into chunks that fully contain either signal or no signal making further analysis much easier.</div>
<span class="im" style="color: #500050;"></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
</div>
<span class="im" style="color: #500050;">
</span>
<br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span class="im" style="color: #500050;"><br clear="none" /></span></div>
<span class="im" style="color: #500050;">
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">Were you surprised by any of your insights?</strong></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
What surprised me most is that after trying several supervised learning methods like neural networks and SVMs with varying success, a very simple approach using an unsupervised method (K-Means) yielded the best results.</div>
<span class="im" style="color: #500050;"></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span class="im" style="color: #500050;"><br clear="none" /></span></div>
<span class="im" style="color: #500050;">
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">Which tools did you use?</strong></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
I started with R for some quick tests but switched to Python with scikit-learn very early, additionally I used the ffnet module for neural networks. To get a better grasp on the data, I did a lot of charting using matplotlib.</div>
<span class="im" style="color: #500050;"></span><br />
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span class="im" style="color: #500050;"><br clear="none" /></span></div>
<span class="im" style="color: #500050;">
<strong style="color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px;">What have you taken away from this competition?</strong></span></div>
<div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<span style="background-color: white;">First of all obviously I learned a lot about how morse code works, how it can be represented mathematically and which patterns random morse texts always have in common. I also deepened my knowledge about signal processing and filtering, even though in the end this only played a minor role in my solution. Like all Kaggle competitions, trying to make sense of data, competing with others and discussing solution approaches was great fun!</span></div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<br /></div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<br /></div>
<div style="border: 0px none; color: #383838; font-family: gotham, helvetica, arial, sans-serif; font-size: 14px; line-height: 22px; margin: 0px; padding: 0px;">
<h2>
Observations & Conclusions</h2>
<div>
I asked advice in the <a href="http://kaggle.a.ssl.fastly.net/forums/t/10219/how-to-promote-kaggle-competitions" target="_blank">Kaggle forum</a> how to promote and attract participants. I tried to encourage people to join the challenge during the first 2 weeks by posting frequent forum updates. Based on download statistics (see the table below) the participation rate of this challenge was roughly 11% as there was 13 actual participants and 120 unique users who downloaded the audio files. I don't know if this is typical in Kaggle competitions but certainly there were much more interested people than actual participants. </div>
<div style="line-height: 100%; margin-bottom: 0in;">
<br /></div>
<table cellpadding="2" cellspacing="2">
<tbody>
<tr bgcolor="#cacaca">
<th style="border: none; padding: 0in;"><div align="left">
<span style="color: black;"><b>Filename</b></span></div>
</th>
<th style="border: none; padding: 0in;"><div align="left">
<span style="color: black;"><b>Size</b></span></div>
</th>
<th style="border: none; padding: 0in;"><div align="left">
<span style="color: black;"><b>Unique Users</b></span></div>
</th>
<th style="border: none; padding: 0in;"><div align="left">
<span style="color: black;"><b>Total Downloads</b></span></div>
</th>
<th style="border: none; padding: 0in;"><div align="left">
<span style="color: black;"><b>Bandwidth Used</b></span></div>
</th>
</tr>
<tr bgcolor="#fafafa">
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">sampleSubmission.csv</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">2.02 KB</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">126</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">226</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">254.21 KB</span></div>
</td>
</tr>
<tr>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">levenshtein.py</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">1.56 KB</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">83</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">120</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">129.69 KB</span></div>
</td>
</tr>
<tr bgcolor="#fafafa">
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">morse.py</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">12.69 KB</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">126</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">196</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">1.56 MB</span></div>
</td>
</tr>
<tr>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">audio_fixed.zip</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">65.91 MB</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">120</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">179</span></div>
</td>
<td style="border: none; padding: 0in;"><div align="left">
<span style="color: #444444;">7.72 GB</span></div>
</td>
</tr>
</tbody></table>
</div>
</div>
</div>
<br />
<div>
I did also a short informal survey among the participants in preparation for the MLM v2 challenge. Here are some examples from the survey:<br />
<br />
<i>Q: Why did you decide to participate MLM v1 challenge?</i><br />
<br />
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="border-collapse: collapse; border: 1px solid #ccc; font-family: arial,sans,sans-serif; font-size: 13px; table-layout: fixed;"><colgroup><col width="461"></col></colgroup><tbody>
<tr style="height: 21px;"><td data-sheets-value="[null,2,"I have a background in signal processing and it seemed like a good way to refresh my memory. "]" style="padding: 2px 3px 2px 3px; vertical-align: bottom;">I have a background in signal processing and it seemed like a good way to refresh my memory. </td></tr>
<tr style="height: 21px;"><td data-sheets-value="[null,2,"By participation I tried to strengthen my Computer Science knowledge. At the university I am attending Machine Learning & Statistics course, so the challenge can help me practice."]" style="padding: 2px 3px 2px 3px; vertical-align: bottom;">By participation I tried to strengthen my Computer Science knowledge. At the university I am attending Machine Learning & Statistics course, so the challenge can help me practice.</td></tr>
<tr style="height: 21px;"><td data-sheets-value="[null,2,"You were enthusiastic about it and it seemed fun/challenging/new"]" style="padding: 2px 3px 2px 3px; vertical-align: bottom;">You were enthusiastic about it and it seemed fun/challenging/new</td></tr>
</tbody></table>
</div>
<div class="yj6qo ajU" style="background-color: white; cursor: pointer; outline: none; padding: 10px 0px; width: 22px;">
</div>
<i>Q: How could we make MLM v2 challenge more enjoyable?
</i><br />
<br />
<table border="1" cellpadding="0" cellspacing="0" dir="ltr" style="border-collapse: collapse; border: 1px solid #ccc; font-family: arial,sans,sans-serif; font-size: 13px; table-layout: fixed;"><colgroup><col width="494"></col></colgroup><tbody>
<tr style="height: 21px;"><td data-sheets-value="[null,2,"More realistic problem setting. "]" style="padding: 2px 3px 2px 3px; vertical-align: bottom;">More realistic problem setting. </td></tr>
<tr style="height: 21px;"><td data-sheets-value="[null,2,"The task is interesting and fun by itself, but the details may vary, so making a challenge with accent on different aspects of the task would be exciting"]" style="padding: 2px 3px 2px 3px; vertical-align: bottom;">The task is interesting and fun by itself, but the details may vary, so making a challenge with accent on different aspects of the task would be exciting</td></tr>
</tbody></table>
<br />
<div>
<br /></div>
<div>
In the MLMv1 the 200 audio files had all very similar structure - 20 random characters each without spaces. Participants were able to leverage this structure to overcome poor signal-to-noise ratio in many files. I was surprised to get such good decoding results given that many audio files had only -12 dB SNR.<br />
<br />
My conclusion for the next MLM v2 challenge is that I should provide real world, recorded Morse signals and reduce the impact of audio file structure. Also, to make the challenge more realistic I need to incorporate RF propagation effects in the audio files. I could also stretch the SNR down to -15 ... -20 dB range, making it harder to search correct answers.<br />
<br />
<br /></div>
<div>
<br /></div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-58442929247992502002014-11-16T21:13:00.000-05:002014-11-25T20:53:16.114-05:00High Resolution, Low Power Temperature Logger using ArduinoI am working on a new project with the goal of developing miniaturized high resolution temperature logger device. The objective is to make it small enough to be a wearable device that would measure temperature with better than 0.1 °C resolution and store the measured data on a memory card for plotting and analysis. The logger needs to run on a chargeable battery for several weeks and must be comfortable enough to wear 24x7.<br />
<br />
I used TMP102 that is a <a href="https://www.sparkfun.com/datasheets/Sensors/Temperature/tmp102.pdf" target="_blank">digital sensor manufactured by Texas Instruments</a> with I2C a.k.a. two wire interface (TWI) and the following features:<br />
<br />
<ul>
<li>12-bit, 0.0625°C resolution</li>
<li>Accuracy: 0.5°C (-25°C to +85°C)</li>
<li>Low quiescent current</li>
<li>10µA Active (max)</li>
<li>1µA Shutdown (max)</li>
<li>1.4V to 3.6VDC supply range</li>
<li>Two-wire serial interface</li>
</ul>
<br />
This is a very handy high resolution sensor that requires a very low current to operate.<br />
<br />
There are quite many data logger boards available but only a few that are fully hack-able and small in size. I ended up selecting Sparkfun OpenLog board that looked small enough and that had all software, firmware and hardware designs available as open source. The only small problem was I2C (TWI) interface signals SCL and SDA. I had to solder wires directly to ATmega328 CPU pins, as they are not available on external interface. Soldering to surface mounted components requires steady hand and small soldering tip on your iron. Having a 20x optical microscope does also help. <br />
<h3>
Hardware components</h3>
The components for this project are listed below with links and cost at the time of writing this article. Most of these are available at Sparkfun.<br />
<ol>
<li><a href="https://www.sparkfun.com/products/9530" target="_blank">Sparkfun OpenLog</a> $24.95</li>
<li><a href="https://www.sparkfun.com/products/11931" target="_blank">Digital Temperature Sensor Breakout - TMP102</a> $5.96</li>
<li><a href="https://www.sparkfun.com/products/12722" target="_blank">Lithium Polymer USB Charger and Battery</a> $24.95</li>
<li><a href="http://www.amazon.com/gp/product/B00IVPU7KE/ref=oh_aui_detailpage_o03_s00?ie=UTF8&psc=1" target="_blank">Samsung 16GB EVO Class 10 Micro SDHC</a> $10.99</li>
<li>Enclosure - 3D printed $9.50 </li>
</ol>
Total cost of components was $76.35
<br />
<br />
<h3>
Software Development </h3>
You can download the latest integrated software development environment from <a href="http://arduino.cc/en/Main/Software" target="_blank">Arduino IDE page</a>. I used the Arduino 1.5.8 with the following configuration<br />
<br />
<br />
<ul>
<li>Tools/Board - Arduino Uno</li>
<li>Tools/Port - /dev/ttyUSB0</li>
</ul>
<br />
<br />
To connect my Lenovo X301 laptop running Linux Mint 17 operating system I used <a href="https://www.sparkfun.com/products/11736" target="_blank">FT231X Breakout</a> with a <a href="https://www.sparkfun.com/products/10660" target="_blank">Crossover Breakout for FTDI</a>. To make it easier to connect I soldered <a href="https://www.sparkfun.com/products/9429" target="_blank">Header - 6-pin Female (0.1", Right Angle)</a> and<br />
<a href="https://www.sparkfun.com/products/553" target="_blank">Break Away Male Headers - Right Angle</a> connectors for these small breakout boards.<br />
<br />
My focus in software development was to re-use existing OpenLog software, implement temperature measurement over I2C bus and try to minimize battery consumption by implementing power shutdown between measurements. This software is still works-in-progress but even with this software version (v3) the average current consumption is about 200 uA level @3.3 Volts. During measurement over I2C bus and when storing the results to the microSD card the current peaks to 25 mA for a few milliseconds.<br />
<br />
The software makes a measurement every 5000 milliseconds. Since this design does not have a RTC chip the timekeeping is done with software. Software has a routine to compensate for timing errors but it is not very accurate yet. It meets my needs for data logging purposes for now. For the final product it might be useful to have RTC chip built into the design.<br />
<br />
A fully charged 3.7 V 850mAh LiPo battery is estimated to provide power for this data logger over 170 days, though I haven't tested it for longer period than 15 days so far.<br />
<br />
The latest <a href="https://github.com/ag1le/logger" target="_blank">logger software</a> is posted in Github.<br />
<h3>
3D Printed Enclosure </h3>
I could not find a suitable small enclosure so I took <a href="http://www.openscad.org/" target="_blank">OpenSCAD software</a> into use and built a 3D model using <a href="http://www.thingiverse.com/thing:43899/#files" target="_blank">some examples I found in Thingverse</a>. OpenSCAD allows you to write 3D objects in simple language so it was quite easy to experiment with different designs and view them before creating the .STL file required for 3D printing.<br />
<br />
The enclosure size was determined mostly by the LiPo battery dimensions. With a smaller battery it would be possible to make a much smaller enclosure. Also, it would be possible to design a single circuit board with all the required components and make it smaller. Since this project is still in concept phase I chose to use the selected components and 3D print an enclosure where I can fit them in.<br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxKweDhCQi_4Tz95QD6u6LTfn0EV1c5h4qwo2Vc7etpOgEPd_AUxhQpCLlwQhVx3zu7jkWrHFf28RLEX3OGJlLfTsFOcTylw8-qukhz-pxdoVZwhGmZ_ErN13mzEuogcvUt_L0JcJTppM/s1600/enclosure4b.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxKweDhCQi_4Tz95QD6u6LTfn0EV1c5h4qwo2Vc7etpOgEPd_AUxhQpCLlwQhVx3zu7jkWrHFf28RLEX3OGJlLfTsFOcTylw8-qukhz-pxdoVZwhGmZ_ErN13mzEuogcvUt_L0JcJTppM/s1600/enclosure4b.png" height="338" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">3D model of enclosure</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
I sent the .STL file to a local company with some email instructions. I my first version the dimensions were 1/8" units - this was not clear so I had to call them to clarify. In the 2nd version I used millimeters as units and the enclosure came out fabricated as I expected. The enclosure is big enough for the LiPo battery and it has also a small indention at the bottom to get the TMP102 sensor chip closer to the outside surface. The wall thickness was set to 1.0 mm and sensor chip is only 0.5 mm from the bottom surface.<br />
<br />
The cover dimensions were designed it to be tightly fitted on the top. <br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrYfy5PrU6a3vEt_RPnXnFtkGdXjx-ota74I4jJYlBC0QQk8z6ZMzd7HZcYo7rgKwIIE0EGcZZsUli-IcFxxMR9gFTMNux0sXKxJvGeDNss4vjJE2eubTmzJaSLV_n8WFh711SiKBjO9E/s1600/enclosure_v2.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhrYfy5PrU6a3vEt_RPnXnFtkGdXjx-ota74I4jJYlBC0QQk8z6ZMzd7HZcYo7rgKwIIE0EGcZZsUli-IcFxxMR9gFTMNux0sXKxJvGeDNss4vjJE2eubTmzJaSLV_n8WFh711SiKBjO9E/s1600/enclosure_v2.jpg" height="474" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">3D printed enclosure - top view</td></tr>
</tbody></table>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
The enclosure has also fixtures both sides attaching a 22mm standard wristband using spring bars.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsrjI8NoXsKoq4araGxIsdZc_xewaMYRp7hCRyoDKrnR1W0u2qvZ9CuoAJaV3d16MgXuqR6MPanoRhLziS7B9l6gkfJva1hQg4X1lK3CY0I8NtGvMY9CpJQeQEW7rrrQ37F6xtLsvw5mg/s1600/enclosure_v2b.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjsrjI8NoXsKoq4araGxIsdZc_xewaMYRp7hCRyoDKrnR1W0u2qvZ9CuoAJaV3d16MgXuqR6MPanoRhLziS7B9l6gkfJva1hQg4X1lK3CY0I8NtGvMY9CpJQeQEW7rrrQ37F6xtLsvw5mg/s1600/enclosure_v2b.jpg" height="308" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">3D printed enclosure - side view </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
Below is a photograph of 3D enclosure model v1 with wristbands attached.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfewflEiA3H8jKxknbWeWD2792LEEZclpM8QiUDvEF9HWjsLdIDOFSrSp7eY1rfRulVMM3WtxtVxM5npauGUmx2lTX93xRqcCLVesg3oN7wvQk-Wx_IC_j38hGS-56y_JY8n4aiWFO-fk/s1600/enclosure_v1_wristband.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjfewflEiA3H8jKxknbWeWD2792LEEZclpM8QiUDvEF9HWjsLdIDOFSrSp7eY1rfRulVMM3WtxtVxM5npauGUmx2lTX93xRqcCLVesg3oN7wvQk-Wx_IC_j38hGS-56y_JY8n4aiWFO-fk/s1600/enclosure_v1_wristband.jpg" height="474" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">3D printed enclosure with wristbands attached</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
In the photo below the hardware components fit in perfectly and the LiPo battery comes on the top. The power is connected using red and black wires to OpenLog board.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXwSIyCVG1rn-wJdKpWHO2jvpB5fYH9RxRHhbyWtlSY61FCzhmsHvFPAZF9SeI-WbYvA9t8VOMCKXMCWncizxh2UaD6r24Wa3jc0LlSfL5qhUTJp6aFYs06luJLIXQFpGElQFmpq7AjXw/s1600/logger.jpg" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiXwSIyCVG1rn-wJdKpWHO2jvpB5fYH9RxRHhbyWtlSY61FCzhmsHvFPAZF9SeI-WbYvA9t8VOMCKXMCWncizxh2UaD6r24Wa3jc0LlSfL5qhUTJp6aFYs06luJLIXQFpGElQFmpq7AjXw/s1600/logger.jpg" height="474" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">3D printed enclosure with components inside</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
Measurement Results </h3>
I tested the temperature logger by putting it from room temperature T0 22.6 °C into a freezer at Tm -21.0 °C.<br />
<br />
Based on the measurement results below (red graph) it took 54 minutes for the sensor to reach this temperature. The thermal mass of the sensor itself is small, but it was in a closed enclosure with the LiPo battery that has a much larger thermal mass. I did the same test but this time left the enclosure open so that the sensor was exposed to freezer temperature from both sides. The blue graph below shows that the sensor reached -21 °C in 30 minutes.<br />
<br />
<br />
<div style="text-align: left;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwtPi43RwZa0F1J5PtMDJkg1oevFt2j1E5CvZW2F_OgCtHIm3VZeQlyPBUV88lVLua4qbLILPZ9w02yQbUK9ELJiDb_DbPwAhZyzB-hmhERrEsvW4WkjyPjv7_R6bDMFJ0Lax9NFiF0Xc/s1600/logger_freezer_test_open_and_closed.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgwtPi43RwZa0F1J5PtMDJkg1oevFt2j1E5CvZW2F_OgCtHIm3VZeQlyPBUV88lVLua4qbLILPZ9w02yQbUK9ELJiDb_DbPwAhZyzB-hmhERrEsvW4WkjyPjv7_R6bDMFJ0Lax9NFiF0Xc/s1600/logger_freezer_test_open_and_closed.png" height="358" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Freezer Test</td></tr>
</tbody></table>
<br />
<br />
<h3>
</h3>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
<br />
<br />
<br />
<br />
<br />
The thermal time constant is defined as the time required by a sensor to reach 63.2% of a step change in temperature under a specified set of conditions.</div>
<div>
<br /></div>
<div>
The response of a sensor to a sudden change in the surrounding temperature is exponential and it is described by the following equation:</div>
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDzIL2gvqa8BDKQjzNOWgHuCPShK7ifRlSrrl2Am0T277czAKoSdmphNPzfn2ASB5nBMAeGvzSvCED4E2LmDrBTWMQbBO8irKUpXl3qA_dzGIgxfqQMIhOVDuiWUW9KyiKlxo2kRxS6BI/s1600/theory1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDzIL2gvqa8BDKQjzNOWgHuCPShK7ifRlSrrl2Am0T277czAKoSdmphNPzfn2ASB5nBMAeGvzSvCED4E2LmDrBTWMQbBO8irKUpXl3qA_dzGIgxfqQMIhOVDuiWUW9KyiKlxo2kRxS6BI/s1600/theory1.png" height="38" width="320" /></a></div>
<div>
<br /></div>
<div>
where T is sensor temperature, Tm is the surrounding medium temperature, T0 is the initial sensor temperature, t is the time and τ is the time constant.</div>
<div>
<br /></div>
<div>
Looking at the results above and trying to find best fit of measurement data to this model using <a href="http://en.wikipedia.org/wiki/Root_mean_square" target="_blank">RMS error method</a> we can estimate that </div>
<div>
<br /></div>
<div>
<div>
τ = 0.009732 for closed enclosure</div>
</div>
<div>
τ = 0.003132 for open enclosure</div>
<div>
<br /></div>
See graph of models vs. actuals is below. To correctly estimate temperatures this thermal time constant need to be taken into account.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhitKDxQl5dL5aofZi8Ui9typ2VjUiqA5IDPAekcX8DiYP1ZTf22xD839CTUvqeo7qGwJPlMg0VjM0wyp-GGS5jcf2btW60qQptg4lizctXO3XzGDmTqmnEnBLZIcagVxDwJ9Nfx34dbq8/s1600/logger_freezer_test_open_and_closed_tau.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhitKDxQl5dL5aofZi8Ui9typ2VjUiqA5IDPAekcX8DiYP1ZTf22xD839CTUvqeo7qGwJPlMg0VjM0wyp-GGS5jcf2btW60qQptg4lizctXO3XzGDmTqmnEnBLZIcagVxDwJ9Nfx34dbq8/s1600/logger_freezer_test_open_and_closed_tau.png" height="396" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Model of thermal time constants <span style="font-size: xx-small; text-align: start;">τ</span></td></tr>
</tbody></table>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<div>
<br /></div>
<div>
In the datasheet there are the following guidelines to maintain accuracy.</div>
<div>
<br /></div>
<div>
<i><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">The temperature sensor in the TMP102 is the chip itself. Thermal paths run through the package leads, as well as the plastic package. The lower thermal</span></i></div>
<div>
<i><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">resistance of metal causes the leads to provide the primary thermal path.</span></i></div>
<div>
<i><span style="font-family: Courier New, Courier, monospace; font-size: x-small;">To maintain accuracy in applications requiring air or surface temperature measurement, care should be taken to isolate the package and leads from ambient air temperature. A thermally-conductive adhesive is helpful in achieving accurate surface temperature measurement.</span></i></div>
</div>
<div>
<br /></div>
<div>
One improvement would be to apply thermally-conductive materials between the sensor chip and the bottom surface of the enclosure. Some biologically inert metal like gold or titanium might provide better thermal time constant. Some insulation might be needed to isolate sensor from ambient air. </div>
<h3>
Conclusions </h3>
Temperature logging using modern, high resolution and accurate digital sensors is quite easy. You need only simple a micro-controller and few lines of code to build a data logger that is small enough to be wearable. <br />
<br />
Working with Arduinos is not only easy but also a lot of fun. With minimal investment you can build a powerful data logging device and add new sensors in incremental fashion. The software development is also straightforward and for most problems there is already some open source examples available.<br />
<br />
The real challenges seem to be on the physics and mechanical enclosure design side. Building an enclosure for the sensor and electronics with a small thermal time constant is not trivial. For highly accurate wearable data logging devices you need also to consider many other topics such as<br />
<br />
<ul>
<li>physical appearance </li>
<li>hypoallergic materials </li>
<li>fit and comfort for different size of subjects</li>
<li>thermal properties of materials</li>
<li>any variability in sensor contact with subject or ambient air </li>
</ul>
<br />
It would be interesting to see the thermal design details of popular fitness trackers that claim accurate body temperature measurements. In particular the <a href="http://en.wikipedia.org/wiki/Basal_body_temperature" target="_blank">basal body temperature</a> measurements where accuracy and high resolution is required poor thermal design would impact results significantly.<br />
<br />
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-15324128394113991852014-09-03T20:32:00.000-04:002014-09-03T21:07:30.993-04:00Morse Learning Machine - Challenge<h3>
MACHINE LEARNING CHALLENGE</h3>
I was astonished to get email acknowledgement that my <a href="https://inclass.kaggle.com/c/morse-challenge" target="_blank">Kaggle Morse Challenge</a> was approved today. I have spent last few days preparing materials and editing the description and the rules for this competition. <br />
<br />
<span style="background-color: yellow;"><b>The goal of this competition is to build a machine that learns how to decode audio files containing Morse code</b>. </span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCtaSUu6PRv6dMNKGvl1ilZYz94XrIeoZTvXJoYcLgTv8QgyRMLwfA-TuGmXuJdX-jFSPfbwjWoi45bj70S5Xx1PoqyTpPytVit6aUhNFgL0r8ZoYr0Mz3vjTjIJbwrBv3PcPeqzkXPD8/s1600/just_numbers.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjCtaSUu6PRv6dMNKGvl1ilZYz94XrIeoZTvXJoYcLgTv8QgyRMLwfA-TuGmXuJdX-jFSPfbwjWoi45bj70S5Xx1PoqyTpPytVit6aUhNFgL0r8ZoYr0Mz3vjTjIJbwrBv3PcPeqzkXPD8/s1600/just_numbers.jpg" height="308" width="400" /></a></div>
<br />
For humans it takes many months effort to learn Morse code and after years of practice the most proficient operators can decode Morse code up to 60 words per minute or even beyond. Humans have also extraordinary ability to quickly adapt to varying conditions, speed and rhythm. <br />
<br />
I want to find out if it is possible to create a machine learning algorithm that exceeds human performance and adaptability in Morse decoding. I have <a href="http://www.slideshare.net/mniinine/fldigi-neai-presentation" target="_blank">shared some of these ideas</a> in <a href="http://www.meetup.com/intelligence/events/119478492/" target="_blank">New England Artificial Intelligence meetup</a> about one year ago and got enthusiastic feedback from the participants.<br />
<br />
<br />
<br />
<br />
<h3>
WHY KAGGLE? </h3>
<a href="http://en.wikipedia.org/wiki/Kaggle" target="_blank">Kaggle is a platform</a> for predictive modelling and analytics competitions on which companies and researchers post their data and statisticians and data miners from all over the world compete to produce the best models. This crowdsourcing approach relies on the fact that there are countless strategies that can be applied to any predictive modelling task and it is impossible to know at the outset which technique or analyst will be most effective. Kaggle aims at making data science a sport.<br />
<br />
Kaggle's community of data scientists comprises tens of thousands of PhDs from quantitative fields such as computer science, statistics, econometrics, maths and physics, and industries such as insurance, finance, science, and technology. They come from over 100 countries and 200 universities. In addition to the prize money and data, they use Kaggle to meet, learn, network and collaborate with experts from related fields.<br />
<br />
For the Morse Learning Machine competition I hope to attract people from the Kaggle community who are interested in solving new, difficult challenges using their predictive data modeling, computer science and machine learning expertise. For students this challenge provides a great opportunity to put theoretical concepts into practice and see how they can solve tough problems by applying knowledge gained in class rooms.<br />
<br />
<br />
<h3>
<b>COMPETITION DETAILS</b></h3>
<br />
During the competition, the participants build a learning system capable of decoding Morse code. To that end, they get development data consisting of 200 .WAV audio files containing short sequences of randomized Morse code. The data labels are provided for a training set so the participants can self-evaluate their systems. To evaluate their progress and compare themselves with others, they can submit their prediction results on-line to get immediate feedback. A real-time leaderboard shows participants their current standing based on their validation set predictions.<br />
<br />
I have also provided <a href="https://inclass.kaggle.com/c/morse-challenge/data" target="_blank">sample Python Morse decoder</a> to make it easier too get started. While this software is purely experimental version it has some features of the <a href="http://www.w1hkj.com/FldigiHelp-3.21/html/cw_configuration_page.html" target="_blank">FLDIGI Morse decoder</a> but implemented using Python instead of C++.<br />
<br />
You can of course leverage the <a href="http://ag1le.blogspot.com/2014/07/new-morse-decoder-part-6.html" target="_blank">experimental multichannel CW decoder I recently implemented on FLDIGI</a> or the <a href="https://github.com/ag1le/morse-wip" target="_blank">standalone version of Bayesian decoder</a> written in C++. There is also <a href="https://github.com/ag1le/morse.py" target="_blank">some new tools</a> I posted to Github.<br />
<br />
Please help me to spread this message to attract participants for the <a href="https://inclass.kaggle.com/c/morse-challenge" target="_blank">Morse Learning Machine challenge</a>!<br />
<br />
73<br />
Mauri AG1LE<br />
<br />
<br />
<br />
<div>
<br /></div>
<div>
<br /></div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com9tag:blogger.com,1999:blog-3326773214329183284.post-27198726063257087442014-08-24T20:00:00.004-04:002014-08-24T22:03:01.160-04:00Cortical Learning Algorithm for Morse code - part 1<h2>
Cortical Learning Algorithm Overview </h2>
Humans can perform many tasks that computers currently cannot do. For example understanding spoken language in noisy environment, walking down a path in complex terrain or winning in CQWW WPX CW contest are tasks currently not feasible for computers (and might be difficult for humans, too).<br />
Despite decades of machine learning & artificial intelligence research, we have few viable algorithms for achieving human-like performance on a computer. Morse decoding at the best human performance level would be a good target to test these new algorithms. <br />
<br />
<a href="http://numenta.com/" target="_blank">Numenta Inc.</a> has developed technology called <a href="http://numenta.org/cla.html" target="_blank">Cortical Learning Algorithm</a> (CLA) that was recently made available as <a href="http://numenta.org/nupic.html" target="_blank">open source project NuPIC</a>. This software provides an online learning system that learns from every data point fed into the system. The CLA is constantly making predictions which are continually verified as more data arrives. As the underlying patterns in the data change the CLA adjusts accordingly. CLA uses Sparse Distributed Representation (SDR) in similar fashion as neocortex in human brain stores information. SDR has many advantages over traditional ways of storing memories, such as ability to associate and retrieve information using noisy data. <br />
<br />
Detailed description on how CLA works can be found from <a href="http://numenta.org/resources/HTM_CorticalLearningAlgorithms.pdf" target="_blank">this whitepaper</a>.<br />
<h2>
Experiment </h2>
To learn more how CLA works I decided to start with a very simple experiment. I created a Python script that uses Morse code book and calculates Sparse Distributed Representation (SDR) for each character. Figure 1 below shows the Morse alphabet and numbers 0...9 converted to SDRs.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh97TlpqgqVBihsFXcN-A2dCUMs8zwfHYKMlXwdhcFd7iJjk_WP4aQGsgEjGlfYo6jPr8aYjrsLfbJsOXf6zkE2LNszILF8d2q31GaFJZu1Sh-5YcOaqcQQmhoOoETCBEGdIqva_TjNBxU/s1600/HTM_CW_SDR.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh97TlpqgqVBihsFXcN-A2dCUMs8zwfHYKMlXwdhcFd7iJjk_WP4aQGsgEjGlfYo6jPr8aYjrsLfbJsOXf6zkE2LNszILF8d2q31GaFJZu1Sh-5YcOaqcQQmhoOoETCBEGdIqva_TjNBxU/s1600/HTM_CW_SDR.png" height="378" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. SDR for Morse characters A...Z, 0...9</td></tr>
</tbody></table>
<br />
<br />
<h2>
</h2>
<h2>
</h2>
<h2>
</h2>
<h2>
</h2>
<h2>
</h2>
<h2>
</h2>
<h2>
</h2>
<h2>
</h2>
NuPIC requires input vector to be a binary representation of the input signal. I created a function that packs "dits" and "dahs" into 36x1 vector, see two examples below. Each "dit" is represented as 1. followed by 0. and each "dah" is represented by 1. 1. 1. followed by 0. to accomodate 1:3 timing ratio between "dit" and "dah". This preserves <a href="http://en.wikipedia.org/wiki/Morse_code" target="_blank">the semantic structure of Morse code</a> that is important from sequence learning perspective.<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">H ....</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">[ 1. 0. 1. 0. 1. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]</span><br />
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">O ---</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">[ 1. 1. 1. 0. 1. 1. 1. 0. 1. 1. 1. 0. 0. 0. 0. 0. 0. 0.</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]</span></div>
</div>
<br />
The Spatial Pooler uses 64 x 64 vector giving SDR of size 4096. As you calculate the SDR random bits get active on this vector. I plotted all active cells (value = 1) per columns 0...4096 for each letters and numbers as displayed in Fig 1. above. The respective character is shown on the right most column.<br />
<br />
To see better the relationships between SDR and Morse character set I created another SDR map with letters 'EISH5' and 'TMO0' next to each other. These consequent letters and numbers differ from each other only by one "dit" or one "dah". See Fig 2. for SDR visualization of these characters. <br />
<br />
There is no obvious visible patterns across these Morse characters, all values look quite different. In the Numenta CLA whitepaper page 21 it says <i>"Imagine now that the input pattern changes. If only a few input bits change, some columns will receive a few more or a few less inputs in the “on” state, but the set of active columns will not likely change much. Thus similar input patterns (ones that have a significant number of active bits in common) will map to a relatively stable set of active columns."</i><br />
<br />
This doesn't seem to apply in these experiments so I need to investigate this a bit further.<br />
<br />
<br />
<div style="text-align: left;">
</div>
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisrqNWBcouI3oACBWHgO2xSM1ZZeDqfhB_UuO0e5bxQ2NyPqD5GLEhCHIxRf8gAlzrHqgf5__ahamFYxeXqhgqHtahyphenhyphennVM7j8nkUO_wHkzNepTAerBb7CWUnY0bqG9aeqP4KO_nvfkzsY/s1600/HTM_CW_SDR1.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisrqNWBcouI3oACBWHgO2xSM1ZZeDqfhB_UuO0e5bxQ2NyPqD5GLEhCHIxRf8gAlzrHqgf5__ahamFYxeXqhgqHtahyphenhyphennVM7j8nkUO_wHkzNepTAerBb7CWUnY0bqG9aeqP4KO_nvfkzsY/s1600/HTM_CW_SDR1.png" height="140" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 2. SDR for Morse characters EISH5 and TMO0</td></tr>
</tbody></table>
<br />
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
I did another experiment by reducing SDR size to only 16x16 so 256 cells per SDR. In Fig 3. it is now easier to see common patterns between similar characters - for example compare C with K and Y. These letters have 3 common cells active. <br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYz8KZB3AsqzKSXDDgM3XBfz-E9c4rRQU1KoI6LJtayGYT4g9NS3YPn29EH7oLKh31pLjNUyv801xhoLR6MHNBKzbavb6_kE2u1vaQRsiyokKFqlQ7N0FgEZdvywr9-ZxZopSjiNs3awM/s1600/HTM_CW_SDR_256(16x16).png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhYz8KZB3AsqzKSXDDgM3XBfz-E9c4rRQU1KoI6LJtayGYT4g9NS3YPn29EH7oLKh31pLjNUyv801xhoLR6MHNBKzbavb6_kE2u1vaQRsiyokKFqlQ7N0FgEZdvywr9-ZxZopSjiNs3awM/s1600/HTM_CW_SDR_256(16x16).png" height="388" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 3. SDR map with reduced size 16x16 = 256 cells</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br /></div>
<div>
The Python software to create the SDR pictures is below:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">"""A simple program that demonstrates the working of the spatial pooler"""</span></div>
<div>
<span style="font-family: 'Courier New', Courier, monospace; font-size: xx-small;">import numpy as np</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">from matplotlib import pyplot as plt</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">from random import randrange, random</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">from nupic.research.spatial_pooler import SpatialPooler as SP</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">CODE = {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> ' ': '',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'A': '.-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'B': '-...',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'C': '-.-.', </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'D': '-..',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'E': '.',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'F': '..-.',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'G': '--.', </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'H': '....',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'I': '..',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'J': '.---',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'K': '-.-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'L': '.-..',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'M': '--',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'N': '-.',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'O': '---',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'P': '.--.',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'Q': '--.-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'R': '.-.',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'S': '...',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'T': '-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'U': '..-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'V': '...-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'W': '.--',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'X': '-..-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'Y': '-.--',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> 'Z': '--..',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '0': '-----',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '1': '.----',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '2': '..---',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '3': '...--',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '4': '....-',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '5': '.....',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '6': '-....',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '7': '--...',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '8': '---..',</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> '9': '----.' }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">class Morse():</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> def __init__(self, inputShape, columnDimensions):</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> """</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> Parameters:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> ----------</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> _inputShape<span class="Apple-tab-span" style="white-space: pre;"> </span>:<span class="Apple-tab-span" style="white-space: pre;"> </span>The size of the input. (m,n) will give a size m x n</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> _columnDimensions<span class="Apple-tab-span" style="white-space: pre;"> </span>:<span class="Apple-tab-span" style="white-space: pre;"> </span>The size of the 2 dimensional array of columns</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> """</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputShape = inputShape</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.columnDimensions = columnDimensions</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputSize = np.array(inputShape).prod()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.columnNumber = np.array(columnDimensions).prod()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray = np.zeros(self.inputSize)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.activeArray = np.zeros(self.columnNumber)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.sp = SP(self.inputShape, </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span> self.columnDimensions,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span> potentialRadius = self.inputSize,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span> numActiveColumnsPerInhArea = int(0.02*self.columnNumber),</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span> globalInhibition = True,</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span> synPermActiveInc = 0.01</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><span class="Apple-tab-span" style="white-space: pre;"> </span> )</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> def createInputVector(self,elem,chr):</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> print "\n\nCreating a Morse codebook input vector for character: " + chr + " " + elem</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> #clear the inputArray to zero before creating a new input vector</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[0:] = 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> j = 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> i = 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> while j < len(elem) :</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> if elem[j] == '.' :</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[i] = 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[i+1] = 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> i = i + 2</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> if elem[j] == '-' :</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[i] = 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[i+1] = 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[i+2] = 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.inputArray[i+3] = 0</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> i = i + 4</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> j = j + 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> <span class="Apple-tab-span" style="white-space: pre;"> </span></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> print self.inputArray</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> def createSDRs(self,row,x,y,ch):</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> """Run the spatial pooler with the input vector"""</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> print "\nComputing the SDR for character: " + ch</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> #activeArray[column]=1 if column is active after spatial pooling</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> self.sp.compute(self.inputArray, True, self.activeArray)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> # plot each row above previous one</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> z = self.activeArray * row</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> # Plot the SDR vectors - these are 4096 columns in the plot, with active cells visible</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> plt.plot(x,y,z,'o')</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> plt.text(4120,row-0.5,ch,fontsize=14);</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> print self.activeArray.nonzero()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Testing NuPIC for Morse decoding </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Create SDRs from Morse Codebook input vectors</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">print "\n \nCreate SDRs from Morse Codebook input vectors"</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># vector size 36x1 for input, 64x64 = 4096 for SDR </span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">example = Morse((36, 1), (64, 64))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x,y = np.meshgrid(np.linspace(1,1,4096), np.linspace(1,1,32))</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">plt.ylim([0,38])</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">plt.xlim([0,4170])</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"># Select the characters to be converted to SDRs</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">#str = 'ABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789'</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">str = 'EISH5 TMO0'</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">row = 1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">for ch in str:</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> example.createInputVector(CODE[ch],ch)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> example.createSDRs(row,x,y,ch)</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"> row = row +1</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">plt.show()</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">plt.clf()</span></div>
</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<h2>
Conclusions
</h2>
<div>
CLA provides promising new technology that is now open for ham radio experimenters to start tinkering with. Building a CLA based Morse decoder would be a good performance benchmark for CLA technology. There are plenty of existing Morse decoders available to compare with and it is fairly easy to test decoder accuracy under noisy conditions. There is also a plenty of audio test material available, including streaming sources like <a href="http://websdr.org/" target="_blank">WebSDR stations</a>.</div>
<div>
<br /></div>
<div>
73</div>
<div>
Mauri </div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0tag:blogger.com,1999:blog-3326773214329183284.post-47413664363661172942014-07-25T01:30:00.002-04:002014-08-24T22:03:13.404-04:00New Morse Decoder - part 6<h2>
Multichannel CW decoder </h2>
Development of the Bayesian CW decoder is moving forward. Many thanks to Dave W1HKJ there is now an alpha build available also for Windows platform. The v3.21.83cw-a4 tarball sources and Windows version are available in <a href="http://www.w1hkj.com/ag1le/" target="_blank">http://www.w1hkj.com/ag1le/</a><br />
<br />
This version has still multiple problems that need to fixed. In Fig 1. below I have screenshot where I have the multichannel signal browser and Fldigi configuration screen for Modems / CW visible. I am running Windows FLDIGI version v3.21.83cw-a4 connected to PowerSDR v2.6.4 and my Flex3000 radio.<br />
<br />
The following description explains the options and expected behavior in this alpha software version. Things are not yet well optimized so you are likely to see a lot of misdecoded signals. I am interested getting feedback and improvement ideas to make the Bayesian decoder better.<br />
<br />
Checkbox "Bayesian decoding" enables multichannel operation. If you have Signal Browser open you can slide the horizontal control at the bottom to adjust the signal decoding threshold. With lower threshold you can enable weaker CW stations to be decoded but often you get just noise and the decoder produces garbage as visible in Fig 1. The software detects peaks in the spectrum and starts a new decoder instance based on the detected peak signal frequency. It tracks each instance on a channel which is rounded at 100 Hz of the audio signal frequency. Number of channels and timeout value can be set in Configure/User Interface/Browser menu. <br />
<br />
If there are two stations in nearby frequencies less than 20 Hz apart the other one is eliminated. This is done to reduce impact of frequency splatter - otherwise one strong CW station would spin up decoders on multiple channels. Also, in this version this process is done for every FFT row in the waterfall display. Because I am not doing any kind of averaging yet the detected peak signal center frequency may be incorrect and therefore decoder is tuned on the wrong frequency. With narrow filter bandwidth setting this may cause garbage in the decoder output. <br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinW-zkdtd_2omRZWmTHeQywJmGm7xEgxKPf7_2rTNTTqeTCNIhmRpQwRfqVUo4r5XQ7yTiUfWBXeuDc0RSz_61IgfreNJ6oZ158dgEyxHfayniCRE5gYd7ZMEhN871Fq51gmWxXK7_0_A/s1600/FLDIGI-3.21.83cwa4.PNG" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinW-zkdtd_2omRZWmTHeQywJmGm7xEgxKPf7_2rTNTTqeTCNIhmRpQwRfqVUo4r5XQ7yTiUfWBXeuDc0RSz_61IgfreNJ6oZ158dgEyxHfayniCRE5gYd7ZMEhN871Fq51gmWxXK7_0_A/s1600/FLDIGI-3.21.83cwa4.PNG" height="640" width="595" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Fig 1. Multichannel CW decoder </td></tr>
</tbody></table>
<br />
In this version each decoder instance uses the same filter bandwidth that is set manually in Configure/Modem/CW/General tab. This means that Bayesian decoder does not have optimal, speed dependent filtering. For faster stations the bandwidth should be larger and for slow stations it can be narrow.<br />
This means that decoding accuracy suffers if there are multiple stations operating at different speeds. Once again this functionality can be improved as the Bayesian decoder does provide a speed estimate automatically but I haven't had time to implement the automatic "Matched filter" feature yet. The filter bandwidth is taken from the "Filter bandwith" slider value for all Bayesian decoder instances.<br />
<br />
On receiver speed estimation Rx WPM value is updated only for selected CW signal on the waterfall. You can also observe that with Bayesian decoder enabled the speed display is updated much more frequently than with the legacy CW decoder. Bayesian decoder calculates speed estimate every 5 msec and provides a WPM value whenever there is a state change (mark -> space or space->mark) in the signal. Sometimes the speed estimator gets "stuck" in lower or upper extreme. You can adjust the "Lower limit" or "Upper Limit" on the CW / General Transmit section - this will give a hint to the speed estimator to re-evaluate speed. Obviously, if the speed estimate is incorrect you will get a lot of garbage text out from the decoder.<br />
<br />
Tracking feature is not implemented properly yet for the Bayesian decoder. This means that if you start to transmit your speed may be different than the station that you were listening. TX WPM is visible in the configuration screen.<br />
<br />
Once again, this is an alpha release provided in order to get some feedback and improvement ideas from FLDIGI users. You can provide feedback by submitting comments below or sending me email to ag1le at innomore dot com. It would be very helpful if you could provide audio samples (WAV files can be recorded using File / Audio / RX Capture feature of FLDIGI), screenshot of what CW parameter settings you are using and general description of the problem or issue you discovered. <br />
<br />
If you are interested in software development I would be very grateful to get some additional help. Building a Bayesian Morse decoder has been a great learning experience in signal processing, machine learning algorithms and probability theory. There are plenty of problems to solve in this space as we build more intelligent software to use Morse code, the international language of hams. <br />
<br />
73<br />
Mauri AG1LE<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<div>
<br /></div>
ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com5tag:blogger.com,1999:blog-3326773214329183284.post-8216018635527173262014-07-17T21:04:00.002-04:002014-07-20T20:56:33.127-04:00New Morse Decoder - part 5<h2>
Multichannel CW decoder for FLDIGI</h2>
I have been working on the Bayesian Morse decoder for a while. The latest effort was focused on making it possible to automatically detect all CW signals in the audio band and spin up a new instance of the Bayesian decoder for each detected signal.<br />
<br />
Figure 1. shows a running version implemented on top of FLDIGI version 3.21.77. The waterfall shows 9 CW signals from 400Hz to 1200 Hz. The software utilizes the FLDIGI Signal Browser user interface and you can set the signal threshold using the slider bar below the signal browser window. The user interface works very much like the PSK or RTTY browser.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCmDJIBsyaO3lETwTEhtq6qtx0rSrA7JO5Niay7B1H7RgUXrPi9BJblfGinyEjqs99xUPLV1JPCqbtAG34ko4Y0O42uC16me8ZHt_QC6vcwRfHDqUeos9UJjUldkYvLk7-q9rUYh1FWY8/s1600/FLDIGI_multichannel_CW_decoding.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCmDJIBsyaO3lETwTEhtq6qtx0rSrA7JO5Niay7B1H7RgUXrPi9BJblfGinyEjqs99xUPLV1JPCqbtAG34ko4Y0O42uC16me8ZHt_QC6vcwRfHDqUeos9UJjUldkYvLk7-q9rUYh1FWY8/s1600/FLDIGI_multichannel_CW_decoding.png" height="514" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. Multichannel CW decoder for FLDIGI</td></tr>
</tbody></table>
The audio file in this demo was created using Octave script below<br />
<br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Fs = 8000; % Fs is sampling frequency - 8 Khz</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">Ts = 10*Fs; % Total sample time is 10 seconds</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">% create 9 different parallel morse sessions - 10 seconds each at 20-35 WPM speed</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">% TEXT audio file noiselevel Hz speed WPM </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x1=morse('CQ TEST DE AG1LE','cw1.wav', 20,1200,Fs,20, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x2=morse('TEST DE SP3RQ CQ','cw2.wav', 15, 1100,Fs,35, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x3=morse('DE W3RQS CQ TEST','cw3.wav', 20, 1000,Fs,30, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x4=morse('SM0LXW CQ TEST DE','cw4.wav',15, 900,Fs, 25, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x5=morse('CQ TEST DE HS1DX','cw5.wav', 20, 800,Fs, 20, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x6=morse('TEST DE JA1DX CQ','cw6.wav', 10, 700,Fs, 20, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x7=morse('DE JA2ATA CQ TEST','cw7.wav',20, 600,Fs, 20, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x8=morse('UA2HH CQ TEST DE','cw8.wav', 15, 500,Fs, 20, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">x9=morse('CQ TEST DE CT1CX','cw9.wav', 20, 400,Fs, 20, Ts);</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">% weighted sum - merge all the audio streams together </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">% 9 signals arranged in frequency order 1200Hz ... 400Hz</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">y = 0.1*x1 + 0.1*x2 + 0.1*x3 + 0.1*x4 + 0.1*x5 + 0.1*x6 + 0.1*x7 + 0.1*x8 + 0.1*x9;</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;"><br /></span>
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">% write to cwcombo.wav file </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: xx-small;">wavwrite(y,Fs,'cwcombo.wav');</span><br />
<br />
I have saved the full sources of this experimental FLDIGI version in Github: <a href="https://github.com/ag1le/morse-wip/blob/master/test/fldigi-3.21.77.multich.tar.gz" target="_blank">FLDIGI source</a><br />
<span style="background-color: yellow;">UPDATE July 20, 2014: I rebased this using Source Forge latest branch - new version is here: <a href="http://www.w1hkj.com/ag1le/" target="_blank">fldigi-3.22.0CN.tar.gz</a></span><br />
<br />
Let me know if you are interested in testing this software. I would be interested in getting feedback on scalability, any performance problems as well as how well the CW decoder works with real life signals.<br />
<br />
73<br />
Mauri AG1LE<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com31tag:blogger.com,1999:blog-3326773214329183284.post-14708808038139362162014-06-11T22:04:00.003-04:002014-07-10T00:58:01.682-04:00New Morse Decoder - Part 4In <a href="http://ag1le.blogspot.com/2013/12/new-morse-decoder-part-3.html" target="_blank">the previous blog entry</a> I shared some test results of the new experimental Bayesian Morse decoder. Thanks to the alpha testers I found the bug that was causing the sensitivity to signal amplitudes and causing overflow. I have fix this bug in the software over the last few months.<br />
<br />
CQ WW WPX CW contest was in May 24-25 and it presented a good opportunity to put the new FLDIGI software version to test. I worked some stations and let the software running for 48 hours to test the stability.<br />
<br />
In the figure 1 the first 2 1/2 lines are decoded using the new Bayesian CW decoder. The following 2 lines is the same audio material decoded using the legacy CW decoder. CW settings are also visible in the picture. Matched filter bandwidth was set to 50Hz based on Rx speed of 32 WPM.<br />
<br />
The legacy decoder is doing a pretty good job following ZF1A working CW contest at 7005.52 KHz. At first look it appears that the new Bayesian decoder is having a lot of difficulties. Let's have a closer look what is really going on here.<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7cvUnNpm6mLY1uD89T2ke_NSsdgs2iACUSTjqynLb7gzB4XTqpEE-RUFsrHCd9-hrT0m4738KvnNOxJnHJSti7PdSAkxXR_pgHOm4hL-SMRJ4m0JMItE6PvnTWsVpL2nEWNDs5-WHquY/s1600/CQ_WW_WPX_bayesian2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh7cvUnNpm6mLY1uD89T2ke_NSsdgs2iACUSTjqynLb7gzB4XTqpEE-RUFsrHCd9-hrT0m4738KvnNOxJnHJSti7PdSAkxXR_pgHOm4hL-SMRJ4m0JMItE6PvnTWsVpL2nEWNDs5-WHquY/s1600/CQ_WW_WPX_bayesian2.png" height="334" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 1. FLDIGI CW Decoder testing </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
In the audio recording ZF1A made 5 contacts, with VE1RGB, UR4MF, KP2M, SM6BZV and WA2MCR in 1 min 31 seconds.<br />
<br />
I let the captured audio file run in a loop for two times using both FLDIGI CW decoder versions. The decoded text is shown below, broken into separate lines by each QSO for clarity.<br />
<br />
<b>FLDIGI - the new experimental Bayesian Morse decoder: </b><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">RN 512 X </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N VE1RGB 5NN513 X </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A --R4MF 5NN 0MO0 N T E E E E E E E TU<br /> ------TM N T E XJ0TT 6NN 515 X </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N QT SM6BZV 5NM516 X </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N WA2MCR 5NN 517 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">NN 5 --2 B</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N VE1RGB 5MK 613 X </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N KR4MF 5NN 514 X </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N KP2M 6NN 515 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A N OT SM6BZV 5NN 516 X</span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZH1A N WA2MCR 5NN 517 </span><br />
<br />
The first line should have been decoded as <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">NN 512 TU</span> but the Bayesian decoder misdecoded last 2 letters. What was originally TU was decoded as X.<br />
<br />
Let's look at the signal waveform (Fig 2.). It is a close call when looking at the waveform timing...same decoding error happens in almost all the cases above. <br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMZE1gaoDY-a2D2KFkQ0ChhpvkL3SS2CBxbMnaWIWhGOdtPrsrKsAFBCFBe6YJS4jgZdt1e7JXiGEFUkzNNQ9n1Y3yvf9WPPaYrn6pkWnxzrkZPRDVxngfxmOTYLi4LwctB1zO63Kv8h8/s1600/CQ_WW_WPX_TU_vs_X2.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMZE1gaoDY-a2D2KFkQ0ChhpvkL3SS2CBxbMnaWIWhGOdtPrsrKsAFBCFBe6YJS4jgZdt1e7JXiGEFUkzNNQ9n1Y3yvf9WPPaYrn6pkWnxzrkZPRDVxngfxmOTYLi4LwctB1zO63Kv8h8/s1600/CQ_WW_WPX_TU_vs_X2.png" height="200" width="171" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 2. TU or X ? </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
So what happened in the QSO with UR4MF? Let's look at waterfall picture (Fig 3.) for possible clues.<br />
UR2MF is very visible at 692 Hz frequency but what is the faint signal that starts 200 msec before letter U? It is approximately 70Hz below UR2MF and starts "dah-dit".<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4_juHKx0e4luQp2mI8RkfoaQLAUmCpXAlAAuPgSZpS4y_BS50Vd_c0Sfunn7_db6WVUlSAfakK8qOfbtkCw36MXGE56z3b8qBGrsodWp-rYobx5xDyo_AjxgVMxN_TRHnTAL1bLn6u6o/s1600/CQ_WW_WPX_UR4MF.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4_juHKx0e4luQp2mI8RkfoaQLAUmCpXAlAAuPgSZpS4y_BS50Vd_c0Sfunn7_db6WVUlSAfakK8qOfbtkCw36MXGE56z3b8qBGrsodWp-rYobx5xDyo_AjxgVMxN_TRHnTAL1bLn6u6o/s1600/CQ_WW_WPX_UR4MF.png" height="72" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 3. UR4MF with another station 70 Hz below</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
The new Bayesian decoder is sensitive enough to be able to pick up this other station, but unfortunately the selected 50 Hz filter bandwidth is not enough to separate these two stations from each other, thus causing what appears an error in decoding. Legacy decoder did not even notice this other station.<br />
<br />
<br />
<br />
<b>FLDIGI - legacy Morse decoder: </b><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">6N S W2 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">F1 VE1RGB 5NN 513 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A UR4MF N 514 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A KP2M 5NN 515 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1 SM6BZV 5NN 516 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A WA2MCR 5NN 17 </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">NN S W2 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">F1 VE1RGB 5NN 513 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A UR4MF E N 514 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A KP2M 5NN 515 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1 SM6BZV 5NN 516 TU </span><br />
<span style="font-family: Courier New, Courier, monospace; font-size: x-small;">ZF1A WA2MCR 5NN 17</span><br />
<br />
<br />
First line should be <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">NN 512 TU</span> but deep QSB in the signal mangled the decoding into <span style="font-family: Courier New, Courier, monospace; font-size: x-small;">6N S W2 TU</span>.<br />
See figure 4 on how the amplitude goes down rapidly between 5 and 1. The first "dit" in number one is barely visible in waterfall figure 5 below. While legacy decoder made an error in this case the new Bayesian decoder didn't seem to have any problem despite this deep and rapid QSB.<br />
<br />
Once again, the Bayesian decoder appears to be much more sensitive and automatically able to pickup signals even under deep QSB conditions.<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMzSFl5tnWpPtP5py48fm-IL5wESZuxFSF7BXKxw-Vq9UPYxUafa-kQUp70Vv9ek20VhiBTxEpMCwugGmp-E8JTEz6CGvYITa8JxaMBtt89RThQYcf5kzdVXVFusko6_qgjTORddMFtJs/s1600/CQ_WW_WPX_QSB.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgMzSFl5tnWpPtP5py48fm-IL5wESZuxFSF7BXKxw-Vq9UPYxUafa-kQUp70Vv9ek20VhiBTxEpMCwugGmp-E8JTEz6CGvYITa8JxaMBtt89RThQYcf5kzdVXVFusko6_qgjTORddMFtJs/s1600/CQ_WW_WPX_QSB.png" height="207" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 4. QSB </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<table cellpadding="0" cellspacing="0" class="tr-caption-container" style="float: left; margin-right: 1em; text-align: left;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZVaLx9NuWrJGwHsA_irr1_29lWychoR-0vbHxEMnyEN3R_0tgDFnusgu2lgU8IKA6168W_A3yIHr2SYLxVrgINVvItQkKo7mncF8GD47XdsfT0TBcO9YiwkTyOHdhggcQh1tiyTncm7Q/s1600/CQ_WW_WPX_QSB_waterfall.png" imageanchor="1" style="clear: left; margin-bottom: 1em; margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjZVaLx9NuWrJGwHsA_irr1_29lWychoR-0vbHxEMnyEN3R_0tgDFnusgu2lgU8IKA6168W_A3yIHr2SYLxVrgINVvItQkKo7mncF8GD47XdsfT0TBcO9YiwkTyOHdhggcQh1tiyTncm7Q/s1600/CQ_WW_WPX_QSB_waterfall.png" height="101" width="400" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 5. QSB in waterfall</td></tr>
</tbody></table>
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<br />
<h3>
CONCLUSIONS</h3>
Testing the performance of the new, experimental Bayesian Morse decoder with real world contest signals did show some surprising behaviors. What appeared initially to be errors in decoding turned out to be real signals that impacted the probabilistic decoding process. It also appears that the Bayesian method is much more sensitive and may need a bit different strategy related to signal pre-processing and pre-filtering compared to the legacy Morse decoder currently implemented in FLDIGI.<br />
<br />
I am working on a better test framework to experiment more systematically the impact of various parameters to find the performance limits of the new experimental Bayesian Morse decoder. Early results show for example that minimum CER (character error rate) is dependent on SNR of signal as expected, but the CW speed dependent pre-filter bandwidth has some interesting effects on CER as demonstrated in figure 6. Different graphs show how the speed (in WPM) setting impacts CER at different pre-filter bandwidth (for example 20 WPM ~ 33.3Hz, 80 WPM ~ 133.3 Hz ). You would expect best CER performance with the most narrow filter setting that matches the actual speed (in this case actual speed was 20 WPM). However, as seen in figure 6 this is not consistently the case with signals between SNR +2 ... +15 dB. <br />
<br />
<br />
<div style="text-align: left;">
</div>
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkkZ-IiBued4CgUZNMXL6NB1PqFeuAeW73NNuXTwXtah0TYcLlys53NoM7CpQxgTRuM_l8ccCm17RjALyeAGx39NFbO5Urc96vMazm4DIYyzrzTBmkAsEh92z05fF8V0ro5bEVvU5-J3E/s1600/CER_vs_SNR_Bayesian_June_11_2014.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkkZ-IiBued4CgUZNMXL6NB1PqFeuAeW73NNuXTwXtah0TYcLlys53NoM7CpQxgTRuM_l8ccCm17RjALyeAGx39NFbO5Urc96vMazm4DIYyzrzTBmkAsEh92z05fF8V0ro5bEVvU5-J3E/s1600/CER_vs_SNR_Bayesian_June_11_2014.png" height="578" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Figure 6. CER vs. SNR testing at different WPM settings </td></tr>
</tbody></table>
<br />
<br />
<br />
<br />ag1lehttp://www.blogger.com/profile/16415319751367496314noreply@blogger.com0