Ganssle Sign Up Information

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• The Rust Programming Language
• Encoder Bouncing
• On R&D
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia.

Arm has released a free e-book of interest to people integrating Cortex-M parts into their SoCs: System-on-Chip Design with Arm Cortex-M Processors.

Be careful what you read. Renesas's new S1JA SoC is based on the Cortex-M23 and appears to be a heck of a part, with just about every peripheral you can imagine. But a writeup on embedded.com reads, in part: "The microcontrollers' ultra-low power extends battery life for battery-operated portable and battery backup applications. The software standby mode consumes only 500 nA to enable 20-year battery-operated applications that spend extended periods in sleep mode." Unless the "battery" is a radioisotope generator the 20-year claim exceeds the shelf-life of the batteries used in portable electronics. Getting years of life from a battery is a hard problem with many subtle issues. For more info see this (though that paper focuses on the CR2032 cell, the concepts scale to most battery types).

"It is better to define your system up front to minimize errors, rather than producing a bunch of code that then has to be corrected with patches on patches" Margaret Hamilton of Apollo Guidance Computer fame.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Thor wrote:

This month's giveaway is a copy of Embedded Systems World-Class Designs.

Enter via this link.

I'm getting a lot of email about the Rust language, in particular, in using Rust on embedded projects.

There's a lot to like about the language. It superficially looks like C/C++.

At compile time a borrow-checker tests for memory safety. Basically, each value has an owner. There's only one owner of a variable at a time, and when the owner goes out of scope, the value is dropped. Many people scream at the borrow-checker as it can cause headaches, but users report after a while they learn to abide by Rust's rules, and get a lot of benefit from the memory-safety checks, as it eliminates data races and other common problems.

Here's an example factorial program, from the Wikipedia entry:

fn factorial(i: u64) -> u64 

{

  let mut acc = 1;

  for num in 2..=i 

    {

    acc *= num;

    }      

  acc

}

Where's the return argument? In Rust, if there's no return statement the last expression is the return value. I find that strangely appealing. The "let" keyword reminds me of the bad old days of Dartmouth BASIC.

Rust is used in embedded development, and the main site has a sub-page about this, with examples using a Cortex M processor.

While C is a great language for low-level programming and embedded development, it lacks modern features like strong typing (and so much more) that helps create correct programs. The SPARK people call this "correctness by construction." Ada, SPARK, and to a lesser extent Rust enforce rules which lead to safer code. Studies (e.g., Software Static Code Analysis Lessons Learned, Andy German, QinetiQ Ltd., Nov 2003 Crosstalk) show programs in Ada and SPARK have only a fraction of the defects found using C. To my knowledge there's no such data about Rust, but it seems reasonable that Rust code will be less error-prone than C code.

I have seen no data on Rust's efficiency (in size and speed), bus suspect it's similar to C.

However, as fascinating as it is to learn new languages, there's one place where Rust fails. It's evolving at a rapid pace, and there is no ANSI standard. That's generally not much of a problem in web development or for small embedded projects. But many embedded systems have lives of years and decades. In the railroad industry support has to be guaranteed for 30 years. What will Rust look like a decade or two hence? Will you be able to get a compiler that handles the old dialect? If you have hundreds of thousands of lines of Rust legacy code, reuse might be impossible if all of that is coded using a long-obsolete version.

C made its way into the embedded space in the 1980s. Lots of compilers abounded. None were compatible. Remember Manx C? That decade was a veritable Babel of incompatible Cs which was resolved only with the advent of standardization under ANSI C 89/90.

As technologists we want the best possible solution. Yet sometimes that has to take second place to business realities. Every engineering effort involves some amount of risk, but we do want to minimize that risk, especially if it could have devastating long-term consequences. Megabucks invested in developing code that can no longer be compiled is a risk that is not, in my opinion, worth taking.

I look forward to the day when Rust joins the stable of standardized languages, and have no doubt we'll see that happen in the next decade or so.

Standardization efforts move slowly. ANSI spent six years working on the first C standard.

Recent Muses (TEM 384 and TEM 385) had discussions about encoders, which engendered quite a bit of interesting email correspondence. I decided to look at bounce characteristics of a few models. I picked low-cost devices as these are common in controls (e.g., radio volume controls). My focus was (mostly) on mechanical encoders, as it's reasonable to assume these will have significant bounce issues.

Bourns's PEC11R-4215F mechanical encoder gives 24 PPR (pulses per revolution). The datasheet specs 2 ms bounce though that is footnoted "Devices are tested using standard noise reduction filters." It's not clear to me if the 2 ms was with or without the filter, but the unit behaved much better than specified. With no filter, this is the worst bouncing I saw:

That's pretty ugly, but the bouncing lasted only about 250 µs.

The datasheet recommends this circuit to clean up the noise:

With it, I saw no bouncing. The resulting waveform gives rise and fall times greater than expected from the RC time constant (3.7 ms rise, 1.9 ms fall); presumably there's some internal resistance:

CTS's 288V232R161B2 mechanical encoder gives 4 PPR. 5 ms bounce is specified. No filter circuit is recommended. The worst bounce I saw was about 300 µs.:

CUI Devices's ACZ11BR4E-20KQ01-12C gives 12 PPR. No bounce data is given in the datasheet. Here's the worst I saw - about 5 ms of terror:

The recommended filter circuit is identical to the one Bourns suggests (see above). With it, well, the results were not great:

The scope is set to 10 ms/division. Some of those supposedly-filtered zits exceed 2V. Even a Schmitt trigger, like the 74HCT14, will pass these on as valid logic signals. At least the bounces are very narrow. One could write some code that rejects pulses shorter than a ms or so.

Just to put this data in perspective, I also tried an optical encoder, expecting clean outputs. And indeed, there was no bouncing. This is the Grayhill 62AG11-L0-P, which sports 32 PPR:

Note the slow rise and fall times, which I saw at times exceeding 8 ms. That is better, however, than the rated 30 ms worst-case numbers. I presume this is an artifact of the optical interrupter swinging into the light beam. Be sure to figure this into your max rotation speed. Assuming 30 ms rise time, and another 30 ms fall time, with 32 PPR the max speed will be under 0.5 rotations/second. I recently rented a Toyota; the radio was almost unusable as turning the tuning knob caused wild and unpredicable results. A clockwise rotation often increased the frequency, as expected, but sometimes the the tuner jumped down or crazily up. One wonders if the engineers used an encoder yet didn't anticipate

For ideas about debouncing, see this.

Research and Development. R&D. It's the lifeblood of tech companies, and it's what we engineers do all day, every day.

Nonsense.

There's no such thing as R&D. There's R, and there's D, and the two are completely separate activities.

Research is all about discovering new things. It's the science that ultimately enables the products we build, the metaphorical man-behind-the-curtain pulling the levers to control the machines we create.

Research might also involve discovering new algorithms, like new ways to smooth signals or compress data. "New" might mean new to us but not to the world. So we research an idea or a need, and then switch to development mode. The result of research is an approach that one can then implement.

Development is taking known ideas and using them to build products. That's the bulk of an engineer's work. We transform an algorithm to something physical, like converting a CRC algorithm to C code, to VHDL inside an FPGA, or to logic components.

One of my top ten reasons for failed projects is "bad science," or the inability to separate R from D. When a company starts building a product before really understanding what is being measured the schedule is doomed. Start coding an algorithm without it being sharply defined and, at best, you'll wander aimlessly till, with luck, settling on an approach that works.

Research simply can't be scheduled. If you don't believe that, please develop a (realistic) schedule for discovering the cure for cancer.

You might be able to guesstimate simple research schedules, like doing a search for a known algorithm, but even that is, in my experience, very difficult to estimate. The first "Eureka" is often followed by disappointment when a little experimenting reveals some fatal flaw, requiring more research to find a better approach.

Yet I constantly see teams conflating R and D, leading inevitably to late or failed projects.

Sure, there are some projects that necessarily pursue R and D in parallel. But those care rarely be scheduled with any accuracy.

What do you think? Have you ever had a project disaster because you were doing R and the same time as D?

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Standards, n.: The principles we use to reject other people's code.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• The Rust Programming Language
• Encoder Bouncing
• On R&D
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia.

Arm has released a free e-book of interest to people integrating Cortex-M parts into their SoCs: System-on-Chip Design with Arm Cortex-M Processors.

Be careful what you read. Renesas's new S1JA SoC is based on the Cortex-M23 and appears to be a heck of a part, with just about every peripheral you can imagine. But a writeup on embedded.com reads, in part: "The microcontrollers' ultra-low power extends battery life for battery-operated portable and battery backup applications. The software standby mode consumes only 500 nA to enable 20-year battery-operated applications that spend extended periods in sleep mode." Unless the "battery" is a radioisotope generator the 20-year claim exceeds the shelf-life of the batteries used in portable electronics. Getting years of life from a battery is a hard problem with many subtle issues. For more info see this (though that paper focuses on the CR2032 cell, the concepts scale to most battery types).

"It is better to define your system up front to minimize errors, rather than producing a bunch of code that then has to be corrected with patches on patches" Margaret Hamilton of Apollo Guidance Computer fame.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Thor wrote:

This month's giveaway is a copy of Embedded Systems World-Class Designs.

Enter via this link.

I'm getting a lot of email about the Rust language, in particular, in using Rust on embedded projects.

There's a lot to like about the language. It superficially looks like C/C++.

At compile time a borrow-checker tests for memory safety. Basically, each value has an owner. There's only one owner of a variable at a time, and when the owner goes out of scope, the value is dropped. Many people scream at the borrow-checker as it can cause headaches, but users report after a while they learn to abide by Rust's rules, and get a lot of benefit from the memory-safety checks, as it eliminates data races and other common problems.

Here's an example factorial program, from the Wikipedia entry:

fn factorial(i: u64) -> u64 

{

  let mut acc = 1;

  for num in 2..=i 

    {

    acc *= num;

    }      

  acc

}

Where's the return argument? In Rust, if there's no return statement the last expression is the return value. I find that strangely appealing. The "let" keyword reminds me of the bad old days of Dartmouth BASIC.

Rust is used in embedded development, and the main site has a sub-page about this, with examples using a Cortex M processor.

While C is a great language for low-level programming and embedded development, it lacks modern features like strong typing (and so much more) that helps create correct programs. The SPARK people call this "correctness by construction." Ada, SPARK, and to a lesser extent Rust enforce rules which lead to safer code. Studies (e.g., Software Static Code Analysis Lessons Learned, Andy German, QinetiQ Ltd., Nov 2003 Crosstalk) show programs in Ada and SPARK have only a fraction of the defects found using C. To my knowledge there's no such data about Rust, but it seems reasonable that Rust code will be less error-prone than C code.

I have seen no data on Rust's efficiency (in size and speed), bus suspect it's similar to C.

However, as fascinating as it is to learn new languages, there's one place where Rust fails. It's evolving at a rapid pace, and there is no ANSI standard. That's generally not much of a problem in web development or for small embedded projects. But many embedded systems have lives of years and decades. In the railroad industry support has to be guaranteed for 30 years. What will Rust look like a decade or two hence? Will you be able to get a compiler that handles the old dialect? If you have hundreds of thousands of lines of Rust legacy code, reuse might be impossible if all of that is coded using a long-obsolete version.

C made its way into the embedded space in the 1980s. Lots of compilers abounded. None were compatible. Remember Manx C? That decade was a veritable Babel of incompatible Cs which was resolved only with the advent of standardization under ANSI C 89/90.

As technologists we want the best possible solution. Yet sometimes that has to take second place to business realities. Every engineering effort involves some amount of risk, but we do want to minimize that risk, especially if it could have devastating long-term consequences. Megabucks invested in developing code that can no longer be compiled is a risk that is not, in my opinion, worth taking.

I look forward to the day when Rust joins the stable of standardized languages, and have no doubt we'll see that happen in the next decade or so.

Standardization efforts move slowly. ANSI spent six years working on the first C standard.

Recent Muses (TEM 384 and TEM 385) had discussions about encoders, which engendered quite a bit of interesting email correspondence. I decided to look at bounce characteristics of a few models. I picked low-cost devices as these are common in controls (e.g., radio volume controls). My focus was (mostly) on mechanical encoders, as it's reasonable to assume these will have significant bounce issues.

Bourns's PEC11R-4215F mechanical encoder gives 24 PPR (pulses per revolution). The datasheet specs 2 ms bounce though that is footnoted "Devices are tested using standard noise reduction filters." It's not clear to me if the 2 ms was with or without the filter, but the unit behaved much better than specified. With no filter, this is the worst bouncing I saw:

That's pretty ugly, but the bouncing lasted only about 250 µs.

The datasheet recommends this circuit to clean up the noise:

With it, I saw no bouncing. The resulting waveform gives rise and fall times greater than expected from the RC time constant (3.7 ms rise, 1.9 ms fall); presumably there's some internal resistance:

CTS's 288V232R161B2 mechanical encoder gives 4 PPR. 5 ms bounce is specified. No filter circuit is recommended. The worst bounce I saw was about 300 µs.:

CUI Devices's ACZ11BR4E-20KQ01-12C gives 12 PPR. No bounce data is given in the datasheet. Here's the worst I saw - about 5 ms of terror:

The recommended filter circuit is identical to the one Bourns suggests (see above). With it, well, the results were not great:

The scope is set to 10 ms/division. Some of those supposedly-filtered zits exceed 2V. Even a Schmitt trigger, like the 74HCT14, will pass these on as valid logic signals. At least the bounces are very narrow. One could write some code that rejects pulses shorter than a ms or so.

Just to put this data in perspective, I also tried an optical encoder, expecting clean outputs. And indeed, there was no bouncing. This is the Grayhill 62AG11-L0-P, which sports 32 PPR:

Note the slow rise and fall times, which I saw at times exceeding 8 ms. That is better, however, than the rated 30 ms worst-case numbers. I presume this is an artifact of the optical interrupter swinging into the light beam. Be sure to figure this into your max rotation speed. Assuming 30 ms rise time, and another 30 ms fall time, with 32 PPR the max speed will be under 0.5 rotations/second. I recently rented a Toyota; the radio was almost unusable as turning the tuning knob caused wild and unpredicable results. A clockwise rotation often increased the frequency, as expected, but sometimes the the tuner jumped down or crazily up. One wonders if the engineers used an encoder yet didn't anticipate

For ideas about debouncing, see this.

Research and Development. R&D. It's the lifeblood of tech companies, and it's what we engineers do all day, every day.

Nonsense.

There's no such thing as R&D. There's R, and there's D, and the two are completely separate activities.

Research is all about discovering new things. It's the science that ultimately enables the products we build, the metaphorical man-behind-the-curtain pulling the levers to control the machines we create.

Research might also involve discovering new algorithms, like new ways to smooth signals or compress data. "New" might mean new to us but not to the world. So we research an idea or a need, and then switch to development mode. The result of research is an approach that one can then implement.

Development is taking known ideas and using them to build products. That's the bulk of an engineer's work. We transform an algorithm to something physical, like converting a CRC algorithm to C code, to VHDL inside an FPGA, or to logic components.

One of my top ten reasons for failed projects is "bad science," or the inability to separate R from D. When a company starts building a product before really understanding what is being measured the schedule is doomed. Start coding an algorithm without it being sharply defined and, at best, you'll wander aimlessly till, with luck, settling on an approach that works.

Research simply can't be scheduled. If you don't believe that, please develop a (realistic) schedule for discovering the cure for cancer.

You might be able to guesstimate simple research schedules, like doing a search for a known algorithm, but even that is, in my experience, very difficult to estimate. The first "Eureka" is often followed by disappointment when a little experimenting reveals some fatal flaw, requiring more research to find a better approach.

Yet I constantly see teams conflating R and D, leading inevitably to late or failed projects.

Sure, there are some projects that necessarily pursue R and D in parallel. But those care rarely be scheduled with any accuracy.

What do you think? Have you ever had a project disaster because you were doing R and the same time as D?

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Standards, n.: The principles we use to reject other people's code.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• DO-178C On Requirements
• Benchmarking Firmware Teams
• Rust Redux
• More on R&D
• Jobs!
• Joke for the Week
• About The Embedded Muse

Happy Beethoven's birthday!

Some HP SSDs "stop running after the 32,768-hour mark." The citation is from an article in Electronic Design. The language is a bit muddy; I expect they fail as the time rolls over from 32,767 to 32,768 hours. Bugs are all-too-common but consider this: That's almost four years. No reasonable amount of testing would uncover the problem. While test is critical, it's inadequate. Without access to the code it's impossible to be certain, but I suspect even achieving 100% test coverage would not have picked up the problem.

Test is just one of many filters needed to insure correct code.

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Re: Wilbur Wright: "Wilbur was a man who established a goal with care, then never lost sight of it. He was the perfect engineer - isolating a basic problem, defining it in the most precise terms, and identifying the missing bits of information that would enable him to solve it. Other students of the subject lost themselves in a welter of confusing details; they were lured into extraneous, if fascinating, blind alleys that led away from the basic problem." From the book The Bishop's Boys by Tom Crouch.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Mat Bennion raised one of the issues we hear about from time to time with the use of C:

In a time when so many articles in the trade press are thinly-disguised press releases, it's important to acknowledge those pushing back against this sort of uber-marketing. Bill Schweber's articles in Electronic Design consistently provide information of use to engineers. They typically describe how some component or circuit works, like this week's excellent piece that describes flyback circuits.

Curt Bruns won the power supply in the November contest.

This month's giveaway is a copy of Embedded Systems World-Class Designs.

Enter via this link.

It's believed, until the recent 737-MAX crashes, that no one had ever lost a life in a commercial aircraft due to a software problem. Why is this?

No one really knows. All commercial aircraft software in the USA is crafted to DO-178C. Is it the use of this standard, or is it a safety culture that is willing to embrace the burdensome regs?

Regardless, few industries regularly achieve the high quality of code done to DO-178C. Harold Kraus sent in the following about the importance of requirements in avionics.

But first, a couple of definitions for those unfamiliar with DO-178C:

• DO-254: The avionics standard for electronic hardware
• DER: Designated Engineering Representative - an individual who may approve, or recommend that the FAA approves, the technical data produced.

Here's Harold's write-up:

How does your team compare to others in the industry? Is it world-class, or sub-par?

Benchmarking is the practice of comparing some aspect of a business with other companies in the same industry. For instance, a business might average 46 days to collect a receivable. What does that mean? If the industry average is 35 days, 46 is nothing to brag about. If 80% of their competitors do worse, then 46 is a great number. It's likely going to be hard to gain much improvement.

I've advocated benchmarking firmware teams for many years. Alas, it's hard to get good numbers, and the standard deviations are high. But we do have some data. In my Better Firmware Faster class I stress the importance of engineering with numbers. As David Aikin said: "Engineering is done with numbers. Analysis without numbers is only an opinion."

For instance, most teams average a 5 to 10% defect rate after getting a clean compile before any testing starts. Worst-in-class engineers generate 250 defects per KLOC. There's no measurable difference between C and C++. So for a 10KLOC project figure on chasing 500 to 1,000 problems.

Ever wonder why projects run late? Clearly these numbers are abysmal. Cut them by half and the schedule will improve dramatically.

Ada teams are about an order of magnitude better, and SPARK developers crush even Ada.

Where do bugs come from? This table shows defects/KLOC for a poorly-performing team, and one rated at Capability Maturity Level 3, considered a pretty disciplined outfit:

And here's where defects are removed:

Defect Removal Efficiency (DRE) is one of those tortured terms academics love. It's merely the percentage of defects removed during engineering and up to 90 days post-initial delivery. You'd hope for 100%. Here's where the industry stands. "Bugs/KLOC" is the number of shipped bugs per thousand lines of code:

Poor quality teams tend to rely mostly on test to find defects. Capers Jones identifies over 60 ways to minimize these, though no one does or should use all of those. I view this as a series of filters, each stage removing some percentage of problems. It has been shown that test, by itself, typically finds only 50%.

For more numbers Capers Jones' books are a great resource. The sadly-defunct Crosstalk Magazine was also a valuable resource.

How does your team measure up?

Muse 387's comments about the Rust programming language generated a lot of discussion. Nick P's thoughts were useful:

Tock is an RTOS written in Rust. The Case for Writing a Kernel in Rust is a worthwhile paper about how Rust is a good choice for RTOS work, though I stand by my thoughts in Muse 387.

A number of people responded to last issue's comments on R&D.

Charles Manning wrote:

From Mike Davis:

Rob Milne commented:

Emil Imrith had a suggestion:

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

bug, n: A son of a glitch.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• DO-178C On Requirements
• Benchmarking Firmware Teams
• Rust Redux
• More on R&D
• Jobs!
• Joke for the Week
• About The Embedded Muse

Happy Beethoven's birthday!

Some HP SSDs "stop running after the 32,768-hour mark." The citation is from an article in Electronic Design. The language is a bit muddy; I expect they fail as the time rolls over from 32,767 to 32,768 hours. Bugs are all-too-common but consider this: That's almost four years. No reasonable amount of testing would uncover the problem. While test is critical, it's inadequate. Without access to the code it's impossible to be certain, but I suspect even achieving 100% test coverage would not have picked up the problem.

Test is just one of many filters needed to insure correct code.

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Re: Wilbur Wright: "Wilbur was a man who established a goal with care, then never lost sight of it. He was the perfect engineer - isolating a basic problem, defining it in the most precise terms, and identifying the missing bits of information that would enable him to solve it. Other students of the subject lost themselves in a welter of confusing details; they were lured into extraneous, if fascinating, blind alleys that led away from the basic problem." From the book The Bishop's Boys by Tom Crouch.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Mat Bennion raised one of the issues we hear about from time to time with the use of C:

In a time when so many articles in the trade press are thinly-disguised press releases, it's important to acknowledge those pushing back against this sort of uber-marketing. Bill Schweber's articles in Electronic Design consistently provide information of use to engineers. They typically describe how some component or circuit works, like this week's excellent piece that describes flyback circuits.

Curt Bruns won the power supply in the November contest.

This month's giveaway is a copy of Embedded Systems World-Class Designs.

Enter via this link.

It's believed, until the recent 737-MAX crashes, that no one had ever lost a life in a commercial aircraft due to a software problem. Why is this?

No one really knows. All commercial aircraft software in the USA is crafted to DO-178C. Is it the use of this standard, or is it a safety culture that is willing to embrace the burdensome regs?

Regardless, few industries regularly achieve the high quality of code done to DO-178C. Harold Kraus sent in the following about the importance of requirements in avionics.

But first, a couple of definitions for those unfamiliar with DO-178C:

• DO-254: The avionics standard for electronic hardware
• DER: Designated Engineering Representative - an individual who may approve, or recommend that the FAA approves, the technical data produced.

Here's Harold's write-up:

How does your team compare to others in the industry? Is it world-class, or sub-par?

Benchmarking is the practice of comparing some aspect of a business with other companies in the same industry. For instance, a business might average 46 days to collect a receivable. What does that mean? If the industry average is 35 days, 46 is nothing to brag about. If 80% of their competitors do worse, then 46 is a great number. It's likely going to be hard to gain much improvement.

I've advocated benchmarking firmware teams for many years. Alas, it's hard to get good numbers, and the standard deviations are high. But we do have some data. In my Better Firmware Faster class I stress the importance of engineering with numbers. As David Aikin said: "Engineering is done with numbers. Analysis without numbers is only an opinion."

For instance, most teams average a 5 to 10% defect rate after getting a clean compile before any testing starts. Worst-in-class engineers generate 250 defects per KLOC. There's no measurable difference between C and C++. So for a 10KLOC project figure on chasing 500 to 1,000 problems.

Ever wonder why projects run late? Clearly these numbers are abysmal. Cut them by half and the schedule will improve dramatically.

Ada teams are about an order of magnitude better, and SPARK developers crush even Ada.

Where do bugs come from? This table shows defects/KLOC for a poorly-performing team, and one rated at Capability Maturity Level 3, considered a pretty disciplined outfit:

And here's where defects are removed:

Defect Removal Efficiency (DRE) is one of those tortured terms academics love. It's merely the percentage of defects removed during engineering and up to 90 days post-initial delivery. You'd hope for 100%. Here's where the industry stands. "Bugs/KLOC" is the number of shipped bugs per thousand lines of code:

Poor quality teams tend to rely mostly on test to find defects. Capers Jones identifies over 60 ways to minimize these, though no one does or should use all of those. I view this as a series of filters, each stage removing some percentage of problems. It has been shown that test, by itself, typically finds only 50%.

For more numbers Capers Jones' books are a great resource. The sadly-defunct Crosstalk Magazine was also a valuable resource.

How does your team measure up?

Muse 387's comments about the Rust programming language generated a lot of discussion. Nick P's thoughts were useful:

Tock is an RTOS written in Rust. The Case for Writing a Kernel in Rust is a worthwhile paper about how Rust is a good choice for RTOS work, though I stand by my thoughts in Muse 387.

A number of people responded to last issue's comments on R&D.

Charles Manning wrote:

From Mike Davis:

Rob Milne commented:

Emil Imrith had a suggestion:

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

bug, n: A son of a glitch.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• On Requirements Documents
• More New Years Resolutions
• On E-Voting
• The Secret of My Success
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Engineering changes at an accelerating pace. The boss wants more bug-free code, today. Tomorrow he'll want even more in the same time-frame. Are you on top of the best ways to deliver it?

Some tell me they just don't have time to attend seminars. I'm reminded of this:

It is important to fight today's engineering battle. But the war will be lost if you don't find better ways to win.

This is what my one-day Better Firmware Faster seminar is all about: giving your team the tools they need to operate at a measurably world-class level, producing code with far fewer bugs in less time. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends today. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Jack's latest blog: A Lack of Forethought

I never give anyone hell. I only tell the truth. They think it's hell. Harry S. Truman

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

This month's giveaway is a Cypress CY8KIT-044 PSoC 4 M-series Pioneer kit.

Enter via this link.

Brian Cuthie reports that the schematic editor/simulation software Micro-Cap is now free. See this.

How long are your requirements documents? How complete are they?

Getting hard data is notoriously difficult in the software world. Capers Jones probably has more than anyone. He stresses the large standard deviation, but makes it clear that one can gain some insight from it. I view the data as an impressionist painting, where the outlines might be fuzzy, but one can still make out the general shape of things.

The following table is derived from data in Jones' The Economics of Software Quality. It shows how many pages of requirements are found for various-sized projects. The "Requirements Completeness" column illustrates all-to-pathetically how short those pages are of the delivered functionality.

Jones uses function points instead of lines of code. Most of us are uncomfortable with function points, so I've converted them to lines of C code using the generally-accepted average of one FP being 130 lines of C.

Playing with the numbers, for 100% completeness you'd need about one page of requirements documentation for every ten lines of code. It's not clear if a "page" means 500 words, or perhaps just a sentence or two, with perhaps each requirement standing proudly on its own page.

What do your numbers look like?

Vlad Z sent more New Year's resolutions:

Later this year the US will chose a new president. But who will make that choice?

I last wrote about electronic voting machines in the election year of 2008. Twelve years later little has changed.

Vermin the world over will exploit any computer weakness to extort tens of thousands of dollars from the unsuspecting populace. Those are pretty puny stakes compared with the presidential election, which will consume billions of dollars of campaign funds and will shape the direction of the most powerful nation on the planet for at least four years.

So how did we react? By acquiring voting machines with many known security flaws, developed without using best engineering practices, whose code is proprietary, and that use hacker-friendly operating systems. In effect, the government, by making these purchasing decisions, sanctioned a "we're the government, you can trust us" appeal to the electorate.

Ironically, the citizenry has never trusted the government less than in recent years.

We know how to build reliable software systems. We know how to build secure software systems. We decided not to. How dumb is that?

Democracy is hard. We citizens have to be involved. We need to hold our elected officials accountable, and toss them out when they fail to meet our standards. That requires we trust the integrity of the ballot box.

E-voting travails are not new; ballot-stuffing is as old as the Republic. The difference is that now it may be possible to stuff millions of votes with a few clicks.

I believe that e-voting holds the promise of eliminating compromised elections. But not with the approaches used now. I'm sickened to read of various commissions demanding that vendors add printers to their machines. That's does nothing to prove that each vote is recorded properly. The printer could show one candidate and record another. And the voting machine itself is not the issue: it's the entire system, including the computers that hold the databases, which are even more attractive targets for bad actors.

We citizens must demand a system that works. That requires the following:

• Open source both the software and hardware. Publish the code and schematics on the web. Sure, we can hire a private company to develop and manufacture the equipment, but the IP should be owned by those footing the bill. Us.
• A dead simple design with minimal features. Prune anything not completely necessary. That means no Windows or Linux.
• Use provably-correct components. No proprietary code is allowed.
• No changes may be made to the design within six months of an election, to give the community time to evaluate the proposed change. If the design is dead simple, few changes should be needed.
• Certify the code to DO-178B level A, the avionics standard..
• Certify the code to Common Criteria EAL7, the highest security standard.

Will this happen? I doubt it. Conspiracy theorists probably feel the overlords who control the government want to have hackable elections. But I suspect there's more truth in: Never ascribe to malice, that which can be explained by incompetence.

The blue light of the TV flickered on the blank wall, but it went unnoticed by me as I slumped in my old armchair.

It had been a bad week. She left on Monday, screaming that she couldn't take all of the equipment piled everywhere. Me, I thought that old Tek 545 was a collector's item, an antique. Sure, the dust was pretty bad, but you can bet we were the only couple in town with a living room populated by old scopes and CP/M machines.

Yeah, I do mess up some times, like that time my homebrew furnace controller burped and drove the house to 115 degrees when we were gone for the weekend. Hey, I never liked those pets anyway, and the smell did eventually come out of the carpets. Pretty much. I mean, it was just a little software bug; we all have those!

And there was the fire. Yeah, next time I'll put a bigger heat sink on the power supply. I admit it - I learned a lesson. The scorch marks on her dresses don't really look all that bad.

I reached for another bag of chips as the chair groaned a bit more. One of these days I'm gonna have to work off some of the excess pounds. A decade spent in the lab drinking Jolt and munching fries had taken its toll. Despite the flab I still know calculus and can program in C; surely a dream dude for any discerning woman. I bet I could wow them at the local watering hole with my great stories about TCP/IP!

Well, this is Silicon Valley after all, where relationships, jobs and careers are measured in milliseconds and loyalty doled out by the microgram. Electronics is a dog-eat-dog business and I'm an old hand at crawling out from the wreckage. Like that last startup I worked for. I told 'em we'd get that product out the door, eventually anyway. We woulda survived if that idiot president just got another couple of mil of venture capital. For a while at least.

Ya know, maybe it was losing that job that ticked her off. I figure, what's the big deal? She should be used to this by now. Check out my resume - it shows lots of experience at lots of places. No one can beat this!

I picked up the phone but heard only the accusing silence of a non-payment disconnection. No matter. Time to find another company looking for my embedded expertise. There's a startup a minute here, pigeons ripe for picking.

I clumped out of the trailer's front door and found Big Al, the usual wild look in his eyes, mouth working hard on this morning's sugar raised, the white powder spotting his beard.� "Al, buddy, you're outa work too, huh? How's the wife and kids?"

"Kids? Kids? Yeah, come to think of it I did notice some little people living with me. I wonder where they came from? Check this out." With that he shoved a coffee-stained fragment of the San Jose Mercury News into my hands. I quickly took in the circled want ad. "ENGINEERs � microprocessor savvy designers and programmers needed. C, FPGAs, assembly a plus. Exciting opportunity for a motivated developers in a new high-growth company."

A slow smile spread across my face. Here was our pigeon; I was already mentally spending the signing bonus.

That afternoon, T-shirts cleaned and pressed, with most of the donut detritus hurriedly wiped from Al's beard, we met with the president of Galaxar Enterprises. Yep, just as usual, this man was the typical harried executive desperate for people, so desperate he had neither the time nor resources to do much of a background check. Not that my background is so terrible; it's just that there's so much of it.

"You know C? Schematic capture? What's the last project you worked on?" he mumbled, looking at his watch while the cell pinged an urgent tune.

"We did that Internet cappuccino maker for Kitchen Services; you must have read about it in the press. Yep, that puppy had a RISC-V based coffee engine with 10 gigs of RAM�"

"Didn't they go Chapter 7?", he interrupted, interested� now.

"Trust me on this. The boss was an idiot. He just didn't understand how much compute power we needed to blend the perfect cuppa joe. That sucker could crank some coffee, believe me. If they hadn't been so stuck on the cost of goods we coulda cleaned up the cappuccino market. We were practically done with development when the SEC raided us."

"OK, OK, look, when can you start? Now? Don't you guys ever shave? Hell, just sit here and Bob will tell you what to do."

Bob, engineering VP, was one of those snotty-nosed brats with a degree and an attitude. "We're building a new marine VHF radio for the recreational boating market. That means there are three main design parameters. First, the unit must be totally sealed to insure it's waterproof. Second, the sell price can't exceed $250. And obviously the unit must be simple enough that even the most casual boater can use it."

He went on to tell us how we were going to design the product. Us! Can you imagine? As if I don't understand project planning, structured design, discipline design, and all of that utter crap. Me, I prefer to skip all of that non-productive nonsense and just bang it out.

I zoned out, the drone of Bob's voice barely noticeable, nodding at the right time while planning my next move. Clearly it was time for the old end-run. Saturday night Al and I marched into the president's office. "Herb," I started, "we know you're running out of venture money and an IPO is at least a year away. Bob's planning to spend another three months just doing preliminary design. Whatdoya want, a design or a product? Trust me on this � we can pound out a design in a week, max, and then get the radio done in no time."

Herb's eyes gleamed. It seems that he, too, was frustrated by Bob's methodical approach to engineering. This valley is the land of Steve Jobs, where unbridled passion and hope fuels the dream of tomorrow's big score. Discipline? Bah. Just lemme at a problem and I'll get it done. With a bit more prodding Herb agreed that this project was so important he'd give it skunk-works treatment, get Bob off of our backs, and let us report directly to his president's office.

The week sped by like a read from cache memory. Al slouched into my cubicle, let out a long, satisfying-sounding belch, and asked "didn't we promise Herb a spec or something?" Right! Never let the boss, down, that's my motto. Unless there's a good reason, of course.

"Sure, look, just grab those header files we've been working on and edit a bit of descriptive stuff at the beginning. They'll never read it all anyway. If he complains we'll tell 'em not only is the spec done, we've incorporated it into the firmware. How can he get upset if he sees we're already coding?"

Herb swallowed our header files hook, line and sinker. He's thrilled that we're already cranking out software, and giddily reported our progress to the venture capitalists. I think they're already mentally spending their IPO profits. Bob is muttering vague threats, but he's been squeezed into the user-interface group.� He wants Al and I to take on that new college grad, Marty. We're supposed to show him how to get projects done. It's� not all bad; the kid has a car so can get us beers and carry-out.

The secret to success in this business is to look busy, keep a prototype in a state that looks like it has some level of functionality, and always agree with the boss.� And you can't act like you have a personal life when battling a schedule! Heck, after just three days on the job Marty asked if he could leave at 5 to celebrate his first anniversary. I straightened him out. "Kid, trust me on this. We all go through one or two starter marriages, you know, no kids, no property, no regrets. Don't take it too seriously. Now let's order a pizza and get back to work." It was probably a good thing that I turned off the switchboard that night, so he wouldn't get distracted by all of those frantic calls from home.

And that kid did need some attention. I caught him late one night doing a spell check on his comments! Somehow he missed the fact that a ship date loomed; comments are always the first thing to go. "Kid, trust me on this. Never include a comment that will help someone else understand your code. If they understand it, they don't need you." I think he gets the picture now.

As time moved on we started having trouble fitting the binary image into the CPU's address space. "This always happens", I reassured Herb, "them 16 bitters just can't handle the sort of code we're cranking out for you. Look, we'll just stuff a bigger part in there this afternoon. No problemo; I've done this a million times."

Big Al's eyes lit up when I suggested we look into a 64 bit processor. "I've got just the ticket. There's one I've been itching to try; it's totally reconfigurable, you can even define your own instruction set. Man, this is gonna look great on my resume!"

Ah, resume fodder, the grease of the industry. Herb didn't seem to concerned about the increased cost of goods � at least he wasn't asking any questions � so I set out trying to find some way to cool the sucker. With luck a big old heat sink and decent-sized fan might be adequate. Jeez, maybe I'll use the next size up; those burnt dresses still haunt me at times.

We optimized the instruction set on the CPU to play DoomStar III at awesome speeds. The best part of using a custom architecture was that I got to port the entire GNU toolchain to our chip. That compiler sure is tricky! First time I'd ever fiddled with a code generator, so it naturally took a bit longer than planned to get working � mostly � tools.

As the weeks passed Herb got noticeably more antsy, checking on our progress on a daily, and then hourly, basis. This always happens, and is a sign that the old cash reserve is evaporating. I started running to the bank the minute paychecks came out. No one's gonna stick me with bouncing paper! Been there, done that.

Bob � remember Bob? � strolled into the lab one afternoon to check on our progress. It seems the fool had actually invested his own money into the company! He's correspondingly annoying about what we do, even though my end-run had gotten him off the project months before. Oddly, he seemed upset about the cooling fan. "This thing has got to be totally sealed, so no water gets in!" he whined.

"Yeah, yeah, just mount it in a dry place or something", I replied. "I can't be bothered with that sort of stuff. You know how much power this sucker uses?" These company men are all stress puppies. Not me; I'll be going strong when he suffers his first mid-30s myocardial infarction.

Christmas rolled around � or was it Easter? I dunno, we were plenty busy chasing down bugs and making feature changes. Bob's paycheck bounced. I knew that Herb had been doing some fancy footwork to keep things afloat, but when everyone in accounting quit, complaining about insolvency or something, the standard exodus began. As usual, engineering remained untouched by the various rounds of layoffs. They needed the products we make to survive. I love this field!

This seemed like a great time for a two week vacation, though Marty seemed almost hysterical that I'd take off now. "Kid, trust me on this. Never complete a project on time. If you do, they will think it was easy and anyone can do it and they don't need you. Now I'm outa here for a while. Look busy and we'll sort it all out when I get back."

I got back, more or less sober but feeling great, to find the front door padlocked and a sheriff standing guard. Marty, skulking in a dark corner, grabbed my arm and moaned that he couldn't build the code at all while I was away. It seems he had trouble locating all of the source.

"Kid, trust me on this. Never archive all the sources necessary to build a binary. Always hide a few on your own disk. If they can build your binary, they don't need you. What do they teach you in college, anyway?"

He said the creditors got fed up and were demanding their money. Half the employees were suing because their paychecks bounced. A satisfied grin spread across my face as I recalled beating the rest of those idiots to the bank.

Marty shrieked that Herb was suing all of us in engineering for not meeting promised dates, specs, or features. "Kid, trust me on this. They always sue. That's why I own nothing. What do they think they'll get, my trailer? The bank owns that!"

Well, it seems my two week holiday might extend itself a bit. No worries there! After such a tough project I needed a break. It's time to sleep in for a while, build up those reserves.

Days later an awful booming interrupted my sleep. "My god, it's not even noon!" I shouted, "go away". The door slammed open and Big Al loomed over me. "Check this out." He unwrapped the newspaper from around his BLT and handed me a section from the want ads. Yep, old Al was right on top of things again. Another startup, as usual desperate for a pair of gurus like us, no doubt willing to hire at any price.

A harried president briefly interviewed the two of us, asking lots of questions about our most recent experience. We gave them the scoop on the VHF radio, but had to parry his request for references. "Sorry � they went out of business. Shame, that. There's no one there you can call. But we built a hell of a radio for those guys. It's too bad management was so screwed up they folded. Hey, it happens all the time in this industry."

"But trust me on this � you need a graybeard like me to mentor your young engineers, and to get this project out now! I'm ready to start coding today. What is it we're building?"

Bob Paddock sent in a link to an article about a new-fangled battery. The key specs:

It's SMT-compatible in an EIA 1812 package. The article doesn't list a price (a practice that is unacceptable), but Digi-Key lists these at $8.30 in small quantities.

While this is an interesting part, with only 100 µAh of capacity and the relatively high cost, I'm not sure where such a part makes sense.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Tomislav Razov sent this:

Changing random stuff until your program works is "hacky" and "bad coding practice".

But if you do it fast enough it is "Machine learning" and pays 4x your current salary.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• New Year's Resolutions for Firmware Developers
• Principles for Interfacing With Humans
• Just How Big is a Million Lines of Code?
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Engineering changes at an accelerating pace. The boss wants more bug-free code, today. Tomorrow he'll want even more in the same time-frame. Are you on top of the best ways to deliver it?

Some tell me they just don't have time to attend seminars. I'm reminded of this:

It is important to fight today's engineering battle. But the war will be lost if you don't find better ways to win.

This is what my one-day Better Firmware Faster seminar is all about: giving your team the tools they need to operate at a measurably world-class level, producing code with far fewer bugs in less time. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends January 20. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Jack's latest blog: How I Write Code.

A recent article in Electronic Design spotlights crashing DRAM and NAND prices. The latter are off 79% from their mid-2017 numbers. But a stunning graph later in the article shows that 1 GB of DRAM cost around $70k in 1991! It's around $3 today. I don't know how accurate that $70k number is, but even if it's off by an order of magnitude that shows astonishing progress in electronics over the last three decades.

Coding is "90% finished" for half the total coding time. Fred Brooks

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

John Carter likes the D language, and he also included a useful link to Stroustup's and Sutter's C++ guidelines:

Interestingly, D does not support 8 or 16 bit processors. From the overview page: "No consideration is given in D for mixed near/far pointers and all the machinations necessary to generate good 16 bit code. The D language design assumes at least a 32 bit flat memory space and supports 64 bit as well." I assume that means no MMU or MPU, which is a shame, as these resources can be useful in preventing complete system crashes.

This month's giveaway is a Cypress CY8KIT-044 PSoC 4 M-series Pioneer kit.

Enter via this link.

Percepio is giving away one-year licenses for their TraceAlyzer debugging tool. I've played with it and find it a great way to visualize how your code interacts with the RTOS (many RTOSes supported). The giveaway is not exactly free - you have to send a description of what you'll do with it, and follow-up with a story about solving the problem. More info here. I image the story could be pretty short.

Not a freebie for Muse readers, but here's an astonishing giveaway: a business card that runs Linux. Production cost: $3 in low volumes. Linux for three bucks? For all the sturm and drang in the political news, sometimes we need to take a step back and just be amazed at how cool things are. Thanks for Martin Buchanan for sending the link.

Happy New Year!

I thought it would be fun to Google for New Year's resolutions for software people. Most of the results were things like "share more," "work with the community," "contribute to open source," "cut back to one quart of whiskey per day," and the like. Nice thoughts, but what technical resolutions are worthwhile?

First, I don't like aspirations like "work harder at..." That's impossible to measure. Without numbers, we're lost in a fog. Without numbers, we can't understand if we're improving. Without numbers, we don't know how we measure up against others (e.g., am I in the bottom quartile of developers?). And, as Harry Roberts demonstrates in Quality is Personal, "trying harder" rarely works.

So here are some thoughts:

• Aim to elicit x% of a project's final requirements. (Requirements churn is pretty easy to measure, either with a spreadsheet or a tool like Jira.)
• Solicit criticism. Have a trusted mentor review x of your best modules.
• Achieve x% test coverage (more about this coming next issue of the Muse).
• Eliminate x% of defects pre-test. (Testing will never be adequate by itself. Most test schemes only exercise half the code. Tons of data shows how test is just not enough, or not even possible.) Our goal should be to write correct code, not to beat poor code into submission.
• x% of my new code will have a cyclomatic complexity under y. (In general, complexities over 10 to 15 mean trouble. Plenty of tools will measure this).
• Find x hours per day for uninterrupted work on one project. (Multitasking on three projects consumes 40% in context switching time. Each interruption costs 15 minutes in overhead).
• Achieve an x% comment density. (Plenty of tools measure this. Best-in-class code has 60 to 70% comment density. That is the percentage of non-blank lines that are, or contain, comments. Plenty of data supports this, such as this. Many argue that comments aren't needed, but after reviewing oceans of code I disagree).
• Reduce shipped defects to under x% (measured as a percentage of all defects found in engineering up to 90 days after shipping). Only 0.4% of companies remove more than 99% of all defects pre-shipping.
• Read x technical books per year. (The average developer reads one. For book recommendations see this. I take notes about the most important points when reading tech content.)

John Carter sent an excerpt from Standard Practice for Human Engineering Design for Marine Systems, Equipment and Facilities. These notions should be embraced by all engineers. Humans are flawed, which this standard addresses explicitly.

But first, a story. I read (somewhere recently, maybe in the Smithsonian's Air and Space magazine) that during WWII B-17s crashed on landing far more often than one would expect. Turns out, the gear and flaps levers were located near each other and looked similar. Tired, stressed pilots would mix up the controls. Relocating and changing the appearance eliminated those sorts of accidents. The moral: design with real people in mind.

Here are the practices:

4.2 Principles of Human Behavior:

4.2.1 There are basic principles of human behavior that control or influence how each person performs in their workplace. Some of these behaviors are culturally derived, while others are general and uniform across all cultures and geographical regions of the world. These behaviors influence a person's physical, social, and psychological approach toward the work they do and how safely they do that work. Failure to satisfy these behavioral principles in the design of a ship or maritime structure can encourage, or even coerce, maritime personnel into taking unsafe risks in their everyday activities. It is, therefore, imperative that designers of ships and maritime equipment, systems, and facilities know these principles to provide a safe and efficient workplace for maritime personnel.

4.2.2 These principles include:

4.2.2.1 If the design of the ship or maritime facility is considered to be unsafe or inefficient by the crew, it will be modified by the users, often solving the initial problem but introducing others that may be as bad, or worse, than the original.

4.2.2.2 Equipment design shall be such that it encourages safe use, that is, does not provide hardware and software that can be used in an unsafe manner.

4.2.2.3 If the equipment or system is not designed to operate as the users' cultural and stereotypical expectations lead them to think that it will operate, the chance for human error is significantly increased.

4.2.2.4 If equipment or systems are perceived by operators/maintainers to be too complex or require more effort to operate or maintain than they believe is necessary, they will always look for a "shortcut." Further, this "shortcut" may be perceived as being safe when it is not.

4.2.2.5 No amount of training, company or organizational policy, threats of retaliatory action, warning notes in a technical manual or training guide, or pleading with personnel to be safe on the job can overcome poor design that encourages, leads, or even coerces personnel into unsafe acts on the job. The most efficient way to prevent unsafe design from contributing to an accident is to eliminate the unsafe design.

4.2.2.6 Equipment users tend to be very unimaginative when it comes to identifying unsafe features and they do not visualize the consequences of unsafe acts. Therefore, do not expect that an "obviously dangerous" task will always be recognized as such by every user.

4.2.2.7 Designers shall consider the possibility for human error and design equipment so that incorrect use (deliberate or accidental) will result in little or no harm to the user.

4.2.2.8 Equipment operators and maintainers tend to make guesses as to what a label, instruction, or operational chart states if it is not complete, legible, readable, and positioned correctly.

4.2.2.9 Designers and engineers shall never use themselves as the standard against which a particular design is evaluated. People come in many shapes, sizes, mental capacities, and capabilities. Therefore, design for the full range of potential users, physically, mentally, and socially.

4.2.2.10 People shall be protected against themselves. Designers cannot create an unsafe piece of equipment or system and expect the users to assume full responsibility for its safe use.

4.2.2.11 Ease of equipment maintenance affects the equipment's reliability, that is, the harder it is to be maintained, the less it will be maintained.

4.2.2.12 Equipment designed to require multiple operators working together simultaneously increases the likelihood of operator errors.

4.2.2.13 Operational/maintenance procedures shall be clear, definitive, and comprehensive, otherwise, they will be misinterpreted or ignored.

4.2.2.14 Structural items such as piping, cable trays, or any other item that appears strong enough to be used by a person to hold onto or stand on, and is placed in a convenient location to use for that purpose, will eventually be used for that purpose.

4.2.2.15 Humans expect consistency in the design and arrangement of their workplace. Therefore, if that workplace, or any part thereof, appears in more than one place in their work environment, it is expected to be located and look the same way at every location.

4.2.2.16 When controls and displays associated with particular pieces of equipment are placed on a console or control panel, they shall be located on that console or panel to replicate the actual location of the equipment on the ship or structure as both are viewed by the operator. Therefore, equipment that is to the operator's left as he/she faces the control station shall appear on the left of the control panel or console, and equipment to the right shall appear on the right side of the console or panel. This "spatial relationship" between the real world and the controls and displays that are associated with the equipments and systems of that world is extremely important in the design of ships and maritime structures.

4.2.3 Humans develop behavioral patterns based on their cultural experiences. Designing a ship or structure that ignores or violate those culturally derived behavior patterns will inevitably lead to human error.

A million lines of code. It's a number bandied about more than ever as software sizes develop overactive pituitaries. Some cell phones use upwards of twenty million. Last year Ford announced that the F150 truck has 150 million. Everett Dirksen once may have said: "A billion here, a billion there, pretty soon you're talking real money." Well, a million lines of code here, a million there, pretty soon you're talking about a program that is as mind boggling and incomprehensible as our national debt.

A million lines of code printed out would be 18,000 pages. That's a stack six feet tall (on typical 20 pound paper). Ironically, the listing weighs in at 180 pounds while the actual operating code is mass-free; it'll live in a fraction of a gram of silicon. Like DNA, code's human-readable description requires tremendously more mass than its actual instantiation.

A million lines of code is probably on the order of 20 million instructions, or around half a billion bits. That's not far off of the 3 billions base pairs in human DNA. Unlike DNA, which has redundancies and so-called "junk" sequences, every single bit in the code must be perfect. A single error causes greater or lesser failure.

Since a typical atom is around 0.3 nm in diameter, if one had as many atoms lined up as the number of instructions needed for a million lines of code, they would stretch 10 cm. That many Ebola viruses would stretch 15 meters.

A million lines of code is as long as 14 copies of War And Peace, 25 of Ulysses, 63 copies of The Catcher in the Rye, or 66 copies of K&R's C Programming Language.

A million lines of code is not ten times more than 100,000. It's well-known that schedules grow exponentially faster than the code. Barry Boehm estimates the exponent is around 1.35 for embedded software. So the schedule for developing a million lines of code is 22 times bigger than for 100,000 LOC.

In the March, 1996 issue of IEEE Computer Watts Humphrey published crude rules of thumb for estimating software projects. Though hardly scientific, they do give a sense of scale. Using his estimates:

• A million lines of code requires 40,000 pages of external documentation.
• A million lines of code will typically have 100,000 bugs pre-test. Best-in-class organizations will ship with around 1k bugs still lurking. The rest of us will do worse by an order of magnitude.
• A million lines of code will occupy 67 people (including testers, tech writers, developers, etc) for 40 months, or 223 person-years. Darwin needed just 1.5 person-years to write The Origin of the Species. Scale that to the 26 copies equal in length of a million lines of code, and it appears writing code is some 6 times more time-consuming than writing a revolutionary scientific tome.

A million lines of code costs $20m to $40m. That's the cost of an F-18 jet, a thousand cars (in America), �ten million gallons of gas, seven times the inflation-adjusted cost of the ENIAC, and something like ten million times the cost of the flash chips it lives in.

Think about that last analogy: Ten million times the cost of the flash chips. Accounting screams over each added penny in recurring costs, while chanting the dual mantras "software is free," and "hey, it's only a software change."

Hex-Five Security's MultiZone product is a software layer that uses hardware to enforce high security when using untrusted code. Currently RISC-V versions are available.

It's a small (under 2 KB) bit of code that is formally verifiable. That's a big selling point when pursuing a Common Criteria security standard. As I understand it, the product enforces separation zones to keep bad actors out of others' sandboxes, which sounds like the Common Criteria's separation kernels.

A datasheet is here. It's free for non-commercial use.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of pi change. - FORTRAN manual for Xerox Computers

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• On Requirements Documents
• More New Years Resolutions
• On E-Voting
• The Secret of My Success
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Engineering changes at an accelerating pace. The boss wants more bug-free code, today. Tomorrow he'll want even more in the same time-frame. Are you on top of the best ways to deliver it?

Some tell me they just don't have time to attend seminars. I'm reminded of this:

It is important to fight today's engineering battle. But the war will be lost if you don't find better ways to win.

This is what my one-day Better Firmware Faster seminar is all about: giving your team the tools they need to operate at a measurably world-class level, producing code with far fewer bugs in less time. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends today. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Jack's latest blog: A Lack of Forethought

I never give anyone hell. I only tell the truth. They think it's hell. Harry S. Truman

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

This month's giveaway is a Cypress CY8KIT-044 PSoC 4 M-series Pioneer kit.

Enter via this link.

Brian Cuthie reports that the schematic editor/simulation software Micro-Cap is now free. See this.

How long are your requirements documents? How complete are they?

Getting hard data is notoriously difficult in the software world. Capers Jones probably has more than anyone. He stresses the large standard deviation, but makes it clear that one can gain some insight from it. I view the data as an impressionist painting, where the outlines might be fuzzy, but one can still make out the general shape of things.

The following table is derived from data in Jones' The Economics of Software Quality. It shows how many pages of requirements are found for various-sized projects. The "Requirements Completeness" column illustrates all-to-pathetically how short those pages are of the delivered functionality.

Jones uses function points instead of lines of code. Most of us are uncomfortable with function points, so I've converted them to lines of C code using the generally-accepted average of one FP being 130 lines of C.

Playing with the numbers, for 100% completeness you'd need about one page of requirements documentation for every ten lines of code. It's not clear if a "page" means 500 words, or perhaps just a sentence or two, with perhaps each requirement standing proudly on its own page.

What do your numbers look like?

Vlad Z sent more New Year's resolutions:

Later this year the US will chose a new president. But who will make that choice?

I last wrote about electronic voting machines in the election year of 2008. Twelve years later little has changed.

Vermin the world over will exploit any computer weakness to extort tens of thousands of dollars from the unsuspecting populace. Those are pretty puny stakes compared with the presidential election, which will consume billions of dollars of campaign funds and will shape the direction of the most powerful nation on the planet for at least four years.

So how did we react? By acquiring voting machines with many known security flaws, developed without using best engineering practices, whose code is proprietary, and that use hacker-friendly operating systems. In effect, the government, by making these purchasing decisions, sanctioned a "we're the government, you can trust us" appeal to the electorate.

Ironically, the citizenry has never trusted the government less than in recent years.

We know how to build reliable software systems. We know how to build secure software systems. We decided not to. How dumb is that?

Democracy is hard. We citizens have to be involved. We need to hold our elected officials accountable, and toss them out when they fail to meet our standards. That requires we trust the integrity of the ballot box.

E-voting travails are not new; ballot-stuffing is as old as the Republic. The difference is that now it may be possible to stuff millions of votes with a few clicks.

I believe that e-voting holds the promise of eliminating compromised elections. But not with the approaches used now. I'm sickened to read of various commissions demanding that vendors add printers to their machines. That's does nothing to prove that each vote is recorded properly. The printer could show one candidate and record another. And the voting machine itself is not the issue: it's the entire system, including the computers that hold the databases, which are even more attractive targets for bad actors.

We citizens must demand a system that works. That requires the following:

• Open source both the software and hardware. Publish the code and schematics on the web. Sure, we can hire a private company to develop and manufacture the equipment, but the IP should be owned by those footing the bill. Us.
• A dead simple design with minimal features. Prune anything not completely necessary. That means no Windows or Linux.
• Use provably-correct components. No proprietary code is allowed.
• No changes may be made to the design within six months of an election, to give the community time to evaluate the proposed change. If the design is dead simple, few changes should be needed.
• Certify the code to DO-178B level A, the avionics standard..
• Certify the code to Common Criteria EAL7, the highest security standard.

Will this happen? I doubt it. Conspiracy theorists probably feel the overlords who control the government want to have hackable elections. But I suspect there's more truth in: Never ascribe to malice, that which can be explained by incompetence.

The blue light of the TV flickered on the blank wall, but it went unnoticed by me as I slumped in my old armchair.

It had been a bad week. She left on Monday, screaming that she couldn't take all of the equipment piled everywhere. Me, I thought that old Tek 545 was a collector's item, an antique. Sure, the dust was pretty bad, but you can bet we were the only couple in town with a living room populated by old scopes and CP/M machines.

Yeah, I do mess up some times, like that time my homebrew furnace controller burped and drove the house to 115 degrees when we were gone for the weekend. Hey, I never liked those pets anyway, and the smell did eventually come out of the carpets. Pretty much. I mean, it was just a little software bug; we all have those!

And there was the fire. Yeah, next time I'll put a bigger heat sink on the power supply. I admit it - I learned a lesson. The scorch marks on her dresses don't really look all that bad.

I reached for another bag of chips as the chair groaned a bit more. One of these days I'm gonna have to work off some of the excess pounds. A decade spent in the lab drinking Jolt and munching fries had taken its toll. Despite the flab I still know calculus and can program in C; surely a dream dude for any discerning woman. I bet I could wow them at the local watering hole with my great stories about TCP/IP!

Well, this is Silicon Valley after all, where relationships, jobs and careers are measured in milliseconds and loyalty doled out by the microgram. Electronics is a dog-eat-dog business and I'm an old hand at crawling out from the wreckage. Like that last startup I worked for. I told 'em we'd get that product out the door, eventually anyway. We woulda survived if that idiot president just got another couple of mil of venture capital. For a while at least.

Ya know, maybe it was losing that job that ticked her off. I figure, what's the big deal? She should be used to this by now. Check out my resume - it shows lots of experience at lots of places. No one can beat this!

I picked up the phone but heard only the accusing silence of a non-payment disconnection. No matter. Time to find another company looking for my embedded expertise. There's a startup a minute here, pigeons ripe for picking.

I clumped out of the trailer's front door and found Big Al, the usual wild look in his eyes, mouth working hard on this morning's sugar raised, the white powder spotting his beard.� "Al, buddy, you're outa work too, huh? How's the wife and kids?"

"Kids? Kids? Yeah, come to think of it I did notice some little people living with me. I wonder where they came from? Check this out." With that he shoved a coffee-stained fragment of the San Jose Mercury News into my hands. I quickly took in the circled want ad. "ENGINEERs � microprocessor savvy designers and programmers needed. C, FPGAs, assembly a plus. Exciting opportunity for a motivated developers in a new high-growth company."

A slow smile spread across my face. Here was our pigeon; I was already mentally spending the signing bonus.

That afternoon, T-shirts cleaned and pressed, with most of the donut detritus hurriedly wiped from Al's beard, we met with the president of Galaxar Enterprises. Yep, just as usual, this man was the typical harried executive desperate for people, so desperate he had neither the time nor resources to do much of a background check. Not that my background is so terrible; it's just that there's so much of it.

"You know C? Schematic capture? What's the last project you worked on?" he mumbled, looking at his watch while the cell pinged an urgent tune.

"We did that Internet cappuccino maker for Kitchen Services; you must have read about it in the press. Yep, that puppy had a RISC-V based coffee engine with 10 gigs of RAM�"

"Didn't they go Chapter 7?", he interrupted, interested� now.

"Trust me on this. The boss was an idiot. He just didn't understand how much compute power we needed to blend the perfect cuppa joe. That sucker could crank some coffee, believe me. If they hadn't been so stuck on the cost of goods we coulda cleaned up the cappuccino market. We were practically done with development when the SEC raided us."

"OK, OK, look, when can you start? Now? Don't you guys ever shave? Hell, just sit here and Bob will tell you what to do."

Bob, engineering VP, was one of those snotty-nosed brats with a degree and an attitude. "We're building a new marine VHF radio for the recreational boating market. That means there are three main design parameters. First, the unit must be totally sealed to insure it's waterproof. Second, the sell price can't exceed $250. And obviously the unit must be simple enough that even the most casual boater can use it."

He went on to tell us how we were going to design the product. Us! Can you imagine? As if I don't understand project planning, structured design, discipline design, and all of that utter crap. Me, I prefer to skip all of that non-productive nonsense and just bang it out.

I zoned out, the drone of Bob's voice barely noticeable, nodding at the right time while planning my next move. Clearly it was time for the old end-run. Saturday night Al and I marched into the president's office. "Herb," I started, "we know you're running out of venture money and an IPO is at least a year away. Bob's planning to spend another three months just doing preliminary design. Whatdoya want, a design or a product? Trust me on this � we can pound out a design in a week, max, and then get the radio done in no time."

Herb's eyes gleamed. It seems that he, too, was frustrated by Bob's methodical approach to engineering. This valley is the land of Steve Jobs, where unbridled passion and hope fuels the dream of tomorrow's big score. Discipline? Bah. Just lemme at a problem and I'll get it done. With a bit more prodding Herb agreed that this project was so important he'd give it skunk-works treatment, get Bob off of our backs, and let us report directly to his president's office.

The week sped by like a read from cache memory. Al slouched into my cubicle, let out a long, satisfying-sounding belch, and asked "didn't we promise Herb a spec or something?" Right! Never let the boss, down, that's my motto. Unless there's a good reason, of course.

"Sure, look, just grab those header files we've been working on and edit a bit of descriptive stuff at the beginning. They'll never read it all anyway. If he complains we'll tell 'em not only is the spec done, we've incorporated it into the firmware. How can he get upset if he sees we're already coding?"

Herb swallowed our header files hook, line and sinker. He's thrilled that we're already cranking out software, and giddily reported our progress to the venture capitalists. I think they're already mentally spending their IPO profits. Bob is muttering vague threats, but he's been squeezed into the user-interface group.� He wants Al and I to take on that new college grad, Marty. We're supposed to show him how to get projects done. It's� not all bad; the kid has a car so can get us beers and carry-out.

The secret to success in this business is to look busy, keep a prototype in a state that looks like it has some level of functionality, and always agree with the boss.� And you can't act like you have a personal life when battling a schedule! Heck, after just three days on the job Marty asked if he could leave at 5 to celebrate his first anniversary. I straightened him out. "Kid, trust me on this. We all go through one or two starter marriages, you know, no kids, no property, no regrets. Don't take it too seriously. Now let's order a pizza and get back to work." It was probably a good thing that I turned off the switchboard that night, so he wouldn't get distracted by all of those frantic calls from home.

And that kid did need some attention. I caught him late one night doing a spell check on his comments! Somehow he missed the fact that a ship date loomed; comments are always the first thing to go. "Kid, trust me on this. Never include a comment that will help someone else understand your code. If they understand it, they don't need you." I think he gets the picture now.

As time moved on we started having trouble fitting the binary image into the CPU's address space. "This always happens", I reassured Herb, "them 16 bitters just can't handle the sort of code we're cranking out for you. Look, we'll just stuff a bigger part in there this afternoon. No problemo; I've done this a million times."

Big Al's eyes lit up when I suggested we look into a 64 bit processor. "I've got just the ticket. There's one I've been itching to try; it's totally reconfigurable, you can even define your own instruction set. Man, this is gonna look great on my resume!"

Ah, resume fodder, the grease of the industry. Herb didn't seem to concerned about the increased cost of goods � at least he wasn't asking any questions � so I set out trying to find some way to cool the sucker. With luck a big old heat sink and decent-sized fan might be adequate. Jeez, maybe I'll use the next size up; those burnt dresses still haunt me at times.

We optimized the instruction set on the CPU to play DoomStar III at awesome speeds. The best part of using a custom architecture was that I got to port the entire GNU toolchain to our chip. That compiler sure is tricky! First time I'd ever fiddled with a code generator, so it naturally took a bit longer than planned to get working � mostly � tools.

As the weeks passed Herb got noticeably more antsy, checking on our progress on a daily, and then hourly, basis. This always happens, and is a sign that the old cash reserve is evaporating. I started running to the bank the minute paychecks came out. No one's gonna stick me with bouncing paper! Been there, done that.

Bob � remember Bob? � strolled into the lab one afternoon to check on our progress. It seems the fool had actually invested his own money into the company! He's correspondingly annoying about what we do, even though my end-run had gotten him off the project months before. Oddly, he seemed upset about the cooling fan. "This thing has got to be totally sealed, so no water gets in!" he whined.

"Yeah, yeah, just mount it in a dry place or something", I replied. "I can't be bothered with that sort of stuff. You know how much power this sucker uses?" These company men are all stress puppies. Not me; I'll be going strong when he suffers his first mid-30s myocardial infarction.

Christmas rolled around � or was it Easter? I dunno, we were plenty busy chasing down bugs and making feature changes. Bob's paycheck bounced. I knew that Herb had been doing some fancy footwork to keep things afloat, but when everyone in accounting quit, complaining about insolvency or something, the standard exodus began. As usual, engineering remained untouched by the various rounds of layoffs. They needed the products we make to survive. I love this field!

This seemed like a great time for a two week vacation, though Marty seemed almost hysterical that I'd take off now. "Kid, trust me on this. Never complete a project on time. If you do, they will think it was easy and anyone can do it and they don't need you. Now I'm outa here for a while. Look busy and we'll sort it all out when I get back."

I got back, more or less sober but feeling great, to find the front door padlocked and a sheriff standing guard. Marty, skulking in a dark corner, grabbed my arm and moaned that he couldn't build the code at all while I was away. It seems he had trouble locating all of the source.

"Kid, trust me on this. Never archive all the sources necessary to build a binary. Always hide a few on your own disk. If they can build your binary, they don't need you. What do they teach you in college, anyway?"

He said the creditors got fed up and were demanding their money. Half the employees were suing because their paychecks bounced. A satisfied grin spread across my face as I recalled beating the rest of those idiots to the bank.

Marty shrieked that Herb was suing all of us in engineering for not meeting promised dates, specs, or features. "Kid, trust me on this. They always sue. That's why I own nothing. What do they think they'll get, my trailer? The bank owns that!"

Well, it seems my two week holiday might extend itself a bit. No worries there! After such a tough project I needed a break. It's time to sleep in for a while, build up those reserves.

Days later an awful booming interrupted my sleep. "My god, it's not even noon!" I shouted, "go away". The door slammed open and Big Al loomed over me. "Check this out." He unwrapped the newspaper from around his BLT and handed me a section from the want ads. Yep, old Al was right on top of things again. Another startup, as usual desperate for a pair of gurus like us, no doubt willing to hire at any price.

A harried president briefly interviewed the two of us, asking lots of questions about our most recent experience. We gave them the scoop on the VHF radio, but had to parry his request for references. "Sorry � they went out of business. Shame, that. There's no one there you can call. But we built a hell of a radio for those guys. It's too bad management was so screwed up they folded. Hey, it happens all the time in this industry."

"But trust me on this � you need a graybeard like me to mentor your young engineers, and to get this project out now! I'm ready to start coding today. What is it we're building?"

Bob Paddock sent in a link to an article about a new-fangled battery. The key specs:

It's SMT-compatible in an EIA 1812 package. The article doesn't list a price (a practice that is unacceptable), but Digi-Key lists these at $8.30 in small quantities.

While this is an interesting part, with only 100 µAh of capacity and the relatively high cost, I'm not sure where such a part makes sense.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Tomislav Razov sent this:

Changing random stuff until your program works is "hacky" and "bad coding practice".

But if you do it fast enough it is "Machine learning" and pays 4x your current salary.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• The Futility of Testing
• Review of SEGGER's J-Trace and Ozone
• Final Thoughts on New Year's Resolutions
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

I'll present the Better Firmware Faster seminar in Melbourne, Australia February 20. All are invited. More info here. Please note that the seminar is nearly full. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Software specifications and documentation are often an afterthought, and the development of requirements after the design is created is sometimes even touted as a benefit or necessity. This simply isn't true for safety-critical software where the loss might involve not only human life but physical property, critical mission loss, and damage to the environment. It should be considered malpractice for software to be created before the safety requirements are identified. - Nancy Leveson

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

The next article is a review of a debugger that includes code coverage, which insures every statement and even every instruction in a program gets tested. However test is simply not adequate to insure an application is correct. Test does nothing to evaluate the correctness and completeness of the requirements, for example. But absent coverage, it's impossible to know if all of the code has been evaluated. A big project may have so many possible paths that insuring 100% coverage could be a huge effort.

In The Software Quality Challenge by the brilliant and sadly-deceased Watts Humphrey, published in the also sadly-deceased CrossTalk magazine (June 2008), Humphrey shows that the number of paths (and therefore required tests) a program can take depends on the quantity of decisions contained. Here's his results:

His analysis is based on an extreme assumption that every branch is likely to affect every other. However, the numbers are stunning and thought-provoking. One takeaway is that it's incredibly important to decouple modules as much as possible so Humprey's assumption (every branch is likely to affect every other) doesn't haunt our efforts.

He goes on to graph pre-test error rates for 810 experienced developers:

(Note that test finds only about half of all defects, a result consistent with many other studies).

I generally don't do traditional reviews of software, as most software tools have a vast number of features. However, I do make general observations about these tools, which is what follows for SEGGER's J-Trace Pro and their "Ozone" debugger. First, full disclosure: SEGGER places paid ads in the Muse and on my web site. The J-Trace Pro I evaluated is for Cortex-M/A/R, processors.

The motivation for these observations started at last summer's Embedded Systems Show in California. I was walking past SEGGER's booth and saw a demo of their coverage feature (which shows the lines of code that were executed). While I think measuring coverage is important, it's also problematic as most tools instrument the code. For line-by-line coverage that can add a lot of overhead, undesirable in a real-time system. I inquired and was told SEGGER's version adds no overhead. It's measured by watching the real-time trace data supplied by the Arm processor. That's a bit of a game changer.

Part of the problem with usual test regimens is that they are usually incomplete; one just doesn't know if the entire program was exercised. With coverage that ambiguity goes away. While coverage alone doesn't prove the code is correct, it does show that everything was at least executed. That is mandated by the most demanding standards, like the highest category of avionics testing. (The recent 737-MAX woes also demonstrate that test alone is inadequate).

SEGGER's J-Trace is a hardware debugging probe. While it works with IDEs from many other vendors, for my testing I used SEGGER's Ozone debugger program. The package costs $2498 and is currently available. A Cortex-M only (no -A or -R processors) is $1748.

Installation of Ozone and starting the demo on the supplied Cortex-M4 demo board took only a few minutes and was brain-dead easy. The demo program is a simple two-task LED blinker using the company's embOS RTOS. Ozone appears to have SEGGER's SystemView monitor (I reviewed it in TEM299) program integrated.

There are so many useful windows in Ozone that you'll want a big monitor. Mine is 24" and I wouldn't want anything less.

The application appears deceptively simple when first opened. File, View, Find, Debug, Tools, Window and Help are the only pull-down menu items; with the exception of the View menu, each of those pull-downs has only a few entries. It's mostly controlled via a vast number of right-click context-sensitive items. Ozone is very flexible in letting you add buttons to the toolbar, and I suspect that after a few minutes using it you'll add your favorite actions to reduce two clicks to one.

Here are my observations:

• At a breakpoint the Instruction Trace shows, unsurprisingly, trace data up to the breakpoint. As in every debugger, other windows show the current registers, disassembly and the like. If you click on an executed instruction in the trace window the disassembly and register windows show their values for that time - you can go back in time to see those values.
• The trace window can show the elapsed time for each instruction in cycles, instruction count, or absolute time, with a resolution of 1 ns.
• The call graph shows stack usage. In some cases that can't be determined in which case a "+" is appended to the size. In that case, in the depth column the reason for the uncertainty is given: "R" meaning recursion and "FP" denotes function pointer:
• 
• A "Code Profile" window gives execution statistics about the program's functions, either at the source or instruction level. The window shows code coverage on a source and instruction level, and the number of times each was invoked. In some cases the "fetch" and "run" counts differ. For example, a conditional statement may only be partially executed:
• 
• Similarly, the source code window shows how often each line of code has been executed. The numbers in green on the left of this image are those counts, which are updated in real time:
• 
• Ozone can monitor "expressions": C-like constructs that it samples from time to time (down to 1 us intervals). It appears to me that these don't steal execution time from the running program:
• 
• Like most debuggers, hovering over a variable will display that items value. But Ozone does this in real time, without slowing the code. And it will also show the value of expressions.
• Need to watch variables change? The Data Sampling window will monitor selected variables and expressions. This image is better than 1000 words. In this case it's sampling the two variables at 10 KHz. The max update rate is dependent on several parameters, such as the number of expressions being monitored:
• 
• An alternative way to watch expressions is the use the Timeline window. This image shows it monitoring two incrementing variables in the upper two lines, and the call stack at the bottom.
• 
• This demo program is in loops nearly all the time so little calling is being done. However, zooming in shows the call graph. And remember, this is in real-time:
• 
• Like a few other debuggers, Ozone will profile power consumption. That's displayed in the Timeline window. The following image, taken from the user manual, shows a series of LEDs being turned on as a function of the value in a variable (top red trace) and the resulting power consumed, in mA, in the bottom red trace.
• 
• RTOS-awareness features are common today, and Ozone likewise peers deeply into the system's multitasking. Many options are supported, from looking at task structures, task execution, stacks, mailboxes, queues and more. Of course, embOS, SEGGER's own OS, is supported as are those from other vendors. But interestingly, the debugger will support any RTOS. A Javascript plugin, documented in the manual, lets the developer link Ozone to even proprietary operating systems.

There's much more in the Ozone/J-Trace tool. The above comes from reading the first 135 pages of the 312 page manual; I simply ran out of time.

The manual is pretty complete but I felt it's hard to use; it needs more examples of using the features. And there are a lot of forward references which can make understanding a feature a bit difficult. The manual is arranged in alphabetical order of the windows which means some complex ideas are introduced before the basics.

Ozone is extremely responsive, and I encountered no notable delays. Click on this short video to see how quickly things are updated as I slew the cursor around:

The application is pretty intuitive, though there are so many features you'll want to curl up with the manual. Did I say feature-rich? I can't think of anything extra we'd want in a debugger other than the mythical "find bug" and "fix bug" buttons.

A very important feature is that acquired data can be exported to a file. That's crucial in many safety-critical projects where regulatory agencies require proof about testing completeness. And it's valuable to QA departments as well. Examples of exportable data include code coverage, count of statements executed, register, variable and even expression results.

In the 1980s and 90s I ran a company that made In-Circuit Emulators. These were debugging tools that offered tons of features that all ran in real-time. The ICE business imploded early this century for a number of reasons, but the death knell was that they just couldn't keep up with emerging fast clock rates. The embedded community then regressed to simpler debugging probes which were inexpensive, but feature-poor. Ironically, our applications were growing in complexity yet our tools fell behind. The J-Trace has gone beyond what ICEs could do. I'm very impressed.

You can find an overview of the J-Trace Pro and a tutorial about using it here.

Wouter van Ooijen has thoughts about C++"

Finally, Capers Jones published Twenty Five Software Engineering Targets from 2015 Through 2019 five years ago. These aspirational goals might make a basis for New Year's resolutions. I disagree with some of these, and many, while as desirable as a resolution to achieve world peace, are about as hard to attain. (Jones uses function points, each of which is about 130 lines of C.) His yearning to reach measurable goals is praiseworthy:

1. Raise defect removal efficiency (DRE) from < 90.0% to > 99.5%.
2. Lower software defect potentials from > 4.0 per function point to < 2.0 per
   function point.
3. Lower cost of quality (COQ) from > 45.0% of development to < 15.0% of
   development.
4. Reduce average cyclomatic complexity from > 25.0 to < 10.0.
5. Raise test coverage from < 75.0% to > 98.5% for risks, paths, and requirements.
6. Eliminate error-prone modules (EPM) in large systems.
7. Eliminate security flaws in all software applications.
8. Reduce the odds of cyber attacks from > 10.0% to < 0.1%.
   
9. Reduce bad-fix injections from > 7.0% to < 1.0%.
10. Reduce requirements creep from > 1.5% per calendar month to < 0.25% per
    calendar month.
11. Lower the risk of project failure or cancellation on large 10,000 function point
    projects from > 35.0% to < 5.0%.
12. Reduce the odds of schedule delays from > 50.0% to < 5.0%.
13. Reduce the odds of cost overruns from > 40.0% to < 3.0%.
14. Reduce the odds of litigation on outsource contracts from > 5.0% to < 1.0%.
15. Lower maintenance and warranty repair costs by > 75.0% compared to 2015
    values.
16. Improve the volume of certified reusable materials from < 15.0% to > 75.0%.
17. Improve average development productivity from < 8.0 function points per month
    to >16.0 function points per month.
18. Improve work hours per function point from > 16.5 to < 8.25.
19. Improve maximum productivity to > 100 function points per staff month for 1000
    function points.
20. Shorten average software development schedules by > 35.0% compared to 2015
    averages.
21. Raise maintenance assignment scopes from < 1,500 function points to > 5,000
    function points.
22. Replace today's static and rigid requirements, architecture, and design methods
    with a suite of animated design tools combined with pattern matching.
23. Develop an interactive learning tool for software engineering based on massively
    interactive game technology.
24. Develop a suite of dynamic, animated project planning and estimating tools that
    will show growth of software applications.
25. Introduce licensing and board certification for software engineers and specialists.
    

The full paper with his reasoning behind each goal can be found here (registration required).

Not quite yet a product, but for ultra-low-power products, why not use the triboelectric current from falling snow?

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

At the source of every error which is blamed on the computer you will find at least two human errors, including the error of blaming it on the computer.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• The Futility of Testing
• Review of SEGGER's J-Trace and Ozone
• Final Thoughts on New Year's Resolutions
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

I'll present the Better Firmware Faster seminar in Melbourne, Australia February 20. All are invited. More info here. Please note that the seminar is nearly full. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Software specifications and documentation are often an afterthought, and the development of requirements after the design is created is sometimes even touted as a benefit or necessity. This simply isn't true for safety-critical software where the loss might involve not only human life but physical property, critical mission loss, and damage to the environment. It should be considered malpractice for software to be created before the safety requirements are identified. - Nancy Leveson

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

The next article is a review of a debugger that includes code coverage, which insures every statement and even every instruction in a program gets tested. However test is simply not adequate to insure an application is correct. Test does nothing to evaluate the correctness and completeness of the requirements, for example. But absent coverage, it's impossible to know if all of the code has been evaluated. A big project may have so many possible paths that insuring 100% coverage could be a huge effort.

In The Software Quality Challenge by the brilliant and sadly-deceased Watts Humphrey, published in the also sadly-deceased CrossTalk magazine (June 2008), Humphrey shows that the number of paths (and therefore required tests) a program can take depends on the quantity of decisions contained. Here's his results:

His analysis is based on an extreme assumption that every branch is likely to affect every other. However, the numbers are stunning and thought-provoking. One takeaway is that it's incredibly important to decouple modules as much as possible so Humprey's assumption (every branch is likely to affect every other) doesn't haunt our efforts.

He goes on to graph pre-test error rates for 810 experienced developers:

(Note that test finds only about half of all defects, a result consistent with many other studies).

I generally don't do traditional reviews of software, as most software tools have a vast number of features. However, I do make general observations about these tools, which is what follows for SEGGER's J-Trace Pro and their "Ozone" debugger. First, full disclosure: SEGGER places paid ads in the Muse and on my web site. The J-Trace Pro I evaluated is for Cortex-M/A/R, processors.

The motivation for these observations started at last summer's Embedded Systems Show in California. I was walking past SEGGER's booth and saw a demo of their coverage feature (which shows the lines of code that were executed). While I think measuring coverage is important, it's also problematic as most tools instrument the code. For line-by-line coverage that can add a lot of overhead, undesirable in a real-time system. I inquired and was told SEGGER's version adds no overhead. It's measured by watching the real-time trace data supplied by the Arm processor. That's a bit of a game changer.

Part of the problem with usual test regimens is that they are usually incomplete; one just doesn't know if the entire program was exercised. With coverage that ambiguity goes away. While coverage alone doesn't prove the code is correct, it does show that everything was at least executed. That is mandated by the most demanding standards, like the highest category of avionics testing. (The recent 737-MAX woes also demonstrate that test alone is inadequate).

SEGGER's J-Trace is a hardware debugging probe. While it works with IDEs from many other vendors, for my testing I used SEGGER's Ozone debugger program. The package costs $2498 and is currently available. A Cortex-M only (no -A or -R processors) is $1748.

Installation of Ozone and starting the demo on the supplied Cortex-M4 demo board took only a few minutes and was brain-dead easy. The demo program is a simple two-task LED blinker using the company's embOS RTOS. Ozone appears to have SEGGER's SystemView monitor (I reviewed it in TEM299) program integrated.

There are so many useful windows in Ozone that you'll want a big monitor. Mine is 24" and I wouldn't want anything less.

The application appears deceptively simple when first opened. File, View, Find, Debug, Tools, Window and Help are the only pull-down menu items; with the exception of the View menu, each of those pull-downs has only a few entries. It's mostly controlled via a vast number of right-click context-sensitive items. Ozone is very flexible in letting you add buttons to the toolbar, and I suspect that after a few minutes using it you'll add your favorite actions to reduce two clicks to one.

Here are my observations:

• At a breakpoint the Instruction Trace shows, unsurprisingly, trace data up to the breakpoint. As in every debugger, other windows show the current registers, disassembly and the like. If you click on an executed instruction in the trace window the disassembly and register windows show their values for that time - you can go back in time to see those values.
• The trace window can show the elapsed time for each instruction in cycles, instruction count, or absolute time, with a resolution of 1 ns.
• The call graph shows stack usage. In some cases that can't be determined in which case a "+" is appended to the size. In that case, in the depth column the reason for the uncertainty is given: "R" meaning recursion and "FP" denotes function pointer:
• 
• A "Code Profile" window gives execution statistics about the program's functions, either at the source or instruction level. The window shows code coverage on a source and instruction level, and the number of times each was invoked. In some cases the "fetch" and "run" counts differ. For example, a conditional statement may only be partially executed:
• 
• Similarly, the source code window shows how often each line of code has been executed. The numbers in green on the left of this image are those counts, which are updated in real time:
• 
• Ozone can monitor "expressions": C-like constructs that it samples from time to time (down to 1 us intervals). It appears to me that these don't steal execution time from the running program:
• 
• Like most debuggers, hovering over a variable will display that items value. But Ozone does this in real time, without slowing the code. And it will also show the value of expressions.
• Need to watch variables change? The Data Sampling window will monitor selected variables and expressions. This image is better than 1000 words. In this case it's sampling the two variables at 10 KHz. The max update rate is dependent on several parameters, such as the number of expressions being monitored:
• 
• An alternative way to watch expressions is the use the Timeline window. This image shows it monitoring two incrementing variables in the upper two lines, and the call stack at the bottom.
• 
• This demo program is in loops nearly all the time so little calling is being done. However, zooming in shows the call graph. And remember, this is in real-time:
• 
• Like a few other debuggers, Ozone will profile power consumption. That's displayed in the Timeline window. The following image, taken from the user manual, shows a series of LEDs being turned on as a function of the value in a variable (top red trace) and the resulting power consumed, in mA, in the bottom red trace.
• 
• RTOS-awareness features are common today, and Ozone likewise peers deeply into the system's multitasking. Many options are supported, from looking at task structures, task execution, stacks, mailboxes, queues and more. Of course, embOS, SEGGER's own OS, is supported as are those from other vendors. But interestingly, the debugger will support any RTOS. A Javascript plugin, documented in the manual, lets the developer link Ozone to even proprietary operating systems.

There's much more in the Ozone/J-Trace tool. The above comes from reading the first 135 pages of the 312 page manual; I simply ran out of time.

The manual is pretty complete but I felt it's hard to use; it needs more examples of using the features. And there are a lot of forward references which can make understanding a feature a bit difficult. The manual is arranged in alphabetical order of the windows which means some complex ideas are introduced before the basics.

Ozone is extremely responsive, and I encountered no notable delays. Click on this short video to see how quickly things are updated as I slew the cursor around:

The application is pretty intuitive, though there are so many features you'll want to curl up with the manual. Did I say feature-rich? I can't think of anything extra we'd want in a debugger other than the mythical "find bug" and "fix bug" buttons.

A very important feature is that acquired data can be exported to a file. That's crucial in many safety-critical projects where regulatory agencies require proof about testing completeness. And it's valuable to QA departments as well. Examples of exportable data include code coverage, count of statements executed, register, variable and even expression results.

In the 1980s and 90s I ran a company that made In-Circuit Emulators. These were debugging tools that offered tons of features that all ran in real-time. The ICE business imploded early this century for a number of reasons, but the death knell was that they just couldn't keep up with emerging fast clock rates. The embedded community then regressed to simpler debugging probes which were inexpensive, but feature-poor. Ironically, our applications were growing in complexity yet our tools fell behind. The J-Trace has gone beyond what ICEs could do. I'm very impressed.

You can find an overview of the J-Trace Pro and a tutorial about using it here.

Wouter van Ooijen has thoughts about C++"

Finally, Capers Jones published Twenty Five Software Engineering Targets from 2015 Through 2019 five years ago. These aspirational goals might make a basis for New Year's resolutions. I disagree with some of these, and many, while as desirable as a resolution to achieve world peace, are about as hard to attain. (Jones uses function points, each of which is about 130 lines of C.) His yearning to reach measurable goals is praiseworthy:

1. Raise defect removal efficiency (DRE) from < 90.0% to > 99.5%.
2. Lower software defect potentials from > 4.0 per function point to < 2.0 per
   function point.
3. Lower cost of quality (COQ) from > 45.0% of development to < 15.0% of
   development.
4. Reduce average cyclomatic complexity from > 25.0 to < 10.0.
5. Raise test coverage from < 75.0% to > 98.5% for risks, paths, and requirements.
6. Eliminate error-prone modules (EPM) in large systems.
7. Eliminate security flaws in all software applications.
8. Reduce the odds of cyber attacks from > 10.0% to < 0.1%.
   
9. Reduce bad-fix injections from > 7.0% to < 1.0%.
10. Reduce requirements creep from > 1.5% per calendar month to < 0.25% per
    calendar month.
11. Lower the risk of project failure or cancellation on large 10,000 function point
    projects from > 35.0% to < 5.0%.
12. Reduce the odds of schedule delays from > 50.0% to < 5.0%.
13. Reduce the odds of cost overruns from > 40.0% to < 3.0%.
14. Reduce the odds of litigation on outsource contracts from > 5.0% to < 1.0%.
15. Lower maintenance and warranty repair costs by > 75.0% compared to 2015
    values.
16. Improve the volume of certified reusable materials from < 15.0% to > 75.0%.
17. Improve average development productivity from < 8.0 function points per month
    to >16.0 function points per month.
18. Improve work hours per function point from > 16.5 to < 8.25.
19. Improve maximum productivity to > 100 function points per staff month for 1000
    function points.
20. Shorten average software development schedules by > 35.0% compared to 2015
    averages.
21. Raise maintenance assignment scopes from < 1,500 function points to > 5,000
    function points.
22. Replace today's static and rigid requirements, architecture, and design methods
    with a suite of animated design tools combined with pattern matching.
23. Develop an interactive learning tool for software engineering based on massively
    interactive game technology.
24. Develop a suite of dynamic, animated project planning and estimating tools that
    will show growth of software applications.
25. Introduce licensing and board certification for software engineers and specialists.
    

The full paper with his reasoning behind each goal can be found here (registration required).

Not quite yet a product, but for ultra-low-power products, why not use the triboelectric current from falling snow?

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

At the source of every error which is blamed on the computer you will find at least two human errors, including the error of blaming it on the computer.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• New Year's Resolutions for Firmware Developers
• Principles for Interfacing With Humans
• Just How Big is a Million Lines of Code?
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

Engineering changes at an accelerating pace. The boss wants more bug-free code, today. Tomorrow he'll want even more in the same time-frame. Are you on top of the best ways to deliver it?

Some tell me they just don't have time to attend seminars. I'm reminded of this:

It is important to fight today's engineering battle. But the war will be lost if you don't find better ways to win.

This is what my one-day Better Firmware Faster seminar is all about: giving your team the tools they need to operate at a measurably world-class level, producing code with far fewer bugs in less time. It's fast-paced, fun, and uniquely covers the issues faced by embedded developers.

I'll present the Better Firmware Faster seminar in Melbourne and Perth, Australia February 20 and 26th. All are invited. More info here. The early registration discount ends January 20. As I noted in a recent blog, I'm cutting back significantly on travel, so this will be my last trip to Australia. It's a beautiful country with great people, but a long way from my homestead in Maryland.

Jack's latest blog: How I Write Code.

A recent article in Electronic Design spotlights crashing DRAM and NAND prices. The latter are off 79% from their mid-2017 numbers. But a stunning graph later in the article shows that 1 GB of DRAM cost around $70k in 1991! It's around $3 today. I don't know how accurate that $70k number is, but even if it's off by an order of magnitude that shows astonishing progress in electronics over the last three decades.

Coding is "90% finished" for half the total coding time. Fred Brooks

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

John Carter likes the D language, and he also included a useful link to Stroustup's and Sutter's C++ guidelines:

Interestingly, D does not support 8 or 16 bit processors. From the overview page: "No consideration is given in D for mixed near/far pointers and all the machinations necessary to generate good 16 bit code. The D language design assumes at least a 32 bit flat memory space and supports 64 bit as well." I assume that means no MMU or MPU, which is a shame, as these resources can be useful in preventing complete system crashes.

This month's giveaway is a Cypress CY8KIT-044 PSoC 4 M-series Pioneer kit.

Enter via this link.

Percepio is giving away one-year licenses for their TraceAlyzer debugging tool. I've played with it and find it a great way to visualize how your code interacts with the RTOS (many RTOSes supported). The giveaway is not exactly free - you have to send a description of what you'll do with it, and follow-up with a story about solving the problem. More info here. I image the story could be pretty short.

Not a freebie for Muse readers, but here's an astonishing giveaway: a business card that runs Linux. Production cost: $3 in low volumes. Linux for three bucks? For all the sturm and drang in the political news, sometimes we need to take a step back and just be amazed at how cool things are. Thanks for Martin Buchanan for sending the link.

Happy New Year!

I thought it would be fun to Google for New Year's resolutions for software people. Most of the results were things like "share more," "work with the community," "contribute to open source," "cut back to one quart of whiskey per day," and the like. Nice thoughts, but what technical resolutions are worthwhile?

First, I don't like aspirations like "work harder at..." That's impossible to measure. Without numbers, we're lost in a fog. Without numbers, we can't understand if we're improving. Without numbers, we don't know how we measure up against others (e.g., am I in the bottom quartile of developers?). And, as Harry Roberts demonstrates in Quality is Personal, "trying harder" rarely works.

So here are some thoughts:

• Aim to elicit x% of a project's final requirements. (Requirements churn is pretty easy to measure, either with a spreadsheet or a tool like Jira.)
• Solicit criticism. Have a trusted mentor review x of your best modules.
• Achieve x% test coverage (more about this coming next issue of the Muse).
• Eliminate x% of defects pre-test. (Testing will never be adequate by itself. Most test schemes only exercise half the code. Tons of data shows how test is just not enough, or not even possible.) Our goal should be to write correct code, not to beat poor code into submission.
• x% of my new code will have a cyclomatic complexity under y. (In general, complexities over 10 to 15 mean trouble. Plenty of tools will measure this).
• Find x hours per day for uninterrupted work on one project. (Multitasking on three projects consumes 40% in context switching time. Each interruption costs 15 minutes in overhead).
• Achieve an x% comment density. (Plenty of tools measure this. Best-in-class code has 60 to 70% comment density. That is the percentage of non-blank lines that are, or contain, comments. Plenty of data supports this, such as this. Many argue that comments aren't needed, but after reviewing oceans of code I disagree).
• Reduce shipped defects to under x% (measured as a percentage of all defects found in engineering up to 90 days after shipping). Only 0.4% of companies remove more than 99% of all defects pre-shipping.
• Read x technical books per year. (The average developer reads one. For book recommendations see this. I take notes about the most important points when reading tech content.)

John Carter sent an excerpt from Standard Practice for Human Engineering Design for Marine Systems, Equipment and Facilities. These notions should be embraced by all engineers. Humans are flawed, which this standard addresses explicitly.

But first, a story. I read (somewhere recently, maybe in the Smithsonian's Air and Space magazine) that during WWII B-17s crashed on landing far more often than one would expect. Turns out, the gear and flaps levers were located near each other and looked similar. Tired, stressed pilots would mix up the controls. Relocating and changing the appearance eliminated those sorts of accidents. The moral: design with real people in mind.

Here are the practices:

4.2 Principles of Human Behavior:

4.2.1 There are basic principles of human behavior that control or influence how each person performs in their workplace. Some of these behaviors are culturally derived, while others are general and uniform across all cultures and geographical regions of the world. These behaviors influence a person's physical, social, and psychological approach toward the work they do and how safely they do that work. Failure to satisfy these behavioral principles in the design of a ship or maritime structure can encourage, or even coerce, maritime personnel into taking unsafe risks in their everyday activities. It is, therefore, imperative that designers of ships and maritime equipment, systems, and facilities know these principles to provide a safe and efficient workplace for maritime personnel.

4.2.2 These principles include:

4.2.2.1 If the design of the ship or maritime facility is considered to be unsafe or inefficient by the crew, it will be modified by the users, often solving the initial problem but introducing others that may be as bad, or worse, than the original.

4.2.2.2 Equipment design shall be such that it encourages safe use, that is, does not provide hardware and software that can be used in an unsafe manner.

4.2.2.3 If the equipment or system is not designed to operate as the users' cultural and stereotypical expectations lead them to think that it will operate, the chance for human error is significantly increased.

4.2.2.4 If equipment or systems are perceived by operators/maintainers to be too complex or require more effort to operate or maintain than they believe is necessary, they will always look for a "shortcut." Further, this "shortcut" may be perceived as being safe when it is not.

4.2.2.5 No amount of training, company or organizational policy, threats of retaliatory action, warning notes in a technical manual or training guide, or pleading with personnel to be safe on the job can overcome poor design that encourages, leads, or even coerces personnel into unsafe acts on the job. The most efficient way to prevent unsafe design from contributing to an accident is to eliminate the unsafe design.

4.2.2.6 Equipment users tend to be very unimaginative when it comes to identifying unsafe features and they do not visualize the consequences of unsafe acts. Therefore, do not expect that an "obviously dangerous" task will always be recognized as such by every user.

4.2.2.7 Designers shall consider the possibility for human error and design equipment so that incorrect use (deliberate or accidental) will result in little or no harm to the user.

4.2.2.8 Equipment operators and maintainers tend to make guesses as to what a label, instruction, or operational chart states if it is not complete, legible, readable, and positioned correctly.

4.2.2.9 Designers and engineers shall never use themselves as the standard against which a particular design is evaluated. People come in many shapes, sizes, mental capacities, and capabilities. Therefore, design for the full range of potential users, physically, mentally, and socially.

4.2.2.10 People shall be protected against themselves. Designers cannot create an unsafe piece of equipment or system and expect the users to assume full responsibility for its safe use.

4.2.2.11 Ease of equipment maintenance affects the equipment's reliability, that is, the harder it is to be maintained, the less it will be maintained.

4.2.2.12 Equipment designed to require multiple operators working together simultaneously increases the likelihood of operator errors.

4.2.2.13 Operational/maintenance procedures shall be clear, definitive, and comprehensive, otherwise, they will be misinterpreted or ignored.

4.2.2.14 Structural items such as piping, cable trays, or any other item that appears strong enough to be used by a person to hold onto or stand on, and is placed in a convenient location to use for that purpose, will eventually be used for that purpose.

4.2.2.15 Humans expect consistency in the design and arrangement of their workplace. Therefore, if that workplace, or any part thereof, appears in more than one place in their work environment, it is expected to be located and look the same way at every location.

4.2.2.16 When controls and displays associated with particular pieces of equipment are placed on a console or control panel, they shall be located on that console or panel to replicate the actual location of the equipment on the ship or structure as both are viewed by the operator. Therefore, equipment that is to the operator's left as he/she faces the control station shall appear on the left of the control panel or console, and equipment to the right shall appear on the right side of the console or panel. This "spatial relationship" between the real world and the controls and displays that are associated with the equipments and systems of that world is extremely important in the design of ships and maritime structures.

4.2.3 Humans develop behavioral patterns based on their cultural experiences. Designing a ship or structure that ignores or violate those culturally derived behavior patterns will inevitably lead to human error.

A million lines of code. It's a number bandied about more than ever as software sizes develop overactive pituitaries. Some cell phones use upwards of twenty million. Last year Ford announced that the F150 truck has 150 million. Everett Dirksen once may have said: "A billion here, a billion there, pretty soon you're talking real money." Well, a million lines of code here, a million there, pretty soon you're talking about a program that is as mind boggling and incomprehensible as our national debt.

A million lines of code printed out would be 18,000 pages. That's a stack six feet tall (on typical 20 pound paper). Ironically, the listing weighs in at 180 pounds while the actual operating code is mass-free; it'll live in a fraction of a gram of silicon. Like DNA, code's human-readable description requires tremendously more mass than its actual instantiation.

A million lines of code is probably on the order of 20 million instructions, or around half a billion bits. That's not far off of the 3 billions base pairs in human DNA. Unlike DNA, which has redundancies and so-called "junk" sequences, every single bit in the code must be perfect. A single error causes greater or lesser failure.

Since a typical atom is around 0.3 nm in diameter, if one had as many atoms lined up as the number of instructions needed for a million lines of code, they would stretch 10 cm. That many Ebola viruses would stretch 15 meters.

A million lines of code is as long as 14 copies of War And Peace, 25 of Ulysses, 63 copies of The Catcher in the Rye, or 66 copies of K&R's C Programming Language.

A million lines of code is not ten times more than 100,000. It's well-known that schedules grow exponentially faster than the code. Barry Boehm estimates the exponent is around 1.35 for embedded software. So the schedule for developing a million lines of code is 22 times bigger than for 100,000 LOC.

In the March, 1996 issue of IEEE Computer Watts Humphrey published crude rules of thumb for estimating software projects. Though hardly scientific, they do give a sense of scale. Using his estimates:

• A million lines of code requires 40,000 pages of external documentation.
• A million lines of code will typically have 100,000 bugs pre-test. Best-in-class organizations will ship with around 1k bugs still lurking. The rest of us will do worse by an order of magnitude.
• A million lines of code will occupy 67 people (including testers, tech writers, developers, etc) for 40 months, or 223 person-years. Darwin needed just 1.5 person-years to write The Origin of the Species. Scale that to the 26 copies equal in length of a million lines of code, and it appears writing code is some 6 times more time-consuming than writing a revolutionary scientific tome.

A million lines of code costs $20m to $40m. That's the cost of an F-18 jet, a thousand cars (in America), �ten million gallons of gas, seven times the inflation-adjusted cost of the ENIAC, and something like ten million times the cost of the flash chips it lives in.

Think about that last analogy: Ten million times the cost of the flash chips. Accounting screams over each added penny in recurring costs, while chanting the dual mantras "software is free," and "hey, it's only a software change."

Hex-Five Security's MultiZone product is a software layer that uses hardware to enforce high security when using untrusted code. Currently RISC-V versions are available.

It's a small (under 2 KB) bit of code that is formally verifiable. That's a big selling point when pursuing a Common Criteria security standard. As I understand it, the product enforces separation zones to keep bad actors out of others' sandboxes, which sounds like the Common Criteria's separation kernels.

A datasheet is here. It's free for non-commercial use.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

The primary purpose of the DATA statement is to give names to constants; instead of referring to pi as 3.141592653589793 at every appearance, the variable PI can be given that value with a DATA statement and used instead of the longer form of the constant. This also simplifies modifying the program, should the value of pi change. - FORTRAN manual for Xerox Computers

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor's Notes
• Quotes and Thoughts
• Tools and Tips
• On Management
• The 1802 Processor
• This Week's Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Over 400 companies and more than 7000 engineers have benefited from my Better Firmware Faster seminar held on-site, at their companies. Want to crank up your productivity and decrease shipped bugs? Spend a day with me learning how to debug your development processes.

Attendees have blogged about the seminar, for example, here, here and here.

In another blow for Boeing, 737-NG aircraft can't land going west. If the runway is facing 270 degrees (true) all six display units blank. Happily, there are only 7 such runways in the world. FAA Airworthiness Directive here.

Dave Gutow won last month's contest, for a PSoC dev kit. There's no contest this month due to a few weeks of travel. Stay tuned for March's giveaway.

From Eric Smith: My favorite quote regarding programming languages in general is C.A.R. Hoare on Algol 60: "Here is a language so far ahead of its time, that it was not only an improvement on its predecessors, but also on nearly all its successors."

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Brian Cuthie likes Visual Studio Code and the Saleae logic analyzer:

Tomasz Kos wrote:

We've made some improvements to our tools page, which is comprised of short reviews and comments of various hardware and software tools we embedded people use.

The voice on the other end of the phone was quiet, furtive. In guarded tones "Joe" recounted the sordid details of his story, as if I were an investigative reporter. "I can send you documents, copies of our listings, memos, you name it", he murmured.

He worked for a small under-capitalized firm which was intolerant of any sort of failure. Joe's boss had a rule: software bugs were unacceptable. At first I mistakenly assumed he meant delivered products were to be defect-free, a noble goal I'd hope we all share.

"Nah, we're not allowed to do any debugging," he almost whispered. "The boss says we've got to write code correctly the first time, and he's not spending any money on fixing our mistakes. So there are no tools and we all work OT to fix problems."

I was reminded of my teenaged technician years, when the QA manager decreed that we were to replace only defective components, under threat of dismissal. From time to time we techs screwed up, of course, removing parts that in fact were just fine. Since Mr. QA had us send all of these extracted components to an outside lab for failure analysis, we learned to manually fry those that we had replaced in error. His in-basket consequently received some quite puzzling failure reports from the lab. Once I killed as op amp in a metal can by applying 110 VAC to the leads. It blew a hole through the can. I never saw the failure analysis for that part but imagine much head-scratching ensued.

Part of me wanted to believe that Joe's boss was a man of deep insight, who had created a dynamic development environment where the product was so well designed all - or most - bugs disappeared before test started. Unhappily the man was as capricious as my old QA tormentor, as he mandated a terribly difficult goal without providing any system or framework in which to achieve it.

Developers almost universally complain about managers, for good reasons and bad. The number one complaint I hear is about fickle decision making. Schedules with imposed end-dates, or those arbitrarily revised. Disappearing resources. Marketing overrides very real technical requirements, and even impose plainly impossible technical goals.

Too many of today's bosses are often preoccupied with generating reports, preparing for shows, and endlessly meeting with vendors. Though these are important roles, managers have two critical functions that are often neglected. First, managers must manage. This means tracking the day by day, week by week progress of an engineering effort, which presumes there's an honest schedule which forms the project's roadmap. When something changes - and things always change - then an effective manager revises the roadmap.

Management means acting as a buffer between the doers and upper level decision makers, insulating developers from distractions from on high, and waging battles to secure needed resources.

Management means looking forward and anticipating changes and problems, clearing minor potholes before they become roadblocks. When disaster strikes it means working with the team to find reasonable solutions, and negotiating with the manager's superiors to adapt to the new realities, without imposing impossible requirements on the development team.

But there's another aspect to proactive management. Your boss is immediately tasked with building a product, but much more important his role is to create an effective, constantly improving, way of building products. Keep your shoulder to the grindstone and you will eventually fail.

Truly enlightened managers transcend an incessant focus on "shipping the damn thing now" to look a year or more ahead, and to develop the skills and technologies that will surely be required.

This is an extension to the philosophy promoted in The E-Myth Revisited by Michael Gerber, one of the better books around for people building businesses. Gerber suggests that too many small companies (and by extension, engineering departments) fail because the leader spends too much time doing productive work, and not enough time inventing a company (department) that does better work in less time. We've all discovered a tool that could save us weeks of effort, but being trapped by an impossible deadline we feel we can't spend time to install and learn how to use it.

Then there's us.

I've observed that all too many developers seem to reach the zenith of their abilities in college. Yes, building real products in industry hones our skills though repetitive practice, but the raw knowledge of how we should design systems and write code seems cast in concrete when the diploma arrives.

How many engineers over 30 ever truly master a new concept like OOP or UML? For every success story there are too many who stayed locked into the concepts with which they have long been comfortable.

We can acquire new skills only by dint of struggle and practice. With today's development effort always overwhelming every other consideration it's no surprise we careen from project to project with little chance to improve. Home life gets ever more complicated with time; children and spouses require attention that detracts from after-hours study opportunities.

Yet this field changes daily. Those not changing with the times are doomed to professional lives in ever more restrictive corners of the field. My dad told a story of a mechanical engineer he knew in the 1960s who was the world's expert at designing wheels for lunar roving vehicles. Surely that poor fellow became unemployable during the engineering crash of the early 70s.

We should take charge of our own careers, and manage those careers aggressively. Figure out what you need to know, and then take action.

But managers, too, must enhance their peoples' skill sets. A manager's most expensive resource is his or her people. Let them stagnate and the entire team will degrade.

We know how to improve software development. The Capability Maturity Model (CMM) is one such process. Though the CMM is no panacea, it does describe a disciplined way to create software products in a predictable manner: predictable bug rates, predictable schedules, and the like.

Like ISO9000, the CMM requires a major commitment from management as it changes the nature of the entire organization. Effective? Yep. Hard? You bet.

In my opinion software development is so hard that we need every asset possible to help us. Pursuing the CMM is one strategy.

Another is Watts Humphrey's Personal Software Process (PSP). Where the CMM requires a total management buy-in from the top down, the PSP is an approach individuals and small teams can use.

Yet when I talk to firmware developers just about 0% work for a company pursuing the CMM. Another 0% know anything about the PSP.

Managers must manage their people's skills. They've got to cajole, plead, push and order their folks to study these and other ideas to improve. Stagnation is death.

And we need to learn from our mistakes. There is no industry I admire more for their management of failure than the commercial airplane business. Every crash brings out an army of investigators who are tasked with finding the accident's cause. The planes themselves are equipped with the notorious "black boxes", tools designed to record and report the parameters that lead up to the crash. When possible, investigators recommend changes in training, documentation or plane design so the chances of a similar problem will be reduced.

In other words, the system has a built-in learning process which is highly targeted towards making changes

Why don't we work as hard to learn from our mistakes? Failure will always be part of doing anything difficult. We'll miss schedules, deliver products whose quality falls below expectations, run over budget, and even suffer massive redesigns due to our own problems and changes imposed from without.

A wise manager simply has to understand that if we don't manage failure, if we don't study our mistakes and turn them into learning experiences, we'll repeat the same old dysfunctional problems forever.

The airplane industry has a feedback loop that improves the process. If we don't emulate their example via aggressive failure management, we'll also have a loop: Repeat forever: the usual mistakes. End Program.

I've often wondered what happened to Joe. Did his boss resign him to he scrap heap of failed employees?

How many of us either manage only to meet just a project's goals, or manage to build an awesome engineering department?

Back in the late 70s processors consumed a fair amount of power. It wasn't unusual for one to need two or even three power supplies, one of which was generally a negative voltage. RCA took a different approach: they bet heavily on CMOS technology, which seemed a backwater to most cutting-edge designers.

CMOS was both extremely low power and highly radiation resistant, factors critical for space and a few other applications. CMOS, though, was slow, very slow. Laughably slow. Switching speeds could be in the hundreds of nanoseconds, a far cry from the 10-20 nsec speeds achievable by reasonably fast TTL. Most of us considered CMOS an interesting idea that would soon die off. We were all so wrong; today CMOS's speed problems are gone, and it's the logic of choice for most applications. In fact, only the use of CMOS now avoids complex chips from self destructing.

RCA joined the microprocessor fray with their 1802, a fully CMOS 8 bit device with a very unusual architecture. It was intensely register based, having many times the number of internal registers as any of the competing NMOS devices. Any one of those could be the program counter. Today we find that one of the tenants of RISC technology is to similarly maximize registers. RCA was clearly ahead of its time.

Its greatest flaw, though, was a total lack of a stack. No push, no pop. No call; no return.

Indeed, the 1802's innate lack of call instructions was a source of continuous grief. We worked around this by building a non-reentrant pseudo-subroutine structure: you do a call by loading a fixed register with the return address, another with the destination of the subroutine, and then jumping to a "subroutine" that stored the return in memory on a manually-managed stack before finally vectoring off to the destination. Returns were managed in a reverse manner. Rather like a primitive version of Arm's R14, aka LR (link register).

The overhead was enormous, but the 1802 was never known as a speed demon. Instead, it found its way into ultra low power applications.

My business partner and I managed to get a number of jobs designing deep sea probes that monitored temperature, pressure and other parameters, running for months on small batteries. The 1802 was our only option, as it required only microamps when operating at slow clock rates.

We used an assembler which ran on a floppy-based CP/M machine. Happily, the assembler included macro capability which made generated the convoluted call/return sequences much easier.

We wrote a very simple monitor program that permitted us to download hex files, fiddle with memory and registers, and stop/stop program execution. The cost of an emulator - about $20k in 1981 dollars - was far beyond our bankroll.

The 1802's fully static design greatly simplified debugging. We'd run the machine at one hertz or less at times, using a scope or even a voltmeter to probe address and data lines.

Later we found a customer doing research into the aging of fruit. Apparently, during the course of a month-long trans-oceanic voyage fruit emits various gasses as it ripens. Our customer wanted to understand this process, and so desired a system that could sample the air around a fruit container a number of times over the course of the voyage.

The solution was to fill (empty?) 30 aluminum containers (ex-fire extinguishers) with a vacuum. A valve on the top of each opened for a few seconds at programmed intervals to suck in the ambient environment. We quickly concluded that only an 1802 could stay alive for the 30 required days, running from a small lead acid battery.

The CPU ran a simple program that accepted timing parameters, and that then counted time till it made sense to open a valve. As I recall we ran the 1802 at about 100 hertz to conserve power.

The data collection program was a great success... until a dockworker dumped a 50 ton container on the instrument.

In the early embedded days we debugged code using text-based tools. Set a breakpoint and the terminal (or, earlier, the TTY) would print a character when the program stopped. That morphed into IDEs using windows, a huge improvement, but we're still interacting with the project via a mouse and keyboard. Thirty years ago I dreamed of a debugging tool that would let an engineer manipulate firmware via a VR headset. Grab that interrupt vector and reroute it. That never happened.

A new tool for engineers working on PCB hardware, though, is somewhat available (somewhat, as the free, limited, version is out, but it appears the pro versions have not yet been released). It's called inspectAR, and utilizes augmented reality.

The website is a bit light on details, but watch the second (3 minute) video on the home screen to get a sense of what the tool can do. It's pretty cool!

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Elephant: (n) A mouse with an operating system.

The Embedded Muse is Jack Ganssle's newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• More on Reuse
• Sequence Points in C
• Precedence Rules
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

It has been two years since my last salary survey. Follow this link to take the 2020 version (it''s only a dozen questions), which also has a question probing your work in these coronavirus times. I''ll publish the results in the May 4 Muse. Three copies of The Art of Designing Embedded Systems will be awarded to three randomly-selected participants. Based on early results, the vast majority of us report we''re working from home due to the virus.

Jack''s latest blog: Lessons From a Failure

I''m on the AmpHour ("Keeping You Current") this week.

The schedule you develop will seem like a complete work of fiction up until the time your customer fires you for not meeting it. David Akin

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

When faced with an intractable bug... wash your hands!

A new(ish) site for industrial IOT is interesting.

Percepio, responding to engineers working at home during the Coronavirus pandemic, are offering two months free use of their Tracealyzer.

This month''s giveaway is a BBC Micro:bit Go Bundle.

Enter via this link.

In reply to the last Muse''s comments on reuse, Stuart Jobbins wrote:

John Carter commented:

Do you know what this code does?

  sum = sum * 10 - ''0'' + (*p++ = getchar());   

Don''t feel bad; no one does. As the standard says: "the actual increment of p can occur at any time between the previous sequence point and the next sequence point (the ;), and the call to getchar can occur at any point prior to the need of its returned value."

The standard lets compilers evaluate parts of an expression with all of the discipline of sailors at a bar on shore leave. It says "The order in which subexpressions are evaluated and the order in which side effects take place, except as specified for the function-call (), &&, ||, ?:, and comma operators" is unspecified.

To give us predictability the standard has the concept of sequence points, To quote the standard again: "The presence of a sequence point between the evaluation of expressions A and B implies that every value computation and side effect associated with A is sequenced before every value computation and side effect associated with B."

A sequence point occurs on:

• A call to a function, after the arguments have been evaluated
• The end of the first operand of the following operators:
  • ? logical AND &&
  • ? logical OR ||
  • ? conditional ?
  • ? comma ,
• The end of a full declarator (a declarator is the part of a declaration (definition or initialization) that is not the type)
• The end of a full expression:
  • ?? an initializer
  • ?? the expression in an expression statement (e.g., p(0);)
  • ?? the controlling expression of a if, switch, while or do
  • ?? each of the expressions of a for statement
  • ?? the expression in a return statement
• Immediately before a library function returns
• After the actions associated with each formatted input/output function conversion specifier
• Immediately before and after each call to a comparison function (e.g., qsort), and also between any call to a comparison function and any movement of the arguments

So be very wary of expressions like:

b = a + a++;

The value of b is going to depend on the order of evaluation of the expression.

Because logical ANDs cause sequence points an expression like:

a++ && a++

is perfectly legitimate. But my preference is to avoid such constructs that may confuse the unwary.

Do note that adding a Lisp-like swarm of parenthesis will not change the compiler-generated order of evaluation.

MISRA rule 13.2 addresses sequence points: "The value of an expression and its persistent side effects shall be the same under all permitted evaluation orders." (Emphasis in original).

CERT rule EXP30-C is more succinct: "Do not depend on order of evaluation between sequence points. "

Do you know C operator precedence rules? I don''t. At least, not well. Most of us stumble when queried about this. In fact, in Developer Beliefs About Binary Operator Precedence by Derek M. Jones shows that even experienced developers strike out when asked which of C''s 18 operators is evaluated before the others:

John Watson was surprised when he found out that Sherlock Holmes didn''t know that the Earth revolves around the sun. Holmes said "What the deuce is it to me? You say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work." I feel the same way about precedence rules. They are arcane and uninteresting. Much better: use parenthesis when the order of evaluation isn''t crystal clear.

Clarity is our goal.

iRobot is launching a robot programming education initiative. They have (free!) on-line tools which let students develop and simulate robots. Looks like fun!

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Do you know Cobol? See this.

These jokes are archived here.

Tis the dream of each programmer,
Before his life is done,
To write three lines of APL,
And make the damn thing run.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• More on Knowledge Capture
• More on Hardware, Software or Hardware/Software?
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

Last issue''s articles about Knowledge Capture and Hardware, Software or Hardware/Software generated a flood of correspondence. It wasn''t possible to include all of your replies, but so many had so much to say that this issue of the Muse is focused on some of those replies. Thanks for all of the feedback, which I always appreciate.

Design is an iterative process. The necessary number of iterations is one more than the number you have currently done. This is true at any point in time. David Akin

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Referring to a note in the last Muse, David Strip wrote:

Jerry Mulchin responded to thoughts on cheap PCBs:

And Dave Smart wrote:

Sergio Caprile responded to my thoughts about state machines:

Francois Baldassari also had a useful link for finite state machines:

Dainius Stankevicius won the Firia Labs Explorer Kit in the July contest.

Firia Labs is donating one of their CodeBots (a Python-enabled robot with a host of sensors) for this month''s giveaway.

Enter via this link.

Anton Ivanovsky was one of many responding to the article on capturing knowledge:

Ted wrote:

Eric Geller corrected my reference to OneNote:

Graham wrote:

George sent this:

Carl Palmgrenhas a couple of suggestions:

Neil Cherry is in DevOps:

Frequent contributor Luca Matteini wrote:

In Robert Lange''s submission, (DFMEA = Design Failure Mode and Effects Analysis, CAR = Corrective Action Report, typically a root cause analysis of a failure:

Daniel McBrearty likes those who can look at a problem from 10,000 meters:

Martin Glunz is a systems person:

J.G.Harston was slightly off topic but brings up an important point:

John Sloan and many others had thoughts about this article in the last Muse:

Boi Okken wrote:

Steve Taylor made an important point about the Socratic Method::

I''m a fan of the Socratic Method, which is basically a process of helping people figure things out by asking good questions. It''s far better to mentor a young engineer by using questioning to help that person gain understanding than simply tossing off an answer.

John Kougoulos wrote:

John''s comments on specialization resonated with me. My dad was a mechanical engineer who worked on Apollo at Grumman. He told me one of the engineers there was the world''s expert in designing wheels for lunar roving vehicles. I''ve always wondered what happened to that person post-Apollo? Specialization can be useful, but remember that technology moves on and that certain skill can be obsolete overnight.

Francois Baldassari figures it''s all about seniority:

S. Stewart sent this. When the Wyze watch Android app doesn''t sync on Blue Tooth it seems to fill some data fields with 72123:

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Outage (as in Internet outage): Isn''t it interesting how close that is in spelling to "outrage"?

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Microcontroller Basics with PIC
• Requirements Traceability
• Failure of the Week
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Jack''s latest blog: Does Anyone Remember In-Circuit Emulators?

When in doubt, estimate. In an emergency, guess. But be sure to go back and clean up the mess when the real numbers come along. David L. Akin.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Though I''m hesitant to enter the editor wars, I''ve been using Ultraedit for many, many years and it is my preferred editor on Windows. They''re up to version 27; I''m using 24 as I just can''t think of any additional features I''d need or want. The Ultraedit folks do sell a file compare, but I''m addicted to Beyond Compare from the oddly-named Scooter Software. It''s fast and it shows deltas brilliantly.

Bob Paddock wrote:

July''s giveaway, courtesy of Firia Labs, is one of their Explorer Kits, which includes 2 micro:bit circuit board computers and associated components for experimenting.

Enter via this link.

What, another book about the Microchip PIC? Aren''t there shelves groaning under the weight of these tomes? Well, yes, but Tam Hanna''s new book Microcontroller Basics with PIC is a bit different. It''s aimed at students and embedded wanna-bees rather than practitioners.

This is a hands-on book that teaches both hardware and software design in a manner analogous to how practicing engineers work. Many simple projects are introduced and explained. The wise reader will build and test the projects. However, unlike every other hands-on book, there is no specified evaluation or demo board used. He recommends using a PIC16F18877 with a solderless breadboard like this:

Then, each of the projects is built simply by plugging in a few parts on the board. This is an excellent approach as demo boards come and go; if gone, so would the value of the book be. (Alas, Tam doesn''t use the word "solderless breadboard" and I''m afraid a novice wouldn''t know how to go about obtaining this item).

The book starts with the basics and begins with assembly language (another nit: it doesn''t explain what assembly is or why it''s important). MPLAB is the IDE used, and Tam gives a great overview of installing and using that product.

Hardware and software are presented as a unified gestalt, as is proper in the embedded world. Code to toggle a pin is accompanied with oscilloscope screen shots showing the result. Even better, drawings annotated with events show what''s happening when, such as:

Assembly instructions are presented with example usage and resulting scope outputs. I''d recommend that a reader have a scope handy to follow along; even a low-bandwidth USB scope would suffice.

Sometimes blinking an LED is that "ah-ha" moment that turns people on to embedded systems. I think the real exciting bit from the book is a table-driven sine wave generator and the accompanying scope shot.

The second half of the book is about using C. As the author notes, it''s not a C text and assumes the reader has some familiarity with the language. But it does dig into using interrupts and various serial protocols.

The book is chatty, perhaps too much so with many references to the author. But it is a great introduction to embedded systems and I recommend it.

I''d pair this book with Alex Dean''s Embedded Systems Fundamentals with ARM Cortex-M based Microcontrollers as two works, taken together, which would give a decent foundation to this field. Tam''s will get you going in the 8-bit world; Alex pushes that to 32 bits and more complex subjects. Both are eminently practical and readable.

Responding to last issue''s article about Software Elements Out Of Context, Harold Kraus wrote:

From Scott Sweeting:

Not quite a product, but researchers have a way of creating short pulses. How short? On the order of 5 ps. There''s more here.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

From Clive Souter:

I''ll never forget the first time I saw a universal remote.  I thought, "This changes everything!"

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Get it Wrong, Pay the Price
• Responses to Expanding One''s Capabilities
• Why Did You Become an Engineer?
• Failure of the Week
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

In any project, large or small, each engineer''s work is expected to be consistent and transparent so that another engineer can check it - by following its assumptions, logic and computations - for inadvertent errors. Henry Petroski

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Brian Cuthie addresses an issue I think is important with regards to watchdog timers:

#define  THREAD1 0x01 // arbitrary thread names here

#define  THREAD2 0x02

#define  THREAD3 0x04

  . 

#define  THREAD_MASK (THREAD1 | THREAD2 | THREAD3 .)

static  uint8_t wdt_accum; 

void  kickWDT(uint8_t thread) {

disableInterrupts();

wdt_accum  |= thread;

if  (wdt_accum == THREADMASK) {

   <do  whatever it is that kicks the hardware WDT>

wdt_accum  = 0;

}

enableInterrupts();

}

Jeremy Overesch and several others recommend Compiler Explorer:

The Sticky Bits blog has an article about using Compiler Explorer with test-driven development.

This month''s giveaway is a LA104 mini logic analyzer. I reviewed it in Muse 406.

Enter via this link.

In one of Capers Jones'' many papers he presents the following:

Defect removal efficiency (DRE) is the percentage of bugs found in all phases of developing a system up to 90 days after initial delivery. A 75% efficiency means that percentage of bugs was removed; another 25% lurk even three months after customer delivery.

What is the cost to fix these defects? There are a lot of numbers bandied about. Remember, these errors are those found in the field, post shipping, and are the most expensive of all. Some data points to something like $5000/shipped bug, but I''ve worked with customers where the numbers are much, much worse, typically in regulated environments. These post-shipping problems probably involve multiple people, including software engineers, managers, QA, telephone support and even field support. That number neglects the lousy image bad code gives of the vendor, which may ultimately be even more expensive to the business.

The post-compile total number of bugs in firmware typically runs 5 to 10% of the total number of lines of code. Assuming the mean of that, 7.5%, and we find these numbers for different sized projects with each of the three defect removal efficiencies, assuming $5k/fix:

I can''t tell you the number of times people have told me they don''t have time or a budget for process improvement. Given that the engineering costs to build a system is $20 to $40 per line of code, fixing fielded bugs in a 100KLOC device at a 75% DRE will consume about a third of all of the money originally spent on engineering the code in the first place.

(Disclaimer: the numbers vary tremendously. Use these to get a feel for the problem rather than an exact number. An impressionistic painting shows the shape and nature of an image; it isn''t a high-res photo. These metrics are similarly impressionistic, though that does not diminish their importance.)

A reader asked how one can go about getting less pigeonholed; widening one''s skills and adding value to a career. Several people responded.

Steve Johnson doesn''t specialize:

I agree with Richard Prosser about courting startups:

Nick Soroka has a trifecta of suggestions:

In the last Muse a reader asked why we became engineers. Christopher Eddy''s contribution makes me wonder about genetics: can 20k or so genes encode an engineer''s personality? Or, as some believe epigenetic factors affected later generations after the Dutch Hongerwinter in 1944-45 could there be a similar effect for nerds? (For a wonderful book on genetics which has some - in my opinion, speculative - discussion of the Hongerwinter and epigenetics, see The Gene: An Intimate History, by Siddhartha Mukherjee). Christopher writes:

Gordon Dick''s path was a little different:

To add a twist to these stories, I''ll include another personal anecdote. As I mentioned last issue, my dad was a mechanical engineer who spent his career mostly in the space program. He did not fall into the field in the same way as those quoted above. Just before WWII his parents bought a diner just outside of Grumman''s gates in Bethpage, NY. With the war Grumman''s business boomed and the diner prospered. In senior year of high school his dad promised to set him up with his own diner, which was all my dad wanted. A teacher more or less forced him to take a scholarship exam, pretty much against his will, and he earned a full ride to MIT. So he sort of fell into engineering accidentally.

Ray Keefe sends this week''s failure. He writes "This is from a running app on my phone. The numbers come good after a bit so it looks like it draws the GUI before it has completed the calculations and must not initialise the variables beforehand or mark them as not ready."

Bill Schweber has an article in EDN about using bus bars on a PCB to provide low-impedance power rails across a PCB. This is important in higher-power large boards. For example, Storm Power Components sells bars used like this:

This is hardly a new concept. Designing boards for Nova minicomputers in the 1970s we used these as these were 15 x 15" boards: enormous, and with hundreds of TTL ICs quite a few amps were needed. Multilayer boards weren''t around back then. The bus bars distributed the massive columbic flow and eliminated board flex.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

Flon''s Law: There is not now, and never will be, a language in which it is the least bit difficult to write bad programs.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Assembly - Dead as a Dodo?
• The LA104 Pint-Sized Logic Analyzer
• Taking Charge
• Failure of the Week
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

The real miracle of the Wright Brothers'' flight was than they accomplished it without the use of any electronics at all - Allen Puckett

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Carl Dreher wrote:

September''s giveaway is courtesy of Binho, and is one of their Nova Multi-Protocol USB Host Adapters with accessories.

Enter via this link.

A very long time ago, when dinosaurs still roamed the Earth, I designed a computer (using the fire-blackened end of a spear on a cave''s wall.)? Still in high school, I was geekily-somewhat competent with digital circuits but knew nothing about computer architecture. But I had learned Fortran and naively assumed some parallel between that language and how computers worked. The result: a machine that would have been a complete failure, and which used a greatly subsetted and compressed form of Fortran as its native instruction set.

College gave me grade-destroying access to a Univac 1108 and I learned its assembly language. Suddenly computer architecture became crystal clear. The one-to-one mapping of machine instructions to simple logic circuits was beautiful; the stored program that substituted instructions in memory for massive amounts of hardware breathtaking.

Since then I''ve read many books about computer design but feel none would reveal a fundamental insight into CPU architecture without relying heavily on the essentials of assembly language. The ALU, program counter and stack pointers are dead lifeless things,? capable of nothing till animated like Frankenstein''s monster with instructions stored in memory.

Assembly is both the basis of all computers and the name of a class of languages. Often used to specify a particular variant (e.g,. "8051 assembly"), it oddly doesn''t even get a capitalized first letter as all other languages do. Or, did, until grammar died a horrible death at the hands of clowns sporting marketing degrees. Proper nouns like Fortran, Ada, C, and Pascal gave way to iPhone, dBASE, and eEverything. The nuns at St. Camillus would have beaten us senseless for peppering our writing with uPPeraNdLowercAse, yet today that affectation is not only common, one is relieved when at the very least the spelling is correct.

In the early days of microprocessors all programs were written in assembly. No C compilers existed for the minimal CPUs of the day and memory was so expensive and processors so slow that no one dreamed of sacrificing any form of efficiency for reduced development costs. All firmware folk were experts at at least one assembly language. Usually several.

C and IDEs were desperately-needed additions to our toolbox. Though I still think assembly is more fun than using a high-level language, C reduces development costs so much I''d never dream of cranking much assembly code anymore.

But one effect of this great gift is a new distance between engineers and the underlying hardware. That has both positive and negative consequences. On the down side I worry that a lot of us no longer have that deep insight into computer architecture. A brief review of the nature of assembly in Programming 201 hardly forces one through PC-relative versus absolute jumps, or how the Cortex M uses a link register to preserve a return address. In real-time systems interrupts reign, but their very real costs are disguised by simple C structures that hide the stacking and unstacking of the system''s context.

Many people ask me how to go about becoming an embedded developer. Part of my advice is to learn - really learn - an assembly language.

Do you use much assembly anymore? Are you fond of programming at the bare metal level?

About a year ago I reviewed the MiniDSO, which is an iPhone-sized oscilloscope. It''s pretty cool and could be a great gift for an engineer who has everything. Recently Guangzhou E-design Intelligent Technology Co., Ltd. sent me one of their LA104 logic analyzers for a review. My understanding of Chinese companies is a bit vague as they have complex interconnections, but I suspect Guangzhou manufactures both the MiniDSO and the LA104. Both are a similar size and have pretty much the same user interface.

And the LA104 logic analyzer, like it''s scope brother, is a pretty cool bit of tech.

First, the specs: it''s rated at 100 MHz with 100 MSa/s sample rate. It sports four input channels with 8 MB of memory, though just how that memory is apportioned isn''t clear to me. I²C, SPI and UART protocol decoders are included, as are four programmable outputs. I couldn''t figure out how to set up the outputs.

The UI is unusual, and is much like that on the MiniDSO. Two wheels control a menu; one advances selections left and right, the other selects from vertical submenus. OK, MENU, ESCAPE and SAMPLE buttons complete the interface. It''s a little awkward at first but quickly becomes natural.

A ribbon cable plugs into the left side of the unit and forms the probes. There are nine connections: 4 inputs and ground, 4 outputs and +3V. The back of the unit has very nice labels indicating which connection is which.

Triggering seems to be pretty decent for such a tiny unit. Here''s the options from the user manual:

Here''s a closeup of the screen:

Note there are six channels displayed, yet there are only four probes. I don''t know what channels 5 and 6 refer to. The solid bar across the bottom shows the entire buffer; in this case acquiring a 25 MHz square wave on channel 1. Up from that are statistics for channels 1 and 2.

Though the unit has the ability to save screens as .CSVs or .BMPs, I didn''t see a way to recall them or to upload them to a PC.

The unit did not come with a manual, but you can download one from here. It''s rather incomplete.

The LA104 is available from many sources, such as Seeed Studio for $89.

You can''t beat the price. Me, I prefer a logic analyzer that sends data to a big PC screen. But if you''re price or space-constrained, this is a pretty remarkable bit of test equipment.

This will be February''s giveaway.

In one of the made-for-TV pseudo-histories of the Apollo program, an astronaut figured landing on the moon would be something like flying helicopters. So he learned to fly choppers. He didn''t ask permission. His boss wasn''t involved. No one told him to acquire this new skill. In the every-man-for-himself astronaut office astronauts took action to fill holes in their experience.

How often do you listen to a developer whine about his dead-end job? How many of us wind up in dead-end jobs because we refuse to gather new skills and experiences?

Back in the old days, paternal corporations benevolently managed their employees'' careers. You could leave college and work for IBM till you retired. They''d take care of you. Need a new skill? Your boss would identify the weakness and schedule some sort of training.

No more. In these troubled times companies are worried about their own survival. Cost-cutting and squeezing out a profit is all that matters. Get the product out the door today no matter what the personal cost may be. Joe is still struggling with C++? Replace the SOB! There''s an army of applicants knocking at the door. The company has neither the time nor the incentive to engage in training and career management.

I believe it''s our personal responsibility to upgrade our skills. constantly. Relentlessly. Technology changes so fast, and even the first derivative of that change is accelerating. Too many developers stop learning when they leave college. Do that, and your career will end in your twenties. Companies will use your skills as long as they are still relevant and then spit you out, a burned-out husk obsolete at 30.

It''s up to you, and to no one else, to learn new things, to become more productive, and to embrace the new technologies that will influence your career. There''s a wealth of resources available, from books to magazines, and websites. Some colleges have put their classes on-line, free of charge.

No doubt the astronauts have an incredible amount of personal freedom in their jobs, as well as a budget we can only dream of. They are all dynamic personalities who look at a problem, figure what resources might be needed, and immediately take action. That sounds like a great description of the engineering method.

Take action. Learn. Grow. Become expert at something new. Stasis in this industry is death. Especially now, when jobs are scarce and layoffs all too common.

From Glenn Hamblin:

RADAR has traditionally been an expensive sort of sensor, but that is changing. An interesting article (written by TI) enumerates a number of cool scenarios where low-cost RADAR can be useful. The focus is TI''s IWR6843, a 60 GHz RADAR on a chip (just add a power supply and antenna). It includes three transmitters and four receivers so can support beam forming. In singles it''s about $30, falling to $22 in 500 lots.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

All programmers are playwrights and all computers are lousy actors.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• More on Reuse
• Sequence Points in C
• Precedence Rules
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

It has been two years since my last salary survey. Follow this link to take the 2020 version (it''s only a dozen questions), which also has a question probing your work in these coronavirus times. I''ll publish the results in the May 4 Muse. Three copies of The Art of Designing Embedded Systems will be awarded to three randomly-selected participants. Based on early results, the vast majority of us report we''re working from home due to the virus.

Jack''s latest blog: Lessons From a Failure

I''m on the AmpHour ("Keeping You Current") this week.

The schedule you develop will seem like a complete work of fiction up until the time your customer fires you for not meeting it. David Akin

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

When faced with an intractable bug... wash your hands!

A new(ish) site for industrial IOT is interesting.

Percepio, responding to engineers working at home during the Coronavirus pandemic, are offering two months free use of their Tracealyzer.

This month''s giveaway is a BBC Micro:bit Go Bundle.

Enter via this link.

In reply to the last Muse''s comments on reuse, Stuart Jobbins wrote:

John Carter commented:

Do you know what this code does?

  sum = sum * 10 - ''0'' + (*p++ = getchar());   

Don''t feel bad; no one does. As the standard says: "the actual increment of p can occur at any time between the previous sequence point and the next sequence point (the ;), and the call to getchar can occur at any point prior to the need of its returned value."

The standard lets compilers evaluate parts of an expression with all of the discipline of sailors at a bar on shore leave. It says "The order in which subexpressions are evaluated and the order in which side effects take place, except as specified for the function-call (), &&, ||, ?:, and comma operators" is unspecified.

To give us predictability the standard has the concept of sequence points, To quote the standard again: "The presence of a sequence point between the evaluation of expressions A and B implies that every value computation and side effect associated with A is sequenced before every value computation and side effect associated with B."

A sequence point occurs on:

• A call to a function, after the arguments have been evaluated
• The end of the first operand of the following operators:
  • ? logical AND &&
  • ? logical OR ||
  • ? conditional ?
  • ? comma ,
• The end of a full declarator (a declarator is the part of a declaration (definition or initialization) that is not the type)
• The end of a full expression:
  • ?? an initializer
  • ?? the expression in an expression statement (e.g., p(0);)
  • ?? the controlling expression of a if, switch, while or do
  • ?? each of the expressions of a for statement
  • ?? the expression in a return statement
• Immediately before a library function returns
• After the actions associated with each formatted input/output function conversion specifier
• Immediately before and after each call to a comparison function (e.g., qsort), and also between any call to a comparison function and any movement of the arguments

So be very wary of expressions like:

b = a + a++;

The value of b is going to depend on the order of evaluation of the expression.

Because logical ANDs cause sequence points an expression like:

a++ && a++

is perfectly legitimate. But my preference is to avoid such constructs that may confuse the unwary.

Do note that adding a Lisp-like swarm of parenthesis will not change the compiler-generated order of evaluation.

MISRA rule 13.2 addresses sequence points: "The value of an expression and its persistent side effects shall be the same under all permitted evaluation orders." (Emphasis in original).

CERT rule EXP30-C is more succinct: "Do not depend on order of evaluation between sequence points. "

Do you know C operator precedence rules? I don''t. At least, not well. Most of us stumble when queried about this. In fact, in Developer Beliefs About Binary Operator Precedence by Derek M. Jones shows that even experienced developers strike out when asked which of C''s 18 operators is evaluated before the others:

John Watson was surprised when he found out that Sherlock Holmes didn''t know that the Earth revolves around the sun. Holmes said "What the deuce is it to me? You say that we go round the sun. If we went round the moon it would not make a pennyworth of difference to me or to my work." I feel the same way about precedence rules. They are arcane and uninteresting. Much better: use parenthesis when the order of evaluation isn''t crystal clear.

Clarity is our goal.

iRobot is launching a robot programming education initiative. They have (free!) on-line tools which let students develop and simulate robots. Looks like fun!

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

Do you know Cobol? See this.

These jokes are archived here.

Tis the dream of each programmer,
Before his life is done,
To write three lines of APL,
And make the damn thing run.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Replies to "Is Assembly Language Dead as a Dodo?
• On Taking Charge
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive. If you include the phrase "embedded muse" in the subject line your email will wend its way to me.

Bad news for router users: a recent study looked at 127 routers (of the style used in homes) and found all have security problems. Major security problems. 50 of them have hard-coded passwords and the like. All run Linux, generally older versions. This raises the question: How does one support a product, especially as it ages? A big OS like Linux will always have bugs. How long should a vendor provide support for a product?

Phil Matthews poses this interesting question: What childhood experiences influenced you to become an engineer? For me it was probably that my dad was a mechanical engineer working in the space program, though how I got interested in electronics I don''t remember. I did build a crystal radio at 8 or 9 years of age, and always liked building "forts" and other items. I got a ham license at 16 and had a lab in the basement from an early age. My interest has always been in building stuff, an interest that has never waned. What was your experience?

One should expect that the expected can be prevented but that the unexpected should have been expected. Unknown author

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

There has been some discussion of inexpensive PCB vendors in the Muse lately. I recently found that Digikey has a PCB service that gets quotes from a half-dozen vendors here.

Chris Bucsko wrote:

Joe Vandzura responded to an article in the last Muse:

Jake Michael won last month''s Binho Nova Multi-Protocol USB Host Adapters.

This month''s giveaway is a LA104 mini logic analyzer. I reviewed it in Muse 406.

Enter via this link.

A ton of people replied to this article in the last Muse. Most said they write little Assembler code but read a fair amount. And there was some discussion about whether "Assembler" should be capitalized. I bow to the general consensus that it should be... and wonder where the word came from. Here are a handful of reader replies.

Dan Daly wrote:

From Chris van Niekerk:

James Thayer, like your editor, goes back to the early days of this field:

Tom Mazowiesky often writes and sent this:

Steve Paik''s insights are worthwhile:

Tony Garland dives into Assembler when needed:

Daniel McBreaty bridges comments on Assembler with another Muse article on taking charge/continuous learning:

Plenty of email also came in regarding the article in the last Muse about Taking Charge. Some highlights follow.

Neil Cherry wrote:

Steve Paik sent this:

Finally, a reader who wishes to remain anonymous poses an interesting question:

From Leon Bazdar:

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

A bug in the code is worth two in the documentation.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Knowledge Capture
• Efficacy of Bug-Finding Tools
• Hardware, Software or Hardware/Software?
• Failure of the Week
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

A recent EEWorldonline article states "TIRIAS Research forecasts that 98% of all edge devices will use some form of machine learning/artificial intelligence by 2025." In my opinion, this AIn''t gonna happen anytime soon.

A long but fascinating history of the transistor is here.

Jack''s latest blog: Networking Did Not Start With the IoT

58% of innovations fail, except for those originated by top management, which fails at a rate of 74%. - The Economist

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Recent Muses have had comments about using state machines. But what about tools? I have heard good things about IAR''s Visual State. Michele Corradin wrote that they are using Yakindu. I had not heard of it but he says they''ve had great success with it and the web site looks interesting. Does anyone have any experience with either of these tools?

Firia Labs is donating one of their CodeBots (a Python-enabled robot with a host of sensors) for this month''s giveaway.

Enter via this link.

In Knowledge Capture and Management for Space Flight Systems author John Goodman argues that, particularly in aerospace projects, much vital project knowledge exists only in engineers'' heads. Imperfect memories, career changes and retirement means this information goes missing, often to the detriment of a current program, or a future one that could have drawn on what might have been painfully-acquired experience.

Goodman relates that early space efforts took just a few years to complete; Project Mercury ran just over four years from contract award till final flight, Gemini took 5 years. The Shuttle program spanned 38 years, an engineer''s entire career. How many original engineers were associated with that system over the entire program?

Just two years after contract award the P-38 fighter had made its first flight, and was introduced to the service two years later. The F-4 flew three years after the contract. But the F-22, despite the contractor having built a flying prototype before winning the competition, required an additional six years for the first flight of the contracted aircraft, and a further eight before the first production model made it to the Air Force. The F-35''s schedule appears unbounded. These are more ambitions projects than the primitive, by today''s standards, P-38 and F-4, so comparisons are not completely fair. But it''s clear that both the complexity of the programs and the engineering duration means preserving design decisions is increasingly important.

In the commercial world the problems are little different. I''ve often wondered how the automotive companies preserve corporate knowledge. Ford learned at great expense how to position gas tanks due to the Pinto fiasco, but do young engineers who weren''t even alive at the time know those lessons learned so painfully, so long ago?

Middle-aged readers who can''t find their keys understand that even a single person''s memory is pretty unreliable. I figure the brain is a FIFO queue. When it fills, around age 50, something has to go when one learns something new.

So how do we preserve knowledge? Traditional tools aren''t up to snuff. Goodman suggests the use of "Brain Books," more or less engineering notebooks used by developers to record design decisions, test results, and the like. He cites a case where a Brain Book saved a project. But hardcopy is tough to search and poor penmanship hinders understanding.

I keep a notebook in my woodworking shop to record things I learn that I''ll surely forget. There''s no computer in that room and the volume of data is very low, so that works well. A variety of pretty organized files on my PCs keep resource notes, interesting quotes, facts and the like. But perhaps a wiki is the ultimate Brain Book repository, and reading Goodman''s article has given me motivation to set up a personal Brain Book on a wiki. We''re drowning in information and, unless preserved and categorized, an excess of data is the same as no data.

Microsoft Notes is an interesting system for recording ideas. I''ve had pretty good success with it. But not trusting any cloud provider to be around forever I do download the data from time to time as a backup.

How do you preserve your design notes and other important bits of information that exist outside of normal tool flow?

I stumbled across a 2005 paper by Stefan WagnerJan J?rjensClaudia KollerPeter Trischberger called Comparing Bug Finding Tools with Reviews and
Tests. The key takeaway: bug-finding tools are useful, but they don''t replace tests or code inspections/reviews.

Now, the study was about Java code and the bug-finding tools were open source, not the commercial versions now available. Here are their conclusions:

Interestingly, code reviews were much more effective at finding the worst bugs than bug-finding tools. Tests were not as good as reviews, but were far better than the tools for these impactful bugs. The tools shine at detecting the least consequential errors. Tests found none of these.

My takeaway: use the tools to cheaply find some errors. Apply reviews, and then tests. Think in terms of a series of filters, as no one approach will give quality code.

When the embedded field was young most developers wore at least two hats, that of firmware and hardware designer. Some also came up with analog, power and maybe even RF circuits. A single engineer might be responsible for most of the electronics in a product. Sometimes the same person would even lay out the PCB and assemble prototypes.

That was a lot of fun!

But systems were simpler back then. 8 KB of code was a lot, and SMT hadn''t turned board assembly into an art practiced only by the caffeine-deprived. 12 bit A/D converters were expensive, which meant those of us working in the analog domain had, in general, smaller noise issues than today''s systems that measure femtoamps. Low clock rates dodged Maxwell''s Laws and programmable logic was merely a dream.

Though a huge number of embedded systems are still small, often containing a bare sliver of silicon embodied in a small PIC or other minimal MCU, big applications are common. Firmware can consume hundreds of thousands or even millions of lines of code. Hardware people design with multi-hundred pin chips running with tiny noise margins, while wrestling with EMI and ESD requirements that were unheard-of decades ago. It''s natural that engineers started to specialize, at first to divide the work into manageable chunks, and later because of the specialized knowledge required to deal with advancing technology and requirements.

Today an increasing number of embedded systems developers live in their own domain, with little knowledge of other aspects of the system they''re building. That''s a natural part of the progression of knowledge: as a technology or science advances people''s expertise at their own arcane area grows while their breadth narrows. But it''s interesting how that, in a single generation, we''ve progressed to the point where many firmware developers can''t answer simple questions about their hardware platform. In casual discussions with engineers I''ll often ask about their system''s clock rate, or supply voltage, or any of a number of hardware issues. Quite a few can''t answer; in some cases the software people don''t know what the target processor is.

The reverse is often true as well: ask a hardware developer about the firmware''s data structures or which RTOS is used and you might get a blank look. The FPGA whizzes are sometimes clueless about classes and analog gurus can be completely ignorant about PWM control.

I''ve worked with groups that use embedded Windows or Linux simply to minimize their need for firmware people, who can be hard to find and a lot more expensive than Visual Studio programmers. These folks may have little idea about the physical instantiation of their product.

This natural and necessary specialization has created a new job category, which I call the systems person. The person who does have a deep understanding of all of these interrelated issues, who can sling some VHDL and C while probing a low-noise preamp. Sometimes they''re the computer experts who can tie together the digital hardware and code to interface with the RF crowd to help the entire team solve a challenging multipath problem.

These systems people are invaluable and increasingly hard to find. At times I wonder if they are a vanishing breed. But other fields have managed to keep systems folks with armies of specialists. Medical GPs, for instance, look at the whole body.

What is your experience? Are systems folks a dying breed?

I took this picture at an airport in Norway. The word "Bug" plastered on the plane didn''t leave a feeling of confidence:

Need a custom PCB? Here''s a site that will quote boards instantly. Change a parameter (say from 2 layer to 4 layer) and the price comes up in under a second. And, small boards are dirt cheap.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

From Dan Daly, who writes: Seen in Silicon Valley, at a park along a bicycling/hiking trail. There were many casts in the pond, but the most visible, for obvious reasons, were (float).

(The jokes section of my website now has 200 computer-related bits of humor.)

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Design by Contract
• RTOS... or a State Machine?
• The (Scope) New Math
• Failure of the Week
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Jack''s latest blog: Is the Future All Linux and Raspberry Pi?

We should continue to collect and analyze data on how, when, where and why people make software errors in order to find means to avoid people making these errors. Barry Boehm

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Sometimes we have very limited I/O for debugging. One approach I have used is to generate Morse code messages to a speaker. Here''s a program that will do this. This site gives implementations in 62 languages; the C version is very clunky, but the Forth version is intriguing. A couple of years ago a reader wrote that he did this, but the beeping drove other engineers crazy!

Max Maxfield has a 9 (!) part series on switch debouncing. Part 9 is here, and that page has links to the earlier parts as well.

Firia Labs is donating one of their CodeBots (a Python-enabled robot with a host of sensors) for this month''s giveaway.

Enter via this link.

Design by Contract (DbC) is an idea that I believe was first implemented in the Eiffel¹ programming language. Ada-2012 included the notion. Essentially, the idea is the components of a program have contractually-governed relationships to each other. A function, for instance, can only accept parameters that meet certain constraints (e.g., code that computes a rocket''s velocity might only accept positive values).

DbC usually includes three types of contracts:

• Preconditions are those that must be satisfied when a function is called. An example: a parameter that will be used in a square root (in the Real number domain) must be positive.
• Postconditions are those that must be satisfied as a function exits. A postcondition is a check on the validity of the function''s code.
• Invariants are those conditions which remain unchanged throughout the execution of the function.

I''ve tried to torture C via macros and other convoluted processes into supporting DbC, without much success. Assertions are a poor-person''s crude-DbC alternative, and I''ve written extensively about better ways of using those here.

I recently stumbled across a nice writeup on using DbC in Ada. Recommended, even for C developers, as that page has wisdom worth pondering.

1. Eiffel is an object-oriented programming language that no one in the embedded industry uses. It is described in Betrand Meyer''s Object Oriented Software Construction, which is probably the best book written about OO design. Though it teaches Eiffel most of the book''s ideas scale directly to C++. Meyer is a brilliant writer.

Tony Garland responded to my comments in the last issue about using an RTOS:

Miro''s book is indeed excellent. I did find it a bit heavy going at times, but worth the effort.

Last year I was using an opto-interrupter to measure the speed of a rotating wheel. The pulses went to an oscilloscope, and by measuring their period I could derive RPM. It occurred to me that most scopes today can do some math, though that is limited to a handful of pre-defined operations. Wouldn''t it be nice if the instrument could do any sort of math? That is, the user could type in custom formulas. Then one could observe things like RPM directly. I mentioned this to Siglent; they said they''d look into it. Of course I forgot all about it.

Recently I upgraded the firmware in my Siglent SDS5034X and found that the MATH button operates normally, but a little clicking around brings up this screen:

That''s right - the scope can accept many kinds of user-defined math functions. The middle array of soft-buttons select the source: channels 1 through 4 (C1-C4), zoomed subsets of those channels (Z1-Z4) and two user-defined functions. The buttons on the left enable common functions and the grouping on the right includes numbers and the basic arithmetic functions.

In the formula bar (the arrow points to this) I''ve entered (∫(channel 3 times channel 2)) *√ (channel 1).

The computation''s result appears as another trace on the screen. As one moves the cursor a box displays the instantaneous value at the cursor position.

Remember when we used to count divisions on the screen and convert those to volts or time? That''s long gone. So too, it now seems, with figuring some function manually.

Here''s a gyro that''s 10,000 times more accurate than the one in your mobile phone, but is only 10x more expensive. Though not available yet as a product, it''s pretty amazing. This is a golden age of sensors.

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• A Buck50 Logic Analyzer
• Pico 2207B USB Scope Review
• Failure of the Week
• This Week''s Cool Product
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive. If you include the phrase "embedded muse" in the subject line your email will wend its weighty way to me.

Are you comfortable designing with FPGAs? I think this is a skill that is going to be highly marketable. This article shows these parts are becoming much more common. One note: C programmers may find Verilog or VHDL a little odd. Unlike C code, where each statement executes in succession, in general in an HDL everything is running at the same time. In fact, it often doesn''t matter which order the HDL statements are coded.

We''re absolutely going to deliver on time on this project. We were too overconfident on the last one.

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Mark Rubin has what looks like a handy utility:

Bob Paddock sent this link to the definitive computer industry glossary of terms.

Phil Koopman, noted Carnegie Mellon professor, has made his embedded software course material available online. The course covers software quality, safety, and security. For those who don''t do YouTube, videos and handouts are here as well.

David Thomas posed an interesting question: Does anyone have tips on keeping a workbench reasonably tidy? I find that it''s not long before scope probes and other leads are snaking all over the bench, tools scattered around, and components piled up. Much as I admired the man, I try to avoid a Jim Williams'' like bench:

Murdo McLeod won the BattLab-One tool in last month''s contest.

This month''s giveaway is a Pico 2207B 70 MHz USB scope, which I review later in this Muse.

Enter via this link.

Need a logic analyzer but Covid has depleted your cash reserves? Can you shell out $1.50?

Mark Rubin has created the "buck50" logic analyzer which has a UI that runs under Linux/Unix and maybe (this is untested) a Mac.

This isn''t a product that he sells. It''s completely open source. Source code and documents are here. And it is extremely well documented.

The hardware is just a "Blue Pill" STM32F103 eval board, widely available for a dollar-fifty. You buy the board, download the firmware, and install the application. You''ll want to solder a header to the board for test probes.

Features include:

• 8 channel, 6+ MHz logic analyzer
  
  • Approx. 5K sample buffer depth
    
  • Samples stored only at signal edges for efficient memory usage
    
  • Units may be ganged for increased number of channels
    
  • Complex triggering via user-defined state machine supporting combinations of sequential ("A then B then C") and logical-OR ("A or B or C") conditionals
    
  • Output to VCD and other file formats for export to waveform viewing software
    
• Live monitoring and logging of digital, analog, USART (sync/async), SPI (MOSI/MISO), and I2C (master/slave/TX/RX) data
  
• Simple dual-channel approx. 1 MHz digital storage oscilloscope, approx. 5K sample buffer depth (10K if single channel)
  
• 3 channel digital pulse train generator with user-defined frequency and per-channel duty cycle and polarity
  
• Bidirectional bridge/converter from USART/UART (async/synchro), SPI (master/slave), or I2C ... to USB ... to host terminal, UNIX socket, or UNIX pty device file
  
• 8-bit parallel output counter (binary or gray code)
  
• Host terminal ASCII or binary input data to 8-bit parallel output
  
• Firmware written in a combination of C++ and ARM Thumb-2 assembly code, with several non-standard hacks^H^H^H^H^Htechniques of possible general interest to advanced software developers
  
• Python host driver program with comprehensive inline help system and usability features
  

As Mark says: "At $1.50 for the hardware plus $0.00 for the firmware and software, if there''s a competition for the world''s cheapest logic analyzer I want to enter it. Please note this is very much a beta release and bug reports are welcome."

Way cool!

Pico Technology is one of the pioneers in USB oscilloscopes. They sell a huge range of test instruments and related gear; their website, though frustratingly slow, is worth perusing. The company sent me their 2207B MSO, a 2 channel 70 MHz scope with a 64 MS buffer, and 1 GHz sample rate. It collects 80,000 waveforms per second.

Though I tested this using their Windows application, they have beta versions for Linux and Macs.

The unit comes nicely boxed with an excellent full color poster showing the UI, with annotations describing how each element works. It''s too big to reproduce here but I commend Pico for this "lagniappe" (a Louisiana term meaning a bonus gift). Also included is a soft carrying case, two scope probes, and digital logic probes. A quick start guide is included, but you''ll want to download the 239 page user manual from their web site.

If you didn''t know anything about the company, you''d quickly infer that they''ve been in the scope business for some time. The manual, in PDF form and a help file, is extensive and very complete. And the scope simply works. There''s no fiddling with USB setups. No connection hiccups.

Pico UI. Click on image for a bigger version.

The UI is complete and totally intuitive. Channel control, horizontal and vertical, is on the top ribbon. Triggering and measurements on the bottom. Channel A is in blue; on the left side of the screen the voltage levels are shown. Channel B''s voltages, in red, are on the right.

I''ve added an arrow to show the trigger position, denoted by a yellow dot. You can change this by simply dragging that dot to any position on the waveform.
Trigger is extensive and the display is super-intuitive. Here are the options:

In the previous picture I added peak-to-peak and frequency measurements, which are shown on the bottom of the screen. I counted 19 of these which include two kinds of RMS and pulse counts. But those tiny numbers sure are hard to see, aren''t they? The UI, though, allows one to select any font size for these. For instance:

As is common on digital scopes, there''s a zoom feature to narrow which part of the buffer is shown. Unusually, a "picture in a picture" shows a really nice image of where in the buffer the zoomed display is presented. It''s easy to grab the cursor in that small image to drag the zoomed display around.

I was disappointed that you can only select vertical and horizontal ranges using the usual 1-2-5 sequence until discovering that the channel selection button lets you type any range. And in that menu for the vertical signals there''s a bandwidth limit option, which is also freely selectable - you''re not limited to a few canned bandwidths.

The spectrum analyzer mode gives a remarkably accurate and clear display. The following shows a quite pure 10 MHz sine wave. Note the frequency labels below the waveform. Though at the far bottom left the scale is shown in kHz, I would prefer that the axis be labeled. Instead of the bland "1000" shown, it would be better to show "1000 kHz".

23 serial protocol decoders are included, and each can be set up using either the analog or digital channels. Also supported: mask testing, where one defines a "legal" waveform and an alarm or trigger occurs when the sampled wave falls out of the legal range.

Like most scopes, the Pico will do math. For instance, one can display a waveform that represents A+B. Unlike most, you can create your own equations. This is one example:

Not long ago I asked Siglent to add this feature to their scopes, which they did. I had been working on a system with rotating components. My report had screen shots showing RPM indirectly, as a frequency measurement. How much better to show actual RPM!

The model tested has 16 digital channels. These are arranged in two groups, and the threshold voltages can be set to any value for each group.

Finally, the unit includes an arbitrary waveform generator that will produce the usual signals up to 1 MHz with 12 bits of resolution. Nothing ground-breaking there. Want a custom wave? A single click brings up a waveform editor which lets you create anything. This is the easiest-to-use waveform creator I''ve come across... and it''s downright fun to use. The following took about three seconds to build:

The 2000 series of PicoScopes, of which this unit is a member, ranges from 10 MHz, 2 analog channels, 100 MS/S sample rate and no digital channels for $139, to 100 MHz, 4 analog, 128 MS memory for $1125. The unit I tested sells for $769. More info here.

Bottom line: This is the best USB scope I''ve tested.

From Martin Grill:

Want to play with AI? Google is offering their "AIY" (Artificial Intelligence Yourself) kits. Currently available are kits for playing with AI for voice and for vision. Looks like fun!

Note: This section is about something I personally find cool, interesting or important and want to pass along to readers. It is not influenced by vendors.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

TOOLS EXPLAINED

DRILL PRESS : A tall upright machine useful for suddenly snatching flat metal bar stock out of your hands so that it smacks you in the chest and flings your beer across the room, denting the freshly-painted project which you had carefully set in the corner where nothing could get to it.

WIRE WHEEL : Cleans paint off bolts and then throws them somewhere under the workbench with the speed of light. Also removes fingerprints and hard-earned calluses from fingers in about the time it takes you to say, ''Oh sh*t''

CHOP SAW : A portable cutting tool used to make studs too short.

PLIERS : Used to round off bolt heads. Sometimes used in the creation of blood-blisters.

BELT SANDER : An electric sanding tool commonly used to convert minor touch-up jobs into major refinishing jobs.

HACKSAW : One of a family of cutting tools built on the Ouija board principle... It transforms human energy into a crooked, unpredictable motion, and the more you attempt to influence its course, the more dismal your future becomes.

VISE-GRIPS : Generally used after pliers to completely round off bolt heads. If nothing else is available, they can also be used to transfer intense welding heat to the palm of your hand.

OXYACETYLENE TORCH : Used almost entirely for lighting on fire various flammable objects in your shop. Also handy for igniting the grease inside the wheel hub out of which you want to remove a bearing race.

TABLE SAW : A large stationary power tool commonly used to launch wood projectiles for testing wall integrity.

HYDRAULIC FLOOR JACK : Used for lowering an automobile to the ground after you have installed your new brake shoes, trapping the jack handle firmly under the bumper.

BAND SAW : A large stationary power saw primarily used by most shops to cut good aluminum sheet into smaller pieces that more easily fit into the trash can after you cut on the inside of the line instead of the outside edge.

TWO-TON ENGINE HOIST : A tool for testing the maximum tensile strength of everything you forgot to disconnect.

PHILLIPS SCREWDRIVER : Normally used to stab the vacuum seals under lids or for opening old-style paper-and-tin oil cans and splashing oil on your shirt; but can also be used, as the name implies, to strip out Phillips screw heads.

STRAIGHT SCREWDRIVER : A tool for opening paint cans. Sometimes used to convert common slotted screws into non-removable screws and butchering your palms.

PRY BAR : A tool used to crumple the metal surrounding that clip or bracket you needed to remove in order to replace a 50 cent part.

HOSE CUTTER : A tool used to make hoses too short.

HAMMER : Originally employed as a weapon of war, the hammer nowadays is used as a kind of divining rod to locate the most expensive parts adjacent the object we are trying to hit.

UTILITY KNIFE : Used to open and slice through the contents of cardboard cartons delivered to your front door; works particularly well on contents such as seats, vinyl records, liquids in plastic bottles, collector magazines, refund checks, and rubber or plastic parts. Especially useful for slicing work clothes, but only while in use.

ADJUSTABLE WRENCH: aka "Another hammer", aka "Crescent Wrench". Commonly used as a one size fits all wrench, usually results in rounding off nut heads before the use of pliers. Will randomly adjust size between bolts, resulting in busted buckles, curse words, and multiple threats to any inanimate objects within the immediate vicinity.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Replies to "Is Assembly Language Dead as a Dodo?
• On Taking Charge
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

Tip for sending me email: My email filters are super aggressive. If you include the phrase "embedded muse" in the subject line your email will wend its way to me.

Bad news for router users: a recent study looked at 127 routers (of the style used in homes) and found all have security problems. Major security problems. 50 of them have hard-coded passwords and the like. All run Linux, generally older versions. This raises the question: How does one support a product, especially as it ages? A big OS like Linux will always have bugs. How long should a vendor provide support for a product?

Phil Matthews poses this interesting question: What childhood experiences influenced you to become an engineer? For me it was probably that my dad was a mechanical engineer working in the space program, though how I got interested in electronics I don''t remember. I did build a crystal radio at 8 or 9 years of age, and always liked building "forts" and other items. I got a ham license at 16 and had a lab in the basement from an early age. My interest has always been in building stuff, an interest that has never waned. What was your experience?

One should expect that the expected can be prevented but that the unexpected should have been expected. Unknown author

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

There has been some discussion of inexpensive PCB vendors in the Muse lately. I recently found that Digikey has a PCB service that gets quotes from a half-dozen vendors here.

Chris Bucsko wrote:

Joe Vandzura responded to an article in the last Muse:

Jake Michael won last month''s Binho Nova Multi-Protocol USB Host Adapters.

This month''s giveaway is a LA104 mini logic analyzer. I reviewed it in Muse 406.

Enter via this link.

A ton of people replied to this article in the last Muse. Most said they write little Assembler code but read a fair amount. And there was some discussion about whether "Assembler" should be capitalized. I bow to the general consensus that it should be... and wonder where the word came from. Here are a handful of reader replies.

Dan Daly wrote:

From Chris van Niekerk:

James Thayer, like your editor, goes back to the early days of this field:

Tom Mazowiesky often writes and sent this:

Steve Paik''s insights are worthwhile:

Tony Garland dives into Assembler when needed:

Daniel McBreaty bridges comments on Assembler with another Muse article on taking charge/continuous learning:

Plenty of email also came in regarding the article in the last Muse about Taking Charge. Some highlights follow.

Neil Cherry wrote:

Steve Paik sent this:

Finally, a reader who wishes to remain anonymous poses an interesting question:

From Leon Bazdar:

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

A bug in the code is worth two in the documentation.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• The Ruse Programming Language
• Debugging Embedded and Real-Time Systems
• The Binho Nova Multi-Protocol USB Host Adapter
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

The Embedded Muse now goes to over 40,000 subscribers.

Let''s disagree without being disagreeable - Laurene Powell Jobs

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Gerald DeSantis wrote:

September''s giveaway is courtesy of Binho, and is one of their Nova Multi-Protocol USB Host Adapters with accessories (reviewed later in this issue).

Enter via this link.

James Munns is an embedded systems engineer, and Managing Director at Ferrous Systems GmbH in Berlin, Germany. James has experience in a variety of domains, including safety critical avionics, robotics, and gas detection, as well as connected IoT devices.

Ferrous Systems GmbH is the largest software consultancy worldwide focused on the Rust Programming Language. Ferrous Systems provides services in advising, development, and training, and has experience developing embedded systems and systems level software in Rust.

Arnold Berger has a new book out and his publisher graciously sent me a copy. Debugging Embedded and Real-Time Systems is a worthy addition to the literature.

Arnie teaches embedded systems at the University of Washington, and I was stunned to read in this volume that the Circuits 1 students get a breadboard and they actually construct and test circuits. When I went to college circuit theory was entirely theory. A later lab class was a joke of playing with multimeters. So while the pundits complain of education going to hell in a handbasket, my hat is off to the U of W for a class rich in practicum.

This book is really aimed at students, and I think it should be a part of the pedagogy of every CS and EE department. Arnie pushes the idea of using a disciplined approach to debugging. Over the decades I''ve seen (and have participated in) too many shotgunning sessions. Sometimes these work though more often much time is wasted. And sometimes a "fix" is masking tape over the wrong issue, creating more havoc later.

The book is about debugging embedded systems; not hardware, not software, but both the hardware and the software, which is means it''s written assuming the reader is familiar with both. Though it does zero in on each in isolation at times, it''s a broadly-focused work that should appeal to all embedded practitioners, especially those at the intersection of the two disciplines.

When it does come to software Arnie''s first advice is to write bug-free code. For that he presents a strategy. He admits we''ll never be perfect, but the unfortunate truth is that many of us write firmware figuring on long sessions with the debugger instead of employing every means possible to avoid errors.

The same is true for hardware, and he presents a number of ideas for designing the hardware correctly. Some of this will be obvious to old-timers; for newbies, not so much so. An example: Use pullups on the bus to implement a NOP so you can run tests without memory loaded. I prefer a software interrupt instruction, when possible, as that will create write cycles as well. A scope or logic analyzer will show all of the resulting bus activity (if any!).

Chapter five is a good overview of the tools we use. I was surprised that the oscilloscope didn''t get its own section, though many references are made to that tool in other places.

One tidbit I''ve been advocating for years: Keep a debugging log. Have you ever chased a bug for days and find yourself repeating a test, simply because you forgot you ran it? I have. A log documents the process in a disciplined way.

Some concepts will be new to even many experienced hands. Ever hear of SIMICS? It''s a fascinating tool I''ve been following for some years, and I''m told Intel (which owns it) uses it to evaluate most of their products.

Arnie highly praises going to the Embedded Systems Conference. Alas, that show is a shadow of its former self. Once that advice was spot-on. Today I recommend Embedded World (in Germany) as that gathering is huge and dynamic.

Arnie presents lots of anecdotes from his career which makes for fun reading.

Debugging Embedded and Real-Time Systems is a quick and easy read that condenses a lot of useful information into 285 pages. Every student of our profession should read it and embrace the concepts. Experienced readers will pick up useful nuggets as well. If there''s one nit to pick it''s the $80 price tag, a price that''s becoming all too common for tech books. I sheepishly admit my own books are also way too expensive. The Embedded Systems Dictionary, which I co-wrote with Mike Barr, is currently $143 on Amazon. Perhaps that explains its pathetic sales!

Binho sent me one of their Nova Multi-Protocol USB Host Adapters. I wasn''t quite sure what such a device is or does, so went to the company''s web page, which didn''t shed enough light for a dullard like me to understand. A lot of web sites today are very pretty yet they don''t clearly state what the company is selling. We used to talk about having an elevator speech: a 15 second explanation of what you do. Too many web sites neglect that.

However, this page gives an excellent description of the Nova. It is basically an interface between a PC''s USB port and common interfaces found in embedded systems. In particular, the device can interface to devices that communicate via I2C, SPI, UART, 1-WIRE, and SWI. On one side of the device is a USB connector; on the other is a flat ribbon cable with power and ground signals, plus five pins for connecting to these interfaces.

Pinouts are:

Why would you want to do this? Early in development it may be easier to develop prototype code for a device driver on a PC rather than in the less-friendly cross-development environment. The Nova can also enable automatic testing of devices in an embedded system. And it can be a nice way to download code to flash memory in production.

Steve Jobs taught us that beauty can be combined with functionality, and a few makers of test equipment (notably Saleae''s logic analyzers) have profited from that lesson. Binho, too, provides a unit in a slim, elegant metal package, which in turn is shipped in a quite attractive hard case.

A GUI interface named "Mission Control" is available for Windows, Linux and Macs. It''s self explanatory. For example, to control an SPI device:

An alternative mode lets you use these 5 pins as GPIO connections or A/D inputs. One can be a DAC. Four can be PWMs with controllable frequency and duty cycle.

The Mission Control GUI is a clean and easy way to interface to the device, but the real power comes from Python libraries that let you write code to send and receive data at speed via the Nova to your hardware. I did not try this, but the web site contains numerous examples, all too long to list here.

You can also drive the Nova from a command terminal like Tera Term.

Binho''s web site has very clear step-by-step instructions for installing and using the device. I do wish they were available as a single PDF as I like to keep user manuals on my disk, and prefer random access access rather than stepping through web pages.

At $149 the Nova is a very affordable way to tie a PC to an embedded system using any of these serial interfaces. The GUI is simple and powerful. If you''re willing to write some Python code, there''s really no limit to the control you could exercise over a device.

Another heart rate fail to complement the one in the last Muse. According to the CDC the max heart rate is 220-(age in years) which means this poor person is negative four billion years old. I guess that means he or she will be born about the time the sun turns into a cinder. I suspect at that time we''ll be programming 8051s in C0xFFFFFFFF.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

From Phil Matthews:

Everybody trusts a hardware engineer, except themselves.
Nobody trusts a software engineer, except themselves.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• The Ruse Programming Language
• Debugging Embedded and Real-Time Systems
• The Binho Nova Multi-Protocol USB Host Adapter
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

The Embedded Muse now goes to over 40,000 subscribers.

Let''s disagree without being disagreeable - Laurene Powell Jobs

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Gerald DeSantis wrote:

September''s giveaway is courtesy of Binho, and is one of their Nova Multi-Protocol USB Host Adapters with accessories (reviewed later in this issue).

Enter via this link.

James Munns is an embedded systems engineer, and Managing Director at Ferrous Systems GmbH in Berlin, Germany. James has experience in a variety of domains, including safety critical avionics, robotics, and gas detection, as well as connected IoT devices.

Ferrous Systems GmbH is the largest software consultancy worldwide focused on the Rust Programming Language. Ferrous Systems provides services in advising, development, and training, and has experience developing embedded systems and systems level software in Rust.

Arnold Berger has a new book out and his publisher graciously sent me a copy. Debugging Embedded and Real-Time Systems is a worthy addition to the literature.

Arnie teaches embedded systems at the University of Washington, and I was stunned to read in this volume that the Circuits 1 students get a breadboard and they actually construct and test circuits. When I went to college circuit theory was entirely theory. A later lab class was a joke of playing with multimeters. So while the pundits complain of education going to hell in a handbasket, my hat is off to the U of W for a class rich in practicum.

This book is really aimed at students, and I think it should be a part of the pedagogy of every CS and EE department. Arnie pushes the idea of using a disciplined approach to debugging. Over the decades I''ve seen (and have participated in) too many shotgunning sessions. Sometimes these work though more often much time is wasted. And sometimes a "fix" is masking tape over the wrong issue, creating more havoc later.

The book is about debugging embedded systems; not hardware, not software, but both the hardware and the software, which is means it''s written assuming the reader is familiar with both. Though it does zero in on each in isolation at times, it''s a broadly-focused work that should appeal to all embedded practitioners, especially those at the intersection of the two disciplines.

When it does come to software Arnie''s first advice is to write bug-free code. For that he presents a strategy. He admits we''ll never be perfect, but the unfortunate truth is that many of us write firmware figuring on long sessions with the debugger instead of employing every means possible to avoid errors.

The same is true for hardware, and he presents a number of ideas for designing the hardware correctly. Some of this will be obvious to old-timers; for newbies, not so much so. An example: Use pullups on the bus to implement a NOP so you can run tests without memory loaded. I prefer a software interrupt instruction, when possible, as that will create write cycles as well. A scope or logic analyzer will show all of the resulting bus activity (if any!).

Chapter five is a good overview of the tools we use. I was surprised that the oscilloscope didn''t get its own section, though many references are made to that tool in other places.

One tidbit I''ve been advocating for years: Keep a debugging log. Have you ever chased a bug for days and find yourself repeating a test, simply because you forgot you ran it? I have. A log documents the process in a disciplined way.

Some concepts will be new to even many experienced hands. Ever hear of SIMICS? It''s a fascinating tool I''ve been following for some years, and I''m told Intel (which owns it) uses it to evaluate most of their products.

Arnie highly praises going to the Embedded Systems Conference. Alas, that show is a shadow of its former self. Once that advice was spot-on. Today I recommend Embedded World (in Germany) as that gathering is huge and dynamic.

Arnie presents lots of anecdotes from his career which makes for fun reading.

Debugging Embedded and Real-Time Systems is a quick and easy read that condenses a lot of useful information into 285 pages. Every student of our profession should read it and embrace the concepts. Experienced readers will pick up useful nuggets as well. If there''s one nit to pick it''s the $80 price tag, a price that''s becoming all too common for tech books. I sheepishly admit my own books are also way too expensive. The Embedded Systems Dictionary, which I co-wrote with Mike Barr, is currently $143 on Amazon. Perhaps that explains its pathetic sales!

Binho sent me one of their Nova Multi-Protocol USB Host Adapters. I wasn''t quite sure what such a device is or does, so went to the company''s web page, which didn''t shed enough light for a dullard like me to understand. A lot of web sites today are very pretty yet they don''t clearly state what the company is selling. We used to talk about having an elevator speech: a 15 second explanation of what you do. Too many web sites neglect that.

However, this page gives an excellent description of the Nova. It is basically an interface between a PC''s USB port and common interfaces found in embedded systems. In particular, the device can interface to devices that communicate via I2C, SPI, UART, 1-WIRE, and SWI. On one side of the device is a USB connector; on the other is a flat ribbon cable with power and ground signals, plus five pins for connecting to these interfaces.

Pinouts are:

Why would you want to do this? Early in development it may be easier to develop prototype code for a device driver on a PC rather than in the less-friendly cross-development environment. The Nova can also enable automatic testing of devices in an embedded system. And it can be a nice way to download code to flash memory in production.

Steve Jobs taught us that beauty can be combined with functionality, and a few makers of test equipment (notably Saleae''s logic analyzers) have profited from that lesson. Binho, too, provides a unit in a slim, elegant metal package, which in turn is shipped in a quite attractive hard case.

A GUI interface named "Mission Control" is available for Windows, Linux and Macs. It''s self explanatory. For example, to control an SPI device:

An alternative mode lets you use these 5 pins as GPIO connections or A/D inputs. One can be a DAC. Four can be PWMs with controllable frequency and duty cycle.

The Mission Control GUI is a clean and easy way to interface to the device, but the real power comes from Python libraries that let you write code to send and receive data at speed via the Nova to your hardware. I did not try this, but the web site contains numerous examples, all too long to list here.

You can also drive the Nova from a command terminal like Tera Term.

Binho''s web site has very clear step-by-step instructions for installing and using the device. I do wish they were available as a single PDF as I like to keep user manuals on my disk, and prefer random access access rather than stepping through web pages.

At $149 the Nova is a very affordable way to tie a PC to an embedded system using any of these serial interfaces. The GUI is simple and powerful. If you''re willing to write some Python code, there''s really no limit to the control you could exercise over a device.

Another heart rate fail to complement the one in the last Muse. According to the CDC the max heart rate is 220-(age in years) which means this poor person is negative four billion years old. I guess that means he or she will be born about the time the sun turns into a cinder. I suspect at that time we''ll be programming 8051s in C0xFFFFFFFF.

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

From Phil Matthews:

Everybody trusts a hardware engineer, except themselves.
Nobody trusts a software engineer, except themselves.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.

You may redistribute this newsletter for non-commercial purposes. For commercial use contact info@ganssle.com. To subscribe or unsubscribe go here or drop Jack an email.

• Editor''s Notes
• Quotes and Thoughts
• Tools and Tips
• Freebies and Discounts
• Know Your Numbers
• Zone Triggering
• Software Element Out of Context
• Failure of the Week
• Jobs!
• Joke for the Week
• About The Embedded Muse

I''m told Embedded World, held this past February in N?rnberg, was a Coronavirused-shell of its former glory. The 2021 affair will be held in March. Hundreds of exhibitors have already signed up. And the Embedded Systems Conference in Silicon Valley is on in January. I admire the optimism of the promoters and exhibitors, but wonder if the conference world will be reshaped by the pandemic. In many disciplines the future is reasonably clear: history curricula changes slowly, in English literature grammar is nearly immutable. Electronics is different; it has always been hard to guess what we''ll see a couple of years hence, and we engineers have grown comfortable with this uncertainty. But today even the near-future a month or three out seems awfully murky. I hope things improve enough to encourage travel and that these important conferences are big successes. What are your thoughts about the future of in-person conferences? Will you be comfortable attending these, and will your company be willing to pay for the travel?

Jack''s latest blog: The GP-8E, my first computer

All programmers are optimists. Fred Brooks

Please submit clever ideas or thoughts about tools, techniques and resources you love or hate. Here are the tool reviews submitted in the past.

Like many hobbyists, as a teenager I learned to design circuits. College augmented those practical lessons with theoritical rigor. But only when employed as an EE did I learn how to design circuits that worked and were reliable. One mentor helped me with that as he critiqued each design decision, sometimes to an infuriating degree. I don''t see much in writing about this, but Jason Sachs'' recent article on managing tolerance is very worthwhile.

July''s giveaway, courtesy of Firia Labs, is one of their Explorer Kits, which includes 2?micro:bit?circuit board computers and associated components for experimenting.

Enter via this link.

Do you know your how many defects will be injected into the next product during development? Or how many of those your team will fix before releasing it to the customer?

If not. why not?

Capers Jones has written: "All software managers and quality assurance personnel should be familiar with these measurements because they have the largest impact on software quality, cost, and schedule of any known measures."

Jones is one of the world''s authorities on software engineering and software metrics. He often makes it clear that teams that don''t track defect metrics are working inefficiently at best.

He suggests we track "defect potentials" and "defect removal efficiency." His article "Software Benchmarking" in the October 1995 issue of IEEE Computer defines "The defect potential of a software application is the total quantity of errors found in requirements, design, source code, user manuals, and bad fixes or secondary defects inserted as an accidental byproduct of repairing other defects."

In other words, defect potential is the total of all mistakes injected into the product during development. Jones says the defect potential typically ranges between two and ten per function point, and for most organizations is about the number of function points raised to the 1.25 power.

Function points are only rarely used in the embedded industry; we think in terms of lines of code (LOC). Though the LOC metric is hardly ideal we do our analysis with the metrics we have. Referring once again to Jones'' work, his November 1995 article "Backfiring: Converting Lines of Code to Function Points" in IEEE Computer claims one function point equals, on average, about 128 lines of C.

"Defect removal efficiency" tells us what percentage of those flaws will be removed prior to shipping. That''s an appalling 85% in the US. But in private correspondence he provided me with figures that suggest embedded projects are by far the best of the lot, with an average defect removal efficiency of 95%, if truly huge (1m function points) projects are excluded.

Jones claims only 5% of software organizations know their numbers. Yet in the many hundreds of embedded companies I''ve worked with, only two or three track these metrics. Since defects are a huge cause of late projects it''s seems reasonable to track them. And companies that don''t track defects can never know if they are best in class. or the very worst, which, in a glass-half-full way suggests lots of opportunities to improve.

What''s your take? What metrics does your company track?

A common complaint heard from tools vendors is that customers don''t know how to use some of the important features. From a user perspective, you''ve got dozens of instruments, each accompanied by a 500 page manual - who has the time to read all of that material?

But some of those overlooked features can save a lot of time.

Today some oscilloscopes have a very cool feature that can be occasionally a lifesaver called "zone triggering." You define a region of the screen that will start an acquisition if a signal strays into it. For instance, here''s a 10 KHz sine wave with a 6 us pulse that''s added to it once a second:

Normally this signal would be almost impossible to capture as it occurs so infrequently, and happens with a voltage level that''s within the normal sine wave. Here I''ve defined a little square zone trigger visible as a small box on the leading edge of the pulse. Anything that wanders into that box triggers the scope. Nothing else does.

The scope I used is a Siglent SDS5034X, though I know Tektronix and other support this feature on some of their instruments. The box is created using iPhone-like pinch-and-squeeze fingers on the touchscreen, though it''s possible to set this up parametrically. This scope supports a pair of zone triggers; each can cause a trigger if a signal wanders into the zone, or if it is found outside that region. Or, the two can be combined with an AND operation for more nuanced acquisitions.

A common use is to find bus contention where conflicting signals drive the voltage level to some value between a zero and a one. Or locating a short noise burst on an analog waveform.

The notion of a "Software Element Out of Context" is gaining in importance in safety-critical code, and I''ve been getting an increasing amount of email from readers wanting to learn more. I asked Dave Hughes of HCC Embedded if he could provide some background, which he graciously did, as follows:

All of computing history has been about creating building blocks and constantly improving and reconstructing them. This is equally true at both a software and hardware level; think about how you put together a PC or mobile phone; or how you build the software in them. It is a continuous process of improvement and in most cases the building blocks are built without an exact knowledge of the end-product. They just have a general use-case of which simple examples might be a microcontroller in hardware or a RTOS in software.

A great description of components like this is "elements out of context" - they were designed to a specification and the integrator must ensure that these elements do what is required. You can think of standard software libraries in the same way: as something designed without the complete context of its final usage being defined.

When we get to safety things change. Building a product that has safety considerations is completely different to standard products. A complete assessment of the risks associated with the device must be done and development methods used that fully mitigate those risks.

However, safety products need to use proven technology, be it a microcontroller in an air bag or piece of networking software for relaying sensor information. For this reason, we need a methodology for ensuring that the appropriate level of risk mitigation can be done for these elements.

In the automotive industry the ISO 26262 standard developed the concept of the Safety Element out of Context (SEooC) - taking general purpose components and integrating them with the development of a road vehicle. The standard covers the whole process of how you validate the element and gives several options for how you achieve this.

The two main methods are to use proven-in-use arguments which is really only suitable for hardware components (software does not wear out) or by providing a full set of ISO 26262 development artefacts for the element ready to support its integration.

The key to the whole process is the two sets of assumptions that are made. Firstly, there are the assumptions that the target system makes of the SEooC, and secondly the assumptions the SEooC makes of any system to which is integrated. If these are a perfect match, then you can integrate the element and validate it. Normally there will be gaps, particularly if it is software (since any change requires retesting). If changes are required, then you go through a tailoring process. Normally it is the SEooC that needs tailoring and if it has been developed within a safety development process with full traceability then this can be achieved relatively efficiently.

Although this is an automotive standard, I see it as a methodology that should be mappable to any other industry, particularly for software where it is the process is more comprehensive a portable than in any other standard I have seen. Although software safety standards are heavily verticalized (ISO26262 for Automotive, IEC 62304 for medical devices, DO178C for aerospace; to name a few of very many) safety software in all industries has the same basic requirements in terms of specification, design, implementation, test and most importantly traceability. It is traceability that gives you the ability to modify with confidence when there are slightly different assumptions made by the target system.

For a more detailed description of how SEooCs can be used for software please try our white paper that can be downloaded from: www.hcc-embedded.com/seooc

Click on picture to get a full-sized image.

An ATM in Baltimore:

Let me know if you’re hiring embedded engineers. No recruiters please, and I reserve the right to edit ads to fit the format and intent of this newsletter. Please keep it to 100 words. There is no charge for a job ad.

These jokes are archived here.

The Definition of an Upgrade: Take old bugs out, put new ones in.

The Embedded Muse is Jack Ganssle''s newsletter. Send complaints, comments, and contributions to me at jack@ganssle.com.

The Embedded Muse is supported by The Ganssle Group, whose mission is to help embedded folks get better products to market faster. We offer seminars at your site offering hard-hitting ideas - and action - you can take now to improve firmware quality and decrease development time. Contact us at info@ganssle.com for more information.