Firmware Testing Pyramid

Remember that controversial decision from my first article? The one where I removed TEST as a separate step from the "pocket DevOps" diagram?

Let me refresh your memory. Here's what I said back then:

Test - Keeping this as a separate step would suggest that testing happens only at this point. Instead, we should recognize that all other steps include some form of testing. Therefore, let's remove it as a separate step.

Some of you probably thought: "Wait, so we just... don't test?" That's not what I meant. Testing is crucial. But treating it as a single, isolated phase at the end? That's the problem.

Today, I'll show you what I actually meant. Using firmware testing as an example, I'll demonstrate how each step in the diagram should already contain some form of testing.

Quick Recap: The "pocket DevOps" Cycle

Before we dive in, let's revisit that diagram we created together.

Each step has its purpose:

• PLAN - Refine and verify your requirements.
• BUILD - Create the next product increment.
• INTEGRATE - Integrate hardware, firmware and software to catch requirement mismatches or gaps.
• ANALYZE - Monitor your product's behaviour. Draw conclusions from the collected data. Then, adjust your plan based on new insights. Then, start the next iteration.
• RELEASE - After N iterations, release your product to the user's hands.

Now here's the key insight: each of these steps already includes testing. We need different types of tests for different purposes.

But not all tests are created equal. Some run in milliseconds on your development machine. Others need real hardware and might take hours. Some verify individual functions. Others test complete user scenarios. This creates a natural hierarchy. Think of it as a pyramid - and that's exactly what we'll explore next.

Three Levels of Firmware Testing

Think about firmware testing as a pyramid with three levels. Each level serves a different purpose and uses different tools. Here's how they map to our "pocket DevOps" cycle:

1. Unit Tests

Unit tests live in the BUILD stage. These are your fast, automated tests that run on your development machine.

• Fully automated - no human intervention needed
• Run on your host machine - fast feedback, no hardware required
• Developer's perspective - tests individual functions and modules

This is where you, the developer, verify that your code does what you think it does. You write a function, you write a test. Simple as that.

2. Integration Tests

Integration tests belong to the INTEGRATE stage. Now we're getting more realistic. Your code needs to talk to hardware peripherals. Your modules need to work together. This is where integration tests come in.

• Fully automated - still no manual work
• Hardware-in-the-Loop - real target with simulated peripherals
• Business' perspective - tests against actual requirements

Now you're checking if your firmware components actually work together. Does the sensor driver talk to the serial bus? Does data flow through the system the way your requirements say it should?

3. End-to-End Tests

E2E tests happen during the ANALYZE stage. They run on complete, real hardware. Real conditions. Real users (or someone pretending to be one).

• Manual or partly automated - Human-in-the-Loop approach
• Real hardware - the actual product in real conditions
• User's perspective - tests complete user scenarios

This is where you or your users interact with the device. Does it behave as expected? Is the user experience smooth? These tests might uncover issues that automated tests miss.

Making Sense of the Pyramid

Why a pyramid shape? Because the distribution of your tests should follow this pattern:

• Lots of unit tests (bottom layer) - fast, cheap, many of them
• Moderate integration tests (middle layer) - slower, more expensive
• Few end-to-end tests (top layer) - slowest, most expensive, use them wisely

I've worked on projects where this pyramid was completely inverted. Everything ran at the system level. Test suites took hours. Tests failed randomly because of infrastructure issues, not because of actual bugs. We wasted entire weeks debugging the test infrastructure instead of the product.

Don't be that project. Push your tests down the pyramid as far as they can go. Test your logic with unit tests. Test your integration points with integration tests. Use end-to-end tests only for what they're uniquely good at: validating the complete user experience.

It All Starts With Requirements

Let's talk about the PLAN stage for a moment. There's something crucial to understand: Testing doesn't start when you write code. It starts when you write requirements. Every requirement you write during the PLAN phase should eventually be verified by a test. Maybe it's a unit test. Maybe it's an integration test. Maybe it's an end-to-end scenario. The level doesn't matter. What matters is that if you can't test a requirement, you probably wrote it wrong.

This is where you define what you're building. You write requirements. You specify behavior. You set acceptance criteria.

Here's the rule that should guide your planning:

"Every requirement must be verifiable by at least one test."

Notice I didn't say "must have a test written for it." Not yet. But it must be testable. You should be able to imagine how you'd verify it.

If you can't figure out how to test a requirement, that's a red flag. Either:

• The requirement is too vague
• The requirement is not really a requirement
• You need to break it down into smaller, testable pieces

Some requirements naturally map to unit tests: "The CRC calculation function must return 0xABCD for input data 0x12, 0x34, 0x56." Those are easy to test.

Some map to integration tests: "The device must respond to a configuration command within 100ms." You need real hardware interaction for those.

Some need end-to-end testing: "The LED must blink green when the device successfully connects to the network." Someone needs to watch that LED and verify the behavior.

The point is: you decide which level of the pyramid will verify each requirement during planning. Not after implementation. Not when something breaks. During planning.

This changes how you write requirements. You start thinking about testability from the beginning. You write clearer specs. You avoid assumptions. You create a direct link between what you planned and how you'll verify that it worked.

Focus on What Matters

Stop obsessing over code coverage percentages. Seriously. I don't care if your unit tests cover 100% of your code. That metric tells me nothing about whether your product actually works. Here's a better approach:

• "Everything that is important is tested. If it's not tested, then it's not important."
• "If you're afraid to refactor a piece of code for fear of breaking it, you need a test for it."

These rules force you to think clearly about what actually matters. They help you:

• Prioritize what to automate - Focus your test automation efforts on critical functionality, not on achieving an arbitrary coverage number
• Identify gaps - If something is important but not tested, you have a gap. Fix it.
• Make explicit decisions - When you decide not to test something, you're explicitly saying it's not important. Own that decision.
• Prevent testing theater - Stop writing tests just to write tests. Every test should have a clear purpose.

I've seen teams waste weeks writing tests for trivial getters and setters to hit their coverage targets. Meanwhile, critical business logic went untested because "we already hit our coverage goal." Don't fall into that trap. Test what matters. Skip what doesn't. Be honest about the difference.

What About Regression Testing?

Here's my favorite approach:

"Whenever we see a bug, we write an automated regression test for that specific behavior."

Two reasons why this works:

• Prevention. That exact bug will never make it to production again. You have a test for it now.
• Documentation. Your regression tests become a living record of every edge case, every corner case, and every weird behavior your system has encountered. They extend your specification. They capture knowledge that would otherwise live only in bug reports and someone's memory.

Think about it: every bug that reaches your users represents a gap in your tests. A missing scenario. A wrong assumption. When you turn that bug into a test, you're closing that gap permanently.

And here's the bonus: as your regression test suite grows, your product becomes more stable. Not because you're writing better code (though you probably are), but because you're systematically eliminating ways for old bugs to return.

The Role of Automation in Testing

Now let's talk about the RELEASE stage. You might think: "We've already tested everything. What does Release have to do with testing?" Everything, actually.

Here's the thing: all your tests mean nothing if you can't get changes into your users' hands. You discover a critical bug. You write a test for it. You fix it. Great. Now what? Does it take you three months to release that fix? Or can you ship it this week?

That's where Continuous Integration and Continuous Delivery (CI/CD) pipelines come in.

• Continuous Integration means the ability to test every single change you make against your test suite. Every commit triggers your tests. You get immediate feedback. Did your change break something? You know within minutes, not days.
• Continuous Delivery means the ability to deploy changes to users safely, quickly, and sustainably. Once your tests pass, you can ship. On demand. Whenever you choose.

Why does this matter for testing? Two reasons:

• Faster feedback loops. Remember that regression test you just added? With CI, you validate it immediately. With CD, you can deploy it to the field faster. You learn whether your fix actually works in real conditions, not just in your lab.
• Your product must stay deployable. This forces discipline. Your test suite needs to run reliably. Your build needs to be automated. Your deployment process needs to be repeatable. All of this improves your testing infrastructure.

Here's what enables CI/CD:

• Your automated tests must pass consistently
• Your integration pipeline must be reliable
• Your deployment mechanism must be proven
• Your rollback strategy must be tested

Notice a pattern? Testing again. But now you're testing your ability to release, not just your product features. When you commit to CI/CD, you're committing to keeping your product in a constantly deployable state. Every iteration. Every test that runs. Every commit that lands. This changes how you approach testing throughout the entire development cycle.

Putting It All Together

Let's connect this back to the "pocket DevOps" cycle. Testing isn't a separate phase that happens at the end. It's integrated into every step:

• PLAN - Write testable requirements. For each requirement, identify which level of the testing pyramid will verify it. If you can't figure out how to test it, rewrite it.
• BUILD - Write unit tests. Test your logic. Make sure your code does what you designed it to do.
• INTEGRATE - Run integration tests. Verify your components work together. Catch interface mismatches early.
• ANALYZE - Perform end-to-end testing. Watch real users interact with your product. Validate your assumptions.
• RELEASE - Maintain the ability to deploy quickly. Your automated tests enable this. When they pass reliably, you can ship with confidence.

And throughout all of this, grow your regression test suite. Every bug becomes a test. Every test becomes documentation. Every iteration makes your product more robust.

That's what I meant when I said testing happens at every step. Not as a separate phase. Not as something you do "after development." As an integral part of how you build firmware.

What's Next?

You might be wondering: "Okay, this all sounds good in theory. But how do I actually implement this? What tools do I use? How do I set up Hardware-in-the-Loop testing? How do I automate tests for embedded systems?"

Fair questions. And they deserve detailed answers. But those answers are big enough for their own articles. For now, focus on the concept: testing at every layer, for different purposes, from different perspectives. Get the strategy right first. The tactics will follow.

Here's how to start:

• During PLAN, ask yourself: "How will I verify this requirement?" Write it down. Make it explicit.
• During BUILD, write unit tests. They're the easiest to set up and give you the fastest feedback loop.
• During INTEGRATE, add integration tests. Use simulation when you can't access real hardware yet.
• During ANALYZE, formalize your manual testing into repeatable scenarios.
• And think about your RELEASE strategy early. Identify which tests should have the highest priority. Start automating those first in your CI/CD setup.

You don't need all three levels from day one. Build your pyramid one layer at a time. But always start with a plan for how you'll test what you're about to build.

In the next articles, I'll show you the actual tools and setups I use for each layer. We'll start with unit testing on the host machine - the foundation of the pyramid. Subscribe to the newsletter so you don't miss out.