Modernize Embedded Systems Engineering with DevOps [Small Business Edition]

Hi. My name is Jakub Zawadzki. I'm from Poland.

I'm an engineer with a passion for Embedded Systems & IoT projects, where hardware product plays an important role. I have almost two decades of experience in this type of projects as a SW developer, HW designer and in DevOps field.

If you are Embedded Systems Engineer working solo or in small team and want to learn more how DevOps can help you in your daily life, then you are in the right place. I'm starting an educational project that might interest you.

What is the idea and where did it come from?

My long experience as a SW developer (and short experience as a HW designer) was always in small scale projects done by small teams in small scale businesses. I have experienced first-hand the successes of projects that went smoothly, as well as the failures of those that were inadequately managed and got out of hand.

For the last four years, however, I'm developing my skills in the DevOps field as part of a team within a large multinational corporation environment. I'm witnessing the ways in which DevOps can improve the development process for large-scale embedded systems used in the infrastructure of modern mobile networks, as well as simplifying the daily lives of embedded developers and project managers.

And I remember very well the problems I faced as an SW developer on smaller projects, and I see that most of them were very similar, just on a different scale. I believe these problems can be solved using the same DevOps techniques, but they need to be scaled down to fit small teams with fewer resources.

This is where the idea of pocket DevOps comes to life.

DevOps for small businesses and small teams
- where small team starts from one person - You.
DevOps focused on Embedded Systems & IoT projects
- however, other project types could benefit too.
DevOps so small, that you can hide it in your pocket (assuming that you have big pockets ;P)
- You don't need fancy servers or cloud infrastructure. All you need to get started is your computer. You can expand later if you need to.
- You don't need a separate department of DevOps Engineers to maintain your setup. You can do it yourself as part of your daily tasks – it's not rocket science, and it won't take you long to set up if you know what you're doing.

What can you expect to find in this project?

I won't pretend that I have all the answers. I don't have the full picture yet; I just have a few ideas about where to start. But I want to embark on this journey, and I want you to join me. I want to combine my current and previous experience to learn through experimentation and guide you through all the necessary steps.

It is my side project that I work on in my free time, after my day job. I plan to create my own content as frequently as possible, but don't expect that I will publish it regularly. If I do, you can find it on this Blog, Youtube, GitHub or LinkedIn.

However, most of the time I will comment on the work of others and provide links to their resources. I will send this to your email inbox via a newsletter.

The main purpose of this newsletter is to share DevOps related knowledge. I will send you articles, random thoughts and interesting finds from the internet for free. I will also notify you there when I publish something on any of my platforms.

I promise that I won't send you any spam. You can unsubscribe at any time.

What are the elements of this project?

Newsletter - This is the base of the project and the main communication channel. If you want to stay up to date with all my publications or keep in touch, this is the best place to do so.
Blog - You will find here all the archived issues of the newsletter, as well as my public notebook where I post additional information that may interest you.
Youtube - My video channel that is an extension of the materials available on the blog. Following the principle that a picture is worth a thousand words, it is sometimes easier to show something than to try to explain it in words.
GitHub - I post random snippets of source code here. They are either linked to blog articles or YouTube videos, or they are small, standalone projects by themselves.
LinkedIn - My professional profile. In case you're interested in learning more about me, especially what I do besides create content for this project.

But what exactly is DevOps? And why should you care?

DevOps is a mindset, a culture and a set of technical practices, that will help your team to maintain a high quality of product development. Use the following practices and you will not regret it. It will take your development process to a higher level.

Agile Project Management

As a risk mitigation strategy.
Iterate, monitor and adapt - spend time to constantly improve the way you work.

Prototype Driven Development

Build a prototype first - minimize the risk of project failure.
Verify the idea before getting started with a full-scale feature or project.

Behavior Driven Development

Refine and verify your requirements.
Make sure that you are building the right product.

Test Driven Development

Build your product with testability in mind right from the start.
Build trust in your code - give yourself the confidence to modify the code without worrying about breaking it.

Centralized Repositories

Single source of truth for your project files.
Ensure consistent releases - always build and test from the same centralized project configuration.

Managed Environments

Ensure, that each member of the development team is using the same tools and environment configurations.
Use the same configuration across your CI/CD pipelines for consistent builds and releases.

Code Review

Clean code = higher product quality - let someone else look at your code before you will deliver it.
Use automated tools for code analysis - computer will analyze your code quicker and better than any human developer.

Live Documentation

Document everything you do. In a few months you won't remember what you did today. Write it down for your future self.
Leave notes in your design files and later, use automated tools to transform it into living documentation.

Automation

Your time is precious - if you have a cyclic task that is difficult or tedious to perform, automate it.
Focus on important tasks that will drive you forward, and let automation do the rest for you.

CI/CD Pipelines

Be sure that your product is always in releasable state.
Or be aware why it isn't.

Over-The-Air Updates

Make sure your product is bug free.
Even after you have shipped it.

Monitoring

Implement feedback loops.
Make sure that your project planning is based on real data, not on assumptions.

How is DevOps for Embedded Systems & IoT different from regular DevOps?

All of the above points will apply to this type of projects. The real difference is that you are building not only software (SW) part, but hardware (HW) and firmware (FW) as well. And the three must interact closely together. Therefore, there is a need for additional practices in your product development process.

Interdisciplinary Teams

Gather HW, FW and SW engineers in one place and let them build together iteratively, one increment of SW, FW and HW in parallel.

Testing Firmware on Host

Build your FW architecture in this way, so you can test it on the development machine and in CI/CD pipelines without having to use real HW.

Hardware-in-the-Loop Testing

Use simulated peripherals to automate FW integration testing on real HW.

JTAG Boundary Scan

Make it possible to test HW prototypes when the FW is not ready yet.

Fault Injection

Add the ability to simulate conditions that go beyond the capabilities of the real-world components.

System Level Simulation

You can have as many virtual products as you wish, on demand, ready to test your business logic.

Digital Twin

Build a digital copy of your product. You will be able to test it on extreme cases, without the risk of damaging it or injuring anyone.

Factory Tests

Before sending a product to a customer, test every single piece.

Failure Prediction

Collect logs from your devices in one place and analyze them looking for anomalies.

Resources Monitoring

Monitor the resource usage of your device (CPU, RAM, Flash, ...) on every commit to the master. If you need to optimize it later, you will know which parts of code to look at first.

Managing OTA Firmware Updates

Gain full control over whether, when and to which FW version your product will be updated. And verify that it has been done without errors.

Firmware Self-Test

Check that the FW update was successful and does not cause any problems. If not, roll back to the last known working state.

A little more about me and my experience.

Jakub

I consider myself more as an engineer than just a software developer. When working on a project, if it is possible, I like to have broader perspective on every aspect of system design. I have knowledge in many subjects, what gives me possibility to work in different roles and on different types of projects.

I have more than 15 years of professional experience in software development in Python, C, C++, Delphi, and more. Including over 2 years in bare-metal programming of embedded systems based on ARM, PIC and Renesas RL78 family microcontrollers.

Over 4 years (and counting) of experience as a DevOps engineer, working to support software teams and make sure product is delivered successfully, by implementing tools, frameworks, and technologies to enhance code quality and developers productivity.

2 years of experience in technical leadership, team's work coordination and working in the Agile methodology as a Scrum Master.

I have also experience in designing of digital electronics hardware (including 2 years in commercial projects). I don’t have plans to work as an electronics designer anymore, but that knowledge was proven to be useful many times in work on embedded projects.

I'm a big Linux enthusiast on desktop, servers and on embedded systems. I use it in my everyday work.

I am a Polish native speaker with English as my second language. Bear this in mind if you notice any mistakes in my publications.