Plan for Introducing Automation into Zynthian Builds

Development
1

Plan for Introducing Automation into Zynthian Builds

Introduction

This document outlines a rough plan for introducing CI/CD automation into the building and testing of Zynthian software and Zynthian OS images based on the approach taken in the Librem 5 phone. This plan was discussed at a Zynth Club meeting on 2026-08-21 during which it was advised that writing the plan down would be helpful to the project.

Why

Advantages of this effort include but are not limited to:

  • Reduced workload for software maintainers and reduced likelihood of introducing bugs due to the elimination of manual steps.
  • Reduced OS image size and disk requirements due to the removal of compiler toolchain and build dependencies.
  • Reduced number of bugs that make it into the hands of users due to automated testing.
  • Reduced workload for reviewers of code contributions due to the obviation of manual testing.
  • Quicker migration between versions of the underlying distribution due to the ease of identifying regressions.
  • Improved robustness of all software.
  • Improved robustness and speed of software upgrades.
  • Elimination of technical debt.
  • Lower barrier of entry for new contributors due to the automation of software building, testing and integration.

How

The first steps of the rough plan are:

  1. Package all of the software which is currently compiled during the process of preparing an OS image.
  2. Build and test the resulting packages in CI, integrating the reporting of build and test results into merge requests.
  3. Transition the OS image building script to install packages using apt rather than compiling.

These steps will only eliminate compilation from the image building script. After compilation is eliminated, there are still very non-trivial tasks required:

  • Fully automatic image construction with no requirement for manual steps, in particular eliminating any need to execute the image on target hardware as part of the build process.
  • Integration testing of built images in VMs/containers.
  • Integration testing on actual hardware, for example using LAVA and a farm of test devices.

Complications

Currently the Zynthian codebase has problems which complicate the automation of software builds.

Assumption that the build host is the runtime host

There is an assumption throughout the Zynthian software and build systems that the software will be compiled on the same host on which it runs. So far, this seems most acute within the zyncoder package. For just one example, the zyncoder build script sets build flags based on the execution of lsmod on the build host:

The use of CI to build binary packages obviously invalidates this assumption and so the introduction of CI builds will necessitate extensive changes to how at least zyncoder and no doubt other packages operate so that all functionality is compiled in and behaviour adapts
based on runtime checks.

Compile-time software configuration

There are a number of preprocessor macros that are used to configure Zynthian software at compile time. For example:

There are also a number of environment variables in the image build script which describe the hardware the build is intended for. For example:

All of this compile-time configuration will need to be removed and replaced with runtime configuration based on static configuration files, runtime detection and user input. Any hardware-specific static configuration files will be put in hardware-specific packages, for example zynthian-hwconfig-zynthianV5.

Who

I, rah, am working to implement this plan. I hope others will help the effort, especially once CI jobs start to take form and demonstrate the advantages of this approach.

I will try to find some suitable introductory materials for creating Debian packages but I expect the bulk of the work will be in (1) modifying the existing software as described above, (2) implementing CI jobs and (3) implementing unit- and integration-tests.

7 Likes
2

Hi @rah !

Thanks for stepping over this. It’s a hard task.

Let’s forget about zyncoder library by now. It’s a critic piece that works fine, but it’s conditionally built depending of hardware config. Changing this is not trivial and it would require a time we don’t have right now. Build time for zyncoder is really short and it rarely fails.

Better let’s focus in other areas. zynlibs could be a better target.

Also all the LV2 plugins that are currently pre-compiled “by hand” and stored as tar.xz files:

We have build and install scripts for almost all of them. Take a look to the zynthian-sys/scripts/recipes.

And of course, the setup script, that turns bare bones RaspberryPiOS Lite into ZynthianOS.

Regards,

1 Like
3

I agree that we should start with zynlibs which use a similar build system as zyncoder but are cleaner and easier to update. You will have more assistance from us there because either can help and the code is fairly simple to understand. zyncoder is a beast that we may want to visit later, when we are confident of the benefit this brings and the process is robust and supportable.

As mentioned, we have recipes for building most (all?) things so we should be able to create the packages with a good (and simple) common build mechanism.

We have previously tried much of what is suggested and through pragmatism we ended up using precompiled tar packages rather than deb packages. If there is a way to create deb packages simply and reliably which we can support without too much effort then we can consider this but do also consider the alternative of using the existing build scripts and tar packages. (I know how useful a good package manager like dpkg is but also the overhead in maintaining it.) Also consider dependencies and interactions. We were using a branched version of libsndfile which caused issues with installing other packages that depended on it. I reversed this recently but had to remove the functionality that the branched version offered. This may have been down to my error in package management / creation - expertise is very welcome.

Don’t try to boil the ocean. Define a task and its benefit and deliver that. The benefit should inform future tasks, e.g. a simple documented workflow for creating packages supports the future use of a consistent package management system.

Thanks for getting involved with this. I look forward to seeing results. We are likely to be using the current mechanisms for vangelis. I suspect that all the benefits suggested here won’t be deliverable before we need to get vangelis out.

1 Like
4

I like the plan, and the responses, and I also thank you for stepping up to the job. I agree with doing some straightforward tasks before tackling zyncoder.

However, while we’re talking about zyncoder here, I’d like to point out that there may be relatively easy ways to handle it as well. I’m thinking of pre-compiled versions for each major-official hardware variant needed, like V4, V5, etc, and a runtime decision to load based on detecting hardware. People making non-standard DIY hardware would have to do a compile, but that seems acceptable-appropriate.

2 Likes
5

Hi @zynthianers!

No need to fix what is not broken.
We have zero problems with zyncoder as it’s, so no need to waste time discussing this.
Conditional compiling zyncoder on updates is super-well tested, it takes almost no time and it works perfectly OK. I would prefer we put the focus in other areas where we have much more to gain.

Cheers!

6 Likes
6

:+1:

I think we might not be communicating properly. Are you aware that I’m the developer who already raised this MR against zynlibs and hasn’t raised any MRs against zyncoder?

And this MR against the setup script?

7

For me, creating Debian packages is not a problem and so the idea of using shell scripts and tarballs seems inappropriate. It’s like trying to create a fire by rubbing sticks together (shell scripts and tarballs) when there’s a lighter (Debian packages and an apt repository) in my hand.

I would anticipate a huge saving on time by maintaining Debian packages rather than shell scripts and tarballs.

:+1:

I suspect that there may be a number of major releases before Zynthian OS images are released using CI/CD.

From my perspective, it will be ready when it’s ready. I don’t really understand the concerns about timing. And to be clear, I’ve been working on Linux audio for over 20 years. I helped write some of JACK back in the day. I’ve been in the game since long before Zynthian existed.

1 Like
8

:+1:

I think that would be much more work in the long run than modifying zyncoder to use static configuration files and then having hardware-specific packages that drop configuration files in the right place for zyncoder to pick up.

9

I’m very concerned by these words. Are you saying that you’re happy with Zynthian/zyncoder assuming the build host and target host are the same machine and using compile-time configuration? Are you outright precluding modifying zyncoder so that it can be built by CI?

10

Hi @rah!

Don’t be worried about this, please! Everything is fine :face_blowing_a_kiss:

Modifying zyncoder to avoid on-the-fly compiling is ultra-low priority for the project. It works very well and it’s a very critic piece of code. The amount of effort to do the needed changes must be justified with clear benefit. I’m not going to invest a ton of hours refactoring zyncoder just to avoid compiling this library on-the-fly. Testing zyncoder is VERY HARD. You need to test with the real hardware, what implies you have the real hardware at hand.

And compiling things on-the-fly is not wrong at all if the chain tool is at place and working fine, what is the case with zynthian. Lot of pip packages self-compile when installing. Some kernel modules too. Etc. There are many examples.

Perhaps you don’t agree my opinions, and this is perfectly OK. To collaborate, we don’t need to agree on everything or convince the other one. We simply must agree on something we can improve together, and hey!! there are a ton of things that can be improved !! Let’s focus on what we agree and forget about the rest.

Cheers!

3 Likes
11

The problem is that if you’re not prepared to accept patches modifying zyncoder so that it doesn’t assume the build host and target host are the same machine and doesn’t use compile-time configuration (“the rest”), then I’m not going to use Zynthian and so there’s no point in me contributing to anything.

To repeat my previous question:

Are you outright precluding modifying zyncoder so that it can be built by CI?

12

No - we are not saying that zyncoder should never be touched. That would of course be a daft stance. What we are saying is that there are many other parts of zynthian build that can be tackled first. The zynlibs and custom plugins are examples of where we can start this work with minimal impact on current build process. If that work proves successful, particularly providing an easy to understand and implement build process, then we can start to look at the more challenging parts of the code. zyncoder is so critical to zynthian operation and as you have identified, somewhat fragile that we can’t consider looking at it right now.

We are 100% focused on getting the vangelis beta out so have no capacity to look at non-critical issues for a few weeks. Please target your initial focus on the easy wins. This has these benefits:

  • Easier to implement.
  • Less likely to cause issues when merged.
  • More likely for core developers to engage.
  • Validates the proposal.
  • Accounts for 80% of the code that may be migrated in 20% of the time.

This list show how useful this exercise is when focused on this set and how it may be substantially less beneficial if initially targetted at zyncoder.

6 Likes
13

The critical thing is whether it should be modified to enable building with CI. @jofemodo are you on board with me (and/or others) modifying zyncoder at some point in future so that it can be built with CI, as described in this thread?

I feel I need to ask @jofemodo explicitly because (1) it would appear that @jofemodo has the power to dictate such matters, (2) I feel that we’re not communicating well at all, even accounting for the obvious language barrier, (3) I sense resistance, and (4) this is such a central, critical issue for me that I feel I need an explicit, unambiguous “yes, this is OK as a distant goal” or “no, I will never merge such patches” from the apparent BDFL before it’s clear whether there’s any point in me continuing work on Zynthian.

That’s what I’ve been doing :slight_smile:

1 Like
14

It is important to consider that all approaches are risks until blessed.

I cannot speak for others, but this is a reasonably critical time in development and in such a situation definitions of the height of fruit is best left to those who have been picking for some considerable time.

In truth what one does with an open source forked code base is limited only by imagination. I’ve been in frustrating situations like this and it is o your credit you are pursuing specific deadlock clearances.

Better code eventually drives out good, but the control of the timing remains with those processing pull requests, and they are busy little beavers.

Do keep examining and documenting. It’s all of great benefit to those keen to learn.

chris@wyleu

1 Like
15

Dear @rah !

Of course, yes, this could be OK as a distant goal.
But i don’t understand your need to “read” such a thing, as it’s something obvious.
We like to improve things and we work to do it. We have modified the zyncoder library many times and we will keep modifying it as many times as it’s needed.

The fact that this is such a central, critical issue for you is the cause of your perceived resistance from my part :wink:

The best,

1 Like
16

That’s a positive! I think that seeing what benefit this brings and what friction it causes will influence future progress. If the cost/benefit proves beneficial then we push on. It’s not then we stop or limit progress. It sounds like a green light to me. :grinning_face_with_smiling_eyes:

1 Like