I am not seeing the error splash screen but I am able to reproduce this issue in some form.
After loading the snapshot, attempts to power down fail. Running through a debugger I do not see this issue - the UI closes with the expected exit code (0). This only seems to occur if the MIDI Filter rules are applied. I will see if I can diagnose further.
[Edit] I removed the MIDI filter rules and now see the error message: exit code 139. (Actually that only appears after moving a control knob.) Removing the PianoTeq or FluidSynth layer did not change behaviour. Removing the PianoTeq and FluidSynth layers or just ZynAddSubFX allowed me to shutdown.
Fault occurs in Multi-timbral and Stage modes.
I think the error 139 may be the result of a crash in the encoder driver we are looking at which may be triggered by this hang…
[Edit] Removing TAL Reverb from both PT & ZASFx seems to stop the issue… I added a different JALV engine (monosynth) after removing tal reverb and although the machine did shutdown it also showed the 139 error. I added another instance of jalv monosynth and got the hung shutdown. This time the “layer” controller triggered the crash and that is the only controller on the first page of monosynth so may be connected. I think this gives us clues to two separate issues…
Thank you riban for your deeper analysis. At the coming weekend I will check more of my other snapshots in combination with the latest image.
The midi rules are necessary in my setup to prevent unwanted midi events causing unwanted behavior inside Zynthians chains. I only want keys/notes and sustain pedal from my stage piano pass to Zynthians chains. Zynthian acts as an intelligent sound expander in my setup.
Have a nice evening.
The midi rules are fine. I was mistaken thinking they had any adverse effect.
The issue with failure to shutdown was a deadlock between interdependent threads. I believe this is fixed in Testing branch. I would like @jofemodo to review the changes before applying to staging. I think this may have reduced the risk of crash during shutdown - I haven’t seen any since the change.
@pianissimo if you are willing and have the time you could switch to testing (which is currently just ahead of staging) and perform your tests.
[Edit] I have now suffered the crash so that is not fixed but the device is more likely to close down and the shutdown is faster. I continue to investigate the crash issue.
Today I did a quick test, updated my Zynthian, switched to test repos and updated again.
After the update the 3 chains (pianoteq, ZynAddSubFx and fluidsynth) of the test snapshot are working. But when I try to shutdown zynthian it got stuck during the shutdown. The UI display showes “powering off” in larger letters. I also tried to shutdown zynthian via webconf, but it does not work. So I had to cut the power. I tried this two times, but the same effect. The device hangs during shutdown.
Today I updated to the latest image from today and then tried some of my saved snapshots. They are working but shutting down Zynthian is not possible without running into an error.
To prevent hard cutting of power I am using this workaround:
cleanup all (layer menu)
ERROR screen occurs for a while
wait until getting back to main screen and don’t produce any midi events meanwhile
It is odd that you see the error screen during clean all. Something is crashing the Zynthian. I was doing some diagnosis yesterday and have done ideas. It would be good to have a reliable reproducible workflow that triggers the crash. Are you able to provide this? Step by step instructions and snapshots etc. to facilitate further investigation?
[Edit] Also will you check if the folder /zynthian/config/img existis and what is in it? It has disappeared from my machine which is blocking restart / shutdown because it is used to show splash screen.
@pianissimo I just pushed a possible fix to the crashing to testing branch. Please update and let me know if this resolves (or changes behaviour of) your issue.
@riban Today in the evening I updated to the latest testing branch and checked it out with two different snapshots. Hitting “Cleanup all” seems to be now possible without Zynthian hanging. But if I directly try to shutdown Zynthian get stuck with “Powering off”. You can then reach Zynthian via Web UI and hit shutdown from there, but nothing further happens.
Here some log data when you hit cleanup all. The xruns apprear during executing clean all. But there seems to be a error message during executing cleanup at Oct 17 19:49:33:
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:42:24 zynthian startx[493]: WARNING:zynthian_autoconnect.cb_jack_xrun: Jack Audio XRUN! => delayed 10224.0us
Oct 17 19:49:33 zynthian startx[493]: ERROR:zynthian_gui.zyngine_refresh: Control thread failed to terminate
Oct 17 19:49:33 zynthian startx[493]: ZynCore: Setting-up 8 x Zynswitches…
Oct 17 19:49:33 zynthian startx[493]: ZynCore: Setting-up 4 x Zynpots (zyncoders)…
Oct 17 19:49:33 zynthian startx[493]: New instance of zynseq
Oct 17 19:49:33 zynthian startx[493]: zynseq initialising as zynseq
For the moment, doing a “clean all” before doing a shutdown is necessary to prevent Zynthian to hang.
Until now I didn’t try to ssh into Zynthian when it’s getting stuck during shutdown and try to end it from there. This would be a next try. Maybe you have some tips what I can check via ssh. At the moment I cut the power to turn it off in this case.
What version of each repository do you have checked out? (Listed under “SOFTWARE” on the main webconf dashboard.)
This indicates that a thread is taking too long to stop. I thought I had fixed this so it will be interesting to see if you are using a version that has the fix in it.
Ah, ok, I see, build date is only system build date. The interesting things are the fingerprints on the 5 software modules. Now I see a difference between the update I did today and the update 2 days ago. There is a difference in the zyncoder module.
I will do a new test about stability on hitting “clean all” and shutting down Zynthian.
I loaded 3 different snapshots and played a while, 10 min or so. One time I hit cleanup before loading the next snapshot and one time I loaded directly the next snapshot. Cleanup and playing is working but when powering off Zynthian (you see “Powering Off” on the devices screen) it ends up there. Last messages I see in the log are these:
Oct 18 19:14:13 zynthian startx[3416]: ERROR:zynthian_gui.zyngine_refresh: Control thread failed to terminate
Oct 18 19:14:13 zynthian startx[3416]: ZynCore: Setting-up 8 x Zynswitches…
Oct 18 19:14:13 zynthian startx[3416]: ZynCore: Setting-up 4 x Zynpots (zyncoders)…
Oct 18 19:14:13 zynthian startx[3416]: New instance of zynseq
Oct 18 19:14:13 zynthian startx[3416]: zynseq initialising as zynseq
This test was done with the software version I posted above.
And when I try to shutdown with the Web UI it has no effect. Web service keeps on running.
Only via ssh I could shutdown Zynthian, but it took some time. I think the shutdown command is waiting for some time to end processes and when there is no response after a while these processes are terminated the hard way.
Certainly update regularly. We are working to fix bugs every day. I thought this one had been resolved. Please update and retest. If it fails again then remind us of the snapshot that is loaded, show a snapshot of the dashboard and I will try to reproduce. I am not seeing the issue here.
[Edit] I was just able to reproduce this by loading your Test.zss then closing down. I will investigate…
Some chainsaw and splitting axe exercise has given me clarity. I think this is now fixed in testing branch. The audio mixer was hanging on exit whilst trying to shutdown its jack client connection. It is not clear why your snapshot highlighted this but I hope this will fix it for you.
Hello @riban , great. Taking your chainsaw and splitting axe seems to be a good recept when working with Zynthian software . Now I can shutdown Zynthian. This issue was really annoying and it’s been there a long time and it’s independent from specific snapshots. I could take every snapshot I have to force the shutdown error.
Although shutdown is working now, when I take a look at the log there is now one error message left:
ct 22 18:45:54 zynthian startx[490]: ERROR:zynthian_gui.zyngine_refresh: Control thread failed to terminate
Oct 22 18:45:54 zynthian startx[490]: ZynCore: Setting-up 8 x Zynswitches…
Oct 22 18:45:54 zynthian startx[490]: ZynCore: Setting-up 4 x Zynpots (zyncoders)…
Oct 22 18:45:54 zynthian startx[490]: New instance of zynseq
Oct 22 18:45:54 zynthian startx[490]: zynseq initialising as zynseq
No that’s not right! The error indicates that the control thread isn’t stopping (in the allocated time). Something must be locking it up. I will try to reproduce here. I changed that code recently to avoid deadlocks and give better debug messages. It looks like I got one of those things right!
. Now I can shutdown Zynthian. This issue was really annoying and it’s been there a long time and it’s independent from specific snapshots. I could take every snapshot I have to force the shutdown error.
