Error upgrading on Debian 11

[dominic-p]

Issue Summary

When I attempt to upgrade the Twilio CLI package via apt on Debian I get the following error:

dpkg-deb: error: archive '/var/cache/apt/archives/twilio_5.3.0-1_amd64.deb' uses unknown compression for member 'control.tar.zst', giving up
Traceback (most recent call last):
  File "/usr/share/apt-listchanges/DebianFiles.py", line 124, in readdeb
    output = subprocess.check_output(command)
  File "/usr/lib/python3.9/subprocess.py", line 424, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib/python3.9/subprocess.py", line 528, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dpkg-deb', '-f', '/var/cache/apt/archives/twilio_5.3.0-1_amd64.deb', 'Package', 'Source', 'Version', 'Architecture', 'Status']' returned non-zero exit status 2.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/bin/apt-listchanges", line 323, in <module>
    main(config)
  File "/usr/bin/apt-listchanges", line 104, in main
    pkg = DebianFiles.Package(deb)
  File "/usr/share/apt-listchanges/DebianFiles.py", line 358, in __init__
    parser.readdeb(self.path)
  File "/usr/share/apt-listchanges/DebianFiles.py", line 127, in readdeb
    raise RuntimeError(_("Error processing '%(what)s': %(errmsg)s") %
RuntimeError: Error processing '/var/cache/apt/archives/twilio_5.3.0-1_amd64.deb': Command '['dpkg-deb', '-f', '/var/cache/apt/archives/twilio_5.3.0-1_amd64.deb', 'Package', 'Source', 'Version', 'Architecture', 'Status']' returned non-zero exit status 2.
Preconfiguring packages ...
dpkg-deb: error: archive '/tmp/apt-dpkg-install-cwYneZ/0-twilio_5.3.0-1_amd64.deb' uses unknown compression for member 'control.tar.zst', giving up
dpkg: error processing archive /tmp/apt-dpkg-install-cwYneZ/0-twilio_5.3.0-1_amd64.deb (--unpack):
 dpkg-deb --control subprocess returned error exit status 2

From this SE post, I gather that the package is using a compression format that Debian's version of dpkg doesn't understand.

Steps to Reproduce

1. Run apt update && apt upgrade on Debian 11 Bullseye

Technical details:

• twilio-cli version: Current: 5.2.2, Attempting to upgrade to: 5.3.0
• node version: 16.18.0

[dominic-p]

Friendly bump. Anyone else seeing Debian compatibility issues?

[mlncn]

Yeah, Twilio CLI is repeatedly blowing up my upgrades. Currently, same error as is reported here:

dpkg-deb: error: archive '/tmp/apt-dpkg-install-Od2MRa/0-twilio_5.3.0-1_amd64.deb' uses unknown compression for member 'control.tar.zst', giving up

Seems this compression method was added by Ubuntu, and Debian may be grudgingly adding support also, but has not yet:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=892664

Therefore Twilio should avoid using it, at least for deb, for now.

Note: sudo apt install --fix-broken appears to repair the rest of Debian after the update bails due to Twilio, or maybe it was always fine except Twilio.

[charan678]

This issue has been added to our internal backlog to be prioritized. Pull requests and +1s on the issue summary will help it move up the backlog.

[dominic-p]

I'd be happy to submit a pull request, but I could use some help getting oriented.

I searched the code base for zstd the other day and I came up empty. I'm guessing some default setting in a dependency got changed, but I couldn't find that either.

[AsabuHere]

This error upon unpacking CLI is because zstd decompression was supported in ubuntu but not in debian. This issue is fixed in latest version of dpkg, where both zstd compression and decompression is supported. The version 1.21.18 has the fix for this issue. (https://packages.debian.org/sid/all/dpkg-dev/download)
After upgrading dpkg to the version 1.21.18, both install and upgrade steps are working fine for latest version of twilio-cli.
@dominic-p Can you please try upgrading dpkg to the latest version and try upgrading twilio-cli?

[dominic-p]

Thanks for the update! I downloaded the unstable 1.21.18 version of dpkg, and attempted to install it, but I ran into a dependency error. It looks like it depends on libc6 >=2.34 and my machine has 2.31. As this is a production machine it seems risky to start upgrading core components like libc, so I'm kind of stuck at the moment.

Any other ideas for workarounds we can try? Given the pace of Debian development, I'm guessing that dpkg 1.21.18 might take a while to reach stable. What can we do in the meantime? Is zstd compression a key feature? Can we just disable it until Debian catches up to Ubuntu?

Install command output for reference:

# dpkg --install dpkg_1.21.18_amd64.deb
dpkg: regarding dpkg_1.21.18_amd64.deb containing dpkg, pre-dependency problem:
 dpkg pre-depends on libc6 (>= 2.34)
  libc6:amd64 is installed, but is version 2.31-13+deb11u5.

dpkg: error processing archive dpkg_1.21.18_amd64.deb (--install):
 pre-dependency problem - not installing dpkg
Errors were encountered while processing:
 dpkg_1.21.18_amd64.deb

[shrutiburman]

Well, we'd need dpkg 1.21.18 to support zStandard compression & decompression. As you might've already seen zstd is not a direct dependency of twilio-cli, but is internally referenced in one of our pack & build commands, so it's not something we can disable or remove (for the sake of deb users).
Please let me know, if there's anything else that I can help you with?

[dominic-p]

Thanks for the feedback. Could the pack & build commands be tweaked to not use zstd? Something changed a month or so ago in how the package was built. Could that change be reverted until Debian catches up to Ubuntu?

Again, I'd be happy to take a stab at it, but I'm new to the code base. Maybe someone could point me to where the Debian build is configured?

[shrutiburman]

Dominic, the problem you're facing with dpkg's latest version is very specific to your environment. Please understand that we cannot afford not upgrading to the latest version unless it's an issue for a larger number of users.
However, we use dpkg --build command to build the binaries, without refactoring any process at our end.
You can take a look at our build script, and see if there's any possible tweak that could be done to unblock you, and which doesn't break anything.
Would be happy to help with that. :)

[dominic-p]

Thanks for the help, I really appreciate it. I'll start looking into dpkg --build to see if there's anything that could be tweaked.

Also, maybe I'm misunderstanding something, but from this page it appears that dpkg 1.21.18 is still only available in the testing version of Debian. Since I'm running a fully patched version of Bullseye (stable), it seems odd to me that my issue could be specific to my environment. If Debian is meant to be a supported OS, wouldn't the current stable release need to be supported?

I don't mean to be confrontational; I'm just genuinely trying to understand if there's something weird about my setup that I'm not aware of.

[dominic-p]

Just checking in on this. Is anyone available to review the PR is submitted?

[AsabuHere]

Hi @dominic-p Our team tested these changes and here's what we have got : Getting error:
dpkg: error: unknown option -Z

we have also updated the PR with this information. Can you please take a look?

[dominic-p]

Thanks for the update. I pushed a change to the PR that should fix it, but again, I don't really have a good test system available for this. I apologize for the back and forth.

I was kind of hoping the CI system would automatically run on the PR, so I could use its output to test my changes, but it looks like it needs approval to run.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Not stale for me. :) I'm sorry the PR hasn't been especially smooth, but I think the most recent version should do it. Again, I'm not really setup to test this currently, but I think I finally have it right.

[AsabuHere]

I'm testing the code changes in this PR. Testing on amd64 arch of debian is completed and is working fine. Trying to figure out if this is working well on armel arch of debian. Will provide an update soon

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Not stale for me. Have you been able to test on ARM yet?

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Still not stale for me. :) Is there anything I can do to help move this forward at this point?

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Not stale for me.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Still not stale for me. I would love to be able to upgrade twilio on my Debian 11 machines.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Not stale for me.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Still not stale for me.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Still not stale for me.

[github-actions]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or comment or this issue will be closed in 30 days.

[dominic-p]

Still not stale for me.