[Bug 1961542] [NEW] libsmartcols: Revert back to previous behaviour of non-shell parsable column output (lsblk -P)
Matthew Ruffell
1961542 at bugs.launchpad.net
Sun Feb 20 23:47:03 UTC 2022
Public bug reported:
[Impact]
util-linux 2.37 in Jammy has introduced some new behaviour for lsblk and
similar tools which depend on libsmartcols. This switched the -P /
--pairs parameter from printing column names as normal, to changing the
names to shell compatible names instead.
e.g. lsblk -P now outputs LOG_SEC instead of LOG-SEC.
The change broke some core tooling which rely on on the output of lsblk
-P, most notably, curtin and MAAS, but I am sure there will be more
applications affected.
Affected MAAS users will see the following traceback when attempting to
deploy 22.04:
Traceback (most recent call last):
File "/curtin/curtin/block/__init__.py", line 785, in get_blockdev_sector_size
logical = info[parent]['LOG-SEC']
KeyError: 'LOG-SEC'
'LOG-SEC'
curtin: Installation failed with exception: Unexpected error while running
This is documented in MAAS bug 1956613.
MAAS decided to fix it by changing from -P to -J, in the following commit:
https://git.launchpad.net/maas/commit/?id=e2c01963430e6837198a54bc1eadf3efc9fdd9a2
Curtin now checks for MAJ_MIN, and changes it back to MAJ:MIN in:
https://github.com/canonical/curtin/commit/ce811db127fe1ce46498b83615f8faed8c7dfeb6
The issue is that these commits are not tagged to any MAAS release, and
users would be forced to upgrade MAAS to the latest stable release when
available if they want to deploy 22.04.
There are many users out there that don't want to upgrade MAAS, so
returning to the previous column output is the most desirable solution.
[Testcase]
On a Jammy install, simply run lsblk with either -P or --pairs:
$ sudo lsblk -P
...
NAME="sda" MAJ_MIN="8:0" RM="0" SIZE="465.8G" RO="0" TYPE="disk" MOUNTPOINTS=""
...
Affected installs will see MAJ_MIN.
There is a test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf326660-test
If you install it, you will see MAJ:MIN, just like it is on Impish and
previous.
[Where problems could occur]
We are changing the column output for -P and --pairs for the following
applications:
* lsblk
* findmnt
* lsipc
If any application has been modified to depend on the new column output,
it will break. I don't have any examples of something that will break,
because MAAS and curtin were modified such that they would be compatible
with both column name formats.
It should be noted that the manpage documents that lsblk output can
change at any time:
> The default output, as well as the default output from options like --fs and --topology, is subject to change.
> So whenever possible, you should avoid using default outputs in your scripts.
> Always explicitly define expected columns by using --output columns-list and --list in environments where a stable output is required.
If a regression should occur, we will need to fix up these affected
packages also.
[Other info]
The change came about when a user asked upstream to make -P / --pairs
shell parsable, in Issue 1201 upstream:
https://github.com/util-linux/util-linux/issues/1201
Karel Zak obliged, and it was implemented in the following commit:
commit 58b510e5805d8350c31bfb81a47bcd38ea9fdd7e
From: Karel Zak <kzak at redhat.com>
Date: Thu, 3 Dec 2020 12:14:10 +0100
Subject: libsmartcols: sanitize variable names on export output
Link: https://github.com/util-linux/util-linux/commit/58b510e5805d8350c31bfb81a47bcd38ea9fdd7e
I wrote to Karel Zak with the regressions introduced by changing the
format, and asked to revert back, and instead implement the shell
parsable logic as a new parameter.
This happened in upstream issue 1594:
https://github.com/util-linux/util-linux/issues/1594
Karel Zak was happy to oblige again, and we now have the following
commits:
338ad4a93 findmnt: commit missing flag
0f843ab64 lsblk: update --help output for -y
eba05f308 lsipc: add -y,--shell
152c17aa4 findmnt: add -y,--shell
9c7e81ff1 lslogins: add -y,--shell
25fb0638a lsblk: add -y/--shell
39679ea0c lsfd: use new libsmartcols functions
6fd0e3590 column: use new libsmartcols functions
0b3c2e80d include/carefulputc: remove unused function
3b5db50f7 libsmartcols: change "export" behavior, add "shellvar" flag
While we got the intended behaviour, these commits won't land until
util-linux 2.38, which will be after Jammy releases, and the other issue
is that this changes a significant amount of code, like nearly 1k lines,
and is spread over 10+ commits.
I wrote to ubuntu-devel asking for advice, on either 1) not changing
anything 2) backporting the 10+ new commits, or 3) simply reverting the
commit which changed the behaviour.
https://lists.ubuntu.com/archives/ubuntu-devel/2022-February/041870.html
ubuntu-devel had strong support for option (3).
Hence, we will revert the below commit to ensure Jammy can be deployed on all existing MAAS releases.
58b510e580 libsmartcols: sanitize variable names on export output
** Affects: util-linux (Ubuntu)
Importance: High
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Affects: util-linux (Ubuntu Jammy)
Importance: High
Assignee: Matthew Ruffell (mruffell)
Status: In Progress
** Tags: jammy sts
** Also affects: util-linux (Ubuntu Jammy)
Importance: Undecided
Status: New
** Changed in: util-linux (Ubuntu Jammy)
Status: New => In Progress
** Changed in: util-linux (Ubuntu Jammy)
Importance: Undecided => High
** Changed in: util-linux (Ubuntu Jammy)
Assignee: (unassigned) => Matthew Ruffell (mruffell)
** Tags added: jammy sts
--
You received this bug notification because you are a member of Ubuntu
Foundations Bugs, which is subscribed to util-linux in Ubuntu.
https://bugs.launchpad.net/bugs/1961542
Title:
libsmartcols: Revert back to previous behaviour of non-shell parsable
column output (lsblk -P)
Status in util-linux package in Ubuntu:
In Progress
Status in util-linux source package in Jammy:
In Progress
Bug description:
[Impact]
util-linux 2.37 in Jammy has introduced some new behaviour for lsblk
and similar tools which depend on libsmartcols. This switched the -P /
--pairs parameter from printing column names as normal, to changing
the names to shell compatible names instead.
e.g. lsblk -P now outputs LOG_SEC instead of LOG-SEC.
The change broke some core tooling which rely on on the output of
lsblk -P, most notably, curtin and MAAS, but I am sure there will be
more applications affected.
Affected MAAS users will see the following traceback when attempting
to deploy 22.04:
Traceback (most recent call last):
File "/curtin/curtin/block/__init__.py", line 785, in get_blockdev_sector_size
logical = info[parent]['LOG-SEC']
KeyError: 'LOG-SEC'
'LOG-SEC'
curtin: Installation failed with exception: Unexpected error while running
This is documented in MAAS bug 1956613.
MAAS decided to fix it by changing from -P to -J, in the following commit:
https://git.launchpad.net/maas/commit/?id=e2c01963430e6837198a54bc1eadf3efc9fdd9a2
Curtin now checks for MAJ_MIN, and changes it back to MAJ:MIN in:
https://github.com/canonical/curtin/commit/ce811db127fe1ce46498b83615f8faed8c7dfeb6
The issue is that these commits are not tagged to any MAAS release,
and users would be forced to upgrade MAAS to the latest stable release
when available if they want to deploy 22.04.
There are many users out there that don't want to upgrade MAAS, so
returning to the previous column output is the most desirable
solution.
[Testcase]
On a Jammy install, simply run lsblk with either -P or --pairs:
$ sudo lsblk -P
...
NAME="sda" MAJ_MIN="8:0" RM="0" SIZE="465.8G" RO="0" TYPE="disk" MOUNTPOINTS=""
...
Affected installs will see MAJ_MIN.
There is a test package available in the following ppa:
https://launchpad.net/~mruffell/+archive/ubuntu/sf326660-test
If you install it, you will see MAJ:MIN, just like it is on Impish and
previous.
[Where problems could occur]
We are changing the column output for -P and --pairs for the following
applications:
* lsblk
* findmnt
* lsipc
If any application has been modified to depend on the new column
output, it will break. I don't have any examples of something that
will break, because MAAS and curtin were modified such that they would
be compatible with both column name formats.
It should be noted that the manpage documents that lsblk output can
change at any time:
> The default output, as well as the default output from options like --fs and --topology, is subject to change.
> So whenever possible, you should avoid using default outputs in your scripts.
> Always explicitly define expected columns by using --output columns-list and --list in environments where a stable output is required.
If a regression should occur, we will need to fix up these affected
packages also.
[Other info]
The change came about when a user asked upstream to make -P / --pairs
shell parsable, in Issue 1201 upstream:
https://github.com/util-linux/util-linux/issues/1201
Karel Zak obliged, and it was implemented in the following commit:
commit 58b510e5805d8350c31bfb81a47bcd38ea9fdd7e
From: Karel Zak <kzak at redhat.com>
Date: Thu, 3 Dec 2020 12:14:10 +0100
Subject: libsmartcols: sanitize variable names on export output
Link: https://github.com/util-linux/util-linux/commit/58b510e5805d8350c31bfb81a47bcd38ea9fdd7e
I wrote to Karel Zak with the regressions introduced by changing the
format, and asked to revert back, and instead implement the shell
parsable logic as a new parameter.
This happened in upstream issue 1594:
https://github.com/util-linux/util-linux/issues/1594
Karel Zak was happy to oblige again, and we now have the following
commits:
338ad4a93 findmnt: commit missing flag
0f843ab64 lsblk: update --help output for -y
eba05f308 lsipc: add -y,--shell
152c17aa4 findmnt: add -y,--shell
9c7e81ff1 lslogins: add -y,--shell
25fb0638a lsblk: add -y/--shell
39679ea0c lsfd: use new libsmartcols functions
6fd0e3590 column: use new libsmartcols functions
0b3c2e80d include/carefulputc: remove unused function
3b5db50f7 libsmartcols: change "export" behavior, add "shellvar" flag
While we got the intended behaviour, these commits won't land until
util-linux 2.38, which will be after Jammy releases, and the other
issue is that this changes a significant amount of code, like nearly
1k lines, and is spread over 10+ commits.
I wrote to ubuntu-devel asking for advice, on either 1) not changing
anything 2) backporting the 10+ new commits, or 3) simply reverting
the commit which changed the behaviour.
https://lists.ubuntu.com/archives/ubuntu-
devel/2022-February/041870.html
ubuntu-devel had strong support for option (3).
Hence, we will revert the below commit to ensure Jammy can be deployed on all existing MAAS releases.
58b510e580 libsmartcols: sanitize variable names on export output
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/1961542/+subscriptions
More information about the foundations-bugs
mailing list