From mpt at canonical.com  Sun Oct 13 09:55:45 2019
From: mpt at canonical.com (Matthew Paul Thomas)
Date: Sun, 13 Oct 2019 10:55:45 +0100
Subject: [ubuntu-hardened] Making security notices searchable
Message-ID: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>

Hi folks

On Canonical’s Web & Design team, I’m working on a project to design how
the USN tracker <https://usn.ubuntu.com/> can become part of the main
ubuntu.com site.

This will have several benefits, including the tracker being easier to
find, more obvious linking to and from other Ubuntu security info
<https://ubuntu.com/security>, and staying in sync with the look of the
rest of ubuntu.com.

As part of this, I’m considering adding a search function, so that you
can search security notices by Details text, with filters for Release
and Package.

So, I’m interested in knowing:

*   Is there any current method of searching USNs? (Other than using
    “site:usn.ubuntu.com” with a global search engine, or grepping the
    usn.ubuntu.com Git repo.)

*   If any search showed results sorted newest first, would there be any
    use case for searching notices by date? (For example, show me only
    notices posted in 2017.)

*   Anything else you think I should know?

Thanks
-- 
mpt


From alex.murray at canonical.com  Mon Oct 14 11:35:27 2019
From: alex.murray at canonical.com (Alex Murray)
Date: Mon, 14 Oct 2019 22:05:27 +1030
Subject: [ubuntu-hardened] Making security notices searchable
In-Reply-To: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
References: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
Message-ID: <87lftn4l1c.fsf@canonical.com>

Hey mpt,

On Sun, 2019-10-13 at 20:25:45 +1030, Matthew Paul Thomas wrote:

> Hi folks
>
> On Canonical’s Web & Design team, I’m working on a project to design how
> the USN tracker <https://usn.ubuntu.com/> can become part of the main
> ubuntu.com site.

Awesome!

>
> This will have several benefits, including the tracker being easier to
> find, more obvious linking to and from other Ubuntu security info
> <https://ubuntu.com/security>, and staying in sync with the look of the
> rest of ubuntu.com.
>
> As part of this, I’m considering adding a search function, so that you
> can search security notices by Details text, with filters for Release
> and Package.

This sounds very neat.

>
> So, I’m interested in knowing:
>
> *   Is there any current method of searching USNs? (Other than using
>     “site:usn.ubuntu.com” with a global search engine, or grepping the
>     usn.ubuntu.com Git repo.)

The USNs are also available in a big json blob - this may or may not be
easier to search due to it's size - but from this you can search on the
"description" attribute for each USN object.

https://people.canonical.com/~ubuntu-security/usn/database.json

>
> *   If any search showed results sorted newest first, would there be any
>     use case for searching notices by date? (For example, show me only
>     notices posted in 2017.)

I can imagine some folks might want this but this sounds more like a
nice-to-have feature than a must-have - I would assume most would want
to be able to sort based on which Release the USN applies to - so show
me all USNs which apply to bionic for apache2 etc.

>
> *   Anything else you think I should know?
>

Nothing off the top of my head but perhaps other folks might be able to
chime in with stuff I have missed :)

> Thanks
> -- 
> mpt



From dominik.russenberger at terreactive.ch  Mon Oct 14 14:00:26 2019
From: dominik.russenberger at terreactive.ch (=?utf-8?Q?Russenberger_Dominik?=)
Date: Mon, 14 Oct 2019 16:00:26 +0200
Subject: [ubuntu-hardened] Making security notices searchable
In-Reply-To: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
References: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
Message-ID: <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>

Hi,
Yes, a search would be great. My current usecase: we have an internal, 
partial mirror of the Ubuntu repos (multiple releases, different package 
set for each release). I have to check the repos for USNs for packages 
in there. My current solution is very... clumsy.

On Sun, Oct 13, 2019 at 10:55:45AM +0100, Matthew Paul Thomas wrote:
> As part of this, I???m considering adding a search function, so that you
> can search security notices by Details text, with filters for Release
> and Package.
Filtering by arches (e.g. amd64+all) might be useful, for the handful of
bugs that only affect a single architecture.

> So, I???m interested in knowing:
>
> *   Is there any current method of searching USNs? (Other than using
>     ???site:usn.ubuntu.com??? with a global search engine, or grepping the
>     usn.ubuntu.com Git repo.)
Well... my current solution is to get https://usn.ubuntu.com/releases/ubuntu-18.04-lts/
and then parse the HTML. Atom/RSS also just contain HTML for the content,
but no method to filter by release. And of course parsing HTML  breaks 
every once in a while.

> *   If any search showed results sorted newest first, would there be any
>     use case for searching notices by date? (For example, show me only
>     notices posted in 2017.)
My usecase requires to show all USNs after a particular date, or even 
better all USNs after another USN.

> *   Anything else you think I should know?
Machine-readable USNs! Alex Murray posted the link to a big json, but
downloading&parsing 130mb every hour doesn't sound like such a good
idea.

Thanks,
Dominik Russenberger
--


From sbeattie at ubuntu.com  Mon Oct 14 22:06:40 2019
From: sbeattie at ubuntu.com (Steve Beattie)
Date: Mon, 14 Oct 2019 15:06:40 -0700
Subject: [ubuntu-hardened] Making security notices searchable
In-Reply-To: <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>
References: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
 <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>
Message-ID: <20191014220640.GA27801@nxnw.org>

Hey Dominik,

Thanks for chiming in.

On Mon, Oct 14, 2019 at 04:00:26PM +0200, Russenberger Dominik wrote:
> Yes, a search would be great. My current usecase: we have an internal,
> partial mirror of the Ubuntu repos (multiple releases, different package
> set for each release). I have to check the repos for USNs for packages
> in there. My current solution is very... clumsy.
>
> On Sun, Oct 13, 2019 at 10:55:45AM +0100, Matthew Paul Thomas wrote:
> > As part of this, I???m considering adding a search function, so that you
> > can search security notices by Details text, with filters for Release
> > and Package.
> Filtering by arches (e.g. amd64+all) might be useful, for the handful of
> bugs that only affect a single architecture.

That might be possible but I don't think the result is going to be
satisfactory, in that architecture specific vulnerabilities are the
exception and not the rule.

> > So, I???m interested in knowing:
> >
> > *   Is there any current method of searching USNs? (Other than using
> >     ???site:usn.ubuntu.com??? with a global search engine, or grepping the
> >     usn.ubuntu.com Git repo.)
> Well... my current solution is to get https://usn.ubuntu.com/releases/ubuntu-18.04-lts/
> and then parse the HTML. Atom/RSS also just contain HTML for the content,
> but no method to filter by release. And of course parsing HTML  breaks 
> every once in a while.
> 
> > *   If any search showed results sorted newest first, would there be any
> >     use case for searching notices by date? (For example, show me only
> >     notices posted in 2017.)
> My usecase requires to show all USNs after a particular date, or even 
> better all USNs after another USN.

> > *   Anything else you think I should know?
> Machine-readable USNs! Alex Murray posted the link to a big json, but
> downloading&parsing 130mb every hour doesn't sound like such a good
> idea.

Understood. There is a compressed versions of the json blob available.
https://usn.ubuntu.com/usn-db/database.json.bz2 which is currently
"only" ~20MB in size, which may help that. Also, you shouldn't need
to re-download it every hour, a time check to see if it's newer
should suffice.

That said, one of the issues that makes the USN json db explode in
size is that it includes a reference for every binary package URL for
each USN; for something like a linux kernel update, this can result
in a couple of thousand URLs per USN, including things like udebs
that are only used by installer media. I'd like to provide a slightly
minimized USN json db that doesn't not contain those (but would still
contain the relevant source and binary packages). I've not yet done
an experiment to see what the size reduction would be, but I bet it's
significant and would reduce the amount of data to be consumed.

Would it also be useful to split the USN json db into per release
files as well?

Would a more digestible sized json db reduce the needs for complex
queries of the website itself?

Thanks again!

-- 
Steve Beattie
<sbeattie at ubuntu.com>
http://NxNW.org/~steve/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-hardened/attachments/20191014/eac9efcc/attachment.sig>

From mpt at canonical.com  Tue Oct 15 12:21:36 2019
From: mpt at canonical.com (Matthew Paul Thomas)
Date: Tue, 15 Oct 2019 13:21:36 +0100
Subject: [ubuntu-hardened] Making security notices searchable
In-Reply-To: <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>
References: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
 <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>
Message-ID: <b971f57a-24cd-b502-44e9-cfb4c064ef83@canonical.com>

Russenberger Dominik wrote on 14/10/2019 3:00 pm:
>…
> On Sun, Oct 13, 2019 at 10:55:45AM +0100, Matthew Paul Thomas wrote:
>>
>> As part of this, I’m considering adding a search function, so that
>> you can search security notices by Details text, with filters for
>> Release and Package.
>
> Filtering by arches (e.g. amd64+all) might be useful, for the handful
> of bugs that only affect a single architecture.

As far as I can tell, the notices don’t include architecture info in any
structured format. We can’t search data that doesn’t exist.

>> So, I’m interested in knowing:
>>
>> *   Is there any current method of searching USNs? (Other than using
>>     “site:usn.ubuntu.com” with a global search engine, or grepping>>     the usn.ubuntu.com Git repo.)
>
> Well... my current solution is to get
> https://usn.ubuntu.com/releases/ubuntu-18.04-lts/
> and then parse the HTML.

I’m not proposing to port the release pages, since with the introduction
of search, they would be redundant with searching for all notices from a
particular release. (Of course you could continue parsing the search
results page.)

> Atom/RSS also just contain HTML for the content,
> but no method to filter by release. And of course parsing HTML breaks 
> every once in a while.

I wonder if we could provide Atom/RSS feeds for arbitrary search
results. (I realise it might be more expensive for us than generating a
single feed for all notices.)

>> *   If any search showed results sorted newest first, would there be
>>     any use case for searching notices by date? (For example, show me
>>     only notices posted in 2017.)
>
> My usecase requires to show all USNs after a particular date, or even 
> better all USNs after another USN.

I guess your current scraping stops when it gets to a USN that it’s seen
before.

>> *   Anything else you think I should know?
>
> Machine-readable USNs! Alex Murray posted the link to a big json, but
> downloading&parsing 130mb every hour doesn't sound like such a good
> idea.
>…

Cheers
-- 
mpt


From seth.arnold at canonical.com  Sat Oct 19 01:33:50 2019
From: seth.arnold at canonical.com (Seth Arnold)
Date: Sat, 19 Oct 2019 01:33:50 +0000
Subject: [ubuntu-hardened] Making security notices searchable
In-Reply-To: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
References: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
Message-ID: <20191019013350.GA3304@millbarge>

On Sun, Oct 13, 2019 at 10:55:45AM +0100, Matthew Paul Thomas wrote:
> On Canonical’s Web & Design team, I’m working on a project to design how
> the USN tracker <https://usn.ubuntu.com/> can become part of the main
> ubuntu.com site.

Hello Matthew, this sounds wonderful!

> *   If any search showed results sorted newest first, would there be any
>     use case for searching notices by date? (For example, show me only
>     notices posted in 2017.)

Yes, it would be nice to be able to search by all the usual criteria:
freeform text, binary package names, source package names, dates,
affected releases, unaffected releases, etc.

> *   Anything else you think I should know?

Our current publishing process for USNs is not ideal. While the static
site generation approach worked well when the entire site was hosted by a
simple webserver and front end caches, it works less well in the face of
rebuilding VMs to run in kubernetes. The latency for update publication
isn't a good use of time.

I'd much rather see a fully dynamic database backed website where the
webservers are updated only when the application needs to be updated and
we can instead push data into a database to publish or modify USNs. (I
should point out that this is strictly my opinion on the matter and not
an official team position.)

Thanks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-hardened/attachments/20191019/edc58231/attachment.sig>

From seth.arnold at canonical.com  Sat Oct 19 01:34:56 2019
From: seth.arnold at canonical.com (Seth Arnold)
Date: Sat, 19 Oct 2019 01:34:56 +0000
Subject: [ubuntu-hardened] Making security notices searchable
In-Reply-To: <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>
References: <72d489e8-a940-86aa-d449-7e60e8a93c0f@canonical.com>
 <zarafa.5da47f7a.3bf4.3d1756d14d3b34e6@terreactive.ch>
Message-ID: <20191019013456.GB3304@millbarge>

On Mon, Oct 14, 2019 at 04:00:26PM +0200, Russenberger Dominik wrote:
> Filtering by arches (e.g. amd64+all) might be useful, for the handful of
> bugs that only affect a single architecture.

I don't believe we have recorded any metadata that could be used to
determine this. I'm not sure it would be useful: the linux kernel package
is going to be the largest source of architecture-specific issues, and
quite a lot of those aren't necessarily kernel vulnerabilities as much as
mitigations for hardware flaws.

We publish kernels on a three week cadence and it is a rare cycle that
doesn't have security issues. If you've got nothing but POWER9 boxes you
may not need the mitigations included for eg Intel processor issues but
something like 99% of the kernel code is architecture independent.

I can't think of any other packages off-hand that had architecture
specific flaws. I'm not saying that they don't exist, but they're rare.

Thanks
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 488 bytes
Desc: not available
URL: <https://lists.ubuntu.com/archives/ubuntu-hardened/attachments/20191019/3895b014/attachment.sig>

