Provisional snap for DUB (D language package/build manager)

[Didier Roche]

Le 27/10/2016 à 01:13, Joseph Rushton Wakeling a écrit :
> Hello folks,

Hey Joseph,

It would be great to have D support! Thanks for working on that :)
I'm going to try to answer to some of your questions, but I'm sure
Sergio (CCed) would be able to give deeper insights on some parts as
being the snapcraft upstream.
>
> Following my attempt to create a snap for LDC (LLVM-based compiler for
> the D programming language), I thought I'd have a go at another
> D-related project and snap DUB.  This is a package/build manager
> that's popular in the D community, and so having it available as a
> snap could be very useful.
>
> First caveats: this is a command-line developer tool, so some of the
> same limitations are going to apply as were identified with the LDC
> snap (access to system tools, linking against host-system development
> libraries, etc.).  I also had to take some temporary shortcuts to
> ensure that the packaged DUB had a D compiler available.
>
> The draft package is available here:
> https://github.com/WebDrake/dub.snap/pull/1/files
>
> A few things on what went into it, and corresponding requests for
> feedback.
>
> First: DUB can be built two ways; either by calling a shell script,
> `build.sh`, or by DUB itself via an existing install.  I couldn't
> identify an obvious way to handle the former, so (given that DUB is
> packaged in Ubuntu 16.04) I opted to create a `dub.py` snapcraft plugin.
>
> I'm not a Pythonist myself, so any feedback on that code would be
> welcome.  But I have a couple of other questions:
>
>   * Is there any way for the plugin to ask for a `dub` instance to be
> available?
>     Currently I'm just specifying `dub` as a build dependency in
> addition to the
>     plugin.
>
>   * Assuming I'd wanted to go the `build.sh` route, is there any way I
> could
>     have achieved that with existing plugins?

I would really advise going the build.sh route. Doing a custom snapcraft
plugin as you started is definitively the right path.
If you take other plugins having similar problematic, like Go, here is
how they work:
- Golang binaries (compiler tools and standard library) installs itself
in parts/<part_name>/go. This logic is from the golang plugin itself.
You will need in your case to ensure LDC is relocatable and can be
downloaded and extracted this way. You can then build without depending
on any package or distribution.
- Then, the plugin run go build, go install, changing the paths. The
plugin can define optional parameters that it will use then. In you
case, I think the plugin will setup environment variables to refers to
local LDC and standard D libraries. Then, you can have one option being
"entry-point", which will refers in your DUB snap case to build.sh. The
plugin would execute it.


>
> DUB doesn't have an `install` option, only a `build` one, which
> creates some problems in terms of determining what files go into the
> snap.  I compromised on a short term workaround where the plugin
> copies everything in the `build/` dir of the part being built, and the
> user is expected to manually specify what bits of that should actually
> wind up in the final snap.  If there is a target path where the built
> files are placed, the plugin can handle that, too.
>
> Two questions there, too:
>
>   * Is there any way to filter that stuff out already at the staging area
>     or earlier?  I tried using `filesets:` but didn't have much luck.
>
>   * Is there any way to detect what extra files have been created
> after the
>     build completes and just use those?

From your questions and the statement that there is no explicit install
steps, I think you mean that D programs just build in tree (snapcraft
tries to encourage builds out of trees) and then, the install step for
people is to copy the resulting binaries and assets manually?

If so, I can see 2 paths here:
- either you expect people to explicitely list assets and binaries that
should be installed (this could again be a custom option of your plugin)
- either you try to be starter, as you told, and detect extra files
being created in the parts/<part_name>/build. There is no built-in
features in snapcraft for this as far as I know, but, it would be quite
easy to build it as part of your plugin: list at the start of the build
phase files in parts/<part_name>/build and redo the same thing after the
build. Then, in the install phase, use that list (that you store on
disk) to copy newly created content in parts/<part_name>/install. Then,
snapcraft will pick it up from there. Note that I think you still need a
manual options for your users to list eventual assets (or they can use
the dump plugin in another part).

Just pick the flow which would be the more natural to the D plugin users.

If you prefer to copy everything and have your users filtering from the
stage area to the prime area, you can use the "snap:" stenza (don't use
filesets for now if you don't plan to repeat the file list name).

>
> Since DUB needs a D compiler to be able to build anything, I first
> tried to just use the existing Ubuntu `ldc` package, and then reverted
> to copying the entirety of my LDC snap setup because it proved simpler
> to get working.  That's very much a temporary measure, I hope, but
> again, questions here:
>
>   * Is there any way for me to give the DUB snap access to the ldc2
> and/or
>     ldmd2 installed in my LDC snap package, other than explicitly
> defining a
>     `d-compiler` interface and submitting it to `snapd`?
>
>   * Is there any way of defining/using/testing an interface short of
> building
>     my own local snapd?
I would advise in a first step to consider the D compiler used by DUB to
build other things to be part of your snap. You want to control the
version in it and consider it as part of your dependencies you need in
your snap. We can discuss later about an interface, but I would really
go that way as a first step.
There is no way right now on definining new interfaces apart from
building them in snapd, and so, use your own local version. (That's the
reason why I advise you to not bother with this right now, let's take an
easier path first, figure the rest, and then, we can rediscuss)

>
> The last issue is a bit weird: DUB installed by the current snap
> package can build D projects, but exits with an error: "Bad System
> Call".  It's been difficult to track down what was going on here
> (`strace dub build` for example was not very edifying, presumably
> because of the containerization) so any advice on how to identify
> what's wrong?

I would look at /var/log/syslogs. Apparmor and seccomp denials are
listed there. Note that if you didn't already, you should really start
developping your snap in devmode. That way, it will get confinment out
of the equasion to get your relocatable code and dependencies working.
Then, we can turn on confinement and figure out those issues to be able
to publish in the stable channel.

>
> At a guess, it's related to the step where the built binaries are made
> executable, because they get created, but are _not_ executable at the
> end of the process.  This is a bit of a surprise, because the
> standalone LDC snap doesn't have this issue (things get made
> executable just fine).
>
> I did try making ldc2 and ldmd2 apps of the DUB snap too, but that
> seemed to make no difference.  OTOH that might be related to how the
> snap-packaged DUB would invoke the D compiler (because presumably
> you'd need to call `dub.ldc2` in order to ensure that you get the
> `home` plug working properly, while DUB probably calls `ldc2`
> directly).  Anyway, any thoughts on what could be going on here?

Don't make your snap calling other snap endpoints. It's better from your
own snap code to ensure that your local ldc2 is in your snap $PATH, and
call ldc2 directly (it will then use the parent process confinment
profile, of course). I think that will help you a little bit more to
control what you are calling at, and how.

I hope those advice would help you!
Cheers,
Didier

I hope