Some plans for scmproj

[Alexander Belchenko]

Michael Gliwinski пишет:
> On Friday 27 November 2009 17:39:55 Alexander Belchenko wrote:
>> To be honest initial design was too much VCS agnostic. I've dreamed to
>> have support for several VCS in scmproj, as it done in config-manager.
>> But now I can say this was too ambitious and requires too much efforts.
>> Much more than free time I have.
> 
> Yes, I know what you mean, VCS abstraction requires much effort.  On the other 
> hand since support for accessing foreign formats is getting better and better 
> in bazaar I suppose lack of such abstraction wouldn't necessarily be a 
> problem.

Yes, Jelmer and others do awesome work in this area.

>>> That (ability to have project config with different number of components)
>>> would be really awesome.  I was just looking for something like that :)
>> Currently at my work we using separate configs for different number of
>> components. For example our main application called "dispatcher" is
>> constantly growing in number of supported features and supported
>> external devices (we develop specific embedded system). So every time we
>> get new external device, we develop new "driver" for it and it becomes
>> new component in the project. Current design of scmproj does not support
>> simultaneous existence of configurations with different set of
>> components. This can be emulated with subsets, but it introduces
>> additional complexity.
>>
>> So given all this in mind, I think just moving project.cfg from separate
>> .scmproj control branch to top-level branch of project (into
>> .bzrmeta/scmproj) will help to get rid of extra entity (.scmproj) and
>> get it closer to natural way of working with project.
> 
> One question I have here is, does this mean that a bzr working tree will have 
> to be at the root of the project directory?  Or do you plan to work-around 
> this somehow?

Yes, today you can have one branch in the root of project tree, 
specifying in the project.cfg:

[component "foo"]
RELPATH = .
...

But it was mandatory. Now this will be mandatory to have *some* bzr tree 
at the root of the project. If you don't like this you may use your old 
.scmproj control branch as top tree. This is up to you. Such tree can be 
almost empty (only keep .bzrmeta/scmproj/config) and that's all.

I think there is still enough flexibility.

>> Idea to using separate .scmproj branch has its own pros and cons. The
>> main pros is VCS agnostic way. The cons as I see it now: harder to track
>> all things at once.
> 
> I'm not sure what you mean that it's harder to track things?

Commands like project-command, project-commit et al should be aware 
about existence of separate control branch. This is extra complexity for 
the code.

>>> However, do you have any plans to reimplement alt/variant like
>>> functionality on top of the new architecture?  I must say at the moment I
>>> rely on them heavily.
>> I'm sorry to say this, but most likely alts will go away.
>> The main problem for me now is snapshots. Every snapshot should store
>> the revision ids of all components tips. If you have several alts, then
>> you'll have several snapshots (every snapshot dedicated to every alt).
>> And snapshots should be stored in .scmproj branch (in old design). The
>> problem wait you when you start to merge 2 diverged .scmproj branches.
> 
> So, snapshots are in a sense, similar to specifying revision id for a 
> component/branch in project.cfg, except that they are done after checking the 
> project out instead of before?  (and obviously that there can be multiple 
> named snapshots, etc.)

Snapshots will be updated every time any component will commit new 
revision or by user request. Snapshot saves the data about last 
committed revision-id for all components, so you can restore exactly the 
same state of the project later (modulo uncommitted changes of course).

>> If I commit new snapshots to alt A and B, and you commit new snapshots
>> to alt A and B, and our snapshots diverge, then one of us will have big
>> problems trying to merge those. There will be conflicts, and to solve
>> them we need to merge all components corresponding to alt A and B. But
>> you can't have 2 different branches with conflicts in one directory.
>> See?
>>
>> So we need to merge alt A first, and then merge alt B, and then we can
>> solve all conflicts. But bzr works with entire tree, not with separate
>> files, and there is problems.
> 
> Yes, I see what you mean.  One option would be to make snapshots per-alt but 
> IIUC this wouldn't quite work for your use case.
> 
> On the other hand if a snapshot recorded both branch and revision info, that 
> should work, no?

Mmm, no. Problem in divergence of multiple snapshots. Resolving such 
conflicts will be real nightmare.

Removing alts now also has another benefit: every top-level tree 
(branch) is the alt. They will be physically visible and more or less 
consistent (because there is snapshots).

> I know alts as they are (were) could lead to problems but a concept of a 
> light 'variant' as a way of specifying multiple branches for a component and 
> specifying which one is 'active' (i.e. which one is used when 'getting' the 
> project) could still fit in, IMO.

The recent idea of Ian to emulate colocated branches with light checkout 
and treeless repo with branches inside the tree (in .bzrbranches 
directory) makes me think we can have more or less good colocated 
branches. Alts is another way to emulate colocated branches but for 
nested branches case. It was intended to be use with boxed workspace, 
but Ian idea makes this boxed workspace idea less relevant.

> 
>> So my new attempt is to use different top-level project directories to
>> keep different variants of projects (different alts). It will be similar
>> to other concepts existing in bzr.
> 
> Do you mean having separate diverged branches of the control branch for each 
> alt?

Yes, something like that.
Every alt is different from another by 3 parameters:

1) Set of components
2) Component branches origins
3) Snapshot (tip of every component branch)

First 2 items specified by project config, this config belongs to top 
tree of the project, so top tree automatically forms alt. Snapshot lives 
in the top tree too.

> That has it's own problems (maintaining common things for one, as you'd have 
> to cherrypick changes between branches manually), but could work if there's 
> no better option.

Well, today I have to create new control branch if my new alt should 
have different set of components. So this problem of sync between two or 
more project.cfg exists and now, but less often. So yes, there is always 
pros and cons.

>>> I actually see the scmproj plugin as something more than externals/nested
>>> trees support.  In fact I was looking into a possibility of adding
>>> project/component resources, for example my current use case is creating
>>> additional files, hardlinks, softlinks, etc. when the project is checked
>>> out/updated, etc.
>> As I said it looks closer to what bzr-builder doing, or to what
>> config-manager do. As I said the initial project clone/update problem is
>> not so big problem for me. But snapshots really hurts us too much.
> 
> I did actually have a look at bzr-builder but AFAICT it is more oriented 
> towards the process of building packages and similar, i.e. you have a recipe, 
> you run it, you get some result.
> 
> My use case is more for maintaining a project composed of multiple components, 
> so I find scmproj's ability to run e.g. status for all components very 
> useful.  The only peculiarity here is that I can't have a branch in project 
> root but still need to keep some links there, pointing to files within the 
> actual component branches.

You can have a branch in project root. Why you think you can't? I'm not 
quite understand your point.

>> Imagine our project with 15+ components and if you want to recreate past
>> state of all branches? Currently we're using tags. But it's very
>> limited. Snapshots should solve this.
> 
> I don't know if I understand how snapshots should work, but if they record 
> revision ids of component branches in a config file in project control 
> branch, wouldn't you have problems e.g. when a component is removed from 
> project.cfg in control branch?

Snapshot will be stored separately of project config, and will be 
updated often enough. So, no. There should not be a problem if component 
will be removed. Snapshot file will be updated based on current config.

>>> I will follow up next week, am a bit short on time today :)
> 
> So I ended up swamped with work :)  Sorry for the delay.

Not only you. No need to sorry, I appreciate your feedback regardless of 
delays.

Thanks!