Help sell this to management?

[Jac Jones]

Hi,

currently our organisation performs around 1 to 2 releases for different customers each week. Each customer has their own branch as the functionality is slightly different for each - so this results in a huge building/testing and then merging the changes back into trunk overhead. I want our company to perform one release for all customers (with customer variation built on top of the trunk), say once a month.

To help my chances of selling this to management I would like to know of any successful companies that are currently doing this? Other suggestions welcome too!


many thanks,

Jac

[Ilja Preuss]

Originally posted by Jac Jones:
currently our organisation performs around 1 to 2 releases for different customers each week. Each customer has their own branch as the functionality is slightly different for each - so this results in a huge building/testing and then merging the changes back into trunk overhead. I want our company to perform one release for all customers (with customer variation built on top of the trunk), say once a month.

I see you talking about two things here.

I can empathize with the wish to reduce the merging overhead. We also have a lot of different customers with customized functionality, but we solved the problem via configurations and plugins. I'm convinced that it is a much more economical way.

What I have trouble understanding is why you want to go to a monthly release cycle. Without knowing more about your situation, I can only see the drawbacks: reduced feedback and decreased return on investment. What advantages do you hope for?

To help my chances of selling this to management I would like to know of any successful companies that are currently doing this? Other suggestions welcome too!

We are successfully distributing a highly customized system to a dozen different customers without branching the code line, just by using configuraiton files and plugins. We don't have a fixed monthly release cycle, though - our releases are very much customer driven. Inquire if you want to know more.

I have one more important suggestion, though: You can only sell what people want to have. So your first step should be to find out what management would get out of your proposed changes, and then sell *that*. You might even find out that you can't sell (parts of) your changes - because they aren't a good idea from the point of view of management. Then you should think hardly about why *you* want those changes in the first place...

[Jac Jones]

Ilja,

thanks for your reply.

We too have the ability to configure our product for different clients via configuration files, etc, which is want we want to leverage - rather than branching.

The primary reasons for going to a monthly release cycle are:
1) reduce testing duplication. Currently we test each release, so if we release to 6 different customers in a month then we are testing the core functionality 6 times, plus the variation of each client (the product for each client is branched and then those changes are merged back onto the head of CVS once released). Whereas releasing once per month means we would only test the core functionality once.
2) Moving to a monthly release cycle would enable us to develop functionality for different clients off the same codebase, ie without branching.

If we continue to release to customers at different points in time, ie twice per week, then its extremely difficult to do this without branching. For instance, how do you release for customer A next week if changes for customer B & C will still half developed, and you are all working off the same codebase?

Finally - what would management get out of this? Well, you can imagine the immense overhead of branching, testing, merging etc - so developing off one codebase would mean greater productivity, higher quality code, greater ability to respond to market opportunities etc.

Thanks for your feedback, I'd like to hear more of your experiences with this.


thanks,

Jac

[Ilja Preuss]

Originally posted by Jac Jones:
We too have the ability to configure our product for different clients via configuration files, etc, which is want we want to leverage - rather than branching.

Sounds like a good plan to me.

The primary reasons for going to a monthly release cycle are:
1) reduce testing duplication. Currently we test each release, so if we release to 6 different customers in a month then we are testing the core functionality 6 times, plus the variation of each client (the product for each client is branched and then those changes are merged back onto the head of CVS once released). Whereas releasing once per month means we would only test the core functionality once.

Reducing testing overhead is a good goal. Lengthening the testing cycle in my experience only leads to trouble, though: you will find problems way after they were introduced, which makes it much harder to analyze and fix them.

The superior way to reduce the testing overhead is to agressively automate the tests. We have a growing suite of over 6000 unit tests that automatically run a dozen times a day. When you break one, a siren in the team room notifies everyone, typically just a few minutes after the change got checked in.

2) Moving to a monthly release cycle would enable us to develop functionality for different clients off the same codebase, ie without branching.

If we continue to release to customers at different points in time, ie twice per week, then its extremely difficult to do this without branching.

That's not my experience. It might take some creativity, but once you got the right ideas, it's quite simple.

For instance, how do you release for customer A next week if changes for customer B & C will still half developed, and you are all working off the same codebase?

We use mainly two options: If the changes for B & C are only interesting for B & C, they go into plugins that don't get delivered to A, so A wouldn't notice anything about it anyway.

If the changes are so general that A also will see them once they are finished, we temporarily introduce conditionals that remove the feature in the production environment. Typically that's just an if-statement that prevents the user interface elements for the feature to show up when the "development" system property is not set.

Another general strategy is to break down the features into very small, self-contained "User Stories". That is, when breaking down features, we try to break them down into smaller features, so that each step to the full feature makes business sense on its own. That not only reduces the number of "half baked features" you have at a time, it also reduces risk and increases scheduling flexibility, because you can stop the development of a feature midway without the whole effort being lost to nothing.

Finally - what would management get out of this? Well, you can imagine the immense overhead of branching, testing, merging etc - so developing off one codebase would mean greater productivity, higher quality code, greater ability to respond to market opportunities etc.

Yes, I don't think I would have trouble selling that to management.

Slowing down the release cycle, though, will probably reduce the ability to respond to market opportunities, so I see trouble in selling that part. But as probably already has become obvious, I also don't think that it's really necessary - I'm positive that your team can find a way to work from one code base without having to slow down the release cycle, if you want to!

[Susan]

Ilja,

thanks for the reply. I understand what you are saying, however I feel at this point we arent in a position to undertake what you are currently doing, eg our lack of test coverage, and the culture of the company. Over time hopefully we can get to where you are at, but I feel its too big a jump for us now.

So at this point while a monthly release cycle isnt ideal, I feel its definitely a step in the right direction and does give us breathing room to develop off the one codebase for different customers.


thanks,

Jac

[Peer Reynders]

Jac,

Why do feel it necessary to have (at least two) separate profiles?

And why does Susan (formerly Susan Oso) no longer comply with the JavaRanch Naming Policy?

[Frank Carver]

I am currently in a pretty similar position. I'm developing a combined web and WAP user portal for an application. We have a steadily growing number of customers, each of which wants a combination of cosmetic changes, application--page-flow changes and occasional internal code changes. We are continuing to develop the system but also need to deliver all sorts cf customer versions, demo versions, test versions and speculative experimental versions.

We automatically test as much as we can, but testing how each page of each customer variant looks on each of around 100 mobile handsets is not something we have yet been able to automate. I'm currently putting in place a system to automatically validate the generated pages for the full set of variants and user agents, but testing how they actually appear and work on each phone is still a long and tedious manual process.

The nature of the business is that we have to adapt our release dates to each customer's needs, although we have a fair degree of control of the scope of each release.

Because handset testing is such a large hit, we have had to be creative about how we structure the project to avoid clashes and re-testing wherever possible. The nature of the business is that we have to adapt our release dates to each customer's needs, although we have a fair degree of control of the scope of each release.

The application itself is pretty pluggable, but not every possible variant can be accommodated by a pluggable facility - sometimes it just makes overwhelming sense to support different variants of a single file or resource.

The way we have handled this is to have a build process which (essentially) merges two eclipse/subversion projects with the same structure to produce a combined project which is built and deployed.

A "core" project has a generic solution with all default values etc. Then, for each customer variant there is an identically structured project which overlays or replaces any variant files from the core. The delivered artefacts are then built by running an ant build from the merged project.

So far this has worked very well, and we are able to isolate a lot of changes without making every possible decision pluggable.

[Ilja Preuss]

Hi Frank,

to me, what you are doing sounds like it could lead to maintenance hell in the long run, but if it works for you, great!

Regarding automatically testing the appearance of pages on different handsets, have you thought about using Gold Masters? That way you only have too manually look at each combination once at the beginning, and then when something changes.

[Frank Carver]

to me, what you are doing sounds like it could lead to maintenance hell in the long run, but if it works for you, great!

There's always that worry, but currently, other options seem worse.

Regarding automatically testing the appearance of pages on different handsets, have you thought about using Gold Masters? That way you only have too manually look at each combination once at the beginning, and then when something changes.

I'm sure we'll get to that for each customer variant once it begins to stabilise, but while developing each new one we have nothing to compare against. It's the "testing web page look and feel" problem on speed.

[Pj Murray]

Originally posted by Jac Jones:
Hi,

currently our organisation performs around 1 to 2 releases for different customers each week. Each customer has their own branch as the functionality is slightly different for each - so this results in a huge building/testing and then merging the changes back into trunk overhead. I want our company to perform one release for all customers (with customer variation built on top of the trunk), say once a month.

To help my chances of selling this to management I would like to know of any successful companies that are currently doing this? Other suggestions welcome too!


many thanks,

Jac

If the commercial reality is that customer releases are needed a couple of times per week, you can't do much about that.

So it becomes your job to do everything you need to automate the process - and perhaps have extra staff dedicated to this specific issue.