Which challenges do you experience the most with CI?

[Paul Duvall]

For those of you that are implementing CI on your projects, I am interested in hearing the challenges that you may be experiencing. Here are some of the challenges I've seen. Is anyone experiencing these and how do they manifest?

1) Getting developers to commit all code daily?
2) Developers committing code that breaks the build?
3) Integration builds continually broken or staying broken for several hours or even days?
4) Lingering broken tests in Integration Build (tests not failing the build)
5) Others?
[ August 30, 2007: Message edited by: Paul Duvall ]

[Ilja Preuss]

Originally posted by Paul Duvall:
1) Getting developers to commit all code daily?

We had a bit of this at the beginning (years ago). Some (developers from outside who "helped" the team at that time) even complaint that they had to resolve conflicts so often, because they "weren't able" to check in for two weeks, and so much other changes were happening in that time. :roll:

Another interesting first reaction to Cruise Control was that one of our core developers installed an email filter so that he wasn't any longer "pestered" by all those emails telling him that a build failed...

2) Developers committing code that breaks the build?

Happening from time to time. That's one of the things that make me wonder whether a manual CI process might work better.

Another thing that is rather annoying is that CVS commits aren't atomic, so that every now and then we have a build failure simply because of an inconsistent checkout.

3) Integration builds continually broken or staying broken for several hours or even days?

We had this problem before we reduced our build time from two hours to an average of five minutes. It was horrible.

[Paul Duvall]

Another interesting first reaction to Cruise Control was that one of our core developers installed an email filter so that he wasn't any longer "pestered" by all those emails telling him that a build failed...

Ah-ha! I like to call this antipattern "Spam Feedback". There are two sides to this. The first is whether this person is acting professionally and whether the development team's culture has embraced the concept that broken builds are high priority. The second side, and more interesting to me, is whether using an e-mail only approach is an effective communication mechanism for this person or team. In chapter 9, Continuous Feedback, I go over many types of feedback mechanisms including, Task bar notifications, RSS, large displays, sounds (more for co-located teams), Ambient Orb, Lava Lamps and so on. A team may implement some combination of these that works with their team. I rarely find that only one communication mechanism works for a team

Developers committing code that breaks the build?
Happening from time to time. That's one of the things that make me wonder whether a manual CI process might work better.

This is a good reason to go with a manual CI process as long as the team is disciplined to run the integration build *every* time. Other teams use a private build on their workstation that runs the same build that the integration build server will run (this prevents many of the common errors). Other teams may choose not to run a private build (because of the time involved) and will, instead, segment their project into smaller teams that each have their own build so that the build errors are smaller. There is a master integration build that occurs based on the success of these other builds (Owen Rogers from ThoughtWorks discusses similar concepts in a paper called "Scaling Continuous Integration")

Another thing that is rather annoying is that CVS commits aren't atomic, so that every now and then we have a build failure simply because of an inconsistent checkout.

Too bad you can't switch to Subversion Have you tried increasing the quiet period time in CruiseControl?

Thanks for sharing Ilja!
[ August 30, 2007: Message edited by: Paul Duvall ]

[Gian Franco]

We are quite new to (automated) CI in our project...

Although our manual build scripts where written some time ago, and the build and versioning procedures were followed reasonably consistently, it looks like introducing CI is fleshing out the procedures to a much more workable whole.

The issues I've encountered introducing CI where/are:

Maintenance of the build script: I used to have a copy of the same build script for every project in cvs, now there is only a central copy in it's own project. The latter project's files physically have the same relative location be it in CC or in a developers' local working environment. Especially at the begin there was a lot of refactoring involved.

Versioning policy: in a second stage other types of improvements are surfacing, like when to create versions with respect to successful builds, and the phase a build is in (i.e. still in development, or in production and being supported).

Resource dependency of Junit tests: the issue of junit tests that I can run locally but not on the CI machine...

Technical issues: these have been surmountable issues, but have costed time, and range from cvs client problems to CC windows service issues...

Gian
[ August 30, 2007: Message edited by: Gian Franco ]

[Ilja Preuss]

Originally posted by Paul Duvall:
Ah-ha! I like to call this antipattern "Spam Feedback". There are two sides to this. The first is whether this person is acting professionally and whether the development team's culture has embraced the concept that broken builds are high priority.

Yes, it was definitely a culture problem. There wasn't even much feeling of a team - in the sense of a shared responsibility - at that time.

That same developer today can't even imagine how other teams can work without a CI server...

The second side, and more interesting to me, is whether using an e-mail only approach is an effective communication mechanism for this person or team. In chapter 9, Continuous Feedback, I go over many types of feedback mechanisms including, Task bar notifications, RSS, large displays, sounds (more for co-located teams), Ambient Orb, Lava Lamps and so on. A team may implement some combination of these that works with their team. I rarely find that only one communication mechanism works for a team

I totally agree. Today, additionally to the email, we have a traffic light showing the build state, sounds for state changes and lava lamps for a side room.

Actually, for some time now, we are using a homegrown software to show the build state that supports not just the two states "success" and "failure", but also "failure noticed", "fix in progress" and "fix committed".

Too bad you can't switch to Subversion.

Actually, we are actively considering this option.

Have you tried increasing the quiet period time in CruiseControl?

The problem is more with checkins that are happening after the modificationset has been determined, while our build script is doing the cvs update.

This has become much better since we use a compound modification set, first checking for each module whether it has been modified at all (indicated by a "touch" on a specific file), and only then asking the CVS server for the actual changes - and also only updating those modules with changes. (Our project consists of 200+ modules.)

I also have an idea for a final fix: use the same time stamp for both the modification set and the cvs update. Until now, there always have been higher priority things to do, though...

[Paul Duvall]

Resource dependency of Junit tests:[/b] the issue of junit tests that I can run locally but not on the CI machine

Can you expand on this? Why are you unable to run on the CI machine?

[Paul Duvall]

Originally posted by Ilja Preuss:
That same developer today can't even imagine how other teams can work without a CI server...

Nice!

Actually, for some time now, we are using a homegrown software to show the build state that supports not just the two states "success" and "failure", but also "failure noticed", "fix in progress" and "fix committed".

I really like this. One of the techniques I've used with the Ambient Orb and Continuous Feedback is to notice trends rather than finite states (i.e. Success or failure). For instance, as the code duplication increased the color of the Orb would change from green to blue to purple to pink to red - as it got worse. Code becomes unmaintainable a little bit at a time and this can be an approach for noticing it while it's occurring.

[Gian Franco]

Originally posted by Paul Duvall:
Can you expand on this? Why are you unable to run on the CI machine?

Yes...on the local environment, I have control of, for example, the database and its contents. When I run a local build with corresponding tests, I always have some tool like Toad (or Sql Developer or Golden, etc...) to intervene if data needs to be removed or added accordingly.

At the moment I don't have fixtures to do the above for me, that's one reason why I can't run the test on the CI machine. One solution mentioned in a previous post is to stub out the database.

Another reason the tests depend on my local environment, is because it would take too much time to finish a build if I include database related tests on the CI machine. Locally it's no issue.

[Ilja Preuss]

Originally posted by Paul Duvall:
I really like this.

I too, mostly. I sometimes wonder whether it might also be a process smell that we need it...

One of the techniques I've used with the Ambient Orb and Continuous Feedback is to notice trends rather than finite states (i.e. Success or failure). For instance, as the code duplication increased the color of the Orb would change from green to blue to purple to pink to red - as it got worse. Code becomes unmaintainable a little bit at a time and this can be an approach for noticing it while it's occurring.

Sounds interesting. How is your experience with this approach?

[Gian Franco]

I notice your teams (Ilja, Paul) are on a whole different level with respect to CI than mine.

There is one important challenge I forgot to mention, and it is more of organizational nature. The introduction of CI lagged behind because of the company's focus on external projects, in house development used to receive short bursts of attention (by different developers), alternated by long periods of inactivity (or just sporadic support). What happens when this is the case is that justifying CI, or just taking the time to introduce it, is not trivial.

Maybe one day we'll have an Ambient Orb as well
[ September 01, 2007: Message edited by: Gian Franco ]