Can I copy the .m2 directory as a backup?

[Billy Sclater]

I'm relatively new to maven, so please forgive if my question is slightly silly!!

The maven project I am working on has a 'master' branch (created via git).
I am working on a 'test' branch I have created.

Master if often broken ;( My test branch I like to keep as a working branch.
The problem is that when I switch to master and then back to my test branch, I find that the test branch no longer works!!

I am guessing that the .m2 directory has become corrupted by the master branch.

The solution I use is to have 2 copies of the .m2 directory. .m2_test and .m2_master.
I simply rename the .m2_test directory .m2 when I am on test, and vice versa.
This way my working .m2 directory remains uncorrupted.

I also make sure I only mvn clean install --offline when on test to avoid maven pulling any corrupt jars.

Ok some silly questions!!:

#1 In theory should this work and keep my test branch working (together with it's own .m2)?
#2 Is there a more sensible way of doing this?

[Jayesh A Lalwani]

1) Yes it will work. THe .m2 folder is merely a copy of the artifact repository
2) Yes of course. This is really the worst way of doing things . sorry
THere is a better way and a best way

Better way
a) Delete your .m2 folder when you switch branches. The first time you do the build right after switching branches, you should get all the jars back in your .m2 folder. Assuming you are not doing anything wierd like copying jars into .m2 manually
b) One of the principles behind maven is that each build should be self contained. When you do mvn clean install on the source folder, the build should get anything and everything it requires. So, as such, if you are using branches for development, each branch should be self contained. When you run the build, it should download all the 3rd party libraries it needs into your .m2 folder, and any artifacts that you yourself generate should be installed into your .m2 folder. I suspect the reason you are having problems is because you are breaking the principle of self-containment. Your build is using something that is left over from the previous run of the build. So, when you switch from master to test, you see the test build using artifacts generated from the build of the master branch

Making the build self-contained might be a harder problem to solve. It's much better to make your builds to be self-contained from the beginning. However, taking the effort to clean your build now will give you long term benefits

[Billy Sclater]

Thanks(:

About 3rd party jars. The trouble is 'I think', many of the jars downloaded in the build process are not just bog standard 3rd party jars such as apache commons, junit etc. Many of them are proprietary made by my company and are used by the project. At least when I look in .m2 I see many jars named my.company.name.somethingsomething.jar. The code in these jars is periodically broken! So sometimes deleting .m2 and doing an online mvn clean install will work, sometimes it won't. It depends on the state of the code in those 'company made proprietary jars'.

Does this render 'copying .m2' the only solution?

Also, why is copying .m2 'the worst solution?!! (I am psychically guessing you will answer 'it defeats the object of maven'!).

[Jayesh A Lalwani]

Copying folder is the worst solution because it defeats the principle of self-containment. Imagine a new developer joining a team:- your onborading process should be
a) Download source from here
b) Run mvn clean install

If you are following alll the maven conventions s/he should find everything in the conventional places. If you are not, s/he should be able to figure out by looking at the maven pom.xml

You shouldn't have any steps outside of this. It creates points of failure in your process. If you have manual steps, then Murphy's law dictates that at some point, someone will screw up the steps

-------------------------------

We had the problem of broken jars blocking everyone in the company at my previous company, and the way we did is by implementing a multi-step promotion process for the artifacts that we builod. Having a CI server helps in automating the process. So, the steps were

a) Developer makes changes to his own module and tests with the SNAPSHOT version of other jars present in our internal artifact repositorychecks in
b) CI server sees the check in and runs the build along with unit level tests
c) If Unit level tests pass, CI server installs the artifacts into a CI environment, and runs integration smoke tests
d) If smoke tests pass, CI server "promotes" the artifacts by deploying it to our internal artifact repository as snapshot versions. The smoke tests are comprehensive enough to answer the question "Will deploying this build to the development team block everyone?" If the answer is No, then it's ready to be promoted. It doesn't mean something won't be broken. All it means is everything won't be broken. It limits the damage from propagating to all 120 developers and 80 QA
e) CI server triggers a series of automated "QA" tests, which were really comprehensive integration tests
f) If these tests pass, the artifacts are promoted to RELEASE versions in artifact repository. Now, these artifacts are Gold candidates.
g) QA pulls these gold candidates, and starts doing other tests that cannot be automated:- ad hoc tests, tests that are not automated yet, performance tests, blah blah
h) Once QA certifies the build, the build gets packaged into a "binary build" that gets shipped to the Infrastructure & Support people. They take it from there and run UAT or whatever they want to do


So, as far as developers are concerned, their responsibility was to monitor the CI builds until step d) They have to be responsible for unit level tests and smoke test. One day the build goes red, the leads go on alert. Second day, the managers.. and so on. You keep the build red for enough days, and the CTO will come sit at your desk, and I plead the fifth on whether that ever happened or not :p

The problem of developers stepping on each other toes is a huge problem in large organisations. Maven gives you the tools to solve the problem, but it doesn't solve the problem for you. To solve the problem, the developers, as a community, have to agree that they are not going to check in broken code, setup automated builds that can catch problems before they propagate, and make a promise to each other that code that does block other people will get the highest priority. As they say, always check in code assuming that the person using the code is an ax murderer who knows you address. The CI server is much nicer than an ax murderer.

-------------------------------------------------------------

I am assuming that getting this process setup is a lot more that you asked for, and you are more or less looking for a shortcut in the short term. You might want to try switching between repositories a tad bit easier. Try this out

Maven defines the local repsitory inside a file contained in MAVEN_HOME/conf/settings.xml. Search for localRepository in your settings.xml. If you are able to modify the localRepository at build time, then you can keep 2 .m2 folders. localRespository defaults to ${user.home}/.m2/repository, which is why you have the m2 folder in the place that you have.

1) I am pretty sure this will work: You can specify localRepository using environment variables. So, you can set it to ${user.home}/${env.BRANCH}/.m2/repository . Now, if you go and setup a enviornment variable named BRANCH = master, it will create the .m2 folder in USER_HOME/master/.m2. You change it to test, and your m2 folder will become USER_HOME/test/.m2.. It's an easy way to switch your m2 folder per branch by changing environment variable

2) I am not sure if this works, but if it does, it will make it easier on you. Try using maven properties in localRepository. If you can use maven properties, then you can override the maven property using profiles. You can have a master profile, and a test profile, and running mvn clean install -Pmaster will make maven use the master .m2 folder

[Billy Sclater]

Thanks Jayesh. I assume that nobody here is aware of this 'broken jar' phenomenon. Or perhaps they are, and didn't warn me because they assume that I should know that!
I think I'll got with your latter solution about creating a local repo for myself in the settings.xml! But I certainly will forward what you said to our release manager!

Thanks for the advise(:

[Jayesh A Lalwani]

A lot of times developers won't complain about broken jars and/or come up with workarounds, because it breaks the bro-code. You complaining about a bug? What are you? QA? *said with a sneer* Ultimately, someone higher up has to realize how much do broken jars cost in terms of lost effort, and compare that with the cost of putting in measures to avoid the lost effort. I was lucky to be in a company where everyone in the company coded, even the CTO. It helps them see that what the developers are make doing with is stupid. If you have non-techie managers, a thing like this becomes a uphill battle, and many times, it's just easier to come up with your own workarounds.

[Jeanne Boyarsky]

Nobody has mentioned this yet, but see if you can pint to versions rather than SNAPSHOTS. This minimizes the chances of getting a broken jar in the first place.

[Billy Sclater]

I mentioned your suggestions to the release manager. He is in the same camp as you guys. He liked Jayesh's idea and said that I could also use mvn clean install -Dmaven.repo.local /USER_HOME/test/.m2--offline. Apparently, there is a plan to switch over to using versions instead of snapshots ...in the future, but not yet!! Thanks for your help!

[Tim Holloway]

Billy Sclater wrote:I mentioned your suggestions to the release manager. He is in the same camp as you guys. He liked Jayesh's idea and said that I could also use mvn clean install -Dmaven.repo.local /USER_HOME/test/.m2--offline. Apparently, there is a plan to switch over to using versions instead of snapshots ...in the future, but not yet!! Thanks for your help!

Just to clarify: snapshots are for nightly builds or the equivalent. The only snapshots you want to be working with are the ones for the stuff you are actually working on.

For background libraries - whether third-party or in-house, you'll achieve maximum stability and reliability if you peg your pom to specific versions (not simply "latest"). That is one of Maven's primary strengths, after all.

[Billy Sclater]

Hi, I tried the solution of having 2 repos, one in USER_HOME/test, and one in USER_HOME/master. Here's what I tried to do:

When master is stable build test branch with:

mvn clean install -Dmaven.repo.local=/path/to/test/repo

When master is unstable add the --offline switch

The problem is after issuing the command it downloads 'some' jars, but complains that it can't find others!
However, maven never complains if I just use a simple 'mvn clean install' which of course uses the standard .m2.

Quite simply, it doesn't appear to work); (for my situation on my company's project anyway);).

I read a discussion on stackoverflow:

link to stackoverflow discussion

It goes through several ways of having 2 repos, but claims theses techniques are all error prone. The discussion concludes that it's probably best just to have 2 user accounts, one for test and one for master. And to only update the 'test' account when master is stable. One of my co-workers uses a VirtualBox with 2 different VM snapshots, one containing a bleeding edge master repo, and one containing a stable test repo.

If what the article says is true then 'even' my original solution of just having 2 folders (one for master one for test) and then renaming the desired to .m2 when required, might be better ;(

What do you guys think?

#1 Using a maven switch to point to a diff local repo, or adjusting the settings.xml to do so.
#2 Using 2 diff user accounts.
#3 Using 2 diff VM snapshots.
#4 Using 2 diff repo folders that can be renamed to .m2 when desired.

Which is best?

[Hauke Ingmar Schmidt]

Billy Sclater wrote:Hi, I tried the solution of having 2 repos, one in USER_HOME/test, and one in USER_HOME/master.

As already stated this is not a solution but a hack.

Billy Sclater wrote:It goes through several ways of having 2 repos, but claims theses techniques are all error prone.

Exactly. And they are right.

Billy Sclater wrote:
#1 Using a maven switch to point to a diff local repo, or adjusting the settings.xml to do so.
#2 Using 2 diff user accounts.
#3 Using 2 diff VM snapshots.
#4 Using 2 diff repo folders that can be renamed to .m2 when desired.

Which is best?

If you or your organisation don't want to use Maven: Don't use Maven. With these hacks you get additional problems without any of the benefits. You just have a step in your build process that says "mvn". But you are not utilizing it. Drop it completely. Use shell scripts, or ant, or a wiki page that explains how to collect the dependencies and how to call the needed tools manually. Well... that is exactly what you have to do now but you add a mvn call somewhere in the list.

Of course it pays to learn and use Maven. But if your organisation doesn't even use version number then it is useless for you.

[Billy Sclater]

Wow! Thats certainly 'saying it how it is'! Actually, I'm a new starter and learning maven. Having broken jars in the repo and broken code on master has pretty much been the norm since I started at the comapny. I thought this was 'normal' and that 'maven was poor build tool'! From what you guys say, quite the reverse! You all seem to be saying:

#1 Building with maven should be a 2 step process S1:Pull the source code S2:mvn clean install.
#2 Using versions as opposed to snapshots is more stable.
#3 There are ways around broken repos but these are 'hacks'. The proper way to do things is not to have a broken repo in the first place (see #2).

I had a discussion with my project manager and release manager (who are also new starters) and they are in your camp. There is a plan to move to versions, but it is a long way off. For now looks like I have no choice but to 'hack'!

[Hauke Ingmar Schmidt]

Billy Sclater wrote:Wow! Thats certainly 'saying it how it is'! Actually, I'm a new starter and learning maven. Having broken jars in the repo and broken code on master has pretty much been the norm since I started at the comapny. I thought this was 'normal' and that 'maven was poor build tool'! From what you guys say, quite the reverse! You all seem to be saying:

#1 Building with maven should be a 2 step process S1:Pull the source code S2:mvn clean install.
#2 Using versions as opposed to snapshots is more stable.
#3 There are ways around broken repos but these are 'hacks'. The proper way to do things is not to have a broken repo in the first place (see #2).

Just for clarification: The repo is not broken. The code is broken which is a different problem. The problem with the missing version number is that you just can't reference a specific version of the code. If everything is always 0.0.1-SNAPSHOT (you ARE using version numbers, you don't set them) then you can't say: Always use the version from 2014-07-04 and switch only if the new version works. Even this I would consider a hack but at least you could define a working version.

IMHO your first step should be to ensure that the version in main is always working (in the sense of compiling). It is quite simple to set up a CI server like Jenkins for this. Even if you don't have extensive unit tests you could configure this to checkout the code at intervals (at least daily) and build it. At least you then know if it builds and can check in the version control who broke it. Only a working version should be propagated to the repository, and this should be done by the CI server.

In theory you could assign this task to some person to do it manually daily. But the set up is really simple, in my estimation it will pay out in less than a few days.

Billy Sclater wrote:I thought this was 'normal' and that 'maven was poor build tool'! From what you guys say, quite the reverse!

Maven is far from flawless but it has its benefits. But it has to be used right. One problem is that it is a little bit more than just a pure build tool, and if it is used purely to build you are not only leaving out some things; those things will even actually look and work like obstacles to you.

Billy Sclater wrote:For now looks like I have no choice but to 'hack'!

Good luck with that.

[Billy Sclater]

The repo is not broken. The code is broken which is a different problem

Ok, but presumably the code constructs some of the 'in-house jars' inside of the repo. And when my 'working' branch downloads these on a build...it breaks.

The problem with the missing version number is that you just can't reference a specific version of the code. If everything is always 0.0.1-SNAPSHOT

Exactly! I've seen the version number change once in 5 months!

Only a working version should be propagated to the repository, and this should be done by the CI server.

Jenkins is used in my company, but broken code is permitted to be be pushed to master and the branch is often red for days at a time. There are unit tests running on Jenkins, and a sanity test that I wrote. Are you saying that Jenkins can be configured to only allow code that builds successfully (and possibly passes sanity and unit tests) to be propogated to master?

Good luck with that!

As a new guy, and a new dev, I have no authority in the company to enforce these changes, I can only suggest them (feeling slightly inappropriate about it, as I'm sure my more experienced collegaues are probably well aware of these solutions!!);( So if my powers of persuasion don't work, I have a choice between being unable to work with a broken environment, or ... attempting a crude 'hack', and having a chance of getting some work done.

[Jeanne Boyarsky]

Billy Sclater wrote: Are you saying that Jenkins can be configured to only allow code that builds successfully (and possibly passes sanity and unit tests) to be propogated to master?

No. He was saying that Jenkins would provide a way to know the build was broken. It sounds like this is known and "nobody cares." It could be made more visible though. Some teams have a physical sign that the build is broken. Or you could bring it up in meetings.

Billy Sclater wrote: As a new guy, and a new dev, I have no authority in the company to enforce these changes, I can only suggest them (feeling slightly inappropriate about it, as I'm sure my more experienced collegaues are probably well aware of these solutions!!);( So if my powers of persuasion don't work, I have a choice between being unable to work with a broken environment, or ... attempting a crude 'hack', and having a chance of getting some work done.

I became the "process person" with hardly any experience. You don't need to be able to enforce change. You just need to try to get them to see the light. I do think it is important to learn to work within the current system before suggesting improvements though. But I don't think you should be creating the hack. What do your teammates do? These experienced people "clearly have a solution" that they use to deal with this problem. Asking them helps show there is pain in this area. The first step towards change.

[Billy Sclater]

No. He was saying that Jenkins would provide a way to know the build was broken. It sounds like this is known and "nobody cares." It could be made more visible though. Some teams have a physical sign that the build is broken. Or you could bring it up in meetings.

We have that, the Jenkins page clearly displays a list of all the recent builds and informs us if the build has failed or passed. It sends out emails to all devs and QA!

What do your teammates do? These experienced people "clearly have a solution" that they use to deal with this problem. Asking them helps show there is pain in this area. The first step towards change.

Here's what they have told me they do:
#1 one team-mate uses 2 seperate VM guests, one containing a working .m2 and one containing the latest master branch's .m2.
#2 the realease manager recommended the mvn -Dmaven.repo.local=/path/to/repo hack!
#3 another just said she never pulls new code or rebuilds online unless Jenkins informs her that it builds successfully and it has passed integration tests.
#4 another co-worker said to just keep a working local branch on your pc (however this doesn't help because as soon as master fails the .m2 will become corrupted when you re-build).
#5 some co-workers just watch Jenkins build fails and report in the 'stand-up' meeting that they couldn't make any progress because their environment is broken!

It sounds like nobody recommends any of the hacks mentioned in this thread! The list above contains 2 of those hacks! But including 'all' in the list above, if you were in my situation, which of the 5 would you choose?

[Hauke Ingmar Schmidt]

Billy Sclater wrote:Are you saying that Jenkins can be configured to only allow code that builds successfully (and possibly passes sanity and unit tests) to be propogated to master?

No, I was thinking of the generated artefacts that you get from the repo. You don't check out and compile every dependency that you aren't working with, or do you?

Billy Sclater wrote:

Good luck with that!

As a new guy, and a new dev, I have no authority in the company to enforce these changes, I can only suggest them (feeling slightly inappropriate about it, as I'm sure my more experienced collegaues are probably well aware of these solutions!!);( So if my powers of persuasion don't work, I have a choice between being unable to work with a broken environment, or ... attempting a crude 'hack', and having a chance of getting some work done.

You are not in an easy situation. Of course you can't go to your team mates or the persons responsible for the process and tell them that everything is wrong and you refuse to work unless it is fixed. But it seems that there is at least some awareness for the problem even if the pressure seems not high enough to enforce a real solution.

Billy Sclater wrote:It sounds like nobody recommends any of the hacks mentioned in this thread! The list above contains 2 of those hacks! But including 'all' in the list above, if you were in my situation, which of the 5 would you choose?

#2. It is the least immerse, the variable that you need to work with is the one thing where your problems show (i.e. you don't need to recreate your whole development environment), it is the one that the superior who is responsible for that task recommends.

#5 is the Wally solution. Or you can do sword fighting.

[Jeanne Boyarsky]

#2 because it seems the least invasive. this would be to actually get my work done

I'd also start writing down the Jenkins status at the time I did my pulls. Which I'd use as data when bringing up with the team why this approach is suboptimal.

[Billy Sclater]

I can't seem to get the 'mvn -Dmaven.repo.local=/path/to/repo ' appproach to work for me. But, I'll have another crack at it on Monday!

I'll also do as Jeanne suggested and note the Jenkins status at the time I do my pulls for emphasising to the team the drawbacks of having to use such an approach.

Not ideal, but better than my option #5 doing nothing, or sword fighting as Hauke put it!!!

(:

[Hauke Ingmar Schmidt]

Billy Sclater wrote:or sword fighting as Hauke put it!!!

BTW, that was a reference to this XKCD.

[Billy Sclater]

okok LOL!