Automated Regression Testing suite

[Sujai Kaarthik]

Hi guys,
My final year project is to automate the testing process which is done for an application. I did a research and found that combination of JUnit, Clover and ANT shall be used to make the automated testing process going. I use Eclipse as my IDE. Clover is a proprietary software and I want you to suggest a code coverage analysis tool that is an an open source which can be used with both JUnit and Ant on Eclipse IDE. The technology should also be simple to use even by a novice (I am a novice to all these technologies).

PS: The application is built on Struts framework

Thanks in advance

[Jeanne Boyarsky]

Sujai,
I like emma for code coverage. It is open source and free.

[Sujai Kaarthik]

Thanks Jeanne for your reply

I went through some of the FAQ's about Emma. It is stated that

"Independently of any user-provided instrumentation filters, EMMA always excludes the following from coverage:

1. Methods that have no executable Java bytecode: abstract methods, native methods;
2. Java interface classes (even though they may have executable static initializers).


Additionally, the following types of methods are excluded by default, but can be included by setting the appropriate instrumentation property:

1. Methods marked as Synthetic by the compiler: these are usually "helper" methods needed by the compiler to implement certain Java Language constructs like inner class access, etc. These methods have no user-provided code.
2. Bridge methods created by Java 1.5 compilers to graft new language features on the existing class format."


Is that a disadvantage in using Emma?

[Jeanne Boyarsky]

Sujai,
The default behavior gives you less "noise" in the reports. If you really want it, you can turn it on though. So I wouldn't see it as a disadvantage since it is under your control.

[Sujai Kaarthik]

Okay Jeanne. Thank you for clarifying

[Sujai Kaarthik]

emmatask

This is a wonderful article to start with. I did created the directory structure and placed files like in the example

    projectname/
    src/
        com/
            example/
                class1.java
                class2.java
                class3.java
    target/
        classes/
            com/
                example/
                    class1.class
                    class2.class
                    class3.class
    lib/
        emma/
            emma.jar
            emma_ant.jar
    build.xml

I put two simple java files and its respective JUnit test cases along with the same files

I created a build.xml file with just copy pasting the targets from the above link. I copied just the following:

  <!-- ====================================================================== -->
<!-- EMMA SETUP -->
<!-- ====================================================================== -->

<property name="coverage.dir" value="${basedir}/reports/emma" />     <!-- directory which emma coverage reports will be written to -->
<property name="instr.dir" value="${basedir}/target/emmainstr" />    <!-- directory which emma instrumentation classes will be written to -->
<property name="emma.dir" value="${basedir}/lib/emma" />             <!-- directory that contains emma.jar and emma_ant.jar -->

<!-- Set emma.lib to refer to the list of EMMA jar files -->
<path id="emma.lib" >
    <fileset dir="${emma.dir}">
        <include name="*.jar"/>
    </fileset>
</path>

<!-- Define which classes will be reported in the coverage reports (by default, we will include all classes and assume -->
<!-- that your project's output folder is target/classes -->
<path id="emma.coverage.classes" >
    <pathelement location="target/classes" />
</path>

<!-- Load <emma> and <emmajava> custom tasks so that they can be used in ANT -->
<taskdef resource="emma_ant.properties" classpathref="emma.lib" />

<target name="all" depends="emma, clean, update, compile1, compile2, compile3, test, jar"/>

<!-- Enable Emma -->
<target name="emma" description="turns on EMMA's instrumentation/reporting" >

    <property name="emma.enabled" value="true" />

    <!-- EMMA instr class output directory (it is important to create this property only when EMMA is enabled) -->
    <mkdir dir="${instr.dir}" />
    
    <!-- this property, if overriden via -Demma.filter=<list of filter specs>
     on ANT's command line, will set the coverage filter; by default,
     all classes found in 'emma.coverage.classes' pathref will be instrumented:
    -->
    <property name="emma.filter" value="" />

</target>



<!-- ======================================================================= -->
<!-- EMMA INSTRUMENTATION -->
<!-- ======================================================================= -->
<!-- Instrument the classes, this takes the normal classes, and makes instrumented classes instead -->
<!-- Note that EMMA takes regular ANT path elements as instrumentation input, which is exceedingly convenient -->
<emma enabled="${emma.enabled}" >
    <instr instrpathref="emma.coverage.classes"
         destdir="${instr.dir}"
         metadatafile="${coverage.dir}/metadata.emma"
         merge="true"
    >
        <filter value="${emma.filter}" />
    </instr>
</emma>
  

I didn't added anything to the build file. When I run the build file I get the following message:

C:\projectname>ant
Buildfile: build.xml

BUILD FAILED
C:\projectname\build.xml:5: Unexpected element "{}property" {antlib:org.apache.tools.ant}property

Total time: 0 seconds  

I placed my project in C:\ . What is the thing that went wrong. Since, I am starting with these only now, I cannot figure it out. Where should I place the test codes of JUnit, I mean how the emma task will identify JUnit test code from the actual code (I placed both together in the src directory)?

I expect a cookbook like assistance . Its really embarrassing to ask again and again the same thing.

Kindly help me on this problem.

PS: I am not using any IDEs now

[Jeanne Boyarsky]

The first step in debugging is to determine exactly what is failing. I suspect it is the part where you import the taskdef. One way to find out is to remove everything after that and see if it still fails. Also, try outputting the path right before the failure to see if it looks right on your machine. or try hard coding the absolute path of the jar.

[Sujai Kaarthik]

Hi Jeane,
I was able to correct that mistake and my build was successful. But as stated in that example, folders were created, but the instrumented class files and the METADATA file are not generated. I post the build script here:

<?xml version="1.0"?>

<project name="project"  default="emma.instrument">
	<property name="coverage.dir" value="${basedir}/reports/emma" />   
	<property name="instr.dir" value="${basedir}/target/emmainstr" />   
	<property name="emma.dir" value="${basedir}/lib/emma" />           

	<path id="emma.lib" >
    		<fileset dir="${emma.dir}">
        			<include name="*.jar"/>
    		</fileset>
	</path>

	<path id="emma.coverage.classes" >
    		<pathelement location="target/classes" />
	</path>

	<taskdef resource="emma_ant.properties" classpathref="emma.lib" />

	<target name="emma.instrument" description="turns on EMMA's instrumentation/reporting" >

    		<property name="emma.enabled" value="true" />
    		<mkdir dir="${instr.dir}" />
    		<property name="emma.filter" value="" />

		<mkdir dir="reports\emma" />
		<emma enabled="${emma.enabled}" >
			
    			<instr instrpathref="emma.coverage.classes"
         			destdir="${instr.dir}"
         			metadatafile="${coverage.dir}/metadata.emma"
         			merge="true"
    			>
        				<filter value="${emma.filter}" />
    			</instr>
		</emma>

	</target>
	
</project>

I get the following output:

 C:\projectname>ant
Buildfile: build.xml

emma.instrument:
    [mkdir] Created dir: C:\projectname\target\emmainstr
    [mkdir] Created dir: C:\projectname\reports\emma
    [instr] processing instrumentation path ...
    [instr] instrumentation path processed in 47 ms
    [instr] [0 class(es) instrumented, 0 resource(s) copied]
    [instr] no output created: metadata is empty

BUILD SUCCESSFUL
Total time: 0 seconds

what kind of alteration should I do in order to generate those files?

[Sujai Kaarthik]

I also post the actual code. To begin with, I just coded a small sample program which computes sum, difference and product:

  import java.io.*;


class HelloMaths
{
	public long i,j,k,l,m,n;
	private long a;
	public long sumMaths(long i,long j)
	{
		a=i+j;
		return a;
	}
	
	public long subMaths(long k, long l)
	{
		a=k-l;
		return a;
	}
	
	public long mulMaths(long m,long n)
	{
		a=m*n;
		return a;
	}

	public static void main(String[] args)
	{
		HelloMaths ob = new HelloMaths();
		System.out.println(ob.sumMaths(12, 43));
		System.out.println(ob.subMaths(12, 43));
		System.out.println(ob.mulMaths(12, 43));
	}
}

I also wrote a JUnit testcode for this file which also lies in the same directory

[Sujai Kaarthik]

Hi Jeane,
I have finally made it. I am very happy that I was able to successfully integrate JUnit and Emma with Ant. Your little guidance was very helpful along the path.