Managing Complexity: Keeping a Large Java Project on Track

Recent CVS Activity

Showing recent CVS history proved a bit tricky. In order to display commits by branch, we had to use cvs-exp.pl, an open source Perl script that wraps the output of the CVS log command. We further wrapped that in a homegrown Ruby CGI script that allows the recent commit history to be rendered to HTML.

CVS Charts and Graphs

Since we use CVS for revision tracking, there are numerous open source tools available to create reports. We use the StatCVS tool to generate charts and graphs of CVS history. Again, we use a small Ruby script to drive the report generation. Here's the line of code that runs StatCVS itself:

$ java -jar statcvs.jar -output-dir path/to/html/dir/

	project_name project_module/cvslog project_module

Since StatCVS comes in one .jar file, there are no dependencies to track. We run this report nightly, since it takes about 20 minutes to run on all our repositories.

Coding Guidelines

PMD is a Java static analysis tool that checks for unused code, empty catch blocks, and so forth. We run a subset of the standard PMD rules, and we've also written a couple of custom rules to check for Thread creation, Socket creation, and various other coding practices that are not appropriate for this project. The documentation for the PMD Ant task is straightforward, but one thing we found helpful was to always delete the report file from the previous hour before generating a new one. That way, if the code being checked goes from five errors to zero errors and no new file is generated, the previous file won't linger around.

The Dashboard Ruby script then parses the PMD HTML report and determines the number of errors by simply counting the number of rows, as illustrated in this snippet:

count=0

File.new("pmd_report.html").each("<td ") {|x| count += 1}

ruleViolations=(count/4) unless count==0

This result is then displayed on the front page and hyperlinked to the full report.

Duplicate Code

CPD is a Java duplicated-code checker that comes bundled with PMD. We run CPD to check for sequences of more than one hundred duplicate tokens — quite a few, considering that CPD discards whitespace, comments, and various uninteresting sequences like import and package statements. Since CPD has an Ant task, integrating it into the build was similar to integrating PMD.

Note that OnJava.com has published several articles on both PMD and CPD, so there's a lot of information out there on both tools.

JUnit Test Results

JUnit is a popular Java unit testing tool. Some of the developers have begun to write JUnit tests for their code. To encourage this, we run those tests and post the results on the Dashboard. In order to standardize a bit, all tests are to be named by appending Test to the class name (i.e., FooTest), and placed in a separate, parallel directory tree. This lets the Ant task easily find the tests, and it keeps test code separate from the production code. After the tests are run and the results sent to an XML file via the JUnit Ant task's <formatter type="xml"/> element, the Ruby script parses out the number of tests passed/failed:

def parseJunitFile(filename, result)

  (REXML::Document.new(File.new(filename))).elements.each(

  "build/target/task/message[@priority='info']") do |info|

    if (info.text =~ "Tests run: ") != nil

      tmp = info.text.split

      result.testsTotal=result.testsTotal.to_i + tmp[2].to_i

      result.testsFailed=result.testsFailed.to_i + tmp[4].to_i + tmp[6].to_i

    end

  end

end

This allows the totals to be displayed neatly on the Dashboard.

Javadocs

Generating Javadocs is also a straightforward operation with Ant. It's a fairly time-consuming task, though, so we only run it every four hours. Note that Javadocs can consume a considerable amount of disk space; all of the Javadocs on the Dashboard together take up around 500 MB.

Hourly .jar Files of Source and Classfiles

We've found it handy to build an hourly drop of the class files and source files in case someone wants to browse or run the latest code without checking it out and compiling it. Since the code has to be compiled anyway, creating these .jars is a simple matter of using the Ant zip task:

<target name="srczip" if="tic.build.usesSrcZip">

  <delete file="${tic.build.srcZip}"/>

  <zip destfile="${tic.build.srcZip}" basedir="${tic.build.srcDirForZipFile}"

    includes="**/*.java"/>

</target>

<target name="jar" depends="compile">

  <jar jarfile="${tic.build.jarFile}" baseDir="${buildDir}"/>

  <signjar jar="${tic.build.jarFile}" keystore="/var/build/signingCA_keystore" 

                   alias="privileged" storepass="keystore"/>

</target>

Note the dependency in the jar task; there's no need to attempt to jar things up if the compilation step fails.

Future Plans

What else could be added to the hourly build page? In some projects, a test coverage report (a report on the percentage of the code that the unit tests actually cover) has been found useful. Several tools exist to provide such a report — Clover comes to mind. Of course, such a report isn't very useful unless a decent number of unit tests have been written.

Folks who are familiar with the Jakarta open source build tool Maven may notice some similarities. It might be possible to use Maven to do some of the things the Dashboard does, but Maven was not very far along when we first began putting the Dashboard together. It might be worth revisiting Maven to see if that's possible now.

Conclusions

We've discussed some things can make a large Java project hard to manage. We've looked at one large Java project — UltraLog — and how an hourly build status page helped keep things under control. We've also done a quick overview of some open source tools that you may find to be a useful part of your hourly build page. Give them a try!

Credits

Thanks to all of the folks who have donated their time and energy towards the various open source tools mentioned in this article.

References

• UltraLog
• COUGAAR
• UltraLog Dashboard
• Ruby
• Ikko templating engine
• Ant
• PMD
• CPD
• JavaNCSS
• StatCVS
• cvs-exp.pl
• Maven

Tom Copeland started programming on a TRS-80 Model III, but demand for that skill has waned and he now programs mostly in Java and Ruby.

Return to ONJava.com.