Analyze your Maven Project Dependencies with dependency:analyze

When working on a larger Maven project it might happen that you lose track of the dependecies in your project. Over time you are adding new dependencies, remove code or move code to modules so some of the dependencies become obsolete. Though I did lots of Maven projects I have to admit I didn't know until recently that the dependency plugin contains a useful goal for solving this problem: dependency:analyze.

The dependency:analyze mojo can find dependencies that are declared for your project but are not necessary. Additionally it can find dependecies that are used but are undeclared, which happens when you are directly using transitive dependencies in your code.

Analyzing Dependencies

I am showing an example with the Odftoolkit project. It contains quite some dependencies and is old enough that some of them are outdated. ODFDOM is the most important module of the project, providing low level access to the Open Document structure from Java code. Running mvn dependency:tree we can see its dependencies at the time of writing:

mvn dependency:tree
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ODFDOM 0.8.10-incubating-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @ odfdom-java ---
[INFO] org.apache.odftoolkit:odfdom-java:jar:0.8.10-incubating-SNAPSHOT
[INFO] +- org.apache.odftoolkit:taglets:jar:0.8.10-incubating-SNAPSHOT:compile
[INFO] |  \- com.sun:tools:jar:1.7.0:system
[INFO] +- xerces:xercesImpl:jar:2.9.1:compile
[INFO] |  \- xml-apis:xml-apis:jar:1.3.04:compile
[INFO] +- junit:junit:jar:4.8.1:test
[INFO] +- org.apache.jena:jena-arq:jar:2.9.4:compile
[INFO] |  +- org.apache.jena:jena-core:jar:2.7.4:compile
[INFO] |  +- commons-codec:commons-codec:jar:1.5:compile
[INFO] |  +- org.apache.httpcomponents:httpclient:jar:4.1.2:compile
[INFO] |  +- org.slf4j:jcl-over-slf4j:jar:1.6.4:compile
[INFO] |  +- org.apache.httpcomponents:httpcore:jar:4.1.3:compile
[INFO] |  +- org.slf4j:slf4j-api:jar:1.6.4:compile
[INFO] |  +- org.slf4j:slf4j-log4j12:jar:1.6.4:compile
[INFO] |  \- log4j:log4j:jar:1.2.16:compile
[INFO] +- org.apache.jena:jena-core:jar:tests:2.7.4:test
[INFO] +- net.rootdev:java-rdfa:jar:0.4.2:compile
[INFO] |  \- org.apache.jena:jena-iri:jar:0.9.1:compile
[INFO] \- commons-validator:commons-validator:jar:1.4.0:compile
[INFO]    +- commons-beanutils:commons-beanutils:jar:1.8.3:compile
[INFO]    +- commons-digester:commons-digester:jar:1.8:compile
[INFO]    \- commons-logging:commons-logging:jar:1.1.1:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 1.877s
[INFO] Finished at: Mon Jan 20 00:41:05 CET 2014
[INFO] Final Memory: 13M/172M
[INFO] ------------------------------------------------------------------------

The project contains some direct dependencies with a lot of transitive dependencies. When running mvn dependency:analyze on the project we will see that our dependencies don't seem to be correct:

mvn dependency:analyze
[INFO] Scanning for projects...
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building ODFDOM 0.8.10-incubating-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[...] 
[INFO] <<< maven-dependency-plugin:2.1:analyze (default-cli) @ odfdom-java <<<
[INFO] 
[INFO] --- maven-dependency-plugin:2.1:analyze (default-cli) @ odfdom-java ---
[WARNING] Used undeclared dependencies found:
[WARNING]    org.apache.jena:jena-core:jar:2.7.4:compile
[WARNING]    xml-apis:xml-apis:jar:1.3.04:compile
[WARNING] Unused declared dependencies found:
[WARNING]    org.apache.odftoolkit:taglets:jar:0.8.10-incubating-SNAPSHOT:compile
[WARNING]    org.apache.jena:jena-arq:jar:2.9.4:compile
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4.769s
[INFO] Finished at: Mon Jan 20 00:43:27 CET 2014
[INFO] Final Memory: 28M/295M
[INFO] ------------------------------------------------------------------------

The second part of the warnings is easier to understand; we have declared some dependencies that we are never using, the taglets and jena-arq. When comparing this with the output we got above you will notice that the largest set of transitive dependencies was imported by the jena-arq dependency. And we don't even need it.

The first part seems to be more difficult: there are two used but undeclared dependencies found. What does it mean? Shouldn't compiling fail if there are any dependencies that are undeclared? No, it just means that we are directly using a transitive dependency from our code which we should better declare ourselves.

Breaking the Build on Dependency Problems

If you want to find problems with your dependencies as early as possible it's best to integrate the check in your build. The dependency:analyze goal we have seen above is meant to be used in a standalone way, for automatic execution there is the analyze-only mojo. It automatically binds to the verify phase and can be declared like this:

<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <version>2.8</version>
    <executions>
        <execution>
            <id>analyze</id>
            <goals>
                <goal>analyze-only</goal>
            </goals>
            <configuration>
                <failOnWarning>true</failOnWarning>
                <outputXML>true</outputXML>
            </configuration>
        </execution>
    </executions>
</plugin>

Now the build will fail if there are any problems found. Conveniently, if an undeclared dependency has been found, it will also output the XML that you can then paste in your pom file.

A final word of caution: the default analyzer works on the bytecode level so in special cases it might not notice a dependency correctly, e.g. when you are using constants from a dependency that are inlined.

Switch Off Legacy Code Violations in SonarQube

While I don't believe in putting numbers on source code quality, SonarQube (formerly known as Sonar) can be a really useful tool during development. It enforces a consistent style across your team, has discovered several possible bugs for me and is a great tool to learn: You can browse the violations and see why a certain expression or code block can be a problem.

To make sure that your code base stays in a consistent state you can also go as far as mandating that there should be no violations in the code developers check in. One of the problems with this is that a lot of projects are not green field projects and you have a lot of existing code. If your violation number already is high it is difficult to judge if no new violations were introduced.

In this post I will show you how you can start with zero violations for existing code without touching the sources, something I got inspired to do by Jens Schauder in his great talk Working with Legacy Teams. We will ignore all violations based on the line in the file so if anybody touches the file the violations will show again and the developer is responsible for fixing the legacy violations.

The Switch Off Violations Plugin

We are using the Switch Off Violations Plugin for SonarQube. It can be configured with different exclusion patterns for the issues. You can define regular expressions for code blocks that should be ignored or deactivate violations at all or on a file or line basis.

For existing code you want to ignore all violations for certain files and lines. This can be done by inserting something like this in the text area Exclusion patterns:

de.fhopf.akka.actor.IndexingActor;pmd:SignatureDeclareThrowsException;[23]

This will exclude the violation for throwing raw Exceptions in line 23 of the IndexingActor class. When analyzing the code again this violation will be ignored.

Retrieving violations via the API

Besides the nice dashboard SonarQube also offers an API that can be used to retrieve all the violations for a project. If you are not keen to look up all existing violations in your code base and insert those by hand you can use it to generate the exclusion patterns automatically. All of the violations can be found at /api/violations, e.g. http://localhost:9000/api/violations.

I am sure there are other ways to do it but I used jsawk to parse the JSON response (On Ubuntu you have to install Spidermonkey instead of the default js interpreter.. And you have to compile it yourself. And I had to use a specific version. Sigh.).

Once you have set up all the components you can now use jsawk to create the exclusion patterns for all existing violations:

curl -XGET 'http://localhost:9000/api/violations?depth=-1' | ./jsawk -a 'return this.join("\n")' 'return this.resource.key.split(":")[1] + ";*;[" + this.line + "]"' | sort | uniq

This will present a list that can just be pasted in the text area of the Switch Off Violations plugin or checked in to the repository as a file. With the next analysis process you will then hopefully see zero violations. When somebody changes a file by inserting a line the violations will be shown again and should be fixed. Unfortunately some violations are not line based and will yield a line number 'undefined'. Currently I just removed those manually so you still might see some violations.

Conclusion

I presented one way to reset your legacy code base to zero violations. With SonarQube 4.0 the functionality of the Switch Violations Off plugin will be available in the core so it will be easier to use. I am still looking for the best way to keep the exclusion patterns up to date. Once somebody had to fix the violations for an existing file the pattern should be removed.

Update 09.01.2014

Starting with SonarQube 4 this approach doesn't work anymore. Some features of the SwitchOffViolations plugin have been moved to the core but excluding violations by line is not possible anymore and also will not be implemented. The developers recommend to only look at the trends of the project and not the overall violation count. This can be done nicely using the differentials.

JUnit Rule for ElasticSearch

While I am using Solr a lot in my current engagement I recently started a pet project with ElasticSearch to learn more about it. Some of its functionality is rather different from Solr so there is quite some experimentation involved. I like to start small and implement tests if I like to find out how things work (see this post on how to write tests for Solr).

ElasticSearch internally uses TestNG and the test classes are not available in the distributed jar files. Fortunately it is really easy to start an ElasticSearch instance from within a test so it's no problem to do something similar in JUnit. Felix Müller posted some useful code snippets on how to do this, obviously targeted at a Maven build. The ElasticSearch instance is started in a setUp method and stopped in a tearDown method:

private EmbeddedElasticsearchServer embeddedElasticsearchServer;

@Before
public void startEmbeddedElasticsearchServer() {
    embeddedElasticsearchServer = new EmbeddedElasticsearchServer();
}

@After
public void shutdownEmbeddedElasticsearchServer() {
    embeddedElasticsearchServer.shutdown();
}

As it is rather cumbersome to add these methods to all tests I transformed the code to a JUnit rule. Rules can execute code before and after a test is run and influence its execution. There are some base classes available that make it really easy to get started with custom rules.

Our ElasticSearch example can be easily modeled using the base class ExternalResource (see the full example code on GitHub):

public class ElasticsearchTestNode extends ExternalResource {

    private Node node;
    private Path dataDirectory;
    
    @Override
    protected void before() throws Throwable {
        try {
            dataDirectory = Files.createTempDirectory("es-test", new FileAttribute []{});
        } catch (IOException ex) {
            throw new IllegalStateException(ex);
        }

        ImmutableSettings.Builder elasticsearchSettings = ImmutableSettings.settingsBuilder()
                .put("http.enabled", "false")
                .put("path.data", dataDirectory.toString());

        node = NodeBuilder.nodeBuilder()
                .local(true)
                .settings(elasticsearchSettings.build())
                .node();
    }

    @Override
    protected void after() {
        node.close();
        try {
            FileUtils.deleteDirectory(dataDirectory.toFile());
        } catch (IOException ex) {
            throw new IllegalStateException(ex);
        }
    }
    
    public Client getClient() {
        return node.client();
    }
}

The before method is executed before the test is run so we can use it to start ElasticSearch. All data is written to a temporary folder. The after method is used to stop ElasticSearch and delete the folder.

In your test you can now just use the rule, either with the @Rule annotation to have it triggered on each test method, or using @ClassRule to execute it only once per class:

public class CoreTest {

    @Rule
    public ElasticsearchTestNode testNode = new ElasticsearchTestNode();
    
    @Test
    public void indexAndGet() throws IOException {
        testNode.getClient().prepareIndex("myindex", "document", "1")
                .setSource(jsonBuilder().startObject().field("test", "123").endObject())
                .execute()
                .actionGet();
        
        GetResponse response = testNode.getClient().prepareGet("myindex", "document", "1").execute().actionGet();
        assertThat(response.getSource().get("test")).isEqualTo("123");
    }
}

As it is really easy to implement custom rules I think this is a feature I will be using more often in the future.

Gradle is too Clever for my Plans

While writing this post about the Lucene Codec API I noticed something strange when running the tests with Gradle. When experimenting with a library feature most of the times I write unit tests that validate my expectations. This is a habit I learned from Lucene in Action and can also be useful in real world scenarios, e.g. to make sure that nothing breaks when you update a library.

OK, what happened? This time I did not only want to have the test result but also ran the test for a side effect, I wanted a Lucene index to be written to the /tmp directory to manually have a look at it. This worked fine for the first time, but not afterwards, e.g. after my machine was rebooted and the directory cleared.

It turns out that the Gradle developers know that a test shouldn't be used to execute stuff. So once the test is run successfully it is just not run again until its input changes! Though this bit me this time this is a really nice feature to speed up your builds. And if you really need to execute the tests, you can always run gradle cleanTest test.

Running and Testing Solr with Gradle

A while ago I blogged on testing Solr with Maven on the synyx blog. In this post I will show you how to setup a similar project with Gradle that can start the Solr webapp and execute tests against your configuration.

Running Solr

Solr is running as a webapp in any JEE servlet container like Tomcat or Jetty. The index and search configuration resides in a directory commonly referred to as Solr home that can be outside of the webapp directory. This is also the place where the Lucene index files are created. The location for Solr home can be set using an environment variable.

The Solr war file is available in Maven Central. This post describes how to run a war file that is deployed in a Maven repository using Gradle. Let's see how the Gradle build file looks like for running Solr:

import org.gradle.api.plugins.jetty.JettyRunWar

apply plugin: 'java'
apply plugin: 'jetty'

repositories {
    mavenCentral()
}

// custom configuration for running the webapp
configurations {
    solrWebApp
}

dependencies {
    solrWebApp "org.apache.solr:solr:3.6.0@war"
}

// custom task that configures the jetty plugin
task runSolr(type: JettyRunWar) {
    webApp = configurations.solrWebApp.singleFile

    // jetty configuration
    httpPort = 8082
    contextPath = 'solr'
}

// executed before jetty starts
runSolr.doFirst {
    System.setProperty("solr.solr.home", "./solrhome")
}

We are creating a custom configuration that contains the Solr war file. In the task runSolr we configure the Jetty plugin. To add the Solr home environment variable we can use the way described by Sebastian Himberger. We add a code block that is executed before Jetty starts and sets the environment variable using standard Java mechanisms. You can now start Solr using gradle runSolr. You will see some errors regarding multiple versions of slf4j that are very like caused by this bug.

Testing the Solr configuration

Solr provides some classes that start an embedded instance using your configuration. You can use these classes in any setup as they do not depend on the gradle jetty plugin. Starting with Solr 3.2 the test framework is not included in solr-core anymore. This is what the relevant part of the dependency section looks like now:

testCompile "junit:junit:4.10"
testCompile "org.apache.solr:solr-test-framework:3.6.0"

Now you can place a test in src/test/java that either uses the convenience methods provided by SolrTestCaseJ4 or you can instantiate an EmbeddedSolrServer and execute any SolrJ actions. Both of these ways will use your custom config. This way you can easily validate that configuration changes don't break existing functionality. An example of using the convenience methods:

import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.SolrServerException;
import org.junit.BeforeClass;
import org.junit.Test;
import java.io.IOException;

public class BasicConfigurationTest extends SolrTestCaseJ4 {

    @BeforeClass
    public static void initCore() throws Exception {
        SolrTestCaseJ4.initCore("solrhome/conf/solrconfig.xml", "solrhome/conf/schema.xml", "solrhome/");
    }

    @Test
    public void noResultInEmptyIndex() throws SolrServerException {
        assertQ("test query on empty index",
                req("text that is not found")
                , "//result[@numFound='0']"
        );
    }

    @Test
    public void pathIsMandatory() throws SolrServerException, IOException {
        assertFailedU(adoc("title", "the title"));
    }

    @Test
    public void simpleDocumentIsIndexedAndFound() throws SolrServerException, IOException {
        assertU(adoc("path", "/tmp/foo", "content", "Some important content."));
        assertU(commit());

        assertQ("added document found",
                req("important")
                , "//result[@numFound='1']"
        );
    }

}

We extend the class SolrTestCaseJ4 that is responsible for creating the core and instanciating the runtime using the paths we provide with the method initCore(). Using the available assert methods you can execute queries and validate the result using XPath expressions.

An example that instanciates a SolrServer might look like this:

import org.apache.solr.SolrTestCaseJ4;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer;
import org.apache.solr.client.solrj.response.FacetField;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.SolrParams;
import org.junit.After;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;

import java.io.IOException;

public class ServerBasedTalkTest extends SolrTestCaseJ4 {

    private EmbeddedSolrServer server;

    @BeforeClass
    public static void initCore() throws Exception {
        SolrTestCaseJ4.initCore("solr/conf/solrconfig.xml", "solr/conf/schema.xml");
    }

    @Before
    public void initServer() {
        server = new EmbeddedSolrServer(h.getCoreContainer(), h.getCore().getName());
    }

    @Test
    public void queryOnEmptyIndexNoResults() throws SolrServerException {
        QueryResponse response = server.query(new SolrQuery("text that is not found"));
        assertTrue(response.getResults().isEmpty());
    }

    @Test
    public void singleDocumentIsFound() throws IOException, SolrServerException {
        SolrInputDocument document = new SolrInputDocument();
        document.addField("path", "/tmp/foo");
        document.addField("content", "Mein Hut der hat 4 Ecken");

        server.add(document);
        server.commit();

        SolrParams params = new SolrQuery("ecke");
        QueryResponse response = server.query(params);
        assertEquals(1L, response.getResults().getNumFound());
        assertEquals("/tmp/foo", response.getResults().get(0).get("path"));
    }

    @After
    public void clearIndex() {
        super.clearIndex();
    }
}

The tests can now be executed using gradle test.

Testing your Solr configuration is important as changes in one place might easily lead to side effects with another search functionality. I recommend to add tests even for basic functionality and evolve the tests with your project.

Testing Akka actors from Java

If you're looking for a general introduction into using Akka from Java have a look at this post

In a recent project I've been using Akka for a concurrent producer-consumer setup. It is an actor framework for the JVM that is implemented in Scala but provides a Java API so normally you don't notice that your dealing with a Scala library.

Most of my business code is encapsulated in services that don't depend on Akka and can therefore be tested in isolation. But for some cases I've been looking for a way to test the behaviour of the actors. As I struggled with this for a while and didn't find a real howto on testing Akka actors from Java I hope my notes might be useful for other people as well.

The main problem when testing actors is that they are managed objects and you can't just instanciate them. Akka comes with a module for tests that is documented well for using it from Scala. But besides the note that it's possible you don't find a lot of information on using it from Java.

When using Maven you need to make sure that you have the akka-testkit dependency in place:

<dependency>
    <groupId>com.typesafe.akka</groupId>
    <artifactId>akka-testkit</artifactId>
    <version>2.1-SNAPSHOT</version>
    <scope>test</scope>
</dependency>

I will show you how to implement a test for the actors that are introduced in the Akka java tutorial. It involves one actor that does a substep of calculating Pi for a certain start number and a given number of elements.

To test this actor we need a way to set it up. Akka-testkit provides a helper TestActorRef that can be used to set it up. Using scala this seems to be rather simple:

val testActor = TestActorRef[Worker]

If you try to do this from Java you will notice that you can't use a similar call. I have to admit that I am not quite sure yet what is going on. I would have expected that there is an apply() method on the TestActorRef companion object that uses some kind of implicits to instanciate the Worker object. But when inspecting the sources the thing that comes closest to it is this definition:

def apply[T <: Actor](factory: ⇒ T)(implicit system: ActorSystem)

No sign of implicit for the factory. Something I still have to investigate further.

To use it from Java you can use the method apply that takes a reference to a Function0 and an actor system. The actor system can be setup easily using

actorSystem = ActorSystem.apply();

The apply() method is very important in scala as it's kind of the default method for objects. For example myList(1) is internally using myList.apply(1).

If you're like me and expect that Function0 is a single method interface you will be surprised. It contains a lot of strange looking methods that you really don't want to have cluttering your test code:

TestActorRef workerRef = TestActorRef.apply(new Function0() {

    @Override
    public Worker apply() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public void apply$mcV$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public boolean apply$mcZ$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public byte apply$mcB$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public short apply$mcS$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public char apply$mcC$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public int apply$mcI$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public long apply$mcJ$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public float apply$mcF$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }

    @Override
    public double apply$mcD$sp() {
        throw new UnsupportedOperationException("Not supported yet.");
    }
}, actorSystem);

The only method we really are interested in is the normal apply method. Where do those other methods come from? There is no obvious hint in the scaladocs.

During searching for the solution I found a mailing list thread that explains some of the magic. The methods are performance optimizations for boxing and unboxing that are automatically generated by the scala compiler for the @specialized annotation. Still, I am unsure about why this is happening exactly. According to this presentation I would have expected that I am using the specialized instance for Object, maybe that is something special regarding traits?

Fortunately we don't really need to implement the interface ourself: There's an adapter class, AbstractFunction0, that makes your code look much nicer:

@Before
public void initActor() {
    actorSystem = ActorSystem.apply();
    actorRef = TestActorRef.apply(new AbstractFunction0() {

        @Override
        public Pi.Worker apply() {
            return new Pi.Worker();
        }
           
    }, actorSystem);
}

This is like I would have expected it to behave in the first place.

Now, as we have setup our test we can use the TestActorRef to really test the actor. For example we can test that the actor doesn't do anything for a String message:

@Test
public void doNothingForString() {
    TestProbe testProbe = TestProbe.apply(actorSystem);
    actorRef.tell("Hello", testProbe.ref());

    testProbe.expectNoMsg(Duration.apply(100, TimeUnit.MILLISECONDS));
}

TestProbe is another helper that can be used to check the messages that are sent between cooperating actors. In this example we are checking that no message is passed to the sender for 100 miliseconds, which should be enough for execution.

Let's test some real functionality. Send a message to the actor and check that the result message is send:

@Test
public void calculatePiFor0() {
    TestProbe testProbe = TestProbe.apply(actorSystem);
    Pi.Work work = new Pi.Work(0, 0);        
    actorRef.tell(work, testProbe.ref());

    testProbe.expectMsgClass(Pi.Result.class);     
    TestActor.Message message = testProbe.lastMessage();
    Pi.Result resultMsg = (Pi.Result) message.msg();
    assertEquals(0.0, resultMsg.getValue(), 0.0000000001);
}

Now we use the TestProbe to block until a message arrives. When it's there we can have a look at using the lastMessage().

You can look at the rest of the test on Github. Comments are more than welcome as I am pretty new to Scala as well as Akka.

Update

As Jonas Bonér points out I've been using the Scala API. Using the Props class the setup is easier:

    @Before
    public void initActor() {
        actorSystem = ActorSystem.apply();
        actorRef = TestActorRef.apply(new Props(Pi.Worker.class), actorSystem);
    }

Running my Tests again

For some time I've been bugged by a Netbeans problem that I couldn't find any solution to. When running a unit test from within Netbeans from time to time it happended that the tests just failed. They seemed to be executed in an old state. Running them again didn't help either, it seemed that some parts of the project didn't get recompiled. When executing the tests from a command line Maven build there were never any problems and afterwards the tests could be run again from Netbeans. The problem only occured very infrequently but nevertheless it was really annoying. I started not running the tests from Netbeans at all but only using Maven. That is also not a good solution as you either run all tests or have to edit the command line all the time for running only a single test.

Recently I noticed what caused the problem: Netbeans has its compile on save feature on for tests. This means it is using its internal incremental compile feature which doesn't seem to work fine at least for some project setups.


You can disable it in the project properties on the Build/Compile node. I haven't seen any problems since disabling it. Saves me a lot of time to run the tests from the IDE again.

Running Ruby on Rails Tests in Netbeans

I don't get it. Netbeans is often recommended as an excellent IDE for Ruby on Rails development, not only when targeting the JVM. Nevertheless, even some basic features don't seem to be working with the default setup. You can't even run the tests, which is fundamental to developing using a dynamic language.

What's happening? Suppose you have a simple app and you want to run some tests using the test database. Not sure if this is mandatory when using the built in JRuby but it seems to be normal to use the jdbcmysql adapter. When you try to run the tests you will see something like this:

1) Error:
test_index_is_ok(ContactsControllerTest):
ActiveRecord::StatementInvalid: ActiveRecord::JDBCError: Table 'kontakt_test.contacts' doesn't exist: DELETE FROM `contacts`

followed by the stack trace that isn't really helpful as it's not the root cause. Rails somehow doesn't create the tables in the test database. You'll see a more helpful output when starting the rake task "db:test:prepare" directly in debug mode:

** Invoke db:test:prepare (first_time)
** Invoke db:abort_if_pending_migrations (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute db:abort_if_pending_migrations
rake aborted!
Task not supported by 'jdbcmysql'
/path/to/netbeans-6.9.1/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rails-2.3.8/lib/tasks/databases.rake:380
/path/to/netbeans/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `call'
/path/to/netbeans/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:636:in `execute'
/path/to/netbeans/ruby/jruby-1.5.1/lib/ruby/gems/1.8/gems/rake-0.8.7/lib/rake.rb:631:in `each'
[...]
** Execute db:test:prepare
** Invoke db:test:load (first_time)
** Invoke db:test:purge (first_time)
** Invoke environment 
** Execute db:test:purge

The task fails in the database task in the rails lib. You can open up the source code by opening the node Libraries/Built-in-JRuby/rails-2.3.8/lib/tasks/databases.rake in Netbeans.

At line 357 you can see the problem: Rails only expexts some hardcoded adapters, jdbcmysql not being one of them. It skips the task for unknown adapters. Two options to fix it: Insert a regular expression that matches both:

when /mysql/ # instead of when "mysql"

or add the jdbcmysql adapter as a second option:

when "mysql","jdbcmysql"

Now the tests are running and hopefully passing. The same kind of error might occur for other tasks as well as there are some more checks for the mysql adapter in this file. You should be able to fix them the same way.

I wouldn't have expected to have to patch the rails code for using it in Netbeans but this doesn't seem to be uncommon. Using a recent active record version is supposed to fix the problem as you can use mysql as an adapter name then but I didn't find a way to run the jdbc generator from Netbeans. It isn't available in the list of generators and I didn't find a generator gem to download.

What's to be learned for me from this? I got a better understanding of how the build process works using rake. But more importantly: even technologies that have been hyped for a long time might not be that flawless as you would expect.

Using OpenCmsTestCase

These are basically some notes for me because I just had to relearn all of it.

To use OpenCmsTestCase in a project the following steps have to be applied:

• Download the source distribution and unzip it


• set the file encoding: export ANT_OPTS=-Dfile.encoding=iso-8859-1


• run ant bindist


• run ant compile-tests


• Somehow some unittests are always not compiling for me: remove the java files and the entries from the TestSuites


• create a jar file from the folder org in ../BuildCms/build/test, e.g. jar -cf opencms-test-7.5.2.jar org


• add the jar to your project classpath/deploy to your maven repository


• add hsqldb.jar to your project


• copy the folders data and webapp to your project


• copy test/log4j.properties and test/test.properties to your test classpath and adjust the directory paths in test.properties (a good reason to use Maven so you can use the resource filtering mechanism)


• play with the files in data/imports and adjust them to your needs

A simple test case example:

import org.opencms.file.CmsObject;
import org.opencms.file.CmsResource;
import org.opencms.test.OpenCmsTestCase;
import org.opencms.test.OpenCmsTestProperties;

public class DummyOpenCmsTest extends OpenCmsTestCase {

    static {
        OpenCmsTestProperties.initialize(OpenCmsTestProperties.getResourcePathFromClassloader("test.properties"));
    }

    public DummyOpenCmsTest(String name) {
        super(name);
    }

    @Override
    public void setUp() throws Exception {
        super.setUp();
        setupOpenCms("simpletest", "/sites/default/");
    }

    public void testExistingResource() throws Exception {
        CmsObject cms = getCmsObject();
        CmsResource res = cms.readResource("/index.html");
        assertEquals("/sites/default/index.html", res.getRootPath());
    }

    @Override
    public void tearDown() throws Exception {
        super.tearDown();
        removeOpenCms();
    }
}