Category: Java

Diagnosing a multithreaded programming issue in Flink’s unit tests

Unfortunately, my contribution to resolve
FLINK-11568 Exception in Kinesis ShardConsumer hidden by InterruptedException introduced a race condition in the unit test code, leading to flaky unit tests reported in FLINK-12595. I made a pointed effort to fix the issue quickly, once it was brought to my attention. Race conditions can be difficult to experimentally reproduce and explain. In order to pinpoint the problem, I wanted to get a full understanding of the multithreaded test and surrounding code. So, I created a big ad-hoc diagram (mainly inspired by UML sequence diagrams) on a whiteboard which I thought would be fun to post here.

Each color represents a different thread. Blue is the unit test (on the left), black is the consumer thread (middle), and green is the KinesisShardConsumer thread (right). Ordering guarantees only exist within each lifeline (vertical line). Multiple lifelines can be run in parallel on the same object by different threads. By drawing the overall diagram carefully, especially blocking method calls and ordering guarantees, I was able to analyze the logic sufficiently to deduce a hypothesis for the likely cause of the issue (circled in red in the picture), and subsequently prove that hypothesis in the code. After that, fixing it was the easy part! It took a couple hours or so to do this work, but I enjoyed it and I was happy to have been able to provide a solution quickly.

Avro Schemas in Multiple Files

Please don’t follow the advice given in the InfoQ article on building an Avro schema up from multiple files. The article recommends doing string replacement to mutate the schemas in order to combine them. The article was written in 2011, and clearly there have been some improvements to Avro since then.

A better (I’m not sure if it’s the best) way to do this, assuming you don’t want to or can’t use .avdl files, is to parse your various files into the same Schema.Parser object. It will give you a map of type name to Schema object:

        List<String> schemaResourceNames = Arrays.asList("avro/foo.avsc", "avro/bar.avsc");

        Schema.Parser parser = new Schema.Parser();
        for (String schemaResourceName : schemaResourceNames) {
            try (InputStream schemaInputStream = classLoader.getResourceAsStream(schemaResourceName)) {
                if (schemaInputStream == null) {
                    throw new RuntimeException("Resource not found " + schemaResourceName);
                }
                parser.parse(schemaInputStream);
            }
        }
        return parser.getTypes();

Corrupted scala-library jar causes build failure

Had a heck of a time figuring out why, when I run “mvn compile” on my project that uses scalatest and scala-maven-plugin I always got:

[ERROR] java.lang.NoClassDefFoundError: scala/Function1
[INFO] 	at java.lang.Class.getDeclaredMethods0(Native Method)
[INFO] 	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
[INFO] 	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
[INFO] 	at java.lang.Class.getMethod0(Class.java:3018)
[INFO] 	at java.lang.Class.getMethod(Class.java:1784)
[INFO] 	at scala_maven_executions.MainHelper.runMain(MainHelper.java:155)
[INFO] 	at scala_maven_executions.MainWithArgsInFile.main(MainWithArgsInFile.java:26)

The cause? A corrupted scala-library JAR in the local Maven repository. Once I deleted it and allowed Maven to re-download it, everything started working.

Using JFormattedTextField with Swing Data Binding from tornado.no

Although there are many data binding libraries available for Swing, the one I have been dealing with lately is from http://databinding.tornado.no/

After I added a JFormattedTextField as a bound field, I noticed that it wasn’t obeying the text-to-value and value-to-text conversion that I had specified in my Formatter. Instead, the data binding was trying to populate the model with the text from the field instead of the value. A look at the source code of CoreUIBridgeProvider shows that JTextFieldBridge is used for JFormattedTextFields, and that bridge only gets & sets the text of the field. Its behavior is sufficient for JTextField, but not for proper use of JFormattedTextField.

To fix the issue, I wrote a custom bridge specifically for formatted fields. It simply passes the value object straight through it.

@Override
public void setUIValue(JFormattedTextField component, Object value) throws ConversionException {
    component.setValue(value);
}

@Override
public Object getUIValue(JFormattedTextField component) throws ConversionException {
    return component.getValue();
}

After I added a new entry to my bridge provider to map my new bridge to my custom JFormattedTextField class, it immediately started working as expected.

Letters typed into JTextField appear in reverse order

I was working on a JTextField in a NetBeans Platform application (Java Swing), and started to see some bizarre behavior. The cursor would remain to the left of each letter typed, and as a result the letters would appear backwards. Or rather, the word would be spelled backwards as the letters would be in reverse order. Rendering was also messed up, as most new letters would not be visible unless the cursor had passed through them. Once it occurs, the problem spreads to other text fields.

I hadn’t noticed, but an Exception had actually been thrown, probably messing up the JTextField’s listeners in some way. The easy answer is to fix the exception. If I had noticed it earlier I wouldn’t have been so baffled!

AspectJ with IntelliJ IDEA

So you’re trying to use AspectJ in a Maven project with IntelliJ IDEA. You add the basic dependency and plugin to your POM file, when IDEA gives you this message:

IDEA was unable to find AspectJ compiler JAR among plugin dependencies.
Please check Settings | Compiler | Java Compiler

The solution is to add a dependency on aspectjtools. For example:

&lt;dependency&gt;
    &lt;groupId&gt;org.aspectj&lt;/groupId&gt;
    &lt;artifactId&gt;aspectjtools&lt;/artifactId&gt;
    &lt;version&gt;1.6.9&lt;/version&gt;
&lt;/dependency&gt;

When IDEA imports the POM changes it will automatically set the AJC compiler and perform some extra indexing work, and then you’re done!

Spring AOP requires an ApplicationContext

Having problems getting Spring AOP working? Is your bean being created, but your aspect not running? Is the section on @AspectJ support not helping?

Make sure you check out Section 3.8.1 of the Spring documentation, which says, “if you are using just a plain BeanFactory then a fair amount of support such as transactions and AOP will not take effect (at least not without some extra steps on your part), which could be confusing because nothing will actually be wrong with the configuration.”

Wicket problems & solutions

So, I’m learning a lot about Wicket lately. Here’s a couple things I’ve learned:

If you’re using the CheckBoxTree from the openCDMS, make sure you get the most recent version. The revision linked to in the Google search results is old, and does not include CheckBoxIconPanel.html. It took me a while to realize what the problem was, because Wicket Components will use the HTML file of their superclass if one is not provided. Here’s a link to the most recent version.

Another issue I recently ran into is that an AJAX Component on my page would get redrawn for the first AJAX call, but not for subsequent calls, even though I added it to the AjaxRequestTarget. Looking at the handy in-page Wicket AJAX debug, I saw,

“ERROR: Wicket.Ajax.Call.processComponent: Component with id [[selectedFeaturesContainera]] a was not found while trying to perform markup update. Make sure you called component.setOutputMarkupId(true) on the component whose markup you are trying to update.”

I was indeed calling setOutputMarkupId(true). The problem was actually that I was using a <wicket:container> for that Component. The <wicket:container> was present with the proper id in the HTML of the initial page load, but would not be included in the document replacing it during the AJAX call. As a result, subsequent calls could not find the element to replace.

The solution: use a real HTML element for every element that has setOutputMarkupId(true). Also, remember that if you’re using using id selectors in your stylesheets, you need to be careful of setOutputMarkupId(true) because your id hard-coded in the HTML will get replaced/overwritten by Wicket’s automatically generated id.

Eclipse IAM

If you get this error when launching a Run Configuration in Eclipse with the IAM (Maven) plugin:

Referenced classpath provider does not exist: org.maven.ide.eclipse.launchconfig.classpathProvider

It’s probably because your Run Configuration was created on an old project, and has bad entries in the Classpath section. The solution is to delete the Run Configuration and create a new one.