Monday, July 18, 2011

JNI Example with Class Instantiation (C++)

Awhile back I had to create a JNI wrapper for a C++ class that uses a proprietary core C API. Rewriting everything including the core API in Java would be safer and perform better but in this case not duplicating code was more important. I couldn't find a clear code example containing everything that was needed to 1) make the JNI work at all and 2) wrap a class and call its non-static member function, so I'm putting this here in case it might help someone else.

There are two source files you'll need to write:
  1. A Java class that declares native methods, loads the library built using the JNI, and calls native methods.
  2. JNI code that implements the native methods declared in the Java class and uses the C++ class you're wrapping. You'll build a library with this.
The JNI code file will need to #include a file you'll generate using javah, the C Header and Stub File Generator. The javah tool will create the JNI function prototypes that match the native methods you declared in the Java class.

Here is an example, A.java:

package com.company.product.component;
import java.util.Properties;

public class A {
    private long ptr_;

    private native long createA(long value);
    private native void destroyA(long ptr);
    private native long getResult(long ptr, byte[] input, long length);

    static {
        System.loadLibrary("AJni");
    }
    
    public A(long value) {
        ptr_ = createA(value);
    }
    
    /**
     * Destroy the A instance created by the JNI.
     * This must be called when finished using the A.
     */
    public void destroy() {
        destroyA(ptr_);
    }

    /**
     * Do something.
     *
     * @param input The input array
     * @param length The length of the input array to process
     * @return The result
     */
    public long getResult(byte[] input, long length) {
        return getResult(ptr_, input, length);
    }
}

There are three native methods declared in A.java: one that will create an instance of the C++ class, one that calls a member function of the class, and one that destroys the instance. There is a member variable, ptr_, which stores the pointer to the A instance that's created outside the JVM. You need this to call the member function, getResult(). You also need it to destroy the A instance when you're done. Once you have your Java class written you'll need to:
  1. Use javac to compile A.java into A.class.
  2. Run javah on the Java class like this: javah com.company.product.component.A
    to generate com_company_product_component_A.h which you'll include in the JNI code.
This is what com_company_product_component_A.h looks like:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class com_company_product_component_A */

#ifndef _Included_com_company_product_component_A
#define _Included_com_company_product_component_A
#ifdef __cplusplus
extern "C" {
#endif

/*
 * Class:     com_company_product_component_A
 * Method:    createA
 * Signature: (J)J
 */
JNIEXPORT jlong JNICALL Java_com_company_product_component_A_createA
  (JNIEnv *, jobject, jlong);

/*
 * Class:     com_company_product_component_A
 * Method:    destroyA
 * Signature: (J)V
 */
JNIEXPORT void JNICALL Java_com_company_product_component_A_destroyA
  (JNIEnv *, jobject, jlong);

/*
 * Class:     com_company_product_component_A
 * Method:    getResult
 * Signature: (J[BJ)J
 */
JNIEXPORT jlong JNICALL Java_com_company_product_component_A_getResult
  (JNIEnv *, jobject, jlong, jbyteArray, jlong);

#ifdef __cplusplus
}
#endif
#endif

The header contains function prototypes that you'll need to provide implementations for in a C++ source file.

Here is the example JNI code, A.cc:

#include "com_company_product_component_A.h"
#include <A.hh>

JNIEXPORT jlong JNICALL Java_com_company_product_component_createA
    (JNIEnv *env, jobject obj, jlong value)
{
    return reinterpret_cast<jlong>(new A(value));
}

JNIEXPORT void JNICALL Java_com_company_product_component_destroyA
    (JNIEnv *env, jobject obj, jlong ptr)
{
    A *a = reinterpret_cast<A*>(ptr);
    if(a) {
        delete a;
    }
}

JNIEXPORT jlong JNICALL Java_com_company_product_component_getResult
    (JNIEnv *env, jobject obj, jlong ptr, jbyteArray input, jlong length)
{
    jbyte *inputArray = env->GetByteArrayElements(input, NULL);
    jlong result = reinterpret_cast<A*>(ptr)->
        getResult(reinterpret_cast<const char*>(inputArray), length);
    env->ReleaseByteArrayElements(input, inputArray, 0);
    return result;
}

The code includes A.hh, the header for the C++ class you're wrapping. All the use of the C++ class is here. Then, A.cc needs to be compiled and linked with the library you're wrapping into a new library, for example libAJni.so, that gets loaded by the Java code using System.load() as you saw in A.java earlier. That's pretty much it. You can now use the Java class A to interact with the C++ library libAJni.so which interacts with the C++ library you wanted to wrap in the first place.
A.class --> libAJni.so --> libA.so

Friday, July 01, 2011

Alternation without Capture/Extraction/Selection

This drove me crazy for longer than I wanted (since I would want that for varying amounts of time) so I will note it here for other frustrated people to find.

I had a regular expression from which I wanted to capture part of it in $1. I also had an alternation in it that needed grouping with parentheses. It kept capturing the alteration in $1 when I didn't care or want to capture that at all.

This is an example of what I had at first that didn't do what I wanted:

$string =~ /bytes\s+=\s+\d+\.?\d*(K|M)?\s+\(\s*(\d+\.?\d*)\%/;

The above code was giving me either 'K' or 'M' in $1 if either were there instead of what I wanted which was the second grouping (\d+\.?\d*). I just needed to know how to stop the capture since stuff like K|M? and K?|M? without parentheses didn't work right either.

After a lot of online searching using probably the wrong query terms, I found the concept of "non-capturing groupings" which are apparently denoted by (?:regex).

Changed my code to this to finally get what I wanted:

$string =~ /bytes\s+=\s+\d+\.?\d*(?:K|M)?\s+\(\s*(\d+\.?\d*)\%/;

This way the 'K' or 'M' isn't captured and I get the second grouping (\d+\.?\d*) stored in $1.

Friday, May 20, 2011

Storing Options with Multiple Values in a Hash using Getopt::Long

I'm relatively new to Perl and was writing a program that takes long options on the command line. I quickly found the Getopt::Long module and started out getting options where each option's value was stored in a separate variable. One of the options has multiple values, e.g. --option5 0 1 2.

When I started, I had something like:

my ($option1, $option2, $option3, $option4, @option5);
my $result = GetOptions('option1=s'    => \$option1,
                        'option2=i'    => \$option2,
                        'option3=s'    => \$option3,
                        'option4=s'    => \$option4,
                        'option5=i{,}' => \@option5);

I wanted to change it to store the options in a hash since I had several options but got confused with the option that takes multiple values using a repeat specifier. The reason was stated in the documentation that "The destination for the option must be an array or array reference." What I didn't realize is that I needed to use this syntax "Alternatively, you can specify that the option can have multiple values by adding a "@", and pass a scalar reference as the destination" since my destination was now an uninitialized hash entry.

I ended up with this:

my %options;
my $result = GetOptions(\%options, 'option1=s',
                                   'option2=i',
                                   'option3=s',
                                   'option4=s',
                                   'option5=i@{,}');

Thursday, March 10, 2011

Using C Format Code Macros in C++

First, the macros are located in /usr/include/inttypes.h.

If you look at the header file, you'll see that you need to add -D__STDC_FORMAT_MACROS to your compile command.

An example of using them looks like this:

char s[100];
int32_t value = 5;
snprintf(s, sizeof(s), "%"PRIi32, value);
uint64_t andthen = 6;
snprintf(s, sizeof(s), "%"PRIu64, andthen);

Friday, March 04, 2011

Avro Serialization in Java

There are two ways to serialize data in Avro: using code generation and not using code generation. I want to show examples of each way because I didn't find many examples online when I needed to do it. First, I will briefly cover my understanding of the general concept of each method for serialization.

When using code generation, you use the Avro tool binary, external to your application (or programatically internal to your application) to generate classes that represent the data in your schema. You populate the fields in the classes and use a specific datum writer parameterized with the class to produce the serialized data.

When you do not use code generation, you use the GenericData classes along with the schema to do a series of put() operations. You then use a generic datum writer constructed with the schema to produce the serialized data.

Example Avro Schema


This is a simple schema to use in the example. I made it have an array as a record field because I had a bit of trouble with that when I was working with such a schema. Let's say that this schema is in a file called Schema.avsc.

{
  "namespace": "my.pkg.path.avro",
  "type" : "record",
  "name" : "PeopleList",
  "fields" : [
    {"name" : "Version", "type" : "int"},
    {"name" : "People", "type" : {
      "namespace" : "my.pkg.path.avro",
      "type" : "array",
      "items" : {
        "type" : "record",
        "namespace": "my.pkg.path.avro",
        "name" : "Person",
        "fields" : [
            {"name" : "FirstName", "type" : "string"},
            {"name" : "LastName", "type" : "string"}
        ]
      }}
    }
  ]
}

This schema represents data consisting of a version and a list of records describing people. Each record has the person's first and last name.

Serialize Using Code Generation


To generate classes from outside of your program using the tool, you can invoke it like this:

java -cp <needed jars> org.apache.avro.tool.Main compile schema
    <path to avro schema> <path to put generated classes>

For the <needed jars> it will probably be a handful of dependencies. If a needed dependency can't be found when you run the tool, you'll see an error that the specific jar cannot be found. When this happened, I added it to the classpath (the -cp option) and once I had all of them, it worked.

The generator will start at the destination path specified as the second argument and create directories as specified by the namespaces in your schema. Note that you must have namespaces otherwise an exception will be thrown (NullException) and the code generation will fail. In this example, the tool will generate classes and put them in:

<path to put generated classes>/my/pkg/path/avro/.

In this example, two classes will be generated: PeopleList and Person. All of the classes contain i.e. store the schema used to generate them.

Here is a link to the Avro 1.6.1 API to refer to when looking at the code: http://avro.apache.org/docs/1.6.1/api/java/index.html.

This code snippet shows how to use the generated classes to create the serialized data and place it in a java.nio.ByteBuffer. I create a stream, a binary encoder initialized with the stream, and a specific datum writer. Then, I create a PeopleList, add three people to it, serialize it, and store it in the ByteBuffer.

ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder e = new BinaryEncoder(out);
SpecificDatumWriter<PeopleList> w = 
        new SpecificDatumWriter<PeopleList>(PeopleList.class);

PeopleList all = new PeopleList();
all.People = new GenericData.Array<Person>(
        3, all.getSchema().getField("People").schema());

Person person1 = new Person();
person1.FirstName = new Utf8("Cairne");
person1.LastName = new Utf8("Bloodhoof");
all.People.add(person1);

Person person2 = new Person();
person2.FirstName = new Utf8("Sylvanas");
person2.LastName = new Utf8("Windrunner");
all.People.add(person2);

Person person3 = new Person();
person3.FirstName = new Utf8("Grom");
person3.LastName = new Utf8("Hellscream");
all.People.add(person3);

all.Version = 1;
w.write(all, e);
e.flush();
ByteBuffer serialized = ByteBuffer.allocate(out.toByteArray().length);
serialized.put(out.toByteArray());

Serialize Without Using Code Generation


This code snippet shows how to populate GenericData objects, serialize them, and place the result in a java.nio.ByteBuffer. This is similar to the previous code. I create a stream, a binary encoder initialized with the stream, and a generic datum writer. Then, I create a GenericData.Record that will have the version and the list. Next, I create a GenericData.Array for the list, add three people (each being a GenericData.Record) to it, serialize it, and store it in the ByteBuffer. Each GenericData has to be initialized with a schema that describes its format. You can grab different sections of the overall schema using Schema methods. Check them out here: http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html. Keep in mind as you're doing this, each put() gets validated against the schema, so if you mess up, an exception will be thrown. If you leave out any field i.e. you don't do a put() for it, an exception will be thrown when you try to write it.

Schema schema = Schema.parse(new File("Schema.avsc"));

ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder e = new BinaryEncoder(out);
GenericDatumWriter<GenericRecord> w = new GenericDatumWriter<GenericRecord>(schema);

GenericRecord all = new GenericData.Record(schema);
Schema peopleSchema = schema.getField("People").schema();
GenericArray<GenericRecord> people = new GenericData.Array<GenericRecord>(3, peopleSchema);
Schema personSchema = peopleSchema.getElementType();

GenericRecord person1 = new GenericData.Record(personSchema);
person1.put("FirstName", new Utf8("Cairne"));
person1.put("LastName", new Utf8("Bloodhoof"));
people.add(person1);

GenericRecord person2 = new GenericData.Record(personSchema);
person2.put("FirstName", new Utf8("Sylvanas"));
person2.put("LastName", new Utf8("Windrunner"));
people.add(person2);

GenericRecord person3 = new GenericData.Record(personSchema);
person3.put("FirstName", new Utf8("Grom"));
person3.put("LastName", new Utf8("Hellscream"));
people.add(person3);

all.put("People", people);
all.put("Version", 1);
w.write(all, e);
e.flush();
ByteBuffer serialized = ByteBuffer.allocate(out.toByteArray().length);
serialized.put(out.toByteArray());

Thoughts


I tried both methods of serializing my data and found that using the code generation seemed less error prone and also handy for when I made changes to the schema. I use Eclipse with the Maven Integration Plugin (http://maven.apache.org/eclipse-plugin.html) to build. I wrote a bash script to invoke the code generation tool and used the Maven AntRun Plugin (http://maven.apache.org/plugins/maven-antrun-plugin/) to run it as part of my build. That way I could make changes to my schema, do a project clean, and have updated classes generated easily. I think updating the serialization methods is easier using the generated classes instead of changing/adding/removing the GenericData objects and/or their put() operations.

I hope maybe some of this can help someone with their Avro serialization endeavors in Java.

Monday, February 28, 2011

How To Install eUML2 Free Edition

These are instructions for installing the free edition of the eUML2 Eclipse plugin.

From Eclipse:
  1. Help->Install New Software...
  2. Add/use software site http://www.soyatec.com/update/galileo
    • This works for Helios too.
  3. Check "eUML2 Free Edition for Galileo" and click Next.
  4. Keep following the wizard. If you encounter an error for a missing dependency (the GMF Runtime), you will need to install it manually before reattempting the eUML2 install.
  5. If you received the error, start over with Help->Install New Software...
  6. Use software site http://download.eclipse.org/releases/helios
    • This is for Helios, if you're using a different release, choose the right site for your release.
  7. Use filter text "gmf" and wait for it to load matches.
  8. Check "Graphical Modeling Framework (GMF) Runtime" and click Next.
  9. Keep following the wizard until it is installed. Use "Restart Now" or "Apply Changes Now" and return to step 1 to install eUML2.
  10. Enjoy working with UML (this is a required step).

Tuesday, February 22, 2011

eUML2 Class Diagram including Multiple Packages

It didn't seem straightforward to me how to do this so I'll explain the steps here. I wanted to generate a class diagram from my code using the reverse engineering capability of the plugin. This seems backward but I wanted to use it as a way to show my coworkers how the design of something already written looks visually. I have my code separated into different packages and wanted to make a diagram containing every package and show relations between most of the classes.

The overall concept of the class diagrams generated from the plugin is that each diagram represents one package. So first you must create a class diagram representing one of your packages. Once you have that, you can add your other packages to that diagram.
  1. Open the .ucd file for the existing diagram you have so far in Eclipse.
  2. Make sure you're in "Selection mode" in the toolbar.
  3. Make sure nothing is selected*. Then right-click on empty space in the diagram.
  4. Choose Insert->Package and pick the package you want to add to the diagram.
  5. A box will be added that represents that package. Select it by left-clicking on it.
  6. Then right-click and choose "Package Elements...".
  7. Check the boxes for the classes you want to show explicitly in the diagram.
  8. The classes should now be in the package box.
  9. Then you can select nothing* again, right-click and choose "Show All Associations" or "Show all Inheritances" if you want.
  10. When you have picked all the options you want, select nothing*, right-click and choose "Layout" to make it arrange things nicely. If you don't like the layout it comes up with, you can repeat the same action and it seems to cycle through a few layout options.
  11. You can save it as an image by selecting nothing*, right-clicking and choosing "Export as image" and following the wizard.
*Left-click on empty space in the diagram to "select nothing" i.e. make sure that no items are selected.

    Saturday, February 12, 2011

    Brominated Vegetable Oil (BVO) Soda List

    Brominated vegetable oil (BVO) is vegetable oil that has had atoms of the element bromine bonded to it: http://en.wikipedia.org/wiki/Brominated_vegetable_oil. It's used in many citrus flavored drinks that have a cloudy appearance. Clear lemon-lime sodas such as Sprite do not contain it.

    Its use as a food additive may have harmful effects on health. BVO has been banned for use in food in several places including Europe and Japan, but it's legal in the United States. This article from Scientific American explains the history of BVO and its potential dangers: http://www.scientificamerican.com/article.cfm?id=soda-chemical-cloudy-health-history.

    This is a list of brominated drinks that I have found so far. I will keep this list updated for your protection.
    • Squirt
    • Diet Squirt
    • Mountain Dew (all)
    • Orange Crush
    • Pineapple Crush
    • Peach Crush
    • Sun Drop (all)
    • Gatorade ("some")
    • Sunkist Fruit Punch
    • Sunkist Peach
    • Sunkist Pineapple
    • Fanta Orange
    • Fanta Orange Zero
    • Fresca (all)
    • Refreshe Diet Grapefruit
    (Edited 11/02/2011): A commenter today noticed that some flavors of drinks such as Sunkist do not contain BVO and instead have ester gum: http://en.wikipedia.org/wiki/Glycerol_ester_of_wood_rosin, which "...serves as a natural alternative to brominated vegetable oil, particularly in citrus-flavored soft drinks." Check the wiki for a list of drinks that contain ester gum.

    Monday, February 07, 2011

    Null Exception when Generating Classes using SpecificCompiler.compileSchema(File, File)

    I decided to try generating classes corresponding to an Avro schema I had created using SpecificCompiler.compileSchema(src, dest). It threw an exception "null" and I found that what it wants to do is create a File(parent, child) using the destination path as the parent and the namespace as the child. The problem was that my schema lacked a namespace and when child is null instantiating a File, it will throw a null exception.