Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Monday, July 18, 2011

JNI Example with Class Instantiation (C++)

Awhile back I had to create a JNI wrapper for a C++ class that uses a proprietary core C API. Rewriting everything including the core API in Java would be safer and perform better but in this case not duplicating code was more important. I couldn't find a clear code example containing everything that was needed to 1) make the JNI work at all and 2) wrap a class and call its non-static member function, so I'm putting this here in case it might help someone else.

There are two source files you'll need to write:
  1. A Java class that declares native methods, loads the library built using the JNI, and calls native methods.
  2. JNI code that implements the native methods declared in the Java class and uses the C++ class you're wrapping. You'll build a library with this.
The JNI code file will need to #include a file you'll generate using javah, the C Header and Stub File Generator. The javah tool will create the JNI function prototypes that match the native methods you declared in the Java class.

Here is an example, A.java:

package com.company.product.component;
import java.util.Properties;

public class A {
    private long ptr_;

    private native long createA(long value);
    private native void destroyA(long ptr);
    private native long getResult(long ptr, byte[] input, long length);

    static {
        System.loadLibrary("AJni");
    }
    
    public A(long value) {
        ptr_ = createA(value);
    }
    
    /**
     * Destroy the A instance created by the JNI.
     * This must be called when finished using the A.
     */
    public void destroy() {
        destroyA(ptr_);
    }

    /**
     * Do something.
     *
     * @param input The input array
     * @param length The length of the input array to process
     * @return The result
     */
    public long getResult(byte[] input, long length) {
        return getResult(ptr_, input, length);
    }
}

There are three native methods declared in A.java: one that will create an instance of the C++ class, one that calls a member function of the class, and one that destroys the instance. There is a member variable, ptr_, which stores the pointer to the A instance that's created outside the JVM. You need this to call the member function, getResult(). You also need it to destroy the A instance when you're done. Once you have your Java class written you'll need to:
  1. Use javac to compile A.java into A.class.
  2. Run javah on the Java class like this: javah com.company.product.component.A
    to generate com_company_product_component_A.h which you'll include in the JNI code.
This is what com_company_product_component_A.h looks like:

/* DO NOT EDIT THIS FILE - it is machine generated */
#include <jni.h>
/* Header for class com_company_product_component_A */

#ifndef _Included_com_company_product_component_A
#define _Included_com_company_product_component_A
#ifdef __cplusplus
extern "C" {
#endif

/*
 * Class:     com_company_product_component_A
 * Method:    createA
 * Signature: (J)J
 */
JNIEXPORT jlong JNICALL Java_com_company_product_component_A_createA
  (JNIEnv *, jobject, jlong);

/*
 * Class:     com_company_product_component_A
 * Method:    destroyA
 * Signature: (J)V
 */
JNIEXPORT void JNICALL Java_com_company_product_component_A_destroyA
  (JNIEnv *, jobject, jlong);

/*
 * Class:     com_company_product_component_A
 * Method:    getResult
 * Signature: (J[BJ)J
 */
JNIEXPORT jlong JNICALL Java_com_company_product_component_A_getResult
  (JNIEnv *, jobject, jlong, jbyteArray, jlong);

#ifdef __cplusplus
}
#endif
#endif

The header contains function prototypes that you'll need to provide implementations for in a C++ source file.

Here is the example JNI code, A.cc:

#include "com_company_product_component_A.h"
#include <A.hh>

JNIEXPORT jlong JNICALL Java_com_company_product_component_createA
    (JNIEnv *env, jobject obj, jlong value)
{
    return reinterpret_cast<jlong>(new A(value));
}

JNIEXPORT void JNICALL Java_com_company_product_component_destroyA
    (JNIEnv *env, jobject obj, jlong ptr)
{
    A *a = reinterpret_cast<A*>(ptr);
    if(a) {
        delete a;
    }
}

JNIEXPORT jlong JNICALL Java_com_company_product_component_getResult
    (JNIEnv *env, jobject obj, jlong ptr, jbyteArray input, jlong length)
{
    jbyte *inputArray = env->GetByteArrayElements(input, NULL);
    jlong result = reinterpret_cast<A*>(ptr)->
        getResult(reinterpret_cast<const char*>(inputArray), length);
    env->ReleaseByteArrayElements(input, inputArray, 0);
    return result;
}

The code includes A.hh, the header for the C++ class you're wrapping. All the use of the C++ class is here. Then, A.cc needs to be compiled and linked with the library you're wrapping into a new library, for example libAJni.so, that gets loaded by the Java code using System.load() as you saw in A.java earlier. That's pretty much it. You can now use the Java class A to interact with the C++ library libAJni.so which interacts with the C++ library you wanted to wrap in the first place.
A.class --> libAJni.so --> libA.so

Friday, March 04, 2011

Avro Serialization in Java

There are two ways to serialize data in Avro: using code generation and not using code generation. I want to show examples of each way because I didn't find many examples online when I needed to do it. First, I will briefly cover my understanding of the general concept of each method for serialization.

When using code generation, you use the Avro tool binary, external to your application (or programatically internal to your application) to generate classes that represent the data in your schema. You populate the fields in the classes and use a specific datum writer parameterized with the class to produce the serialized data.

When you do not use code generation, you use the GenericData classes along with the schema to do a series of put() operations. You then use a generic datum writer constructed with the schema to produce the serialized data.

Example Avro Schema


This is a simple schema to use in the example. I made it have an array as a record field because I had a bit of trouble with that when I was working with such a schema. Let's say that this schema is in a file called Schema.avsc.

{
  "namespace": "my.pkg.path.avro",
  "type" : "record",
  "name" : "PeopleList",
  "fields" : [
    {"name" : "Version", "type" : "int"},
    {"name" : "People", "type" : {
      "namespace" : "my.pkg.path.avro",
      "type" : "array",
      "items" : {
        "type" : "record",
        "namespace": "my.pkg.path.avro",
        "name" : "Person",
        "fields" : [
            {"name" : "FirstName", "type" : "string"},
            {"name" : "LastName", "type" : "string"}
        ]
      }}
    }
  ]
}

This schema represents data consisting of a version and a list of records describing people. Each record has the person's first and last name.

Serialize Using Code Generation


To generate classes from outside of your program using the tool, you can invoke it like this:

java -cp <needed jars> org.apache.avro.tool.Main compile schema
    <path to avro schema> <path to put generated classes>

For the <needed jars> it will probably be a handful of dependencies. If a needed dependency can't be found when you run the tool, you'll see an error that the specific jar cannot be found. When this happened, I added it to the classpath (the -cp option) and once I had all of them, it worked.

The generator will start at the destination path specified as the second argument and create directories as specified by the namespaces in your schema. Note that you must have namespaces otherwise an exception will be thrown (NullException) and the code generation will fail. In this example, the tool will generate classes and put them in:

<path to put generated classes>/my/pkg/path/avro/.

In this example, two classes will be generated: PeopleList and Person. All of the classes contain i.e. store the schema used to generate them.

Here is a link to the Avro 1.6.1 API to refer to when looking at the code: http://avro.apache.org/docs/1.6.1/api/java/index.html.

This code snippet shows how to use the generated classes to create the serialized data and place it in a java.nio.ByteBuffer. I create a stream, a binary encoder initialized with the stream, and a specific datum writer. Then, I create a PeopleList, add three people to it, serialize it, and store it in the ByteBuffer.

ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder e = new BinaryEncoder(out);
SpecificDatumWriter<PeopleList> w = 
        new SpecificDatumWriter<PeopleList>(PeopleList.class);

PeopleList all = new PeopleList();
all.People = new GenericData.Array<Person>(
        3, all.getSchema().getField("People").schema());

Person person1 = new Person();
person1.FirstName = new Utf8("Cairne");
person1.LastName = new Utf8("Bloodhoof");
all.People.add(person1);

Person person2 = new Person();
person2.FirstName = new Utf8("Sylvanas");
person2.LastName = new Utf8("Windrunner");
all.People.add(person2);

Person person3 = new Person();
person3.FirstName = new Utf8("Grom");
person3.LastName = new Utf8("Hellscream");
all.People.add(person3);

all.Version = 1;
w.write(all, e);
e.flush();
ByteBuffer serialized = ByteBuffer.allocate(out.toByteArray().length);
serialized.put(out.toByteArray());

Serialize Without Using Code Generation


This code snippet shows how to populate GenericData objects, serialize them, and place the result in a java.nio.ByteBuffer. This is similar to the previous code. I create a stream, a binary encoder initialized with the stream, and a generic datum writer. Then, I create a GenericData.Record that will have the version and the list. Next, I create a GenericData.Array for the list, add three people (each being a GenericData.Record) to it, serialize it, and store it in the ByteBuffer. Each GenericData has to be initialized with a schema that describes its format. You can grab different sections of the overall schema using Schema methods. Check them out here: http://avro.apache.org/docs/current/api/java/org/apache/avro/Schema.html. Keep in mind as you're doing this, each put() gets validated against the schema, so if you mess up, an exception will be thrown. If you leave out any field i.e. you don't do a put() for it, an exception will be thrown when you try to write it.

Schema schema = Schema.parse(new File("Schema.avsc"));

ByteArrayOutputStream out = new ByteArrayOutputStream();
Encoder e = new BinaryEncoder(out);
GenericDatumWriter<GenericRecord> w = new GenericDatumWriter<GenericRecord>(schema);

GenericRecord all = new GenericData.Record(schema);
Schema peopleSchema = schema.getField("People").schema();
GenericArray<GenericRecord> people = new GenericData.Array<GenericRecord>(3, peopleSchema);
Schema personSchema = peopleSchema.getElementType();

GenericRecord person1 = new GenericData.Record(personSchema);
person1.put("FirstName", new Utf8("Cairne"));
person1.put("LastName", new Utf8("Bloodhoof"));
people.add(person1);

GenericRecord person2 = new GenericData.Record(personSchema);
person2.put("FirstName", new Utf8("Sylvanas"));
person2.put("LastName", new Utf8("Windrunner"));
people.add(person2);

GenericRecord person3 = new GenericData.Record(personSchema);
person3.put("FirstName", new Utf8("Grom"));
person3.put("LastName", new Utf8("Hellscream"));
people.add(person3);

all.put("People", people);
all.put("Version", 1);
w.write(all, e);
e.flush();
ByteBuffer serialized = ByteBuffer.allocate(out.toByteArray().length);
serialized.put(out.toByteArray());

Thoughts


I tried both methods of serializing my data and found that using the code generation seemed less error prone and also handy for when I made changes to the schema. I use Eclipse with the Maven Integration Plugin (http://maven.apache.org/eclipse-plugin.html) to build. I wrote a bash script to invoke the code generation tool and used the Maven AntRun Plugin (http://maven.apache.org/plugins/maven-antrun-plugin/) to run it as part of my build. That way I could make changes to my schema, do a project clean, and have updated classes generated easily. I think updating the serialization methods is easier using the generated classes instead of changing/adding/removing the GenericData objects and/or their put() operations.

I hope maybe some of this can help someone with their Avro serialization endeavors in Java.

Monday, February 07, 2011

Null Exception when Generating Classes using SpecificCompiler.compileSchema(File, File)

I decided to try generating classes corresponding to an Avro schema I had created using SpecificCompiler.compileSchema(src, dest). It threw an exception "null" and I found that what it wants to do is create a File(parent, child) using the destination path as the parent and the namespace as the child. The problem was that my schema lacked a namespace and when child is null instantiating a File, it will throw a null exception.