Nothing in the world can take the place of persistence. Talent will not; nothing is more common than unsuccessful men with talent. Genius will not; unrewarded genius is almost a proverb. Education will not; the world is full of educated derelicts. Persistence and determination alone are omnipotent. The slogan “press on” has solved and always will solve the problems of the human race.

- Calvin Coolidge

The latest version of the source code that accompanies the article can be cloned from the VoltDB example project on my GitHub account here.

If you have any feedback, please leave a comment.

http://www.ibm.com/developerworks/web/library/os-riak2/index.html

I’ve had a quick look and there appears to be an encoding issue in Listing 2. Ignore the question marks. It should read:

$ curl -i http://localhost:8098/riak/odds/
...
{ "odds":"", "description":"" }

 
Hopefully they will have corrected it by the time you read this.

Other than that it’s more or less how I submitted it (I think). I’ll go over it in more detail later on. Let me know what you think.

VoltDB implements a subset of ANSI-standard SQL. It supports the basic CRUD operations (INSERT, SELECT, UPDATE, DELETE) but it does not have support for automatically generating unique identifiers. It is possible, however, to simulate these in VoltDB, as per this entry in the FAQ. What we can do is create a table that stores the name of the table and the current value that can be used as the unique value for, say, a given column. The schema for the table is shown below:

CREATE TABLE IDENTIFIER (
   TABLE_NAME VARCHAR(100) NOT NULL,
   CURRENT_VALUE INTEGER DEFAULT 1 NOT NULL,
   PRIMARY KEY (TABLE_NAME)
);

The stored procedure looks like this:

import org.voltdb.*;

@ProcInfo (
  partitionInfo = "IDENTIFIER.TABLE_NAME: 0",
  singlePartition = true
)
public class GenerateUniqueIdentifier extends VoltProcedure {

  public final SQLStmt select = new SQLStmt(
    "SELECT CURRENT_VALUE FROM IDENTIFIER WHERE TABLE_NAME = ?"
  );

  public final SQLStmt update = new SQLStmt(
    "UPDATE IDENTIFIER SET CURRENT_VALUE = CURRENT_VALUE + 1 " +
    "WHERE TABLE_NAME = ?"
  );

  public VoltTable[] run(String tableName)
    throws VoltAbortException {

      voltQueueSQL(select, tableName);
      VoltTable[] idResult = voltExecuteSQL();

      voltQueueSQL(update, tableName);
      voltExecuteSQL(true);

      return idResult; // Return the current value for the table
  }
}

As this procedure demonstrates, it is possible to execute multiple SQL statements from inside a stored procedure. It also highlights the advantage of having single-threaded partitions. The table is partitioned on the table name therefore the current value for a given table (each row) is stored in a single partition. As each partition is single-threaded, and stored procedures are run sequentially, there is no risk of the current value for a given table being altered between the time the procedure reads the current value and updates it; this would not be the case in a multi-threaded environment where some kind of locking would have to be used.

With the stored procedure in place, it can be called at will. The returned values can then be used in subsequent calls to other stored procedures. And that is how you can simulate auto-increment in VoltDB. At some point I will upload a complete working solution. Checkout the autoincrement branch of my example VoltDB project to play around with some working code that implements the above. The End.

http://www.ibm.com/developerworks/library/os-riak1/

It’s mostly intact although a few paragraphs appear to have fallen by the wayside. Not really surprising as the article was supposed to be under 3000 words whereas the (final) version I submitted was quite a bit over that.

There’s a second installment but I have no idea when it will be published. Or if for that matter. I guess that may depend on the reaction to the first one

I started thinking about it and I don’t get it. I understand technically how multi-tenancy works; I just don’t see the benefits of making an application multi-tenant aware! With the advancements in virtualisation technology made in recent years, why not just get a couple of servers[1], install some virtualisation software and create a virtual instance for each site?

I can imagine that the development effort required in making an application multi-tenant aware is non-trivial – this implies a significant monetary cost in terms of the number of developers working on it and the time it takes to develop – and then there is the potential security risk of tenant data being exposed to other tenants due to bugs in the software. It could happen, you never know! Presumably you will also need to maintain a separate version of the application that doesn’t support multi-tenancy (?)

I guess upgrades would be easier – you would only have to do it once as opposed to having to upgrade each virtual instance.

It just seems to me to be much simpler to go down the virtualisation route. You don’t need to do anything to the application, all you need to do is install some virtualisation software and off you go. Obviously I’m over simplifying it but you get the idea. Perhaps I have missed something. I would be interested to know what you think.

[1] I realise it’s not fashionable nowadays to actually buy/rent your own dedicated servers as everything has to be in the “cloud”!

You can grab it from my GitHub account: https://github.com/sbuckle/yubico-java-client

If you do decide to use it, you might want to pick and choose which bits you want to pull as I have made other changes not related to the enhancements in version 2.0 of the validation protocol. Now I wonder if I’ll get my five free YubiKeys

Update: I did. Just ordered them. Thanks Yubico

I’ve created an Alfresco extension that implements two-factor authentication using a YubiKey.

What is a YubiKey? It’s a device that you plug into your USB port and it generates one time passwords (OTP). It’s similar to RSA’s SecurID, only a lot cheaper. So now, in addition to specifying your username and password, you also have to submit a OTP when logging in – the OTPs are validated by Yubico’s servers.

Using a key like this makes logging in a lot more secure as it is now no longer possible to log in just using a username and password. In addition, each key is tied to a particular user account – the extension takes care of this – so it’s not possible to just use any key; the user has to use the key that has been (uniquely) assigned to them. The screencast below shows how it works.

I will release the extension shortly. You can download the extension from here. I’ll be attending Alfresco DevCon in London so come and say hi and I can give you a live demo of the system. In the meantime, if you have any questions, feel free to leave a comment or send me an email.

As before, the first thing to do is to get log data into the system so it can be analyzed; in this case the Hadoop Distributed File System (HDFS). I copied all of my log files into /tmp/logs on the server where I was running Hadoop. To import the data into HDFS, run the following command:

./bin/hadoop dfs -put /tmp/logs /var/logs

This copies all the files from the directory /var/logs on the local file system into the location /var/logs in HDFS. Remember: the destination location is the location in HDFS and does NOT correspond to a directory in the local file system. To see if the log files were copied, run the following:

./bin/hadoop dfs -ls /var/logs

You should see a list of the files you just imported.

Just like with Riak, we need two classes: a Mapper and a Reducer. The Mapper class contains a map function, which is called for each input – in this case, each log file. The Reducer class contains a reduce function, which is called once for each key, and takes as input the intermediate results from the map phases.

The code for the Mapper class looks like this:

public static class LogEntryMapper extends Mapper<Object, Text, Text, IntWritable> {

  private final static IntWritable one = new IntWritable(1);
  private Text url = new Text();

  private Pattern p = Pattern.compile("(?:GET|POST)\\s([^\\s]+)");

  public void map(Object key, Text value, Context context)
  throws IOException, InterruptedException {
	String[] entries = value.toString().split("\r?\n");
	for (int i=0, len=entries.length; i<len; i+=1) {
		Matcher matcher = p.matcher(entries[i]);
		if (matcher.find()) {
			url.set(matcher.group(1));
			context.write(url, one);
		}
	}
  }
}

The map function takes the text from each log file and splits it at the newline character to get an array of individual log entries. The URL is then extracted from each line and a count of 1 is assigned to each URL instance. The reduce function takes these individual counts and adds them together to get a total for each key (e.g. URL).  The Reducer class looks like this:

public static class LogEntryReducer extends Reducer<Text, IntWritable, Text, IntWritable> {

	private IntWritable total = new IntWritable();

	public void reduce(Text key, Iterable<IntWritable> values, Context context)
	throws IOException, InterruptedException {
		int sum = 0;
		for (IntWritable value : values) {
		    sum += value.get();
		}
		total.set(sum);
		context.write(key, total);
	}
}

The main() method sets up the map-reduce job and starts it:

public static void main(String[] args) throws Exception {
	Configuration conf = new Configuration();

	if (args.length != 2) {
		System.err.println("Usage: loganalyzer <in> <out>");
		System.exit(2);
	}

	Job job = new Job(conf, "analyze log");
	job.setJarByClass(LogAnalyzer.class);
	job.setMapperClass(LogEntryMapper.class);
	job.setReducerClass(LogEntryReducer.class);
	job.setOutputKeyClass(Text.class);
	job.setOutputValueClass(IntWritable.class);
	FileInputFormat.addInputPath(job, new Path(args[0]));
	FileOutputFormat.setOutputPath(job, new Path(args[1]));

	System.exit(job.waitForCompletion(true) ? 0 : 1);
}

As I mentioned in my previous article, it’s possible with Hadoop to specify a combiner class to perform a reduce-type operation after each map phase. If you want to do that here, you will need to add the following line to the job definition:

job.setCombinerClass(LogEntryReducer.class);

Now that the data is available to analyze, the map-reduce job can be started from a local JAR file:

./bin/hadoop jar loganalyzer.jar loganalyzer /var/logs /var/logs-output

(If you are following along at home you will need to download loganalyzer.jar)

This tells the map-reduce job to take all the files in /var/logs as input and output the results of the job to /var/logs-output (in HDFS). To copy the results locally, once the job is complete, execute the following:

./bin/hadoop dfs -getmerge /var/logs-output /tmp/logs-output

If you open up the file in /tmp/logs-output, you should see something like this:

simonbuckle.com/feed/atom       11
simonbuckle.com/index.php       13
simonbuckle.com/robots.txt      2
...

That’s it! The results could be tidied up a bit by removing the hostname as that doesn’t add anything but other than that, it does the job.

The code that defines the Mapper and Reducer classes is here – I have put everything in one file to make it easier to read. The code for the driver is here. All code was compiled against Hadoop version 0.20.203.0.

As before, feel free to leave any comments.

So what is Riak? According to Wikipedia it’s “a NoSQL database implementing the principles from Amazon’s Dynamo paper”. Or, put another way,  it’s a distributed key-value store that has built-in support for MapReduce. If you aren’t familiar with MapReduce a good starting point would be to read Google’s MapReduce paper. I am not going to go over how to install Riak; there’s a good tutorial for that on the Riak website. Riak also has a lot of other features that won’t be covered here.

Right, off we go.

The first thing to do is get your log data into Riak. The example will use Riak’s HTTP API that allows you to create/delete content using HTTP GET, POST etc. Run the following command from wherever your log data is stored:

curl -v -X PUT http://localhost:8091/riak/logs/2011-08-23 \

    -H "Content-Type: text/plain" --data-binary @2011-08-23.log

In this case I am storing the log data in a bucket called “logs” and I am also providing a key (“2011-08-23″) for this particular log. If you do a POST to just the bucket, e.g. you don’t specify a key, Riak will generate a key for you; you will be able to see the generated key in the “Location” header in the HTTP response. Also, note the use of the --data-binary flag. It’s really important because if you use -d instead, curl will very kindly strip out all of the newline characters from the text – as I eventually found out! Not what you want.

Now that the log data is stored in Riak, you can query it. This is where MapReduce comes in. Riak’s MapReduce supports writing map and reduce functions in either JavaScript or Erlang. Currently there is no support for applying an optional combiner function after each map task; other frameworks, such as Hadoop, do. For example, when counting the number of words in a set of documents, each map task may produce lots of records of the type <”at”, 1>. Rather than sending all of these individual records over the network, it would be beneficial to merge the counts for each individual record on each node before sending them to the reduce phase; however, in this scenario, it wouldn’t be difficult to just add the logic for doing the merge in the map function itself. I used JavaScript for my map and reduce functions. Queries are specified using JSON. The query for analysing the log(s) looks something like this:

{
   "inputs": [["logs", "2011-08-23"]],
   "query": [
	{ "map": { "language": "javascript", "name": "LogAnalyzer.mapLogEntry" } },
	{ "reduce": { "language": "javascript", "name": "LogAnalyzer.reduceLogEntry" } }
   ]
}

The inputs consist of an array of arrays; each entry specifies the bucket name and the corresponding key of the log we want to process. In this case the bucket name and key correspond to the log that was loaded previously. Only one map phase is defined but you can specify more than one if you want to. During the map phase the request URL will be extracted from each line in the logs. The reduce phase will take the results from the map phase and sum the counts for each URL, returning the totals to the client. Both the map and reduce phases in this example use named queries. The source for the functions is here.

You can define anonymous JavaScript functions directly in your queries; there is an example of using an anonymous function halfway down this page. Unless your functions are trivial I recommend that you name your functions and have Riak load them when it starts up. To do that you will need to modify the configuration file (<node>/etc/app.config) for each node in your cluster. Open the config file, locate the variable “js_source_dir” and set it to wherever you have your JavaScript files, e.g. {js_source_dir, "/Users/simon/Projects/Riak/js"}. Make sure it’s uncommented. You will need to restart your nodes for the changes to have an effect.

To run the query, save it to a file, open up a terminal and run the following command:

curl -v -H "Content-Type: application/json" \

      http://localhost:8091/mapred -d @log-query.json

Hopefully, you should get back something like this:

[{"www.simonbuckle.com/feed/" : 19},
    {"www.simonbuckle.com/2006/01/19/design-revamp-2" : 1}, ...]

So that’s how to do some analysis on your Apache logs using Riak. All the code can be found on GitHub. There’s also an example of how to do a distributed word count that I didn’t cover here.

There is a lot more information about Riak on the Riak website. At some point it would be nice to be able to specify queries in languages other than JavaScript and Erlang. The map and reduce phases in this example are trivial but I can envisage a scenario where you might want to do some rather complex analysis during each phase so it would be nice to be able to use external libraries rather than having to write stuff from scratch each time.

Feel free to leave a comment if you have any questions.