Rhizome: Web Helpers Added

Most of my Rhizome applications are Servlet oriented. So I decided to break one of my initial design considerations ("This will not become a web framework") and add some classes for supporting web-based apps.

Really, this is not a web framework. It just provides web-oriented tasks to the Rhizome Controller framework, which is a request queue manager.

With this framework, I developed a prototype of Seriatim -- a project time tracking tool -- in about four hours. I am planning on releasing Seriatim in the near future.

Full Search Added to Rhizome

Yesterday I checked in what I think will be the last major change in Rhizome before I release the 1.0 version.

What was the feature? Full text search. Actually, full text search has always been supported, but it required Rhizome implementors to do lots of the heavy lifting on their own. That has now changed.

Now, from a Rhizome IndexSearcher instance, you can use the simpleSearch() methods to fire off very complex searches without having to do any custom coding.

SimpleSearch supports the following:

• Specifying which metadata should be searched
• Determining whether or not the main body should be searched
• Returning results in sorted order by score
• Handling multiple pages of search data
• Using the ProxyRhizomeDocument class to return certain fields immediately, while delaying the load of entire documents until it is necessary
• Handling very complex search strings, including metadata-specific searches.

On that last item, an example is in order. You could, for example, search for all documents that have a specific tag (or label) and contain a specific word. Here's the search string:

+tag:Nietzsche +zarathustra

This would search for only items with the Nietzsche tag, and with the word zarathustra somewhere in the body or metadata. A whole batch of operators and so on are supported, thanks to the Lucene subsystem.

Rhizome and the PAC Pattern

Rhizome is designed basically according to the PAC (Presentation-Abstraction-Control) design pattern. PAC is similar to MVC, but probably better for web development, and definitely better for Rhizome's goals.

Presentation:
The presentation layer bundled with Rhizome, which makes use of Velocity, is not terribly impressive. It does what a presentation layer ought to do, and no more. It is responsible for taking data and formatting it for return to the user.

Rhizome is pretty flexible on the presentation layer. Velocity is included because I like it enough to use it, and don't hate it enough to look for something else. Also, its introspection abilities are excellent, and make template writing much easier. No special interfaces, etc.

Of course, all that it takes to use a different presentation layer is a few lines of code extending from com.technosophos.rhizome.command.AbstractCommand. And a couple of lines of XML in commands.xml.

But one thing should be noted about the presentation layer as it is: No (non-GUI related) processing is done in the presentation layer. No callbacks to the system are used, and no new resources are retrieved from the system once template rendering is initiated. In short, presentation is passive. It takes what it's given, and it presents it. That's all.

Abstraction:
This is similar to the Model layer in MVC, but in PAC it is the layer responsible for managing the data.

Rhizome has a robust abstraction layer, packaged in com.technosophos.rhizome.repository. This abstraction provides unified methods of accessing data. The current implementation uses a disk-based repository and indexes.

The abstraction engine is broken into three parts: A repository (or database) for storing entire documents, a searcher for finding resources in the repository (usually by search data and metadata), and an indexer. The indexer takes documents and preprocesses them before storing them in the document repository.

Basically, the abstraction layer of Rhizome is a document database (DocumentDB). It is responsible for storage and retrieval of documents.

In some ways, the Rhizome document DB is itself a PAC: The RepositoryManager manages the repository like a controller. The search, indexing, and repository access tools deal directly with data abstraction. And the document representations (in com.technosophos.rhizome.document) providing a common format for the app's client -- in this case, Rhizome's controller. Thus, the RepositoryManager retrieves content, makes sure it is in the correct format, and then returns to the RhizomeController. (Is this a valid instance of PAC? Probably. PAC controllers can be chainable.)

Control:
This layer is responsible for exerting control over the rest of Rhizome. All requests pass through the controller, and the controller is responsible for passing requests to both the repository (abstraction) and whatever the presentation layer is (Velocity, for example).

The com.technosophos.rhizome.controller tree -- and the RhizomeController object specifically -- handle control of the program. RhizomeController is a Front Controller (Which seems to be a compatible pattern with PAC) that can be chained to another controller. Usually, the external controller is a Servlet.

The controller maps requests (from the external controller or user) to a chain of internal commands that must be run in sequence to fulfill the request.

Velocity Variables Persisting Across Templates

Yesterday, I noticed that the Velocity template engine used in Rhizome was behaving strangely. Variables set using the velocity #set directive were persisting over multiple calls to VelocityEngine.mergeTemplate().

So if I set a variable on pageA (using templateA), and then requested pageB (using a different template), the variable would still be available. This has some nasty consequences -- especially considering that Velocity (by default) will not assign nulls to variables.

Example:
Template A:

#set($title= 'My Title');
$title

Template B:

#set($title = $myObj.getTitle())
$title

What happens if getTitle() returns null? This bit is less than intuitive: Velocity ignores the assignment altogether.

So the combination of the variable context bleeding and the null assignment behavior is that if template A is rendered on a first request, and then template B is used on the next request, the title for template B will be
'My Title'.

Surely, this is not good behavior. So what is going on?

First, the Null behavior can be configured. Second, the problem with the variable bleeding is a result of some interesting Velocity internals dealing with VelocityContext objects. Let's look at some Java pseudocode:

VelocityEngine ve = new VelocityEngine();
VelocityContext cxt = newVelocityContext(myHashMap);
ve.mergeTemplate(myTemplateFile,"UTF-8", cxt, myWriter);

In the three lines above, we create a velocity engine, create a context, and then merge a single template.

Line 2 is the important one for us: When Velocity merges a template, it uses the context for storing some variable information. If it finds a Map inside of the context, it uses that Map for storing variables, assuming the Map will be discarded after the template merge.

What if the Map is static? You guessed it: All variables will be stored in the same Map. Bad.

The solution, as it turns out, is fairly simple -- though a bit odd: Remove the Map from the immediate context. This can be done with the Context Chaining feature of Velocity's contexts:

VelocityContext cxt = new VelocityContext(new VelocityContext(myHashMap));

(Alternately, you could just clone the Map into a non-static instance. The above should be less resource intensive, caveat emptor).

Finally, it is also easy (in Velocity 1.5 and later, at least) to get rid of the null assignment behavior. Simply set the SET_NULL_ALLOWED property to true:

ve.setProperty(VelocityEngine.SET_NULL_ALLOWED, true);

These changes are now checked into Rhizome's repository.

The Sinciput Servlet

I have just added the Sinciput servlet to the SVN repository at Google Code.

This new servlet has not been thoroughly tested (it has been tested a little, though), and it is certainly far from finished, but it is checked in and it runs.

Sinciput will work by loading a number of special classes, mainly command classes and descriptive Enums, which will then be used within the Rhizome command architecture.

I anticipate that once I get going on Sinciput basics, things should progress quickly. But at the outset, the difficult tasks will be coding the commands, and getting the dynamic Enum-based data model working.

Middle-Tier Rhzime Construction

The alpha code from the Rhizome backend is done. But in the first stages of Sinciput development, I realized that a good deal could be abstracted, and it seemed a good fit for Rhizome to add a middle-tier controller model.

This new model will make development of applications as simple as adding some commands (Rhizome will map requests to chains of commands), and adding some Velocity templates. Moderately complex implementations might need to create a few Enumerations. More complex applications can implement custom command classes to handle data processing. But for simple apps, it may be as simple as editing some XML and HTML.

The next stage will be to implement a generic Rhizome servlet (to make web applications easy to develop). Then, we will be off to Sinciput.

Rhizome Status

Rhizome is the new backend that will drive Sinciput (it is a complete replacement of Pilaster, which was a good, but limited, architecture).

In a nutshell, Rhizome is a non-hierarchical storage and retrieval engine that includes full text searching, metadata-based data retrieval, relationship support, and an open extension mechanism.

While Pilaster was limited to a single sort of data storage mechanism (namely, Berkeley DB), Rhizome has no such limitation. Further, while Pilaster required that the repository and the index files be located in the same "place", Rhizome makes no such requirement.

As it stands now, Rhizome has the following items completed already:

• A handful of classes implementing the new Rhizome XML format.
• A core set of managing classes and interfaces.
  
• A file system implementation of the repository.
• A Lucene-based implementation of the search indexing code.

The main task left, now, is to implement the searching interface for Lucene, which will provide a method of performing either semi-structured or completely structured searches of the Rhizome repository.