vladimir piskarev's blog

Musings on the Eclipse Handly project and software development in general

Handly — Support for legacy models

You might ask… Handly looks interesting, but can it be of any use if I already have a (handle-based) code model with a well-established API? (Or, perhaps, you’d like to use Handly in a greenfield, but are a bit uncomfortable about the leakage of Handly API to your model’s clients.)

If this sort of questions makes you delay considering Handly adoption, perhaps you should wait no longer. Handly can support legacy models while still providing the benefits of a uniform handle-based API. Now when the dust settled after the Handly 0.3 release, let me show you how.

Although Handly imposes almost no restrictions on the model shape or on the languages modeled, it still requires that model elements implement some of the provided interfaces such as IHandle, ISourceElement and ISourceFile, which might be enough to break API compatibility with an existing model. However, a handle-based design makes it straightforward to implement lightweight adapter elements to the existing model API and expose this fully backwards compatible adapter model to clients. As long as those adapter elements can adapt themselves to underlying Handly-based model elements, they can be fully supported by the generic IDE infrastructure built on top of the uniform Handly API. The adapter model does incur some additional overhead in both time and space, but the approach is architecturally sound and in many cases the overhead should be manageable.

To make it more concrete and as a proof of concept, I have implemented a subset of the JDT Java model API atop the example model for Java and pushed it to the javamodel.jdt branch in the project’s repository. (I’m hesitant about merging it to master as I think it can unnecessarily complicate the Java model example. However, I’m open to reconsidering this decision based on the community feedback.)

Once the adapter model had been in place, it was very easy to use existing JDT UI classes to build an almost fully functional Java Navigator. I said almost because, since the adapter model was based on the deliberately contrived example model, many of its methods just throw UnsupportedOperationException. However, navigation does work, and you can see all the usual actions in the context menu (although it is not recommended to actually use those actions since, as mentioned above, the underlying adapter model is incomplete and not all operations are supported; you can still use the JDT’s Package Explorer for performing the actions). I must say that implementing such an adapter to the JDT Java model API was an exciting (if a bit short) experience!

In the end, it’s all about loose coupling. Your model API may depend on Handly, which can be convenient in many cases but is not strictly required: instead, you can treat your Handly-based model as an implementation detail, exposing only the adapter model to clients. Generic IDE components (such as those already provided by Handly) may depend on the uniform Handly API, but can know nothing about specifics of a model. Finally, there may be (in theory, at least) alternative implementations of the Handly API in addition to the one provided by the project itself.

Support for legacy models as well as overall API quality are going to be important themes on our way to the Handly 0.4 release. As always, feedback is most welcome and can be directed to the project’s forum or right to the developer mailing list.

Advertisements

Announcing Handly 0.3 Release

It is with great pleasure that I announce the on-time availability of the Handly 0.3 release, symbolically coinciding with Eclipse Mars. This release includes major new features, such as an outline framework, integration with text editors (in addition to preexisting integration with Xtext editor), a real-world exemplary implementation (a simplified Java model), and a number of API enhacements. This release is primarily focused on bringing more value to early adopters while further evaluating the Handly core framework.

Downloads

Handly is a relatively new project at Eclipse started with an initial contribution from 1C. In brief, it can be described as a framework for handle-based models — with an emphasis on code models that render Eclipse workspace from a programming language angle.

Because the rest of the IDE usually depends on the code model, the quality of the design and implementation of that model is of the utmost importance. Traditionally, such models were built either entirely from scratch or by forking and modifying preexisting models. The traditional process required much effort, was tedious and error-prone. The resulting models were effectively silos with a completely isolated API, which prevented a possibility of developing reusable IDE components, although the models did seem to have certain traits in common.

The Handly project begs to differ with the traditional approach. It provides a set of flexible building blocks that help developers create handle-based models similar in design principles to the JDT Java model. The framework imposes almost no restrictions on the shape of the models or on the languages modeled. The uniform API makes it possible to develop generic IDE components dealing with such models — some common UI components are already provided by Handly. The intent is to come up with a really nice design in this problem area through an open and transparent development process primarily driven by community feedback. For more information, please visit the project’s website.

Despite the “incubating technology” status of Handly, there already are three known major adopters (in chronological order): 1C:Enterprise Development Tools, erlide (in yet-to-be-released version 0.30), and Codasip Studio — two commercial products and an esteemed open-source IDE. We are very grateful for their vote of confidence to the project, active participation and valuable feedback. With this release, we hope to encourage further adoption and receive broader feedback.

Some design discussions with early adopters are already taking place on the project’s forum, but also (as it stands, chiefly) on the handly-dev mailing list, with potentially even more interesting discussions soon to follow.

So if you are interested in this project and would like to have a say in it, I encourage you to engage as early as possible by playing around with this release and giving feedback.

Thank you!

Handly 0.3 : Integration with TextFileBuffer-based editors

Besides providing out-of-the-box integration with Xtext editor, Handly can also support other editors. In particular, it allows for integration with TextFileBuffer-based editors. The following notes are intended as forward pointers only; full-blown editor implementation is beyond the scope of this article.

To integrate a TextFileBuffer-based editor with working copy functionality, you’ll need to subclass the editor’s document provider from the class SourceFileDocumentProvider (which extends the TextFileDocumentProvider).

For example:

public class FooFileDocumentProvider
    extends SourceFileDocumentProvider
{
    public FooFileDocumentProvider()
    {
        super(new FooElementForEditorInputFactory());
        // ...
    }

    // ...
}

You’ll also need to implement a Handly-based reconciler for the editor:

public class FooReconciler
    extends HandlyReconciler
{
    public FooReconciler(ITextEditor editor, IWorkingCopyManager manager)
    {
        super(editor, manager);
    }

    @Override
    protected void addElementChangeListener(IElementChangeListener listener)
    {
        FooModelCore.getFooModel().addElementChangeListener(listener);
    }

    @Override
    protected void removeElementChangeListener(IElementChangeListener listener)
    {
        FooModelCore.getFooModel().removeElementChangeListener(listener);
    }
}

The reconciler is connected to the editor via a source viewer configuration:

public class FooSourceViewerConfiguration
    extends TextSourceViewerConfiguration
{
    private final ITextEditor editor;
    private final IWorkingCopyManager manager;

    public FooSourceViewerConfiguration(IPreferenceStore preferenceStore,
        ITextEditor editor, IWorkingCopyManager manager)
    {
        super(preferenceStore);
        this.editor = editor;
        this.manager = manager;
    }

    @Override
    public IReconciler getReconciler(ISourceViewer sourceViewer)
    {
        if (editor != null && editor.isEditable())
        {
            return new FooReconciler(editor, manager);
        }
        return null;
    }
}

Then, you can wire it all together in the editor:

public class FooEditor
    extends AbstractDecoratedTextEditor
{
    @Override
    protected void initializeEditor()
    {
        super.initializeEditor();
        FooFileDocumentProvider provider =
            Activator.getDefault().getFooFileDocumentProvider();
        setDocumentProvider(provider);
        setSourceViewerConfiguration(new FooSourceViewerConfiguration(
            getPreferenceStore(), this, provider));
    }
}

Note that the functionality described in this article is only available since Handly 0.3, which will be released tomorrow (June 24th). The initial API should be considered provisional. Feedback can be directed to the project’s forum or right to the developer mailing list (the list requires subscription).

Handly 0.3 : API enhacements

Handly 0.3 includes a number of API enhancements; some of them are breaking changes. Here I would like to focus on the most important ones:

ISourceElement: remove hard dependency on ISourceFile (Bug 461262)

Source elements no longer have the getSourceFile() method. Such method, if necessary or desired for a specific model, can be provided by the model implementor, probably in model-specific terms.

The problem with this method was that it had been originally specified to return the source file containing the source element and could not return null, assuming, thus, that source elements are always contained in a source file in the workspace. It was realized that this assumption didn’t hold for models like the JDT Java model that support source attachment for “binary modules” such as class files. For example, in the Java model, both ICompilationUnit and IClassFile consist of pretty much the same elements: ITypes, IMethods and IFields, which all extend the ISourceReference interface (that roughly corresponds to ISourceElement in Handly).

Fortunately, this design flaw was fixed in Handly 0.3. Unfortunately, a few breaking changes had to be introduced to make it possible. As a rule, we try to preserve backward compatibility, but in this case it was not feasible, and it is hoped that future benefits should outweight present costs.

Here’s the complete list of breaking changes related to this redesign around source elements:

  • ISourceElement#getSourceFile() method got removed
  • ISourceElementInfo#getSnapshot() may now return null (if no source is available)
  • ISourceElementInfo#getFullRange() and #getIdentifyingRange() may now return null, but may no longer return a NULL_RANGE (see below)
  • TextRange#isNull() and #NULL_RANGE got removed. The special source range with a -1 offset and a 0 length was a JDT legacy that I did not readily identified as such. It would be very painful to deprecate it (JDT couldn’t): that’s why we had to do it as soon as possible. For a bit of history, see bug 130161.

Deprecate ISourceFileFactory (Bug 464588)

The interface ISourceFileFactory has been superseded, for all intents and purposes, by IElementForEditorInputFactory, and may be removed in a future release. The new interface is more generic and capable. There is little point in having an imperfect API except for backward compatibility: ISourceFileFactory was allowed (and intended) to be implemented by clients.

So, if you have previously implemented ISourceFileFactory (and, for Xtext languages, bound it in the Xtext UI module for the language, as was required by Handly/Xtext integration), please consider implementing (and binding it in the UI module, for Xtext languages) the new interface IElementForEditorInputFactory instead. A typical implementation might look like this.

Handly 0.3 : Outline framework and quick outline support

A major new feature of the upcoming 0.3 release of Handly is an outline framework (org.eclipse.handly.ui.outline package). Only a part of it depends on the handle-based model API provided by Handly; much of the code is fairly generic and can be reused in other contexts, thanks to a multi-layered design. Handly 0.3 also includes quick outline support (org.eclipse.handly.ui.quickoutline package), which nicely complements the outline framework. A detailed description of both facilities is beyond the scope of this note, but the Basic Example (org.eclipse.handly.examples.basic.ui) in Handly SDK now uses the outline framework and offers quick outline in the Foo editor; it might be a good place to start. The API is fairly well documented (in Javadocs) and is expected to be quite easy to use.

As a teaser, compare the from-scratch previous implementation of the outline page in the Basic Example with the corresponding new implementation that uses the outline framework. The new implementation is both richer in ability and less in size. And you can easily do a lot more with it: this more advanced implementation provides problem marker decoration and a user-toggleable filter with just a few additional lines of code.

The initial API should be considered provisional. I would like to invite potential adopters to take a look and leave a feedback; as always, it would be much easier to fix any discovered deficiences early on. Feedback can be directed to the project’s forum or right to the developer mailing list (the list requires subscription).

Handly 0.3 : Example model for Java

A highlight of the upcoming Handly 0.3 release (just a few days away) is a Java model example (org.eclipse.handly.examples.javamodel*), which demonstrates a Handly-based model for the Java language along the lines of the JDT Java model.

Back in Handly 0.1, a basic exemplary implementation was provided: a complete Handly-based model for a simple Xtext-based language, with the outline page and navigator view implemented atop the model. Intentionally contrived to be simple so as not to explain a lot of stuff that isn’t Handly-related, but at the same time sufficient for demonstrating many design and implementation aspects, this example provided the basis for a later multi-part tutorial, which has received some acclaim from early adopters.

It was obvious, though, that a more advanced example was really needed. Such an example would be called for to demonstrate how to implement some of the more advanced features with Handly and, ideally, spur further Handly adoption.

In particular, the following requirements for this example had been gathered via the project’s forum:

  • “Real-world”;
  • Demonstrate how to deal with a classpath-like concept (e.g. source folders);
  • Exemplify model integration with a custom text editor (i.e. non-Xtext).

After a discussion with early adopters, it was decided that an example model for Java would fit the bill. It could exemplify all the features mentioned above and would be sufficiently real-world (despite the inevitable simplifications vs. the full-fledged JDT implementation). Besides, adopters already know Java (and probably JDT internals too), so it sounded like a useful common ground. Also, JDT provides some important services (like parsing) that could be reused by the example code.

So, I set to work on the Java model example. The outcome of this work for Handly 0.3 is the core model and a bunch of tests. There is no corresponding UI yet. As one whould expect from an example, the model is somewhat contrived. In particular:

  • Only classpath entries of kind CPE_SOURCE, with paths of length 1 are considered (i.e. only source folders that are direct children of the project resource);
  • Inclusion/exclusion filters are ignored — everything is included;
  • Only default output location is supported, with path of length 1;
  • Initializers and local/anonymous types are not represented in the model.

Some of these constraints may be relaxed in the future (especially based on community feedback).

The following UI parts are expected to appear in a subsequent release:

  • Example Java editor, with Handly-based reconciling and outline;
  • Example Java navigator view.

It is hoped that this example will be useful to Handly adopters. In particular, I tried to craft the series of commits carefully to show how one can grow the model organically. For instance, it can be seen that at one point there was no effective model cache/delta processor, only a bare model structure that, however, was fully functional and testable. Only after the model structure was fully fleshed out, the model cache/delta processor were implemented. This is the way I would recommend for developing a Handly-based model.

Feedback is most welcome and can be directed to the project’s forum or right to the developer mailing list (the list requires subscription).

On M-theory and H-models, or can EMF fit every nail

Those readers who are interested in modern physics may know about M-theory that seems to be gathering a lot of hype in recent years. It is the latest toy of theoretical physicists, which many think could become the ultimate “theory of everything”. It is a subject of ongoing debate, however, whether there actually can be a single unified theory capable of explaining everything or, instead, it is a set of theories, each describing different aspects of the universe (with some overlap). And if one adopts model-dependent realism, it doesn’t really matter :-)

In Eclipse world, EMF has been steadily gaining traction. Rightfully, EMF is such a nice, flexible tool that many problems start looking like a nail. But can an entire IDE (like JDT) be essentially EMF-based? Xtext might lead many to believe so, but let’s take a closer look and see how JDT is structured from the modeling point of view. After all, it reflects thirty years of experience!

It may be surprising that there are at least three different models in JDT, each describing different aspects of the IDE (with some overlap) and having very distinctive characteristics of its own:

  • Java model

    I like to think of the Java model as the IDE ground (and glue). This model is the common foundation for many core services as well as all views and actions in JDT. One can also think of it as a Java-oriented projection of the Eclipse workspace.

    The Java model is a handle-based model (H-model for short). Its elements are handles: value objects that act like a key to a model element. This gives it a number of special properties that are very useful for a large-scale in-memory model of the whole IDE:

    • Stable (elements to which a reference can be kept, e.g. to show in a view)
    • Light weight (model non exhaustive, no resolved information kept)
    • Lazily populated, LRU cache (doesn’t hold on resources)
    • Eventually consistent (need not be consistent all the time, allows “dirty reads”)
    • Thread safe without imposing much locking (fosters high concurrency)

    That’s quite unique combination of properties, if you think of it. It’s not a mere coincidence that the underlying workspace resource model is also an H-model (with slightly different properties).

  • Abstract Syntax Tree (AST)

    Each AST node represents a Java source code construct, such as a name, type, expression, statement, or declaration. The parser parses a string containing Java source code and returns an AST for it.

    It is important to note that the AST is really a tree (no cross-references). Since ASTs tend to consume a lot of memory, JDT holds only one AST, for the currently active editor, reconciling it as necessary. Clients should not keep references to elements of the AST.

  • Bindings

    When an AST is created by the parser, there is an option to also create bindings. A binding represents a named entity in the Java language, such as a package, type, field, method, constructor, or local variable.

    The world of bindings (an environment) provides an interconnected picture of the structure of the program as seen from the compiler’s point of view. This is the only JDT model with graph structure, i.e. cross-references. One can think of it as the semantic model of the language.

It seems only one of those models (Bindings) can be naturally represented with EMF. Another (AST) might fit EMF, with some caveats. The remaining one (Java model) is not a good fit for EMF at all. (Beg to differ? You are welcome to leave a comment. I may be missing something.) It looks like “M-theory” for Eclipse-based IDEs (where ‘M’, in this case, stands for Modeling) would indeed include a set of models with different goals, properties, and requirements to the underlying implementation technology.

In particular, the common needs of H-models like the Java model are to be addressed in Handly, an Eclipse technology project that is about to turn one year old. Handly (at its core) supplies basic building blocks that help developers create their own H-models.

A nice bonus is that many useful IDE components (micro-frameworks) could be implemented atop the common API for H-models provided by Handly — there must be a considerable potential for unification. As an example, the upcoming Handly 0.3 release, scheduled to coincide with Eclipse Mars, will provide an outline framework (including quick outline support) for H-models. The outline functionality is already in place, actually, in the project’s git repository and latest I-builds.

The vision behind Handly is not that of an all-encompassing IDE framework with its many inherent assumptions. Rather, it envisages a set of loosely-coupled micro-frameworks around H-models — that common IDE ground. If you are interested in giving Handly a try, there’s a multi-part tutorial on GitHub. Beware though: if you are new to H-models, it takes some getting used to. But if you are keen to know what principles make JDT (or other Eclipse-based IDEs) tick as a seamless whole (the *I*DE), I think it might be worth it.

Of course, Handly doesn’t prescribe a specific implementation technology for the other IDE models. They could be EMF-based (like in Xtext; in fact, Handly already includes a layer of integration with Xtext) or something else; it doesn’t matter much when one adopts model-dependent realism!