Groovy 3 semantics and new MOP








Jochen "blackdrag" Theodorou



Last modification 


Abstract: Groovy 3 semantics and new MOP

For some time we have been thinking about a new MOP for Groovy but haven’t been able to decide on a final design which mostly keeps backwards compatibility but also improves various cases that we want to improve. It is difficult to find the right path between being user-friendly and efficient. Thus the new MOP will contain some features the old did not, but will also remove some old features for the sake of a more straight and more easy to understand MOP.


Removing default null argument

The default null argument is used for method calls that have one parameter, but the call is done without an argument for the parameter. Groovy will here use null, as long as the type of the parameter is not a primitive. In case of a primitive the call fails. This feature was mainly added in the early Groovy pre 1.0 years before Groovy supported default values for parameters. Another limitation of this logic is that it works only with single parameter methods. There have been multiple cases of confused users by this logic. Performance wise this logic doesn’t cost much, and certainly not more than a call with a real argument. But since this is a confusing feature of limited use, it should be removed.

Removing automatic list expansion

A method call done with a list that finds no matching method for that list (a method with one parameter of type List, Collection, Object, etc.), will cause a second method selection iteration. This time the list is "unpacked" and all elements of the list are taken as if the method call had been done with the elements rather than the list. Groovy also supports spreading of lists by a syntax element, making this automatic feature not needed. In fact this can be quite surprising for users and is a problem for performance. A spread version might be still not good in performance, but at least the user will have to use an extra symbol and thus have the visual indicator. As of why this feature was originally added is unclear. Looking at user code you will find barely intended usages of this. Thus it should be removed.

Changing Safe Navigation to stop evaluation

Currently an expression like a?.b.c will fail if a is null. It will not evaluate b, but it will try to evaluate c on null. This defies the intent of safe navigation to avoid a NullPointerException. Thus this should be changed to stop the evaluation of the path expression.

User Wish List

  • Instance based Categories by Wujek Srujek
    Instead of having to provide a class with static methods it would be good to be able to feed an instance to the use-construct and that then we will use this instance along with its instance methods. This allows instance state to be used.

  • Replacing Introspector by blackdrag
    Not only because of bugs like GROOVY-5019 there should be a replacement of the Introspector

Some leading design ideas of this MOP

The plan is to orientate us a lot on the open call site caching the JVM provides with invokedynamic. For this to work all parts of the MOP should no longer be seen as places that do invoke something, but as places that return something, that then will be invoked. An invokeMethod then will for example return instead an object that acts as a kind of handler, which can be invoked. Groovy will then store it and avoid the reselection unless you invalidate it. In the old MOP such caching often did not happen once you interact using meta programming. The tasks to be solved in this are to provide an "extension point" for intercepting methods and to react to missing methods, as well as being able to invalidate a cached version and/or to make an uncached version possible.

MOP2 outline


in metaclass of x in pseudocode:

cachable = getMethodInterceptor(...)
if (cachable==null) cachable = getFittingMethod(...)
if (cachable==null) getMethodMissingHandler(...)
if (cachable==null) throw new MethodMissingException(...)

as an explanation to the methods:

  • getMethodInterceptor
    Extension point for the user to register an interceptor, that will be used for all method calls. This is equal to what GroovyInterceptable does using invokeMethod, only that in MOP1 the method is directly called. In MOP2 we will instead return a handler that then will do whatever is needed. This can for example be a method handle to a invokeMethod method if wanted. TODO: define guards

  • getFittingMethod
    Predefined way to select a method through metaclass. This method will return an object which then can be used to invoke the selected method.

  • getMethodMissingHandler
    Extension point for the user to register a handler for missing methods. The then invoked method is comparable to what methodMissing in MOP1 would be doing.

  • MethodMissingException
    thrown if none of the above methods is bearing a non-null result

  • storeInCallSite
    Predefined method to store the cachable result of the above process in the call site. As long as the cached object is valid the above methods will not be called again for this call site.

  • invoke
    Predefined method to do the Initial invocation of the cachable. Subsequent invocation may be done directly by invokedynamic.

Context Meta Class

For the MOP2 Groovy will use a system of metaclasses with a context element and an always existing default. At each call site only one such view will be valid and it will be constant. Those views can be used to defined "sealed" metaclass, which won’t get influenced by outside metaclasses or to allow for example calling private methods and not allowing them in other cases. This makes 3 possible metaclass variations currently:

  • Default metaclass
    Behaves pretty much like what Java allows in regard to method calls (same class private methods are available) and all meta programming changes are visible

  • Sealed metaclass
    Like the default metaclass, but meta programming changes from outside the class are not visible. This can be especially useful to library writers.

  • Test metaclass (un)sealed
    Like the two before but with the addition that private members will always be available.

Getting a Meta Class

Because of the concept of context a class does not have one direct metaclass that can be generated without its context. The call site defines the place of the context. How the context itself is defined is a TODO. As an implementation strategy it is possible to for example use ClassValue to store a table with the context being a key. The key would probably have to be available as static information, or as easily computable information. Since the resulting metaclass could be stored later in the call site object context changes are to be avoided, since it implies the invalidation of the call sites using that context.

General dispatch rule for methods (and properties)

To define the dispatch rules correctly we need to define some terms first:
Static Sender Class (SSC): This is the static information about the class a call is made from. If there is for example a class A and a class B extends A, and a call in a method in A, then even if your instance is actually a B the SSC will still be A.
Inheritance based Multimethods (short multimethods from now on): Given a class A and a class B extends A, a call made from within A may see a method defined on B as long as the method is visible (not private). Groovy defines a special exception to this though. If the method call from within A is calling a method of the name m, then a m from B can only be visible if there is no private m defined in A.

Given those two definitions a method call in A will select the set of method to decide from based on this: A call m() with the SSC A and done on an instance of B (extends A) will be using the methods defined in A, if A has a private m, otherwise the call is done using B.

Calls to Super:
A call to super in B extends A will have the SSC B, but for the method selection process the super class of SSC (super(SSC)) will be used. In super calls multimethods are not visible. Thus we can directly use the metaclass super(SSC), but we will dispatch only on the public methods of that metaclass.

Module Extension Methods Shadowing Rules

Module Extensions Methods are in the old and new MOP defined by the DefaultGroovyMethods related classes and module extension, like groovy-swing. In the definition here we will use the terms of from "inside" and from "outside" to define a callsite, that lies in the same class as the target method (inside) or not (outside). The general rules are:

  • public methods are shadowed

  • private methods are shadowed for outside callsites, but not for inside callsites

Subclasses of the class the module extension method has been applied to have these extended rules:

  • if the subclass defines a private method of the same signature as the module extension method, then outside callsites will still see the extension method, inside callsites the private method

  • A call to "super" or "this" will call the module extension method. As such the subclass is seen as outside callsite.

Open Blocks are not seen as separate classes.

Property Discovery

Currently MetaClass discovers properties based on the Java Beans conventions. It also allows pseudo properties matching a convention on java.beans.EventSetDescriptor. This allows the following trick in SwingBuilder for example:

button(actionPerformed: { println it })

The pseudo property actionPerformed is inferred from the single method exposed by ActionListener, a type of listener that can be registered on a JButton. The code responsible for discovering these properties is buried in MetaClassImpl and is not accessible to the outside. It would be great if this mechanism be made pluggable.

The Realm concept

In MOP2 a Realm is a tree like structure containing the metaclasses. There is a root realm, used as default, but there can be any number of lower realms. A metaclass change is visible in a realm, if the change is done to the metaclass in the same realm or to a metaclass in a higher realm. Script execution engines are able to set a realm for example to prevent them changing metaclasses they should not change. This can be used for unit tests to isolate metaclass changes done during the tests as well. A library can have its own realm (defined through an annotation) to prevent other classes to leak their changes into that library, while the library can still use a higher realm to make changes more public visible, if the realm allows that. Realms can have a setting that prevents code executed from there to make changes to higher realms. Calling a method is always done using the metaclasses from the current realm, even if the called class then calls other classes using its own realm. A realm is thus not thread local structure, it is more of a lexical scope. A realm can also use a different metaclass flavor, to for example allow access to private methods and fields.

Work Items

This part is to guide the implementors with the course of action and planning of the subtasks.

  • make indy the only compilation target in the build

  • move all non-indy bytecode interfacing code to a module, which may be removed later. This includes ScriptBytecodeAdapter as well as all the custom call site caching classes

  • make a new module for MOP2

  • turn metaclass into an immutable

  • implement metaclass views

Breaking changes trace

groovy.lang.MetaObjectProtocol (currently in groovy.mop.MetaObjectProtocol):

  • getProperties() renamed to getMetaProperties()

  • getMethods() renamed to getMetaMethods()

  • respondsTo(Object, String, Object[]) changed to respondsTo(String, Object…​)

  • respondsTo(Object, String) replaced by getMetaMethods(String, Class…​) with the class argument being null

  • hasProperty(Object,String) replaced by getMetaProperty(String) being null or not

  • getStaticMetaMethod(String, Object[]) replaced by respondsTo(String, Object…​) and inspecting the list for static methods

  • getMetaMethod(name, Object[]) replaced by respondsTo(String, Object…​) in case the arguments are no classes and getMetaMethods(String,Class…​) in case of the arguments being classes

  • invokeConstructor(Object[])NO REPLACEMENT

  • invokeMethod(Object, String, Object[]) NO REPLACEMENT

  • invokeMethod(Object, String, Object) NO REPLACEMENT

  • invokeStaticMethod(Object, String, Object[]) NO REPLACEMENT

  • getProperty(Object, String) replaced by MetaProperty#invoke

  • setProperty(Object, String, Object) replaced by MetaProperty#invoke

  • getAttribute(Object, String) replaced by MetaProperty#getField#invoke

  • setAttribute(Object, String, Object) replaced by MetaProperty#getField#invoke

  • getMetaProperty(String), getTheClass() UNCHANGED

groovy.lang.MetaMethod is split into a public interface groovy.mop.MetaMethod and an internal default implementation groovy.mop.internal.DefaultMetaMethod.

Differences to groovy.mop.internal.DefaultMetaMethod:

  • does no longer extend ParameterTypes and does no longer implement Clonable

  • no protected fields nativeParamTypes, parameterTypes and isVargsMethod

  • the constructor MetaMethod() and MetaMethod(Class[]) are removed and partially replaced by DefaultMetaMethod(Class, String, int, MethodHandle) and DefaultMetaMethod(Class, String, MethodType), which uses the MethodType or the MethodHandle to define the parameter classes

  • coerceArgumentsToClasses(Object[]), correctArguments(Object[]), isValidExactMethod(Class[]), isValidExactMethod(Object[]), isValidMethod(Class[]), isValidMethod(Object[]), isVargsMethod(), isVargsMethod(Object[]) NO REPLACEMENT

  • getNativeParameterTypes()replaced by getParameterClasses()

  • equal(CachedClass[], CachedClass[]), equal(CachedClass[], Class[]), checkParameters(Class[]), clone(), doMethodInvoke(Object, Object[]), getDescriptor() NO REPLACEMENT

  • getDeclaringClass(), getModifiers(), getName(), getReturnType(), isAbstract(), isPrivate(), isProtected(), isPublic(), isStatic(), toString() UNCHANGED

  • getMopName(), getSignature(), invoke(Object, Object[]), isCacheable(), isMethod(MetaMethod), isSame(MetaMethod), processDoMethodInvokeException(Exception, Object, Object[]) NO REPLACEMENT

groovy.lang.MetaProperty is split into a public interface groovy.mop.MetaProperty and an internal default implementation groovy.mop.internal.DefaultMetaProperty.

Differences to groovy.mop.internal.DefaultMetaProperty:

  • the public static field PROPERTY_SET_PREFIX is removed NO REPLACEMENT

  • the protected fields name and type are now private and have to be requested through getName and getType

  • getModifiers(), getName(), getType(), DefaultMetaProperty(String, Class) UNCHANGED

  • getGetterName(String, Class), getSetterName(String)NO REPLACEMENT

  • getProperty(Object) replaced by getter(boolean)

  • setProperty(Object, Object) replaced by setter(boolean)

Mailing-list discussions

Reference implementation

Update history

3 (2013-10-11)

Version as extracted from Codehaus wiki

4 (2018-10-28)

Numerous minor tweaks