Developer’s tool in Pharo

I’ve finished an internship in RMOD team in Inria and I was doing some retrospective in the work I’ve done.

The idea for my internship was to improve the developer experience while using Pharo [http://www.pharo-project.org/], (quite challenging!), so in order to do this I was mainly focused on 4 points:

  • Suggestions
  • Code Navigation
  • Syntax Coloring
  • Class Definition representation

Suggestions
While we are coding we usually want to apply actions depending on the element we are writing/seeing, for example if it’s a variable we may want to rename it. But in order to do this, we have big menus to find what we want, usually with lot of options that don’t apply.
The idea in smart suggestions is that based on the context offer only the relevant options.
We use the current AST to do this through RBParser (parseFaultyMethod:parseFaultyExpresion:) and Opal Semantic Analysis.

We choose RBParser because we can parse faulty expressions, with this feature we can offer suggestions while the user types the method.

Opal is the new compiler integrated in Pharo, it’s a very clean implementation and we think that will replace the old compiler, so we didn’t want to couple suggestions with an old compiler. The semantic analysis helps us to complete information form a code, because there is information that we can not know from the code without the context, by example: if we have a temporary variable or an instance one.
With the best AST node for the selection we have the available suggestions and with the semantic analysis we go one level deeper for the variables getting to know the nature: temporary, instance or class to refine the suggestions.

asd

If you want to define a new suggestion you only have to annotate a method with the scope where you would like to offer the suggestion and return an object respecting the SugsSuggestions API.
For returning the object you can extend SugsSuggestion and redefine the desired methods, you can see various examples with the current implementation in the SmartSuggestions-Suggestion package.
Also you can instantiate a generic suggestion: with SugsSuggestion class>>#for:named:icon: where the first parameter is a block with the action to execute and receives a valid SugsAbstractContext.

The available scopes are:

  • <assignmentCommand>
  • <classCommand>
  • <classVarCommand>
  • <instVarCommand>
  • <literalCommand>
  • <messageCommand>
  • <methodCommand>
  • <sourceCodeCommand>
  • <tempVarCommand>
  • <undeclaredVarCommand>
  • <globalCommand>

Here we have an example defining a Dummy action in

DummyAction class >>


newFormattingSuggestion
 <globalCommand>
 ^ SugsSuggestion
 for: [ :context | context formatSourceCode ]
 named: 'My global formater'
 icon: Smalltalk ui icons smallFindIcon

Navigation
Sometimes while browsing code we think in programming terms instead of text, for example we think in a messageSend or a statement instead of word, spaces or symbols.
The idea is to use context information and let the programmer navigate the code thinking in those terms.
In order to do this we find the best ast node through the RBParser and we offer navigations in different directions:
– Parent: The node that contains the selected one. For example if we have the code ‘aNumber between: anotherNumber’ and we are selecting the variable anotherNumber if we navigate to the parent we will go to the message send.
– Sibling: The node in the same level as the selected node. For example in a temporary variables definition: ‘| one two three |’if we are in the variable one we can go to the siblings two or three.
– Child: Node contained by the selected node. For example if we are in a message send: ‘aNumber between: anotherNumber’ we will go the parameter anotherNumber.

It’s very easy to see how the Suggestions + Navigation work together, and how with not too much effort you can improve a lot.
If you want to activate the Node navigation using Command (or Control if unix) + Arrows go to the System Settings and activate: AST Navigation.
ASTColoring
We want to color the syntax we are writing having the most information possible, in order to be able of select the scope where we are or show information associated to that piece of code.
In order to do that we use the AST and the semantic analysis (we need the semantic analysis because we want to show different kinds of variables with different colors, like undeclared variables), through the RBParser (parseFaultyMethod:/parseFaultyExpression:) to obtain the AST representation. The implementation it’s very simple because we can define a new Visitor to which we delegate the coloring algorithm and once we define the coloring from each syntax representation we just visit the tree: ast acceptVisitor: “visitorImplmentation”.
To enable the syntax coloring activate in System Settings: Enable AST based coloring.

Class Definition
Include a representation modeling the arguments and the message as 1st class object. About this point I have written a whole blog post.
Summary
After working with the AST I can say that is a very powerful tool and also is very easy to use. The visitor model is great and very flexible. But we must be careful because we can easily add too many responsibilities there and complicate the design.
Changing the AST structure has an obvious impact in the visitor implementors, and in the current Pharo implementation this can be complicated because we have some interesting users of the AST (like the new Compiler), and if we make a mistake it is very easy to break the image, so if we are going to do some experiments with this it is better to change the compiler first.
Obtaining an AST representation is not so hard to do when we have valid code, but when we are trying to find the best representation for a faulty expression things get nasty, and if we want to colour the text while we are writing we need this feature.

But the main point of everything is the re-affirmation that Pharo is a great tool fordoing experiments, the power of a live environment and reflectivity give us an edge over other environments.
With not so much effort we can implement some tools that improve a lot the programming experience a lot, and everyone can do this!

A look into Spec

Today I’m going to write about implementing the UI.
Since I want to have a nice UI in Pharo I choose Spec to build it, because it’s an on going project and there are lot of effort there, also I want to try it :).
Searching I found some tutorials and demos but some were quite old and not fully compatible with current version of Spec, so I will do a brief tutorial.

Spec was developed by Benjamin Van Ryseghem, you can find more information in: https://pharo.fogbugz.com/default.asp?spec.5.5.2

Anyway I think that the best place to take a look is in the examples that comes under Spec in Spec-Examples-PolyWidgets (mainly) and Spec-Examples-Widgets.

Doing the minimun for opening a Window
We want an empty window… but our empty window, so… how?
1. We are going to create a class extending ComposableModel

ComposableModel subclass: #TestSpec
instanceVariableNames: ”
classVariableNames: ”
poolDictionaries: ”
category: ‘Flamel-UI’

2. We have to implement 2 methods
2.1. initializeWidgets which responsability is to setup the components we will use (right now nothing)

initializeWidgets
“nothing to do here”

2.2 defaultSpec (class side) which returns a Layout saying which components will be render. There is a number of different layouts implemented: SpecColumnLayout, SpecRowLayout, and in the class methods of SpecLayout you can find a lot more. We will choose a very easy one:

defaultSpec
<spec>
^ SpecLayout composed

In fact you can name the method as you want, only need to be annotated with the <spec> pragma.

3. Open the window, we evaluate in a workspace:

TestSpec new openWithSpec

And we obtain:

emptyWindow

Nice… but quite empty… let’s put something inside!

Adding a component
1. Choose the widget to add, you can see some in Spec-Widgets, we want a text area so we will choose a TextModel.
2. In order to render the component we need to: create it and the tell the layout that we want to render it
2.1. Create the widget: define an instance variable and instantiate it during the initializeWidget using the Spec way to do it:

initializeWidgets

self instantiateModels: #(
text TextModel
)

We need to define text as an instance variable because instantiateModels assume that and try the elements of the array as an association: variable – kind where variable is a property in the object and makes it point to a new instance of the kind, in order to avoid the troubles we will define the variable.

Be careful with this (as Sean mention) if you define the variable while you are debugging you can get some nasty errors, so the advice is to define the variable first and then open the window.

At this point if we get anxious to see our cool text component and we open the window we will see that is still empty, this is because we never added it to the layout! So it’s not rendered.
2.2. Add the component in the layout, for this we will modify the defaultSpec class method and add a getter for the text property:

defaultSpec
^SpecColumnLayout new
add: #text;
yourself

We need the getter because the layout tries to access the variable sending the symbol as a message
3. Open it

withComponent

Communicating Components
Now we can open a window and have our components, but now we want to have several components that depends between them.
We will build a window with a text editor, when the content in the text editor changes we will see the message “The text changed” we can accept that pressing a button that says “Ok” and then the message will change to “You are up to date” (until someone change the content in the text …).
For doing this we will use:
– a TextModel called: example
– a LabelModel called: changes, starts with the text: “You are up to date”
– a ButtonModel called: acceptChanges, starts disabled and with the label: “Ok”

initializeWidgets
self instantiateModels: #(
example TextModel
changes LabelModel
acceptChanges ButtonModel ).

changes text: ‘You are up to date’ .
acceptChanges
label: ‘Ok’;
disable.

This is like we had before, only that we use other components that needs a little more of configuration.

The real challenge here is to link some effects: when the contents of example changes we want to change the message in changes and enable acceptChanges.
For defining this kind of interactions we should define the method initializePresenters defining the interactions, lucky our models understand several useful message to define an action to perform for an event, an example for a TextModel #whenTextIsAccepted: , #whenTextChanged:, …
Let’s define the actions:

initializePresenter
example whenTextIsAccepted: [ changes text: ‘The text Changed’. acceptChanges enable ].
acceptChanges action: [ changes text: ‘You are up to date’. acceptChanges disable ].

In this part we are done, now remember that in order to render the components we need to add it to the layout:

defaultSpec
<spec>

^ SpecColumnLayout new
add: #example;
add: #changes;
add: #acceptChanges;
yourself.

And we see:

3iguales

Quite easy and good, but lets improve the layout and play a little:

defaultSpec
<spec>
^SpecLayout composed
newColumn: [ :mainColumn| mainColumn
add: #example;
newRow: [ :feedbackRow | feedbackRow
add: #changes;
add: #acceptChanges]];
yourself.

2iguales

Better but still a little detail:

defaultSpec
<spec>
^SpecLayout composed
newColumn: [ :mainColumn| mainColumn
add: #example;
newRow: [ :feedbackRow | feedbackRow
add: #changes;
add: #acceptChanges
] height: 26.];

yourself.

final

A lot better! And It was quite easy, we can see that is well separated the layout from the components, and that we have a nice proposal for organizing our UI-Code:

  • We define the layout in #defaultSpec
  • We instantiate the components we are going to use in #initializeWidget
  • We define the interactions in: #initializePresenter.

This is very nice, but now I have a UI that separates the logic but still has a lot of responsibilities, is the UI who knows if I’ve accepted a change or if I’m up to date, also knows exactly what to tell me… I don’t think that this is the UI responsibility I want to model a class and that the UI shows it and not all in one like we have now.
And how do we do it?

Using a Model
We will have a model behind our window in 3 easy steps:
1. Create the model when initializing (you can create it or receive it, as you want, I will show the simplest way)

initialize
model := AcceptableText new. “replace for your domain class”
super initialize .

2. define the widgets, is almost the same that before

initializeWidgets
self instantiateModels: #(
example TextModel
changes LabelModel
acceptChanges ButtonModel ).

acceptChanges
label: ‘Ok’;
disable.

3. define the interaction

initializePresenter
example whenTextChanged: [:changed|
model text: changed.
self updateContents].

acceptChanges action: [
model acceptChanges.
self updateContents ]

updateContents
changes text: model status .
acceptChanges enabled: model hasUnreadChanges .

Look that when we are defining the actions we do 2 things:

1) modify the model
2) update the content of the widgets to show the new state of the model

We can see this actions that are using blocks as controllers (in a MVC pattern) since they modify the model from an input in the view and then update the view to show the new state in the model, in other frameworks we can use Bindings that do exactly the same.

In fact Ben pointed that Spec is far more closer from a MVP pattern than an MVC

In order to write a good UI we have to be very careful in how much code goes there, we probably will like to have small controllers delegating in the model instead of big controllers that does all the work for the model if we are not careful there we may end with anemic objects and spaghetti uis.

But to much talk and see how it ends:

final

Basically the same that before but now we have the linked ui using our domain object! Quite nice, and really really really simple to use.

The class definition message

Q: What is a class definition in Smalltalk?

A: Just another message send!

I love that simplicity… We don’t need control structures because we send messages and as we have an abstraction that represents some piece of code everything is happiness (yeah… blocks are cool stuff).

But still knowing it I go to the System Browser… I see the class definition and I don’t think that is another common message send, in fact when I realize that I can write that “magic code” in a workspace and have the same effect (creating the class) and see it in the system browser I said wow! After a while I thought “so lame… of course is just a message send”.

But however I go to the class definition again select an instance variable and I want to apply the rename refactoring… so… I ask for suggestions… and I got… nothing… just selecting a string and that confused me and then I realize, ok it’s just a message send and is just a string as argument so… that’s logic. The thing is that is not true that is just a message send… because for some reason we expect more in some points, by example when we are in the class definition…

So I did some prototypes for changing the representation and meanwhile I change my mind in the process.

1. The class definition is not represented as a message node

I was radical… I said: A Class Definition is always a class definition, so I changed the parser and when parsing I tried to parse a class definition
The prototype for this can be found in:

MCHttpRepository
location: ‘http://smalltalkhub.com/mc/gisela/ParsingClassDefinition/main&#8217;
user: ”
password: ”

I started with some issues here… The main one is:
How do I know I’m in a class definition, in the prototype you will see a very “innocent” implementation for realizing this… I just assume that if the message identifier I parsed is #subclass: it’s because a full message definition. There is no way to know if you are in a class definition until you finish the parsing.
Lot of the behavior is similar to a message send, and to implement the program node API we have to see how to share this code, it seems an efort without a real motive, because all the problems I was having relies in the fact that I assumed that a class definition is not a message send…

2.The class definition is created from a message send

Something in between, I still changed the parser but… transforming the message node parsed only if it was a class definition.
The prototype is in:

MCHttpRepository
location: ‘http://smalltalkhub.com/mc/gisela/MessageClassDefinition/main&#8217;
user: ”
password: ”

I had less problems 🙂 but still some strange behaviors, it was because I changed the AST and all the users of the ast should change, specially the visitors because now you have classDefinitionNode, categoryNode, variableDefinitionNode (…).

And then I have lots of failling test and for olving in the major part I endedUp delegating in the message node reference I had hold.
But this was expected, since I changed the AST structure in some point all the users of that structures need to be modified, but ended up having like the same that before… lot of behavior shared between a message node and a class definition, and sometimes the “new representation wasn’t even desiarable.
By example asking for suggestions… if I’m writting a method and write somthing like:

Object
subclass: #JunkClass
instanceVariableNames: ‘zzz’
classVariableNames: ”
poolDictionaries: ”
category: ‘DeleteMe-1’.

If I ask for suggestions in zzz I expect something like… “Extract local” and no “Rename” but in the definition space… I want that behavior, so… since that I decided that in some ocassions you look the class definition message as a special stuf, and that depends on the context where you are, so changed the parser wasn’t so great because at that point you don’t how they would like to treat that particular message, that it’s mostly user-decision.

3. Request the class definition explicitly

I don’t change the parser, I add a message to the nodes “asClassDefinition”

MCHttpRepository
location: ‘http://smalltalkhub.com/mc/gisela/ClassDefEnhaceTree/main&#8217;
user: ”
password: ”

And then let the user of that tree decide in which case he is and if it’s who apply to do the transformation or not.

With this approach the direct user of the ast must know if he wants this representation or not, the original tree doesn’t change, and we can think that is that we let that two representation exists, is a similar approach to the semantic analysis by default you don’t have the information and if you want that you explicitly need to say… ok do the analysis.
And in the end

I finish in the beginning, the AST represents the syntax structure and that is how we understand it and how we manipulate it, sometimes we want a transformation, we want more and we enhance that structure for better understanding.

At least for the suggestions it was only in the class definition view. And after dealing with the consequences of change I get convinced that we send message and that’s it, we don’t have and neither need an special syntax for defining a class, is just sending the right message to the right object, it’s simple and it’s very good.

And when we want another approach now we can have it, very easily.

 

Defining the UI

The main feature we want for the rewriting tool UI it’s to be example driven. Two very similar examples:

Option 1 – All in the same view

Imagen

Option 2 – A tabed view

Imagen

Imagen

Option 3 – A 3-Column layout

Screenshot from 2013-06-14 21:57:42

We also want to be possible:

  • select a method and drop into the example text editor.
  • manipulate the ASTs nodes in the text editor to make simple define transformations and matching.
  • highlight the matched parts from the example.
  • select scopes to apply the rule.
  • preview the changes.
  • show the transformation in the example expression.
  • modify an expression throw the example or writing the pattern.
  • output the smalltalk valid code that produces the effect we see in the example.

Next steps…

Now we have a very powerful tool but quite complex to use… The pattern’s definitions ends up been confusing and lot of people dismiss the tool for this.

The fact that the patterns are strings is one of the causes in misunderstanding and encourage to produce lot of mistakes, when we want to match an expression thinking in a method definition we think in terms of: a method contains this statement, or declares this variable and not in complex regular expressions, it’s also true that sometimes we end up with a cryptic expression, hard to understand by others (even for ourselves).

It would be great to have a very expressive api for what we want to do, for example make possible to say something similar to:

“some scope” pattern
includesStatement: anStatement;
hasTemporaryVariable: aVariable.

We should write some nice examples of usage and also take a look into the rewriting engine, we could do lot of improvements.

All of this sounds very ambitious, so I will start for defining an UI for make the patterns auto generated in an easy way with lot of cool features, specially example driven when building expressions.

Once we have the UI and a nice video showing how to use it… we can improve the API and make it more expressive and if we have time we will take a look in the rewrite engine… but first let’s do these easy to use!