smalltalkhubUser: ‘gisela’ project: ‘Flamel’;
If take a look in the metavariable patterns you can see that it can get as complex as you want.
But, let’s focus in a particular token. We end up with a metavariable, but we can see a pattern from simplicity to complexity:
1. Just abstract the token as a metavariable: we do not care if the method contains the exact word we want everything that contains something acting as that.
Maybe we do not care about who is the sender of the message #size. In order to do that, we will abstract the variable ‘aVariable’ as a metavariable:
2. Abstract a list of tokens: we do not care if the method cointain not only one element also a lot of elements.
If the pattern is `object size we are too specific because we are only matching the send to size whom receiver is an object, if we have something like this:
(aCollection select: #even) asOrderedCollection size.
Is not a result, if we want that, we have to specify that we do not care:
3. Abstract a recursive pattern: still maybe we want matches if the pattern is repeated
aVariable size size.
If our pattern is: `object size, this does not match. If it’s: `@some size we have one match when some = ‘aVariable size’.
If we put a recursive pattern:
We have two matches: one when some=’aVariable size’ and another when some= ‘aVariable’
4. Abstract a statement: Maybe we do not even care about the hole statement, and for this we should use the dot.
We can think that in the abstraction patterns you have a variety in abstraction, since one more concret with more information until one more abstract.
And for that we can conclude that we have a flow during the metavariable abstraction:
NoAbstraction <=> Simple <=> List <=> Recursive <=>Statement
In order to support this in flamel I’ve added #FlamelAbstractionStep that basically have 3 main responsabilities:
- next step: abstract
- previous step: unAbstract
- pattern: pattern
You can take an expression, obtain the AST representation and then just send to the node the messages abstract and unAbstract to let the abstration proccess to continue, once you finish just use the expression builder to obtain the expression.
testAbstractVariableNodeWritesRightPattern | searchExpression variable root expression | expression := 'aVariable aMessage'. root := RBParser parseExpression: expression. variable := root receiver . variable abstractOnMatching. searchExpression := FlamelSearchExpressionBuilder new searchExpressionFor: root. self assert: searchExpression equals: '`aVariable aMessage'
If we do:
variable variable abstractOnMatching. variable variable abstractOnMatching.
We will end up with: `@aVariable aMessage.
If you want to get fun annotating nodes I added some protocols to maintain the state in a node:
- abstractOnMatching: instead the source code for the node we are interested in the metavariable. Each time you say abstractOnMatching you are going up in the matching steps
- deleteOnMatching: when we want to match, we should not include the code contained inside the node
- ignoreOnMatching: we go to the last step in the matching process
- unAbstractOnMatching: we want the previous step for the metavariable
I Also wrote a visitor to genrate the matching expression, so in order to obtain the expression, you can evaluate:
FlamelSearchExpressionBuilder new searchExpressionFor: root.
Ok let’s start to understand the rewrite engine.
Let’s do it with an example, our goal:Find all the methods that send the consecutives mesages: #globlals #at:
In our example we want as result:
findClassesForCategories: aCollection | items | aCollection isEmpty ifTrue: [ ^ self baseClass withAllSubclasses asSet ]. items := aCollection gather: [ :category | ((Smalltalk organization listAtCategoryNamed: category) collect: [ :each | Smalltalk globals at: each ]) select: [ :each | each includesBehavior: self baseClass ] ]. ^ items asSet
But not with this variation:
findClassesForCategories: aCollection | items | aCollection isEmpty ifTrue: [ ^ self baseClass withAllSubclasses asSet ]. items := aCollection gather: [ :category | ((Smalltalk organization listAtCategoryNamed: category) collect: [ :each | Smalltalk globals at: each ifAbsent: [ "ignore" ]]) select: [ :each | each includesBehavior: self baseClass ] ]. ^ items asSet
Why? because in the second example we send the messages #globals (ok we want this!) but then we send #at:ifAbsent: (we don’t want this)
Ok so… Which object do I send wich message to find my “problematics” methods?
The answer is very simple a RULE… For each rule you should extend one flavor of rule and define what you want to do. Sounds easy let’s check the rules hierarchy
All of those are abstract classes, so we will take a look into them and choose the one that is better for our case.
- RBLintRule: defines the protocol used for execute a rule.
This is a very general class, doesn’t define:
– what to do when we are cheeking a class, for this we have to implement: #checkClass:. In our case we don’t care about the class, so we can let the default that does nothing with the class
– what to do when we are cheeking methods, for this we have to implement: #checkMethod:.
In our case we should obtain the source code form the method and search in the source code all the variants for the consecutive messages #globals #at:, an inocent version of this can be:
A way to do it:
checkMethod: aContext (aContext compiledMethod source findString: 'globals at:') ~= 0 ifTrue:["we have a match in aContext selectedClass>> aContext selector we should handle it"]
I said that it is an innocent implementation because if we have a method with the code: “(…) globals “yes a comment here!” at: (…)” or “(…) globals at: number” we aren’t causing a match and if we have: “(…) globals at: something ifAbsent: (…)” we are and we shouldn’t.
Probably we should allways use one subclass of LintRule with more features implemented otherwise we have to extend LintRule to define what to do in case of matching, model a result, transforming code (…)
- RBBasicLintRule: it’s another abstract class that adds a result associated to a rule. We still should redefine checkMethod and/or check class but we have a proposal for a result:
checkMethod: aContext (aContext compiledMethod source findString: 'globals at:') ~= 0 ifTrue:[result addClass: aContext selectedClass selector: aContext selector ]
This can be a good extension point, but again… you have a lot to implement here.
- RBBlockLintRule: it’s an abstract class, that by default specifies that the resultClass corresponds to a selector environment.
This class doesn’t add a lot of behavior but we reduce errors related to the result handling.
- RBParseTreeLintRule: here is where everything gets interesting.
The main point in this class is to offer an implementation for #checkMethod: the implementation motivation is to check methods using ast’s representations.
We will have a wanted tree and then we will obtain the AST for the checked method, in order to find a match we observe if the wanted tree is subtree of the method tree.
Also it’s posible to define metavariables where we can specify that we want to match with a node type but not necesary the value of it.
In our example our matching expression will look like:
“@lotOfStuffBefore globals at: “@lotOfStuffAfter
With this expresion we are saying:
– “@lotOfStuffBefore: ignore everything before globals (receiver / message send):
– “@lotOfStuffAfter: ignore everything after at: (object / messages)
(For more information about metavariables see my first post)
The solution implemented for the check method introduces new objects:
– Matcher: Is the responsable for visit an AST and verify the matches. Is the matcher the one who interprets the matching expressions. First of all we have to specify if we are matching with a method (matchesMethod: aString do: aBlock) or an expreesion (matches: aString do: aBlock) pattern.
We can add more than one matching expression, it’s very important to know that every time we add an expression in the matcher inside we are adding a RBSearchRule for that expression.
When the tree rule sends the message executeTree: to the matcher at the end the matchers iterates over all the rules visiting the nodes delegating in the RBSearchRules to perform the match.
It’s important to say that the search rules aren’t deleted automatically, so, if you want to reuse the object probably you should reset the rule, reseting the matcher.
– ParseTreeEnvironment: Is an specialized SelectorEnvironment that makes it possible to detect the selection interval for an expression inside the method, using the ast matcher
This class is still abstract because we should categorize it, adding a name and initilize the rule with the matching expression patterns.
In functionality is almost the same that before but we have other abstraction level, the result is handled automatically, and we don’t have to worry about checking a method or a class only to define the desired matching expression.
- RBCompositeLintRule: Is just a composite for rules.
- RBTransformationRule: The main idea here is to produce a transformation in the system, for this implements: #checkMethod: in a similar way that RBParseTreeLintRule the diference is that if we find a match we will produce a modification in the code and then we change the method with the new version of the code (recompiling the new method).
In order to solve this the rule adds some objects:
– RBParseTreeEnvironment: with all the results, the results are: RBAddMethodChange to track a change in a method
– RBParseTreeRewriter: It’s a subclass off the matcher (RBParseTreeSearcher) and again the main point is that this is a visitor that works over a method AST changing it depending in the matching and transforming expression.
The transforming expression also works with metavariable and usually we use the metavariables defined in the matching expression to specify the transformation.
- Before start you should choose if you want to:
perform a search
do a match
- Implement your rules because all of them are abstract, probably you will end up using the Tree rules because are more automatics and powerfull than to basics.
- When you are defining your rule:
– give a name for it
– define if you will use a method or an expression type pattern
– write your patterns and add them to the rule
– if you are in a search rule to the matcher
– if you are in a transformation rule rewriteRule
– define what to do with a result
- run your rule
- use your results
- if you want to reuse it reset your rule, again if it’s a search rule, reset the matcher, if it’s a transformation rule the rewriteRule.
In our example:
- we want to match, so let’s create an object that extends: RBParseTreeLintRule:
RBParseTreeLintRule subclass: #SearchGlobalsAtUsage instanceVariableNames: '' classVariableNames: '' poolDictionaries: '' category: 'Blog-example'
- we have to implement the abstract methods:
name ^ 'Find all potential wrong usage in with globals'
- I want to match an expression type because I do not care about the rest of the method, I want everything that contains the messages: #globals #at:, we also have to say what to do with the matching node in this example I will open an inspector:
initialize super initialize. self matcher matches: '``@lotOfStuffBefore globals at: ``@lotOfStuffAfter' do: [:theMatch :theOwner | theMatch inspect].
- now we should run the rule:
WARNING: this can take a time because you will check the hole system
SearchGlobalsAtUsage new run.
To avoid this you can restrict the environment for your rule, an example:
rule := SearchGlobalsAtUsage new. environment := RBClassEnvironment class: Result. RBSmalllintChecker runRule: rule onEnvironment: (environment).
If you have matches then you will see the inspector.
So, as we can see this is quite complex, and in the sinposis you can see that before doing anything you have to make too many decisions, the idea behind Flamel is to make it easier.
The equivalent code (with a restricted environment) using Flamel for all this is:
FlamelMatchAndTransformRule new matchingExpression: '``@lotOfStuffBefore globals at: ``@lotOfStuffAfter'; scope: environment; run; result
If you evaluate that and inspect it you can search your results 🙂
I think this is quite cool to replace all that code (with class creation included) with 5 lines.
And this was all for understanding a little bit the rewrite engine and see Flamel in action for today
I want to show a proposal for using the matching expression builder.
The idea is to keep it simple, so if you think this is very complex, let me know!
First of all open Flamel-UI, in a workspace:
Flamel new openWithSpec.
For those non familiarized with the project, you will see something like this:
- Scope for the rule: Here we restrict the scopes for the rule, for those familiarized with the RW-Engine behind this we have the RBEnvironment objects.
- Define match: The idea here is to help to define the matching rule, for this we have:
- Starting point example method: We choose a method to use as an example for the pattern, you can change for the method you want, and you have more options available (just, right click and see)
- Pattern expression type: 3 options here:
- Method: The pattern corresponds to a method
- Expression: The pattern corresponds to an expression, so we don’t have the selector definition plus the optional temporary variables definition
- Both: Basically, try to search the matching expression as a method and as an expression
- Matching expression: The pattern we will search
- Define transformation: The same idea for the matching but in this case to help you with the transformation
How to use Flamel in 5 stepChoose the scope for your rule
if we don’t clarify this we will look in the hole image and this takes time
- Select your base example: You can use the default example or… choose an example method that is better for you
The example will help you to define the pattern
For change the example, right click in the Starting point example method and choose: “Change example”, you will see a window like this:
- Choose the Pattern expression type: If you are not sure, just let the both and we will try to do our best, but if you really know if you want to define a pattern with a method syntax or not (an expression) choose the best option.
- Define the matching expression: For this we use the right-click menu options to help us.
We can start choosing “Reset expression” and then we will see the same code in the example as a pattern
Then we use the menu to modify the expression
- Perform the search just clicking in the Search button.
And then see your results!
It took me many hours (and too much coffee) to realized all of this and I want to share with you some tips to help you define the matching pattern you really want to aplly
- Pay attention to the dots! `sel “@.Statements1. self subclassResponsibility. “@.Statements2 is very different to: `sel “@.Statements1 self subclassResponsibility. “@.Statements2
- Matching the selectors is not a simple task
- Is not the same matching an expression that matching a method one line can change everything!
If you say:
</pre> matcher <strong>matchesMethod:</strong> aMatchingExpression
You are particullary saying that aMatchingExpression should be parsed doing: RBParser parseRewriteMethod: aMatchingExpression instead of RBParser parseRewriteExpression: aMatchingExpression
And this mean that:
- Your string is well formed, if you have a syntactic error you will see the window saying that the expression is not correct
- Your string has a method structure
- Your metavariables will play as a message send or as a variable depending on the context
A fast example, imagine this expression:
Let’s play we are the a cool parser and someone told us hey… this is a string, but you should interpret it as a mehod… so:
`anObject = the selector so I want all the unary methods
size = a variable named ‘size’ because it can’t be a parameter because my message is unary, it’s not a variable definition it has to be a variable.
So as a cool parser I say that I will match with all the unary methods that the body is just an unique sentence that contains a variable called ‘size’
Now, imagine someone told us this is an expression… so:
`anObject = the object, I do not know nothing, only that here goes an unique object
size = before I had an object so here I should have an invocation for the message #size.
So as a cool parser I say that I will match with all the methods the expressions that sends the message size
- If you look for a particular message send you should care about the structure:
- Do you want all the senders? for defining the expresion is not the same object messageToFind that object message anotherMessage messageToFind moreMessages
- It can be inside a block?
- It can be invoked as a symbol? object perform: #messageTofind
- It can be in any part of method? (the first sentence? after that can be more message sends?)
- The matcher uses pattern matching! You give an alias and when it matches then is bounded and that’s
`sel “@.Statements1. self messageTofind. “@.Statements1
Does not match with:
example self oneMessage. self messageTofind. self otherMessage.
Because we use the same variable name! Statements1 and in the matching it bounded to the statement self oneMessage that is NOT identical to self otherMessage so we don’t have the match.
But! It should match with:
example self oneMessage. self messageTofind. self oneMessage.
But this feature is really cool if you want to search repeated code in the same method…
- If you are matching inside a method, does the method defines temporary variables?
- Remember that when you parse if the variable definition is wmpty it get’s ignored but if you do not put it in your pattern and the method defines a temp you are excluding it.
example |aTempVar| self messageTofind.
matches with method pattern:
`sel |`@vars| `object messageTofind
but does NOT match with:
`sel `object messageTofind
In the end I can understand that all this “tinny” details are very important… but if you want to match a common case it can be really ugly and it’s VERY easy to do it wrong.
But do not worry too much one big goal for Flamel it’s to offer a simple API to avoid all this commons errors.
I think that this had happend to a lot of smalltalkers with the time… After a while you just get used to do some tasks that aren’t very intuitive for a newcommer and started to feel “natural”. Yes… the classic user that get used to a system…
And these days I’ve been watching the frustation of someone that is starting the path and trying to help I just realize that I have a lot of tips and tricks that I follow unconcious and decided to make an entry here maybe this help and more people with more expirience would share their tips.
Before start your coding you have to setup your environment, an one choose is which image? Wich version? I strongly suggest the lastest but maybe you should keep reading before choose.
Why does the title say developing under develop?
Because when I start a new project in pharo I want to use the lastest features, Pharo is getting really cool and has improved a lot and to use all those feautures I need the lastest image.
Other reason is that while you are programming your system you are testing Pharo also and time to time you find some problems and correct them are not so hard, so it pushes you to improve Pharo itself!
So in order to have all the benefits you have to use the lastest image (under develop) and with that you should have concience in the consecuencies and take some considerations.
Ok you are in a system under developing, IS NOT PERFECT but it is in the path to be one step near to the perfection and to achive that needs your help. Basically applies to the system the same rule that applies to your development:
- The system may change: some apis can change and maybe you are an user of that API. This is not bad! Probably it’s a refactor to improve the code, so you get benefit because your code will be more expressive.
- The system can have bugs! Yes as I said is not perfect and it’s not on purpose.
- Sometimes the lastest image have problems, yes we would love to be perfect but sometimes there are some nusty side effects and some functionality that stop working, but to find it sometimes you have to release and wait for feedback and you are a great candidate to give that feedback.
- The debugger does not mean that everything is lost, your first approach sholud NOT be close the debuger and say abandom abandom!
- Get attach to an image is not a great idea… Your “lastest” image is not there for stay with you for a long time, you have chossen to use under develop image, this means that your image will change (in the happiness path but change anyway) and deliberated decide to attach with an old particular version from a changing image has the worst of two worlds: Now you are not in an stable but you are stuck in a version that nobody is using and by consecuence you don’t have the new fratures/fixes
- The tests are your friends 🙂 (As usual!)
Tips and tricks!
If you are new at Pharo y strongly recomend to take a look in Mariano Peck‘s blog particullary the post called: Pharo tips and tricks. The shortcuts are really cool and from your previous experience maybe you would say… I can’t do that only with my keyboard… but the true is that probably you can do it! Just take a look in Key mappings
Mitigate the errors
Some tips when you are developing:
- Use fresh images, if you are in lastest Pharo a week with the same image is really A LOT
- Allways use a repository to share your code, I strongly recommend smalltalkhub
- If your project has a complex setup invest some time making a configuration
- Do not make a lot of actions before commit, small and numerous commit is too much easier than one big commit
- Reverting the changes is not so easy. There are a lot of effor in this topic and is comming, but is not ready! So, try to keep it simple
- If you find extrange behavior send email to the user mailing list: email@example.com nobody will judge you for asking!
- Lenguage barrier is not so hard with a tolerating comunity, Pharo comunity is full of non native-english speaker and is more important to improve Pharo than writing poetry, just ask and try to make that understandable (look… if I’m writing this blog with my horrible english, you shouldn’t care at all)
- If you can reproduce a problem open an issue in the issue tracker
- Save your image regulary, if you have worked for a while and you have uncommited changes save your image regulary!
- Before killing your image because “does not respond” you have the resource to interrupt the process just press alt + ‘.’ or cmd + ‘.’
- Contribute and give feedback, it’s a way to keep growing
I’m sure we have a lot more, but at least is a beginning!
One image it’s a great way to show the current status:
As you can see we still have (a lot of) work to do… But we can:
- Choose the scope where we run the run (right now a class/es or a package/packages)
- Enter the expresion for matching and transforming
- Search the matches
- Apply the entire rule (matching expresion + transforming)
- We have some actions implemented for the menus (same code, containing code, defining a variable)
- Inspect the result for a Search or Transformation action
But still now I have some thing that I observed: The rewrite engine its in fact very cool, and surely does everything we want to do.
But it’s very complex you have to choose the matcher , modify the pattern (is an expression? is a method? you have multiple options? is about an argument?), set the environment (do you want to run where? each Environment is represented but wait! Which matcher are you using? Because not all have the feature, and they are not all polimorfic when you want to match. And of course do not even talk about the regular expresions and the usual doubts we have (with temps? with N arguments? previous statements? and don’t forget the dot!), and… before starting… do you want to match? or do you want to transform? You have to choose!
And usually you don’t have all the information to choose that in an early moment, so what I’ve done it’s to implement a rule that delegates in existing components but takes the desitions for you… By example: Do you want to transform? Or just match? That depends… do we have a transformation expression? If that is true we want to search and replace.
testMatchAnUndefinedExpressionNotFind | rule | rule := FlamelMatchAndTransformRule new. rule matchingExpression: 'testFlamelFlamelNotDefinedInAnyPlaceNever'. rule scope: (RBClassEnvironment class: FlamelSearchMock ). rule run. self assert: (rule result isEmpty )
I want to change that to be something like:
testMatchAnUndefinedExpressionNotFind | rule | rule := FlamelMatchAndTransformRule new. rule matchingExpression: 'testFlamelFlamelNotDefinedInAnyPlaceNever'. rule applyToClass: FlamelSearchMock. rule run. self assert: (rule result isEmpty )
So, I have lot of work!
Some known bugs, and some unknown you can check the todo list in a Trello dashboard: https://trello.com/b/XqfJGqeB/flamel
It would be great to improve the UI because it’s quite “rustic”.
And my plan for this week it’s:
- Implement result views ==> with this one we have a first real prototype for the tool because we can: filter the scope, enter a matching/transforming expression and understand the results
- Implement actions for transforming menu
To open the browser just:
Flamel new openWithSpec
All the comments that you have are very wellcome, please don’t hestiate in comment or send me an email directly to me: firstname.lastname@example.org.
Have a great week and nice coding!