Background

I was looking at a way to process auto-generated tweets, like the ones on http://twitter.com/moneyvidya_com. Some sample tweets:

• #moneyvidya arunthestocksguru (5 Star rated) says Buy Vijay Shanthi Builders – 6m (Monday 29 March 2010 @ 09:55 … http://bit.ly/bd5JgC
• #moneyvidya arunthestocksguru (5 Star rated) says Buy Bhagwati Banquets & Hotels – 6m (Monday 29 March 2010 @ 09… http://bit.ly/9MzRDG
• #moneyvidya NSV (5 Star rated) says Buy ACC – 6m (Wednesday 24 March 2010 @ 09:55 AM): http://bit.ly/b5xTrN
• #moneyvidya justsurjit (5 Star rated) says Sell Sesa Goa – Intraday (Monday 22 March 2010 @ 10:31 AM): http://bit.ly/9lLo8U

As it is clear, the text follows a specific format, but has its own little variations. I intended to process the ‘insights’ and see each expert’s success rate. Although I never got around actually completing the task, I did learn a lot about text processing.

Approach

The apprentice – Regular Expressions

The first approach was the most obvious one – regular expressions. I am sure RegEx would have addressed the particular task at hand. But the parsing expression would become a convoluted mess very soon. So I started looking for better alternatives.

The strict teacher – Lexical Analysis

Lexical analysis starts where regular expression give up. This also needs pretty strict rules on the allowed input text, but the rules could be a lot more flexible and easy to comprehend.

I especially enjoyed using Irony, which makes it trivial to convert BNF formed rules to C# code. There is a good gentle introduction to lexical analysis using Irony on code project.

The guru – Natural Language Processing

Processing test using tools like NLTK, allows you to parse and understand any unstructured text and understand it. Although this gives you maximum freedom, it also needs a lot of work to get right. For this to produce good results, be sure you have lots of data to be able to tweak and test your implementation. I guess this is the reason Google and co., can do so much better at translation, since they have huge data available for improving.

Conclusions

I don’t have one . I guess, there are several ways to solve a problem, and half the solution is to identify the best way to solve the given problem. As for me, it was a good learning exercise and may come in handy if I ever write a DSL.

Here is a list of programming languages available on these runtimes.

1. Can run on CLI using IKVM.NET
2. Can run on JVM using Mainsoft solution
3. Not yet usable
4. Can run on CLR, but is behind the JVM implementation

The main reason for the research was to identify a new language I should pick-up. I looked at Python and Ruby, but both have some sore thumbs that I just can’t stand. I really liked Boo and Groovy; they are similar to C#/Java in syntax and incorporate the good things from Python. Although I like Boo’s syntax and approach more than Groovy, Groovy has a more mature implementation and ecosystem. I will try to use Groovy for some hobby project and get a feel to things.

Sample program with obligatory screenshot

Showing Flickr Image

   1: url = Flickr.GetPictureOfMoment()

   2: img = ImageList.LoadImage(url) 

   3: GraphicsWindow.Title = url 

   4: GraphicsWindow.DrawImage(img,0,0) 

   5: GraphicsWindow.Height = ImageList.GetHeightOfImage(img) 

   6: GraphicsWindow.Width = ImageList.GetWidthOfImage(img)

The Good:

• The interface is toy like and very simple. Will surely win favours with 10 year olds.
• The limited with just 15 keywords is good too.
• The learner does not need to initialise variables, there is no new.
• No variable scopes, everything is global.
• Before you think it will bring forth another set of programmers, like the much maligned VB programmers, remember this is for 10 yr olds.
• In general, it feels light weight and fast.

The Bad

• The application itself depends on .NET 3.5 which still does not have a large installed base.
• For sake of simplicity, all libraries are more or less global objects. This approach might not scale when more libraries are added.
• The turtle is non-interactive and verbose. How I long for the good old code which went something like:

   1: FD 100 

   2: RT 90 

   3: REPEAT 4

• Same application in SmallBasic:

   1: For i=1 To 4

   2: Turtle.Move(100) 

   3: Turtle.TurnRight() 

   4: EndFor

• The result:
• 

  Notice there is no area to type the next command, all commands to turtle have to be specified upfront.

The Ugly

• The syntax is very ugly. Why the curly brackets?

   1: 'You need the curly brackets for all function calls.

   2: GraphicsWindow.Show() 

   3: 'Why not drop the brackets for zero argument functions?

   4: GraphicsWindow.Show 

• The current syntax forces the learner to understand the different between a property and a function. Completely unimportant for the target audience.
• To add to that it is not consistent, you still have a few functions that have a property pattern (GetSomething, SetSomething)
• What would look better, VB.NET style or a mix of original BASIC and ruby?

Conclusion

I really dig for anything related to teaching programming to children. Although my daughter is too young to program, I draw from my experiences of learning programming. I, like countless others, was introduced to BASIC and Logo as the first programming language. But I was not interested in adding 2 numbers or drawing a square. I became interested in programming when I was introduced to dBase III (remember it?). When I found I could save and retrieve data and manipulate it, I was sold.

Similarly, I think, today’s internet generation might be sold on network based interactivity. SmallBasic takes initial steps in that direction by having a Network object and a Flickr object. This will be a good direction to pursue and add more libraries for APIs from other web apps. Hackety Hack approach, I guess.

The other thing interesting to this generation is gaming and multimedia. Although Scratch, Alice, Phun and others like Phrogram are filling that need, a few libraries towards that end (animations, effects, sound, video), will not hurt at all.

In conclusion, I think this is a very interesting start, but needs to cover much ground, while remaining true to the promise of simplicity.

Things I liked:

• Type inference: Could have been better, but I will take this any day over the verbose Java alternate
• Binding: In fact this is a great thing. Two way binding and binding with expression.
• Triggers: Need to explore more, but shows promise.
• Timelines: This along with the exceptional support for multimedia will help in creating the next killer app.
• Collections: You can iterate over collections in SQL like syntax which,to me, looks better than LINQ.
• Strings and Dates: Finally they get treated with the respect they deserve, since most of the time one is juggling text and dates.

Things I am meh about:

• Declarative UI design: I believe UI design is best left to designers (software not people)
• Using all the available brackets: All examples look like they have used up every possible punctuation mark

  :,=,{},[],(),;

• Init vs. assignment: The ambiguity on where to use a variable : value and where to use a variable = value

Overall looks good and I am going to spend some time learning the innards of Javafx.

Finally, what I will like to see guidance on how patterns will evolve to address this new form of development. I immediately see a lot of older patterns not longer needed like Singleton, Visitor Pattern, Lazy Loading, Thread Pool Pattern, Observer Pattern and more. Similarly we need to recalibrate a few like the MVC, MVP patterns.