My search took me to Lucene’s HindiStemmer, which in turn led me to the paper by Ananthakrishnan Ramanathan and Durgesh D Rao: A Lightweight Stemmer for Hindi [PDF]. It was a good initiation to how some simple rules could stem most Hindi words.
The problem was, it was for words written in Devanagari script not Roman. So I decided to implement the logic for transliterated Hindi. After some refinement, I ended up a large subset of what the paper does, because I wanted to keep the implementation simple.

At the end of a few hours work over a few days, I ended up with one line of code. (I will be paid almost nothing if I were being paid by KLOC written)

re.sub(r'(.{2,}?)([aeiougyn]+$)',r'\1', word)

For people who are regex challenged, the above regex, deletes all vowels along with g,y,n from the end of the word, but leaves at least a 2 character long stem, so that words like ‘aayenga’ do not completely vanish.

The above regex will stem the words as below:

What do you think? How could this be improved? What edge cases are not considered?

We can then predict a positive case if the hypothesis outputs a value above a certain threshold, which normally is 0.5 but could be more or less. The ideal threshold can be derived based on the cross validation result. More on this in a later post.

Hypothesis

As mentioned above, logistic regression is similar to linear regression, but the hypothesis always returns an output between 0 and 1. This is achieved by passing the hypothesis of the linear regression to a Sigmoid function.
Thus the hypothesis is denoted as:

hθ(x) = g(θTx) 

where g is the sigmoid function defined as:

Cost Function

The cost function for classification problems are a bit more involved due to the fact that simple squared error is not sufficient to work with the small differences here, since the max difference possible is 1, predictive 0% probability for a positive case. Therefore to magnify the differences a log scale is used.

Vectorized version:

J = sum(-y .* log(hypo) - (1-y).*log(1-hypo)) / -m

Derivative of cost function (gradient)

The gradient for hypothesis is the same as that of linear regression:

The hypothesis of course is different as stated above.

Multiclass Classification

So far we have only looked at 2 distinct results from the algorithm, either negative (y=0) or positive (y=1). But what if there are more than 2 distinct possibilities? This is covered under multi class or one-vs-all classification.

The basic idea is simple. We train the classifier once per each possible outcome. For e.g. If we need to predict if the given sentence is in English, German or French, we will derive the value of θ for each.

So the θ for English will be able to predict whether the sentence is in English or not, thus changing the problem to a single class (positive/negative) classification. Similarly the θ for German and French are derived.

Then for a given sentence we will calculate the probability of it being English, German or French and return the language having the highest probability.

This model finds weights (θ) to be assigned to each feature, such that sum of the weighted features is closest to the given answer y.

Hypothesis

So given there are n features x₀ to x_n and the related weights θ₀ to θ_n; the hypothesis is given as:

hθ(x) = θ0x0 + θ1x1 + ... + θnxn

If θ and x are stored as matrices, then the above can be represented as matrix multiplication and is called vectorized implementation. Vectorisation really speeds up the processing and was one of the big lessons on the entire ML class. The vectorized version of the above hypothesis is:

hθ(x) = θTx

Cost function

So what remains is to find values of θ those results in the hypothesis predicting values that are close to the correct value given.

There could be several definition of close, but the most accepted definition is average squared error. In other words it is the sum of the square of the difference between the actual value and the predicted value, divided by 2m where m is the total number of samples.

Vectorized version:

sumsq(θTx-y) / (2*m)

Optimisation Algorithms

So far so good and here comes the difficult part. Now we are left with finding the values of theta which minimizes the above cost function. We could do this with trial and error, and maybe come up with an answer in a few years. Thankfully there are several algorithms developed by people much smarter than me and I could use their work to get the results. There were few approaches discussed in the class.

• Normal Equation
• Gradient Descent
• Advanced algorithms
  • Conjugate gradient (fminunc & fmincg)
  • BFGS, L-BFGS (just mentioned)

We mostly used gradient descent and fminunc / fmincg in the class, and both expect a cost function that returns the cost and the gradient.

Derivative of cost function (gradient)

The cost was defined above and the gradient for linear regression hypothesis is given as:

(for j = 0 to n)

Vectorized version:

X' * (X * theta - y) / m

Implementation notes

Details will be covered in another post.

• Apply feature scaling
• Apply mean normalisation
• Add a new feature (x₀) which is all ones
• It helps to visualize the data
• It helps plot the cost after each iteration
• There are several other practical tips given throughout the class, and I hope to blog about it as well.

Machine learning

Arthur Samuel (1959): Field of study that gives computers the ability to learn without being explicitly programmed.

Machine learning algorithms (covered in the class):

• ­Supervised learning
  • Linear Regression
  • Logistic Regression (Classification)
  • Neural Networks
  • Support Vector Machine

• ­Unsupervised learning
  • k-Means clustering
  • Principal Component Analysis

My understanding of machine learning:

Given a set of examples with certain features; the ability of  a computer to approach, and surpass, the ability of a human expert at analysing and extracting meaning out of the given data.

Important points about ML:

1. If a human expert does not find the data sufficient to come up with a conclusion, then the computer is unlikely to perform any better.
2. All machine learning algorithms are based on mathematics, and thus expect all data to be numbers.
3. Usually more data is better, but not if the data is redundant. That is, duplicate examples or features.

More to come in future posts.

The very mention of this word brings out different memories and emotions for different people. More so if is a framework for be used within a large company, by a completely different team than the one writing it.

To make sure that the next time you hear this word, you have positive memories; use the evaluation sheet below, before adopting the framework. If you are the one writing the framework, make sure you have good answers to these.

Why in-house frameworks?

Before we begin, let’s recap why we need in-house frameworks in the first place.

Each company, more so is it is an ‘enterprise’ has a certain set of internal standards and a ‘way to do things’. Be those be related to security, encryption, authentication, authorisation or be it about data retention, traceability, logging or monitoring. The foremost benefit of an in-house framework is out-of-the-box support for these standards and hence any solution built on them is preapproved by the standards committee, while saving the developers from reinventing the wheel for these.

These frameworks also know the company’s domain and can provide a lot of helper functions at the least to a full blown DSL on the other extreme. Thus an in-house framework is a good thing to have if it can pass the evaluation criteria.

How to evaluate an in-house framework?

Well, it is no different from evaluating a external framework, be it commercial or an open-source one.

• What is the origin of the framework?
  • Was it created from a blank slate, based on past project learning? If so, has it been battle-tested? You don’t want to be the guinea pig.
  • Was it extracted out of software delivered? Is it still tightly coupled to the environment the original software ran in?
• What is the team’s motivation to maintain this framework?
  • Is the framework their primary task or is it a distraction? This is especially the case when the framework is either extracted from a project or was primarily written for a specific project.
• What is the development model?
  • Does it mimic a commercial vendor? What is the governance model around support and enhancement request?
  • Or does it mimic open-source model? What is the governance model around roadmap, vision and acceptance of patches?
  • Does the framework team have a consistent vision to avoid the framework from acquiring feature-bloat, and at the same time is it extensible enough for a team using it?
• What is the learning curve? How well documented it the framework?
  • This is the single most important question to ask. For external frameworks, you have the internet to help you out, but for internal ones, the documentation is normally the only hope. In addition, you can hire people with skills in external framework, but will need to train every new hire on the internal framework, so it better have a short learning curve.
• What is the release and support model?
  • You don’t want to be using a framework which does not have a specific release cadence and a know support model for older versions. Neither should the support model allow proliferation of versions in the
    wild, nor should it force and upgrade every 3 months.
• Finally, will it last?
  • Is there organisational and budget support for the framework team?
  • Will the framework continue to live after the lead developers have left?

Only past midnight, did it dawn on me that there is no documentation, because it is so darn simple! So as a note to self, and to save someone an hour or two, here’s how to run a Bottle app on Dreamhost.

In a rush? Get the code at github.

Setup a domain

Follow the steps listed in the wiki, to setup Python/Passenger support for a domain.

Create a passenger_wsgi.py

This is where the Apache Passenger module hooks into our application. There are 4 steps needed to create this file.

Set the path

All dreamhost documentation on python advises that you setup a virtual environment. Since bottle has no dependencies this is not needed. Instead just add the current directory, where bottle.py resides, into the system path.

import os, sys cmd_folder = os.path.dirname(os.path.abspath(__file__)) if cmd_folder not in sys.path:  sys.path.insert(0, cmd_folder) 

Passenger hook method

def application(environ, start_response):  return bottle.default_app().wsgi(environ,start_response) 

Write a main method

This will be used to run a development server, in debug mode and auto reload.

if __name__ == "__main__":  bottle.debug(True)  run(reloader=True) 

Write your application

Well, you will need to write your own application

@route('/') def index():   return "it works!" 

That’s it, save it as passenger_wsgi.py.

For local development, run it as

python passenger_wsgi.py

on dreamhost, stuff will just happen.

Additional notes

Create restart.txt

To restart your application on dreamhost, run this command:

touch tmp/restart.txt

in the base folder, i.e. the parent of the public folder defined in the “Web directory” of the dreamhost domain setup.

Server static files directly

It is better to let apache/passenger take care of serving static files. Any file found in the public directory will be served directly and bottle routing will not be called.

For local development, define routes to server static files out of public directory.

@route('/css/:file') def css_file(file):  send_file(file, root='public/css') 

Full code listing on github.

Some photos taken at the Technology Museum in Munich.

The ENIGMA

One of the first crypto machine.

An Enigma machine is any of a family of related electro-mechanical rotor machines used for the encryption and decryption of secret messages. The first Enigma was invented by German engineer Arthur Scherbius at the end of World War I. This model and its variants were used commercially from the early 1920s, and adopted by military and government services of several countries – most notably by Nazi Germany before and during World War II.

- http://en.wikipedia.org/wiki/ENIGMA

Random set of Keyboards

Mechanical Calculators (Arithmometer)

An Arithmometer or Arithmomtre was a mechanical calculator that could add and subtract directly and could perform long multiplications and divisions effectively by using a movable accumulator for the result. Patented in France by Thomas de Colmar in 1820 and manufactured from 1851 to 1915, it became the first commercially successful mechanical calculator. Its sturdy design gave it a strong reputation of reliability and accuracy and made it a key player in the move from human computers to calculating machines that took place during the second half of the 19th century.

- http://en.wikipedia.org/wiki/Arithmometer

IBM System 360 (Mainframe)

The IBM System/360 (S/360) was a mainframe computer system family first announced by IBM on April 7, 1964, and sold between 1964 and 1978. It was the first family of computers designed to cover the complete range of applications, from small to large, both commercial and scientific. The design made a clear distinction between architecture and implementation, allowing IBM to release a suite of compatible designs at different prices. All but the most expensive systems used microcode to implement the instruction set, which featured 8-bit byte addressing and binary, decimal and floating-point calculations.

- http://en.wikipedia.org/wiki/IBM_System/360

IBM 2560

Multifunction card machine (reader/punch/interpreter/multi-hopper)

Siemens 2002

The Siemens 2002 was a transistor computers manufactured by Siemens & Halske.

In 1954 its development began. The first prototype was fisnished in 1956 and distrubutions started in 1959. The Siemens 2002 was manufactured until 1966.

The system used magnetic core memories as memory had as extension, a drum storage. Different peripheral devices could be attached such as paper tape devices, page printers, card equipment, magnetic tapes and high speed printers.

Japanese Keyboard

IBM 729 V – magnetic Tape Storage

The IBM 729 Magnetic Tape Unit was IBM’s iconic tape mass storage system from the late 1950s through the mid 1960s. Part of the IBM 7 track family of tape units, it was used on late 700, most 7000 and many 1400 series computers.

- http://en.wikipedia.org/wiki/IBM_729

Stack trace in Java

Command Line

If the application is running as a console application, you can try one of these:

Sending a signal to the Java Virtual Machine

On UNIX platforms you can send a signal to a program by using the kill command. This is the quit signal, which is handled by the JVM. For example, on Solaris you can use the command kill -QUIT process_id, where process_id is the process number of your Java program.

Alternatively you can enter the key sequence <ctrl>\ in the window where the Java program was started. Sending this signal instructs a signal handler in the JVM, to recursively print out all the information on the threads and monitors inside the JVM.

To generate a stack trace on Windows, enter the key sequence <ctrl><break> in the window where the Java program is running, or click the Close button on the window.
(Excerpt from http://java.sun.com/developer/technicalArticles/Programming/Stacktrace/)

If the application is not running in a console, you can use the jstack tool that ships with J2EE since 1.5

jstack [ option ] pid
jstack [ option ] executable core
jstack [ option ] [server-id@]remote-hostname-or-IP

GUI Tools

Since the JVM provides APIs to hook in and get the thread state and other information, there are several tools in the market that allows you to see the state of the running application including the thread stack trace. One such freely available option is VisualVM. In fact it ships with JDK these days. Although VisualVM is pretty self explanatory, here are some instructions to get you started. http://docs.oracle.com/javase/6/docs/technotes/guides/visualvm/threads.html

You could also use jConsole for similar purpose, but I prefer VisualVM. Check this for more information: http://java.sun.com/developer/technicalArticles/J2SE/jconsole.html#DeadlockDetection

Programmatically capturing the stack trace

from within an appilcation, you can call : Thread.getAllStackTraces(). But if you want to do the same from a different application, ThreadMXBean exposes the needed data, which can be retrieved by using the JMX interface.

Dumping thread Stack trace in .NET

.NET does not seem to have the variety of options like Java. The only reasonable option I have seen so far is the Managed Stack Explorer. This can be run from the command line as

mse /s /p <pid>

or as a GUI application:

Since it is an open source (MSPL) project, I took a peek into its source to check the API used. It depends on some sample code released by the CLR Team called mdbg. This in turn is a managed layer on the unmanaged CLR Debugging Services API.

From what I see there seems to be no direct way to get a stack trace in .NET, apart from either using the debugger API, or rolling your own on top of the StackTrace class. Check this question on StackOverflow.

In this world of concurrent, multi-threaded programming, functional style of programming makes more sense. And one of the key tenets of functional programming is immutability. Even in OO languages, a few benefits of FP can be derived if the objects are made immutable.

Clone gone wrong

We follow a similar approach in our application. But this is not always possible, especially the mutable by default approach of Java. To overcome this, we pass around clones of instances, instead of the instance itself. This works in most cases, but we experienced a very subtle bug last week, where one thread was changing the values on a clone held by another thread, in spite of having no reference to it.

The cause

After lot of disbelief and head scratching, the reason was there, right in front. We were calling the instance.clone() to create a clone. The clone function defined in the Object class (more or less) takes each variable and makes a copy if it and assigns it to the variable of the new instance. This works great for stack variable, but the for the ones on the heap, only the reference is copied, in effect both the clones have reference to the same instance of the member variable. In other words, it does a shallow copy and you might actually need a deep copy sometimes.

Before Clone

After Clone

The effect

Thus any change made to MyClassInstance.OtherClass after MyClassClone was created also changes the MyClassClone.OtherClass, leading to confusing side effects.

The solution

Once the problem is understood, the solution is obvious. Just override the clone method and clone any reference classes in it.

Check http://stackoverflow.com/questions/2890340/question-about-cloning-in-java for more.

Microsoft LifeCam VX-1000 does not quite work on Ubuntu and even worse with Skype.
To get video working on Skype, start it as :
env LD_PRELOAD=/usr/lib/libv4l/v4l1compat.so skype

And to get the microphone working, run this in a terminal
sudo rmmod gspca_sonixj
sudo modprobe gspca_sonixj