P.S: this post does not absolve me from writing up something (with many thanks and links to people) about the useR2010 conference, but I can see it taking a bit longer till I do that.

—————–

Abstract of the talk

This talk is a basic introduction to blogs: why to blog, how to blog, and the importance of the R blogosphere to the R community.

Because R is an open-source project, the R community members rely (mostly) on each other’s help for statistical guidance, generating useful code, and general moral support.

Current online tools available for us to help each other include the R mailing lists, the community R-wiki, and the R blogosphere. The emerging R blogosphere is the only source, besides the R journal, that provides our community with articles about R. While these articles are not peer reviewed, they do come in higher volume (and often are of very high quality).

According to the meta-blog R-bloggers.com, the (English) R blogosphere has produced, in January 2010, about 115 “articles” about R. There are (currently) a bit over 50 bloggers (now about 100) who write about R, with about 1000 (now ~2200) subscribers who read them daily (through e-mails or RSS). These numbers allow me to believe that there is a genuine interest in our community for more people – perhaps you? – to start (and continue) blogging about R.

In this talk I intend to share knowledge about blogging so that more people are able to participate (freely) in the R blogosphere – both as readers and as writers. The talk will have three main parts:

• What is a blog
• How to blog – using the (free) blogging service WordPress.com (with specific emphasis on R)
• How to develop readership – integration with other social media/networks platforms, SEO, and other best practices

* * *
Tal Galili founded www.R-bloggers.com and blogs on www.R-statistics.com
* * *

Audio recording of the talk

Click here to download the audio file

Slides

Download (PDF, 5.09MB)

—————–

Last week I had the honor of attending the talk given by Richard Stallman, the last keynote speaker on the useR 2010 conference.  In this post I will give a brief context for the talk, and then give the audio files of the talk, with some description of what was said in the talk.

Context for the talk

Richard Stallman can be viewed as (one of) the fathers of free software (free as in speech, not as in beer).

He is the man who led the GNU project for the creation of a free (as in speech, not as in beer) operation systems on the basis of which GNU-Linux, with its numerous distributions, was created.
Richard also developed a number of pieces of widely used software, including the original Emacs,[4] the GNU Compiler Collection,[5], the GNU Debugger[6], and many tools in the GNU Coreutils

Richard also initiated the free software movement and in October 1985 he also founded it’s formal foundation and co-founded the League for Programming Freedom in 1989.

Stallman pioneered the concept of “copyleft” and he is the main author of several copyleft licenses including the GNU General Public License, the most widely used free software license.

You can read about him in the wiki article titles “Richard Stallman”

The useR 2010 conference is an annual 4 days conference of the community of people using R.  R is a free open source software for data analysis and statistical computing (Here is a bit more about what is R).

The conference this year was truly a wonderful experience for me.  I  had the pleasure of giving two talks (about which I will blog later this month), listened to numerous talks on the use of R, and had a chance to meet many (many) kind and interesting people.

Richard Stallmans talk

The talk took place on July 23rd 2010 at NIST U.S.  and was the concluding talk for the useR2010 conference.  The talk consisted of a two hour lecture followed by a half-hour question and answer session.

On a personal note, I was very impressed by Richards talk.  Richard is not a shy computer geek, but rather a serious leader and thinker trying to stir people to action.  His speech was a sermon on free software, the history of GNU-Linux, the various versions of GPL, and his own history involving them.

I believe this talk would be of interest to anyone who cares about social solidarity, free software, programming and the hope of a better world for all of us.

I am eager for your thoughts in the comments (but please keep a kind tone).

Here is Richard Stallmans  (2 hours) talk:

Audio file to download – Richard Stallman talk at the useR! 2010 conference (~2 hours)

The second part of the talk consisted of Richard Stallman answering the following questions:

• What are your thoughts about Data portability?
• What are your thoughts about FaceBook?
• Isn’t it a problem that free software doesn’t create wealth?
• What are your thoughts about innovation?
• What are your thoughts about Software as service (a.k.a: cloud computing)?
• How can we defend your open sourced software from “hackers“?
• What are your thoughts about googles products and services?
• What are your thoughts about the legality/ethically of people changing from GPL to closed-sourced?
• How can a programmer be “compensated” for his contribution for a free “open source” software?
• What are your thoughts about “free games“?
• What are your thoughts about search results?
• What are your thoughts about Taxes and government responsibility for the use of free software?

Audio file to download – Richard Stallman talk at the useR! 2010 conference – Q&A session (~25 minutes)

Final note, more talks from the useR2010 conference are expected to be put online here, thanks to Drew Conway.

(And also consider giving the contender, MetaOptimize a visit)

* * * *

Statistical analysis Q&A website is about to go into BETA

A month ago I invited readers of this blog to commit to using a new Q&A website for Data-Analysis (based on StackOverFlow engine), once it will open (the site was originally proposed by Rob Hyndman).
And now, a month later, I am happy to write that over 500 people have shown interest in the website, and choose to commit themselves. This means we we have reached 100% completion of the website proposal process, and in the next few days we will move to the next step.

The next step is that the website will go into closed BETA for about a week. If you want to be part of this – now is the time to join (<--- call for action people).
From being part in some other closed BETA of similar projects, I can attest that the enthusiasm of the people trying to answer questions in the BETA is very impressive, so I strongly recommend the experience.

If you won't make it by the time you see this post, then no worries - about a week or so after the website will go online, it will be open to the wide public.

(p.s: thanks Romunov for pointing out to me that the BETA is about to open)

p.s: MetaOptimize

I would like to finish this post with mentioning MetaOptimize. This is a Q&A website which is of a more “machine learning” then a “statistical” community. It also started out some short while ago, and already it has around 700 users who have submitted ~160 questions with ~520 answers given. From my experience on the site so far, I have enjoyed the high quality of the questions and answers.
When I first came by the website, I feared that supporting this website will split the R community of users between this website and the area 51 StackExchange website.
But after a lengthy discussion (published recently as a post) with MetaOptimize founder, Joseph Turian, I came to have a more optimistic view of the competition of the two websites. Where at first I was afraid, I am now hopeful that each of the two website will manage to draw a tiny bit of different communities of people (that would otherwise wouldn’t be present in the other website) – thus offering all of us a wider variety of knowledge to tap into.

See you there…

just released new versions of plyr and ggplot2. source versions available on cran, compiled will follow soon #rstats

Going to the CRAN website shows that plyr has gone through the most major update, with the last update (before the current one) taking place on 2009-06-23. And now, over a year later, we are presented with plyr version 1, which includes New functions, New features some Bug fixes and a much anticipated Speed improvements.
ggplot2, has made a tiny leap from version 0.8.7 to 0.8.8, and was previously last updated on 2010-03-03.

Me, and I am sure many R users are very thankful for the amazing work that Hadley Wickham is doing (both on his code, and with helping other useRs on the help lists). So Hadley, thank you!

Here is the complete change-log list for both packages:

plyr 1.0 (2010-07-02) —————————————————

(taken from the CRAN website)
New functions:

* arrange, a new helper method for reordering a data frame.
* count, a version of table that returns data frames immediately and that is
much much faster for high-dimensional data.
* desc makes it easy to sort any vector in descending order
* join, works like merge but can be much faster and has a somewhat simpler
syntax drawing from SQL terminology
* rbind.fill.matrix is like rbind.fill but works for matrices, code
contributed by C. Beleites

Speed improvements

* experimental immutable data frame (idata.frame) that vastly speeds up
subsetting – for large datasets with large numbers of groups, this can yield
10-fold speed ups. See examples in ?idata.frame to see how to use it.
* rbind.fill rewritten again to increase speed and work with more data types
* d*ply now much faster with nested groups

New features:

* d*ply now accepts NULL for splitting variables, indicating that the data
should not be split
* plyr no longer exports internal functions, many of which were causing
clashes with other packages
* rbind.fill now works with data frame columns that are lists or matrices
* test suite ensures that plyr behaviour is correct and will remain correct
as I make future improvements.

Bug fixes:

* **ply: if zero splits, empty list(), data.frame() or logical() returned,
as appropriate for the output type
* **ply: leaving .fun as NULL now always returns list
(thanks to Stavros Macrakis for the bug report)
* a*ply: labels now respect options(stringAsFactors)
* each: scoping bug fixed, thanks to Yasuhisa Yoshida for the bug report
* list_to_dataframe is more consistent when processing a single data frame
* NAs preserved in more places
* progress bars: guaranteed to terminate even if **ply prematurely terminates
* progress bars: misspelling gives informative warning, instead of
uninformative error
* splitter_d: fixed ordering bug when .drop = FALSE

ggplot2 0.8.8 (2010-07-02) —————————————-

(taken from the CRAN website)

Bug fixes:

* coord_equal finally works as expected (thanks to continued prompting from Jean-Olivier Irisson)
* coord_equal renamed to coord_fixed to better represent capabilities
* coord_polar and coord_polar: new munching system that uses distances (as defined by the coordinate system) to figure out how many pieces each segment should be broken in to (thanks to prompting from Jean-Olivier Irisson)
* fix ordering bug in facet_wrap (thanks to bug report by Frank Davenport)
* geom_errorh correctly responds to height parameter outside of aes
* geom_hline and geom_vline will not impact legend when used for fixed intercepts
* geom_hline/geom_vline: intercept values not set quite correctly which caused a problem in conjunction with transformed scales (reported by Seth Finnegan)
* geom_line: can now stack lines again with position = “stack” (fixes #74)
* geom_segment: arrows now preserved in non-Cartesian coordinate system (fixes #117)
* geom_smooth now deals with missing values in the same way as geom_line (thanks to patch from Karsten Loesing)
* guides: check all axis labels for expressions (reported by Benji Oswald)
* guides: extra 0.5 line margin around legend (fixes #71)
* guides: non-left legend positions now work once more (thanks to patch from Karsten Loesing)
* label_bquote works with more expressions (factors now cast to characters, thanks to Baptiste Auguie for bug report)
* scale_color: add missing US spellings
* stat: panels with no non-missing values trigged errors with some statistics. (reported by Giovanni Dall’Olio)
* stat: statistics now also respect layer parameter inherit.aes (thanks to bug report by Lorenzo Isella and investigation by Brian Diggs)
* stat_bin no longer drops 0-count bins by default
* stat_bin: fix small bug when dealing with single bin with NA position (reported by John Rauser)
* stat_binhex: uses range of data from scales when computing binwidth so hexes are the same size in all facets (thanks to Nicholas Lewin-Koh for the bug report)
* stat_qq has new dparam parameter for specifying distribution parameters (thanks to Yunfeng Zhang for the bug report)
* stat_smooth now uses built-in confidence interval (with small sample correction) for linear models (thanks to suggestion by Ian Fellows)
* sta

Imagine you want to give a presentation or report of your latest findings running some sort of regression analysis. How would you do it?

This was exactly the question Wincent Rong-gui HUANG has recently asked on the R mailing list.

One person, Bernd Weiss, responded by linking to the chapter “Plotting Regression Coefficients” on an interesting online book (I have never heard of before) called “Using Graphs Instead of Tables” (I should add this link to the free statistics e-books list…)

Letter in the conversation, Achim Zeileis, has surprised us (well, me) saying the following

I’ve thought about adding a plot() method for the coeftest() function in the “lmtest” package. Essentially, it relies on a coef() and a vcov() method being available – and that a central limit theorem holds. For releasing it as a general function in the package the code is still too raw, but maybe it’s useful for someone on the list. Hence, I’ve included it below.

(I allowed myself to add some bolds in the text)

So for the convenience of all of us, I uploaded Achim’s code in a file for easy access. Here is an example of how to use it:

source("http://www.r-statistics.com/wp-content/uploads/2010/07/coefplot.r.txt")
 
data("Mroz", package = "car")
fm <- glm(lfp ~ ., data = Mroz, family = binomial)
coefplot(fm, parm = -1)

Here is the resulting graph:

I hope Achim will get around to improve the function so he might think it worthy of joining his“lmtest” package. I am glad he shared his code for the rest of us to have something to work with in the meantime

* * *

Update (07.07.10):
Thanks to a comment by David Atkins, I found out there is a more mature version of this function (called coefplot) inside the {arm} package. This version offers many features, one of which is the ability to easily stack several confidence intervals one on top of the other.

It works for baysglm, glm, lm, polr objects and a default method is available which takes pre-computed coefficients and associated standard errors from any suitable model.

Example:
(Notice that the Poisson model in comparison with the binomial models does not make much sense, but is enough to illustrate the use of the function)

library("arm")
data("Mroz", package = "car")
M1<-      glm(lfp ~ ., data = Mroz, family = binomial)
M2<- bayesglm(lfp ~ ., data = Mroz, family = binomial)
M3<-      glm(lfp ~ ., data = Mroz, family = binomial(probit))
coefplot(M2, xlim=c(-2, 6),            intercept=TRUE)
coefplot(M1, add=TRUE, col.pts="red",  intercept=TRUE)
coefplot(M3, add=TRUE, col.pts="blue", intercept=TRUE, offset=0.2)

(hat tip goes to Allan Engelhardt for help improving the code, and for Achim Zeileis in extending and improving the narration for the example)

Resulting plot

* * *
Lastly, another method worth mentioning is the Nomogram, implemented by Frank Harrell’a rms package.

About prize baring contests

Competition with prizes are an amazing thing. If you are not sure of that, I urge you to listened to Peter Diamandis talk about his experience with the X prize (start listening at minute 11:40):

At short – prizes can give up to 1 to 50 ratio of return on investment of the people giving funding to the prize. The money is spent only when results are achieved. And there is a lot of value in terms of public opinion and publicity. And the best of all (for the promoter of the competition) – prizes encourage people to take risks (at their own expense) in order to get results done.

All of that said, I look at prize baring competition as something worth spreading, especially in cases where the results of the winning team will be shared with the public.

About the IEEE ICDM Contest

The IEEE ICDM Contest (“Road Traffic Prediction for Intelligent GPS Navigation”), seems to be one of those cases. Due to a polite request, I am republishing here the details of this new competition, in the hope that some of my R colleagues will bring the community some pride

Introduction

Data mining competition affiliated with IEEE International Conference on Data Mining 2010 (ICDM), Sydney, Australia, Dec 14-17. The task is to predict city traffic based on simulated historical measurements or real-time stream of notifications sent by individual drivers from their GPS navigators. Prizes worth $5,000 will be awarded to the winners.

Road Traffic Prediction for Intelligent GPS Navigation

Over the last century, number of cars engaged in vehicular traffic in cities has increased rapidly, causing many difficulties for all citizens: traffic jams, large and unpredictable communication delays, pollution etc. Excessive traffic became a civilization problem that affects everyone who lives in a city of 50,000 or larger, anywhere in the world. Complexity of processes that stand behind traffic flow is so large, that only data mining algorithms – from the domains of structure mining, graph mining, data streams, large-scale and temporal data mining – may bring efficient solutions for these problems. With the proposed competition, we want to ask researchers to devise the best possible algorithms that tackle problems of traffic flow prediction, for the purpose of intelligent driver navigation and improved city planning.
There are Tyree independent tasks:

• Traffic (link). Traffic congestion prediction, in an elementary setup of time series forecasting: a series of measurements from 10 selected road segments is given and the goal is to make short-term predictions of future values based on historical ones. This task is intended as an introductory one, simpler than the other two.
• Jams (link). Modeling the process of traffic jams formation during morning peak in the presence of roadworks, based on initial information about jams broadcast by radio stations. Input data contain identifiers of road segments closed due to roadworks, accompanied by a sequence of segments where the first jams occurred. The algorithm should predict a sequence of segments where next jams will occur in the nearest future.
• GPS (link). Traffic reconstruction and prediction based on real-time information from individual drivers. Input data consist of a stream of notifications from 1% of vehicles about their current GPS locations in the city road network, sent every 10 seconds. The algorithm receives this stream and predicts traffic congestion on selected road segments for the next 30 minutes. Large volumes of data are involved in this task, requiring the use of scalable data mining methods.

Everyone is welcome to participate. Competition starts now and will last till September 6th, 2010. More details on: http://tunedit.org/challenge/IEEE-ICDM-2010

What is the StackOverFlow Q&A website about?

StackOverFlow.com (“SO” for short) is a programming Q & A site that’s free. Free to ask questions, free to answer questions, free to read. Free, And fast.

For the R community, SO offers a growing database of R related questions and answer (click the link to check them out).

You might be asking yourself what’s so special about SO over other available resources such as R mailing lists, R blogs, R wiki and so on?
That is a great question.

The answer is that SO succeeds in doing a great job synthesizing aspects of Wikis, Blogs, Forums, and Digg/Reddit to offer a very powerful Q&A website.

In SO, the new questions are like forum/blog posts (A main text with comments/answers). After someone answers a question, other users can give a thumb-up or a thumb-down to the answer (like digg/reddit). And all content can be edited, like a wiki page, by the users (provided the user has enough “karma points”).
You also get badges (“awards”) for a bunch of actions (like coming to the website every day for a month. Giving an answer that got X amount of thumb-ups and so on). The awards allows someone who is asking a question to see how much the person who had answered him has good reputation (in terms of acceptance/appreciation of his answers by other SO members).
It also offers a small (but effective) ego-boost for the person who gives answers.

So if StackOverFlow is so great – what is this new website you wrote about in the title?

Well, StackOverFlow has one limitation. It deals ONLY with programming questions. Other questions like:

• Which of the following three graphics best displays this data set? Why?
• Can you give an example of where I might prefer to use a z-test vs a t-test?
• What is the relationship between Bayesian and neural networks?

Will not be answered, and the threads will get closed as being “off topic”. Why? because such questions are dealing with: statistics, data analysis, data mining, data visualization – But in no means in programming.

So there is no StackOverFlow-like Q&A website for data analysis… Until now!

In the past few weeks, Rob Hyndman and other users, have made much effort to push the creation of a new website, based on the StackOverFlow engine, to allow for statistically related Q&A.
His proposal for a new website is almost complete. All it need is for you (yes you), to go to the following link:
Stack Exchange Q&A site proposal: Statistical Analysis
And commit yourself to the website (that is, click the button called “commit” – so to declare that you will have interest in reading, asking and answering questions on such a website)

Once a few more tens 379 more people will commit – the website will go online!

Hope to see you there.

In 2002, Matthias Schonlau published in “The Stata Journal” an article named “The Clustergram: A graph for visualizing hierarchical and . As explained in the abstract:

In hierarchical cluster analysis dendrogram graphs are used to visualize how clusters are formed. I propose an alternative graph named “clustergram” to examine how cluster members are assigned to clusters as the number of clusters increases.
This graph is useful in exploratory analysis for non-hierarchical clustering algorithms like k-means and for hierarchical cluster algorithms when the number of observations is large enough to make dendrograms impractical.

A similar article was later written and was (maybe) published in “computational statistics”.

Both articles gives some nice background to known methods like k-means and methods for hierarchical clustering, and then goes on to present examples of using these methods (with the Clustergarm) to analyse some datasets.

Personally, I understand the clustergram to be a type of parallel coordinates plot where each observation is given a vector. The vector contains the observation’s location according to how many clusters the dataset was split into. The scale of the vector is the scale of the first principal component of the data.

Clustergram in R (a basic function)

After finding out about this method of visualization, I was hunted by the curiosity to play with it a bit. Therefore, and since I didn’t find any implementation of the graph in R, I went about writing the code to implement it.

The code only works for kmeans, but it shows how such a plot can be produced, and could be later modified so to offer methods that will connect with different clustering algorithms.

The function I present here gets a data.frame/matrix with a row for each observation, and the variable dimensions present in the columns.
The function assumes the data is scaled.
The function then goes about calculating the cluster centers for our data, for varying number of clusters.
For each cluster iteration, the cluster centers are multiplied by the first loading of the principal components of the original data. Thus offering a weighted mean of the each cluster center dimensions that might give a decent representation of that cluster (this method has the known limitations of using the first component of a PCA for dimensionality reduction, but I won’t go into that in this post).
Finally all of our data points are ordered according to their respective cluster first component, and plotted against the number of clusters (thus creating the clustergram).

My thank goes to Hadley Wickham for offering some good tips on how to prepare the graph.

Here is the code (example follows)

 
 
clustergram.kmeans <- function(Data, k, ...)
{
	# this is the type of function that the clustergram
	# 	function takes for the clustering.
	# 	using similar structure will allow implementation of different clustering algorithms
 
	#	It returns a list with two elements:
	#	cluster = a vector of length of n (the number of subjects/items)
	#				indicating to which cluster each item belongs.
	#	centers = a k dimensional vector.  Each element is 1 number that represent that cluster
	#				In our case, we are using the weighted mean of the cluster dimensions by 
	#				Using the first component (loading) of the PCA of the Data.
 
	cl <- kmeans(Data, k,...)
 
	cluster <- cl$cluster
	centers <- cl$centers %*% princomp(Data)$loadings[,1]	# 1 number per center
												# here we are using the weighted mean for each
 
	return(list(
				cluster = cluster,
				centers = centers
			))
}		
 
clustergram.plot.matlines <- function(X,Y, k.range, 
											x.range, y.range , COL, 
											add.center.points , centers.points)
	{
		plot(0,0, col = "white", xlim = x.range, ylim = y.range,
			axes = F,
			xlab = "Number of clusters (k)", ylab = "PCA weighted Mean of the clusters", main = "Clustergram of the PCA-weighted Mean of the clusters k-mean clusters vs number of clusters (k)")
		axis(side =1, at = k.range)
		axis(side =2)
		abline(v = k.range, col = "grey")
 
		matlines(t(X), t(Y), pch = 19, col = COL, lty = 1, lwd = 1.5)
 
		if(add.center.points)
		{
			require(plyr)
 
			xx <- ldply(centers.points, rbind)
			points(xx$y~xx$x, pch = 19, col = "red", cex = 1.3)
 
			# add points	
			# temp <- l_ply(centers.points, function(xx) {
									# with(xx,points(y~x, pch = 19, col = "red", cex = 1.3))
									# points(xx$y~xx$x, pch = 19, col = "red", cex = 1.3)
									# return(1)
									# })
						# We assign the lapply to a variable (temp) only to suppress the lapply "NULL" output
		}	
	}
 
 
 
clustergram <- function(Data, k.range = 2:10 , 
							clustering.function = clustergram.kmeans,
							clustergram.plot = clustergram.plot.matlines, 
							line.width = .004, add.center.points = T)
{
	# Data - should be a scales matrix.  Where each column belongs to a different dimension of the observations
	# k.range - is a vector with the number of clusters to plot the clustergram for
	# clustering.function - this is not really used, but offers a bases to later extend the function to other algorithms 
	#			Although that would  more work on the code
	# line.width - is the amount to lift each line in the plot so they won't superimpose eachother
	# add.center.points - just assures that we want to plot points of the cluster means
 
	n <- dim(Data)[1]
 
	PCA.1 <- Data %*% princomp(Data)$loadings[,1]	# first principal component of our data
 
	if(require(colorspace)) {
			COL <- heat_hcl(n)[order(PCA.1)]	# line colors
		} else {
			COL <- rainbow(n)[order(PCA.1)]	# line colors
			warning('Please consider installing the package "colorspace" for prittier colors')
		}
 
	line.width <- rep(line.width, n)
 
	Y <- NULL	# Y matrix
	X <- NULL	# X matrix
 
	centers.points <- list()
 
	for(k in k.range)
	{
		k.clusters <- clustering.function(Data, k)
 
		clusters.vec <- k.clusters$cluster
			# the.centers <- apply(cl$centers,1, mean)
		the.centers <- k.clusters$centers 
 
		noise <- unlist(tapply(line.width, clusters.vec, cumsum))[order(seq_along(clusters.vec)[order(clusters.vec)])]	
		# noise <- noise - mean(range(noise))
		y <- the.centers[clusters.vec] + noise
		Y <- cbind(Y, y)
		x <- rep(k, length(y))
		X <- cbind(X, x)
 
		centers.points[[k]] <- data.frame(y = the.centers , x = rep(k , k))	
	#	points(the.centers ~ rep(k , k), pch = 19, col = "red", cex = 1.5)
	}
 
 
	x.range <- range(k.range)
	y.range <- range(PCA.1)
 
	clustergram.plot(X,Y, k.range, 
											x.range, y.range , COL, 
											add.center.points , centers.points)
 
 
}

Example on the iris dataset

The iris data set is a favorite example of many R bloggers when writing about R accessors , Data Exporting, Data importing, and for different visualization techniques.
So it seemed only natural to experiment on it here.

data(iris)
set.seed(250)
par(cex.lab = 1.5, cex.main = 1.2)
Data <- scale(iris[,-5]) # notice I am scaling the vectors)
clustergram(Data, k.range = 2:8, line.width = 0.004) # notice how I am using line.width.  Play with it on your problem, according to the scale of Y.

Here is the output:

Looking at the image we can notice a few interesting things. We notice that one of the clusters formed (the lower one) stays as is no matter how many clusters we are allowing (except for one observation that goes way and then beck).
We can also see that the second split is a solid one (in the sense that it splits the first cluster into two clusters which are not “close” to each other, and that about half the observations goes to each of the new clusters).
And then notice how moving to 5 clusters makes almost no difference.
Lastly, notice how when going for 8 clusters, we are practically left with 4 clusters (remember – this is according the mean of cluster centers by the loading of the first component of the PCA on the data)

If I where to take something from this graph, I would say I have a strong tendency to use 3-4 clusters on this data.

But wait, did our clustering algorithm do a stable job?
Let’s try running the algorithm 6 more times (each run will have a different starting point for the clusters)

set.seed(500)
Data <- scale(iris[,-5]) # notice I am scaling the vectors)
par(cex.lab = 1.2, cex.main = .7)
par(mfrow = c(3,2))
for(i in 1:6) clustergram(Data, k.range = 2:8 , line.width = .004, add.center.points = T)

Resulting with: (press the image to enlarge it)

Repeating the analysis offers even more insights.
First, it would appear that until 3 clusters, the algorithm gives rather stable results.
From 4 onwards we get various outcomes at each iteration.
At some of the cases, we got 3 clusters when we asked for 4 or even 5 clusters.

Reviewing the new plots, I would prefer to go with the 3 clusters option. Noting how the two “upper” clusters might have similar properties while the lower cluster is quite distinct from the other two.

By the way, the Iris data set is composed of three types of flowers. I imagine the kmeans had done a decent job in distinguishing the three.

Limitation of the method (and a possible way to overcome it?!)

It is worth noting that the current way the algorithm is built has a fundamental limitation: The plot is good for detecting a situation where there are several clusters but each of them is clearly “bigger” then the one before it (on the first principal component of the data).

For example, let’s create a dataset with 3 clusters, each one is taken from a normal distribution with a higher mean:

set.seed(250)
Data <- rbind(
				cbind(rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3)),
				cbind(rnorm(100,1, sd = 0.3),rnorm(100,1, sd = 0.3),rnorm(100,1, sd = 0.3)),
				cbind(rnorm(100,2, sd = 0.3),rnorm(100,2, sd = 0.3),rnorm(100,2, sd = 0.3))
				)				
clustergram(Data, k.range = 2:5 , line.width = .004, add.center.points = T)

The resulting plot for this is the following:

The image shows a clear distinction between three ranks of clusters. There is no doubt (for me) from looking at this image, that three clusters would be the correct number of clusters.

But what if the clusters where different but didn’t have an ordering to them?
For example, look at the following 4 dimensional data:

set.seed(250)
Data <- rbind(
				cbind(rnorm(100,1, sd = 0.3),rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3)),
				cbind(rnorm(100,0, sd = 0.3),rnorm(100,1, sd = 0.3),rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3)),
				cbind(rnorm(100,0, sd = 0.3),rnorm(100,1, sd = 0.3),rnorm(100,1, sd = 0.3),rnorm(100,0, sd = 0.3)),
				cbind(rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3),rnorm(100,0, sd = 0.3),rnorm(100,1, sd = 0.3))
				)				
clustergram(Data, k.range = 2:8 , line.width = .004, add.center.points = T)

In this situation, it is not clear from the location of the clusters on the Y axis that we are dealing with 4 clusters.
But what is interesting, is that through the growing number of clusters, we can notice that there are 4 “strands” of data points moving more or less together (until we reached 4 clusters, at which point the clusters started breaking up).
Another hope for handling this might be using the color of the lines in some way, but I haven’t yet figured out how.

Clustergram with ggplot2

Hadley Wickham has kindly played with recreating the clustergram using the ggplot2 engine. You can see the result here:
http://gist.github.com/439761
And this is what he wrote about it in the comments:

I’ve broken it down into three components:
* run the clustering algorithm and get predictions (many_kmeans and all_hclust)
* produce the data for the clustergram (clustergram)
* plot it (plot.clustergram)
I don’t think I have the logic behind the y-position adjustment quite right though.

Here is an example of how it looks:

Conclusions (some rules of thumb and questions for the future)

In a first look, it would appear that the clustergram can be of use. I can imagine using this graph to quickly run various clustering algorithms and then compare them to each other and review their stability (In the way I just demonstrated in the example above).

The three rules of thumb I have noticed by now are:

1. Look at the location of the cluster points on the Y axis. See when they remain stable, when they start flying around, and what happens to them in higher number of clusters (do they re-group together)
2. Observe the strands of the datapoints. Even if the clusters centers are not ordered, the lines for each item might (needs more research and thinking) tend to move together – hinting at the real number of clusters
3. Run the plot multiple times to observe the stability of the cluster formation (and location)

Yet there is more work to be done and questions to seek answers to:

• The code needs to be extended to offer methods to various clustering algorithms.
• How can the colors of the lines be used better?
• How can this be done using other graphical engines (ggplot2/lattice?) – (Update: look at Hadley’s reply in the comments)
• What to do in case the first principal component doesn’t capture enough of the data? (maybe plot this graph to all the relevant components. but then – how do you make conclusions of it?)
• What other uses/conclusions can be made based on this graph?

I am looking forward to reading your input/ideas in the comments (or in reply posts).

This was published on the R-help mailing list today:

————-

The final registration deadline for the R User Conference is June 20,
2010, one week away.  Later registration will not be possible on site!

Conference webpage:  http://www.R-project.org/useR-2010
Conference program: http://www.R-project.org/useR-2010/program.html

Registration:
http://www.R-project.org/useR-2010/registration/registration.html

The conference is scheduled for July 21-23, 2010, and will take place at
the campus of the National Institute of Standards and Technology (NIST) in
Gaithersburg, Maryland, USA.

Following the successful useR! 2004, useR! 2006, useR! 2007, useR! 2008,
and useR! 2009, conferences, the conference is focused on:

1. R as the `lingua franca’ of data analysis and statistical computing,
2. providing a platform for R users to discuss and exchange ideas on
how R can be used to do statistical computations, data analysis,
visualization and exciting applications in various fields,
3. giving an overview of the new features of the rapidly evolving R
project.

As for the predecessor conferences, the program will consist of two parts:
invited lectures and user-contributed sessions.  Prior to the conference,
there will be tutorials on R, descriptions of which are available at
http://www.R-project.org/useR-2010/tutorials

INVITED LECTURES

Invited speakers will include

Mark Handcock, Frank Harrell Jr, Friedrich Leisch, Michael Meyer,
Richard Stallman, Luke Tierney, Diethelm Wuertz.

USER-CONTRIBUTED SESSIONS

The sessions will be a platform to bring together R users, contributors,
package maintainers and developers in the S spirit that `users are
developers’. People from different fields will show us how they solve
problems with R in fascinating applications.  The sessions are organized
by members of the program committee, including

Dirk Eddelbuettel, John Fox, Virgilio Gomez-Rubio,
Richard Heiberger, Torsten Hothorn, Aaron King, Jan de Leeuw,
Nicholas Lewin-Koh, Andy Liaw, Uwe Ligges, Martin Maechler,
Katharine Mullen, Heather Turner, Ravi Varadhan, H. D. Vinod,
John Verzani, Alan Zaslavsky, Achim Zeileis.

The program will cover topics such as

* Applied Statistics & Biostatistics
* Bayesian Statistics
* Bioinformatics
* Chemometrics and Computational Physics
* Data Mining
* Econometrics & Finance
* Environmetrics & Ecological Modeling
* High Performance Computing
* Machine Learning
* Marketing & Business Analytics
* Psychometrics
* Robust Statistics
* Social network analysis
* Spatial Statistics
* Statistics in the Social and Political Sciences
* Teaching
* Visualization & Graphics
* and many more.

IMPORTANT DATES

*********************************************************************
**   2010-06-20 registration deadline
**                (later registration NOT possible on site)
*********************************************************************
2010-07-20   tutorials
2010-07-21   conference start
2010-07-23   conference end

We hope to meet you in Gaithersburg!

Imprtent update to the post (17.07.10)

I now came across David smith’s post on the REvolution blog, pointing to instruction on the R wiki for how to install R on the iPhone!
I didn’t try it myself since it both requires jailbreaking the iPhone, and I don’t have an iPhone. But it is still interesting to know of.

Preface – I don’t use Mac

I don’t use Mac! Not that there is anything wrong with that, but I don’t use Mac…

Yet at the same time, wonderful people like my wife, my brother, my thesis advisor and even my mother-in-law – all use mac. So one can’t help but wonder if I might be missing out on something.

Still, for a Windows user like me it is a bit difficult to understand the hype around the iPhone 4 release:

Such releases tend to look to me more like this spoof video about the release of the apple “i”.

So while not using apples product, I have a deep respect for the impact it has made in peoples lives. Which begs the question: Could you use R on an iPhone (or an iPad) ??

Can R be run on iPhone/iPad ?

This question (and the motivation for this post) was raised in an R help mailing list thread a week ago.

After receiving permission from the threads author, I am republishing the content that was presented there in the hopes it might be of interest to other R community members.

And here is what “Marc Schwartz” wrote:

Marc Schwartz on R and iPhone/iPad

Hi all,

There have been posts in the past about R being made available for the iPhone and perhaps more logically now, on the iPad. My recollection is that the hurdle discussed in the past was primarily a lack of access to a CLI on the iPhone’s variant of OSX, compelling the development of a GUI interface for R specifically for these devices. R itself, can be successfully compiled with the iPhone development tools.

Well, now there is another, clearly more profound reason.

The FSF has recently communicated with Apple on the presence of a GPL application (GNU Go) in the iTunes store because the iTunes TOS infringes upon the GPL. Apple, given a choice, elected to remove the application, rather than amending their TOS.

The FSF also informed the developers of the iPhone port of GNU Go that their distribution is in violation of the GPL. R Core and any others considering an iPhone/iPad port of R, if you are not already aware, take note…

More information is here:

http://www.fsf.org/news/2010-05-app-store-compliance/

with an update here:

http://www.fsf.org/news/blogs/licensing/more-about-the-app-store-gpl-enforcement

So, until Apple amends their TOS agreement, it looks like there will be no GPL apps available for the iPhone/iPad, since the only way to make applications available for these platforms is via the iTunes store (unless you unlock the device). Hence, no R for these devices in the foreseeable future.

Marc Schwartz second massage

Hi all,

Thanks to an offlist e-mail from Thomas (Lumley), I have spent the past few days getting a bit more edumacated on additional restrictions on the opportunity for R to appear on the iPhone/iPad. These in fact go well beyond the GPL issue that I raised in my initial post, which in and of itself, is not trivial by any means. I now know, or at least think I know, more about these issues than I probably wanted, but I also want to present a better picture of the situation.

Note that I am not a lawyer and am not intending to represent my findings from a legal perspective. I am reporting them here using a common sense approach from my own reading of the relevant Apple iPhone SDK language, as well as being based upon specific examples and discussions that I located on the web.

So, in summary, here are the key issues. I will follow each with some additional details below, but note that all such restrictions would need to be removed or otherwise overcome, before R could in fact appear on these two platforms, at least through Apple approved means in the App Store.

1. Distribution of GPL covered applications is not permissible via the App Store due to the Apple Terms of Service language, which infringes upon rights granted under the GPL.

‘Nuff said.

2. The use of FORTRAN is precluded (explicitly from SDK version 4.x forward)

Version 3.x of the iPhone SDK has the following language:

3.3.1 Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs.

Apple of course does not offer a FORTRAN compiler, as those who build R from source on OSX, as I do, are keenly aware. Thanks to Simon, we have one to use for OSX, but one is not officially available for the iPhone/iPad in the SDK.

Note that there is an important distinction here. The ability to build R versus the ability to have the resultant application pass Apple’s review to be able to appear in the App Store.

The above language has also been interpreted to apply to Java/Flash, precluding those environments from appearing on the iPhone/iPad.

However, the beta release of the 4.x version of the iPhone SDK has the following language in the same section:

3.3.1 — Applications may only use Documented APIs in the manner prescribed by Apple and must not use or call any private APIs. Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine, and only code written in C, C++, and Objective-C may compile and directly link against the Documented APIs (e.g., Applications that link to Documented APIs through an intermediary translation or compatibility layer or tool are prohibited).

Thus, in fact, one can only use Objective-C, C, C++ or JavaScript to develop applications for the iPhone/iPad. No FORTRAN, upon which of course, R is dependent. The above language has also been interpreted to further reinforce restrictions specifically on Java/Flash.

3. The implementation of programming language interpreters, of which R is one, is precluded.

The following language appears in version 3.x of the iPhone SDK:

3.3.2 An Application may not itself install or launch other executable code by any means, including without limitation through the use of a plug-in architecture, calling other frameworks, other APIs or otherwise. No interpreted code may be downloaded or used in an Application except for code that is interpreted and run by Apple’s Documented APIs and built-in interpreter(s).

The above language has been (pardon the pun) interpreted to restrict the implementation of programming language interpreters on these devices. Some sites I found have narrowly focused on the “No interpreted code may be downloaded” part of the language as an “out” of sorts. However, upon review, they seem to ignore the “or used” wording, which would seem to be independent of the action of downloading. If the language was “and used”, then one could envision the use of locally stored or entered code, but this is not the case. In either case, of course, R would not be one of Apple’s “built-in” interpreters.

I found two interesting examples of how Apple has either approved or rejected applications that can be considered interpreters. It is not clear where Apple drew the line between these two, such that it might enable one to better differentiate the reasoning and therefore design an app that would pass muster with them. Thus, one complication may be Apple’s lack of internal consistency in making decisions on allowing or disallowing apps in the App Store.

The first is an application which was intended to be a BASIC interpreter on the iPhone and which was apparently rejected by Apple under the name BasicMatrix:

http://smartcalc.coollittlethings.com/?p=3

After the author substantially restricted the functionality of the application to being an “enhanced calculator” under the name “SmartCalc”, Apple approved the application.

However, a contrarian example is “Frotz” which is a game type of an application currently available at no cost in the App Store for both the iPhone and the iPad. It is in fact an interpreter of so-called “Z code” (http://en.wikipedia.org/wiki/Z-machine) to create interactive fiction adventures. The web page for the app is:

http://code.google.com/p/iphonefrotz/wiki/FrotzMain

The app was initially accepted by Apple. A later version however was rejected, because the updated version could download new game files (Z code files) via the internet. Once the author removed the download functionality, Apple accepted the updated version of the app.

Frotz is also a GPL’d application, so its longevity in the App Store is logically in question and I sent an e-mail to the author to be sure that he was aware of the FSF’s recent actions.

The additional implications for R here, vis-a-vis Frotz, would be the ability to download, install and use CRAN packages. I will touch on this issue again below.

4. The implementation of anything resembling the CRAN network, to facilitate add-on packages for R, would be highly problematic for multiple reasons.

The following language appears in version 3.x of the iPhone SDK

3.3.3 Without Apple’s prior written approval, an Application may not provide, unlock or enable additional features or functionality through distribution mechanisms other than the App Store.

For those who have an iPhone/iPad and are familiar with so-called “In App” purchases, this refers to the mechanism approved by Apple to provide add-on functionality to already installed applications. The notion of In App purchases is somewhat misleading, as in fact the activity may involve no actual direct monetary cost to download the additional component(s).

Thus, arguably between the SDK language, which would preclude a CRAN type network unless approved by Apple, combined with the relevant language I reference from 3.3.2 above, there would be substantive restrictions on the ability of a “default” R installation from being able to download and utilize add-on R packages.

Further, since one cannot compile programs on the iPhone/iPad, there would have to be some means to pre-compile any add-on packages for R on these two platforms, similar to what is done for Windows and OSX presently on CRAN to create package binaries. Further, packages that use FORTRAN, tcl/tk, Java, Perl or other such libraries would of course, also be problematic, both from a functional and where appropriate, a GPL perspective.

Even if one got by some of those issues and pre-compiled add-on packages were to be made available via some distribution process, the entire process of R package installation, updating and management would have to be re-written specifically for these platforms. Thus, there would be a meaningful amount of development work that some group of folks would have to undertake.

That all being said, it is clear that a remote client/server implementation of R would be possible, as long as the client-side R GUI application on the iPhone/iPad meets Apple’s requirements. Anyone using the WolframAlpha app ($1.99 U.S.) on the iPhone or iPad (I do on the former) will understand this model. See http://products.wolframalpha.com/iphone/ and http://products.wolframalpha.com/ipad/index.html for more information. Somebody just has to be willing to fund the backend server farm to provide the service.

Needless to say, a fully browser based client/server implementation would also work and be cross-platform, provided that the input/output web pages are appropriately scaled for the mobile displays. WA also provides a free alternative to their dedicated apps via a mobile site at http://m.wolframalpha.com/ as an example. Similar considerations here, as above, for the backend server platform would be applicable. The key advantage of the dedicated app is a more efficient keyboard layout providing easier access to special character sets, rather than scrolling through them using the default keyboard.

I hope that the above is helpful to folks. Needless to say, I do not present the above as being the definitive reference, but it seems to be at least a logical interpretation of the current situation.

Marc Schwartz’s reply to Ken Williams reply to Marc’s original post

Ken,

See comments inline.

On Jun 1, 2010, at 2:25 PM, Ken Williams wrote:

> Hi Marc,
>
> I want to debate a couple points from your post:
>
>> 1. Distribution of GPL covered applications is not permissible via the App
>
>> Store due to the Apple Terms of Service language, which infringes upon
> rights
>> granted under the GPL.
>>
>> ‘Nuff said.
>
> I’m not sure I agree with this, but there’s so much wiggle room in
> interpreting what the GPL means that there would probably be no way to
> decide without a courtroom & judge, so I’ll leave this part alone. =) I
> also haven’t yet read your other post where you discuss this.

Please do, including the links therein. It’s not my interpretation, it is the FSF’s action and Apple’s response to that action, which sets at least an operational precedence, if not one that could also affect any future litigation pertaining to GPL’d apps in the App Store. That all just took place within the past week or so, which is what prompted my initial post on the matter, since it would be relevant to any R offering via that channel.

>> 3.3.1 Applications may only use Documented APIs in the manner
> prescribed by
>> Apple and must not use or call any private APIs.
>
> I believe that language only refers to *Apple’s* APIs. In other words, they
> don’t want you to call hidden functions that aren’t supposed to be exposed
> to developers. If it meant no use of any APIs private to the developer, it
> would rule out pretty much every application in existence, considering that
> a call from one function to any other is an API call.

I am not in disagreement on that point. The key issue to date has been the lack of a compatible FORTRAN compiler, at least off the shelf, based upon what I can tell. Arguably, that is at least a notable deterrent to use FORTRAN on the iPhone for now.

There are other programming language tools for current iPhone development, but nothing for FORTRAN that I can find.

I can find no references to anyone building iPhone apps using FORTRAN (even in part) and the few queries that I can find that even mention an interest in doing so, reference the same issues that I have. Some have referenced f2c, but it is not clear to me that such an approach would work for R, not to mention the development overhead and the extensive testing of any conversion of critical functionality.

That all changes with the new language in 4.x.
- Show quoted text -
The key wording change relevant to R (and of course for other iPhone developers and tool providers) in 4.x is:

“Applications must be originally written in Objective-C, C, C++, or JavaScript as executed by the iPhone OS WebKit engine”

Ignore the other wording pertaining to API’s and other layers for the time being.

The app must be written natively in one of those four languages. There appears to be no interpretation that I can find that differentiates a scenario where a library of low level functions, written in a language such as FORTRAN, may be called from a higher level language such as those listed above.

Unless there is some subtlety in differentiating the abstraction layers within which the application is executed on the iPhone, I see no recourse here.

Note that the entities that provide iPhone cross-compilation/framework tools (eg. MonoTouch, Titanium, unity3D, Rhodes, etc.) which convert other code directly to native iPhone apps are also trying to figure out where they stand. Similarly, folks who develop natively in other languages are also having headaches over the new SDK wording.

There is even a question relevant to cross-compilation tools that take another language and convert it to, for example, Obj-C, as an intermediate step, before subsequent compilation to an iPhone native binary.

So the message that everyone is coming away with is, if you want to develop for the iPhone, write your code using one of these four languages, period. No doubt, some folks will test the boundaries and we will get more definitive answers in time.

Is it possible that the SDK language will change before 4.x is released as a stable OS/SDK? Sure, but that does not seem likely.

>> 3.3.2 An Application may not itself install or launch other executable
> code
>> by any means, including without limitation through the use of a plug-in
>> architecture, calling other frameworks, other APIs or otherwise. No
>> interpreted code may be downloaded or used in an Application except for
> code
>> that is interpreted and run by Apple’s Documented APIs and built-in
>> interpreter(s).
>
> I think this indeed pretty effectively rules out installation of packages
> from CRAN, which is a bummer – unless those modules are downloaded &
> installed through the app store. Not sure if that would even work though,
> since they’re not apps.

As I note, between 3.3.2 and 3.3.3, any add-on functionality, such as CRAN packages, would be problematic any way you read it.

> As for the “interpreted code” stuff, there’s so much murkiness about what
> constitutes interpreted code that I don’t know if this is a deal-breaker or
> not. At one extreme, it could prohibit pressing buttons in an app and then
> “interpreting” those presses as commands for the app to “do something.” At
> the other extreme, Somewhere in the middle, it would seem to cover language
> translation apps. The notion of “interpreted” is just not very
> well-defined. For instance, most people think of Perl as an interpreted
> language, but it compiles to bytecode before executing just like Java (it
> just doesn’t typically save it to a bytecode file).

I would say that, beyond the SDK language parsing issues relevant to interpreters, given that Apple rejected BasicMatrix and that there are no other programming language interpreters in the App Store, these are pretty goods sign that R would not pass Apple’s review under these parameters. I take a fairly pragmatic approach there.

> Finally, I do agree with the general tone implied in your post – it is a
> major major hassle that Apple’s overlords control the distribution channel
> for software on non-jailbroken iDevices. I don’t like it at all, for the
> exact reason that people like you & me & the rest of the world now have to
> sit around speculating whether our helpful apps will pass muster with the
> cabal.

As I noted in my closing comments in my second post, if one has a desire to make R’s functionality available on smartphones (iPhone, Android, etc.) or iPad-class devices, then a client/server approach may be the most efficient means to do so. That approach also avails you of more powerful computing platforms than the client side mobile devices have, at least at present, which will also limit aspects of portable functionality.

Regards,

Marc

And lastly Gustaf Rydevik reply

Indeed, the client/server approach is what is used in MatLab Mobile,
which is now on sale in the app store.
See

http://blogs.mathworks.com/desktop/2010/05/24/introducing-matlab-mobile-%E2%80%93-an-iphone-app-to-connect-remotely-to-your-matlab/

If matlab can do it, then surely the R community can as well.

Regards,
Gustaf

———————–

I hope the above will be interesting/useful to some of you in the future.
Best,
Tal