This post shows how to print a prettier nested pivot table, created using the {reshape} package (similar to what you would get with Microsoft Excel), so you could print it either in the R terminal or as a LaTeX table. This task is done by bridging between the cast_df object produced by the {reshape} package, and the tabular function introduced by the new {tables} package.

Here is an example of the type of output we wish to produce in the R terminal:

1
2
3
4
5
6
7

       ozone       solar.r        wind         temp       
 month mean  sd    mean    sd     mean   sd    mean  sd   
 5     23.62 22.22 181.3   115.08 11.623 3.531 65.55 6.855
 6     29.44 18.21 190.2    92.88 10.267 3.769 79.10 6.599
 7     59.12 31.64 216.5    80.57  8.942 3.036 83.90 4.316
 8     59.96 39.68 171.9    76.83  8.794 3.226 83.97 6.585
 9     31.45 24.14 167.4    79.12 10.180 3.461 76.90 8.356

Or in a latex document:

Motivation: creating pretty nested tables

In a recent post we learned how to use the {reshape} package (by Hadley Wickham) in order to aggregate and reshape data (in R) using the melt and cast functions.

The cast function is wonderful but it has one problem – the format of the output. As opposed to a pivot table in (for example) MS excel, the output of a nested table created by cast is very “flat”. That is, there is only one row for the header, and only one column for the row names. So for both the R terminal, or an Sweave document, when we deal with a more complex reshaping/aggregating, the result is not something you would be proud to send to a journal.

The opportunity: the {tables} package

The good news is that Duncan Murdoch have recently released a new package to CRAN called {tables}. The {tables} package can compute and display complex tables of summary statistics and turn them into nice looking tables in Sweave (LaTeX) documents. For using the full power of this package, you are invited to read through its detailed (and well written) 23 pages Vignette. However, some of us might have preferred to keep using the syntax of the {reshape} package, while also benefiting from the great formatting that is offered by the new {tables} package. For this purpose, I devised a function that bridges between cast_df (from {reshape}) and the tabular function (from {tables}).

The bridge: between the {tables} and the {reshape} packages

The code for the function is available on my github (link: tabular.cast_df.r on github) and it seems to works fine as far as I can see (though I wouldn’t run it on larger data files since it relies on melting a cast_df object.)

Here is an example for how to load and use the function:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

######################
# Loading the functions
######################
# Making sure we can source code from github
source("http://www.r-statistics.com/wp-content/uploads/2012/01/source_https.r.txt")
 
# Reading in the function for using tabular on a cast_df object:
source_https("https://raw.github.com/talgalili/R-code-snippets/master/tabular.cast_df.r")
 
 
 
######################
# example:
######################
 
############
# Loading and preparing some data
require(reshape)
names(airquality) <- tolower(names(airquality))
airquality2 <- airquality
airquality2$temp2 <- ifelse(airquality2$temp > median(airquality2$temp), "hot", "cold")
aqm <- melt(airquality2, id=c("month", "day","temp2"), na.rm=TRUE)
colnames(aqm)[4] <- "variable2"	# because otherwise the function is having problem when relying on the melt function of the cast object
head(aqm,3)
#  month day temp2 variable2 value
#1     5   1  cold     ozone    41
#2     5   2  cold     ozone    36
#3     5   3  cold     ozone    12
 
############
# Running the example:
tabular.cast_df(cast(aqm, month ~ variable2, c(mean,sd)))
tabular(cast(aqm, month ~ variable2, c(mean,sd))) # notice how we turned tabular to be an S3 method that can deal with a cast_df object
Hmisc::latex(tabular(cast(aqm, month ~ variable2, c(mean,sd)))) # this is what we would have used for an Sweave document

And here are the results in the terminal:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

> 
> tabular.cast_df(cast(aqm, month ~ variable2, c(mean,sd)))
 
       ozone       solar.r        wind         temp       
 month mean  sd    mean    sd     mean   sd    mean  sd   
 5     23.62 22.22 181.3   115.08 11.623 3.531 65.55 6.855
 6     29.44 18.21 190.2    92.88 10.267 3.769 79.10 6.599
 7     59.12 31.64 216.5    80.57  8.942 3.036 83.90 4.316
 8     59.96 39.68 171.9    76.83  8.794 3.226 83.97 6.585
 9     31.45 24.14 167.4    79.12 10.180 3.461 76.90 8.356
> tabular(cast(aqm, month ~ variable2, c(mean,sd))) # notice how we turned tabular to be an S3 method that can deal with a cast_df object
 
       ozone       solar.r        wind         temp       
 month mean  sd    mean    sd     mean   sd    mean  sd   
 5     23.62 22.22 181.3   115.08 11.623 3.531 65.55 6.855
 6     29.44 18.21 190.2    92.88 10.267 3.769 79.10 6.599
 7     59.12 31.64 216.5    80.57  8.942 3.036 83.90 4.316
 8     59.96 39.68 171.9    76.83  8.794 3.226 83.97 6.585
 9     31.45 24.14 167.4    79.12 10.180 3.461 76.90 8.356

And in an Sweave document:

Here is an example for the Rnw file that produces the above table:
cast_df to tabular.Rnw

I will finish with saying that the tabular function offers more flexibility then the one offered by the function I provided. If you find any bugs or have suggestions of improvement, you are invited to leave a comment here or inside the code on github.

(Link-tip goes to Tony Breyal for putting together a solution for sourcing r code from github.)

(The R code for Barnard’s exact test is at the end of the article, and you could also just download it from here)

About Barnard’s exact test

About half a year ago, I was studying various statistical methods to employ on contingency tables. I came across a promising method for 2×2 contingency tables called “Barnard’s exact test“. Barnard’s test is a non-parametric alternative to Fisher’s exact test which can be more powerful (for 2×2 tables) but is also more time-consuming to compute (References can be found in the Wikipedia article on the subject).

The test was first published by George Alfred Barnard (1945). Mehta and Senchaudhuri (2003) explain why Barnard’s test can be more powerful than Fisher’s under certain conditions:

When comparing Fisher’s and Barnard’s exact tests, the loss of power due to the greater discreteness of the Fisher statistic is somewhat offset by the requirement that Barnard’s exact test must maximize over all possible p-values, by choice of the nuisance parameter, π. For 2 × 2 tables the loss of power due to the discreteness dominates over the loss of power due to the maximization, resulting in greater power for Barnard’s exact test. But as the number of rows and columns of the observed table increase, the maximizing factor will tend to dominate, and Fisher’s exact test will achieve greater power than Barnard’s.

About the R implementation of Barnard’s exact test

After finding about Barnard’s test I was sad to discover that (at the time) there had been no R implementation of it. But last week, I received a surprising e-mail with good news. The sender, Peter Calhoun, currently a graduate student at the University of Florida, had implemented the algorithm in R. Peter had  found my posting on the R mailing list (from almost half a year ago) and was so kind as to share with me (and the rest of the R community) his R code for computing Barnard’s exact test. Here is some of what Peter wrote to me about his code:

On a side note, I believe there are more efficient codes than this one.  For example, I’ve seen codes in Matlab that run faster and display nicer-looking graphs.  However, this code will still provide accurate results and a plot that gives the p-value based on the nuisance parameter.  I did not come up with the idea of this code, I simply translated Matlab code into R, occasionally using different methods to get the same result.  The code was translated from:

Trujillo-Ortiz, A., R. Hernandez-Walls, A. Castro-Perez, L. Rodriguez-Cardozo. Probability Test.  A MATLAB file. URL

http://www.mathworks.com/matlabcentral/fileexchange/loadFile.do?objectId=6198

My goal was to make this test accessible to everyone.  Although there are many ways to run this test through Matlab, I hadn’t seen any code to implement this test in R.  I hope it is useful for you, and if you have any questions or ways to improve this code, please contact me at calhoun.peter@gmail.com

Using the R function for Barnard’s exact test

1
2
3
4
5

source("http://www.r-statistics.com/wp-content/uploads/2010/02/Barnard.R.txt") # downloading the code from the website
#Examples:
Convictions <-matrix(c(2, 10, 15, 3), nrow = 2, dimnames = list(c("Dizygotic", "Monozygotic"), c("Convicted", "Not convicted")))
fisher.test(Convictions, alternative = "less")
Barnard(Convictions)

And here is the output

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

> #Examples:
> Convictions <-matrix(c(2, 10, 15, 3), nrow = 2, dimnames = list(c("Dizygotic", "Monozygotic"), c("Convicted", "Not convicted")))
> fisher.test(Convictions, alternative = "less")
 
        Fisher's Exact Test for Count Data
 
data:  Convictions 
p-value = 0.0004652
alternative hypothesis: true odds ratio is less than 1 
95 percent confidence interval:
 0.0000000 0.2849601 
sample estimates:
odds ratio 
0.04693661 
 
> Barnard(Convictions)
 
 2x2 matrix Barnard's exact test: 100 13x19 tables were evaluated
 -----------------------------------------------------------
 Wald statistic =  3.6099
 Nuisance parameter =  0.44446
 p-values:  1-tailed =  0.00015285 2-tailed =  0.00030569
 -----------------------------------------------------------

Final note: I would like to thank Peter Calhoun again for sharing his code with the rest of us – Thanks Peter!

Update (21.04.2010): In case you are facing a table with structural zeros (that is, missing values in the table), the package aylmer might be able to help you (it offers a generalization of Fisher’s exact test)

Update (16.06.2011): A more updated (and faster) code is now available on the post (thanks goes, again, to Peter Calhoun). The post is updated with a newer example. The old R code for the function can still be found here.

I guess this is not the number one post I would like to start with on this blog, but I feel the time is right for it (community-wise).

I’ll move on to the subject matter in a moment, but first a short intro: This blog is written by Tal Galili. I am an aspiring statistician who also loves to use R for his work. At the same time I am also a WordPress blogger, writing mainly at www.TalGalili.com where I can use my native language (Hebrew) for self expression.

This combination of statistics and blogging will lead me to sometimes much less statistical, but more Web/Open-Source oriented posts like this one. So for the statisticians in the audience I extend my apologies and invite you to wait for future posts which will be more fully focused on Statistics and R.

And now for the topic at hand. . .

*         *         *         *         *
I have just noticed the nice article published on the wordpress development blog titled “2.9 Features Vote Results“. The post exemplifies a wonderful trend in the WordPress community (led by Jane Wells) having to do with connecting between the core team and the WordPress user community. The way Jane does this is by giving surveys to WordPress users,  which in turn offers the WordPress core team an opportunity to understand the community needs.

In the post “2.9 Features Vote Results“, Jane presented the results of such survey. The post had tables and barplots, but the barplots were only present for the one dimensional variables. In contrast, more elaborate data, such as that of question 2 (asking to rate each of 11 potential features on a scale of 1 to 5), was shown only with a table, such as this:

The table gives the full information (although I would love it if it was easily downloadable, instead of having to type in the numbers) – but its main limitation is that from a quick look, one can not easily get (let alone understand) anything.

For the goal of understanding more of the results with a quick glance, I offer two simple and well-known visualizations for the results.

1) Parallel barplots (click for bigger image)

This plot can be easily implemented in Excel (although I did it in R) and can allow us to compare the different ranking each potential feature received.

For example, this shows us that most answers were usually given rank 4 (“would be nice”) for each feature.

2) Mosaic plot (click for bigger image)

I don’t know if this can be done in Excel, but with R it is just a simple line of code.

(mosaicplot((DataSet.table), las = 1, col = c(“gray”,”gray”,”blue”,3,”dark green”), main = “”))

The advantage of this plot is that it allows us to compare the different features easily, while not only comparing the top rank, but also combining different rankings for easy comparison (for example, comparing how many rank 4 or 5 each feature received).

So for example, the plot shows me that the most ranked with number 5 was the feature “easier embeds” but the most ranked “number 4 or 5″ was the feature “custom image sizes”. The feature “media album” came close to these two, but didn’t top either.

Conclusions from this post:

1. It would be nice (if possible) to publish the full data of the surveys, not just the results.
2. The second question of the survey gives different answer than the first question. But since the difference in percentage seems to be so small compared to the other options, I would guess that all of the top 4 features are more or less in the same level of interest to the community.

p.s. to Jane – why do none of the numbers in this table add up to 3406 (the number of respondants) ?

p.p.s.  to Jane and the Dev team – great work people!