R-statistics blogR-statistics blog

R 3.0.1 is released

Tal Galili — Mon, 20 May 2013 14:37:54 +0000

R 3.0.1 (codename “Good Sport”) was released last week. As mentioned earlier by David, this version improves serialization performance with big objects, improves reliability for parallel programming and fixes a few minor bugs.

Upgrading to R 3.0.1

You can download the latest version from here. Or, if you are using windows, you can upgrade to the latest version using the installr package (also available on CRAN and github). Simply run the following code:

# installing/loading the package:
if(!require(installr)) { 
install.packages("installr"); require(installr)} #load / install+load installr
 
updateR(to_checkMD5sums = FALSE) # the use of to_checkMD5sums is because of a slight bug in the MD5 file on R 3.0.1. Soon this should get resolved and you could go back to using updateR(), install.R() or the menu upgrade system.

I try to keep the installr package updated and useful. If you have any suggestions or remarks on the package, you’re invited to leave a comment below.

If you use the global library system (as I do), you can run the following in the new version of R:

source("http://www.r-statistics.com/wp-content/uploads/2010/04/upgrading-R-on-windows.r.txt")
New.R.RunMe()

R 3.0.1 NEWS:

This is the relevant part of the NEWS file

CHANGES IN R 3.0.1:

  NEW FEATURES:

    o chooseCRANmirror() and chooseBioCmirror() gain an ind argument
      (like setRepositories()).

    o mcparallel has a new argument mc.interactive which can modify the
      interactive flag in the child process. The new default is FALSE
      which makes child processes non-interactive by default (this
      prevents lock-ups due to children waiting for interactive input).

    o scan() now warns when end-of-file occurs within a quoted string.

    o count.fields() is now consistent with scan() in its handling of
      newlines in quoted strings.  Instead of triggering an error, this
      results in the current line receiving NA as the field count, with
      the next line getting the total count of the two lines.

    o The default method of image() will plot axes of the class of xlim
      and ylim (and hence of x and y if there is a suitable range()
      method).  Based on a suggestion of Michael Sumner.

    o load() now has a verbose argument for debugging support, to print
      the names of objects just before loading them.

    o When loading a serialized object encounters a reference to a
      namespace which cannot be loaded, this is replaced by a reference
      to the global environment, with a warning.

    o pairs() gains a line.main option for title placement.

    o The remaining instances in which serialization to a raw vector
      was limited to 2GB have been unlimited on a 64-bit platform, and
      in most cases serialization to a vector of more than 1GB will be
      substantially faster.

  UTILITIES:

    o R CMD config now make use of personal Makevars files under ~/.R
      and a site file Makevars.site, in the same way as R CMD SHLIB and
      R CMD INSTALL.  This makes the utility more useful in package
      configure scripts.

      On Windows finding the personal files may require the environment
      variable HOME set.

      The old behaviour can be obtained with the new options
      --no-user-files and --no-site-files.

  PACKAGE INSTALLATION:

    o Alternatives to the site and user customization files
      Makevars.site and ~/.R/Makevars can be specified _via_ the
      environment variables R_MAKEVARS_SITE and R_MAKEVARS_USER
      respectively.  These can be used to suppress the use of the
      default files by setting an empty value (where possible) or a
      non-existent path.

  BUG FIXES:

    o sys.source() did not report error locations when keep.source =
      TRUE.

    o as.POSIXct.numeric was coercing origin using the tz argument and
      not "GMT" as documented (PR#14973).

    o The active binding to assign fields in reference classes has been
      cleaned up to reduce dependence on the class' package
      environment, also fixing bug in initializing read-only fields
      (inspired by a report from Hadley Wickham).

    o str(d) no longer gives an error when names(d) contain illegal
      multibyte strings (PR#15247).

    o Profiling of built-in functions with line.profiling= TRUE did not
      record the line from which they were called.

    o citation(pkg) dropped the header and footer specified in the
      CITATION file (PR#15257).

    o Quotes were handled differently when reading the first line and
      reading the rest, so read.table() misread some files that
      contained quote characters (PR#15245).

    o cat() with sep a character vector of length greater than one and
      more than one argument was using separators inconsistently
      (PR#15261).

    o On Windows in R 3.0.0, savePlot() failed because of an incorrect
      check on the argument count.

    o unzip(list = TRUE) returned Names as a factor and not a character
      vector (as documented) for the internal method.  (Noticed by Sean
      O'Riordain.)

    o contourLines() now checks more comprehensively for conformance of
      its x, y and z arguments (it was used incorrectly in package
      R2G2).

    o Saved graphics display lists are R version-specific.  Attempting
      to load workspaces containing them (or some other
      version-specific objects) aborted the load in R 3.0.0 and
      earlier; now it does a partial load and generates a warning
      instead.

    o In R 3.0.0, identify() and locator() did not record information
      correctly, so replaying a graph (e.g. by copying it to another
      device) would fail. (PR#15271)

    o Calling file.copy() or dirname() with the invalid input "" (which
      was being used in packages, despite not being a file path) could
      have caused a segfault.

      dirname("") is now "" rather than "." (unless it segfaulted).

    o supsmu() could read/write outside its input vectors for very
      short inputs (seen in package rms for n = 4).

    o as.dendrogram()'s hclust method uses less memory and hence gets
      considerably faster for large (n ~ 1000) clusterings, thanks to
      Daniel M"ullner. (PR#15174)

    o The return value when all workers failed from
      parallel::mclapply(mc.presechedule = TRUE) was a list of strings
      and not of error objects.  (Spotted by Karl Forner and Bernd
      Bischl.)

    o In R 3.0.0, when help() found multiple pages with the same alias,
      the HTML display of all the selections was not produced.
      (PR#15282)

    o splinefun(method="monoH.FC") now produces a function with first
      argument named x and allows deriv=3, as documented. (PR#15273)

    o summaryRprof() would only read the first chunksize lines of an
      Rprof file produced with line.profiling=TRUE. By default, this is
      the first 100 seconds.  (PR#15288)

    o lsfit() produced an incorrect error message when argument x had
      more columns than rows or x had a different number of rows than
      y. (Spotted by Renaud Gaujoux.)

    o Binary operations on equal length vectors copied the class name
      from the second operand when the first had no class name, but did
      not set the object bit. (PR#15299)

    o The trace() method for reference generator objects failed after
      those objects became function definitions.

    o write.table() did not check that factors were constructed
      correctly, and so caused a segment fault when writing bad ones.
      (PR#15300)

    o The internal HTTP server no longer chokes on POST requests
      without body. It will also pass-through other request types for
      custom handlers (with the method stored in Request-Method header)
      instead of failing.

R 3.0.0 is released! (what’s new, and how to upgrade)

Tal Galili — Wed, 03 Apr 2013 13:28:20 +0000

A few hours ago Peter Dalgaard (of R Core Team) announced the release of R 3.0.0! Bellow you can read the changes in this release.

One of the features worth noticing is the introduction of long vectors to R 3.0.0. As David Smith recently wrote:

Although many people won’t notice the difference, the introduction of long vectors to R is in fact a significant upgrade, and required a lot of work behind-the-scenes to implement in the core R engine. It will allow data frames to exceed their current 2 billion row limit, and in general allow R to make better use of memory in systems with large amounts of RAM. Many thanks go to the R core team for making this improvement.

You can get the source code from: http://cran.r-project.org/src/base/R-3/R-3.0.0.tar.gz

or wait for it to be mirrored at a CRAN site nearer to you. Binaries for various platforms will appear in due course (which often means it will be within the next 2-48 hours).

If you are running R on Ubuntu, you may wish to consult this post.

If you are running R on Windows, you can use the following code to quickly download and install the latest R version using the installr package:

# installing/loading the package:
if(!require(installr)) { 
install.packages("installr"); require(installr)} #load / install+load installr
install.R(to_checkMD5sums = FALSE) # the use of to_checkMD5sums is because of a slight bug in the MD5 file on R 3.0.0. Soon this should get resolved and you could go back to using updateR()

Either way, all users should note that this new release requires that packages will need to be re-installed, which means that after you install the new R, you should run the following command in it:

update.packages(checkBuilt=TRUE)

(thank to Prof. Ripley for the above clarification, and the FAQ pointer)

R 3.0.0 NEWS:

SIGNIFICANT USER-VISIBLE CHANGES

Packages need to be (re-)installed under this version (3.0.0) of R.
There is a subtle change in behaviour for numeric index values 2^31 and larger. These never used to be legitimate and so were treated as NA, sometimes with a warning. They are now legal for long vectors so there is no longer a warning, and x[2^31] <- y will now extend the vector on a 64-bit platform and give an error on a 32-bit one.
It is now possible for 64-bit builds to allocate amounts of memory limited only by the OS. It may be wise to use OS facilities (e.g.ulimit in a bash shell, limit in csh), to set limits on overall memory consumption of an R process, particularly in a multi-user environment. A number of packages need a limit of at least 4GB of virtual memory to load.64-bit Windows builds of R are by default limited in memory usage to the amount of RAM installed: this limit can be changed by command-line option –max-mem-size or setting environment variable R_MAX_MEM_SIZE.
Negative numbers for colours are consistently an error: previously they were sometimes taken as transparent, sometimes mapped into the current palette and sometimes an error.

NEW FEATURES

identical() has a new argument, ignore.environment, used when comparing functions (with default FALSE as before).
There is a new option, options(CBoundsCheck=), which controls how .C() and .Fortran() pass arguments to compiled code. If true (which can be enabled by setting the environment variable R_C_BOUNDS_CHECK to yes), raw, integer, double and complex arguments are always copied, and checked for writing off either end of the array on return from the compiled code (when a second copy is made). This also checks individual elements of character vectors passed to .C().This is not intended for routine use, but can be very helpful in finding segfaults in package code.
In layout(), the limits on the grid size have been raised (again).
New simple provideDimnames() utility function.
Where methods for length() return a double value which is representable as an integer (as often happens for package Matrix), this is converted to an integer.
Matrix indexing of dataframes by two-column numeric indices is now supported for replacement as well as extraction.
setNames() now has a default for its object argument, useful for a character result.
StructTS() has a revised additive constant in the loglik component of the result: the previous definition is returned as the loglik0component. However, the help page has always warned of a lack of comparability of log-likelihoods for non-stationary models. (Suggested by Jouni Helske.)
The logic in aggregate.formula() has been revised. It is now possible to use a formula stored in a variable; previously, it had to be given explicitly in the function call.
install.packages() has a new argument quiet to reduce the amount of output shown.
Setting an element of the graphics argument lwd to a negative or infinite value is now an error. Lines corresponding to elements with values NA or NaN are silently omitted.Previously the behaviour was device-dependent.
Setting graphical parameters cex, col, lty, lwd and pch in par() now requires a length-one argument. Previously some silently took the first element of a longer vector, but not always when documented to do so.
Sys.which() when used with inputs which would be unsafe in a shell (e.g. absolute paths containing spaces) now uses appropriate quoting.
as.tclObj() has been extended to handle raw vectors. Previously, it only worked in the other direction. (Contributed by Charlie Friedemann, PR#14939.)
New functions cite() and citeNatbib() have been added, to allow generation of in-text citations from "bibentry" objects. Acite() function may be added to bibstyle() environments.
A sort() method has been added for "bibentry" objects.
The bibstyle() function now defaults to setting the default bibliography style. The getBibstyle() function has been added to report the name of the current default style.
scatter.smooth() now has an argument lpars to pass arguments to lines().
pairs() has a new log argument, to allow some or all variables to be plotted on logarithmic scale. (In part, wish of PR#14919.)
split() gains a sep argument.
termplot() does a better job when given a model with interactions (and no longer attempts to plot interaction terms).
The parser now incorporates code from Romain Francois’ parser package, to support more detailed computation on the code, such as syntax highlighting, comment-based documentation, etc. Functions getParseData() and getParseText() access the data.
There is a new function rep_len() analogous to rep.int() for when speed is required (and names are not).
The undocumented use rep(NULL, length.out = n) for n > 0 (which returns NULL) now gives a warning.
demo() gains an encoding argument for those packages with non-ASCII demos: it defaults to the package encoding where there is one.
strwrap() converts inputs with a marked encoding to the current locale: previously it made some attempt to pass through as bytes inputs invalid in the current locale.
Specifying both rate and scale to [dpqr]gamma is a warning (if they are essentially the same value) or an error.
merge() works in more cases where the data frames include matrices. (Wish of PR#14974.)
optimize() and uniroot() no longer use a shared parameter object across calls. (nlm(), nlminb() and optim() with numerical derivatives still do, as documented.)
The all.equal() method for date-times is now documented: times are regarded as equal (by default) if they differ by up to 1 msec.
duplicated() and unique() gain a nmax argument which can be used to make them much more efficient when it is known that there are only a small number of unique entries. This is done automatically for factors.
Functions rbinom(), rgeom(), rhyper(), rpois(), rnbinom(), rsignrank() and rwilcox() now return integer (not double) vectors. This halves the storage requirements for large simulations.
sort(), sort.int() and sort.list() now use radix sorting for factors of less than 100,000 levels when method is not supplied. So does order() if called with a single factor, unless na.last = NA.
diag() as used to generate a diagonal matrix has been re-written in C for speed and less memory usage. It now forces the result to be numeric in the case diag(x) since it is said to have ‘zero off-diagonal entries’.
backsolve() (and forwardsolve()) are now internal functions, for speed and support for large matrices.
More matrix algebra functions (e.g. chol() and solve()) accept logical matrices (and coerce to numeric).
sample.int() has some support for n >= 2^31: see its help for the limitations.A different algorithm is used for (n, size, replace = FALSE, prob = NULL) for n > 1e7 and size <= n/2. This is much faster and uses less memory, but does give different results.
approxfun() and splinefun() now return a wrapper to an internal function in the stats namespace rather than a .C() or .Call()call. This is more likely to work if the function is saved and used in a different session.
The functions .C(), .Call(), .External() and .Fortran() now give an error (rather than a warning) if called with a named first argument.
Sweave() by default now reports the locations in the source file(s) of each chunk.
clearPushBack() is now a documented interface to a long-existing internal call.
aspell() gains filters for R code, Debian Control Format and message catalog files, and support for R level dictionaries. In addition, package utils now provides functions aspell_package_R_files() and aspell_package_C_files() for spell checking R and C level message strings in packages.
bibentry() gains some support for “incomplete” entries with a crossref field.
gray() and gray.colors() finally allow alpha to be specified.
monthplot() gains parameters to control the look of the reference lines. (Suggestion of Ian McLeod.)
Added support for new %~% relation (“is distributed as”) in plotmath.
domain = NA is accepted by gettext() and ngettext(), analogously to stop() etc.
termplot() gains a new argument plot = FALSE which returns information to allow the plots to be modified for use as part of other plots, but does not plot them. (Contributed by Terry Therneau, PR#15076.)
quartz.save(), formerly an undocumented part of R.app, is now available to copy a device to a quartz() device. dev.copy2pdf()optionally does this for PDF output: quartz.save() defaults to PNG.
The default method of pairs() now allows text.panel = NULL and the use of .panel = NULL is now documented.
setRefClass() and getRefClass() now return class generator functions, similar to setClass(), but still with the reference fields and methods as before (suggestion of Romain Francois).
New functions bitwNot(), bitwAnd(), bitwOr() and bitwXor(), using the internal interfaces previously used for classes"octmode" and "hexmode".Also bitwShiftL() and bitwShiftR() for shifting bits in elements of integer vectors.
New option "deparse.cutoff" to control the deparsing of language objects such as calls and formulae when printing. (Suggested by a comment of Sarah Goslee.)
colors() gains an argument distinct.
New demo(colors) and demo(hclColors), with utility functions.
list.files() (aka dir()) gains a new optional argument no.. which allows to exclude "." and ".." from listings.
Multiple time series are also of class "matrix"; consequently, head(), e.g., is more useful.
encodeString() preserves UTF-8 marked encodings. Thus if factor levels are marked as UTF-8 an attempt is made to print them in UTF-8 in RGui on Windows.
readLines() and scan() (and hence read.table()) in a UTF-8 locale now discard a UTF-8 byte-order-mark (BOM). Such BOMs are allowed but not recommended by the Unicode Standard: however Microsoft applications can produce them and so they are sometimes found on websites.The encoding name "UTF-8-BOM" for a connection will ensure that a UTF-8 BOM is discarded.
mapply(FUN, a1, ..) now also works when a1 (or a further such argument) needs a length() method (which the documented arguments never do). (Requested by Hervé Pagès; with a patch.)
.onDetach() is supported as an alternative to .Last.lib. Unlike .Last.lib, this does not need to be exported from the package’s namespace.
The srcfile argument to parse() may now be a character string, to be used in error messages.
The format() method for ftable objects gains a method argument, propagated to write.ftable() and print(), allowing more compact output, notably for LaTeX formatting, thanks to Marius Hofert.
The utils::process.events() function has been added to trigger immediate event handling.
Sys.which() now returns NA (not "") for NA inputs (related to PR#15147).
The print() method for class "htest" gives fewer trailing spaces (wish of PR#15124).Also print output from HoltWinters(), nls() and others.
loadNamespace() allows a version specification to be given, and this is used to check version specifications given in the Importsfield when a namespace is loaded.
setClass() has a new argument, slots, clearer and less ambiguous than representation. It is recommended for future code, but should be back-compatible. At the same time, the allowed slot specification is slightly more general. See the documentation for details.
mget() now has a default for envir (the frame from which it is called), for consistency with get() and assign().
close() now returns an integer status where available, invisibly. (Wish of PR#15088.)
The internal method of tar() can now store paths too long for the ustar format, using the (widely supported) GNU extension. It can also store long link names, but these are much less widely supported. There is support for larger files, up to the ustar limit of 8GB.
Local reference classes have been added to package methods. These are a technique for avoiding unneeded copying of large components of objects while retaining standard R functional behavior. See ?LocalReferenceClasses.
untar() has a new argument restore_times which if false (not the default) discards the times in the tarball. This is useful if they are incorrect (some tarballs submitted to CRAN have times in a local timezone or many years in the past even though the standard required them to be in UTC).
replayplot() cannot (and will not attempt to) replay plots recorded under R < 3.0.0. It may crash the R session if an attempt is made to replay plots created in a different build of R >= 3.0.0.
Palette changes get recorded on the display list, so replaying plots (including when resizing screen devices and using dev.copy()) will work better when the palette is changed during a plot.
chol(pivot = TRUE) now defaults to LAPACK, not LINPACK.
The parse() function has a new parameter keep.source, which defaults to options("keep.source").
Profiling via Rprof() now optionally records information at the statement level, not just the function level.
The Rprof() function now quotes function names in in its output file on Windows, to be consistent with the quoting in Unix.
Profiling via Rprof() now optionally records information about time spent in GC.
The HTML help page for a package now displays non-vignette documentation files in a more accessible format.
To support options(stringsAsFactors = FALSE), model.frame(), model.matrix() and replications() now automatically convert character vectors to factors without a warning.
The print method for objects of class "table" now detects tables with 0-extents and prints the results as, e.g., < table of extent 0 x 1 x 2 >. (Wish of PR#15198.)
Deparsing involving calls to anonymous functions and has been made closer to reversible by the addition of extra parentheses.
The function utils::packageName() has been added as a lightweight version of methods::getPackageName().
find.package(lib.loc = NULL) now treats loaded namespaces preferentially in the same way as attached packages have been for a long time.
In Windows, the Change Directory dialog now defaults to the current working directory, rather than to the last directory chosen in that dialog.
available.packages() gains a "license/restricts_use" filter which retains only packages for which installation can proceed solely based on packages which are guaranteed not to restrict use.
New check_packages_in_dir() function in package tools for conveniently checking source packages along with their reverse dependencies.
R’s completion mechanism has been improved to handle help requests (starting with a question mark). In particular, help prefixes are now supported, as well as quoted help topics. To support this, completion inside quotes are now handled by R by default on all platforms.
The memory manager now allows the strategy used to balance garbage collection and memory growth to be controlled by setting the environment variable R_GC_MEM_GROW. See ?Memory for more details.
(‘For experts only’, as the introductory manual says.) The use of environment variables R_NSIZE and R_VSIZE to control the initial (= minimum) garbade collection trigger for number of cons cels and size of heap has been restored: they can be overridden by the command-line options --min-nsize and --min-vsize; see ?Memory.
On Windows, the device name for bitmap devices as reported by .Device and .Devices no longer includes the file name. This is for consistency with other platforms and was requested by the lattice maintainer.win.metafile() still uses the file name: the exact form is used by package tkrplot.
set.seed(NULL) re-initializes .Random.seed as done at the beginning of the session if not already set. (Suggestion of Bill Dunlap.)
The breaks argument in hist.default() can now be a function that returns the breakpoints to be used (previously it could only return the suggested number of breakpoints).
File ‘share/licenses/licenses.db’ has some clarifications, especially as to which variants of ‘BSD’ and ‘MIT’ is intended and how to apply them to packages. The problematic licence ‘Artistic-1.0’ has been removed.

LONG VECTORS

This section applies only to 64-bit platforms.

There is support for vectors longer than 2^31 – 1 elements. This applies to raw, logical, integer, double, complex and character vectors, as well as lists. (Elements of character vectors remain limited to 2^31 – 1 bytes.)
Most operations which can sensibly be done with long vectors work: others may return the error ‘long vectors not supported yet’. Most of these are because they explicitly work with integer indices (e.g. anyDuplicated() and match()) or because other limits (e.g. of character strings or matrix dimensions) would be exceeded or the operations would be extremely slow.
length() returns a double for long vectors, and lengths can be set to 2^31 or more by the replacement function with a double value.
Most aspects of indexing are available. Generally double-valued indices can be used to access elements beyond 2^31 – 1.
There is some support for matrices and arrays with each dimension less than 2^31 but total number of elements more than that. Only some aspects of matrix algebra work for such matrices, often taking a very long time. In other cases the underlying Fortran code has an unstated restriction (as was found for complex svd()).
dist() can produce dissimilarity objects for more than 65536 rows (but for example hclust() cannot process such objects).
serialize() to a raw vector is unlimited in size (except by resources).
The C-level function R_alloc can now allocate 2^35 or more bytes.
agrep() and grep() will return double vectors of indices for long vector inputs.
Many calls to .C() have been replaced by .Call() to allow long vectors to be supported (now or in the future). Regrettably several packages had copied the non-API .C() calls and so failed.
.C() and .Fortran() do not accept long vector inputs. This is a precaution as it is very unlikely that existing code will have been written to handle long vectors (and the R wrappers often assume that length(x) is an integer).
Most of the methods for sort() work for long vectors.rank(), sort.list() and order() support long vectors (slowly except for radix sorting).
sample() can do uniform sampling from a long vector.

PERFORMANCE IMPROVEMENTS

More use has been made of R objects representing registered entry points, which is more efficient as the address is provided by the loader once only when the package is loaded.This has been done for packages base, methods, splines and tcltk: it was already in place for the other standard packages.
Since these entry points are always accessed by the R entry points they do not need to be in the load table which can be substantially smaller and hence searched faster. This does mean that .C / .Fortran / .Call calls copied from earlier versions of R may no longer work – but they were never part of the API.
Many .Call() calls in package base have been migrated to .Internal() calls.
solve() makes fewer copies, especially when b is a vector rather than a matrix.
eigen() makes fewer copies if the input has dimnames.
Most of the linear algebra functions make fewer copies when the input(s) are not double (e.g. integer or logical).
A foreign function call (.C() etc) in a package without a PACKAGE argument will only look in the first DLL specified in the ‘NAMESPACE’ file of the package rather than searching all loaded DLLs. A few packages needed PACKAGE arguments added.
The @<- operator is now implemented as a primitive, which should reduce some copying of objects when used. Note that the operator object must now be in package base: do not try to import it explicitly from package methods.

PACKAGE INSTALLATION

The transitional support for installing packages without namespaces (required since R 2.14.0) has been removed. R CMD build will still add a namespace, but a .First.lib() function will need to be converted.R CMD INSTALL no longer adds a namespace (so installation will fail), and a .First.lib() function in a package will be ignored (with an installation warning for now).
As an exception, packages without a ‘R’ directory and no ‘NAMESPACE’ file can still be installed.
Packages can specify in their ‘DESCRIPTION file’ a line like
```
     Biarch: yes
```
to be installed on Windows with –force-biarch.
Package vignettes can now be processed by other engines besides Sweave; see ‘Writing R Extensions’ and thetools::vignetteEngine help topic for details.
The ‘*.R’ tangled source code for vignettes is now included in tarballs when R CMD build is used to produce them. In R 3.0.0, ‘*.R’ files not in the sources will be produced at install time, but eventually this will be dropped.
The package type "mac.binary" now looks in a path in the repository without any Mac subtype (which used to be universal orleopard): it looks in ‘bin/macosx/contrib/3.0’ rather than ‘bin/macosx/leopard/contrib/2.15’). This is the type used for the CRANbinary distribution for OS X as from R 3.0.0.
File ‘etc/Makeconf’ makes more use of the macros $(CC), $(CXX), $(F77) and $(FC), so the compiler in use can be changed by setting just these (and if necessary the corresponding flags and FLIBS) in file ‘~/.R/Makevars’.This is convenient for those working with binary distributions of R, e.g. on OS X.

UTILITIES

R CMD check now gives a warning rather than a note if it finds calls to abort, assert or exit in compiled code, and has been able to find the ‘.o’ file in which the calls occur.Such calls can terminate the R process which loads the package.
The location of the build and check environment files can now be specified by the environment variables R_BUILD_ENVIRON andR_CHECK_ENVIRON, respectively.
R CMD Sweave gains a –compact option to control possibly reducing the size of the PDF file it creates when –pdf is given.
R CMD build now omits Eclipse’s ‘.metadata’ directories, and R CMD check warns if it finds them.
R CMD check now does some checks on functions defined within reference classes, including of .Call() etc calls.
R CMD check --as-cran notes assignments to the global environment, calls to data() which load into the global environment, and calls to attach().
R CMD build by default uses the internal method of tar() to prepare the tarball. This is more likely to produce a tarball compatible with R CMD INSTALL and R CMD check: an external tar program, including options, can be specified via the environment variableR_BUILD_TAR.
tools::massageExamples() is better protected against packages which re-define base functions such as cat() and get() and so can cause R CMD check to fail when checking examples.
R CMD javareconf has been enhanced to be more similar to the code used by configure.There is now a test that a JNI program can be compiled (like configure did) and only working settings are used.
It makes use of custom settings from configuration recorded in ‘etc/javaconf’.
The –no-vignettes argument of R CMD build has been renamed to the more accurate –no-build-vignettes: its action has always been to (re)build vignettes and never omitted them.R CMD check accepts –no-build-vignettes as a preferred synonym for –no-rebuild-vignettes.

DEPRECATED AND DEFUNCT

The ENCODING argument to .C() is defunct. Use iconv() instead.
The .Internal(eval.with.vis) non-API function has been removed.
Support for the converters for use with .C() has been removed, including the oft misused non-API header ‘R_ext/RConverters.h’.
The previously deprecated uses of array() with a 0-length dim argument and tapply() with a 0-length INDEX list are now errors.
Translation packages are defunct.
Calling rep() or rep.int() on a pairlist or other non-vector object is now an error.
Several non-API entry points have been transferred to packages (e.g. R_zeroin2) or replaced by different non-API entry points (e.g.R_tabulate).
The ‘internal’ graphics device invoked by .Call("R_GD_nullDevice", package = "grDevices") has been removed: usepdf(file = NULL) instead.
The .Fortran() entry point "dqrls" which has not been used by R since version 2.15.1 is no longer available.
Functions traceOn() and traceOff() in package methods are now defunct.
Function CRAN.packages() is finally defunct.
Use of col2rgb(0) is defunct: use par("bg") or NA instead.
The long-defunct functions Rd_parse(), anovalist.lm(), categpry(), clearNames(), gammaCody(), glm.fit.null(),lm.fit.null(), lm.wfit.null(), manglePackageNames(), mauchley.test(), package.contents(), print.coefmat(),reshapeLong(), reshapeWide(), tkclose(), tkcmd(), tkfile.dir(), tkfile.tail(), tkopen(), tkputs(), tkread(),trySilent() and zip.file.extract() have been removed entirely (but are still documented in the help system).
The unused dataPath argument to attachNamespace() has been removed.
grid.prompt() has been removed: use devAskNewPage() instead.
The long-deprecated intensities component is no longer returned by hist().
mean() for data frames and sd() for data frames and matrices are defunct.
chol(pivot = FALSE, LINPACK = TRUE), ch2inv(LINPACK = TRUE), eigen(EISPACK = TRUE), solve(LINPACK = TRUE) andsvd(LINPACK = TRUE) are defunct: LAPACK will be used, with a warning.
The keep.source argument to library() and require() is defunct. This option needs to be set at install time.
Documentation for real(), as.real() and is.real() has been moved to ‘defunct’ and the functions removed.
The maxRasters argument of pdf() (unused since R 2.14.0) has been removed.
The unused fontsmooth argument has been removed from the quartz() device.
All the (non-API) EISPACK entry points in R have been removed.
chol(pivot = TRUE, LINPACK = TRUE) is deprecated.
The long-deprecated use of \synopsis in the Usage section of ‘.Rd’ files will be removed in R 3.1.0.
.find.package() and .path.package() are deprecated: only the public versions without the dot have ever been in the API.

CODE MIGRATION

The C code underlying base graphics has been migrated to the graphics package (and hence no longer uses .Internal() calls).
Most of the .Internal() calls used in the stats package have been migrated to C code in that package.This means that a number of .Internal() calls which have been used by packages no longer exist, including .Internal(cor).Internal(cov), .Internal(optimhess) and .Internal(update.formula).
Some .External() calls to the base package (really to the R executable or shared library) have been moved to more appropriate packages. Packages should not have been using such calls, but some did (mainly those used by integrate()).

PACKAGE parallel

There is a new function mcaffinity() which allows getting or setting the CPU affinity mask for the current R process on systems that supports this (currently only Linux has been tested successfully). It has no effect on systems which do not support process affinity. Users are not expected to use this function directly (with the exception of fixing libraries that break affinity settings like OpenBLAS) – the function is rather intended to support affinity control in high-level parallel functions. In the future, R may supplement lack of affinity control in the OS by its own bookkeeping via mcaffinity() related to processes and threads it spawns.
mcparallel() has a new argument mc.affinity which attempts to set the affinity of the child process according to the specification contained therein.
The port used by socket clusters is chosen randomly: this should help to avoid clashes observed when two users of a multi-user machine try to create a cluster at the same time. To reproduce the previous behaviour set environment variable R_PARALLEL_PORTto 10187.

C-LEVEL FACILITIES

There has been some minor re-organization of the non-API header files. In particular, ‘Rinternals.h’ no longer includes the non-API header ‘R_exts/PrtUtil.h’, and that no longer includes ‘R_exts/Print.h’.
Passing NULL to .C() is now an error.
.C() and .Fortran() now warn if "single" arguments are used with DUP = FALSE, as changes to such arguments are not returned to the caller.
C entry points R_qsort and R_qsort_I now have start and end as size_t to allow them to work with longer vectors on 64-bit platforms. Code using them should be recompiled.
A few recently added C entry points were missing the remapping to Rf_, notably [dpq]nbinom_mu.
Some of the interface pointers formerly available only to R.app are now available to front-ends on all Unix-alikes: one has been added for the interface to View().
PACKAGE = "" is now an error in .C() etc calls: it was always contrary to the documentation.
Entry point rcont2 has been migrated to package stats and so is no longer available.
R_SVN_REVISION in ‘Rversion.h’ is now an integer (rather than a string) and hence usable as e.g. #if R_SVN_REVISION < 70000.
The entry points rgb2hsv and hsv2rgb have been migrated to package grDevices and so are no longer available.
R_GE_version has been increased to 10 and name2col removed (use R_GE_str2col instead). R internal colour codes are now defined using the typedef rcolor.
The REPROTECT macro now checks that the protect index is valid.
Several non-API entry points no longer used by R have been removed, including the Fortran entry points chol, chol2inv, cg, ch andrg, and the C entry points Brent_fmin, fft_factor and fft_work.
If a .External call is registered with a number of arguments (other than -1), the number of arguments passed is checked for each call (as for other foreign function calls).
It is now possible to write custom connection implementations outside core R using ‘R_ext/Connections.h’. Please note that the implementation of connections is still considered internal and may change in the future (see the above file for details).

INTERNATIONALIZATION

The management of translations has been converted to R code: see ?tools::update_pkg_po.
The translations for the R interpreter and RGui.exe are now part of the base package (rather than having sources in directory ‘po’ and being installed to ‘share/locale’). Thus the base package supports three translation domains, R-base, R and RGui.
The compiled translations which ship with R are all installed to the new package translations for easier updating. The first package of that name found on .libPaths() at the start of the R session will be used. (It is possible messages will be used before .libPaths()is set up in which case the default translations will be used: set environment variable R_TRANSLATIONS to point to the location of the intended translations package to use this right from the start.)
The translations form a separate group in the Windows installer, so can be omitted if desired.
The markup for many messages has been changed to make them easier to translate, incorporating suggestions from Łukasz Daniel.

INSTALLATION

There is again support for building without using the C ‘long double’ type. This is required by C99, but system implementations can be slow or flawed. Use configure option –disable-long-double.
make pdf and make install-pdf now make and install the full reference index (including all base and recommended packages).
The ‘reference manual’ on the Windows GUI menu and included in the installer is now the full reference index, including all base and recommended packages.
The Windows installer no longer installs a Start Menu link to the static help pages; as most pages are generated dynamically, this led to a lot of broken links.
Any custom settings for Java configuration are recorded in file ‘etc/javaconf’ for subsequent use by R CMD javareconf.
There is now support for makeinfo version 5.0 (which requires a slightly different ‘.texi’ syntax).
The minimum versions for –use-system-zlib and --use-system-pcre are now tested as 1.2.5 and 8.10 respectively.
On Windows, the stack size is reduced to 16MB on 32-bit systems: misguided users were launching many threads without controlling the stack size.
configure no longer looks for file ‘~/.Rconfig’: ‘~/.R/config’ has long been preferred.

BUG FIXES

When R CMD build is run in an encoding other than the one specified in the package’s ‘DESCRIPTION’ file it tries harder to expand the authors@R field in the specified encoding. (PR#14958)
If R CMD INSTALL is required to expand the authors@R field of the ‘DESCRIPTION’ file, it tries harder to do so in the encoding specified for the package (rather than using ASCII escapes).
Fix in package grid for pushing a viewport into a layout cell, where the layout is within a viewport that has zero physical width OR where the layout has zero total relative width (likewise for height). The layout column widths (or row heights) in this case were being calculated with non-finite values. (Reported by Winston Chang.)
solve(A, b) for a vector b gave the answer names from colnames(A) for LINPACK = TRUE but not in the default case.
La.svd() accepts logical matrices (as documented, and as svd() did).
legend() now accepts negative pch values, in the same way points() long has.
Parse errors when installing files now correctly display the name of the file containing the bad code.
In Windows, tcltk windows were not always properly constructed. (PR#15150)
The internal functions implementing parse(), tools::parseLatex() and tools::parse_Rd() were not reentrant, leading to errors in rare circumstances such as a garbage collection triggering a recursive call.
Field assignments in reference class objects via $<- were not being checked because the magic incantation to turn methods on for that primitive operator had been inadvertently omitted.
setHook(hookname, value, action="replace") set the hook to be the value, rather than a list containing the value as documented. (PR#15167)
If a package used a ‘NEWS.Rd’ file, the main HTML package index page did not link to it. (Reported by Dirk Eddelbuettel.)
The primitive implementation of @<- was not checking the class of the replacement. It now does a check, quicker but less general thanslot<-. See the help.
split(x, f) now recycles classed objects x in the same way as vectors. (Reported by Martin Morgan.)
pbeta(.28, 1/2, 2200, lower.tail=FALSE, log.p=TRUE) is no longer -Inf; ditto for corresponding pt() and pf() calls, such as pt(45, df=5000, lower.tail=FALSE, log.p=TRUE). (PR#15162)
The Windows graphics device would crash R if a user attempted to load the graphics history from a variable that was not a saved history. (PR#15230)
The workspace size for the predict() method for loess() could exceed the maximum integer size. (Reported by Hiroyuki Kawakatsu.)
ftable(x, row.vars, col.vars) now also works when the *.vars arguments are (integer or character vectors) of length zero.
Calling cat() on a malformed UTF-8 string could cause the Windows GUI to lock up. (PR#15227)

Updating R (on Windows) through a menu-bar: installr 0.9 released on CRAN

Tal Galili — Mon, 01 Apr 2013 13:06:25 +0000

In preparation for the upcoming release of R 3.0.0, a new release 0.9 of installr is now on CRAN.

The package can be installed and loaded using:

# installing/loading the package:
if(!require(installr)) { 
install.packages("installr"); require(installr)} #load / install+load installr

The new version includes various bug fixes (as can be seen in the NEWS file) and new functions and features. The most user visible feature is that from now on, whenever loading installr in the Rgui, it will add a new menu-bar for updating your R version (the menu is removed when the package is detached).

When choosing to update R, a new GUI based system will guide you step by step through the updating process. It will first check if a newer version of R is available, if so, it will offer to show the latest NEWS of that release, download and install the new version, and copy/move your packages from the previous library folder, to the one in the new installation. If you have a global library folder, you can simply stop the updating once your new R is installed, and continue as you would otherwise (in the future, I intend to update the package to also allow it to deal with people using a global library folder).

(for using {installr} to update R through R terminal, see my previous post: Updating R from R (on Windows) – using the {installr} package)

Another new feature is the “installr()” function (which can also be run through the menubar), running it will open a window with a list of software you can download and install using the installr package (From Rtools and RStudio to pandoc and MikTeX).

I hope you’ll enjoy this new release, and as always – please let me know in the comments (or via e-mail) if you come across any bugs or have suggestions for new features.

Writing a MS-Word document using R (with as little overhead as possible)

Tal Galili — Mon, 25 Mar 2013 02:52:56 +0000

The problem: producing a Word (.docx) file of a statistical report created in R, with as little overhead as possible.
The solution: combining R+knitr+rmarkdown+pander+pandoc (it is easier than it is spelled).

If you get what this post is about, just jump to the “Solution: the workflow” section.

Preface: why is this a problem (/still)

Before turning to the solution, let’s address two preliminary questions:

Q: Why is it important to be able to create report in Word from R?

A: Because many researchers we may work with are used to working with Word for editing their text, tracking changes and merging edits between different authors, and copy-pasting text/tables/images from various sources.
This means that a report produced as a PDF file is less useful for collaborating with less-tech-savvy researchers (copying text or tables from PDF is not fun). Even exchanging HTML files may appear somewhat awkward to fellow researchers.

Q: But wasn’t this problem solved already?

A: Yes and no. There have been many attempts at solving the problem in the past several years, but many of them came with an overhead which made the solutions un-friendly (the developers and heavy users of these technologies are asked to not be offended – this is only my opinion, and you’re welcome to respond and expand my point of view).
Previous solutions include SWord and R2wd, both rely on the rcom package (and the statconnDCOM or RDCOMClient servers). Or using online converters to turn PDF files into Word files.

Q: Any more issues?
A: Yes. Another big issue is formatting the output. If I would like my tables to look nice in the output file, I would often need to start wrapping ALL of my output functions with the some function (taken from packages such as xtable, rms, quantreg, stargazer, pander, and more.

Sources/links

The solution I propose here is a combination of using the following R packages: knitr, rmarkdown, pander. Combined with the external tool pandoc (easily installed using the installr package).

Combining these ideas together has been discussed before in various places in the past half year or so, here are just a few:

Reproducible research with R, Knitr, Pandoc and Word
Reproducible research with markdown, knitr and pandoc
Formatting tables in markdown
Workflow w/ reports package
Other links… (to be extended based on your comments)

Solution: the workflow

An overview of the steps:

Write text with R code chunks weaved-together (I do it using RStudio, markdown, knitr – in an .rmd file)
At the beginning of the file – make sure to replace the “print” method with that of the markdown wrapping package (see example bellow)
Compile the doc into .md using knitr
Turn the .md into .docx using pandoc

Here is an example rmarkdown code for steps 1 and 2:

 
Doc header 1
============
```{r set_knitr_chunk_options}
opts_chunk$set(echo=FALSE,message=FALSE,results = "asis") # important for making sure the output will be well formatted.
```
 
```{r load_pander_methods}
require(pander)
replace.print.methods <- function(PKG_name = "pander") {
   PKG_methods <- as.character(methods(PKG_name))
   print_methods <- gsub(PKG_name, "print", PKG_methods)
   for(i in seq_along(PKG_methods)) {
      f <- eval(parse(text=paste(PKG_name,":::", PKG_methods[i], sep = ""))) # the new function to use for print
      assign(print_methods[i], f, ".GlobalEnv")
   }   
}
replace.print.methods()
## The following might work with some tweaks:
## print <- function (x, ...) UseMethod("pander")
```
Some text explaining the analysis we are doing
```{r}
summary(cars)# a summary table
fit <- lm(dist~speed, data = cars)
fit
plot(cars) # a plot
```

The above code can be saved into an .rmd file, for example: example.rmd
This file can now be compiled using knitr:

library(knitr)
knit2html("example.rmd")

This will produce an example.md file, which can be compiled into a Word file using pandoc.
If you don’t yet have pandoc, and are running a Windows OS, you can quickly install pandoc by running the following code in R:

# installing/loading the package:
if(!require(installr)) { install.packages("installr"); require(installr)} #load / install+load installr 
# Installing pandoc
install.pandoc(use_regex = FALSE)

Once pandoc is installed, simply run:

FILE <- "example"
system(paste0("pandoc -o ", FILE, ".docx ", FILE, ".md"))

And your .docx file is ready!

Possible expansions and caveats

The first caveat of this method is that it relies on markdown and pander, which is (by definition) more limited than using something like LaTeX. For that purpose, one can decide to work with LaTeX based solutions. Here is an example of how to do it with several existing packages (this code bellow is not very debugged – so more careful attention should be given to using it – I welcome comments and suggestions):

 
```{r load_pander_methods}
replace.print.methods <- function(PKG_name = "pander") {
   PKG_methods <- as.character(methods(PKG_name))
   print_methods <- gsub(PKG_name, "print", PKG_methods)
   for(i in seq_along(PKG_methods)) {
      f <- eval(parse(text=paste(PKG_name,":::", PKG_methods[i], sep = ""))) # the new function to use for print
      assign(print_methods[i], f, ".GlobalEnv")
   }   
}
require(xtable)
replace.print.methods("xtable")
```

Similar solutions can probably be found for HTML documents also. (credit: The above code is based on the help of Ramnath to my question on SO)

The second caveat is that the above solution (at least the part that makes sure we can use the R code as is, without wrapping it with things like “pander(summary(cars))”), is basically a dirty hack. It is a hack in the sense that it overrides basic R commands (which is quite ugly really). This issue is being thought about and discussed for over a month now in the knitr github page, I hope a better solution will come out of it.

The third issue is that if you use a function for which there is an issue with the method, it might cause problems in compiling the code (for example, pander still needs a pander.summary.lm method…).

To conclude: Thanks to the amazing work by Yihui on knitr, by the people at RStudio, by Jeffrey Horner on markdown, Gergely Daróczi for pander, and many others – it is now easier than ever to quickly create a docx report based on analysis performed using R. It seems that 2012 was a great year for reproducible research, I’m looking forward to 2013…

Updating R from R (on Windows) – using the {installr} package

Tal Galili — Tue, 05 Mar 2013 09:37:38 +0000

Upgrading R on Windows is not easy. While the R FAQ offer guidelines, some users may prefer to simply run a command in order to upgrade their R to the latest version. That is what the new {installr} package is all about.

The {installr} package offers a set of R functions for the installation and updating of software (currently, only on Windows OS), with a special focus on R itself. To update R, you can simply run the following code:

# installing/loading the package:
if(!require(installr)) { 
install.packages("installr"); require(installr)} #load / install+load installr
 
# using the package:
updateR() # this will start the updating process of your R installation.  It will check for newer versions, and if one is available, will guide you through the decisions you'd need to make.

Running this function will perform the following steps:

Check what is the latest R version. If the current installed R version is up-to-date, the function ends (and returns FALSE)
If a newer version of R is available, you will be asked if to review the NEWS of the latest R version – in order to decide if to install the
newest R or not.
If you wish it – the function will download and install the latest R version. (you will need to press the "next" buttons on your own)
Once the installation is done, you should press "any-key", and the function will proceed with copying all of your packages from your old (well, current) R installation, into your newer R installation.
You can then erase all of the packages in your old R installation.
After your packages are moved (and the old ones possibly erased), you will get the option to update all of your packages in the new version of R.
Lastely – you can open the new Rgui and close the current session of your old R. (This is a bit buggy in version 0.8, but has been fixed in version 0.8.1)

If you know you wish to upgrade R, and you want the packages moved (not copied, MOVED), you can simply run:

# installing/loading the package:
if(!require(installr)) { install.packages("installr"); require(installr)} #load / install+load installr
 
updateR(F, T, T, F, T, F, T) # install, move, update.package, quit R.

Since the various steps are broken into individual functions, you can also pick and choose what to run using the relevant function:

# installing/loading the package:
if(!require(installr)) { install.packages("installr"); require(installr)} #load / install+load installr
 
# step by step functions:
check.for.updates.R() # tells you if there is a new version of R or not.
install.R() # download and run the latest R installer
copy.packages.between.libraries() # copy your packages to the newest R installation from the one version before it (if ask=T, it will ask you between which two versions to perform the copying)

If you like using the global library system, you can run the following in the old R:

# installing/loading the package:
if(!require(installr)) { install.packages("installr"); require(installr)} #load / install+load installr
 
updateR(F, T, F, F, F, F, T) # only install R (if there is a newer version), and quits it.

And then run the following in the new version of R:

source("http://www.r-statistics.com/wp-content/uploads/2010/04/upgrading-R-on-windows.r.txt")
New.R.RunMe()

The {installr} package also offers functions for installing various other software on Windows. These functions include: install.pandoc (which was mentioned on this blog recently), install.git, install.Rtools, install.MikTeX, install.RStudio, and a general install.URL and install.packages.zip functions. You can see these further explained in the package’s Reference manual.

Feature requests, bug reports – and your help in improving the package

You can see the latest version of installr on github, where you can also submit bug reports (you may also just leave a comment in this post). Since this is my first R package, I might have (e.g: probably have) missed something here or there. So any comment on how to improve my code/documentation/R-fu, will be most welcomed (here or on github).

If this type of coding is fun/easy for you, you can help me improve this package on github. Cool new features I think may be added (by me or others) are:

Add an uninstall.R function – to remove the old R version.
Add more support for upgrading R for people who uses a global library for their packages.
Add support for Linux and Mac! This one I am less likely to do on my own – and would love to see someone else extend my code to other operation systems.
GUI – add a menu based option for running updateR. Something like help->”check for updates” would be great. (p.s: this idea came from Yihui Xie)
add even more install.software functions. If you have functions for which you’d like to be able to easily install them – just let me know and it could be included in future releases.

Thanks

Final note, I would like to thank the many people who have developed WONDERFUL tools for making R package development possible (and even somewhat fast), on Windows. These include Prof. Brian Ripley and Duncan Murdoch for Rtools, also Uwe Ligges for his work on CRAN, Hadley Wickham for devtools (in general, and for its documentation), Yihui Xie for roxygen2, JJ and others in the RStudio team for RStudio, the people behind git and github, and more. There are probably more things I can thank these people for, and many more people I should thank, but I can’t figure who you are probably (feel free to e-mail me, I appreciate you work even if it is not clear to me your are behind it).

Installing Pandoc from R (on Windows) – using the {installr} package

Tal Galili — Wed, 27 Feb 2013 13:19:01 +0000

The R blogger Rolf Fredheim has recently wrote a great piece called “Reproducible research with R, Knitr, Pandoc and Word“, where he advocates for Pandoc as an essential part of reproducible research workflow in R, in helping to turn documents which are knitted in R into high quality Word for exchanging with our colleagues. It is a great post, with many useful bits of code, and I wanted to supplement it with one missing function: “install.pandoc“.

Update: the install.pandoc function is now part of the {installr} package.

The install.pandoc() function, as the name implies, will recognize the latest version of pandoc, download its installer to your computer, and run it. Making it even simpler for you to experiment with adding pandoc into your workflow.

Here is how to use it:

 
# installing/loading the package:
if(!require(installr)) { install.packages("installr"); require(installr)} #load / install+load installr
 
# Installing pandoc
install.pandoc(use_regex = FALSE)  # The use of use_regex here is due to a change in pandoc download page.  This parameter will not be needed in installr versions after 0.9

Note: the code in this function is based on GERGELY DARÓCZI’s coding in his answer on the Q&A forum StackOverflow, and also G. Grothendieck for the non-XML addition to the function. I thank them both!

{stargazer} package for beautiful LaTeX tables from R statistical models output

Tal Galili — Mon, 14 Jan 2013 08:11:11 +0000

stargazer is a new R package that creates LaTeX code for well-formatted regression tables, with multiple models side-by-side, as well as for summary statistics tables. It can also output the content of data frames directly into LaTeX. Compared to available alternatives, stargazer excels in three regards: its ease of use, the large number of models it supports, and its beautiful aesthetics.

Ease of use

stargazer was designed with the user’s comfort in mind. The learning curve is very mild and all arguments are very intuitive, so that even a beginning user of R or LaTeX can quickly become familiar with the package’s many capabilities. The package is intelligent, and tries to minimize the amount of effort the user has to put into adjusting argument values. If stargazer is given a set of regression model objects, for instance, the package will create a side-by-side regression table. By contrast, if the user feeds it a data frame, stargazer will know that the user is most likely looking for a summary statistics table or – if the summary argument is set to false – wants to output the content of the data frame.

A quick reproducible example shows just how easy stargazer is to use. You can install stargazer from CRAN in the usual way:

install.packages("stargazer")
library(stargazer)

To create a summary statistics table from the ‘attitude’ data frame (which should be available with your default installation of R), simply run the following:

stargazer(attitude)

To output the contents of the first four rows of same data frame, specify the part of the data frame you would like to see, and set the summary option to FALSE:

stargazer(attitude[1:4,], summary=FALSE)

Now, let us try to create a simple regression table with three side-by-side models – two Ordinary Least Squares (OLS) and one probit regression model – using the lm() and glm() functions. We can set the align argument to TRUE, so that coefficients in each column are aligned along the decimal point:

## 2 OLS models
linear.1 <- lm(rating ~ complaints + privileges + learning + raises + critical, data=attitude)
linear.2 <- lm(rating ~ complaints + privileges + learning, data=attitude)
 
## create an indicator dependent variable, and run a probit model
 
attitude$high.rating <- (attitude$rating > 70)
probit.model <- glm(high.rating ~ learning + critical + advance, data=attitude, family = binomial(link = "probit"))
 
stargazer(linear.1, linear.2, probit.model, title="Regression Results", align=TRUE)

Many supported models

stargazer supports objects from the most widely used statistical functions and packages. These include objects from betareg (betareg), coxph (survival), clm (ordinal), clogit (survival), ergm (ergm),gam (mgcv), gee (gee), glm (stats), glmer (lme4), gls (nlme), hurdle (pscl), ivreg (AER), lm (stats), lmer (lme4), lmrob (robustbase), multinom (nnet), nlmer (lme4), plm (plm), pmg (plm), polr (MASS), rlm (MASS), svyglm (survey), survreg (survival), tobit (AER), zeroinfl (pscl), as well as from the implementation of these in Zelig. In addition, stargazer also supports several Zelig models for social network analysis: cloglog.net, gamma.net, probit.net, and logit.net. The number of models and objects can stargazer can accommodate puts it ahead of most of the alternative R-to-LaTeX options. As the development of the package continues, this list will continue expanding to include linear mixed effects models, matching models, as well as new, user-made, or customized statistical models.

Beautiful aesthetics

stargazer is very pleasing to the eye, and allows the user to customize the formatting of the resulting table, including all variable labels. Below is an example of a good-looking, complex regression table created by stargazer:

If you’d like to create tables that look like those from your discipline’s leading journal, stargazer can help you with that as well. You can use the style argument to choose a template of your choice. Economics and management scholars can thus create tables that resemble those published in the American Economic Review, in the Quarterly Journal of Economics, or in Administrative Science Quarterly. Political scientists can avail themselves of templates based on the American Political Science Review, the American Journal of Political Science, and on International Organization. For sociologists and demographers, the American Sociological Review, the American Sociological Reviews and Demography are available.

stargazer, of course, is not the only R package that creates LaTeX code from R statistical output. Other packages with similar capabilities include apsrtable, xtable, memisc, texreg and outreg. Each of these has its own strengths and weaknesses, and users should explore all of them to find the best fit for their needs.

Some extra Q&A with Marek Hlavac (the package author)

What was your motivation for starting the package? (self use, for students, for other people etc.)

As a doctoral student in Political Economy and Government at Harvard University, I saw an urgent need for an easy-to-use tool to create well-formatted stargazer tables. Although other packages were available for this task, none of them combined a large number of supported models, good aesthetics, and simplicity of use in a way that I thought would be ideal.

Why is the “summary” parameter turned on as default for data.frames?

In published papers in the social sciences, summary statistics table are found more commonly than direct print-outs of chunks of data sets. For this reason, I thought the stargazer package should default to the user’s most likely need.

This has been a guest post by Marek Hlavac, the author of the {stargazer} R package for beautiful LaTeX tables from R’s statistical models’ outputs.

Finishing note: You (the readers) are invited to leave a comment to the author, or suggest your own guest post here, by contacting me.

100 most read R posts for 2012 (stats from R-bloggers) – big data, visualization, data manipulation, and other languages

Tal Galili — Tue, 01 Jan 2013 12:03:45 +0000

R-bloggers.com is now three years young. The site is an (unofficial) online journal of the R statistical programming environment, written by bloggers who agreed to contribute their R articles to the site.

Last year, I posted on the top 24 R posts of 2011. In this post I wish to celebrate R-bloggers’ third birthmounth by sharing with you:

Links to the top 100 most read R posts of 2012
Statistics on “how well” R-bloggers did this year
My wishlist for the R community for 2013 (blogging about R, guest posts, and sponsors)

1. Top 100 R posts of 2012

R-bloggers’ success is thanks to the content submitted by the over 400 R bloggers who have joined r-bloggers. The R community currently has around 245 active R bloggers (links to the blogs are clearly visible in the right navigation bar on the R-bloggers homepage). In the past year, these bloggers wrote around 3200 posts about R!

Here is a list of the top visited posts on the site in 2012 (you can see the number of unique visitors in parentheses, while the list is ordered by the number of total page views):

2. Statistics – how well did R-bloggers do in 2012?

Short answer: quite well.

In 2012, R-bloggers has reached around 11,000 regular subscribers (which you can also subscribe to: via RSS, or e-mail), serving the content of about 245 R bloggers. In total, the site was visited around 2.7 million times, by over 1.1 million people. Bellow you can see a few figures comparing the statistics of 2012 with those of 2011 (just click the image to enlarge it):

3. My wishlist for 2013 – about the future of the R blogosphere

Well now, this has been an AMAZING year for the R-project in general, the R community, and consequently also for R-bloggers. Here are a few things I wish for 2013:

Reproducible R blogging – make it to blog from R to WordPress and blogger (via knitr, RStudio, etc.)

The past year has been wonderful regarding progress in making reproducible research with R using Sweave, knitr, RStudio, and many new R packages. For 2013 I wish someone (or some-company, RStudio, cough cough) would take on themselves to make it as easy as possible to do Reproducible R blogging. The seeds are already there, thanks to people like JJ Allaire, Jeffrey Horner, Vicent Marti, and Natacha Porte we now have the markdown package, which combined with Yihui Xie knitr package and the wonderful RStudio (R IDE), allows us all to easily create HTML documents of R analysis. Combine this with something like one of Duncan Temple Lang’s R packages (XMLRPC, RWordPress) and one can imagine the future.

The next step will be to have a “publish to your WordPress/blogger” button right from the RStudio console – allowing for the smoothest R blogging experience one could dream of.

I hope we’ll see this as early as possible in 2013.

Creating online interactive visualization using R

There can never be enough of this really.

So far, I should give props to Markus Gesmann, Diego de Castillo for authoring and maintaining the awesome googleVis R package. This package is great for online publishing of interesting results. For example, see the site StatIL.org – visualizing over 25,000 Time series of Israel’s statistics using html files produced (also) with the googleVis package (example: population of Israel between 1950 to 2011).

The second promising project is Shiny, which Shiny makes it incredibly easy to build interactive web applications with R. Since they intend to release an open source server of Shiny, which can run on Apache, we can expect very interesting developments on that front this year.

More guest posts on R-bloggers

If you have valuable knowledge and insights to share with the R community, the best way I suggest is to start your own free blog on WordPress.com. Create a dedicated R category for your R posts, and ask to join r-bloggers (make sure to read and follow the guidelines mentioned there).

This year I am considering allowing non-bloggers to also take part in the party. The idea is to create a simple form which will allow you to write a guest article which (after review) will go live on r-bloggers (without the need to first start your own blog). If you are interested to submit such a guest article in the future (even if you are not sure exactly what you will write about), please fill out this form with your e-mail. IF I see people are interested, I will go ahead and create this service.

Your help in sharing/linking-to R-bloggers.com

Sharing: If you don’t alreayd know, R-bloggers is not a company. The site is run by just one guy (Tal Galili). There is no marketing team, marketing budget, or any campaign. The only people who know about the site are your and the people YOU will send the link to (through facebook, your personal website, blog, etc.). So if you haven’t already – please help share r-bloggers.com in whatever way you can online.

Subscribe to R-bloggers.com

You can also subscribe to daily updates of new R posts via RSS, or by filling in your e-mail address (I don’t give it to strangers, I promise). You can also join the R-bloggers facebook page, but make sure (once liked) to press the “like” button and mark V by “get notifications” and “show in news feed” (see in the image bellow)

Sponsoring

If you are interested in sponsoring/placing-ads/supporting R-bloggers, then you are welcome to contact me. Currently there is not much place left, but you can still contact me and I will update you once an ad placement is freed up.

Stay in touch

As always, you are welcome to leave a comment on this blog, and/or contact me (keeping in mind it might take me some time to get back to you, but I promise I will).

Happy new year!
Yours truly,
Tal Galili

Generation of E-Learning Exams in R for Moodle, OLAT, etc.

Tal Galili — Thu, 20 Dec 2012 12:11:25 +0000

(Guest post by Achim Zeileis)
Development of the R package exams for automatic generation of (statistical) exams in R started in 2006 and version 1 was published in JSS by Grün and Zeileis (2009). It was based on standalone Sweave exercises, that can be combined into exams, and then rendered into different kinds of PDF output (exams, solutions, self-study materials, etc.). Now, a major revision of the package has been released that extends the capabilities and adds support for learning management systems. It is still based on the same type of
Sweave files for each exercise but can also render them into output formats like HTML (with various options for displaying mathematical content) and XML specifications for online exams in learning management systems such as Moodle or OLAT. Supplementary files such as graphics or data are
handled automatically. Here, I give a brief overview of the new capabilities. A detailed discussion is in the working paper by Zeileis, Umlauf, and Leisch (2012) that is also contained in the package as a vignette.

The basic idea…

…is to have standard standalone Sweave files for each exercise. These typically comprise (1) R code chunks (as usual within <<>>= and @) for random data generation. (2) Question and solution descriptions contained in LaTeX environments of corresponding names, typically with \Sexpr{} for including data
for the question. (3) Metainformation about the exercise type (numeric, multiple choice, …), its correct solution etc. Then many different replications of (collections of) such an exercise can be easily created by repeating the following four steps:
Weaving the exercise (i.e., executing the R code for random data generation and embedding that into the text), reading the resulting LaTeX into R, transforming the LaTeX to another format such as HTML (if necessary), writing the corresponding output files, e.g., in PDF, HTML, XML, …
The package provides convenience interfaces exams2pdf(), exams2html(), exams2moodle(), and exams2qti12() (for OLAT) that directly carry out all of these steps but the package also provides the underlying building blocks such that new interfaces can be easily created.

A simple example

To get a quick impression of the look and feel of the package, consider the following simple example: computation of a t statistic from given mean, variance, and sample size. The corresponding Sweave template tstat.Rnw is provided within the package. After installing the package, you can produce random versions of this exercise in PDF and HTML by running the following code:

library("exams")
exams2pdf("tstat.Rnw")
exams2html("tstat.Rnw")

The former needs a working LaTeX installation and then pops up a PDF with the exercise.
The latter does not need LaTeX but just the R package tth (interfacing Ian Hutchinson’s wonderful TtH
TeX-to-HTML converter) and then pops up the HTML with the exercise in a browser.
MathML support is needed for this version of the HTML exercise, e.g., by using the Firefox browser.

How it works

The tstat.Rnw is a relatively simple Sweave file: It first draws some random data and then uses the textbook formula to compute the correct t statistic. Subsequently, there are two LaTeX environments with the {question} and {solution}, respectively. Finally, some simple metainformation is provided
in LaTeX style (but commented):

<>=
## DATA GENERATION
n n}}}
    = \frac{\Sexpr{Mean} - \Sexpr{mu}}{\sqrt{\frac{\Sexpr{Var}}{\Sexpr{n}}}}
    = \Sexpr{tstat}.
  \end{eqnarray*}
  The absolute value of the $t$~test statistic is thus equal to
  $\Sexpr{format(abs(tstat), nsmall = 3)}$.
\end{solution}

%% META-INFORMATION
%% \extype{num}
%% \exsolution{\Sexpr{format(abs(tstat), nsmall = 3)}}
%% \exname{t statistic}
%% \extol{0.01}

After running Sweave() on it, the resulting LaTeX code has the random numbers filled in and the data-generating code omitted:

\begin{question}
  A machine fills milk into $500$ml packages. It is suspected that the 
  machine is not working correctly and that the amount of milk filled differs 
  from the setpoint $\mu_0 = 500$. A sample of $226$ packages 
  filled by the machine are collected. The sample mean $\bar{y}$ is equal to 
  $517.2$ and the sample variance $s^2_{n-1}$ is equal to $262.56$.

  Test the hypothesis that the amount filled corresponds on average to the 
  setpoint. What is the absolute value of the $t$~test statistic?
\end{question}

\begin{solution}
  The $t$~test statistic is calculated by:
  \begin{eqnarray*}
    t & = & \frac{\bar y - \mu_0}{\sqrt{\frac{s^2_{n-1}}{n}}}
    = \frac{517.2 - 500}{\sqrt{\frac{262.56}{226}}}
    = 15.958.
  \end{eqnarray*}
  The absolute value of the $t$~test statistic is thus equal to
  $15.958$.
\end{solution}

%% META-INFORMATION
%% \extype{num}
%% \exsolution{15.958}
%% \exname{t statistic}
%% \extol{0.01}

The LaTeX code is then read into R, can be transformed into HTML using tth(), embedded into a simple HTML file, and then shown in a browser:

Support for learning management systems

The same HTML code shown in the example above can also be embedded into learning management
systems. These typically have XML-based exchange formats, e.g., Moodle uses its own Moodle XML
while OLAT and several other systems use the international IMS QTI 1.2 standard. The package provides functions exams2moodle() and exams2qti12() which provide output files that can be easily imported into Moodle and OLAT, respectively.
As a result, the tstat.Rnw looks in Moodle like this:

Note that this now has a field for entering the answer (and only after that the correct solution
can optionally be shown).

To generate a Moodle quiz with three random replications for five exercises drawn from collections
of different templates, consider the following example (based on templates provided within the
package):

myexam <- list(
  "boxplots",
  c("confint", "ttest", "tstat"),
  c("anova", "regression"),
  "scatterplot",
  "relfreq")
exams2moodle(myexam, n = 3)

This produces a file moodlequiz.xml that can be imported into the question bank in Moodle. (In this step, the HTML markup for formulas and graphics is not formatted by Moodle but don’t worry: the final quiz will render correctly.) Subsequently, all replications of each exercise can be added as “random” questions into the quiz. Some more configuration of the quiz as a whole can then be done in Moodle but otherwise everything else is ready for the participants of the quiz.

If you want to use this for your own course…

You need an installation of a suitable learning management system but many universities have
support for this anyway.
You need to build your collection of Sweave exercise templates. This is, of course,
the most work. However, the package provides a suite of examples that can be helpful as
starting points.
You need to export the exams/quizzes/assessments from R to the correct format for the
learning management system. Moodle and OLAT are directly supported. But through QTI 1.2
other systems could, in principle, be interfaced as well. We haven’t tested this yet, though.

More details are provided in the working paper and we also provide a support forum on R-Forge if you experience any problems etc.

Comparing Shiny with gWidgetsWWW2.rapache

Tal Galili — Mon, 12 Nov 2012 21:34:48 +0000

(A guest post by John Verzani)

A few days back the RStudio blog announced Shiny, a new product for easily creating interactive web applications (http://www.rstudio.com/shiny/). I wanted to compare this new framework to one I’ve worked on, gWidgetsWWW2.rapache – a version of the gWidgets API for use with Jeffrey Horner’s rapache module for the Apache web server (available at GitHub). The gWidgets API has a similar aim to make it easy for R users to create interactive applications.

I don’t want to worry here about deployment of apps, just the writing side. The shiny package uses websockets to transfer data back and forth from browser to server. Though this may cause issues with wider deployment, the industrious RStudio folks have a hosting program in beta for internet-wide deployment. For local deployment, no problems as far as I know – as long as you avoid older versions of internet explorer.

Now, Shiny seems well suited for applications where the user can parameterize a resulting graphic, so that was the point of comparison. Peter Dalgaard’s tcltk package ships with a classic demo tkdensity.R. I use that for inspiration below. That GUI allows the user a few selections to modify a density plot of a random sample.

gWidgetsWWW2

We’ll start with a gWidgets approach and work our way to the shiny code. A basic GUI can be generated with the following code:

 
width <- 600; height <- 480
 
## The plot commands. Uses GUI objects defined below
make_plot <- function(...) {
  y <- get(svalue(distribution))(svalue(size))
  plot(density(y, bw=svalue(bandwidth)/100, kernel=svalue(kernel)))
  points(y, rep(0, svalue(size) ))
}
 
update_plot <- function(...) {
  f <- tempfile()
  require(canvas)
  canvas(f, width=width, height=height)
  make_plot()
  dev.off()
  svalue(cnv) <- f ## update canvas widget
}
 
## The layout
w <- gwindow("tkdensity example")
gstatusbar("Powered by gWidgetsWWW2.rapache and rapache", cont=w)
 
## containers
bl <- gborderlayout(cont=w)
fl <- gformlayout(cont=bl, where="west")
 
## controls
distribution <- gcombobox(data.frame(value=c("rnorm", "rexp"),
                                     labels=c("Normal", "Exponential"),
                                     stringsAsFactors=FALSE),
                          cont=fl, label="distribution")
kernel <- gcombobox(c("gaussian", "epanechnikov", "rectangular",
             "triangular", "cosine"),
                    cont=fl, label="kernel")
size <- gcombobox(c(5,50, 100, 200, 300),
                  coerce.with="as.numeric",
                  cont=fl, label="size")
bandwidth <- gslider(0.05*100, 2*100, by=0.05*100, value=1*100,
                     width=250,
                     coerce.with="as.numeric",
                     cont=fl, label="bandwidth")
refresh <- gbutton("refresh",
                   cont=fl, label="")
 
cnv <- gcanvas(cont=bl, where="center", width=width, height=height)
 
## connect controls
for(i in list(kernel, size, bandwidth, refresh))
  addHandlerChanged(i, update_plot)

All right, that may look like a lot, but lots of people have used gWidgets to write such GUIs. They aren’t that hard. basically there are three parts:

The definition of the controls and their layout that comprise the
user interface.
The definition of some callback to create the graphic when a control
is updated.
Some means to specify when to invoke this call back.

The gWidgets API is meant to be cross-toolkit, but this is only mostly true. In the above we use a few web-specific features, including the canvas package to provide a JavaScript canvas for drawing R graphics.

Manipulate

This particular example would be really easy were we able to use RStudio’s manipulate package. That is only for RStudio users though. Well, not really. There is a simple map available in the gWidgetsWWW2.rapache package that allows us to easily use that specification. Here is the code:

 
## load in map for manipulate -> gWidgets
source(system.file("demo", "manipulate.R", package="gWidgetsWWW2.rapache"), local=TRUE)
 
w <- gwindow("Manipulate example")
gstatusbar("Powered by gWidgetsWWW2.rapache and rapache", cont=w)
 
manipulate(
           {
             y <- get(distribution)(size)
             plot(density(y, bw=bandwidth/100, kernel=kernel))
             points(y, rep(0, size))
           },
           ##
           distribution=picker("Normal"="rnorm", "Exponential"="rexp"),
           kernel=picker("gaussian", "epanechnikov", "rectangular",
             "triangular", "cosine"),
           size=picker(5, 50, 100, 200, 300),
           bandwidth=slider(0.05 * 100, 2.00 * 100, step=0.05 * 100, initial=1* 100), # integers needed
           button=button("Refresh"),
           ## gWidgetsWWW2.rapache extras
           container=w,       
           device="svg",                # svg or canvas or png
           delay=1000                   # delay to let data load for combo boxes
           )

Okay, I admit. I’m a fan of manipulate and it really is a perfect way to specify an interface as straightforward as this. This one is an example in the gWidgetsWWW2.rapache package and you can see it in action at http://www.math.csi.cuny.edu/gw/ex-manipulate.R (though this redirects to a private server which is often not online). The manipulate interface handles the layout of the controls and the invocation of the callback, all you add is an expression to plot and a specification of
the controls.

Shiny

Now for Shiny. We noted above there are really three pieces to writing this app, with Shiny we only need to worry about two: the definition and layout of the user interface, and the specification of how the resulting graphic is made. For this, we write two functions: ui.R and server.R.

The ui.R files is used to layout the interface. Here is ours:

library(shiny)
 
shinyUI(pageWithSidebar(
 
  # Application title
  headerPanel("tkdensity"),
 
  # Sidebar with a slider input for number of observations
  sidebarPanel(
               selectInput("distribution", "Distribution:", 
                           choices = c("Normal"="rnorm", "Exponential"="rexp")),
               selectInput("kernel", "Kernel:", 
                           choices = c("gaussian", "epanechnikov", "rectangular",
                             "triangular", "cosine")),
               selectInput("size", "Size:", 
                           choices = c(5,50, 100, 200, 300)),
               sliderInput("bandwidth", "bandwidth:", 
                           min = 0.05, 
                           max = 2, 
                           value = 1)
  ),
 
  # Show a plot of the generated distribution
  mainPanel(
    plotOutput("distPlot")
  )
))

There are a few containers (“panels”) and a few controls specified (“inputs”). The basic usage of shiny involves just a handful of each and they are easy to learn. The above code simply makes a header, puts four controls into a side panel and in the main panel readies a place to display our plots. The name distPlot is odd because I stole this basic set up from one of shiny‘s examples and made just minor changes.

The server.R file is not difficult either. We need to make the graphic so that it displays. Here it is:

library(shiny)
 
shinyServer(function(input, output) {
 
  make_plot <- function(distribution, size, bandwidth, kernel) {
    y <- get(distribution)(size)
    plot(density(y, bw=bandwidth, kernel=kernel))
    points(y, rep(0, size))
  }
 
  output$distPlot <- reactivePlot(function() {
    make_plot(input$distribution, input$size, input$bandwidth, input$kernel)
  })
})

Again, this is only slightly modified from an example. The input object is not a list or an environment, but we can extract the values associated with the user interface as though it were. In the above we see the distribution, size, bandwidth and kernel values are passed along to the make_plot call. The only new thing here, besides that, is the magic way we create a plot (passing a function as a call back to reactivePlot and assigning it to distPlot in the output object). The shiny package takes care of the third part of our GUI, using this “reactive” programming style to synchronize various parts of
the interface.

Want to see it at work? Well fire up your copy of shiny and run it from a GitHub gist:

require(shiny)
shiny:::runGist("4009017")

Before you gloss over the above lines, think how neat this is. Just as you can install packages from GitHub within an R session through the devtools package, you can run shiny apps within R from GitHub with the runGist function. This is an awesome feature for deploying interactive web apps for local usage.

I think you’ll see, if you compare, for this task the user experience is nicer with shiny than that of gWidgetsWWW2.rapache. The specification of the GUI is easiest with the manipulate style, but using shiny wasn’t difficult. I personally like very much the use of the “bootstrap” CSS libraries (http://twitter.github.com/bootstrap/), as opposed to the ExtJS CSS (http://www.sencha.com/products/extjs/). The interface looks very modern. (Not only looks, the slider is much better than that in gWidgetsWWW2.rapache, where the underlying control only wants integer values.) Though the gWidgetsWWW2.rapache package might have more controls and more layout options than the basic use of shiny, I can see that there will be a huge number of great shiny apps written. Within its range of use, it is a great choice for easily making R-driven web applications.

About:

John Verzani is the maintainer of the gWidgets set of packages and co-author with Michael Lawrence of the book Programming Graphical User Interfaces in R.