Parallel Multicore Processing with R (on Windows)

Parallel Processing backend for R under windows – installation tips and some examples.

This post offers simple example and installation tips for “doSMP” the new Parallel Processing backend package for R under windows.
* * *

Update:
The required packages are not yet now available on CRAN, but until they will get online, you can download them from here:
REvolution foreach windows bundle
(Simply unzip the folders inside your R library folder)

* * *

Recently, REvolution blog announced the release of “doSMP”, an R package which offers support for symmetric multicore processing (SMP) on Windows.
This means you can now speed up loops in R code running iterations in parallel on a multi-core or multi-processor machine, thus offering windows users what was until recently available for only Linux/Mac users through the doMC package.

Installation

For now, doSMP is not available on CRAN, so in order to get it you will need to download the REvolution R distribution “R Community 3.2” (they will ask you to supply your e-mail, but I trust REvolution won’t do anything too bad with it…)
If you already have R installed, and want to keep using it (and not the REvolution distribution, as was the case with me), you can navigate to the library folder inside the REvolution distribution it, and copy all the folders (package folders) from there to the library folder in your own R installation.

If you are using R 2.11.0, you will also need to download (and install) the revoIPC package from here:
revoIPC package – download link (required for running doSMP on windows)
(Thanks to Tao Shi for making this available!)

Usage

Once you got the folders in place, you can then load the packages and do something like this:

require(doSMP)
workers <- startWorkers(2) # My computer has 2 cores
registerDoSMP(workers)

# create a function to run in each itteration of the loop
check <-function(n) {
	for(i in 1:1000)
	{
		sme <- matrix(rnorm(100), 10,10)
		solve(sme)
	}
}


times <- 10	# times to run the loop

# comparing the running time for each loop
system.time(x <- foreach(j=1:times ) %dopar% check(j))  #  2.56 seconds  (notice that the first run would be slower, because of R's lazy loading)
system.time(for(j in 1:times ) x <- check(j))  #  4.82 seconds

# stop workers
stopWorkers(workers)

Points to notice:

  • You will only benefit from the parallelism if the body of the loop is performing time-consuming operations. Otherwise, R serial loops will be faster
  • Notice that on the first run, the foreach loop could be slow because of R's lazy loading of functions.
  • I am using startWorkers(2) because my computer has two cores, if your computer has more (for example 4) use more.
  • Lastly - if you want more examples on usage, look at the "ParallelR Lite User's Guide", included with REvolution R Community 3.2 installation in the "doc" folder

Updates

(15.5.10) :
The new R version (2.11.0) doesn't work with doSMP, and will return you with the following error:

Loading required package: revoIPC
Error: package 'revoIPC' was built for i386-pc-intel32


So far, a solution is not found, except using REvolution R distribution, or using R 2.10
A thread on the subject was started recently to report the problem. Updates will be given in case someone would come up with better solutions.

Thanks to Tao Shi, there is now a solution to the problem. You'll need to download the revoIPC package from here:
revoIPC package - download link (required for running doSMP on windows)
Install the package on your R distribution, and follow all of the other steps detailed earlier in this post. It will now work fine on R 2.11.0


Update 2: Notice that I added, in the beginning of the post, a download link to all the packages required for running parallel foreach with R 2.11.0 on windows. (That is until they will be uploaded to CRAN)

Update 3 (04.03.2011): doSMP is now officially on CRAN!

0 thoughts on “Parallel Multicore Processing with R (on Windows)”

      1. Are there any new updates for 64 bit doSMP. I am new to parallel computing but I could make it to work on 32 bit. I installed on 64 bit without errors but the problem is when I start running, its not responding (even for stopWorkers function).
        My machine is Windows 7 with 4 cores and 8 logical processors (intel(R) Core(TM) i7 CPU).

  1. Great post – I’m very excited about parallel computing. So far I’m doing things on a single PC, but maybe someday I will be able to set up a cloud (as soon as sky falls down and my faculty starts using linux). It would be also nice if something similar would be done for Windows (sensu the SETI project), with a single application that would turn host’s PC into a node that would share resources for R over the internet or local network.

    1. Thank you for the kind words Romunov.
      I hear your pain regarding the cloud in your local system.

      My University uses Condor, which I still haven’t fully figured out how to use for my needs (I did once, but never got the motivation to systematically do it again, and more scalable for my work).

      In the next few months I’ll probably write a bit more on options to do this using Amazon cloud.
      But only after I have learned it a bit more myself 🙂

      Cheers,
      Tal

      1. has anyone had any luck with this? parallel processing in windows 7 with a 64 bit build would make r much more powerful for me..

  2. Has anyone bothered to find out how to “kill” all SMP processes? I was sloppy and didn’t clean after myself before powering down R and now I’m without the primary object I used to registerDoSMP. Is there a way to “reboot” all the cores?

  3. Hi Tal,
    Thanks for the helpful instructions and example code. It seems that doSMP is now available from R-Forge (https://r-forge.r-project.org/projects/dosmp/), and can be installed on native R with the install.packages() command.

    install.packages(“doSMP”, repos=”http://R-Forge.R-project.org”)

    It works fine on R-2.12.0 on Windows 7 64bit.

    1. Failed with error: ‘package ‘revoIPC’ is not installed for ‘arch=i386’’

      win7 64 bit

      It still can not work for win7 64 bit system. Is that right?

  4. I read your parallel multicore processing in R post and I’d like to try it out.  Unfortunately, I haven’t been able to install the doSMP package.  It isn’t listed on CRAN (I found a .tar.gz file in the archives, but I’m not sure how to install it).

    I’d appreciate any help that you can give me.  Thanks!

    1. Hi Jon,
      It is (sadly) the case that doSMP was removed from CRAN due to bugs in the package (when used with the newest R).  You will either have to wait for a newer version, use an older version of R (not suggested), or try one of the other options (for example, the {cluster} package in R).

      With regards,
      Tal

  5. The doSMP package is no longer actively supported in R 2.15 – it is now a deprecated package.
     The doParallel package has taken over the functionality of doSMP.

    Stephen Weller
    Revolution Analytics Technical Support Engineer

Leave a Reply to Tal GaliliCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.