[DEPRECATED] Building OpenBLAS-based Rblas.dll for 64-bit R for Windows: R ≥ 3.3.0

Alternating Sign Matrices of Size 3

Updated: July 19, 2016

With the release of R 3.3.0 and Rtools34, these instructions needed yet another update. The most important difference is that Rtools34 is based on gcc 4.9, and can be used to compile OpenBLAS as well as R. MSYS2 is still necessary, but there is no more need for a separate MinGW64 GCC installation.

Compiling OpenBLAS

  1. Download OpenBLAS (hereafter referred to as OPB) source code. The version used for this example is 0.2.18.
  2. Download a 64-bit MSYS environment. I used 64-bit MSYS2 date-stamped 201602605.
  3. Install MSYS2.
    • Be careful to follow the directions in Section III of the installation instructions as there are some unique quirks to an installation. Specifically, the installed pacman (package manager) is 5.0.0.6348, which means that the update-core method must be used. This also may require multiple forced closings and restarts of MSYS. The batch files used to start MSYS may also change. What fun!
    • Once the instructions in Section III are completed, also install make using pacman -S make from within the MSYS2 shell.
    • Once that is complete, edit the msys2.ini and mingw64.ini files to uncomment the line MSYS2_PATH_TYPE=inherit in both.
    • For some reason, there is a conflict between the key authentication of the MSYS2 package manager and GnupG4win, so you will have to resort to uninstalling GnuPG2 and sticking with GnuPG1 for the time being.
    • Once MSYS2 is installed, there will be a directory called home and underneath it one that is probably your log-in name. This second subdirectory is your actual “home”. Create a subdirectory underneath here. This will be referred to as OPB_HOME.
  4. Expand/unzip the OpenBLAS source code into OPB_HOME.
  5. Edit your path to remove the reference to “C:\Rtools\bin” and replace it with “X\msys64\usr\bin;“, substituting the appropriate drive letters and subdirectories as necessary X.
  6. Open a command shell and type PATH ensuring that the MSYS2 paths is there and the Rtools\bin path is not. In my experience, not calling PATH runs the risk of the change not “registering” somehow.
  7. Open the file Makefile.rule in the OPB source code directory with a text editor (I use Notepad++) and make the follow changes. Some of them require uncommenting, which means removing the # at the beginning of the line:
    1. Set FC = gfortran (Line 25 in v0.2.18)
    2. Set BINARY=64 (Line 46 in v0.2.18)
    3. Set USE_THREAD = 1 (Line 52 in v0.2.18). Personal experimentation indicates that in most cases multi-thread outperforms single-threaded (eigen on non-symmetric matrices is an exception).
    4. Set NUM_THREADS = 4 (Line 60 in v0.2.18). Personal experimentation indicates that setting the number of threads equal to half the number of available threads—equal to the number of physical processors factoring in hyperthreading—leads to the best results.
    5. Set NO_SHARED = 1 (Line 66 in v0.2.18)
    6. Set NO_CBLAS = 1 (Line 69 in v0.2.18)
    7. Set NO_LAPACK = 1 (Line 77 in v0.2.13)
    8. Set and comment out # NO_WARMUP = 0 (Line 96 in v0.2.18)
      • You want OpenBLAS to find more optimal settings!
    9. Set NO_PARALLEL_MAKE = 1 (Line 112 in v0.2.18). In my experience the GCC-based compilers for Windows do not handle parallel makes well.
    10. Set GEMM_MULTITHREAD_THRESHOLD = 4 (Line 145 in v0.2.18)
      • This speeds up computation by using the single thread version for small matrices where the multithread overhead outweighs the speed gain.
  8. Run msys2.exe file to open an instance of MSYS2.
  9. Navigate to the directory in which the source code and makefiles sit, and type “make
  10. After a while and lots of fast-scrolling text, the compilation should finish successfully. If it does not, and you followed all of the above steps, you’ll probably have a few tens of hours of online searching to do to try and figure out what went wrong, I’m afraid.

At this point, in the OPB folder within the MSYS install, there will be a file with the name libopenblas_sandybridgep-r0.2.18.a where sandybridge may be something different depending on your processor (Nehalem, Prescott, etc.). This is the compiled static BLAS which will be fed into the R compilation.

Compiling R

I am not going to go through an exhaustive step-by-step explanation of how to compile R, the manual does a much better job than I could. However, there are some steps that need to be taken to allow R to use the BLAS created above. If you want to try link-time optimization for base R, I will probably address that in a future post.

  1. Download and install the latest version of Rtools, including any additional files, if you haven’t already.
    • During the install it is very important to allow Rtools to add its binaries to the PATH, and to delete the MSYS2 entry from the PATH. Otherwise, there will be two versions of GCC related tools in the path, and mayhem may ensue.
    • If Rtools is already installed, make sure to replace the MSYS bin directory in the path with the Rtools one.
  2. Re-read the R installation guide, especially if you have compiled R from source for Windows before. There are a lot of changes, such as the need for perl, texi2any, the localsoft directory, ICU, curl, etc.
  3. Expand the R source code into R_HOME as described in the manual, including any extra needed files such as the Tcl installed by Rtools, cairodevices, etc.
  4. I strongly recommend having a proper install of MIKTEX, Inno, and qpdf as described in the installation manual. Even if you are not interested in creating a full-fledged installer, creating it will help minimize the potential errors found in the checking stage, and thus will make it easier to see if the BLAS is causing a failure.
  5. Open R_HOME/src/gnuwin32/MkRules.dist and uncomment and set the following changes, if necessary:
    1. Set USE_ATLAS=YES (Line 30 in R-patched_2016-07-18)
    2. Set ATLAS_PATH (Line 31 in R-patched_2016-07-18) to wherever you have the libopenblas file. For example, if you moved the static library to F:\R\BLAS then set ATLAS_PATH=F:/R/BLAS. Note the forward slashes (/) instead of backslashes (\). This applies to any path in the makefile, such as Localsoft, ICU, cairo, Inno, qpdf, etc.
    3. Set BINPREF64 (Line 59 in R-patched_2016-07-18) to wherever you have Rtools installed. This needs the trailing /! So, if Rtools is installed under F:\, you would set BINPREF64 = F:/Rtools/mingw_64/bin/
    4. Set WIN = 64 (Line 73 in R-patched_2016-07-18)
    5. Uncomment and set the appropriate paths for Localsoft, ICU, curl, Cairo, Inno, qpdf, etc. as necessary
      • Perl is the exception. There /path/to/perl has to terminate in perl.exe. It isn’t just the path, but the full name of the executable.
    6. I would recommend researching the particular machine for which you are compiling, and setting the EOPTS with the appropriate march or at least mtune flag. Using march will make the installation specific to the architecture on which it is built, but the purpose of this exercise is to try and wring as much speed out of the machine on which you are working, is it not? Personally, I use EOPTS = -march=native -pipe. For more information for Rtool‘s version’s GCC settings, see the entries in its documentation.
  6. Navigate to and open R_HOME\src\extra\blas\Makefile.win and change line 15 from:-L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -lf77blas -latlas to -L../../../$(IMPDIR) -lR -L"$(ATLAS_PATH)" -lopenblas_sandybridgep-r0.2.18 where -lopenblas_sandybridgep-r0.2.18 is the name of the .a file but starting with “-l” and not “lib”.
  7. If you are using Rtools34 but compiling R-3.3.x (as I am) you need to make one more change. Navigate to R_HOME/src/gnuwin32/fixed/etc and edit the Makeconf so that TCL_VERSION = 86 (Line 61 in R-patched_2016-07-18). This will not be necessary for R-devel or R-3.4.x.
  8. Compile the base R by running make distribution. If there is a significant problem with the BLAS and/or its path, this process may fail. The process may need to be restarted a few times, especially if you get a file not found error. Just run make distribution again.
  9. At this point, run make check-all, preferably piping it into a text file (such as make check-all > CheckAll.txt). This can take a long time (20 minutes on a fast machine; over an hour on a slow one). If there is an issue with the BLAS, the check will fail somewhere, although failing the internet check is not a problem. [As of 2015-03-09, there is a bug causing standard internet.dll, but internet2 (and libcurl) work.] I pay careful attention to the times when the check is working on the base, stats, Matrix, and mgcv packages, as most of the time, failures occur in these routines.
  10. When completed, check the R_HOME\tests folder and any subdirectory for .Rout.fail files. If you created a log file, then search it for the word ERROR. Many of them will be fine (statements that errors have not been found or the internet failure) but a true error should be found if it exists.
  11. If you find a .Rout.fail you may want to try and re-create the error in your newly-compiled R. For example, the only error I get is in example 6, where the call is:

    and the actual error between iM6$varStruct["Female",] and the constants is 5.720938e-05 > 5e-5. To me this is close enough. Now if the errors were of the order 1e-3 or so, that may be different. It’s your call.

If the check completes and there are no fail files or no significant differences, then congratulations! The Rblas.dll file in R_HOME\bin\x64 is a BLAS tuned to your specific machine.

If anyone does successfully complete this process, I would be very interested to know how much, if at all, the matrix/linear algebra have sped up. Good Luck!


15 Responses

  1. Distilled News | Data Analytics & R
    Distilled News | Data Analytics & R March 17, 2015 at 4:52 AM |

    […] Building OpenBLAS-based Rblas.dll for 64-bit R for Windows: R ≥ 3.2.0 With the release of R 3.2.0 going to significantly change how to compile R from source for Windows, it is a good time to update the steps of how to build an OpenBLAS-based Rblas. Of course, the previous caveats still apply: I am not a programmer nor do I play one on the internet so I probably cannot debug any issue anyone may have. Use the BLAS at your own risk and reward. If you use it win the Nobel prize, the credit is all yours (and that of R-core and the OpenBLAS team). If your computer explodes into a pile of slag, opening a dimensional portal to the netherworld and starting the zombie apocalypse, you’re on your own—although I’d appreciate a quick e-mail telling me to run for the hills. The instructions generally assume that you are familiar with compiling programs on Windows, especially versions of R. If not, it is crucial to read and understand the R Installation and Administration manual, especially the portions on compiling for Windows. Even though I have done so scores of times over the years, I still keep the manual open when I am performing a new compilation. […]


  2. R with GotoBLAS on Windows 10 | Matt Moores

    […] Clearly, this is far from the desired outcome in switching BLAS implementations. As described by Avraham Adler, the alternative is to install MSYS2 & MINGW64, then compile both OpenBLAS & R from source. […]


  3. Updated R & BLAS timings
    Updated R & BLAS timings March 31, 2016 at 12:51 AM |

    […] it’s actually easier now to compile OpenBLAS for R and in R on Windows, my instructions are a bit dated, and so I’ll have to update those eventually. I have considered hosting some […]


  4. Updated R & BLAS Timings – Mubashir Qasim

    […] it’s actually easier now to compile OpenBLAS for R and in R on Windows, my instructions are a bit dated, and so I’ll have to update those eventually. I have considered hosting some […]


  5. Updated OpenBLAS instructions for R-3.3+ and Rtools34

    […] just updated the instructions for building a 64-bit OpenBLAS-based Rblas.dll for Windows to reflect changes to R 3.3+ and […]


  6. James Chang
    James Chang September 18, 2017 at 1:11 AM |

    Both OpenBLAS and R have newer version, coud you please kindly updat this atrical?
    thanks in advance!

    Reply
    1. Avraham
      Avraham September 18, 2017 at 3:46 PM |

      Hello, James. Nothing in the process has changed due to R 3.4.1 and OPB 2.20—with the exception of some line numbers. It’s actually easier now that the TCL version is up to date and internet2 is deprecated. I’ve built R/Rblas a number of times, including 3.41patched & OPB2.20 with no issues. Are you having specific issues?

      It may change when R-devel is released (as R3.5?) as Luke Tierney recently updated R-devel to change something as regards compilation. When the new major version is released, I’ll try to clean up the above. Thank you.

      Reply
      1. Avraham
        Avraham September 18, 2017 at 3:50 PM |

        Whoops, there is one issue I forgot. I happen to be compiling one right now which reminded me. In the MSYS2 portion, where the BLAS library is being compiled, it invariably stops mid-process with an error. Just restart ‘make’. It finishes (or at least has for me in the past) and the resulting static library compiles seamlessly into Rblas.dll and (at least for me) passes make check-devel.

        Reply
  7. Ricky
    Ricky December 8, 2017 at 2:48 PM |

    Is that possible to build OpenBLAS for 32bit R? for some legacy reasons we’d like to have both 32 and 64 bit version of R

    Reply
    1. Avraham
      Avraham January 24, 2018 at 12:46 PM |

      You would need to use the 32bit version of Rtools, but I don’t see why it wouldnt work if you make the proper changes in the OpenBLAS and R make files.

      Reply
  8. Eduardo Alvim
    Eduardo Alvim May 9, 2018 at 7:29 PM |

    Excelent Post! I just compiled the R 3.5.0 here, after a lot of Work in compiling R (I tried to use Cygwin, but It didn’t helped much), but the steps that you described here saved a lot of my time (and also a help from here too, https://github.com/thequackdaddy/R-OpenBLAS, I just did’nt give up the compilation because of these helps). I wasted lots of time in figuring out the paths from CURL and CAIRO… But your description about the procedures for compiling the OpenBLAS, it worked seanlesly, it was a really good tip, actually the better tip of R that I have read. If you want, I can send you my “MkRules.local” file to help the newcomers (like me) to understand better the R compilation. In the end, the command “make check-all” returned an error in the file ”reg-tests-1c.Rout”. Then I executed the same scripts in my compiled R 3.5.0 and in the installed MS R Open 3.4.4, and the results were the same both versions of R… Thank You!

    Reply
  9. Tom Wenseleers
    Tom Wenseleers September 3, 2019 at 2:59 PM |

    I found that on Windows the fastest mutlithreaded matrix operations you get with Intel MKL, (faster than OpenBlas) which you can get just by installing Microsoft Open R (https://mran.microsoft.com/open). Also, you can just copy the files libiomp5md.dll, Rblas.dll and Rlapack.dll from that distribution to directory Program Files/R/R-XX.XX.XX/bin/x64 (corresponding to any R version) to get multithreaded BLAS. That’s a whole lot easier than having to recompile the whole of R… 🙂

    ith their Intel MKL counterpart that you will have multithreaded matrix operations. For me this usually results in a ca 10 fold increase in the speed of matrix operations, so it’s well worth it! Let me know if this works for you! Aside from via the dropbox link given above you can get those files also if you just install Microsoft Open R, and move the files from there to your latest R installation (just remember to also tell RStudio to use your regular version of R instead of Microsoft Open R).

    if in directory Program Files/R/R-XX.XX.XX/bin/x64 (e.g. the latest R distribution) you can just overwrite files libiomp5md.dll, Rblas.dll and Rlapack.dll with their Intel MKL counterpart that you will have multithreaded matrix operations. For me this usually results in a ca 10 fold increase in the speed of matrix operations, so it’s well worth it! Let me know if this works for you! Aside from via the dropbox link given above you can get those files also if you just install Microsoft Open R, and move the files from there to your latest R installation (just remember to also tell RStudio to use your regular version of R instead of Microsoft Open R).

    Reply
  10. Tom Wenseleers
    Tom Wenseleers December 26, 2019 at 7:52 AM |

    Easier solution than having to recompile R against the Intel MKL libraries on Windows is just to

    1. Install the latest version of R from https://cran.r-project.org/bin/windows/base/, i.e. currently R 3.6.2
    2. Install Microsoft R Open from https://mran.microsoft.com/download, which comes with the outdated R version 3.5.3 but also with the Intel MKL multithreaded BLAS libraries
    3. copy files libiomp5md.dll, Rblas.dll and Rlapack.dll from C:\Program Files\Microsoft\R Open\R-3.5.3\bin\x64 to C:\Program Files\R\R-3.6.2\bin\x64 (you can back up your existing default non-hyperthreaded Rblas.dll and Rlapack.dll files first if you like)
    4. copy Microsoft R Open libraries/packages MicrosoftR, RevoIOQ, RevoMods, RevoUtils and RevoUtilsMath from C:\Program Files\Microsoft\R Open\R-3.5.5\library to your default package directory, e.g. C:\Documents\R\win-library\3.6
    5. copy files Rprofile.site and Renviron.site from directory C:\Program Files\Microsoft\R Open\R-3.5.5\etc to C:\Progral Files\R\R-3.6.2\etc
    6. replace line 24 in file Rprofile.site options(repos=r) with options(repos="https://cran.rstudio.com") (or your favourite CRAN repository) to make sure that it will install the latest CRAN packages as opposed to the outdated mran.microsoft.com mirror that has outdated package versions, frozen at the 15th of April 2019. Also comment out lines 153, 154 and 155 with a #

    Then restart RStudio to check that it works, with small SVD benchmark on my Intel Core i7-4700HQ 2.4GHz 4 core/8 thread laptop:

    getMKLthreads()
    4

    # Singular Value Decomposition
    m <- 10000
    n <- 2000
    A <- matrix (runif (m*n),m,n)
    system.time (S 8 fold speed increase here!

    Reply
  11. Building R 4+ for Windows with OpenBLAS | Strange Attractors

    […] with OpenBLAS. The release of R 4.0 includes significant changes to the Windows build system from prior versions—for the better! Before anything, we all owe Jeroen Ooms significant gratitude for the many hours […]


  12. Building R 4+ for Windows with OpenBLAS – Data Science Austria

    […] with OpenBLAS. The release of R 4.0 includes significant changes to the Windows build system from prior versions—for the better! Before anything, we all owe Jeroen Ooms significant gratitude for the many hours […]


Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.