Trouble building R package wtih devtools when it uses RcppArmadillo - rcpp

This is my first stackoverflow question, so please be kind, folks!
I have greatly enjoyed my recently-found power to build R packages using devtools. However, as soon as I try building a package that uses RcppArmadillo, my workflow of running devtools::document(), devtools::check(), and devtools::build() no longer works.
For example, I have a (hopefully pretty minimal + complete) test version of the package I'm trying to develop here: https://github.com/suztolwinskiward/fooR/. fooR contains only one functions, which is a C++ implementation of the rdist.earth function from the fields package.
Running devtools::document("fooR") spits out lots of messages (several alluding to "undefined references" to variables that do not live in my source co that are not interpretable to me, and then fails:
collect2: ld returned 1 exit status
no DLL was created
ERROR: compilation failed for package 'fooR'
* removing 'C:/Users/I53794/AppData/Local/Temp/RtmpWgC8nD/devtools_install_1ea473123086/fooR'
Error: Command failed (1)
One the other hand, when I source the C++ function that depends on RcppArmadillo, it seems to run just fine:
> Rcpp::sourceCpp('./src/rdist_earth_cpp.cpp')
> data('miami')
> data('new_orleans','katrina_path')
> rdist_earth_cpp(katrina_path,new_orleans)
[,1]
[1,] 1042.36073
[2,] 998.96793
[3,] 957.69315
[4,] 917.91486
[5,] 868.07791
[6,] 805.73485
[7,] 763.01476
[8,] 726.10133
[9,] 692.14482
[10,] 670.15133
[11,] 662.23353
[12,] 625.55592
[13,] 601.08682
[14,] 579.73940
[15,] 560.32660
[16,] 539.14192
[17,] 510.15438
[18,] 481.40037
[19,] 442.52322
[20,] 391.96619
[21,] 331.66378
[22,] 271.79088
[23,] 201.24749
[24,] 128.12647
[25,] 56.99198
[26,] 45.80297
[27,] 32.96609
[28,] 81.71237
[29,] 189.31050
[30,] 296.92104
[31,] 406.12593
[32,] 516.08458
[33,] 654.81113
[34,] 808.21670
This leads me to think there's something wrong with the way I'm trying to use RcppArmadillo in my package, but I haven't been able to figure out what. Any advice much appreciated!
P.S. I'm surprised there's no RcppArmadillo tag here....

In addition to the answers of jtilly and the comment from Dirk:
RcppArmadillo.package.skeleton() generates the correct namespace file, but after running roxygen2 via document() the namespace just contains one line
# Generated by roxygen2: do not edit by hand
and the DynLib/export directives are overwritten. To let roxygen2 automatically generate the correct namespace, add a new R file to the R-subdirectory of your package directory containing the following:
#' #useDynLib YourPackageName
#' #importFrom Rcpp evalCpp
#' #exportPattern "^[[:alpha:]]+"
NULL
The name of this file doesn't matter, but YourPackageName.r is usual for this (kind of) "main file".
When running "document()", the following namespace file is generated:
# Generated by roxygen2: do not edit by hand
exportPattern("^[[:alpha:]]+")
importFrom(Rcpp,evalCpp)
useDynLib(YourPackageName)
This is the same namespace which is generated via RcppArmadillo.package.skeleton() by RcppArmadillo 0.6.700.6.0.

Your NAMESPACE file is empty. It should contain something like this:
useDynLib(fooR)
exportPattern("^[[:alpha:]]+")

What eventually worked was to initialize a new package with RcppArmadillo.skeleton.package, move all my previous files therein, document manually, and then check and build with the GUI buttons in RStudio. This feels pretty kludgy and I really liked using roxygen2 much better for documentation-- but as a relative novice in package development with RcppAmadillo dependence, I am just happy for now to have found a way to build successfully!

'devtools::document()' compiles your code, honestly, I'm not sure why. This means that if the compilation fails, the documentation isn't completed. It seems in your case this means you don't get your NAMESPACE written.
Start as Dirk suggests, and add stuff in, but it will have to compile before the docs get done.

Related

Converting a package backend from Rcpp to cpp11

I'm trying to use cpp11 with a fork of fixest (I need to use vendoring). I put a new branch at https://github.com/pachadotdev/fixest/tree/cpp11, then started committing step by step as cpp11's FAQ says, and I fixed all the errors one by one.
Now when I run devtools::install(), the process fails, and now it doesn't say "I didn't like line 10 in means.cpp" or similar.
I'm looking for any help to be able to solve this problem. I listed my doubts at https://github.com/pachadotdev/fixest/issues.
The problem appears to be in this file just to show diff in the file quf.cpp (https://github.com/pachadotdev/fixest/pull/8/files#diff-b9826484794eda5aaf5e9687abf2c971b0b71666b0c44c0b5fa985ce24f3ef26).

How do I write a SCons script with hard-to-predict dynamic sources?

I'm trying to set up a build system involving a code generator. The exact files generated are unknown until after the generator is run, but I'd like to be able to run further build steps by pattern matching (run some program on all files with some extension). Is this possible?
Some of the answers here involving code generation seem to assume that the output is known or a listing of generated files is created. This isn't impossible in my case, but I'd like to avoid it since it makes things more complicated.
https://bitbucket.org/scons/scons/wiki/DynamicSourceGenerator seems to indicate that it's possible to add additional targets during Builder actions, but while I could get the build to run and list the generated files, any build steps introduced don't run.
https://bitbucket.org/scons/scons/wiki/NonDeterministicDependencies uses Scanners to add build steps. I put a glob(...) in a scanner, and it succeeds in detecting the generated files, but the files are inexplicably deleted before it actually runs the dependent step.
Is this use case possible? And why is SCons deleting my generated files?
A toy example
source (the file referenced in SConscript)
An example generator, constructs 3 files (not easily known to the build system) and puts them in the argument folder
echo "echo 1" > $1/gen1.txt
echo "echo 2" > $1/gen2.txt
echo "echo 3" > $1/gen3.txt
SConstruct
Just sets up a variant_dir
SConscript('SConscript', variant_dir='build')
SConscript
The goal is for it to:
"Compile" the generator (in this toy example, just copies a file called 'source' and adds execute permissions
Run the "compiled" generator ('source' is a script that generates files)
Perform some operation on each of those generated files by extension. This example just runs the "compile" copy operation on them (for simplicity).
env = Environment()
env.Append(BUILDERS = {'ExampleCompiler' :
Builder(action=[Copy('$TARGET', '$SOURCE'),
Chmod('$TARGET', 0755)])})
generator = env.ExampleCompiler('generator', 'source')
env.Append(BUILDERS = {'GeneratorRun' :
Builder(action=[Mkdir('$TARGET'),
'$SOURCE $TARGET'])})
generated_dir = env.GeneratorRun(Dir('generated'), generator)
Everything's fine up to here, where all the targets are explicitly known to the build system ahead of time.
Attempting to use this block of code to glob over the generated files causes SCons to delete (!!) the generated files:
for generated in generated_dir[0].glob('*.txt'):
generated_run = env.ExampleCompiler(generated.abspath + '.sh', generated)
Attempting to use an action to update the build tree results in additional actions not being run:
def generated_scanner(target, source, env):
for generated in source[0].glob('*.txt'):
print "scanned " + generated.abspath
generated_target = env.ExampleCompiler(generated.abspath + '.sh', generated)
Alias('TopLevelAlias', generated_target)
env.Append(BUILDERS = {'GeneratedOperation' :
Builder(action=[generated_scanner])})
dummy = env.GeneratedOperation(generated_dir[0].File('#dummy'), generated_dir)
Alias('TopLevelAlias', dummy)
The Alias operations are suggested in above dynamic source generator guide, but don't seem to do anything. The prints do execute and indicate that the action gets run.
Running some build pattern on special file extensions is possible with SCons. For C/CPP files this is the preferred scheme, for example:
env = Environment()
env.Program('main', Glob('*.cpp'))
The main task of SCons, as a build system, is to do the minimum amount of work such that all your targets are up-to-date. This makes things complicated for the use case you've described above, because it's not clear how you can reach a "stable" situation where no generated files are added and all targets are built.
You're probably better off by using a simple Python script directly...I really don't see how using SCons (or any other build system for that matter) is mission-critical in this case.
Edit:
At some point you have to tell SCons about the created files (*.txt in your example above), and for tracking all dependencies properly, the list of *.txt files has to be complete. This the task of the Emitter within SCons, which is responsible for returning the list of resulting target and source files for a Builder call. Note, that these files don't have to exist physically during the "parse" phase of SCons. Please also have a look at my answer to Scons: create late targets , which goes into some more detail.
Once you have a proper Emitter in place (see also https://bitbucket.org/scons/scons/wiki/ToolsForFools , "Using Emitters") you should be able to use the Glob('*.txt') call, which will detect and track your created files automatically.
Finally, on our page "Talks and Slides" ( https://bitbucket.org/scons/scons/wiki/TalksAndSlides ) you can find my talk from the PyCon FR.2014, "Why SCons is Not Slow", which explains shortly how SCons works internally. This might be helpful in understanding this problem better and coming up with a full solution.

Read DICOM images

i am trying to read dicom images without using imageviewer and i came across VtkGdmReader.. when i am trying to execute it, its giving me an error:
code => vtkGdmReader example
error C2039: 'SetInput' : is not a member of 'vtkTexture'
error C2039: 'SetInput' : is not a member of 'vtkPolyDataMapper'
please can any one tell me why am i facing this problem, is this error related to vtk version ? if so then how can i go about it ?
please help me resolving the problem..
As said in the comments, this error is related to the VTK version. SetInput() was removed in VTK 6.
You can either work in VTK 5 or update the code. If you decide to update it, this error gets fixed by replacing SetInput() with either SetInputData() or SetInputConnection() with a few modifications. You should use SetInputConnection() if you want the filters to go through the pipeline.
As an example of fixing the errors you mentioned, you should replace the following lines in the code you provided:
VTKtexture->SetInput(ima); and
VTKplaneMapper->SetInput(VTKplane->GetOutput());
to:
VTKtexture->SetInputConnection(reader->GetOutputPort()); and
VTKplaneMapper->SetInputConnection(VTKplane->GetOutputPort());
In the second modification (VTKplaneMapper), note that we just changed GetOutput() to GetOutputPort(), while in the first one (VTKtexture) we completely changed the argument passed to SetInputConnection(). That happens because data objects (such as ima) no longer have dependency on pipeline objects (such as algorithms and executives). In this case, we give the algorithm that generated that data object as the argument - if you look for it, you can see the line vtkImageData* ima = reader->GetOutput();, you just have to replace GetOutput() by GetOutputPort() as we did in the second modification.
I recommend looking into the VTK Wiki's VTK 6 Migration pages (and guide) for more information on this error and other errors that you might run into.

kraftwagen is not working

I'm trying to do setting a kraftwagen project using http://kraftwagen.org/get-started.html
I can go through the document until Building topic. But after that, when I try to run drush kw-b is it give some error.
$ drush kw-b
Makefile generated from template (/home/dinuka/drush_test/src/tools/build.make.tpl).[success]
Beginning to build /tmp/kwLL5uil. [ok]
drupal-7.22 downloaded. [ok]
file_exists() expects parameter 1 to be string, array given filesystem.inc:185 [warning]
Source directory p is not readable or does not exist. [error]
drush_test copied from p. [ok]
Called command make returned an error.
I can't find drupal-7.22 anywhere. I think it is a problem. I'm a newcomer to Drupal and these things.
Seems to me to be a bug, like this one : https://drupal.org/node/1078318
Maybe you should report it directly (and try to get the patch for it.)
Hope it helps! :)

Runtime error R6034 in embedded Python application

I am working on an application which uses Boost.Python to embed the Python interpreter. This is used to run user-generated "scripts" which interact with the main program.
Unfortunately, one user is reporting runtime error R6034 when he tries to run a script. The main program starts up fine, but I think the problem may be occurring when python27.dll is loaded.
I am using Visual Studio 2005, Python 2.7, and Boost.Python 1.46.1. The problem occurs only on one user's machine. I've dealt with manifest issues before, and managed to resolve them, but in this case I'm at a bit of a loss.
Has anyone else run into a similar problem? Were you able to solve it? How?
The problem was caused by third-party software that had added itself to the path and installed msvcr90.dll in its program folder. In this case, the problem was caused by Intel's iCLS Client.
Here's how to find the problem in similar situations:
Download Process Explorer here.
Start your application and reproduce runtime error R6034.
Start Process Explorer. In the "View" menu go to "Lower Pane View" and choose "DLLs".
In the top pane, locate your application and click on it. The bottom pane should show a list of DLLS loaded for your application.
Locate "msvcr??.dll" in the list. There should be several. Look for the one that is not in the "winsxs" folder, and make a note of it.
Now, check the path just before your application runs. If it includes the folder you noted in step 5, you've probably found the culprit.
How to fix the problem? You'll have to remove the offending entry from the path before running your program. In my case, I don't need anything else in the path, so I wrote a simple batch file that looks like this:
path=
myprogram.exe
That's it. The batch file simply clears the path before my program runs, so that the conflicting runtime DLL is not found.
This post elaborates on #Micheal Cooper and #frmdstryr and gives a better alternative than my earlier answer.
You can put the following in front of a python script to purge the problematic entries.
import os, re
path = os.environ['PATH'].split(';')
def is_problem(folder):
try:
for item in os.listdir(folder):
if re.match(r'msvcr\d\d\.dll', item):
return True
except:
pass
return False
path = [folder for folder in path if not is_problem(folder)]
os.environ['PATH'] = ';'.join(path)
For the vim with YouCompleteMe case, you can put the following at the top of your vimrc:
python << EOF
import os, re
path = os.environ['PATH'].split(';')
def is_problem(folder):
try:
for item in os.listdir(folder):
if re.match(r'msvcr\d\d\.dll', item):
return True
except:
pass
return False
path = [folder for folder in path if not is_problem(folder)]
os.environ['PATH'] = ';'.join(path)
EOF
A more general solution is:
import os
os.environ['path'] = ";".join(
[path for path in os.environ['path'].split(";")
if "msvcr90.dll" not in map((lambda x:x.lower()), os.listdir(path))])
(I had the same problem with VanDyke SecureCRT)
(This might be better as a comment than a full answer, but my dusty SO acct. doesn't yet have enough rep for that.)
Like the OP I was also using an embedded Python 2.7 and some other native assemblies.
Complicating this nicely was the fact that my application was a med-large .Net solution running on top of 64-Bit IIS Express (VS2013).
I tried Dependency Walker (great program, but too out of date to help with this), and Process Monitor (ProcMon -- which probably did find some hints, but even though I was using filters the problems were buried in thousands of unrelated operations, better filters may have helped).
However, MANY THANKS to Michael Cooper! Your steps and Process Explorer (procexp) got me quickly to a solution that had been dodging me all day.
I can add a couple of notes to Michael's excellent post.
I ignored (i.e. left unchanged) not just the \WinSxS\... folder but also the \System32\... folder.
Ultimately I found msvcr90.dll being pulled in from:
C:\Program Files (x86)\Intel\OpenCL SDK\2.0\bin\x64
Going through my Path I found the above and another, similar directory which seemed to contain 32-bit versions. I removed both of these, restarted and... STILL had the problem.
So, I followed Michael's steps once more, and, discovered another msvcr90.dll was now being loaded from:
C:\Program Files\Intel\iCLS Client\
Going through my Path again, I found the above and an (x86) version of this directory as well. So, I removed both of those, applied the changes, restarted VS2013 and...
No more R6034 Error!
I can't help but feel frustrated with Intel for doing this. I had actually found elsewhere online a tip about removing iCLS Client from the Path. I tried that, but the symptom was the same, so, I thought that wasn't the problem. Sadly iCLS Client and OpenCL SDK were tag-teaming my iisexpress. If I was lucky enough to remove either one, the R6034 error remained. I had to excise both of them in order to cure the problem.
Thanks again to Michael Cooper and everyone else for your help!
Using Michael's answer above, I was able to resolve this without a bat file by adding:
import os
# Remove CLS Client from system path
if os.environ['PATH'].find("iCLS Client")>=0:
os.environ['PATH'] = "".join([it for it in os.environ['PATH'].split(";") if not it.find("iCLS Client")>0])
to the main python file of the application. It just makes sure system path didn't include the paths that were causing the issue before the libraries that loaded the dll's were imported.
Thanks!
This post elaborates on #Micheal Cooper and #frmdstryr.
Once you identified the problematic PATH entries, you can put the following in front of a
python script, assuming here that iCLS Client and CMake are problematic.
import os
for forbidden_substring in ['iCLS Client', 'CMake']:
os.environ['PATH'] = ';'.join([item for item in os.environ['PATH'].split(';')
if not item.lower().find(forbidden_substring.lower()) >= 0])
Concerning the vim with YouCompleteMe case, you can put the following at the top of your vimrc:
python << EOF
import os
for forbidden_substring in ['iCLS Client', 'CMake']:
os.environ['PATH'] = ';'.join([item for item in os.environ['PATH'].split(';')
if not item.lower().find(forbidden_substring.lower()) >= 0])
EOF
If none of these solutions is applicable for you, you can try to remove the problem causing
entries from you PATH manually, but you want to make sure you don't break anything else on your
system that depends on these PATH entries. So, for instance, for CMake you could try to remove
its PATH entry, and only put a symlink (or the like) pointing to the cmake.exe binary into some
other directory that is in your PATH, to make sure cmake is still runnable from anywhere.
Thanks for the solution. I just little modified this sample code as the path variable in my system contains the string "ICLS CLIENT" instead of "iCLS Client"
import os
# print os.environ['PATH']
# Remove CLS Client from system path
if os.environ['PATH'].find("iCLS Client") >= 0 or os.environ['PATH'].find("ICLS CLIENT") >= 0:
os.environ['PATH'] = "".join([it for it in os.environ['PATH'].split(";") if not (it.find("iCLS Client")>0 or it.find("ICLS CLIENT")>0)])
I also had the same problem with embedding Python27.dll from a C-program using the Universal-CRT.
A <PYTHON_ROOT>\msvcr90.dll was the offender. And <PYTHON_ROOT> is off-course in my PATH. AFAICS the only users of msvcr90.dll are the PyWin32 modules
<PYTHON_ROOT>\lib\site-packages\win32\win32*.pyd.
The fix was just move <PYTHON_ROOT>\msvcr90.dll to that directory.
PS. PyWin32 still has this as an issue 7 years later!
In my case the rebuilding of linked libraries and the main project with similar "Runtime execution libraries" project setting helped. Hope that will be usefull for anybody.
In my case, I realised the problem was coming when, after compiling the app into an exe file, I would rename that file. So leaving the original name of the exe file doesn't show the error.
The discussion on this page involves doing things way far advanced above me. (I don't code.) Nevertheless, I ran Process Explorer as the recommended diagnostic. I found that another program uses and needs msvcr90.dll in it's program folder. Not understanding anything else being discussed here, as a wild guess I temporarily moved the dll to a neighboring program folder.
Problem solved. End of Runtime error message.
(I moved the dll back when I was finished with the program generating the error message.)
Thank you all for your help and ideas.
Check any library having user specified path by Process Explorer. It is not necessary must be msvcr??.dll
I solved same problem except I run Python 3. Present solutions not helped because they not indicate unusual paths of msvcr90.dll. I debug code step by step inside till error dialog appears after rows (called when my code was importing PyTables module):
import ctypes
ctypes.cdll.LoadLibrary('libbz2.dll')
Then Process Explorer helps to find path to old libbz2.dll caused the problem (steps 3, 4 of #Micheal Cooper algorithm)
Adding this answer for who is still looking for a solution. ESRI released a patch for this error. Just download the patch from their website (no login required), install it and it will solve the problem. I downloaded the patch for 10.4.1 but there are maybe patches for other versions also.

Resources