Pytorch Tensor::data_ptr<long long>() not working on Linux - pytorch

I cannot link my program to pytorch under Linux, get the following error:
/tmp/ccbgkLx2.o: In function `long long* at::Tensor::data<long long>() const':
test.cpp:(.text._ZNK2at6Tensor4dataIxEEPT_v[_ZNK2at6Tensor4dataIxEEPT_v]+0x14): undefined reference to `long long* at::Tensor::data_ptr<long long>() const'
I am building a very simple minimal example:
#include "torch/script.h"
#include <iostream>
int main() {
auto options = torch::TensorOptions().dtype(torch::kInt64);
torch::NoGradGuard no_grad;
auto T = torch::zeros(20, options).view({ 10, 2 });
long long *data = (long long *)T.data<long long>();
data[0] = 1;
return 0;
}
The command used to build it:
g++ -w -std=c++17 -o test-torch test.cpp -D_GLIBCXX_USE_CXX11_ABI=1 -Wl,--whole-archive -ldl -lpthread -Wl,--no-whole-archive -I../libtorch/include -L../libtorch/lib -ltorch -ltorch_cpu -lc10 -Wl,-rpath,../libtorch/lib
Pytorch has been downloaded from the link https://download.pytorch.org/libtorch/cpu/libtorch-cxx11-abi-shared-with-deps-1.7.0%2Bcpu.zip and unzipped (so I have the libtorch folder next to the folder with test.cpp).
Any ideas how to solve this problem? Same program works just fine under Visual C++.
P.S. I know pytorch is kind of designed for cmake, but I have zero experience with cmake and no desire to write a cmake-based build system for my app. Also, the examples they give are seemingly supposed to only work if pytorch is "installed" in the system. So I cannot just download the .zip with libs? And if I "install" it (e.g. from sources or in whatever other way) on an AVX512 system, will the binary I link to it and distribute to end-users work on non-AVX512? The documentation is completely incomprehensible for newbies.
UPDATE: I tried to do this via CMake following the tutorial https://pytorch.org/cppdocs/installing.html and got exactly the same error. Specifically, I renamed my directory to example-app and the source file to example-app.cpp. Then I created CMakeLists.txt in this directory with the following contents:
cmake_minimum_required(VERSION 3.0 FATAL_ERROR)
project(example-app)
find_package(Torch REQUIRED)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} ${TORCH_CXX_FLAGS}")
add_executable(example-app example-app.cpp)
target_link_libraries(example-app "${TORCH_LIBRARIES}")
set_property(TARGET example-app PROPERTY CXX_STANDARD 14)
Then
mkdir build
cd build
cmake -DCMAKE_PREFIX_PATH=../../libtorch ..
cmake --build . --config Release
And here's the output:
CMakeFiles/example-app.dir/example-app.cpp.o: In function `long long* at::Tensor::data<long long>() const':
example-app.cpp:(.text._ZNK2at6Tensor4dataIxEEPT_v[_ZNK2at6Tensor4dataIxEEPT_v]+0x14): undefined reference to `long long* at::Tensor::data_ptr<long long>() const'
Makes me think, maybe I forgot to include some header or define some variable?
Oh, this is all on Mint 19.2 (equivalent to Ubuntu 18.04), g++ version is 7.5.0, glibc is 2.27. Compiling with g++-8 gives the same result.

This is not a cmake-related error, it's just how the library was implemented. I do not know why, but it appears that the specialization of T* at::Tensor::data<T> const with T = long long was forgotten/omitted.
If you want to get your signed 64-bits pointer, you can still get it with int64_t:
auto data = T.data<int64_t>();
It's good practice to use these types for which the size is explicit in general, in order to avoid compatibility issues.

Related

Embeding V8 engine in a Hello_World console application on Linux - GCC 7 linker errors (undefined references)

Here are some details:
Host/Test OS, Toolchain:
Linux Mint 19.3 Tricia (Ubuntu 18.04), GCC 7
V8 GN build arguments:
is_debug=true
target_os="linux"
target_cpu="x64"
is_clang = false
is_component_build=false
use_glib=false
use_custom_libcxx = false
v8_static_library=true
v8_enable_i18n_support=false
v8_use_external_startup_data=false
The v8 engine is successfully built and I got all expected static libraries. Then I tried to build Hello_World app (provided in v8 source), linking with generated static libs. I got the following linking errors:
/usr/bin/g++-7 -o ./build-Debug/bin/Hello_World #./build-Debug//ObjectsList.txt -L. -L../../out.gn/linux.x86_64.Debug -L../../out.gn/linux.x86_64.Debug/obj -L../../out.gn/linux.x86_64.Debug/obj/tools/debug_helper -L../../out.gn/linux.x86_64.Debug/obj/third_party/zlib -L../../out.gn/linux.x86_64.Debug/obj/third_party/zlib/google -lv8_libbase -lv8_libplatform -lv8_base_without_compiler -lv8_bigint -lv8_compiler_opt -lv8_compiler -lv8_cppgc_shared -lv8_debug_helper -lv8_init -lv8_initializers -lv8_snapshot -lchrome_zlib -ltorque_base -ltorque_ls_base -ltorque_generated_initializers -ltorque_generated_definitions -lwee8 -lcompression_utils_portable -lpthread
../../out.gn/linux.x86_64.Debug/obj/torque_generated_initializers/promise-misc-tq-csa.o: In function `v8::internal::PromiseInit_0(v8::internal::compiler::CodeAssemblerState*, v8::internal::TNode<v8::internal::JSPromise>)':
/media/hongkun/Windows/Users/hongkun/v8/v8/out.gn/linux.x86_64.Debug/gen/torque-generated/src/builtins/promise-misc-tq-csa.cc:297: undefined reference to `v8::internal::PromiseBuiltinsAssembler::ZeroOutEmbedderOffsets(v8::internal::TNode<v8::internal::JSPromise>)'
../../out.gn/linux.x86_64.Debug/obj/torque_generated_initializers/promise-misc-tq-csa.o: In function `v8::internal::InnerNewJSPromise_0(v8::internal::compiler::CodeAssemblerState*, v8::internal::TNode<v8::internal::Context>)':
/media/hongkun/Windows/Users/hongkun/v8/v8/out.gn/linux.x86_64.Debug/gen/torque-generated/src/builtins/promise-misc-tq-csa.cc:377: undefined reference to `v8::internal::PromiseBuiltinsAssembler::AllocateJSPromise(v8::internal::TNode<v8::internal::Context>)'
../../out.gn/linux.x86_64.Debug/obj/torque_generated_initializers/promise-misc-tq-csa.o: In function `v8::internal::NewJSPromise_2(v8::internal::compiler::CodeAssemblerState*, v8::internal::TNode<v8::internal::Context>, v8::Promise::PromiseState, v8::internal::TNode<v8::internal::Object>)':
/media/hongkun/Windows/Users/hongkun/v8/v8/out.gn/linux.x86_64.Debug/gen/torque-generated/src/builtins/promise-misc-tq-csa.cc:1143: undefined reference to `v8::internal::PromiseBuiltinsAssembler::ZeroOutEmbedderOffsets(v8::internal::TNode<v8::internal::JSPromise>)'
Obviously, linker cannot find the class v8::internal::PromiseBuiltinsAssembler in any of the provided libraries. I have added all generated v8 libraries in the linker options.
How can I fix this error? Thanks for any suggestions.
Neither PromiseBuiltinsAssembler nor promise-misc-tq-csa.o should go into final binaries, both are only used by mksnapshot.
Have you tried following the official documentation?

undefined reference to `__gcov_exit'?

while I am building glibc library using yocto project it is giving
error: missing attribute ((constructor)) support??
after adding the coverage flags:
TARGET_CFLAGS += "-fprofile-arcs -ftest-coverage"
TARGET_LDFLAGS += "-lgcov -fprofile-arcs -ftest-coverage"
still, I am getting an error for glibc.
Please find the link of config log file : https://drive.google.com/file/d/14tiQJ8JIFE_tDWt3H9tS8zBBQROcZDNa/view
It is not working even after adding the following line in conf/local.conf :
EXTRA_OECONF = "libc_cv_ctors_header=yes"
Even i tried this
EXTRA_OECONF_append = "libc_cv_ctors_header=yes"
please find the config log file generated during compilation : https://drive.google.com/open?id=1kxTu8pt7h_9ty55OywP9Ilmmp04T61Rr
So, How to resolve this error?
Log file error Point
poky-linux/gcc/i586-poky-linux/8.2.0/ld: /tmp/ccxetEc1.o: in function `_GLOBAL__sub_D_00100_1__start':
conftest.c:(.text.exit+0x40): undefined reference to `__gcov_exit'<br>
collect2: error: ld returned 1 exit status<br>
configure:5682: $? = 1<br>
configure:5702: error: missing __attribute__ ((constructor)) support??
You are trying to build glibc with -fprofile-arcs -ftest-coverage in CFLAGS. That will not work. The errors you see are a result of these incorrect compiler flags.
A profiling glibc requires fairly substantial changes throughout the library and needs to be created by building with --enable-profile (which is not the default).
I had this error while I tried to enable coverage on a C project using a C++ test harness (CppUTest). Build system was handled by CMake.
Compilers and gcov were aligned on the same version (gcc --version, g++ --version and gcov --version gave the same version) but it seems that my build system was generated with a gcc 5 (resulting to an additional included directory by the linker: usr/lib/gcc/x86_64-linux-gnu/5). I clean the build tree and generated it again thanks to CMake which fixed the error.

linking SDL2 libraries with pkg-config

I am using Ubuntu 14.04LTS. I have installed the SDL2 libraries both by compiling from source (method1 https://askubuntu.com/questions/344512/what-is-the-general-procedure-to-install-development-libraries-in-ubuntu) and using sudo apt-get install libsdl2-dev.
As I understand, the former installed the libraries and headers in /usr/local/(lib and include), while the latter installs them system wide in /usr/(lib and include).
When I tried to compile a simple code to test the functionality:
#include <SDL.h>
#include <stdio.h>
int main(int argc, char* argv[]) {SDL_Window *window;
// Declare a pointer
SDL_Init(SDL_INIT_VIDEO); // Initialize SDL2
// Create an application window with the following settings:
window = SDL_CreateWindow(
"An SDL2 window", // window title
SDL_WINDOWPOS_UNDEFINED, // initial x position
SDL_WINDOWPOS_UNDEFINED, // initial y position
640, // width, in pixels
480, // height, in pixels
SDL_WINDOW_OPENGL // flags - see below
);
// Check that the window was successfully created
if (window == NULL) {
// In the case that the window could not be made...
printf("Could not create window: %s\n", SDL_GetError());
return 1;
}
// The window is open: could enter program loop here (see SDL_PollEvent())
SDL_Delay(3000); // Pause execution for 3000 milliseconds, for example
// Close and destroy the window
SDL_DestroyWindow(window);
// Clean up
SDL_Quit();
return 0;
using: g++ sdl_test.cpp -o sdlout
the compiler outputs:
sdltest.cpp:2:17: fatal error: SDL.h: No such file or directory
#include <SDL.h>
^
compilation terminated.
if I change to #include <SDL2/SDL.h> I get the following error:
/tmp/cc05JSKn.o: In function `main':
sdltest.cpp:(.text+0x15): undefined reference to `SDL_Init'
sdltest.cpp:(.text+0x3a): undefined reference to `SDL_CreateWindow'
sdltest.cpp:(.text+0x4a): undefined reference to `SDL_GetError'
sdltest.cpp:(.text+0x6d): undefined reference to `SDL_Delay'
sdltest.cpp:(.text+0x79): undefined reference to `SDL_DestroyWindow'
sdltest.cpp:(.text+0x7e): undefined reference to `SDL_Quit'
collect2: error: ld returned 1 exit status
Which are the basic functions, so I assume that the shared object libraries are not linked correctly.
I also tried: g++ -Wall sdltest.cpp -o outsdl -I /usr/local/include -L /usr/local/lib
to specify the paths, but again I get:
sdltest.cpp:2:17: fatal error: SDL.h: No such file or directory
#include <SDL.h>
^
compilation terminated.
The only command that worked and successfully compiled, is when using pkg-config g++ sdltest.cpp -o outsdl $(pkg-config --cflags --libs sdl2)
Therefore, I have the following questions:
1) Why is pkg-config necessary and how do compilation and linking flags work?
2) Is it possible to do something else in order to make the compilation command simpler?
3) (if not explained previously) What is the difference between pkg-config and using -I and -L which do not work?
4) what does $(...) actually do in the command line and is it completely the same as `...` ?
Thank you.
The pkg-config command is a more-or-less cross-platform or cross-distro way to provide the correct flags to your compiler to allow it to find header and library files. That way, your system can store files in different locations and everyone can use the same commands to compile your code. It also helps resolve any special requirements of the library you're trying to use.
Using $() is the same as using backticks, so you can execute what is inside the parentheses in order to see what extra arguments are being passed to your compiler to make it work. Here is what I get on my machine when I run pkg-config --cflags --libs sdl2:
-D_REENTRANT -I/usr/include/SDL2 -lSDL2
The reason you're getting SDL.h: No such file or directory is because pkg-config adds -I/usr/include/SDL2 to the include search paths so you can include SDL.h in your code (without the SDL2 subdirectory).
The reason you get undefined reference errors is because you don't have -lSDL2 (which tells the compiler to link libSDL2).

Linking GSL library to RcppGSL on Windows machine

I try to link the GSL library to the RcppGSL package. The following is my test function:
# colNorm.cpp
// [[Rcpp::depends(RcppGSL)]]
#include <RcppGSL.h>
#include <gsl/gsl_matrix.h>
#include <gsl/gsl_blas.h>
extern "C" SEXP colNorm(SEXP sM) {
try {
RcppGSL::matrix<double> M = sM; // create gsl data structures from SEXP
int k = M.ncol();
Rcpp::NumericVector n(k); // to store results
for (int j = 0; j < k; j++) {
RcppGSL::vector_view<double> colview = gsl_matrix_column (M, j);
n[j] = gsl_blas_dnrm2(colview);
}
M.free() ;
return n; // return vector
} catch( std::exception &ex ) {
forward_exception_to_r( ex );
} catch(...) {
::Rf_error( "c++ exception (unknown reason)" );
}
return R_NilValue; // -Wall
}
The GSL library is succesfully linked to the RcppGSL if the following command does not throw a compiler error:
sourceCpp("colNorm.cpp")
Since I am using Windows machine, I need to define environment variables such that RcppGSL knows where the GSL library is located.
I tried editing the environment variable, but the following compiler error shows that the package is still unable to find the GSL library:
g++ -m64 -I"C:/PROGRA~1/R/R-31~1.1/include" -DNDEBUG -I"C:/CodeLibrary/lib"/include - I"C:/PROGRA~1/R/R-31~1.1/library/Rcpp/include" -I"C:/PROGRA~1/R/R-31~1.1/library/RcppGSL/include" -I"d:/RCompile/CRANpkg/extralibs64/local/include" -O2 -Wall -mtune=core2 -c colNorm.cpp -o colNorm.o
g++ -m64 -shared -s -static-libgcc -o sourceCpp_38624.dll tmp.def colNorm.o -LC:/CodeLibrary/lib/lib -lgsl -lgslcblas -LC:/PROGRA~1/R/R-31~1.1/bin/x64 -lRlapack -LC:/PROGRA~1/R/R-31~1.1/bin/x64 -lRblas -lgfortran -Ld:/RCompile/CRANpkg/extralibs64/local/lib/x64 -Ld:/RCompile/CRANpkg/extralibs64/local/lib -LC:/PROGRA~1/R/R-31~1.1/bin/x64 -lR
c:/program files/r/r-3.1.1/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lgsl
c:/program files/r/r-3.1.1/rtools/gcc-4.6.3/bin/../lib/gcc/i686-w64-mingw32/4.6.3/../../../../i686-w64-mingw32/bin/ld.exe: cannot find -lgslcblas
collect2: ld returned 1 exit status
I set an environment variable LIB_GSL equal to "C:/CodeLibrary/lib". In the first line, the compiler takes my environment variable and adds /include. In the second line, the compiler adds /lib. Those locations do not exist on my C drive, maybe that's the reason why it cannot find the library.
I would be really happy, if someone with a lot of compiler experience could show how to successfully link a 3rd party library to a package on Windows machine.
Maybe more environment variables need to be defined?
Thanks Dirk for your handholding! I got it finally up and running.
Three things have to be done:
1) Download the local300 folder from your link and allocate folder on your drive. The path cannot contain any whitespaces, i.e. C:/Program Files/local300 will not work but
C:/local300 will work
2) Set the environment variable LIB_GSL equal to this path, e.g. LIB_GSL to C:/local300
3) The compiler looks at LIB_GSL/lib for libgsl.a and libgslcblas.a (-lgsl and -lgslcblas). However, in LIB_GSL/lib are subfolders i386 and x64. I didn't know how to change the place the compiler looks for the files, thus I copied everything from inside x64 and put it into LIB_GSL/lib folder (one folder level above).
This allows RcppGSL to compile code without errors.
Look at this page which is (after a link or two) pointed to from the R Installation and Administration manual, appendix D for Windows.
Expand that, see where it puts headers and the library for GSL and adjust LIB_GSL accordingly.
This is what CRAN itself uses, so we know it works.

How to link to VTK library on Windows using g++

I want to use the command line to compile a simple program using the VTK library:
g++ -IC:\VTK\Install\includes\vtk-5.10 SimpleTest.cpp -LC:\VTK\Install\lib
\vtk_5.10 -lvtkCommon -lvtkGraphics
SimpleTest.cpp
#include "vtkConeSource.h"
int main()
{
vtkConeSource* cone = vtkConeSource::New();
cone->SetHeight(5.0);
return 1;
}
but I always get many errors about "undefined references". For example:
C:\VTK\Install\lib\vtk_5.10/libvtkGraphics.a(vtkConeSource.cxx.obj):vtkConeSource.cxx:(.text+0x1c): undefined reference to `vtkInformationVector::GetInformationObject(int)'
C:\VTK\Install\lib\vtk_5.10/libvtkGraphics.a(vtkConeSource.cxx.obj):vtkConeSource.cxx:(.text+0x24): undefined reference to `vtkStreamingDemandDrivenPipeline::MAXIMUM_NUMBER_OF_PIECES()'
C:\VTK\Install\lib\vtk_5.10/libvtkGraphics.a(vtkConeSource.cxx.obj):vtkConeSource.cxx:(.text+0x36): undefined reference to `vtkInformation::Set(vtkInformationIntegerKey*, int)'
C:\VTK\Install\lib\vtk_5.10/libvtkGraphics.a(vtkConeSource.cxx.obj):vtkConeSource.cxx:(.text+0x85): undefined reference to `vtkPolyDataAlgorithm::PrintSelf(std::ostream&, vtkIndent)'
C:\VTK\Install\lib\vtk_5.10/libvtkGraphics.a(vtkConeSource.cxx.obj):vtkConeSource.cxx:(.text+0x9f): undefined reference to `operator<<(std::ostream&, vtkIndent const&)'
and many more ...
I can't tell if I am using wrong syntax for g++ or if there is something wrong with my VTK library.
I am using:
VTK 5.10.1
minGW 4.7.2, 32 bit
VTK was compiled from source with minGW using CMake 2.8.11.2
update:
I did find this tutorial http://vtkblog.blogspot.com/2008/05/build-vtk-from-source-using-mingwmsys.html about this. However I used minGW32-make instead of installing MSYS and using it's "make". Could this be part of the problem?
As a very first test, try explicitly including all vtk libraries contained in Install\lib\vtk_5.10 - to ensure those undefined references are not coming from those missing libraries.

Resources