Automatically add all files in a folder to a target using CMake? - visual-c++

I am considering switching a cross platform project from separate build management systems in Visual C++, XCode and makefiles to CMake.
One essential feature I need is to add automatically all files in a directory to a target. While this is easy to do with make, it is not easily doable with Visual C++ and XCode (correct me if I am wrong). Is it possible to do it in directly in CMake? How?

As of CMake 3.1+ the developers strongly discourage users from using file(GLOB or file(GLOB_RECURSE to collect lists of source files.
Note: We do not recommend using GLOB to collect a list of source files from your source tree. If no CMakeLists.txt file changes when a source is added or removed then the generated build system cannot know when to ask CMake to regenerate. The CONFIGURE_DEPENDS flag may not work reliably on all generators, or if a new generator is added in the future that cannot support it, projects using it will be stuck. Even if CONFIGURE_DEPENDS works reliably, there is still a cost to perform the check on every rebuild.
See the documentation here.
There are two goods answers ([1], [2]) here on SO detailing the reasons to manually list source files.
It is possible. E.g. with file(GLOB:
cmake_minimum_required(VERSION 2.8)
file(GLOB helloworld_SRC
"*.h"
"*.cpp"
)
add_executable(helloworld ${helloworld_SRC})
Note that this requires manual re-running of cmake if a source file is added or removed, since the generated build system does not know when to ask CMake to regenerate, and doing it at every build would increase the build time.
As of CMake 3.12, you can pass the CONFIGURE_DEPENDS flag to file(GLOB to automatically check and reset the file lists any time the build is invoked. You would write:
cmake_minimum_required(VERSION 3.12)
file(GLOB helloworld_SRC CONFIGURE_DEPENDS "*.h" "*.cpp")
This at least lets you avoid manually re-running CMake every time a file is added.

The answer by Kleist certainly works, but there is an important caveat:
When you write a Makefile manually, you might generate a SRCS variable using a function to select all .cpp and .h files. If a source file is later added, re-running make will include it.
However, CMake (with a command like file(GLOB ...)) will explicitly generate a file list and place it in the auto-generated Makefile. If you have a new source file, you will need to re-generate the Makefile by re-running cmake.
edit: No need to remove the Makefile.

Extension for #Kleist answer:
Since CMake 3.12 additional option CONFIGURE_DEPENDS is supported by commands file(GLOB) and file(GLOB_RECURSE). With this option there is no needs to manually re-run CMake after addition/deletion of a source file in the directory - CMake will be re-run automatically on next building the project.
However, the option CONFIGURE_DEPENDS implies that corresponding directory will be re-checked every time building is requested, so build process would consume more time than without CONFIGURE_DEPENDS.
Even with CONFIGURE_DEPENDS option available CMake documentation still does not recommend using file(GLOB) or file(GLOB_RECURSE) for collect the sources.

To use Visual Studio project hierarchy inside Clion with cmake:
cmake_minimum_required(VERSION 3.17)
project(MyProject)
set(CMAKE_CXX_STANDARD 17)
file(GLOB APP_SOURCES */*.cpp)
foreach (testsourcefile ${APP_SOURCES})
get_filename_component(testname ${testsourcefile} NAME_WLE)
get_filename_component(dirname ${testsourcefile} DIRECTORY)
file(GLOB dir_src CONFIGURE_DEPENDS
"${dirname}/*.h"
"${dirname}/*.cpp"
)
message("${testname}.cpp | ${dir_src}")
add_executable("${testname}.cpp" ${dir_src})
endforeach (testsourcefile ${APP_SOURCES})

So Why not use powershell to create the list of source files for you. Take a look at this script
param (
[Parameter(Mandatory=$True)]
[string]$root
)
if (-not (Test-Path -Path $root)) {
throw "Error directory does not exist"
}
#get the full path of the root
$rootDir = get-item -Path $root
$fp=$rootDir.FullName;
$files = Get-ChildItem -Path $root -Recurse -File |
Where-Object { ".cpp",".cxx",".cc",".h" -contains $_.Extension} |
Foreach {$_.FullName.replace("${fp}\","").replace("\","/")}
$CMakeExpr = "set(SOURCES "
foreach($file in $files){
$CMakeExpr+= """$file"" " ;
}
$CMakeExpr+=")"
return $CMakeExpr;
Suppose you have a folder with this structure
C:\Workspace\A
--a.cpp
C:\Workspace\B
--b.cpp
Now save this file as "generateSourceList.ps1" for example, and run the script as
~>./generateSourceList.ps1 -root "C:\Workspace" > out.txt
out.txt file will contain
set(SOURCE "A/a.cpp" "B/b.cpp")

Related

CMake Precompiled Headers not including correctly for Visual Studio

So I've recently started using the "new" target_precompile_headers command for CMake to generate and include my precompiled headers.
But for some reason it doesn't seem to work at all the way I want it.
I do understand that the usage of this command is to provide the list of headers to include in your pch and that CMake itself generates the actual pch. This seem to work just fine as I can see the pchs generated for each config & project in my solution.
However looking at the project properties, and more specifically looking at the .cpp properties the /FI command is always empty, which means no files are force included.
This also means that none of my cpp catches that I am using pch at all and my compile fails.
Does anyone have any suggestions on what could be wrong?
// Example of a CMakeLists.txt which defines a static lib using PCH
file(GLOB ALL_SOURCE_FILES "*.cpp" "*.h")
add_library(MyLib STATIC ${ALL_SOURCE_FILES})
target_precompile_headers(MyLib PRIVATE MyHeader.h)
Below you see my own lib created with the snippet above. As you can see, no Forced Include file is added, thus no cpp files gets the PCH included.
I found this workaround: Adding the compiler option manually...
if(MSVC)
target_compile_options(YourProject PRIVATE "/FI${CMAKE_CURRENT_BINARY_DIR}/CMakeFiles/${PROJECT_NAME}.dir/$<CONFIG>/cmake_pch.hxx")
endif()
(Currently using Cmake 3.24.1 with cmake_minimum_required(VERSION 3.18) in the file)

"undefined reference" error from static lib of external parquet project

I want to link expernal 'parquet' project ( https://github.com/apache/arrow/tree/master/cpp ) as part of my current project on Linux.
For this purposes I ran cmake of parquet with such parameters
cd build_parquet
cmake -DCMAKE_BUILD_TYPE=Release -DARROW_PARQUET=ON \
-DBoost_NO_BOOST_CMAKE=TRUE -DBoost_NO_SYSTEM_PATHS=TRUE -DBOOST_ROOT=${BOOST_BUILD_DIR}/include -DBOOST_LIBRARYDIR=${BOOST_BUILD_DIR}/lib/boost -DARROW_BOOST_USE_SHARED=OFF -DBOOST_INCLUDEDIR=${BOOST_BUILD_DIR}/include/boost ..
cmake --build . --config Release
// There are a lot of dependencies except boost, but only boost required to be installed on system, since other could be downloaded and installed by cmake script
Project successfully compiled. I got executable which could launch, generated static libs libarrow.a, libparquet.a, shared libraries libarrow.so, libparquet.so
In my main project I want to use such libraries and I use such commands in cmake to find them
find_path(PARQUET_INCLUDE_DIR NAMES arrow/api.h PATHS ${PARQUET_DIR}/src)
find_library(PARQUET_LIBRARY_RELEASE NAMES parquet.a
PATHS build_parquet/release/Release/ )
find_library(ARROW_LIBRARY_RELEASE NAMES arrow.a
PATHS build_parquet/release/Release/ )
set(PARQUET_LIBRARIES_RELEASE ${PARQUET_LIBRARY_RELEASE} ${ARROW_LIBRARY_RELEASE} )
include(FindPackageHandleStandardArgs)
find_package_handle_standard_args(Parquet DEFAULT_MSG PARQUET_INCLUDE_DIR
${PARQUET_LIBRARIES_RELEASE } )
That's work okay, libraries and includes are found.
Then I link this libraries to my project
target_link_libraries(${myExe} ${PARQUET_LIBRARIES_RELEASE} ${mySomeOtherLibraries} )
after this I got enormous amount of linker errors such that
libparquet.a(column_writer.cc.o): In function `apache::thrift::transport::TMemoryBuffer::~TMemoryBuffer()':
column_writer.cc:(.text._ZN6apache6thrift9transport13TMemoryBufferD0Ev[_ZN6apache6thrift9transport13TMemoryBufferD5Ev]+0x3): undefined reference to `vtable for apache::thrift::transport::TMemoryBuffer'
.....
so that's what I don't understand much, why lib compiled well in parquet project itself but has a lot of unresolved now, when I use it to link in my own project? Moreover I compiled project for windows and when I did the same things, but with arrow.lib and parquet.lib (instead of libparquet.a and libarrow.a ) things worked fine! I was needed only to put arrow.dll, parquet.dll to executables to run project. But in Linux I've already crashed my head
So, why it doesn't work, what I should do to finally link the project with library ?
Update
I found the problem, I had to link libraries with adding .so files (not only .a files) like this
find_library(PARQUET_LIBRARY_RELEASE NAMES parquet.so parquet.a
PATHS build_parquet/release/Release/ )
find_library(ARROW_LIBRARY_RELEASE NAMES arrow.so arrow.a
PATHS build_parquet/release/Release/ )
set(PARQUET_LIBRARIES_RELEASE ${PARQUET_LIBRARY_RELEASE} ${ARROW_LIBRARY_RELEASE} )
Project is builded. So now the question is, why I need to add .so files to linker (in Windows only static .lib are enough), is it always a case when I build project in Linux ? Is order of linkage important ( .so files first and .a files next ? )
As Uwe wrote in a comment, the https://github.com/apache/parquet-cpp repository is deprecated, and the Parquet C++ library is being developed as part of the Apache Arrow C++ codebase https://github.com/apache/arrow/tree/master/cpp. Can you try building based on that, and if you have trouble can you post on the dev#arrow.apache.org mailing list?
You have succeed to build the project when link with the shared (.so) libraries instead of the static (.a) ones.
(The command find_library actually looks for one library, which name is listed in NAMES option. In your case it found .so library because its name comes before the .a one).
Actually, both shared and static parquet libraries contain the same set of symbols, and both sets are insufficient for link. The difference is that the shared library contains information, where to find remaining symbols (in the thrift library in your case), but the static library doesn't.
For correctly link with the static libraries, you need to list dependent libraries manually.
On Windows .lib file may mean either a static library, or a import file for the shared (.dll) one. It seems that you link with the dynamic one (it has no lib prefix), which succeed like on Linux.

CMake - compile with /MT instead of /MD

I'm a newbie to cmake (2.8.12.1) and I'm using it on Windows to generate the project files to build cpp-netlib using Visual Studio 2012.
By default it compiles with the /MDd compiler switch. I want to change it so that it uses /MTd.
I followed the advice given here https://stackoverflow.com/a/14172871 but it isn't working for me.
Specifically, I added the second line shown below in the if statement to CmakeLists.txt.
if (MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /bigobj")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")
endif()
When I open the Visual Studio sln file I can see that the /MDd option is still set. Furthermore, I see the following in CMakeCache.txt:
//Flags used by the compiler during debug builds.
CMAKE_CXX_FLAGS_DEBUG:STRING=/D_DEBUG /MDd /Zi /Ob0 /Od /RTC1
I've also tried setting the flags from scratch like this:
set(CMAKE_CXX_FLAGS_DEBUG "/MTd")
but that doesn't work either.
If I pass the option via the command line like this:
-DCMAKE_CXX_FLAGS_DEBUG="/MTd"
the option is successfully set in the Visual Studio projects.
Can anyone tell me what I'm doing wrong?
I would also appreciate it if someone could enlighten me as to where the values in the cache come from that I don't specify on the command line or aren't in CmakeLists.txt.
Adding CMakeList.txt as requested. I've never posted before so apologies if I've not done this right.
# Original from cpp-netlib.org with my edits
cmake_minimum_required(VERSION 2.8)
project(CPP-NETLIB)
set(Boost_USE_STATIC_LIBS ON)
set(Boost_USE_MULTI_THREADED ON)
find_package( Boost 1.45.0 REQUIRED unit_test_framework system regex date_time thread filesystem program_options chrono )
find_package( OpenSSL )
find_package( Threads )
set(CMAKE_VERBOSE_MAKEFILE true)
if (CMAKE_BUILD_TYPE MATCHES Debug)
add_definitions(-DBOOST_NETWORK_DEBUG)
endif()
if (OPENSSL_FOUND)
add_definitions(-DBOOST_NETWORK_ENABLE_HTTPS)
include_directories(${OPENSSL_INCLUDE_DIR})
endif()
if (${CMAKE_CXX_COMPILER_ID} MATCHES GNU)
set (CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall")
endif()
if (Boost_FOUND)
################# added #################
add_definitions(-DBOOST_ALL_NO_LIB)
#########################################
if (MSVC)
add_definitions(-D_SCL_SECURE_NO_WARNINGS)
endif(MSVC)
if (WIN32)
add_definitions(-D_WIN32_WINNT=0x0501)
endif(WIN32)
include_directories(${Boost_INCLUDE_DIRS})
enable_testing()
add_subdirectory(libs/network/src)
add_subdirectory(libs/network/test)
if (NOT MSVC)
add_subdirectory(libs/mime/test)
endif(NOT MSVC)
add_subdirectory(libs/network/example)
endif(Boost_FOUND)
if (MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /bigobj")
################# added #################
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /MTd")
#########################################
endif()
enable_testing()
Adding the code inside your if block should work fine. However, since it doesn't produce the required effect in your .sln, either the code inside the if block isn't being executed or the flags are being replaced later in the CMakeLists.txt.
As to where the cached values for these flags come from; CMake applies and caches what are considered to be useful default values for each default build type.
Have you added your changes to the flags before the project call? project sets up a lot of things, including the compiler flags. If you set these flags without caching them (i.e. calling set as you have shown), project will overwrite them. If you set the flags and cache them (e.g. using set(... CACHE), or by adding them via the command line arg -D as you have shown), then project won't overwrite them.
Also, since calling set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /bigobj") doesn't use the CACHE option of set, then this change to the variable CMAKE_CXX_FLAGS won't appear in your CMakeCache.txt file.
In order to get this into the cache, you'd have to also use the FORCE option of set, since you'd be telling CMake to replace an already-cached value (CMake itself caches these flag variables by default). I wouldn't recommend this however - I can't see why you'd need to do this, and you'd be best to look for a better way than appending the flag(s) you want, since every time CMake runs the appended flags would be re-appended.
If this doesn't allow you to resolve your issue, perhaps you could show a full (small) CMakeLists.txt which exhibits the problem?
UPDATE:
OK - The problem is a scoping issue. The changes to the compiler flags will be applied to all targets (exes/libs) which are defined in that CMakeLists.txt, but only to subdirectory targets which are added after the flags are modified.
The add_subdirectory command introduces a new scope (although this isn't mentioned in the docs) which inherits a copy of the parent's variables. However, subsequent changes to the variables in the parent scope don't affect the child copies, and likewise changing the child copies doesn't affect the value of the parent's copies (unless you force it using set(... PARENT_SCOPE)).
So, to fix your issue, just move the if(MSVC) block to before all the add_subdirectory calls.

Reusing custom makefile for static library with cmake

I guess this would be a generic question on including libraries with existing makefiles within cmake; but here's my context -
I'm trying to include scintilla in another CMake project, and I have the following problem:
On Linux, scintilla has a makefile in (say) the ${CMAKE_CURRENT_SOURCE_DIR}/scintilla/gtk directory; if you run make in that directory (as usual), you get a ${CMAKE_CURRENT_SOURCE_DIR}/scintilla/bin/scintilla.a file - which (I guess) is the static library.
Now, if I'd try to use cmake's ADD_LIBRARY, I'd have to manually specify the sources of scintilla within cmake - and I'd rather not mess with that, given I already have a makefile. So, I'd rather call the usual scintilla make - and then instruct CMAKE to somehow refer to the resulting scintilla.a. (I guess that this then would not ensure cross-platform compatibility - but note that currently cross-platform is not an issue for me; I'd just like to build scintilla as part of this project that already uses cmake, only within Linux)
So, I've tried a bit with this:
ADD_CUSTOM_COMMAND(
OUTPUT scintilla.a
COMMAND ${CMAKE_MAKE_PROGRAM}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/scintilla/gtk
COMMENT "Original scintilla makefile target" )
... but then, add_custom_command adds a "target with no output"; so I'm trying several approach to build upon that, all of which fail (errors given as comment):
ADD_CUSTOM_TARGET(scintilla STATIC DEPENDS scintilla.a) # Target "scintilla" of type UTILITY may not be linked into another target.
ADD_LIBRARY(scintilla STATIC DEPENDS scintilla.a) # Cannot find source file "DEPENDS".
ADD_LIBRARY(scintilla STATIC) # You have called ADD_LIBRARY for library scintilla without any source files.
ADD_DEPENDENCIES(scintilla scintilla.a)
I'm obviously quote a noob with cmake - so, is it possible at all to have cmake run a pre-existing makefile, and "capture" its output library file, such that other components of the cmake project can link against it?
Many thanks for any answers,
Cheers!
EDIT: possible duplicate: CMake: how do i depend on output from a custom target? - Stack Overflow - however, here the breakage seems to be due to the need to specifically have a library that the rest of the cmake project would recognize...
Another related: cmake - adding a custom command with the file name as a target - Stack Overflow; however, it specifically builds an executable from source files (which I wanted to avoid)..
You could also use imported targets and a custom target like this:
# set the output destination
set(SCINTILLA_LIBRARY ${CMAKE_CURRENT_SOURCE_DIR}/scintilla/gtk/scintilla.a)
# create a custom target called build_scintilla that is part of ALL
# and will run each time you type make
add_custom_target(build_scintilla ALL
COMMAND ${CMAKE_MAKE_PROGRAM}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/scintilla/gtk
COMMENT "Original scintilla makefile target")
# now create an imported static target
add_library(scintilla STATIC IMPORTED)
# Import target "scintilla" for configuration ""
set_property(TARGET scintilla APPEND PROPERTY IMPORTED_CONFIGURATIONS NOCONFIG)
set_target_properties(scintilla PROPERTIES
IMPORTED_LOCATION_NOCONFIG "${SCINTILLA_LIBRARY}")
# now you can use scintilla as if it were a regular cmake built target in your project
add_dependencies(scintilla build_scintilla)
add_executable(foo foo.c)
target_link_libraries(foo scintilla)
# note, this will only work on linux/unix platforms, also it does building
# in the source tree which is also sort of bad style and keeps out of source
# builds from working.
OK, I think I have it somewhat; basically, in the CMakeLists.txt that build scintilla, I used this only:
ADD_CUSTOM_TARGET(
scintilla.a ALL
COMMAND ${CMAKE_MAKE_PROGRAM}
WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/scintilla/gtk
COMMENT "Original scintilla makefile target" )
... and then, the slightly more complicated part, was to find the correct cmake file elsewhere in the project, where the ${PROJECT_NAME} was defined - so as to add a dependency:
ADD_DEPENDENCIES(${PROJECT_NAME} scintilla.a)
... and finally, the library needs to be linked.
Note that in the commands heretofore, the scintilla.a is merely a name/label/identifier/string (so it could be anything else, like scintilla--a or something); but for linking - the full path to the actual `scintilla.a file is needed (which in this project ends up in a variable ${SCINTILLA_LIBRARY}). In this project, the linking basically occurs through a form of a
list(APPEND PROJ_LIBRARIES ${SCINTILLA_LIBRARY} )
... and I don't really know how cmake handles the actual linking afterwards (but it seems to work)
For consistency, I tried to use ${SCINTILLA_LIBRARY} instead of scintilla.a as identifier in the ADD_CUSTOM_TARGET, but got error: "Target names may not contain a slash. Use ADD_CUSTOM_COMMAND to generate files". So probably this could be solved smarter/more correct with ADD_CUSTOM_COMMAND - however, I read that it "defines a new command that can be executed during the build process. The outputs named should be listed as source files in the target for which they are to be generated."... And by now I'm totally confused so as to what is a file, what is a label, and what is a target - so I think I'll leave at this (and not fix it if it ain't broken :) )
Well, it'd still be nice to know a more correct way to do this eventually,
Cheers!

How to work around "scons: warning: Two different environments were specified for target"

Suppose I have an SConstruct file that looks like this:
env = Environment()
env.Program("a", ["a.c", "util.c"])
env.Program("b", ["b.c", "util.c"])
This build works properly with no SCons warning messages. However, if I modify this to specify different libraries for each Program build (the actual libraries are not relevant):
env.Program("a", ["a.c", "util.c"], LIBS="m")
env.Program("b", ["b.c", "util.c"], LIBS="c")
then I get the warning:
scons: warning: Two different environments were specified for target util.o,
but they appear to have the same action: $CC -o $TARGET -c $CFLAGS $CCFLAGS $_CCCOMCOM $SOURCES
This appears to be caused by the Program builder automatically creating a new environment for building the sources, even though it is just the LIBS variable that is different (and so only the link step needs to have a different environment). I can work around this by doing something like:
util = env.Object("util.c")
env.Program("a", ["a.c"] + util, LIBS="m")
env.Program("b", ["b.c"] + util, LIBS="c")
This uses a single Object builder for building util.c, then using the precompiled object file in each Program build, thus avoiding the warning. However, this should not really be necessary. Is there a more elegant way to work around this problem? Or is this actually a bug in SCons that should be fixed?
Context: I have nearly 2000 C source files compiled into about 20 libraries and 120 executables with lots of shared sources. I created the SConstruct file from the previous proprietary build system using a conversion script I wrote. There are about 450 "Two different environments" warning messages produced by SCons for a full build using my current SConstruct.
I found a workaround that doesn't involve creating extra variables to hold the object file nodes:
env.Program("a", ["a.c", env.Object("util.c")], LIBS="m")
env.Program("b", ["b.c", env.Object("util.c")], LIBS="c")
This isolates the build of util.c within a single environment. Although it is specified twice, once for each Program, SCons doesn't warn about this because it's the same source built with the same env object. Of course SCons only compiles the source once in this case.
You may use the Split function and a custom helper to simplify the build process for large projects:
def create_objs(SRCS, path=""):
return [env.Object(path+src+".cpp") for src in SRCS]
prg1 = Split("file_1 file_2 file_N")
prg2 = Split("file_2 file_5 file_8")
env.Program("a", create_objs(prg1), LIBS="x")
env.Program("b", create_objs(prg2), LIBS="y")
The object files are created only once, and they can be used in multiple builds. Hope this helps...
One issue I found in my code was that I was not using the target object path correctly. Or in otherwords I had a variant dir directive, but instead of using BUILDPATH i ended up using my original source code path. This way Scons was finding the object generated in target BUILDPATH and source path.
Creating a static library out of the first set of files and linking the library to the next set of files (which have some files in common with the first set) to create a target works as well.
env.StaticLibrary ("a", ["a.c","util.c"], LIBS = "m")
env.Program ("b", ["b.c","util.c"], LIBS = ["c","a"])

Resources