I need to find out why some module gets included into compilation.
There is some class that should not be included and I think there are some unused imports or bad architecture that requires unnecessary imports. Is there a way to find which modules import some module, which modules import these modules that include this module, and so on, tracking that down to main class of application?
You could use -D dump-dependencies for this, in which case the compiler will generate two files that can be used to follow the dependency graph in both directions:
dump/<target>/.dependants.dump
dump/<target>/.dependencies.dump
There is also a handy online tool created by Mark Knol that helps a lot with analyzing these files. To answer the question "what does Array depend on?", you can just upload the two files and enter "array" into the search field:
Conveniently, the results are also clickable.
I've just came up with very simple idea: just delete this file and there will be compilation errors in places where this module is imported.
Related
I am trying to create a Python package. You can have a look at my awful attempt here.
I have a module called imguralbum.py. It lives in a directory called ImgurAlbumDownloader which I understand is the name of the package -- in terms of what you type in an import statement, e.g.
import ImgurAlbumDownloader
My module contains two classes ImgurAlbumDownloader and ImgurAlbumException. I need to be able to use both of these classes in another module (script). However, I cannot for the life of me work out what I am supposed to put in my __init__.py file to make this so. I realize that this duplicates a lot of previously answered questions, but the advice seems very conflicting.
I still have to figure out why (I have some ideas), but this is now working:
from ImgurAlbumDownloader.imguralbum import ImgurAlbumDownloader, ImgurAlbumException
The trick was adding the package name to the module name.
It sure sounds to me like you don't actually want a package. It's OK to just use a single module, if your code just does one main thing and all its parts are closely related. Packages are useful when you have distinct parts of your code that might not all be needed at the same time, or when you have so much code that a single module would be very large and hard to find things in.
Suppose we have the following project structure:
E:\demo_proj\
utilities_package\
__init__.py
util_one.py
util_two.py
demo_package\
__init__.py
demo_sub_package\
__init__.py
demo_sub_sub_package\
demo_file.py
What is a sensible way for demo_file.py to import from utilities_package?
The utilities_package is in a much higher level directory than demo_file.py.
Although it is not shown in the little diagram I gave above, suppose that utilities_package is used everywhere throughout the project. It is not sensible to make utilities_package a sub-directory of demo_sub_sub_package
Some similar questions have been asked here and here.
The asker of the first question neglected to include a __init__.py in their packages. That is not an issue in my case.
The answers given to the the second question linked-to above are problematic:
Most of the answers are quite old. For example, Loader.load_module was deprecated after Python 3.4
Most of the answers are specific to the question in that they only go up one directory, whereas, I would like to go up several levels.
Some of the answers are high-level descriptions without code. Other answers have the opposite problem: they contain ugly-looking code, but don't bother to explain it.
I am open to solutions which require restructuring the project. I'm also interested in solutions which leave the project structure intact, but which insert the relevant code into demo_file.py
It already has the very epic solution for your problems. Please refer there: Relative imports for the billionth time. Just as a notation, the python3 and python2 have the totally different definitions for the relative and absolute import. For version 3+, any modules import directly using import module_name are treated as the absolute importation, which means this module must be included in the sys.path. Otherwise, for any modules import like from one_module import something or from . import something are treated like the relative importation. The dot-notation represent the current directory. However, any module is executed directly by itself, it will lose its namespace, instead it will always be treated as __main__, and all the files in the same directory will automatically add to sys.path by default. Hope it will help you.
Is there a way to have custom behaviour for import statements in Python? How? E.g.:
import "https://github.com/kennethreitz/requests"
import requests#2.11.1
import requests#7322a09379565bbeba9bb40000b41eab8856352e
Alternatively, in case this isn't possible... Can this be achieved in a standard way with function calls? How? E.g.:
import gitloader
repo = gitloader.repo("https://github.com/kennethreitz/requests")
requests = repo.from_commit("7322a09379565bbeba9bb40000b41eab8856352e")
There are two reasons for why I would like to do this. The first reason is convenience (Golang style imports). The second reason is that I need cryptographic verification of plugin modules for a project. I'm using Python 3.x
What you're essentially asking is customization of the import steps. This was made possible with PEP 302 which brought about certain hooks for for customization.
That PEP is currently not the source from which you should learn how importing works; rather, look at the documentation for the import statement for reference. There it states:
Python includes a number of default finders and importers. The first one knows how to locate built-in modules, and the second knows how to locate frozen modules. A third default finder searches an import path for modules. The import path is a list of locations that may name file system paths or zip files. It can also be extended to search for any locatable resource, such as those identified by URLs.
The import machinery is extensible, so new finders can be added to extend the range and scope of module searching.
In short, you'll need to define the appropriate finder (to find the modules you're looking for) and an appropriate loader for loading these.
I am building a custom binary of NodeJS from the latest code base for an embedded system. I have a couple modules that I would like to ship as standard with the binary - or even run a custom script the is compiled into the binary and can be invoked through a command line option.
So two questions:
1) I vaguely remember that node allowed to include custom modules during build time but I went through the latest 5.9.0 configure script and I can't see anything related - or maybe I am missing it.
2) Did someone already do something similar? If yes, what were the best practices you came up with?
I am not looking for something like Electron or other binary bundlers but actually building into the node binary.
Thanks,
Andy
So I guess I figure it out much faster that I thought.
For anyone else, you can add any NPM module to it and just add the actual source files to the node.gyp configuration file.
Compile it and run the custom binary. It's all in there now.
> var cmu = require("cmu");
undefined
> cmu
{ version: [Function] }
> cmu.version()
'It worked!'
> `
After studying this for quite a while, I have to say that the flyandi's answer is not quite true. You cannot add any NPM module just by adding it to the node.gyp.
You can only add pure JavaScript modules this way. To be able to embed a C++ module (I deliberately don't use the word "native", because that one is quite ambiguous in nodeJS terminology - just look at the sources).
To summarize this:
To embed a JS module to your custom nodejs, just add it in the library_files section of the node.gyp file. Also note that it should be placed within the lib folder, otherwise you'll have troubles requiring the module. That's because the name/path listed in node.gyp / library_files is used to encode the id of the module in the node_javascript.cc intermediate file which is then used when searching for the built-in modules.
To embed a native module is much more difficult. The best way I have found so far is to build the module as a static library instead of dynamic, which for cmake(-js) based module you can achieve by changing the SHARED parameter to STATIC like this:
add_library(${PROJECT_NAME} STATIC ${SRC})
instead of:
add_library(${PROJECT_NAME} SHARED ${SRC})
And also changing the suffix:
set_target_properties(
${PROJECT_NAME}
PROPERTIES
PREFIX ""
SUFFIX ".lib") /* instead of .node */
Then you can link it from node.gyp by adding this section:
'link_settings': {
'libraries' : [
"path/to/my/library.lib",
#...add other static dependencies
],
},
(how to do this with node-gyp based project should be quite ease to google)
This allows you to build the module, but you won't be able to require it, because require() function in node can only be used to load built-in JS modules, external JS modules or external dynamic node modules. But now we have a built-in C++ module. Well, lot of node integrated modules are C++, but they always have a JS wrapper in /lib, and those wrappers they use process.binding() to load the C++ module. That is, process.binding() is sort of a require() function for integrated C++ modules.
That said, we also need to call require.binding() instead of require to load our integrated module. To be able to do that, we have to make our module "built-in" first.
We can do that by replacing
NODE_MODULE(mymodule, InitAll)
int the module definition with
NODE_BUILTIN_MODULE_CONTEXT_AWARE(mymodule, InitAll)
which will register it as internal module and from now on we can process.binding() it.
Note that NODE_BUILTIN_MODULE_CONTEXT_AWARE is not defined in node.h as NODE_MODULE but in node_internals.h so you either have to include that one, or copy the macro definition over to your cpp file (the first one is of course better because the nodejs API tends to change quite often...).
The last thing we need to do is to list our newly integrated module among the others so that the node knows to initialize them (that is include them within the list of modules used when searching for the modules loaded with process.binding()). In node_internals.h there is this macro:
#define NODE_BUILTIN_STANDARD_MODULES(V) \
V(async_wrap) \
V(buffer) \
V(cares_wrap) \
...
So just add the your module to the list the same way as the others V(mymodule).
I might have forgotten some step, so ask in the comments if you think I have missed something.
If you wonder why would anyone even want to do this... You can come up with several reasons, but here's one most important to me: Those package managers used to pack your project within one executable (like pkg or nexe) work only with node-gyp based modules. If you, like me, need to use cmake based module, the final executable won't work...
Are there any known issues with mixing nodejs modules (require) with typescript definition files (d.ts) multiple times over files within a module?
My scenario is that I have a module namespace per folder (much like I would in C#), then I basically compile them all via tsc to an outputted my-module.js. However I keep getting really odd errors like Could not find type HTMLElement but lots of people have pointed out that tsc includes the typescript lib file by default which contains all those types.
I have noticed a few people having odd errors when they are including the same d.ts files over multiple files which are all compiled with the --out flag to get it all into one file, so could this be causing my issues?
An example of my usage would be:
///<reference path="path/to/knockout.d.ts" />
import ko = require("knockout");
This would then be put in each file which requires knockout js, which is at least 10 files in the module i'm trying to compile currently. It just bombs out constantly saying knockout.d.ts cannot find the type HTMLElemet, Element, Document etc.
If you are using external modules (which you are if you have a top level "import" - as shown above), then you can't use the --out switch to combine multiple source files. It is a limitation that with external modules that one source file = one module. With source that is not in an external module (i.e. contributes to the 'global' scope), you can combine input source to one output JavaScript file using --out.
I have no idea about the "could not find HTMLElement" issues. If you can provide a repro (and outline which version you are using) I can take a look.