I have about 40-42 python modules to use in my lambda functions, does it mean i have to make a zip of all with my main handler.py and upload. I know about boto-3 but could not get through it as the documentation is not specific. Could someone help to find the easiest way to get around this.
Unfortunately (with exception of the aws-sdk) it's up to you to package and include all your modules when deploying to Lambda.
Luckily this is such a common task that there are a few good frameworks out there to help you automate this process, such as:
Serverless Framework (with Python Requirements): https://github.com/UnitedIncome/serverless-python-requirements
Chalice: https://github.com/aws/chalice
n.b. this isn't unique to Python- node developers have to include their node_modules, and Golang devs have to compile to a static executable.
Related
Let's say we have a package on NPM like got.
I want to figure out which core NodeJS modules this package is using as I want to polyfill it on environments (like React native, or the browser).
Is there any way where I can just get a list of which core NodeJS modules this package is using without manually searching through the source code?
I know there are ways to figure out the dependency tree -- that is not what I am asking. I specificially want to know which core NodeJS modules are being used.
There is no way to do this that is not manual unfortunately.
However, it is pretty simple to do.
Simply search all the files in your project via your IDE. CMD+F for things like crypto, buffer, stream, etc.
https://flaviocopes.com/node-core-modules/ is a list of all the core node modules
Currently, I’m experimenting with building Node.js projects (different bundlers and other kinds of stuff). And I got a simple idea about bundling Node.js to a single binary for Linux, macOS, or Windows.
There are two packages nexe and pkg both of them don’t really support native .node modules that are used by packages like bcrypt or realm. Were you performing a similar compilation? I would be extremely happy to see some real-world examples (I could not find them on GitHub).
But... for what? It's more like an idea for fun what we're able to archive with Node.js. Even if it will be working (I get it working on cjs packages without native modules) example project with hello wold may have a size above 30MB.
One cool observation is that #vercel/ncc is able to compile my project into a single directory, but it also doesn’t work without node_modules (probably, I had no environment to test it)
Update V1: Building realm is impossible in such kinds of projects, the simplest solution is to not use realm in node apps because it will crash the binary build (it's because realm is using tons of other packages such as electron or react-native which doesn't work at all when it comes to desktop platforms.
My code is available at the following repository: https://github.com/keinsell/typecraft
After days of struggling with researching how pkg works and searching for potential alternatives, I've found caxa which were working for me in a good way, and on the road, I also got pkg working fine with (actually only tested on bcrypt) native modules. My core insights on using pkg for building Node.js binaries are:
Use matrix-builds on CI to build package ex. win package from windows host, mac from macos host etc.
Be sure to include native modules to assets (there you can use require() function in JavaScript (example below) or use pkg.assets object in package.json - I don't really get it but it's contained in my issue on vercel/pkg - vercel/pkg#1473
+ require('bcrypt/lib/binding/napi-v3/bcrypt_lib.node')
export async function main() { /* ... */ }
Some modules are still build-crashing, and the good example is realm but I'll still search for a way of building it and including it into node.js binary application. All of my progress on researching Node.js builds will be available on https://github.com/keinsell/typecraft and this StackOverflow answer will be updated over time and my new discoveries.
Read this resource with examples and you will be able to compile it to a binary release. Of course, nexe is necessary but with pty.js you can successfully compile it with all the native libraries. Have a look at the source: https://www.jedi.be/blog/2013/05/14/compiling-packaging-a-nodejs-project-as-a-single-binary/
I wrote a package that uses pywin32 to sync GitLab issues with Microsoft Projects.
I would like to use readthedocs to host the documentation.
The problem is that I can't install pywin32 as a Linux environment is used there.
Any suggestion on how to get autodoc to build the documentation if a package is not available on the build machine?
The easiest way to solve this is setting autodoc_mock_imports = ["pywin32"]. This only requires setting the root package and any subsequent use your code makes of the library (calling or importing submodules, classes, etc) will be mocked by Sphinx.
Notice the mocked library won't provide any functionality besides allowing its components to be declared and importable. So your code should be structured to not have any module level execution depending on pywin32 callables, because the returns of those callbles will also be mocked.
I also found this guide that elaborates focused on ReadTheDocs builds and it suggests the same. Parallel to this problem I found the extension sphinxcontrib-mockautodoc that addresses the problem of making builds when you simultaneously have different Python versions in a given project.
I found a more or less ugly solution: Mocking the used modules and functions.
Needs quite some manual work.
Create a mocking folder in your project and create modules, classes and function stub for each class/function used.
After this edit the doc/conf.py and add:
try:
import win32com
except ImportError:
sys.path.insert(0, os.path.join(__location__, '../mocking'))
to automatically load the mocking if the real package is not available (and only then!).
Even so the solution is quite cumbersome it has one advantage. It allows for static typing, that would not be possible without.
Can someone give me some advice or links for discussion on whether I should bundle JS for backend?
I tried to Google with this title (and similar words) and I can't get any useful links.
Just want to know, say I am using latest Node.JS (es6-ready), should I bundle/compile the JS? If not, how am I suppose to use typescript/flow?
Thank you.
I feel like you are asking two different questions. I'll try to answer both.
How can I just run TypeScript code?
This is the one your question's title seems to ask ("How to use typescript/flow in nodejs without compiling it"). For this, you can use the ts-node package on npm. But it's usually not a good idea to use ts-node over just compiling when running in production because it tends not to be as fast.
How should TypeScript code get distributed to be run?
Any TypeScript code will need to get compiled from .ts files to .js files to eventually be run. Basically something like the same thing applies to Flow code.
If you plan to distribute a package written in TypeScript, you should be publishing the .js and .d.ts files together. This is so that
Your package consumers don't have to recompile your package. (they already get .js files.
Your non-TypeScript consumers don't need to install TypeScript to use your package. (they already have runnable .js files)
Your TypeScript consumers can get good type safety and completions. (they get your .d.ts files)
For more information, see the TypeScript documentation on Publishing Declaration Files.
I have installed metasploit-framework from git. It's working fine. I have followed tutorial from Metasploit Framework.
Now I would like to add more scripts to this framework, like scripts from Avg Security Scripts
I would like to know, How can we install and tell metasploit to use these scripts?
Any help would be appreciated as it may help a step forward.
Metasploit is written in Ruby and it doesn't support modules or scripts written in Python. However, Metasploit does have an RPC interface using MSGPACK.
Here are some tutorials on using Python + MSGPACK + Metasploit:
https://www.fishnetsecurity.com/6labs/blog/scripting-metasploit-python
And here is a library that SpiderLabs put out for MSF RPC written in Python: https://github.com/SpiderLabs/msfrpc
Check out the guide here.
it's pretty easy, using Metasploit's default local module search path, $HOME/.msf4/modules, and there are just a couple caveats:
Mirror the "real" Metasploit module paths
Create an appropriate category
So if you're root user it is just a case of dropping in the modules into the correct directory strucutre at /root/.msf4/modules.