I'm trying to use ghostscript to convert pdfs to tiffs in AWS Lambda. Everything works locally and I get good, fast conversions. I was told Lambda has an (old) version of ghostscript built in that is ready to use but I can't seem to be able to call on it.
I tried using the os.system command to call ghostscript directly and also tried using the pip installed version of ghostscript but neither worked.
os.system("gs -q -dBATCH -dNOPAUSE "
"-sDEVICE=tiffg4 -r600 "
"-sOutputFile=" + upload_path + " " + download_path)
I'm getting this following error :
sh: gs: command not found
and I don't understand why or how to get around it.
I followed the instructions in this doc by starting up an ec2 instance with the correct AMI (first link here for me), but that only helps you get the shared libraries out, which ended up being the following command when that part comes:
yumdownloader ghostscript.x86_64 avahi-libs.x86_64 cairo.x86_64 cups-libs.x86_64 ghostscript-fonts.noarch gnutls.x86_64 jasper-libs.x86_64 jbigkit-libs.x86_64 libXdamage.x86_64 libXfixes.x86_64 libXt.x86_64 libXxf86vm.x86_64 libdrm.x86_64 libglvnd.x86_64 libglvnd-glx.x86_64 libpciaccess.x86_64 libtiff.x86_64 libxshmfence.x86_64 mesa-dri-drivers.x86_64 mesa-filesystem.x86_64 mesa-libGL.x86_64 mesa-libglapi.x86_64 pixman.x86_64 urw-fonts.noarch
So I installed ghostscript 8.70 with yum, got the share & bin folders from /tmp/ghostscript-8.70-24.26.amzn1.x86_64/usr after installation.
The libraries you'll get by following the link above go in /lib and I put the /bin and /shared in /ghostscript in the lambda zip.
So now the zip for my node.js function looks like:
src/
ghostscript/
lib/
...etc
I changed the script to use LAMBDA_TASK_ROOT to build the binary path for gs (which is now /var/task/ghostscript/bin/gs), but got an error saying it couldn't find gs_init.ps.
In this version they used share/Resource/Init for these files, so I added this location as an environment variable on the lambda function (KEY: GS_LIB, VALUE: /var/task/ghostscript/share/ghostscript/8.70/Resource/Init) and all is working again.
Aws have just upgraded the version of linux that Lambda currently runs on.
https://aws.amazon.com/blogs/compute/upcoming-updates-to-the-aws-lambda-execution-environment/
Ghostscript was working fine on the previous AMI version however it seems to be broken on the new one.
https://forums.aws.amazon.com/thread.jspa?threadID=306787&tstart=0
Related
I realize there are other threads already, but none of the solutions are working for me. This is my error when trying to use the protoc command: Protoc.
I'm using Windows10 and a virtual environment in PyCharm using python 3.8 and pip is updated using the console command.
I installed protobuf with pip and then I've manually downloaded protoc following the installation guide on https://grpc.io/docs/protoc-installation/.
I have tried both adding the path to the protoc.exe file in my venv path variables and copying it to Tensorflow/models/research, but neither yields a positive result.
I've tried both protoc-21.1-win64.zip and protobuf-python-4.21.1.zip. I added the path to the protoc.exe folder when using the win64 version but I'm not sure which folder contains the file for the python version, although I see most people recommend to use the windows version anyways.
Through my PyCharm terminal in my C:\Users\Name\PycharmProjects\Project_Name I'm attempting to run
protoc object_detection/protos/*.proto --python_out=.
I have tried to run it from C:\Users\Name\PycharmProjects\Project_Name\venv as well if that should make a difference.
These are the most prominent threads I've attempted solutions from:
Cannot find `protoc` command
protoc object_detection/protos/*.proto: No such file or directory
Make sure you add the Protoc compiler to the path instead of adding it in PyCharm as that usually brings problems.
I have a python function which didn't work initially on cloud9(Amazon Linux2). After having installed libX11(the code uses headless chrome with selenium) the problem is gone. I just wanted to do the same on lambda so tried to add libX11 as a layer as explained in the link below but that didn't resolve the issue.
https://aws.amazon.com/premiumsupport/knowledge-center/lambda-linux-binary-package/.
My packages are here
I packaged them directly and didn't work, on my 2nd try i packaged them under /lib directory but not joy. I actually don't want docker as an option as it gives you no control on modification of the code. Any alternative guys?
Aside from the obvious "no such file or directory" which is true...there is in fact no aws file in that location, this is coming up after attempting the final installation step of the V2 AWS-CLI installation routine found here. (aws --version)
Now, I have two Ubuntu systems side by side. I ran the same install on both and one succeeded, but the other did not. On the one that succeeded, there also is no AWS file in the path suggested by the error.
Furthermore, on both systems, the folder structure and symlinks of the installation appear to be identical. I'm using the root user on both, and comparing the file permissions on the system that failed with the one that works yields identical setups.
I suspect that AWS has been setup to point to the wrong path? Both $PATH environments are also identical on each machine.
The ONLY difference between the two machines is that one is Ubuntu 18.04 and the other is 20.04.
Any ideas what would be different about that and why I am unable to run AWS commands on this stubborn box?
Short answer:
run hash aws on shell
Details:
awscli v1 points to /usr/bin/aws.
awscliv2 points to /usr/local/bin/aws
On uninstalling awscliv1 and installing awscliv2, aws was still pointing to /usr/bin/aws, while which aws resulted in /usr/local/bin/aws.
Seems bash has cached the path /usr/bin/aws for aws executable.
$ which aws
/usr/local/bin/aws
$ aws
-bash: /usr/bin/aws: No such file or directory
So running any aws command would look for /usr/bin/aws (non-existing)
-bash: /usr/bin/aws: No such file or directory
hash aws clears this cache. After this, firing aws commands uses the correct path
$ aws --version
aws-cli/2.2.32 Python/3.8.8 Linux/5.4.0-77-generic exe/x86_64.ubuntu.18 prompt/off
Follow these steps from [https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html]:
In your browser, download the macOS pkg file:
[https://awscli.amazonaws.com/AWSCLIV2.pkg]
Run your downloaded file and follow the on-screen instructions.
To verify
$ which aws
/usr/local/bin/aws
$ aws --version
aws-cli/2.7.24 Python/3.8.8 Darwin/18.7.0 botocore/2.4.5
I want to compile my .proto file into stubs, but when entering the command:
`protoc -I=. ./protos/dummy.proto
--js_out=import_style=commonjs,binary:./server --grpc_out=./server --plugin=protoc-gen-grpc=which grpc_tools_node_protoc_plugin
I got the following error :
--grpc_out: protoc-gen-grpc: %1 is not a valid Win32 application.
Thigs I have installed :
Windows 10
npm install -g grpc-tools
npm install google-protobuf
protoc
NOTE: I noticed there are a few similar questions already, but I didn't find one for Node.js, also the already asked questions all have different solutions.
On Windows, I messed around with the grpc-tools for Node a bit and came across some issues. I installed the tools (and the TS plugin) via npm install grpc-tools grpc_tools_node_protoc_ts --save-dev to my local node_modules/.bin folder. Thanks to this post, I can provide these solutions. This was my original shell script
#!/usr/bin/env bash
PROTO_DIR="./src/grpc/proto"
PROTOC_GEN_TS_PATH="./node_modules/.bin/protoc-gen-ts"
GRPC_TOOLS_NODE_PROTOC_PLUGIN="./node_modules/.bin/grpc_tools_node_protoc_plugin"
GRPC_TOOLS_NODE_PROTOC="./node_modules/.bin/grpc_tools_node_protoc"
# Generate JS and corresponding TS d.ts codes for each .proto file using the grpc-tools for Node.
$GRPC_TOOLS_NODE_PROTOC \
--plugin=protoc-gen-grpc="$GRPC_TOOLS_NODE_PROTOC_PLUGIN" \
--plugin=protoc-gen-ts="$PROTOC_GEN_TS_PATH" \
--js_out=import_style=commonjs,binary:"$PROTO_DIR" \
--ts_out="$PROTO_DIR" \
--grpc_out="$PROTO_DIR" \
-I "$PROTO_DIR" \
"$PROTO_DIR"/*.proto
If you just provide the plugin by its name, e.g. --plugin=protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin, Windows will complain about an invalid Win32 application. You can solve this issue by adding the .cmd extension:
--plugin=protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin.cmd
Unfortunately, the following issue '.' Not an internal or external command, or a runnable program or batch file arises, which indicates that Windows cannot resolve the relative path to the plugin. Therefore, you have to provide the absolute path, e.g.
--plugin=protoc-gen-grpc=C:/.../<project-dir>/node_modules/.bin/grpc_tools_node_protoc_plugin.cmd
Now the real magic happens:
Because of a simple typo (I typed --plugin=proto-gen-grpc instead of --plugin=protoc-gen-grpc), I figured out that, if you have grpc-tools and additional plugins installed in your local Node environment, you can simply omit --plugin.
It seems that grpc_tools_node_protoc will automatically lookup the executables required to generate the code specified by the output flags --grpc_out, --js_out or --ts_out in ./node_modules/.bin/.
Therefore, I updated the following LOC in my script
$GRPC_TOOLS_NODE_PROTOC \
--js_out=import_style=commonjs,binary:"$PROTO_DIR" \
--ts_out="$PROTO_DIR" \
--grpc_out="$PROTO_DIR" \
-I "$PROTO_DIR" \
"$PROTO_DIR"/*.proto
Maybe, others could share their experience and more clarifications to this issue.
You can solve this issue by adding the .cmd extension:
--plugin=protoc-gen-grpc=./node_modules/.bin/grpc_tools_node_protoc_plugin.cmd
And the following issue '.' Not an internal or external command, or a runnable program or batch file, you can solve replacing the '.' by '%cd%':
--plugin=protoc-gen-grpc=%cd%/node_modules/.bin/grpc_tools_node_protoc_plugin.cmd
SIMPLE SOLUTION :
For me the "which" command was pointing to the wrong path so I removed it and replaced it with an absolute path to the plugin instead. It looks like this :
protoc -I=. ./protos/dummy.proto
--js_out=import_style=commonjs,binary:./server --grpc_out=./server --plugin=protoc-gen-grpc=C:\Users\myUser\AppData\Roaming\npm\node_modules\grpc-tools\bin\grpc_node_plugin.exe
Explanation :
I am not sure why this error (--grpc_out: protoc-gen-grpc: %1 is not a valid Win32 application) was happening, but I have an theory... The "which" command pointed to the folder and not to the .exe file. How do I know this? When I try to only run the command (in the terminal)
which grpc_tools_node_protoc_plugin
It returns the folder
/c/Users/myUser/AppData/Roaming/npm/grpc_tools_node_protoc_plugin
As grpc-tools has been installed globally (-g flag per OP), setting the plugin path beforehand works.
GRPC_TOOLS_PLUGIN="$HOME/AppData/Roaming/npm/grpc_tools_node_protoc_plugin.cmd" && \
protoc -I=. ./protos/*.proto \
--js_out=import_style=commonjs,binary:./server \
--grpc_out=./server \
--plugin=protoc-gen-grpc=$GRPC_TOOLS_PLUGIN
Thanks to both answers before me for pointers!
Tested and found to be working on Windows 10 Pro and Git Bash.
This piece of code worked for me.
protoc --js_out=import_style=commonjs,binary:./server ./proto/dummy.proto
I'm trying to get a Lambda happy version of XPDF's pdftohtml to work but am having no luck.
So far the following has been tried:
Created Docker container running the latest amazonlinux image
I've copied the source code into this container and ran:
yum install cmake, gcc, gcc-c++, freetype-devel
Compiling the code with cmake produces a binary which executes perfectly in the container which should be the same OS and environment as Lambda.
I've verified the version of libc.so.6 as 2.26 within the container.
I've copied this into my AWS zip folder and included the following dependencies in a lib folder ready to upload:
libfreetype.so.6.10.0, libpng15.so.15, libstdc++.so.6.0.24
These dependencies are copied directly from the container used to compile the code.
Python function then connects these via
os.environ.update(dict(LD_LIBRARY_PATH='/var/task/lib'))
At the end of this, I run the function and get the following error code:
/var/task/pdftohtml: /lib64/libc.so.6: version `GLIBC_2.18' not found (required by /var/task/lib/libstdc++.so.6)
I've no idea where the GLIBC_2.18 comes from as this version isn't present in the container used to compile it.
Really stumped but keen to get it finished as it would produce a lightweight binary perfect for a Lambda function!
Where am I going wrong?
EDIT
SOLVED - see the comments below. There are two versions of AWS Linux and Lambda runs this version
I ran in an EC2 instance as one of the commenters suggested. Whilst the libstdc++.so.6.24 looked to be the right version, as it was itself compiled with a different GLIBC version, it throws an error. Compiling in EC2 from the source code worked fine. The other trick was making sure the CXX_FLAGS included -std=c++11. Thanks to those who contributed to help me solve this!
I've no idea where the GLIBC_2.18 comes from as this version isn't present in the container used to compile it.
I think you don't understand symbol version dependencies (see here).
The error message is telling you that your libstdc++.so.6 was built against GLIBC-2.18 or newer, and you are running against GLIBC-2.17 or older.
Where am I going wrong?
Your build environment is targeting something much newer than what your deployment environment contains.
You need to either find a built environment that matches your deployment target, or you need to change your deployment target to be not older than your build environment.