I know there are a lot of similar questions out there, but none of them has a proper answer. I am trying to deploy my code using GitLab cicd pipeline. While executing the deployment stage, my pipeline failed and got this error.
My serverless.yml has this code related to exclude
package:
patterns:
- '!nltk'
- '!node_modules/**'
- '!package-lock.json'
- '!package.json'
- '!__pycache__/**'
- '!.gitlab-ci.yml'
- '!tests/**'
- '!README.md'
The error I am getting is
Serverless Error ----------------------------------------
No file matches include / exclude patterns
I forgot to mention, I have a nltk layer which I am deploying in the same serverless.yml as my lambda function and other resources.
I am not sure what has to be done exactly to get rid of the error. Any help would be appreciated. thank you.
Your directives do not define any inclusive patterns. Perhaps you want to list the files & directories you need packaged. Each directive builds on the next.
Something like:
package:
patterns:
- "**/**"
- '!nltk'
- '!node_modules/**'
- '!package-lock.json'
- '!package.json'
- '!__pycache__/**'
- '!.gitlab-ci.yml'
- '!tests/**'
- '!README.md'
See https://www.serverless.com/framework/docs/providers/aws/guide/packaging/#patterns
Related
I'm setting up a test framework for my project. After configuring pytest and coverage, it shows 100% coverage for all the python files, but there are no tests yet. I'm guessing it is counting the source code also as tests and giving 100% coverage for all the scripts.
Apologies, I cannot really post the image from my work account. Please let me know if there is anything wrong with my configuration, or the way I'm running it.
Project structure:
etl-orchestrator
- etl_api
- com.etl.api
- com.etl.tests
- etl_core
- com.etl.core
- com.etl.tests
- etl_services
- com.etl.services
- com.etl.tests
- .coveragerc
- pytest.ini
- setup.py
.coveragerc
[run]
source = .
omit =
*/__init__.py
*tests*
*virtualenvs*
.venv/*
*build/*
pytest.ini
[pytest]
python_files = tests/*.py
addopts = --cov-config=.coveragerc --cov=etl_api --cov=etl_core --cov=etl_services
command to run:
cd <project root directory>
pytest
I did as much research as I could but I can`t seem to find a way to structure my folder the way I want to.
My folder structure looks like this:
aws-lambdas
database_credentials.yml (just a file to read the creds from in a var)
serverless.yml
functions
ETLTool
somefile1.py
somefile2.py
lambda_function.py
ETLToolFullLoadSLS.yml
ETLToolSLS.yml
TriggerSnowflakeETL
somefile1.py
somefile2.py
lambda_function.py
TriggerSnowflakeETLSLS.yml
What I want to do is to pull in all the .yml from inside the functions folder into the serverless.yml at the root folder. My main serverless.yml file looks as such:
service: SnowflakePoc
frameworkVersion: '2'
custom:
database_credentials: ${file(./database_credentials.yml):database_credentials}
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
timeout: 90
memorySize: 2048
stage: dev
region: eu-west-2
vpc:
securityGroupIds:
- sg-013059b0cbf4054b5
- sg-02c6fcaa9f2bfac7f
subnetIds:
- subnet-04aa5cacdb8d9d077
- subnet-0ea7eb629fbc6f6a8
iam:
role: arn:aws:iam::309161096106:role/LamdaRDSAccess
functions:
- ${file(./functions/ETLTool/ETLToolSLS.yml)}
- ${file(./functions/ETLTool/ETLToolFullLoadSLS.yml)}
- ${file(./functions/TriggerSnowflakeETL/TriggerSnowflakeETLSLS.yml)}
plugins:
- serverless-python-requirements
The issue is that the whole functions/* folder is picked up by each of the lambdas even if I have something like this in each inner function yml file.
TriggerETLTool:
timeout: 600
memorySize: 5000
reservedConcurrency: 3
handler: functions/TriggerSnowflakeETL/lambda_function.lambda_handler
layers:
- arn:aws:lambda:eu-west-2:309161096106:layer:Snowflake:3
- arn:aws:lambda:eu-west-2:309161096106:layer:DatabaseUtilities:5
package:
patterns:
- '!functions/TriggerSnowflakeETL/**'
- functions/TriggerSnowflakeETL/lambda_function.py
Inside AWS it looks like this:
Pic from AWS Lambda Source Code
Is there a better pattern than having 1 directory per lambda?
I would like just the files inside each function to be at the root of my lambdas without them being inside a folder once they reach AWS as they are in the image. Also I`d like to have just the files from each inner-function inside the functions folder rather than the whole functions directory.
If you want to package each function individually, you'll need two things (one of which you've already done)
Configure your serverless.yml file to package functions individually:
service: SnowflakePoc
package:
individually: true
provider:
...
In each function, specify the pattern to correctly zip just that part (you've already done this)
Packaging individually is configurable globally or at a per-function level, so you can choose what's best for you.
You can find more information in the documentation
I like to keep one node_modules for multiple Angular projects since it does not add around 500mb data per project into my HDD - ( yes this question has been asked many times before , i tried almost all tips in them but did not work)
This is my folder structure
:~/Public/node_modules/angular/ <-- this is where contents of usual node_modules folder end up
:~/Public/angular/project1/ <- ng new -skip-install
:~/Public/angular/project2/ <- copy of above and not running above command
and i have modified the usual 2nd line of angular.json to this
"$schema": "../../node_modules/angular/#angular/cli/lib/config/schema.json",
and when i do ng serve
i get the following error
An unhandled exception occurred: Cannot find module '#angular-devkit/build-angular/package.json'
Require stack:
- /usr/local/lib/node_modules/#angular/cli/node_modules/#angular-devkit/architect/node/node-modules-architect-host.js
- /usr/local/lib/node_modules/#angular/cli/node_modules/#angular-devkit/architect/node/index.js
- /usr/local/lib/node_modules/#angular/cli/models/architect-command.js
- /usr/local/lib/node_modules/#angular/cli/commands/serve-impl.js
- /usr/local/lib/node_modules/#angular/cli/node_modules/#angular-devkit/schematics/tools/export-ref.js
- /usr/local/lib/node_modules/#angular/cli/node_modules/#angular-devkit/schematics/tools/index.js
- /usr/local/lib/node_modules/#angular/cli/utilities/json-schema.js
- /usr/local/lib/node_modules/#angular/cli/models/command-runner.js
- /usr/local/lib/node_modules/#angular/cli/lib/cli/index.js
- /usr/local/lib/node_modules/#angular/cli/lib/init.js
- /usr/local/lib/node_modules/#angular/cli/bin/ng
See "/tmp/ng-kyWfPF/angular-errors.log" for further details.
How do i get rid of this and get the functionality i desire of having one node_modules for many projects like i have mentioned above ?
If I understood correctly the node_modules and angular are on the same level, so in this case, you set incorrect path in angula.json. You need
"$schema": "./node_modules/angular/#angular/cli/lib/config/schema.json"
I am pretty new to puppet. I configured a hiera file, whose path is /etc/puppetlabs/puppet/hiera.yaml, as so
version: 5
hierarchy: []
backends:
- yaml
yaml:
- datadir: /etc/puppetlabs/puppet/some_dir
and I get this error
Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://docs.puppet.com/puppet/5.3/reference/deprecated_language.html
(file & line not available)
Error: Evaluation Error: Error while evaluating a Function Call, Lookup of key 'user_dir' failed: The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'backends'
The Lookup Configuration at '/etc/puppetlabs/puppet/hiera.yaml' has wrong type, unrecognized key 'yaml' at /etc/puppetlabs/code/environments/production/manifests/site.pp:30:17 on node puppet,some_cluster_DNS.internal
Initially, I had those kinds of format for the keys :backends: :yaml: but it seemed it is not regular one for the 5 version, so that I deleted the : sign
Someone has an idea ?
First, in terms of that warning, you should definitely switch over to the Puppet lookup function from the Hiera hiera functions if you are using Hiera >= 4: https://puppet.com/docs/puppet/4.10/hiera_use_function.html
Second, in terms of that error, I would consult the documentation on how to setup a Hiera 5 config file: https://puppet.com/docs/puppet/4.10/hiera_config_yaml_5.html
Using the proper format, your config file would look like:
# /etc/puppetlabs/puppet/hiera.yaml
version: 5
defaults:
- data_hash: yaml_data
- datadir: /etc/puppetlabs/puppet/some_dir
hierarchy: []
What you are trying to do on the last line (specify a specific datadir for the yaml_data backend) is not allowed in Hiera 5. If you want to specify a datadir for a specific backend, then you need to specify a level of the hierarchy for just that backend (or just that backend's datadir; you can customize in several depths of matrices with it) and specify a datadir there. For example:
hierarchy:
- name: yaml data
data_hash: yaml_data
datadir: /etc/puppetlabs/puppet/some_dir
paths:
- "%{trusted.certname}.yaml"
- common.yaml
I have already:
Downloaded the Cucumber Java, Gherkin plugin
I already have the steps and features directories:
My directory structure looks like this:
- test
- java
- features
- featureSet1
- oneFeature.feature
- anotherFeature.feature
- featuresSet2
- twoFeature.feature
- CucumberTests.java
- steps
- step1.java
- step2.java
Under the features folder, I have a file called, CucumberTests.java. I'm able to run the tests via mvn test but the red error marks reallllly annoy me.
I have these tags in CucumberTest.java, which is supposed to run the tests:
#RunWith(Cucumber.class)
#CucumberOptions(plugin = { "pretty", "html:target/surefire-
reports/cucumber", "json:target/surefire-
reports/cucumberOriginal.json"},
features = {"src/test/java/features/featuresSet1",
"src/test/java/features/featuresSet2",
},
tags = {"~#ignore"},
glue = {"steps"})
The issue is from Substeps IntelliJ Plugin that IntelliJ suggests you install when it locates a .feature file inside your project.
Just ignore this extension when it pops up or uninstall if you already have it.
Cucumber for Java and Gherkin should be enough.