I am attempting to get the simplest example of creating a S3 bucket with the AWS CDK Python with no luck.
I want to put the code to create the bucket in another file (which file exists in a subdirectory).
What I am doing works with every other Python project I have developed or started.
Process:
I created an empty directory: aws_cdk_python/. I then, inside that directory ran:
$cdk init --language python to layout the structure.
This created another subdirectory with the same name aws_cdk_python/, and created a single .py within that directory where I could begin adding code in the __init__(self) method (constructor)
I was able to add code there to create a S3 bucket.
Now I created a subdirectory, with an __init__.py and a file called: create_s3_bucket.py
I put the code to create a S3 bucket in this file, in a method called 'main'
file: create_s3_bucket.py
def main(self):
<code to create s3 bucket here>
When I run the code, it will create the App Stack with no errors, but the S3 bucket will not be created.
Here is my project layout:
aws_cdk_python/
setup.py
aws_cdk_python/
aws_cdk_python_stack.py
my_aws_s3/
create_s3_bucket.py
setup.py contains the following two lines:
package_dir={"": "aws_cdk_python"},
packages=setuptools.find_packages(where="aws_cdk_python"),
The second line here says to look in the aws_cdk_python/ directory, and search recursively in sub-folders for .py files
In aws_cdk_python_stack.py, I have this line:
from my_aws_s3.create_s3_bucket import CreateS3Bucket
then in __init__ in aws_cdk_python_stack.py, I instantiate the object:
my_aws_s3 = CreateS3Bucket()
and then I make a call like so:
my_aws_s3.main() <== code to create the S3 bucket is here
I have followed this pattern on numerous Python projects before using find_packages() in setup.py
I have also run:
$python -m pip install -r requirements.txt which should pick up the dependencies pointed to in setup.py
Questions:
- Does anyone that uses the AWS CDK Python done this? or have recommendations for code organization?
I do not want all the code for the entire stack to be in aws_cdk_python_stack.py __init__() method.
Any ideas on why there no error displayed in my IDE? All dependencies are resolved, and methods found, but when I run, nothing happens?
How can I see any error messages, no error messages appear with $cdk deploy, it just creates the stack, but not the S3 bucket, even though I have code to call and create a S3 bucket.
This is frustrating, it should work.
I have other sub-directories that I want to create under aws_cdk_python/aws_cdk_python/<dir> , put a __init__.py there (empty file) and import classes in the top level aws_cdk_python_stack.py
any help to get this working would be greatly appreciated.
cdk.json looks like this (laid down from cdk init --language python
{
"app": "python app.py",
"context": {
"#aws-cdk/aws-apigateway:usagePlanKeyOrderInsensitiveId": true,
"#aws-cdk/core:enableStackNameDuplicates": "true",
"aws-cdk:enableDiffNoFail": "true",
"#aws-cdk/core:stackRelativeExports": "true",
"#aws-cdk/aws-ecr-assets:dockerIgnoreSupport": true,
"#aws-cdk/aws-secretsmanager:parseOwnedSecretName": true,
"#aws-cdk/aws-kms:defaultKeyPolicies": true,
"#aws-cdk/aws-s3:grantWriteWithoutAcl": true,
"#aws-cdk/aws-ecs-patterns:removeDefaultDesiredCount": true,
"#aws-cdk/aws-rds:lowercaseDbIdentifier": true,
"#aws-cdk/aws-efs:defaultEncryptionAtRest": true,
"#aws-cdk/aws-lambda:recognizeVersionProps": true,
"#aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true
}
}
app.py looks like this
import os
from aws_cdk import core as cdk
from aws_cdk import core
from aws_cdk_python.aws_cdk_python_stack import AwsCdkPythonStack
app = core.App()
AwsCdkPythonStack(app, "AwsCdkPythonStack",
)
app.synth()
to date: Tue 2021-12-31, this has not been solved
Not entirely sure, but I guess it depends on what your cdk.json file looks like. It contains the command to run for cdk deploy. E.g.:
{
"app": "python main.py", <===== this guy over here assumes the whole app is instantiated by running main.py
"context": {
...
}
}
Since I don't see this entrypoint present in your project structure it might be related to that.
Usually after running cdk init you should at least be able to synthesize. usually in app.py you keep your main App() definition and stack and constructs go in subfolders. Stacks are often instantiated in app.py and the constructs are instantiated in the stack definition files.
I hope it helped you a bit further!
Edit:
Just an example of a working tree is shown below:
aws_cdk_python
├── README.md
├── app.py
├── cdk.json
├── aws_cdk_python
│ ├── __init__.py
│ ├── example_stack.py
│ └── s3_stacks <= this is your subfolder with s3 stacks
│ ├── __init__.py
│ └── s3_stack_definition.py <== file with an s3 stack in it
├── requirements.txt
├── setup.py
└── source.bat
aws_cdk_python/s3_stacks/s3_stack_definition.py:
from aws_cdk import core as cdk
from aws_cdk import aws_s3
class S3Stack(cdk.Stack):
def __init__(self, scope: cdk.Construct, construct_id: str, **kwargs) -> None:
super().__init__(scope, construct_id, **kwargs)
bucket = aws_s3.Bucket(self, "MyEncryptedBucket",
encryption=aws_s3.BucketEncryption.KMS
)
app.py:
from aws_cdk import core
from aws_cdk_python.s3_stacks.s3_stack_definition import S3Stack
app = core.App()
S3Stack(app, "ExampleStack",
)
app.synth()
So I tried many things (from SO and more) getting my tests running but nothing worked this is my current code:
test.py which I call to run the tests: python3 ./src/preprocess/python/test.py
import unittest
if __name__ == '__main__':
testsuite = unittest.TestLoader().discover('.')
unittest.TextTestRunner(verbosity=2).run(testsuite)
the test file looks like this:
import unittest
from scrapes.pdf import full_path_to_destination_txt_file
print(full_path_to_destination_txt_file)
class PreprocessingTest(unittest.TestCase):
def path_txt_appending(self):
self.assertEqual(full_path_to_destination_txt_file(
"test", "/usr/test"), "/usr/test/test.txt")
if __name__ == '__main__':
unittest.main(verbosity=2)
But the output is always like this:
python3 ./src/preprocess/python/test.py
----------------------------------------------------------------------
Ran 0 tests in 0.000s
OK
Additional Information:
As you can see I call this not from my root directory. The test folder is in ./src/preprocess/python/test/ and has a __init__.pyfile included (there is also a init file on the level of test.py)
it would be okay for me if I have to code down all the calls for all the tests I just want to finish this
automatic search with -t does not work either so I thought the more robust method here with test.py would work...
using this framework is a requirement I have to follow
test_preprocessing.py is in the test folder and from scrapes.pdf import full_path_to_destination_txt_filescrapes is a module folder on the same level as test
When I call the single unit test directly in the command line it fails because of the relative import. But using the test.py (obviously) finds the modules.
What is wrong?
By default, unittest will only execute methods whose name starts with test:
testMethodPrefix
String giving the prefix of method names which will be interpreted as test methods. The default value is 'test'.
This affects getTestCaseNames() and all the loadTestsFrom*() methods.
from the docs.
Either change that attribute or (preferably) prefix your method name with test_.
Need a little help on aws lambda if ya'll came across this issue with deployment package uploading in aws lambda.
Regards,
xxSoumya----
[find snippet of issue ] [1]: https://i.stack.imgur.com/2QeGe.png
Your deployment package structure should be something like this,
main.py <---------- lambda entry/handler file
(can be name anything, just config your aws lambda to use it)
demo.py
mylib/
__init__.py
foo.py
bar.py
numpy/
...
pandas/
...
If demo.py is in a another folder from where your main lambda handler file is locate, then you will need to put a "__init__".py in that folder.
main.py <---------- lambda entry/handler file
mylib/
__init__.py
demo.py
foo.py
bar.py
numpy/
...
pandas/
...
Now in main, you will need to do, from mylib.demo import .....
I have the following code;
#!/usr/bin/env python3
import yaml
with open('config.yml', 'r') as config_file:
config = yaml.load(config_file)
The file is called __init__.py which is in the directory ~/bin/myprogram/myprogram/ and in the same directory, I have a file called config.yml
My symlink is as follows;
user$ ls -la /usr/local/bin/
lrwxr-xr-x 1 user admin 55 27 Nov 13:25 myprogram -> /Users/user/bin/myprogram/myprogram/__init__.py
Every time I run myprogram, I get the error FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'. I believe this is because the config.yml is not in /usr/local/bin/. What is the best way to work around this issue?
You can use __file__ to access the location of the __init__.py file when executing code in that file. It returns the full path, but care has to be taken as it may be the .pyc (or .pyo) version. Since you are using Python3 I would use the pathlib module:
import yaml
from pathlib import Path
my_path = Path(__file__).resolve() # resolve to get rid of any symlinks
config_path = my_path.parent / 'config.yaml'
with config_path.open() as config_file:
config = yaml.safe_load(config_file)
Please note:
If you have to use PyYAML, use safe_load(), even PyYAML's own documentation indicates .load() can be unsafe. It almost never necessary to use that. And in the unlikely event that safe_load() cannot load your config, e.g. if it has !!python/... tags, you should explicitly add register the classes that you actually need to the SafeLoader).
Since September 2006 the recommended extension for YAML files has been .yaml
After you create your first project with django-admin startproject mysite you get a directory that looks like this:
mysite/
manage.py - script for running tasks
mysite/
__init__.py
settings.py - for general setting configuration
urls.py - for routing configuration
wsgi.py - for webserver configuration
What I want is 3 directories for me to put various files that are usable across all django apps.
mysite/
manage.py
javascript_css/
...
html_templates/
...
custom_modules/
...
mysite/
__init__.py
settings.py
urls.py
wsgi.py
What settings would I need to change in settings.py in order to configure this behaviour and how would I reference the javascript_css/html_templates/custom_modules in my django apps?
Thanks for any help.
You can find everything in the django docs:
static files
html templates
And it's probably better to keep folder named as static and templates, it's more readable.