This is my cookiecutter.json file:
{
"day": "1",
"directory_name": "day-{{ cookiecutter.day }}"
}
Now I only want to be prompted for the day, but not for the directory_name which is derived from it. How do I get that to happen?
The documentation for no_input is less than helpful.
The no-input option is general option, allowing you to use all the defaults and skipping the questions overall.
What you need is a private variable
{
"day": "1",
"__directory_name": "day-{{ cookiecutter.day }}"
}
Related
I'm trying to execute an query on my users of my Azure B2C Active Directory.
So far everything works fine with the following query:
https://graph.windows.net/myTentant/users?$filter=
startswith(displayName,'test')%20
or%20startswith(givenName,'test')%20
or%20startswith(surname,'test')%20
or%20startswith(mail,'test')%20
or%20startswith(userPrincipalName,'test')
&api-version=1.6
The thing about that is, that this properties are just simple values like this:
"displayName: "testValue",
"givenName": "testValue",
"displayName: "testValue",
"surname": "testValue",
"mail: "testValue",
"userPrincipalName": "testValue",
In my case I need to use one more statement, in which I need to check an array if it contains 'test' like the others. This array look like that:
"signInNames": [
{
"type": "emailAddress",
"value": "test#mail.com"
}, {
"type": "emailAddress",
"value": "test2#mail.com"
}
]
I Already search in the official documentation but had no luck....
Any ideas?
In theory, we should use the following format to determine whether the value starts with "test".
GET https://graph.windows.net/myorganization/users?$filter=signInNames/any(c:startswith(c/value, 'test'))
Unfortunately, it will show an error: value only supports equals-match. PrefixMatch is not supported.
And the contains string operator is currently not supported on any Microsoft Graph resources. So we can't use contains neither.
You need to use equal to find the exact match dataļ¼
GET https://graph.windows.net/myorganization/users?$filter=signInNames/any(c:c/value eq '***')
It is not a solution. But there seems not to be a way to meet your needs.
Maybe you could query all the signInNames and handle them in your code.
I want to reference a different model(as discribed here: https://loopback.io/doc/en/lb2/Embedded-models-and-relations.html) but the by a nested id:
{
"name" : "person",
...
"relations": {
"cars": {
"type": "referencesMany",
"model": "car",
"foreignKey": "cars.id"
}
}
Person json will actually be something like:
{
...
cars: [{"id": 1, "name": "car1"}, ...]
}
And car model will be the full car details
Do I have to write my own remote method to do this?
Yosh DaafVader,
I've came accross this issue also and took time to find a solution ^^ but actually you just have to play with the parameter options inside your target relation property. The documentation states how the relation should be defined (sure the loopback cli does not include in version 3.x yet the way to use embeds nor references).
In your person model you have to change the foreignKey and to add the following options to be able to only use id to reference cars.
{
"name" : "person",
...
"relations": {
"cars": {
"type": "referencesMany",
"model": "car",
"foreignKey": "",
"options": {
"validate": true,
"forceId": true
}
}
}
Now you will be able to see in the explorer the new routes to add, remove and see the cars that belongs to the target person.
[Edit]
the foreignKey shall be blank, in order to be able to add items properly in the list of cars, or you can test and give some feedbacks about it
The validate option ensures the id exists in your database
forceId option will ensure it accepts only ids as a parameter
[/Edit]
Hope it will help :)
Cheers
In Cloudformation I have two stacks (one nested).
Nested stack "ec2-setup":
{
"AWSTemplateFormatVersion" : "2010-09-09",
"Parameters" : {
// (...) some parameters here
"userData" : {
"Description" : "user data to be passed to instance",
"Type" : "String",
"Default": ""
}
},
"Resources" : {
"EC2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"UserData" : { "Ref" : "userData" },
// (...) some other properties here
}
}
},
// (...)
}
Now in my main template I want to refer to nested template presented above and pass a bash script using the userData parameter. Additionally I do not want to inline the content of user data script because I want to reuse it for few ec2 instances (so I do not want to duplicate the script each time I declare ec2 instance in my main template).
I tried to achieve this by setting the content of the script as a default value of a parameter:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Parameters" : {
"myUserData": {
"Type": "String",
"Default" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash \n",
"yum update -y \n",
"# Install the files and packages from the metadata\n",
"echo 'tralala' > /tmp/hahaha"
]]}}
}
},
(...)
"myEc2": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL": "s3://path/to/ec2-setup.json",
"TimeoutInMinutes": "10",
"Parameters": {
// (...)
"userData" : { "Ref" : "myUserData" }
}
But I get following error while trying to launch stack:
"Template validation error: Template format error: Every Default
member must be a string."
The error seems to be caused by the fact that the declaration { Fn::Base64 (...) } is an object - not a string (although it results in returning base64 encoded string).
All works ok, if I paste my script directly into to the parameters section (as inline script) when calling my nested template (instead of reffering to string set as parameter):
"myEc2": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL": "s3://path/to/ec2-setup.json",
"TimeoutInMinutes": "10",
"Parameters": {
// (...)
"userData" : { "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash \n",
"yum update -y \n",
"# Install the files and packages from the metadata\n",
"echo 'tralala' > /tmp/hahaha"
]]}}
}
but I want to keep the content of userData script in a parameter/variable to be able to reuse it.
Any chance to reuse such a bash script without a need to copy/paste it each time?
Here are a few options on how to reuse a bash script in user-data for multiple EC2 instances defined through CloudFormation:
1. Set default parameter as string
Your original attempted solution should work, with a minor tweak: you must declare the default parameter as a string, as follows (using YAML instead of JSON makes it possible/easier to declare a multi-line string inline):
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
myUserData:
Type: String
Default: |
#!/bin/bash
yum update -y
# Install the files and packages from the metadata
echo 'tralala' > /tmp/hahaha
(...)
Resources:
myEc2:
Type: AWS::CloudFormation::Stack
Properties
TemplateURL: "s3://path/to/ec2-setup.yml"
TimeoutInMinutes: 10
Parameters:
# (...)
userData: !Ref myUserData
Then, in your nested stack, apply any required intrinsic functions (Fn::Base64, as well as Fn::Sub which is quite helpful if you need to apply any Ref or Fn::GetAtt functions within your user-data script) within the EC2 instance's resource properties:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
# (...) some parameters here
userData:
Description: user data to be passed to instance
Type: String
Default: ""
Resources:
EC2Instance:
Type: AWS::EC2::Instance
Properties:
UserData:
"Fn::Base64":
"Fn::Sub": !Ref userData
# (...) some other properties here
# (...)
2. Upload script to S3
You can upload your single Bash script to an S3 bucket, then invoke the script by adding a minimal user-data script in each EC2 instance in your template:
AWSTemplateFormatVersion: "2010-09-09"
Parameters:
# (...) some parameters here
ScriptBucket:
Description: S3 bucket containing user-data script
Type: String
ScriptKey:
Description: S3 object key containing user-data script
Type: String
Resources:
EC2Instance:
Type: AWS::EC2::Instance
Properties:
UserData:
"Fn::Base64":
"Fn::Sub": |
#!/bin/bash
aws s3 cp s3://${ScriptBucket}/${ScriptKey} - | bash -s
# (...) some other properties here
# (...)
3. Use preprocessor to inline script from single source
Finally, you can use a template-preprocessor tool like troposphere or your own to 'generate' verbose CloudFormation-executable templates from more compact/expressive source files. This approach will allow you to eliminate duplication in your source files - although the templates will contain 'duplicate' user-data scripts, this will only occur in the generated templates, so should not pose a problem.
You'll have to look outside the template to provide the same user data to multiple templates. A common approach here would be to abstract your template one step further, or "template the template". Use the same method to create both templates, and you'll keep them both DRY.
I'm a huge fan of cloudformation and use it to create most all my resources, especially for production-bound uses. But as powerful as it is, it isn't quite turn-key. In addition to creating the template, you'll also have to call the coudformation API to create the stack, and provide a stack name and parameters. Thus, automation around the use of cloudformation is a necessary part of a complete solution. This automation can be simplistic ( bash script, for example ) or sophisticated. I've taken to using ansible's cloudformation module to automate "around" the template, be it creating a template for the template with Jinja, or just providing different sets of parameters to the same reusable template, or doing discovery before the stack is created; whatever ancillary operations are necessary. Some folks really like troposphere for this purpose - if you're a pythonic thinker you might find it to be a good fit. Once you have automation of any kind handling the stack creation, you'll find it's easy to add steps to make the template itself more dynamic, or assemble multiple stacks from reusable components.
At work we use cloudformation quite a bit and are tending these days to prefer a compositional approach, where we define the shared components of the templates we use, and then compose the actual templates from components.
the other option would be to merge the two stacks, using conditionals to control the inclusion of the defined resources in any particular stack created from the template. This works OK in simple cases, but the combinatorial complexity of all those conditions tends to make this a difficult solution in the long run, unless the differences are really simple.
Actually I found one more solution than already mentioned. This solution on the one hand is a little "hackish", but on the other hand I found it to be really useful for "bash script" use case (and also for other parameters).
The idea is to create an extra stack - "parameters stack" - which will output the values. Since outputs of a stack are not limited to String (as it is for default values) we can define entire base64 encoded script as a single output from a stack.
The drawback is that every stack needs to define at least one resource, so our parameters stack also needs to define at least one resource. The solution for this issue is either to define the parameters in another template which already defines existing resource, or create a "fake resource" which will never be created becasue of a Condition which will never be satisified.
Here I present the solution with fake resource. First we create our new paramaters-stack.json as follows:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Description": "Outputs/returns parameter values",
"Conditions" : {
"alwaysFalseCondition" : {"Fn::Equals" : ["aaaaaaaaaa", "bbbbbbbbbb"]}
},
"Resources": {
"FakeResource" : {
"Type" : "AWS::EC2::EIPAssociation",
"Condition" : "alwaysFalseCondition",
"Properties" : {
"AllocationId" : { "Ref": "AWS::NoValue" },
"NetworkInterfaceId" : { "Ref": "AWS::NoValue" }
}
}
},
"Outputs": {
"ec2InitScript": {
"Value":
{ "Fn::Base64" : { "Fn::Join" : ["", [
"#!/bin/bash \n",
"yum update -y \n",
"# Install the files and packages from the metadata\n",
"echo 'tralala' > /tmp/hahaha"
]]}}
}
}
}
Now in the main template we first declare our paramters stack and later we refer to the output from that parameters stack:
{
"AWSTemplateFormatVersion": "2010-09-09",
"Resources": {
"myParameters": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL": "s3://path/to/paramaters-stack.json",
"TimeoutInMinutes": "10"
}
},
"myEc2": {
"Type": "AWS::CloudFormation::Stack",
"Properties": {
"TemplateURL": "s3://path/to/ec2-setup.json",
"TimeoutInMinutes": "10",
"Parameters": {
// (...)
"userData" : {"Fn::GetAtt": [ "myParameters", "Outputs.ec2InitScript" ]}
}
}
}
}
Please note that one can create up to 60 outputs in one stack file, so it is possible to define 60 variables/paramaters per single stack file using this technique.
I'm trying to write script in bash, for AWS Autoscaling Group. That means even if instance is terminated, Autoscaling Group reinstall instance and all packages from tags by Package name and Value Package number.
Here is LaunchConfiguration group from AWS Cloudformation template:
"WorkerLC": {
"Type" : "AWS::AutoScaling::LaunchConfiguration",
"Properties" : {
"ImageId": {"Ref" : "SomeAMI"},
"InstanceType" : "m3.medium",
"SecurityGroups" : [{"Ref": "SecurityGroup"}],
"UserData" : {
"Fn::Base64": {
"Fn::Join": [ "", [
{"Fn::Join": ["", ["Engine=", {"Ref": "Env"},".app.net"," \n"]]},
{"Fn::Join": ["", [
"#!/bin/bash\n",
"cd /app/\n",
"./worker-install-package.sh"
]]}
]]
}
}
}
}
And I want to take from tags of AutoscalingGroup like that:
"Worker": {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties": {
"LaunchConfigurationName": {"Ref": "Worker"},
"LoadBalancerNames": [{"Ref": "WorkerELB"}],
"AvailabilityZones": {"Ref": "AZs"},
"MinSize" : "1",
"MaxSize" : "1",
"HealthCheckGracePeriod": 300,
"Tags" : [
{"Key": "WorkersScalingGroup", "Value": {"Fn::Join": ["", ["Offering-", {"Ref": "Env"} "-Worker-1"]]}, "PropagateAtLaunch": true},
{"Key": "EIP", "Value": {"Ref": "WorkerIP"}, "PropagateAtLaunch": true},
{"Key": "Environment", "Value": {"Ref": "Env"}, "PropagateAtLaunch": true}
]
}
}
So, now is hard part. Now I tried to find in Userdata tags with text "worker". Because I have couple types of instances and each one comes with other couple packages.
It first time when I wrote something in bash.
Here is worker-install-package.sh:
#read tag for the installed package
EC2_REGION='us-east-1'
AWS_ACCESS_KEY='xxxxx'
AWS_SECRET_ACCESS_KEY='xxxxxxxxxxxxxxxxxx'
InstanceID=`/usr/bin/curl -s http://169.254.169.254/latest/meta-data/instance-id`
PackageName=`/opt/aws/apitools/ec2/bin/ec2-describe-tags -O $AWS_ACCESS_KEY -W $AWS_SECRET_ACCESS_KEY --filter resource-id=$InstanceID --filter key='worker' | cut -f5`
while read line
if [ "$PackageNmae" = "worker" ]; then
sudo -- sh -c "./install-package.sh ${PackageName} ${Value}"
/opt/aws/apitools/ec2/bin/ec2-create-tags $InstanceID -O $AWS_ACCESS_KEY -W $AWS_SECRET_ACCESS_KEY --tag "worker-${PackageName}"=$Value
fi
done
I have two questions. First, if I'm doing that in right way. And second, is How I can take value of package name value(it some number of package version)?
Thanks!
First of all, as a best practice don't include your AWS keys in your script. Instead attach a role to your instance at launch (this can be done in the launch configuration of your autoscaling group).
Secondly, what you do is one way to go, and it can definitely work. Another way (proper but slightly more complex) to achieve this would be to use a tool like puppet or AWS opsworks.
However, I don't really get what you are doing in your script, which seem overcomplicated for this purpose: why don't you include your package name in your userdata script? If this is only a matter of agility when it comes to change/update the script, you can outsource this script to an S3 bucket and have the instances download / execute it at creation time. This way you don't need to read from the tags.
That been said, and more as a comment, if you do want to remain reading tags, then I don't really understand you script. If you do need help on the script, please provide more details in that sense (e.g debug samples etc):
when you evaluate PackageName, does this work?
PackageName=`/opt/aws/apitools/ec2/bin/ec2-describe-tags -O $AWS_ACCESS_KEY -W $AWS_SECRET_ACCESS_KEY --filter resource-id=$InstanceID --filter key='worker' | cut -f5`
not sure why you filter with "key=worker", and not "WorkersScalingGroup"
Then you call the below if condition:
if [ "$PackageNmae" = "worker" ]; then
(I assume there is typo here, and should be PackageName) and right below you call:
"worker-${PackageName}"
which would give "worker-worker"?
I am investigate possibility to switch to ElasticSearch from SphinxSearch.
What is good about SphinxSearch - full-text search just work out of the bot on pretty good level. Make it work on ElasticSearch appeared not as easy as I expected.
In my project I have search box with typeahead, means I stype Clint E and see dropdown with results including Clint Eastwood on the first place. Type robert down and see Robert Downey Jr. on the first place. All this I achieved with SphinxSearch out of the box just providing it my DB credentials and SQL query to pull the necessary fields.
On the other hand, with ElasticSearch I can't get satisfying results even after a day of reading about Fuzzy Like This Query, matching, partial matching and other. A lot of information but it does not make task easier. I feel like I need to be PhD in search just to make it work at simplest level.
So far I ended up with such configuration
{
"settings": {
"analysis": {
"analyzer": {
"stem": {
"tokenizer": "standard",
"filter": [
"standard",
"lowercase",
"stop",
"porter_stem"
]
}
}
}
},
"mappings": {
"movies": {
"dynamic": true,
"properties": {
"title": {
"type": "string",
"analyzer": "stem"
}
}
}
}
}
The Query look like this:
{
"query": {
"query_string": {
"query": "clint eastw"
"default_field": "title"
}
}
}
But quality of search in this case is not satisfying at all - back to my example, it can not find Clint Eastwood profile until I type his name completely.
Then I tried to use
{
"query": {
"fuzzy_like_this": {
"fields": [
"title"
],
"like_text": "clint eastw",
"max_query_terms": 25,
"fuzziness": 0.5
}
}
}
It helps but not much, now I can find what I need with shorter request clint eastwo and after some manipulations with parameters with clint eastw but still not encouraging.
So I wonder, is there a simple recipe how to cook full-text search with ElasticSearch and get decent quality of results. I spend a day reading but didn't find the solution.
Couple of images to demonstrate what I am talking about:
Elastic, name almost complete but no expected result, note that there is no better match as well.
One letter after, elastic found it!
At the same moment Sphinx shining :)
Elasticsearch ships with auto completion suggester.
You need not put this into query functioanility , the way it works is on token level and not on partial token level.
Go for completion suggester , it also have support for fuzzy logic.