Passing Jinja variables in airflow params dict - python-3.x

I am trying to use templated params dict in SnwoflakeOperator. I have 2 tasks one to create a S3 object and then delete and replace the content of the table with s3 object content
execution_date = "{{ execution_date.int_timestamp | int }}" # built in macro
task1 = PythonOperator(task_id='s3_create',
op_kwargs={'s3_key': f'{execution_date}_{file}'})
task_2 = SnowflakeOperator(task_id='load_data',
sql=["""copy into {{params.table}} from {{ params.stage }}
files = ('{{ params.files }}')
file_format = 'csv'
"""]
params={'table':'test_table',
'files'=f'{execution_date}_{file}'}
)
task_1 >> task_2
It is not rendering as expected and have tried almost all formatting combinations.
Note: I cannot use S3ToSnowflakeOperator as I have do more than just copying the contents into the table.

sql is templated filed thus it can automatically render execution_date (since it's Airflow build-in macro) there is no need to set it via params. The params is used to render python variables or just strings that you want to pass.
You can just set it directly as:
file = "test_file.csv"
SnowflakeOperator(
task_id='load_data',
sql=""" copy into {{ params.table }} from {{ params.stage }}
files = ('{{ execution_date }}_{{ params.file }}')
file_format = csv
""",
params={
'table': 'test_table',
'stage': 'test_stage',
'file': file
}
)
It will be rendered as:
I'm not sure this is the file name you wanted but you can change execution_date to any other macro you wish.
Edit:
You can replace {{ execution_date }} with {{ execution_date.int_timestamp | int }} it will produce:

Related

How to create global variable as html

$value = sha1("anything");
$secure = '<input type = "hidden" value = "'.$value.'"/>';
$this->twig->addGlobal('anything',$secure);
After this I access this variable like this on template
{{ anything }}
The big surprise print this variable don't escaped like this
<input type = "hidden" value = "8867c88b56e0bfb82cffaf15a66bc8d107d6754a"/>
I want to print as html tag. But it worked when I used {{ anything|raw }}, I wish it worked without using raw filter
You can mark the variable as safe, without the filter raw, if you wrap it in a Twig\Markup instance, e.g.
$value = sha1("anything");
$secure = new \Twig\Markup('<input type = "hidden" value = "'.$value.'"/>', 'UTF-8');
$this->twig->addGlobal('anything',$secure);

ansible - changing a specific line in configuration file using a jinja template

I have an .ini config file that I'd like to template, but I only want to manipulate one or two lines with the jinja2 files.
example config file:
[configuration]
config1 = this_is_fine
config2 = to_be_templated
config3 = to_be_templated
config_4 = this_is_fine
What would be the best way to define a configuration.ini.j2 that would only change
config2 = {{ assigned_value }}
config3 = {{ assigned_value }}
while keeping the formatting and remaining configuration in place when running the playbook?
Q: "Manipulate one or two lines with the Jinja2 files."
A: "One or two lines" means a block. Use blockinfile to manipulate a block in a file. You'll need markers to do this, e.g. given the file
shell> cat conf.ini
[configuration]
config1 = this_is_fine
config2 = to_be_templated
config3 = to_be_templated
config_4 = this_is_fine
the two tasks below put markers around the block starting config2 and ending config3
- replace:
path: conf.ini
regexp: '(config2.*)'
replace: |-
{{ '#' }} BEGIN ANSIBLE MANAGED BLOCK c2-3
\g<1>
- replace:
path: conf.ini
regexp: '(config2.*[\s\S]*?config3.*)'
replace: |-
\g<1>
{{ '#' }} END ANSIBLE MANAGED BLOCK c2-3
gives
shell> cat conf.ini
[configuration]
config1 = this_is_fine
# BEGIN ANSIBLE MANAGED BLOCK c2-3
config2 = to_be_templated
config3 = to_be_templated
# END ANSIBLE MANAGED BLOCK c2-3
config_4 = this_is_fine
These two tasks are not idempotent. Run them only once. Either put them in a separate playbook to set up the project or test the presence of the markers.
Now you can use the template
shell> cat conf.ini.j2
config2 = {{ assigned_value }}
config3 = {{ assigned_value }}
For example, the task below
- blockinfile:
path: conf.ini
marker: '# {mark} ANSIBLE MANAGED BLOCK c2-3'
block: |
{{ lookup('template', 'conf.ini.j2') }}
vars:
assigned_value: AVALUE
gives
shell> cat conf.ini
[configuration]
config1 = this_is_fine
# BEGIN ANSIBLE MANAGED BLOCK c2-3
config2 = AVALUE
config3 = AVALUE
# END ANSIBLE MANAGED BLOCK c2-3
config_4 = this_is_fine

How to separate the two values? Twig return sql group concat

i need a help for make the result.
my sql request :
$posts = $db->prepare('SELECT
CM.idpost,
CM.title,
CM.slug,
CM.content,
CM.cover,
CM.date,
GROUP_CONCAT(PC.id SEPARATOR ";") AS list_id,
GROUP_CONCAT(PC.cat_name SEPARATOR ";") AS list_cat_name,
GROUP_CONCAT(PC.icon SEPARATOR ";") AS list_icon,
GROUP_CONCAT(PC.bg SEPARATOR ";") AS list_bg,
GROUP_CONCAT(PC.slug_cat SEPARATOR ";") AS list_slug_cat,
U.iduser,
U.username,
U.avatar
FROM cms_posts CM
LEFT JOIN relation_posts RP ON RP.id_post = CM.idpost
LEFT JOIN cms_postcategory PC ON PC.id = RP.id_category
LEFT JOIN users U ON U.iduser = CM.author
GROUP BY CM.idpost
ORDER BY CM.date DESC LIMIT '.$paginationStart.','.$limit.'');
this is what my sql query for my table returns: Marketing;Général in Twig :
{{ post.list_cat_name }}
Picture demo
I need the result :
Marketing
Général
Thanks for help <3
So you've got a string like Marketing;Général. You need to create 2 links using the text for each as the link text.
I assume you similarly need to split the list_id for your idcategory value.
So something like this:
{% set categoryNames = post.list_cat_name | split(';') %}
{% set categoryIds = post.list_id | split(';') %}
{% for category in categoryNames %}
{{category}}
{% endfor %}
https://twig.symfony.com/doc/2.x/tags/for.html#the-loop-variable
https://twig.symfony.com/doc/2.x/filters/split.html

How to use flask to check whether every folder has specific files or not and using jinja2 to show the result?

I am now trying to check whether every folder has specific files or not.
First, I create two list to store the is_file result.
folderlist[folder1,folder2], has_data[file1 ok,file2 nok,file1 ok, file2 ok]
But I don't know how to use jinja2 to deal with them.
I want the output can be like below.
folder1
file1 ok
file2 nok
folder2
file1 ok
file2 ok
And below is my code to check file.
Could you give me some advice or better solution?
def has_data():
folderlist = []
has_data = []
for root, dirs, files in walk(target_folder):
qq = list(root.split('\\'))
filefolder = qq[-1]
folderlist.append(filefolder)
file1_site = os.path.join(target_folder+'\\'+filefolder+'\\' + 'file1.txt')
file2_site = os.path.join(target_folder+'\\'+filefolder+'\\' + 'file2.txt')
if not os.path.isfile(file1_site):
has_data.append('file1.txt NOK')
else:
has_data.append('file1.txt OK')
if not os.path.isfile(file2_site):
has_data.append('file2.txt NOK')
else:
has_data.append('file2.txt OK')
del folderlist [0]
del has_data [0]
return render_template('hasdata.html',folderlist=folderlist,has_data=has_data)
I didn't solve your problem directly, don't know values of those list. But you can wrap your html output in pre tag for preformatted text and add - sign to for loop.
From jinja2 documentation:
You can also strip whitespace in templates by hand. If you add a minus sign (-) to the start or end of a block (e.g. a For tag), a comment, or a variable expression, the whitespaces before or after that block will be removed
So your template can look something like this to get output you need:
...
<pre>
{% for folder in folderlist -%}
{{ folder }}
{% for file in has_data -%}
{{ file }}
{% endfor %}
{% endfor %}
</pre>
...

Coding Implode In Twig

How can one use the equivalent of the implode function in a Twig environment?
For example, the contents of the variable $data1 = "input your name" and the variable $data2 = "input your address".
How to create a variable $result = "input your name, input your address" in Twig?
I'm guessing you're looking for the join filter. It works exactly like implode does in php. Quoting from the manual page:
The join filter returns a string which is the concatenation of the items of a sequence:
{{ [1, 2, 3]|join }}
{# returns 123 #}
The separator between elements is an empty string per default, but you can define it with the optional first parameter:
{{ [1, 2, 3]|join('|') }}
{# returns 1|2|3 #}

Resources