In nifi I have a simple flow,
GenerateFile -> ConvertCharacterSet -> UpdateAttribute -> PutSFTP
in ConvertCharacterSet processor, there're properties
"Input Character Set" , "Output Character Set" ,
both have expression language scope : "Variable registry and Flow file attributes"
When I want to use "Output Character Set" value attribute in filename (inside UpdateAttribute processor), it isnt reflected
(placed value).
filename: converted_to_${"Output Character Set"}_${now():format('yyyy-MM-dd_HHmmss', 'CET')}
Filename I finally get is (eg) : converted_to__2021-02-10_200201
As of this answer, you can't access component properties in NiFi Expression Language. When the "Output Character Set" says it has an expression language scope, it means you can use Expression Language in the value of that property, and the attributes/variables you can use in your expression can come from a Variable Registry and/or the current FlowFile attributes.
You could instead set a FlowFile attribute for which character set to use (let's call it charset) then you can set Output Character Set to ${charset}. You'd also use charset in your UpdateAttribute to add it to the filename.
Related
I have been working with Exchange Online unified groups and Microsoft Teams teams in powershell: New-UnifiedGroup, Set-UnifiedGroup, Get-UnifiedGroup, New-Team, Get-Team etc. I have been creating groups and teams from PowerShell.
I am interested in understanding a bit better when a value for a parameter requires quotes and when it does not.
When I import from CSV, it seems the values are automatically interpreted as strings. When I supply them to a parameter that requires a string, the value does not require quotes even if it has spaces e.g New-UnifiedGroup -DisplayName $item.displayName does not require quotes even when the display name has spaces.
But when I want to create a team from an existing group, and I get the ID of the group, the group id requires quotes: New-Teams -GroupId "$group.ExternalDirectoryObjectId". In this case the parameter -GroupId requires a string value, although the ExternalDirectoryObjectId that it requires is not a string.
Is there a rule that a value does not require quotes if it is a string, and a string value is expected? Does it help to declare a variable as a string before passing it to a parameter that requires a string? For example, if I have a $path variable, I usually have to provide it as -Path "$path". If I declared the path as [String]$path =, would I then not need to use the quotes in -Path $path
Generally, only ever use quoting in PowerShell to explicitly pass a value as a string ([string]).
String literals require quoting if they contain any of the following: spaces or, more generally, PowerShell metacharacters[1], and commands or expressions as part of a larger string (which must then be enclosed in $(...) - see below).
To pass the value of a variable, one of its properties, or even the result of a method call on it, you do not need quoting in PowerShell, which will pass the resulting value with its original data type; however, when the value is bound to its target parameter, PowerShell may automatically convert it to the parameter's type.
If the target parameter is [string]-typed (as is the case with New-Team's -GroupId parameter), PowerShell will automatically convert any non-string argument to a string, essentially by calling .ToString() on it[2]. If the resulting string isn't the right representation, you must perform explicit stringification, by way of an expression or command.
E.g., both -GroupId $groupId and -GroupId $group.ExternalDirectoryObjectId would work - even if the resulting string contains embedded spaces or other PowerShell metacharacters.
If you need to pass an object's property, a method call, or any type of command or expression as part of a larger string, enclose the argument in "..." and use $(...), the subexpression operator around the expression / command (e.g., "$($group.ExternalDirectoryObjectId)/more"; referencing a variable by itself inside "..." does not require $(...)).
"$group.ExternalDirectoryObjectId" definitely does not work as intended, because only variable reference $group by itself is recognized - and stringified - whereas the .ExternalDirectoryObjectId part is treated literally - see first link below for why.
Further reading:
Overview of PowerShell's expandable strings (string interpolation, "...")
String literals in PowerShell (bottom section)
How unquoted tokens are parsed as arguments.
[1] The metacharacters are (some only need quoting if at the start of the argument):
<space> ' " ` , ; ( ) { } | & < > # #
[2] The exact stringification rules, where culture-sensitivity factors in as well, are detailed in this answer.
Generally, PowerShell has a very flexible automatic type-conversion system whose rules are complex and not explicitly documented - a peek at the source code may help.
PowerShell always tries to automatically convert a given value to the target type, where the target type may be dictated by a parameter's type or the (usually) LHS operand of an operator-based expression (e.g., 42 + "1" yields 43).
That's strange. Usually you only need quotes to pass a literal string that has a space in it.
get-childitem -path 'foo 2'
I can pass an object property without quotes usually:
$a = [pscustomobject]#{path = 'foo 2'}
get-childitem -path $a.path
This is more rare, but if a string looks like an array element, I've found I have to quote it:
select-xml -XPath "//*[#a='hi']" -Path file.xml
Even using something like an integer works without quotes, because it can be 'coerced' into a string.
I have written an ETL in Talend Open Studio that loads a CSV/TSV file in a database. To do so, I want to provide the delimiter in tFileInputDelimited component using dynamic context load from a text file. I have specified it in the context file as fieldDelimiter="\t" and in the tFileInputDelimited component as shown in the screenshot. But, it doesn't work as a delimiter. I have also tried using fieldDelimiter="\\t" or fieldDelimiter="\u0009" (unicode character for tab).
What should I provide in the context file so that the delimiter is a tab character and not "\t" string as is happening in this case?
I notice a difference in the context variable names. In the screen shot you have mentioned (String)context.get("fileDelimiter"). But in the text you are saying "I have specified it in the context file as fieldDelimiter="\t" ".
just keeping the context as follows in the .properties file should work
fieldDelimiter=\t
Also use context.fieldDelimiter instead of (String)context.get("fileDelimiter").
In your context file, just put fileDelimiter = \t
(Without quotes)
And then access the variable in field delimiter. Talend will automatically handle it as string.
Hope this works.
There is no function (String)context.get("key") that I know of. If you have set the separator as a String element in the Context, just access it directly. Now there will be an empty String set as the field separator I suppose.
So if your field is called fileDelimiter simply put context.fileDelimiter into the Field Separator.
As pointed out by others, you should use context.ParamName syntax, the benefit of this method is syntax checking at compile time which eliminates the risk of typos in your variable names.
This parameter must be declared in your job (contexts tab) in order for Talend to recognize it. You can either create it as a built-in or import it if it's in the repository.
What is the different between Empty String and none Created Variable in Batch File?
and if they have any different, can you show me an example of using empty string vs none created string?
#echo off
title Empty String Vs None Created Variable
set String=
set variable=exist
if [%String%] == [] echo its be a Empty String
if [%variable%] == [] echo its be a Empty String
if [%none_exist_var%] == [] echo its be a Empty String
Thanks a Lot!
Variables in batch files can be
Defined: there is a value stored and a associated name used to retrieve the value, that is, the variable name.
Undefined: there is no value and in consecuence there is not any need for an associated name, so it does not exist.
This two simple rules handle how the environment block (where variables/value are stored) is updated. If there is a value, the environment block has an entry for the value and the name to retrieve it. Without a value, the environment block does not create the entry or, when setting the variable to nothing, the entry is removed.
So, if
a never defined variable has not any entry in the environment block
a variable with not value has not any entry in the environment block
there is not any difference between the two cases.
note: While the traditional way to check if a variable stores a value / a variable exists is (as dbenham has commented, this syntax is not recommeded as quotes inside the variable value lead to syntax problems)
if "%varName%"=="" ....
if command extensions are enabled (and the default configuration is to have them enabled) you can also use a safer alternative
if not defined varName ....
if defined varName ....
note that in this syntax, as we are not trying to read the value in the variable, varName is used, not %varName%
There is no difference. Read Delete a variable:
Type SET with just the variable name and an equals sign:
SET _department=
Better still, to be sure there is no trailing space after the = place
the expression in parentheses or quotes:
(SET _department=)
or
SET "_department="
The script (hivequery.hql:) looks like this:
Use ${platformType:platformName};
select * from hivetablename;
And this script is being called in a bash script as
#!/usr/bin/env bash
hive -f hivequery.hql
Within an hql file, the use command sets the default database. See Use Database.
The ${platformType:platformName} is Hive's variable notation where platformType is the namespace and platformName is the variable name. This is explained in the Using Variables section of the Language Manual.
If you want to see what value a specific variable has, you can just use set like:
set platformType:platformName;
and it will print out the value. You can also run set; to get a full listing of known variables in all namespaces.
The more correct way to write the construct ${word:word} would be to write ${parameter:offset} . It cause parameter expansion, it expands to the portion of the value of parameter starting at the character (counting from 0 ) determined by expanding offset to the end of the parameter . It has one more variant as ${parameter:offset:length } - Expands to the portion of the value of parameter starting at the character (counting from 0 ) determined by expanding offset as an arithmetic expression and consisting of the number of characters determined by the arithmetic expression defined by length.
So I think basically the in your case , it is meant to get the name of the database from platformType.
For more details on this look into the
Look for Parameter Expansion in the bash man page.
trying out upgrading antlr4, I have 2 lines in the grammar that produce the error message:
label tok assigned to a block which is not a set
Specifically for a grammar line that looks like this:
contextRadius: tok=('radius' 'change-authorize-nas-ip') (IP4_ADDRESS|IP6_ADDRESS) 'encrypted' 'key' ID 'port' INT_TOK 'event-timestamp-window' INT_TOK 'no-reverse-path-forward-check'
;
What does this imply, exactly - to be a "block which is not set" and is there a general solution?
The improper label is the following:
tok=('radius' 'change-authorize-nas-ip')
In this case, ANTLR doesn't know whether to assign the token 'radius' or the token 'change-authorize-nas-ip' to the label tok. Starting with ANTLR 4, rather than generate code with unclear semantics an error is produced. You'll want to either remove the label tok or move it to the intended item. In other words, use one of the following three forms.
('radius' 'change-authorize-nas-ip')
(tok='radius' 'change-authorize-nas-ip')
('radius' tok='change-authorize-nas-ip')
The reason labels are allowed on blocks in grammars is to support items like the following. This block is a set, meaning the contents can be collapsed to matching exactly one token from a fixed set of allowed tokens. The particular item matched by the set is then assigned to x.
x=('a' | 'b')