how to skip 1 st line of payload - groovy - groovy

I have final payload in csv format with some amount of employee records. I have 2 main values to work with:
-Sequence field (added for each record)
-personal_id (unique for every employee)
Because of the fact that each employee can have multiple records, the need is to have according sequence number for each ID. In other words If employee with ID "123" have 5 records, the sequence value should be 1,2,3,4,5 instead of 1,1,1,1,1. All records are grouped by ID.
Also the payload below shows only 2 fields needed for the description, normally it has much more fields and is an example of static payload. Normally it will be dynamically.
here are the input payload:
Sequence;ID
123456
232323
232323
232323
111111
111111
222222
222222
222222
222222
222222
222222
222222
and here is expected payload
Sequence;ID
1;123456
1;232323
2;232323
3;232323
1;111111
2;111111
1;222222
2;222222
3;222222
4;222222
5;222222
6;222222
7;222222
Here is the actual payload after using groovy script:
1;Sequence;ID
1;123456
1;232323
2;232323
3;232323
1;111111
2;111111
1;222222
2;222222
3;222222
4;222222
5;222222
6;222222
7;222222
I am using the following groovy script but the problem is that the first line of payload is also numbered by 1. Can you show me how to skip the first line of payload?
The script is adding a number in every record to have it counted. The clue is to have all records of the same id counted from 1 incrementally. The script does that perfectly but the issue is as I said while i want to skip the first line od payload "Sequence;..." to not have it counted.
import com.sap.gateway.ip.core.customdev.util.Message
def Message processData(Message message) {
def payload = message.getBody(java.lang.String)
def prevId = ''
def sequence = 1
def sb = new StringBuilder()
def line = new StringReader()
payload.eachLine { line ->
def values = line.split(';')
if (values[1] != prevId) {
// New personal ID
sequence = 1
prevId = values[1]
} else {
// Another line of the same personal ID
sequence += 1
}
line = readLine()
values[0] = sequence
sb.append(values.join(';')).append(System.lineSeparator())
}
message.setBody(sb.toString())
return message
}

Assuming message is just a String, .eachLine can be given two arguments: the line and the index. You could use that to do whatever logic you want in your closure: skip index 0 entirely, output it as-is, etc.
Basic example:
payload.eachLine { line, idx ->
if (idx == 0) {
// do something with the first line
} else {
// everything you currently have
}
}

Related

fetch specific protobuf members

I want to get an array of all the lines which start by text: (till the first asset_performance_label)
I saw this post, but wasn't sure how to apply it.
Should I convert the proto to string, as I have tried?
text = extract_text_from_proto(r"(\w+)text:(\w+)asset_performance_label:", '''[pinned_field: HEADLINE_1
text: "5 Best Products"
asset_performance_label: PENDING
policy_summary_info
{
review_status: REVIEWED
approval_status: APPROVED
}
, pinned_field: HEADLINE_1
text: "10 Best Products 2021"
asset_performance_label: PENDING
policy_summary_info
{
review_status: REVIEWED
approval_status: APPROVED
}''')
def extract_text_from_proto(regex, proto_string):
regex = re.escape(regex)
result_array = [m.group() for m in re.finditer(regex, proto_string)]
return result_array
# return [extract_text(each_item, regex) for each_item in proto],
def extract_text(regex, item):
m = re.match(regex, str(item))
if m is None:
# text = "MISSING TEXT"
raise Exception("Ad is missing text")
else:
text = m.group(2)
return text
Expected result: ["5 Best Products","10 Best Products 2021"]
What if I want to match (optional) pinned_field: (word)? so the result could be: [HEADLINE_1: 5 Best Products', 'HEADLINE_1:10 Best Products 2021', 'some_text_without_pinned_field']` ?
You can use a single capture group, and match assert_performance_label in the next line. Use re.findall to return the group values.
\btext:\s*"([^"]+)"\n\s*asset_performance_label\b
The pattern matches
\btext:\s*" Match text: predeced by a word boundary \b to prevent a partial match
([^"]+) Capture group 1, match 1+ chars other than a double quote
"\n\s* Match a newline an optional whitespace chars
asset_performance_label\b Match `asset_performance_label followed by a word boundary
For example
import re
def extract_text_from_proto(regex, proto_string):
return re.findall(regex, proto_string)
text = extract_text_from_proto(r'\btext:\s*"([^"]+)"\n\s*asset_performance_label\b', '''[pinned_field: HEADLINE_1
text: "5 Best Products"
asset_performance_label: PENDING
policy_summary_info
{
review_status: REVIEWED
approval_status: APPROVED
}
, pinned_field: HEADLINE_1
text: "10 Best Products 2021"
asset_performance_label: PENDING
policy_summary_info
{
review_status: REVIEWED
approval_status: APPROVED
}''')
print(text)
Output
['5 Best Products', '10 Best Products 2021']

Analyze log data with occassional multiline logs

I need to scan a log file for users that ran certain SQL statements (DROP,CREATE,etc.), and return an array with the users and which SQL ddl's they tried(drop,create,etc.).
I have log files where each line normally looks like this:
'2019-01-14T-19:23:50Z UTC' [ db=dev user=joeschmoe pid=123 userid=1 xid=1234]' Log: Select *
however, sometimes the select statement will span multiple lines like this:
'2019-01-14T19:23:50Z UTC [ db=dev user=rb pid=16 userid=1 xid=8 ]' LOG: SELECT SUM (num_queries) num_all_queries
,SUM (CASE WHEN lalala is not null THEN num_queries ELSE 0 END) num_b
,SUM (CASE WHEN lalala is null THEN num_queries ELSE 0 END) num_non_b
,SUM (total_queue_time_min) total_queue_time_min
,SUM (CASE WHEN lalala is not null THEN total_queue_time_min ELSE 0 END) b_total_queue_time_min
,SUM (CASE WHEN lalala is null THEN total_queue_time_min ELSE 0 END) non_b_total_queue_time_min
,SUM (CASE WHEN lalala is not null THEN duration_s ELSE 0 END)/60.0 total_burst_usage_min
,SUM (CASE WHEN lalala is not null THEN 1 ELSE 0 END) num_lalalas
,MIN(firsttime) mintime
,MAX(lasttime) maxtime
,DATEDIFF (seconds, mintime, maxtime) workload_duration_s
,wration_s/60.0 workload_duration_min
LEFT JOIN (SELECT b FROM STfdaf LIMIT 1) sq ON sq.but_reon < 100
;
I am combing these logs for certain keywords in the SQL statements. I can write the regex to handle that but I need help getting this log in a format I can work with. I was originally using a for loop and a regex
for line in input:
user_match = re.search("DROP", line, re.IGNORECASE)
This wouldn't be accurate because when a sql statement spans multiple lines I wouldn't be able to tie the "DROP" back to the "USER" if the DROP occurred many lines after the initial line.
I'm not sure how to go about doing this. Whether its turning this text file into a list in python and programmatically combining multiple lines into one or whatever other options there may be.
I solved this issue by storing the user value in a variable that only updates if a match is found using RegEx. So if a keyword is found, the user value last stored in the variable would be returned along with the keyword.
for line in input:
error = []
user_match = re.search("USER=b[0-9]{6}", line, re.IGNORECASE)
serviceuser = re.search("USER=[a-z0-9]*", line, re.IGNORECASE)
if serviceuser:
user = (serviceuser.group().split("=")[1])
elif user_match:
user = (user_match.group().split("=")[1])
ddl = re.search(
"LINK|.DELETE.|INSERT|TRIGGER|TRUNCATE|UPDATE|WRITE", line, re.IGNORECASE)
if ddl:
error.append(user)
error.append(ddl.group())

Validate row length when importing a txt file with fix length columns in SSIS 2016

I have a fixed length file that I have to read and validate. That file is produced by another system, but sometimes, employees are making manual changes to it. Example:
Layout
Variable: Surname size: 30 1 -30
Variable: Name size: 30 31-60
Variable: Email size: 30 61-90
Variable: Comments size: 30 91-120
Variable: CarriageReturn size: 2 121-123
So the system produces the following text file:
Source file
But then there is a manual intervention and the person does not respect the column length:
Source file after manual intervention
So before even starting to validate the values in the columns, everything is offset because my first carriage return is now splitting my "Comments" column when I read it in the SSIS.
Is there a way to tell the system that, if the length row is more than 2033, output in error file and continue ? What is the best way to do this?
Mylene
I found it!!
//Pass the file path and file name to the StreamReader and StreamWriter constructors
StreamReader sr = new StreamReader(inputFile);
StreamWriter sw = new StreamWriter(Dts.Connections["CE802CleanInput"].ConnectionString);
StreamWriter swe = new StreamWriter(Dts.Connections["CE802PreValidationErrors"].ConnectionString);
//Read the first line
line = sr.ReadLine();
while (line != null)
{
int length = line.Length;
if (length > 2033)
{
if
{
swe.WriteLine("Some records have been rejected at the pre validation phase.");
swe.WriteLine("Those records will not be included in the process.");
swe.WriteLine("Please review the records below, fix and re submit if applicable.");
swe.WriteLine("Input file: " + Dts.Connections["CE802Input"].ConnectionString.ToString());
swe.WriteLine();
swe.WriteLine(line);
count++;
}
else
{
swe.WriteLine(line);
count++;
}
}
if (length <= 2033)
{
sw.WriteLine(line);
}
line = sr.ReadLine();
}

NetSuite - get csv values using suitescript

I would like to do update a whole bunch of 'rev rec plan' records. Updating using mass update isn't available nor is an import possible.
So the only way I think I could do this is via script (probably scheduled), but I am not sure how to retrieve the 3 fields in my excel (internal id, start date and end date). I need the script to get the date fields from my file and using the internal id update the rec rec dates.
Thanks
You should be able to get started with the following code. Just get the internal id of the file you want to parse.
function parseCSVFromFile(fileId){
var csvFile = nlapiLoadFile(fileId);
var csv = csvFile.getValue();
var parsedCSV = CSVParser().parse(csv);
var data= parsedCSV.data;
return data;
}
function CSVParser(){function e(r,n){if(Array.isArray(r)){var i=[];return r.forEach(function(t){"object"==typeof t?i.push(e(t.file,t.config)):i.push(e(t,n))}),i}var i={data:[],errors:[]};if(!/(\.csv|\.txt)$/.test(r))return i.errors.push({type:"",code:"",message:"Unsupported file type.",row:""}),i;try{var a=fs.readFileSync(r).toString();return t(a,n)}catch(s){return i.errors.push(s),i}}function t(e,t){var r=a(t),i=new n(r),s=i.parse(e);return f(r.complete)&&r.complete(s),s}function r(e,t){function r(){"object"==typeof t&&("string"==typeof t.delimiter&&1==t.delimiter.length&&-1==l.BAD_DELIMITERS.indexOf(t.delimiter)&&(o=t.delimiter),("boolean"==typeof t.quotes||t.quotes instanceof Array)&&(f=t.quotes),"string"==typeof t.newline&&(d=t.newline))}function n(e){if("object"!=typeof e)return[];var t=[];for(var r in e)t.push(r);return t}function i(e,t){var r="";"string"==typeof e&&(e=JSON.parse(e)),"string"==typeof t&&(t=JSON.parse(t));var n=e instanceof Array&&e.length>0,i=!(t[0]instanceof Array);if(n){for(var s=0;s<e.length;s++)s>0&&(r+=o),r+=a(e[s],s);t.length>0&&(r+=d)}for(var f=0;f<t.length;f++){for(var l=n?e.length:t[f].length,u=0;l>u;u++){u>0&&(r+=o);var p=n&&i?e[u]:u;r+=a(t[f][p],u)}f<t.length-1&&(r+=d)}return r}function a(e,t){if("undefined"==typeof e||null===e)return"";e=e.toString().replace(/"/g,'""');var r="boolean"==typeof f&&f||f instanceof Array&&f[t]||s(e,l.BAD_DELIMITERS)||e.indexOf(o)>-1||" "==e.charAt(0)||" "==e.charAt(e.length-1);return r?'"'+e+'"':e}function s(e,t){for(var r=0;r<t.length;r++)if(e.indexOf(t[r])>-1)return!0;return!1}var f=!1,o=",",d="\r\n";if(r(),"string"==typeof e&&(e=JSON.parse(e)),e instanceof Array){if(!e.length||e[0]instanceof Array)return i(null,e);if("object"==typeof e[0])return i(n(e[0]),e)}else if("object"==typeof e)return"string"==typeof e.data&&(e.data=JSON.parse(e.data)),e.data instanceof Array&&(e.fields||(e.fields=e.data[0]instanceof Array?e.fields:n(e.data[0])),e.data[0]instanceof Array||"object"==typeof e.data[0]||(e.data=[e.data])),i(e.fields||[],e.data||[]);throw"exception: Unable to serialize unrecognized input"}function n(e){function t(){if(E&&m&&(p("Delimiter","UndetectableDelimiter","Unable to auto-detect delimiting character; defaulted to '"+l.DefaultDelimiter+"'"),m=!1),e.skipEmptyLines)for(var t=0;t<E.data.length;t++)1==E.data[t].length&&""==E.data[t][0]&&E.data.splice(t--,1);return r()&&n(),a()}function r(){return e.header&&0==w.length}function n(){if(E){for(var e=0;r()&&e<E.data.length;e++)for(var t=0;t<E.data[e].length;t++)w.push(E.data[e][t]);E.data.splice(0,1)}}function a(){if(!E||!e.header&&!e.dynamicTyping)return E;for(var t=0;t<E.data.length;t++){for(var r={},n=0;n<E.data[t].length;n++){if(e.dynamicTyping){var i=E.data[t][n];"true"==i||"TRUE"===i?E.data[t][n]=!0:"false"==i||"FALSE"===i?E.data[t][n]=!1:E.data[t][n]=u(i)}e.header&&(n>=w.length?(r.__parsed_extra||(r.__parsed_extra=[]),r.__parsed_extra.push(E.data[t][n])):r[w[n]]=E.data[t][n])}e.header&&(E.data[t]=r,n>w.length?p("FieldMismatch","TooManyFields","Too many fields: expected "+w.length+" fields but parsed "+n,t):n<w.length&&p("FieldMismatch","TooFewFields","Too few fields: expected "+w.length+" fields but parsed "+n,t))}return e.header&&E.meta&&(E.meta.fields=w),E}function o(t){for(var r,n,a,s=[","," ","|",";",l.RECORD_SEP,l.UNIT_SEP],f=0;f<s.length;f++){var o=s[f],d=0,u=0;a=void 0;for(var p=new i({delimiter:o,preview:10}).parse(t),c=0;c<p.data.length;c++){var h=p.data[c].length;u+=h,"undefined"!=typeof a?h>1&&(d+=Math.abs(h-a),a=h):a=h}u/=p.data.length,("undefined"==typeof n||n>d)&&u>1.99&&(n=d,r=o)}return e.delimiter=r,{successful:!!r,bestDelimiter:r}}function d(e){e=e.substr(0,1048576);var t=e.split("\r");if(1==t.length)return"\n";for(var r=0,n=0;n<t.length;n++)"\n"==t[n][0]&&r++;return r>=t.length/2?"\r\n":"\r"}function u(e){var t=g.test(e);return t?parseFloat(e):e}function p(e,t,r,n){E.errors.push({type:e,code:t,message:r,row:n})}var c,h,m,g=/^\s*-?(\d*\.?\d+|\d+\.?\d*)(e[-+]?\d+)?\s*$/i,y=this,v=0,b=!1,w=[],E={data:[],errors:[],meta:{}};if(f(e.step)){var x=e.step;e.step=function(n){if(E=n,r())t();else{if(t(),0==E.data.length)return;v+=n.data.length,e.preview&&v>e.preview?h.abort():x(E,y)}}}this.parse=function(r){if(e.newline||(e.newline=d(r)),m=!1,!e.delimiter){var n=o(r);n.successful?e.delimiter=n.bestDelimiter:(m=!0,e.delimiter=l.DefaultDelimiter),E.meta.delimiter=e.delimiter}var a=s(e);return e.preview&&e.header&&a.preview++,c=r,h=new i(a),E=h.parse(c),t(),!f(e.complete)||b||y.streamer&&!y.streamer.finished()||e.complete(E),b?{meta:{paused:!0}}:E||{meta:{paused:!1}}},this.pause=function(){b=!0,h.abort(),c=c.substr(h.getCharIndex())},this.resume=function(){b=!1,h=new i(e),h.parse(c),b||(y.streamer&&!y.streamer.finished()?y.streamer.resume():f(e.complete)&&e.complete(E))},this.abort=function(){h.abort(),f(e.complete)&&e.complete(E),c=""}}function i(e){e=e||{};var t=e.delimiter,r=e.newline,n=e.comments,i=e.step,a=e.preview,s=e.fastMode;if(("string"!=typeof t||1!=t.length||l.BAD_DELIMITERS.indexOf(t)>-1)&&(t=","),n===t)throw"Comment character same as delimiter";n===!0?n="#":("string"!=typeof n||l.BAD_DELIMITERS.indexOf(n)>-1)&&(n=!1),"\n"!=r&&"\r"!=r&&"\r\n"!=r&&(r="\n");var f=0,o=!1;this.parse=function(e){function l(){return w.push(e.substr(f)),v.push(w),f=c,y&&p(),u()}function d(t){v.push(w),w=[],f=t,O=e.indexOf(r,f)}function u(e){return{data:v,errors:b,meta:{delimiter:t,linebreak:r,aborted:o,truncated:!!e}}}function p(){i(u()),v=[],b=[]}if("string"!=typeof e)throw"Input must be a string";var c=e.length,h=t.length,m=r.length,g=n.length,y="function"==typeof i;f=0;var v=[],b=[],w=[];if(!e)return u();if(s){for(var E=e.split(r),x=0;x<E.length;x++)if(!n||E[x].substr(0,g)!=n){if(y){if(v=[E[x].split(t)],p(),o)return u()}else v.push(E[x].split(t));if(a&&x>=a)return v=v.slice(0,a),u(!0)}return u()}for(var D=e.indexOf(t,f),O=e.indexOf(r,f);;)if('"'!=e[f])if(n&&0===w.length&&e.substr(f,g)===n){if(-1==O)return u();f=O+m,O=e.indexOf(r,f),D=e.indexOf(t,f)}else if(-1!==D&&(O>D||-1===O))w.push(e.substring(f,D)),f=D+h,D=e.indexOf(t,f);else{if(-1===O)break;if(w.push(e.substring(f,O)),d(O+m),y&&(p(),o))return u();if(a&&v.length>=a)return u(!0)}else{var A=f;for(f++;;){var A=e.indexOf('"',A+1);if(-1===A)return b.push({type:"Quotes",code:"MissingQuotes",message:"Quoted field unterminated",row:v.length,index:f}),l();if(A===c-1)return w.push(e.substring(f,A).replace(/""/g,'"')),v.push(w),y&&p(),u();if('"'!=e[A+1]){if(e[A+1]==t){w.push(e.substring(f,A).replace(/""/g,'"')),f=A+1+h,D=e.indexOf(t,f),O=e.indexOf(r,f);break}if(e.substr(A+1,m)===r){if(w.push(e.substring(f,A).replace(/""/g,'"')),d(A+1+m),D=e.indexOf(t,f),y&&(p(),o))return u();if(a&&v.length>=a)return u(!0);break}}else A++}}return l()},this.abort=function(){o=!0},this.getCharIndex=function(){return f}}function a(e){"object"!=typeof e&&(e={});var t=s(e);return("string"!=typeof t.delimiter||1!=t.delimiter.length||l.BAD_DELIMITERS.indexOf(t.delimiter)>-1)&&(t.delimiter=o.delimiter),"\n"!=t.newline&&"\r"!=t.newline&&"\r\n"!=t.newline&&(t.newline=o.newline),"boolean"!=typeof t.header&&(t.header=o.header),"boolean"!=typeof t.dynamicTyping&&(t.dynamicTyping=o.dynamicTyping),"number"!=typeof t.preview&&(t.preview=o.preview),"function"!=typeof t.step&&(t.step=o.step),"function"!=typeof t.complete&&(t.complete=o.complete),"boolean"!=typeof t.skipEmptyLines&&(t.skipEmptyLines=o.skipEmptyLines),"boolean"!=typeof t.fastMode&&(t.fastMode=o.fastMode),t}function s(e){if("object"!=typeof e)return e;var t=e instanceof Array?[]:{};for(var r in e)t[r]=s(e[r]);return t}function f(e){return"function"==typeof e}var o={delimiter:"",newline:"",header:!1,dynamicTyping:!1,preview:0,step:void 0,comments:!1,complete:void 0,skipEmptyLines:!1,fastMode:!1},l={};return l.parse=t,l.parseFiles=e,l.unparse=r,l.RECORD_SEP=String.fromCharCode(30),l.UNIT_SEP=String.fromCharCode(31),l.BYTE_ORDER_MARK="\ufeff",l.BAD_DELIMITERS=["\r","\n",'"',l.BYTE_ORDER_MARK],l.DefaultDelimiter=",",l.Parser=i,l.ParserHandle=n,l}
Example:
var parsedCSV = parseCSVFromFile(123);
nlapiLogExecution('DEBUG', "parsedCSV ", JSON.stringify(parsedCSV));

groovy read a file, resolve variables in file content

I am new to Groovy and I could not get around this issue. I appreciate any help.
I want to read a file from Groovy. While I am reading the content, for each line I want to substitute the string '${random_id}' and '${entryAuthor}' with different string values.
protected def doPost(String url, URL bodyFile, Map headers = new HashMap() ) {
StringBuffer sb = new StringBuffer()
def randomId = getRandomId()
bodyFile.eachLine { line ->
sb.append( line.replace("\u0024\u007Brandom_id\u007D", randomId)
.replace("\u0024\u007BentryAuthor\u007D", entryAuthor) )
sb.append("\n")
}
return doPost(url, sb.toString())
}
But I got the following error:
groovy.lang.MissingPropertyException:
No such property: random_id for class: tests.SimplePostTest
Possible solutions: randomId
at foo.test.framework.FooTest.doPost_closure1(FooTest.groovy:85)
at groovy.lang.Closure.call(Closure.java:411)
at groovy.lang.Closure.call(Closure.java:427)
at foo.test.framework.FooTest.doPost(FooTest.groovy:83)
at foo.test.framework.FooTest.doPost(FooTest.groovy:80)
at tests.SimplePostTest.Post & check Entry ID(SimplePostTest.groovy:42)
Why would it complain about a property, when I am not doing anything? I also tried "\$\{random_id\}", which works in Java String.replace(), but not in Groovy.
You are doing it the hard way. Just evaluate your file's contents with Groovy's SimpleTemplateEngine.
import groovy.text.SimpleTemplateEngine
def text = 'Dear "$firstname $lastname",\nSo nice to meet you in <% print city %>.\nSee you in ${month},\n${signed}'
def binding = ["firstname":"Sam", "lastname":"Pullara", "city":"San Francisco", "month":"December", "signed":"Groovy-Dev"]
def engine = new SimpleTemplateEngine()
template = engine.createTemplate(text).make(binding)
def result = 'Dear "Sam Pullara",\nSo nice to meet you in San Francisco.\nSee you in December,\nGroovy-Dev'
assert result == template.toString()
you better use groovy.text.SimpleTemplateEngine class; check this for more details http://groovy.codehaus.org/Groovy+Templates
The issue here is that Groovy Strings will evaluate "${x}" by substituting the value of 'x', and we don't want that behaviour in this case. The trick is to use single-quotes which denote plain old Java Strings.
Using a data file like this:
${random_id} 1 ${entryAuthor}
${random_id} 2 ${entryAuthor}
${random_id} 3 ${entryAuthor}
Consider this code, which is analogous to the original:
// spoof HTTP POST body
def bodyFile = new File("body.txt").getText()
StringBuffer sb = new StringBuffer()
def randomId = "257" // TODO: use getRandomId()
def entryAuthor = "Bruce Eckel"
// use ' here because we don't want Groovy Strings, which would try to
// evaluate e.g. ${random_id}
String randomIdToken = '${random_id}'
String entryAuthorToken = '${entryAuthor}'
bodyFile.eachLine { def line ->
sb.append( line.replace(randomIdToken, randomId)
.replace(entryAuthorToken, entryAuthor) )
sb.append("\n")
}
println sb.toString()
The output is:
257 1 Bruce Eckel
257 2 Bruce Eckel
257 3 Bruce Eckel

Resources