I need help, I want to compare 2 or more messages containce kv in logstash
examples :
first message : X < 10=5.4|9=14|36=V|3=9|49=360T_SEP|5=Good|220=p48
second messages : y1 > 8=pap4|10=495|37=d|34=7|49=SEP|220=p48
y2 > 8=pap4|10=495|34=d|34=7|49=SEP|220=p48
iteration 1 : I get two key : 5 and 220
iteration 2 : I check if y1 has not 5 and 220 from x equals 220 to y1 then set in y1 5.
Basically, I want retrieved in each message the key 220 which corresponds to 5
Any Suggestion please.
Unless things have really changed, logstash typically concerns itself with one event at a time. The elapsed filter is one of the only exception where it is considering prior events in the processing.
You could use ruby to create your own cache, or perhaps use the redis inputs and outputs to that effect, but I'd suggest changing the format of the original message to include the data you need.
Related
I'm newbie and try to get ichimoku data on 4 hour timeframe but it not showing the correct value when I shift.
//#version=4
study(title="test1", overlay=true)
conversionPeriods = input(9, minval=1, title="Conversion Line Length")
basePeriods = input(26, minval=1, title="Base Line Length")
laggingSpan2Periods = input(52, minval=1, title="Leading Span B Length")
displacement = input(26, minval=1, title="Displacement")
donchian_M240(len) => avg(security(syminfo.tickerid, 'D' , lowest(len)), security(syminfo.tickerid, 'D', highest(len)))
tenkanSen_M240 = donchian_M240(conversionPeriods)
kijunSen_M240 = donchian_M240(basePeriods)
senkoSpanA_M240 = avg(tenkanSen_M240, kijunSen_M240)
plot(senkoSpanA_M240[25], title="senkoSpanA_M240[25]")
The value senkoSpanA_M240[25] keep changing when I'm in M5, M15, M30, H1, H4 or D1.
Can you help please?
the reason it keeps changing when you change time frames is because you are using a historical bar reference [25] on your senkoSpanA_M240.
This means it will look for the senkoSpanA_M240 condition that occurred 25 bars ago.
Depending on which time frame you are selecting, it will look back 25 bars of that time frame and perform the calculation.
What exactly are you trying to achieve by using the [25]?
I've been trying to optimise a bokeh server to calculate live stats by selected country on Covid19.
I found myself repeating a groupby function to calculate new columns and was wondering, having selected the groupby, if I could then apply it in a similar way to .agg() on multiple columns ?
For example:
dfall = pd.DataFrame(db("SELECT * FROM C19daily"))
dfall.set_index(['geoId', 'date'], drop=False, inplace=True)
dfall = dfall.sort_index(ascending=True)
dfall.head()
id date geoId cases deaths auid
geoId date
AD 2020-03-03 70119 2020-03-03 AD 1 0 AD03/03/2020
2020-03-14 70118 2020-03-14 AD 1 0 AD14/03/2020
2020-03-16 70117 2020-03-16 AD 3 0 AD16/03/2020
2020-03-17 70116 2020-03-17 AD 9 0 AD17/03/2020
2020-03-18 70115 2020-03-18 AD 0 0 AD18/03/2020
I need to create new columns based on 'cases' and 'deaths' and applying various functions like cumsum(). Currently I do this the long way
dfall['ccases'] = dfall.groupby(level=0)['cases'].cumsum()
dfall['dpc_cases'] = dfall.groupby(level=0)['cases'].pct_change(fill_method='pad', periods=7)
.....
dfall['cdeaths'] = dfall.groupby(level=0)['deaths'].cumsum()
dfall['dpc_deaths'] = dfall.groupby(level=0)['deaths'].pct_change(fill_method='pad', periods=7)
I tried to optimise the groupby call like this:-
with dfall.groupby(level=0) as gr:
gr = g['cases'].cumsum()...
But the error suggest the class doesn't support this
AttributeError: __enter__
I thought I could use .agg({}) and supply dictionary
g = dfall.groupby(level=0).agg({'cc' : 'cumsum', 'cd' : 'cumsum'})
but that produces another error
pandas.core.base.SpecificationError: nested renamer is not supported
I have plenty of other bits to optimise, I thought this python part would be the easiest and save a few ms!
Could anyone nudge me in the right direction?
To avoid repeating dfall.groupby(level=0) you can just save it in a variable:
gb = dfall.groupby(level=0)
gb_cases = gb['cases']
dfall['ccases'] = gb_cases.cumsum()
dfall['dpc_cases'] = gb_cases.pct_change(fill_method='pad', periods=7)
...
And to run multiple aggregations using a single expression, I think you can use named aggregation. But I have no clue whether it will be more performant or not. Either way, it's better to profile the code and improve the actual bottlenecks.
I am having issues using rampUser() method in my gatling script. The request is getting stuck after the following entry which had passed half way through.
Version : 3.3
================================================================================
2019-12-18 09:51:44 45s elapsed
---- Requests ------------------------------------------------------------------
> Global (OK=2 KO=0 )
> graphql / request_0 (OK=1 KO=0 )
> rest / request_0 (OK=1 KO=0 )
---- xxxSimulation ---------------------------------------------------
[##################################### ] 50%
waiting: 1 / active: 0 / done: 1
================================================================================
I am seeing the following in the log which gets repeated for ever and the log size increases
09:35:46.495 [GatlingSystem-akka.actor.default-dispatcher-2] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
09:35:47.494 [GatlingSystem-akka.actor.default-dispatcher-6] DEBUG io.gatling.core.controller.inject.open.OpenWorkload - Injecting 0 users in scenario xxSimulation, continue=true
The above issue is happening only with rampUser and not happening with
atOnceUsers()
rampUsersPerSec()
rampConcurrentUsers()
constantConcurrentUsers()
constantUsersPerSec()
incrementUsersPerSec()
Is there a way to mimic rampUser() in some other way or is there a solution for this.
My code is very minimal
setUp(
scenarioBuilder.inject(
rampUsers(2).during(1 minutes)
)
).protocols(protocolBuilder)
I am stuck with this for some time and my earlier post with more information can be found here
Can any of the gatling experts help me on this?
Thanks for looking into it.
It seems you have slightly incorrect syntax for a rampUsers. You should try remove a . before during.
I have in my own script this code and it works fine:
setUp(userScenario.inject(
// atOnceUsers(4),
rampUsers(24) during (1 seconds))
).protocols(httpProtocol)
Also, in Gatling documentation example is also without a dot Open model:
scn.inject(
nothingFor(4 seconds), // 1
atOnceUsers(10), // 2
rampUsers(10) during (5 seconds), // HERE
constantUsersPerSec(20) during (15 seconds), // 4
constantUsersPerSec(20) during (15 seconds) randomized, // 5
rampUsersPerSec(10) to 20 during (10 minutes), // 6
rampUsersPerSec(10) to 20 during (10 minutes) randomized, // 7
heavisideUsers(1000) during (20 seconds) // 8
).protocols(httpProtocol)
)
My guess is that syntax can't be parsed, so instead 0 is substituted. (Here is example of rounding. Not applicable, but as reference: gatling-user-injection-constantuserspersec)
Also, you mentioned that others method work, could you paste working code as well?
I've a requirement, where I need to extract 2 types of records from a single input file & join them for EZT report processing.
Currently, I've written an ICETOOL step to perform the extraction followed by the join. The output of the ICETOOL step is fed to the Easytrieve report step.
Extraction card is as below -
SORT FIELDS=(14,07,PD,A)
OUTFILE FNAMES=FILE010,INCLUDE=(25,03,CH,EQ,C'010')
OUTFILE FNAMES=FILE011,INCLUDE=(25,04,CH,EQ,C'011')
OPTION DYNALLOC=(SYSDA,05)
Here is the join card -
SORT FIELDS=(14,07,PD,A)
JOINKEYS F1=FILE010,FIELDS=(14,07,A),SORTED,NOSEQCHK
JOINKEYS F2=FILE011,FIELDS=(14,07,A),SORTED,NOSEQCHK
REFORMAT FIELDS=(F1:14,07,
F2,25,10)
OUTREC BUILD=(1,17,80:X),VTOF
OPTION DYNALLOC=(SYSDA,05)
I'm wondering if it was possible to perform the above SORT/ICETOOL operations within EasyTrive. I've used the Easytrieve internal SORT but it was for the simple extractions. Can we perform the join operation within the Easytrieve?
Note - The idea is to have a single EZT step.
You can make use Synchronized File Processing facility (SFP) in Easytrieve to acheive the task. Read more about it here.
FILE FILE010
KEY1 14 7 N
*
FILE FILE011
KEY2 14 7 N
FIELD1 25 10 A
*
FILE OUTFILE FB(80 0)
OKEY 1 7 N
OFIELD 8 10 A
*
WS-COUNT W 5 N VALUE 0
*
JOB INPUT FILE010 KEY KEY1 FILE011 KEY KEY2 FINISH(DIS)
*
IF EOF FILE010
STOP
END-IF
*
IF MATCHED
OKEY = KEY1
OFIELD = FIELD1
WS-COUNT = WS-COUNT + 1
PUT OUTFILE
END-IF
*
DIS. PROC
DISPLAY 'RECORDS WRITTEN: ' WS-COUNT
END-PROC
Please note,
Above code isn't tested, it's just a draft showing the idea on
file matching using Easytrieve to achieve the task.
Data types to the data items are assumed. You may have to change them suitably.
You may have to define the variable input datasets in the FILE
statement.
You may add more statements within the IF MATCHED condition for the
creation of report.
Hope this helps!
I would like to be able to query several handles at once, where the tables have the same format like:
handles: 8000,8001,8003
tables: foo
Want to do something like:
x:hopen `8000`8001`8003
x select from foo col1,col2
So i get rows from each foo table on each handle.
Is there a way to achieve this?
Thank you
Use 'each' to hopen each handle
q)h:hopen each 8000 8001 8002
q)h
476 480 484i
Use apply each-left to send the same query to each server
q)r:h#\:"select col1,col2 from foo"
q)r
+`col1`col2!(1 2;2 3)
+`col1`col2!(1 2;2 3)
+`col1`col2!(1 2;2 3)
Then you'll have to raze the result:
q)show res:raze r
col1 col2
---------
1 2
2 3
1 2
2 3
1 2
2 3
If you are not planning to reuse the handles, you can do
q)raze`::8000`::8001`::8003#\:"select from foo col1,col2"
Same as other answers, but more complicated using set (neg h) rather than get (h)
The cookbook/load-balancing helps in this example, too.
q)h:hopen each 8000 8001 8002
q)h
476 480 484i
q)r:(0#0i)!() /dictionary of handles and results
Set the callback for the response from the servers
q).z.ps:{#[`r;.z.w;:;x]}
Send a "set" query to each handle
q)(neg h)#\:({(neg .z.w)value"select col1,col2 from foo"};`)
Wait until all messages have a response
q)h .\:()
Finally, putting the result together
q)raze r h
The only advantage is concurrency.
As #AlexanderBelopolsky pointed out, don't forget
q)hclose each h