expand collapsed data in nushell - nushell

I'm trying to learn nushell, and from this example in the documentation,
> echo [[column1, column2]; [Value1, Value2]]
───┬─────────┬─────────
# │ column1 │ column2
───┼─────────┼─────────
0 │ Value1 │ Value2
───┴─────────┴─────────
but my output is:
echo [[column1, column2], [Value1, Value2]]
╭───┬────────────────╮
│ 0 │ [list 2 items] │
│ 1 │ [list 2 items] │
╰───┴────────────────╯
How do you expand the collapsed information?

It sounds like what you really want to do is convert a Nushell list of lists to a table (a list of records). I'm a bit surprised myself that there isn't (at least that I could find), an easier way (i.e. a built-in command) to do this coercion, but there are several ways to accomplish it. However, I still feel that there's an easier way that I'm just not seeing myself.
Anyway, the easiest that I could come up with, for your particular example:
〉[[column1, column2], [Value1, Value2]] | each { { 0: $in.0, 1: $in.1 }} | headers
╭───┬─────────┬─────────╮
│ # │ column1 │ column2 │
├───┼─────────┼─────────┤
│ 0 │ Value1 │ Value2 │
╰───┴─────────┴─────────╯
Note, in most cases, you can drop the echo commands and have exactly the same results.
Also note that there may be a simpler answer by transforming the data earlier, but you don't mention the source of the data. It's unlikely that the original source of the data would be a Nushell "list literal" in the "real world". But for purely academic/learning purposes, it's certainly worth understanding.
Explanation:
The first data type you have above (from the doc example) is truly a Nushell table:
〉[[column1, column2]; [Value1, Value2]] | describe
table<column1: string, column2: string>
The second is a list of lists:
〉 [[column1, column2], [Value1, Value2]] | describe
list<list<string>>
Also note that the doc example above might be considered a bit outdated. When it was written, it was the only way to construct a table-literal in Nushell (and is still probably considered the "table-literal" syntax). But the addition of the record-syntax in 0.40 brought us a new way to specify a table literal, as a "list of records". You can now use the (IMHO) more readable (or at least "standard"):
[{ 'column1':'value1', 'column2':'value2' }]
That said, the [[column1, column2]; [Value1, Value2]] form is more concise if you ever really needed to specify multiple rows for the table. But whether or not this is a "real world" use is another question, and it wouldn't surprise me to see the "old" table-literal syntax removed before Nushell reaches 1.0.
It's that new "list-of-records" form that we use to answer your question:
[[column1, column2], [Value1, Value2]] | each { { 0: $in.0, 1: $in.1 }} | headers
For each pair (e.g. [Value1, Value2]) in the list, creates a record with the first value set as key 0 and the second set as key 1.
The results from each are automatically collected into a list.
Because the end result is a "list of records where every record is the same type", it results in a table. If one of the rows was different (e.g. it had an integer instead of a string), then the results would look the same, but the actual output would be a "list of records".
The headers command is used to take your heading names and use them instead of 0 and 1.
Bonus:
The following will display lists with an arbitrary number of values:
[[Column1, Column2, Column3 ], [Value1, Value2], [Value3, Value4, Value5], [Value6]] |
each { |values|
reduce -f {} { |it,acc|
$acc | insert $"Field($acc | transpose | length)" $it
}
} | headers
Result:
╭───┬─────────┬─────────┬─────────╮
│ # │ Column1 │ Column2 │ Column3 │
├───┼─────────┼─────────┼─────────┤
│ 0 │ Value1 │ Value2 │ │
│ 1 │ Value3 │ Value4 │ Value5 │
│ 2 │ Value6 │ │ │
╰───┴─────────┴─────────┴─────────╯
Explanation (in case you aren't familiar with reduce yet):
Loops over each (inner) list of values in the (outer) list
"Reduces" each of those inner lists of values to a record:
Starts with an empty record -f {} as $acc
Inserts each field as Field0, Field1, and so on.
The results are appended to the $acc which becomes the complete record at the end of the reduction.
The | transpose | length is a hacky workaround due to a bug that I need to file. I'll link it back here when I do.

Related

Terraform - inconsistent conditional result types, from same map

I have a map of values, and I'm basically trying to replace a number of different values while in a giant loop.
I've gotten so far, and feel like I've almost got this working but stumbling at what maybe the final step.
Error: Inconsistent conditional result types
│
│ on main.tf line 25, in locals:
│ 25: source_ranges = distinct(flatten([for source in [for source in v.source_ranges_names : source == "#Env" ? replace(source, "#Env", env) : (length(regexall("#Env_customer_cidr", source)) > 0 && env == "prod" ? local.defined_ranges["prod_customer_cidr"] : (length(regexall("#Env_customer_cidr", source)) > 0 && env == "test" ? local.defined_ranges["test_customer_cidr"] : local.defined_ranges[source] ))]: local.defined_ranges[source] ]))
│ ├────────────────
│ │ local.defined_ranges is object with 18 attributes
│ │ local.defined_ranges["prod_customer_cidr"] is tuple with 1 element
│ │ local.defined_ranges["test_customer_cidr"] is tuple with 1 element
│
│ The true and false result expressions must have consistent types. The given
│ expressions are string and list of string, respectively.
I don't understand the cause of the error, as I'm referring back to the same object each time and the values in the object are all defined the same way:
"defined_ranges": {
"random_cidr": [
"192.168.1.0/24"
],
"prod_customer_cidr": [
"192.168.2.0/24"
],
"test_customer_cidr": [
"192.168.3.0/24"
],
}
In the example error above, source would have a value such as 'random_cidr'.
I realise this can likely be refactored into something far easier to read/understand but I'm still getting my head around what the terraform is doing.

Enterprise Architect - Activity diagram simulation

I'm simulating an activity diagram using dynamic simulation with python code and I have few questions regarding this topic:
let's make a simple example : Apple counter - activity diagram
START
xxx
xxx
│
│
┌─────────▼─────────┐
│ │
│ ├──────────────────────────┐
│ PICK A FRUIT │ │
│ │ │
│ │ │
└─────────┬─────────┘ │
│ │
│ │
│ │
│ │
▼ │
xxx FRUIT iS AN APPLE? │
xxxxxx ┌──────────┴──────────┐
xxxxxxxxxx │ │
xxxxxxxxxxxxx ───────────────► │ INCREMENT APPLE │
xxxxxxxxxxx fruit == "apple" │ COUNTER │
xxxxxxxxx └─────────────────────┘
xxxxx
xxx
│
│
▼
EXIT
Question:
I want to embed apple_counter +=1 inside "INCREMENT APPLE COUNTER" action
I appreciate some practical examples (also in other programming languages) in order to understand better how to use code in simulation.

How to copy a Julia DataFrame in a threaded loop?

I've got the following problem :
A DataFrame named cr1 with 553 columns
Then, I make two loops as follow :
cr2=copy(cr1)
#-----------------#
#Loop on scenarios#
#-----------------#
#threads for k in 0:499
cr1=copy(cr2)
cr1[!,"scn"].=k
cr1 = leftjoin(cr1,tra,on = [:scn])
for i in 1:40
#----------------------#
##Projection on 40 years
#----------------------#
cr1[!,"ch_prime"*str(i)].=cr1[!,"ch_prime"].*(cr1[!,"coll_"*str(i)]
...job on other columns
At the beginning, I make a copy of cr1 to keep the vision of the DataFrame at this stage. Indeed I need to restart of this vision at each iteration of the k-loop.
Then I consider the scenario = k to make the right join with the DF called tra with 40 columns (tra1 to tra40) and k lines.
Finally, I make another loop to project all my variables on 4O years.
But it doesn't work. Julia do some job but cr1 is not modified.
I tried adding a global on the first line of the loop and squeezing the #threads
for k in 0:499
global cr1=copy(cr2)
Then it works, but with no threading and I want to keep it for performance
Have you any idea another way to do this or a way to thread the last solution ?
EDIT
Here is the full code (a bit long sorry), if it could help:
#Définit la fonction string avec une syntaxe plus courte
#Permet de conserver le code Python
function str(x)
string(x)
end
# %% Début du programme
#time begin
##+ Import des tables
donnee=CSV.File("/Users/Python/Python_VIF_sto/donnee.csv")|> DataFrame!
lapse=CSV.File("/Users/Python/Python_VIF_sto/lapse.csv")|> DataFrame!
dc=CSV.File("/Users/Python/Python_VIF_sto/dc.csv")|> DataFrame!
ind_action=CSV.File("/Users/Python/Python_VIF_sto/ind_action.csv")|> DataFrame!
tra=CSV.File("/Users/Python/Python_VIF_sto/tra.csv")|> DataFrame!
ts=CSV.File("/Users/Python/Python_VIF_sto/ts.csv")|> DataFrame!
ech=CSV.File("/Users/Python/Python_VIF_sto/ech.csv")|> DataFrame!
sw=CSV.File("/Users/Python/Python_VIF_sto/sw.csv")|> DataFrame!
df=CSV.File("/Users/Python/Python_VIF_sto/df.csv")|> DataFrame!
tx_is=CSV.File("/Users/Python/Python_VIF_sto/tx_is.csv")|> DataFrame!
fga=CSV.File("/Users/Python/Python_VIF_sto/fga.csv")|> DataFrame!
fgp=CSV.File("/Users/Python/Python_VIF_sto/fgp.csv")|> DataFrame!
fgacq=CSV.File("/Users/Python/Python_VIF_sto/fgacq.csv")|> DataFrame!
inf=CSV.File("/Users/Python/Python_VIF_sto/inf.csv")|> DataFrame!
#Definition des parametres communs
ppb0=1347911164; # PPB initiale
const global dur_proj=40; # Durée de projection
nb_coll=6; # 6 années de collecte possible
tcomp_mod=0 # Mode de determination composante technique PB min (1=fixe ou 0=libre)
tcomp_fixe=1 # Composante si fixe (1=prev 0=RT)
fdc=1 # 1 = frontiere des contrats S2 / 0 = désactivée
#CR initial avec les paramètres en t=0#
cr1=copy(donnee)
cr1[!, :annee].= 1
cr1[!, :der_fg].=0.01 #Dérive des frais au delà inflation
#Creation des dummies#
cr1[!,:eu] .= (cr1.supp.=="eu")
cr1[!,:uc] .= (cr1.supp.=="uc")
cr1[!, :prev] .= (cr1.supp.=="prev")
cr1[!, :ve] .= (cr1.supp.=="ve")
#Construction des taux projetés sur horizon de projection(rachats, décès, indices..)#
#Deces - par produit#
cr1 = leftjoin(cr1,dc, on =[:age, :tab_dc,:ab_dc]) # Décès
#----------------------#
#Echeances & Arbitrages#
#----------------------#
cr1 = leftjoin(cr1,ech, on =[:tab_ech]) #Echeances
cr1 = leftjoin(cr1,sw, on =[:tab_sw]) #Arbitrages
#---------------------------#
#Frais admin / acq / prestas#
#---------------------------#
cr1 = leftjoin(cr1,fga, on =[:fg_ad]) #Frais d"administration
cr1 = leftjoin(cr1,fgp, on =[:fg_prest]) #Frais sur prestations
cr1 = leftjoin(cr1,fgacq,on =[:fg_acq]) #Frais d"acquisition
#------------#
#---Lapses---#
#------------#
cr1 = leftjoin(cr1,lapse,on = [:tab_lapse]) #Rachats - vision simple - pas de gestion de la collecte
m = Matrix(cr1[!,columnindex(cr1, :anc1):columnindex(cr1, :anc100)])
v = cr1.anc
insertcols!(cr1, ["lap$k" => getindex.(Ref(m), axes(cr1, 1), v .+ k .- 1) for k in 1:dur_proj]...)
#--------#
#Collecte#
#--------#
# On force la collecte à zero au dela du plan
for i in 7:dur_proj
cr1[!,"coll_"*str(i)].=0
end
# Collecte prevoyance
for i in 1:dur_proj
cr1[!,"ca_prev"*str(i)].=cr1[!,"ca_prev"*str(i-1)].*(1 .-cr1[!,"lap"*str(i)])
.+cr1[!,"coll_"*str(i)].*cr1[!,"prev"]
end
# Application de la frontière des contrats au delà de la première année
for i in 2:dur_proj
cr1[!,"ca_prev"*str(i)].=cr1[!,"ca_prev"*str(i)].*cr1[!,"fdc"]
end
#------------------------------------------#
#Allocation PPB ini aux prorata des PM Euro#
#------------------------------------------#
s=(cr1[!,"eu"].+cr1[!,"ve"]).*cr1[!,"pm_0"]
som=max(1,sum(s)) #Eviter les divisions par zero qd pas de produits euros
ratio=s/som
cr1[!,"ppb0"].=ppb0.*ratio #Quid de la prev
cr2=copy(cr1)
#------------------------#
#Boucle sur les scenarios#
#------------------------#
#threads for k in 0:499
global cr1=copy(cr2)
cr1[!,"scn"].=k #scenario deterministe / parametrable en sto
#---------------------------------#
#Paramètres d"actifs et financiers#
#---------------------------------#
cr1 = leftjoin(cr1,tra,on = [:scn]) #TRA
cr1 = leftjoin(cr1,ts,on = [:tab_ts]) #TS
cr1 = leftjoin(cr1,ind_action,on = [:scn]) #Indice actions
cr1 = leftjoin(cr1,df,on = [:scn]) #Discount Factor
cr1 = leftjoin(cr1,tx_is,on = [:scn]) #Taux d"IS
cr1 = leftjoin(cr1,inf,on = [:scn]) #Inflation
##CR deterministe sur duree de projection
for i in 1:dur_proj
#---------#
##Technique
#---------#
cr1[!,"ch_prime"*str(i)].=cr1[!,"ch_prime"].*(cr1[!,"coll_"*str(i)].*(cr1[!,"eu"]
.+cr1[!,"uc"].+cr1[!,"ve"]).+cr1[!,"ca_prev"*str(i)])##chargements sur primes OK
cr1[!,"dc"*str(i)].=cr1[!,"qx"*str(i)].*(cr1[!,"pm_"*str(i-1)].+cr1[!,"ca_prev"*str(i)])#deces OK
cr1[!,"rach"*str(i)].=cr1[!,"lap"*str(i)].*cr1[!,"pm_"*str(i-1)].*(cr1[!,"eu"].+cr1[!,"uc"].+cr1[!,"ve"]) #Rachats structurels KO - Defo ancienneté avec collecte
cr1[!,"eche"*str(i)].=cr1[!,"ech"*str(i)].*cr1[!,"pm_"*str(i-1)]#Echeances OK
#cr1["arb"*str(i)]=cr1["sw"*str(i)]*cr1["pm_"+str(i-1)] #Arbitrages € vers UC KO lien produit multi
cr1[!,"ch_enc"*str(i)].=cr1[!,"ch_enc"].*cr1[!,"pm_"*str(i-1)].+(cr1[!,"coll_"*str(i)]
.-cr1[!,"rach"*str(i)].-cr1[!,"dc"*str(i)]).*((1 .+cr1[!,"ch_enc"]).^(0.5).- 1)#chargements sur encours OK
#RT Prevoyance (intègre de fait les chargements sur primes => à oter du résultat ass)
cr1[!,"rt"*str(i)].=(1 .-cr1[!,"SP"].-cr1[!,"ch_prime"]).*cr1[!,"ca_prev"*str(i)].*cr1[!,"prev"]
#---------#
##Financier
#---------#
#IT#
cr1[!,"IT"*str(i)].=(cr1[!,"eu"].+cr1[!,"ve"]).*(cr1[!,"IT"].*cr1[!,"pm_"*str(i-1)].+(cr1[!,"coll_"*str(i)]
.-cr1[!,"rach"*str(i)].-cr1[!,"dc"*str(i)]).*((1 .+cr1[!,"IT"]).^(0.5).-1))#IT OK
#PB distribuée
cr1[!,"pb_dist"*str(i)].=(cr1[!,"eu"].+cr1[!,"ve"]).*((cr1[!,"ts"*str(i)].-cr1[!,"IT"]).*cr1[!,"pm_"*str(i-1)]
.+(cr1[!,"coll_"*str(i)].-cr1[!,"rach"*str(i)].-cr1[!,"dc"*str(i)]).*((1 .+cr1[!,"ts"*str(i)]
.-cr1[!,"IT"]).^(0.5).-1))#PB dist OK
#Ajustements ACAV
cr1[!,"acav"*str(i)].=(cr1[!,"uc"].*cr1[!,"pm_"*str(i-1)].*(cr1[!,"inda"*str(i+1)]./cr1[!,"inda"*str(i)].-1)
.+(cr1[!,"coll_"*str(i)].-cr1[!,"ch_prime"*str(i)].-cr1[!,"rach"*str(i)].-cr1[!,"dc"*str(i)]).*cr1[!,"uc"]
.*(cr1[!,"inda"*str(i+1)]./cr1[!,"inda"*str(i)].-1)./2)
#Produits financiers contrats
cr1[!,"pfi"*str(i)].=(cr1[!,"eu"].+cr1[!,"ve"]).*cr1[!,"tra"*str(i)].*(2 .*cr1[!,"pm_"*str(i-1)]
.+cr1[!,"coll_"*str(i)].-cr1[!,"rach"*str(i)].-cr1[!,"dc"*str(i)])./2 #produits financiers par produits - hors PPB - hors FP
#Produits financiers sur PPB <=> Au prorata pfi contrats euros(suffisant en vision marginale)
cr1[!,"pfi_ppb"*str(i)].=cr1[!,"ppb"*str(i-1)].*cr1[!,"tra"*str(i)]
#CSG
cr1[!,"csg"*str(i)].=(cr1[!,"eu"].+cr1[!,"ve"]).*cr1[!,"csg"].*(cr1[!,"pb_dist"*str(i)].+cr1[!,"IT"*str(i)].-cr1[!,"ch_enc"*str(i)])
##PM fin
cr1[!,"pm_"*str(i)].=(cr1[!,"pm_"*str(i-1)].+cr1[!,"coll_"*str(i)].+cr1[!,"ca_prev"*str(i)].-cr1[!,"ch_prime"*str(i)]
.-cr1[!,"ch_enc"*str(i)].-cr1[!,"dc"*str(i)].-cr1[!,"rach"*str(i)].+cr1[!,"acav"*str(i)].-cr1[!,"eche"*str(i)]
.+cr1[!,"IT"*str(i)].+cr1[!,"pb_dist"*str(i)].-cr1[!,"csg"*str(i)]).*(cr1[!,"eu"].+cr1[!,"ve"].+cr1[!,"uc"])
#Retrocessions UC
cr1[!,"retro"*str(i)].=cr1[!,"retro"].*(cr1[!,"pm_"*str(i-1)].+cr1[!,"pm_"*str(i)]).*0.5
#Correctifs ratios de frais
cr1[!,"cor_fg_uc0"].=1
cr1[!,"cor_fg_eu0"].=1
cr1[!,"cor_fg_uc"*str(i)].=cr1[!,"cor_fg_uc"*str(i-1)].*((cr1[!,"inf"*str(i-1)].+cr1[!,"der_fg"])
./(cr1[!,"inda"*str(i+1)]./cr1[!,"inda"*str(i)]))
cr1[!,"cor_fg_eu"*str(i)].=cr1[!,"cor_fg_eu"*str(i-1)].*((cr1[!,"inf"*str(i-1)].+cr1[!,"der_fg"])
./(1 .+cr1[!,"ts"*str(i)].+cr1[!,"IT"]))
#Frais généraux
##FG d"administration epargne
cr1[!,"fg_ad"*str(i)].=cr1[!,"fga"*str(i)].*(cr1[!,"cor_fg_uc"*str(i)].*cr1[!,"uc"].+cr1[!,"cor_fg_eu"*str(i)]
.*(cr1[!,"eu"].+cr1[!,"prev"].+cr1[!,"ve"])).*(cr1[!,"pm_"*str(i-1)].+cr1[!,"pm_"*str(i)].-(cr1[!,"IT"*str(i)]
.+cr1[!,"pb_dist"*str(i)].+cr1[!,"csg"*str(i)]).*(cr1[!,"eu"].+cr1[!,"ve"]))./2
##FG d"aquisition
cr1[!,"fg_acq"*str(i)].=cr1[!,"fgacq"*str(i)].*(cr1[!,"coll_"*str(i)].+cr1[!,"ca_prev"*str(i)])
#FG Prestations
cr1[!,"fg_prest"*str(i)].=cr1[!,"fgp"*str(i)].*(cr1[!,"cor_fg_uc"*str(i)].*cr1[!,"uc"].+cr1[!,"cor_fg_eu"*str(i)]
.*(cr1[!,"eu"].+cr1[!,"prev"].+cr1[!,"ve"])).*(cr1[!,"rach"*str(i)].+cr1[!,"dc"*str(i)].+cr1[!,"ca_prev"*str(i)]
.*cr1[!,"SP"])
#Coms de gestion & admin (mma)
cr1[!,"com_ad"*str(i)].=cr1[!,"com_ad"].*(cr1[!,"pm_"*str(i-1)].+cr1[!,"pm_"*str(i)].-(cr1[!,"IT"*str(i)]
.+cr1[!,"pb_dist"*str(i)].+cr1[!,"csg"*str(i)]).*(cr1[!,"eu"].+cr1[!,"ve"]))./2#OK
cr1[!,"com_acq"*str(i)].=cr1[!,"com_acq"].*cr1[!,"coll_"*str(i)]##=>OK
#PB min
#Solde souscription
cr1[!,"ssous"*str(i)].=(cr1[!,"eu"].+cr1[!,"ve"].+cr1[!,"prev"]).*(cr1[!,"ch_enc"*str(i)].+cr1[!,"rt"*str(i)]
.+cr1[!,"ch_prime"*str(i)].-cr1[!,"fg_ad"*str(i)].-cr1[!,"fg_acq"*str(i)].-cr1[!,"fg_prest"*str(i)]
.-cr1[!,"com_ad"*str(i)].-cr1[!,"com_acq"*str(i)])
#Determination composante technique (4,5% Pprev ou 10% tech)
if tcomp_mod==1
cr1[!,"tcomp"*str(i)].=tcomp_fixe
elseif tcomp_mod==0 && sum(cr1[!,"ca_prev"*str(i)]).*0.045 .> sum(cr1[!,"ssous"*str(i)]).*0.1
cr1[!,"tcomp"*str(i)].=1
else
cr1[!,"tcomp"*str(i)].=0
end
#Quid du changement de contrainte sur le technique en fonction des produits enlevés?
cr1[!,"pb_min"*str(i)].=0.85.*(cr1[!,"pfi"*str(i)].+cr1[!,"pfi_ppb"*str(i)]).+(cr1[!,"ssous"*str(i)]
.-cr1[!,"tcomp"*str(i)].*cr1[!,"ca_prev"*str(i)].*0.045) .-((1 .-cr1[!,"tcomp"*str(i)]).*cr1[!,"ssous"*str(i)].*0.10)
#Variation de PPB par produit
cr1[!,"varppb"*str(i)].=cr1[!,"pb_min"*str(i)].-cr1[!,"IT"*str(i)].-cr1[!,"pb_dist"*str(i)]
#PPB par produit
cr1[!,"ppb"*str(i)].=cr1[!,"ppb"*str(i-1)].+cr1[!,"varppb"*str(i)]
#RT Vie entière
cr1[!,"rtve"*str(i)].=cr1[!,"ve"].*(-1).*(cr1[!,"ch_enc"*str(i)].-cr1[!,"fg_ad"*str(i)].-cr1[!,"fg_prest"*str(i)]
.-(cr1[!,"ca_prev"*str(i)].-cr1[!,"ch_prime"*str(i)].-cr1[!,"varppb"*str(i)].-cr1[!,"fg_ad"*str(i)]
.-cr1[!,"fg_prest"*str(i)].-cr1[!,"dc"*str(i)].-cr1[!,"rach"*str(i)]))
#PM finale yc RT Vie entière
cr1[!,"pm_"*str(i)].=cr1[!,"pm_"*str(i)].+cr1[!,"rtve"*str(i)].-cr1[!,"varppb"*str(i)].*cr1[!,"ve"]
#Impots
cr1[!,"IS"*str(i)].=(cr1[!,"ch_prime"*str(i)].+cr1[!,"ch_enc"*str(i)].+cr1[!,"pfi"*str(i)].+cr1[!,"pfi_ppb"*str(i)]
.-cr1[!,"pb_min"*str(i)].+cr1[!,"retro"*str(i)].+cr1[!,"rt"*str(i)].-cr1[!,"fg_ad"*str(i)].-cr1[!,"fg_acq"*str(i)]
.-cr1[!,"fg_prest"*str(i)].-cr1[!,"com_ad"*str(i)].-cr1[!,"com_acq"*str(i)]).*cr1[!,"tx_is"*str(i)]
#Résultat net Assurance apres IS
cr1[!,"rnai"*str(i)].=(cr1[!,"ch_prime"*str(i)].+cr1[!,"ch_enc"*str(i)].+cr1[!,"pfi"*str(i)].+cr1[!,"pfi_ppb"*str(i)]
.-cr1[!,"pb_min"*str(i)].+cr1[!,"retro"*str(i)].+cr1[!,"rt"*str(i)].-cr1[!,"fg_ad"*str(i)].-cr1[!,"fg_acq"*str(i)]
.-cr1[!,"fg_prest"*str(i)].-cr1[!,"com_ad"*str(i)].-cr1[!,"com_acq"*str(i)]).*(1 .-cr1[!,"tx_is"*str(i)])
#Profits futurs "actualisés (PVFP)
cr1[!,"pvfp"*str(i)].=cr1[!,"rnai"*str(i)].*cr1[!,"df"*str(i)]
end
#---------------------------------------------#
#Variables d"intérêt actualisés sur duree proj
#Principaux éléments de marge
#---------------------------------------------#
cr1[!,"ch_enc_actu"].=0
cr1[!,"ch_prime_actu"].=0
cr1[!,"pfnd_actu"].=0
cr1[!,"retro_actu"].=0
cr1[!,"rt_actu"].=0
cr1[!,"fg_ad_actu"].=0
cr1[!,"fg_acq_actu"].=0
cr1[!,"fg_prest_actu"].=0
cr1[!,"com_ad_actu"].=0
cr1[!,"com_acq_actu"].=0
cr1[!,"IS_actu"].=0
cr1[!,"pvfp_actu"].=0
cr1[!,"pm_actu"].=(cr1[!,"pm_0"].*cr1[!,"df1"].-cr1[!,"pm_"*str(dur_proj-2)].*cr1[!,"df"*str(dur_proj-1)])./2 #initialise
cr1[!,"ca_actu"].=0
cr1[!,"pfi_actu"].=0
cr1[!,"pfi_ppb_actu"].=0
cr1[!,"pb_dist_actu"].=0
cr1[!,"IT_actu"].=0
cr1[!,"varppb_actu"].=0
cr1[!,"dc_actu"].=0
cr1[!,"rach_actu"].=0
cr1[!,"marge_sous_actu"].=0
cr1[!,"marge_acq_actu"].=0
cr1[!,"marge_ges_actu"].=0
cr1[!,"marge_fi_actu"].=0
for i in 1:dur_proj
cr1[!,"ch_enc_actu"].=cr1[!,"ch_enc_actu"].+(cr1[!,"ch_enc"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"ch_prime_actu"].=cr1[!,"ch_prime_actu"].+(cr1[!,"ch_prime"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"pfnd_actu"].=cr1[!,"pfnd_actu"].+(cr1[!,"pfi"*str(i)].+cr1[!,"pfi_ppb"*str(i)].-cr1[!,"pb_min"*str(i)]).*cr1[!,"eu"].*cr1[!,"df"*str(i)]
cr1[!,"retro_actu"].=cr1[!,"retro_actu"].+cr1[!,"retro"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"rt_actu"].=cr1[!,"rt_actu"].+cr1[!,"rt"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"fg_ad_actu"].=cr1[!,"fg_ad_actu"].+cr1[!,"fg_ad"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"fg_acq_actu"].=cr1[!,"fg_acq_actu"].+cr1[!,"fg_acq"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"fg_prest_actu"].=cr1[!,"fg_prest_actu"].+cr1[!,"fg_prest"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"com_ad_actu"].=cr1[!,"com_ad_actu"].+cr1[!,"com_ad"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"com_acq_actu"].=cr1[!,"com_acq_actu"].+cr1[!,"com_acq"*str(i)].*cr1[!,"df"*str(i)]
cr1[!,"IS_actu"].=cr1[!,"IS_actu"].+(cr1[!,"IS"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"pvfp_actu"].=cr1[!,"pvfp_actu"].+cr1[!,"pvfp"*str(i)]#déjà actualisée
cr1[!,"ca_actu"].=cr1[!,"ca_actu"].+(cr1[!,"ca_prev"*str(i)].+cr1[!,"coll_"*str(i)]).*cr1[!,"df"*str(i)]
cr1[!,"pfi_actu"].=cr1[!,"pfi_actu"].+(cr1[!,"pfi"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"pfi_ppb_actu"].=cr1[!,"pfi_ppb_actu"].+(cr1[!,"pfi_ppb"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"pb_dist_actu"].=cr1[!,"pb_dist_actu"].+(cr1[!,"pb_dist"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"IT_actu"].=cr1[!,"IT_actu"].+(cr1[!,"IT"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"varppb_actu"].=cr1[!,"varppb_actu"].+(cr1[!,"varppb"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"dc_actu"].=cr1[!,"dc_actu"].+(cr1[!,"dc"*str(i)].*cr1[!,"df"*str(i)])
cr1[!,"rach_actu"].=cr1[!,"rach_actu"].+(cr1[!,"rach"*str(i)].*cr1[!,"df"*str(i)])
end
for i in 1:dur_proj-1
cr1[!,"pm_actu"].=cr1[!,"pm_actu"].+(cr1[!,"pm_"*str(i)].*cr1[!,"df"*str(i+1)])
end
cr1[!,"marge_sous_actu"].=cr1[!,"rt_actu"]
cr1[!,"marge_acq_actu"].=cr1[!,"ch_prime_actu"].-cr1[!,"com_acq_actu"].-cr1[!,"fg_acq_actu"]
cr1[!,"marge_ges_actu"].=cr1[!,"ch_enc_actu"].+cr1[!,"retro_actu"].-cr1[!,"com_ad_actu"].-cr1[!,"fg_ad_actu"].-cr1[!,"fg_prest_actu"]
cr1[!,"marge_fi_actu"].=cr1[!,"pfnd_actu"]
# cr3 = cr1[:, All(Between(:gamme,:pm_0), Between(:ch_enc_actu,:marge_fi_actu))]
end
end
This is what I would do (not seeing the full code it is hard to write a full solution (also this will eat up a lot of memory, but again - in order to optimize this I would need to see the code; what I write has a benefit of being simple to implement and not require locking).
So the recommendation is:
create cr2_vec = [copy(cr1) for i in 0:499]
in outer loop write cr2 = cr2_vec[k]
Then do all the processing on cr2
after the #threads loop finishes take cr2_vec, which will have updated data frames and from each data frame of this vector take the columns that are needed and add it to your original cr1 data frame.
A more advanced solution would be not to use a vector of data frames, but just a single data frame, as you do, but after the computing is done use a lock and within a lock update the global cr1 with only the computed columns (you need to use lock to avoid race condition).
EDIT
An example of a more efficient implementation:
~$ JULIA_NUM_THREADS=4 julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.4.2 (2020-05-23)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
julia> using DataFrames
julia> function example()
df = DataFrame(id=1:10)
tmp_df = df[!, :] # avoid copying columns, just copy the structure
l = Threads.ReentrantLock()
Threads.#threads for i in 1:500
# avoid copying columns, just copy the structure; I assume that you do not modidy existing columns
# if you do write copy(tmp_df) instead
local_df = tmp_df[!, :]
local_df.x = rand(nrow(local_df)) # here use any operations you want that add columns to local_df
Threads.lock(l)
# and here add new columns to df in a thread safe way
# I have made just a simple example
df[!, string("col", i)] = local_df.x
Threads.unlock(l)
end
return df
end
example (generic function with 1 method)
julia> example()
10×501 DataFrame. Omitted printing of 488 columns
│ Row │ id │ col126 │ col127 │ col128 │ col1 │ col376 │ col251 │ col129 │ col377 │ col252 │ col378 │ col130 │ col253 │
│ │ Int64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │ Float64 │
├─────┼───────┼───────────┼───────────┼───────────┼──────────┼────────────┼───────────┼───────────┼──────────┼──────────┼───────────┼───────────┼──────────┤
│ 1 │ 1 │ 0.0859079 │ 0.196322 │ 0.419307 │ 0.692845 │ 0.0649538 │ 0.271088 │ 0.061715 │ 0.580881 │ 0.962198 │ 0.799577 │ 0.168915 │ 0.147532 │
│ 2 │ 2 │ 0.943435 │ 0.367186 │ 0.999753 │ 0.955678 │ 0.0250972 │ 0.960388 │ 0.0186629 │ 0.360238 │ 0.482318 │ 0.630828 │ 0.638949 │ 0.078192 │
│ 3 │ 3 │ 0.47552 │ 0.73942 │ 0.81966 │ 0.354086 │ 0.00347341 │ 0.889408 │ 0.836998 │ 0.46696 │ 0.467453 │ 0.939678 │ 0.211346 │ 0.28351 │
│ 4 │ 4 │ 0.802997 │ 0.761556 │ 0.848161 │ 0.893623 │ 0.651274 │ 0.564887 │ 0.248991 │ 0.667879 │ 0.644271 │ 0.25262 │ 0.246616 │ 0.759315 │
│ 5 │ 5 │ 0.252624 │ 0.452797 │ 0.925494 │ 0.986887 │ 0.125357 │ 0.593984 │ 0.030369 │ 0.607307 │ 0.572907 │ 0.496147 │ 0.512023 │ 0.166856 │
│ 6 │ 6 │ 0.538396 │ 0.225402 │ 0.800646 │ 0.296837 │ 0.74292 │ 0.327231 │ 0.210256 │ 0.60462 │ 0.611555 │ 0.0593179 │ 0.662082 │ 0.932477 │
│ 7 │ 7 │ 0.0371803 │ 0.7965 │ 0.0444999 │ 0.178894 │ 0.106347 │ 0.780827 │ 0.637981 │ 0.237108 │ 0.627657 │ 0.944787 │ 0.378639 │ 0.878564 │
│ 8 │ 8 │ 0.811234 │ 0.187935 │ 0.227647 │ 0.262164 │ 0.620936 │ 0.0184777 │ 0.438722 │ 0.38764 │ 0.125682 │ 0.194858 │ 0.490097 │ 0.958929 │
│ 9 │ 9 │ 0.540951 │ 0.0577447 │ 0.0718953 │ 0.316012 │ 0.670796 │ 0.867076 │ 0.0432124 │ 0.929919 │ 0.272197 │ 0.410973 │ 0.455775 │ 0.21984 │
│ 10 │ 10 │ 0.471401 │ 0.185443 │ 0.817339 │ 0.757105 │ 0.717144 │ 0.257629 │ 0.528992 │ 0.347383 │ 0.912039 │ 0.905729 │ 0.0434874 │ 0.75463 │
Note that in the result the order of columns will depend in what order the parallel threads run, so probably you will want to reorder the columns after this operation manually.

Python Protobuf (IPv4/IPv6 address) into Clickhouse FixedString(16)

Using a simple protobuf definition like so:
syntax = "proto3";
package flowprotobuf;
message FlowMessage {
bytes SourceIP = 6;
bytes DestIP = 7;
}
What is the proper way to encode an IPv4/IPv6 address so as to insert it into a Clickhouse table with FixedString(16) types set for both SourceIP and DestIP?
After days of riding the struggle bus I'm currently doing the following (Python 3) to dump the protobuf stream to a Kafka topic (and then consuming via Clickhouse Kafka engine & materialized view) with a "good" result:
#!/usr/bin/env python
import flow_pb2
from google.protobuf.internal.encoder import _VarintBytes
from socket import inet_pton, AF_INET, AF_INET6
from binascii import hexlify
def pack_addr(ipaddr):
if ':' in ipaddr:
l = int(hexlify(inet_pton(AF_INET6, ipaddr)), 16)
return l.to_bytes(16, byteorder='big')
else:
l = int(hexlify(inet_pton(AF_INET, ipaddr)), 16)
return l.to_bytes(16, byteorder='big')
fm = flow_pb2.FlowMessage()
fm.SourceIP = pack_addr(ip_src)
fm.DestIP = pack_addr(ip_dst)
size = fm.ByteSize()
fpb = _VarintBytes(size) + fm.SerializeToString()
producer.produce(kafka_producer_topic, fpb)
producer.poll(0)
I put good in quotes because according to the Clickhouse documentation for IPv6NumToString():
Accepts a FixedString(16) value containing the IPv6 address in binary format. Returns a string containing this address in text format.
IPv6-mapped IPv4 addresses are output in the format ::ffff:111.222.33.44.
However my query results are not showing the ::ffff:x.x.x.x format - instead:
de33137dfc80 :) SELECT Date,TimeReceived,IPv6NumToString(SourceIP),IPv6NumToString(DestIP) FROM test LIMIT 5;
SELECT
Date,
TimeReceived,
IPv6NumToString(SourceIP),
IPv6NumToString(DestIP)
FROM test
LIMIT 5
┌───────Date─┬────────TimeReceived─┬─IPv6NumToString(SourceIP)─┬─IPv6NumToString(DestIP)─┐
│ 2020-08-05 │ 2020-08-05 06:41:27 │ ::98.158.157.211 │ ::202.122.147.98 │
│ 2020-08-05 │ 2020-08-05 06:41:27 │ ::98.158.157.211 │ ::217.118.23.125 │
│ 2020-08-05 │ 2020-08-05 06:41:27 │ ::192.34.21.69 │ ::104.34.73.41 │
│ 2020-08-05 │ 2020-08-05 06:41:27 │ ::98.158.157.211 │ ::194.28.167.103 │
│ 2020-08-05 │ 2020-08-05 06:41:27 │ ::98.158.148.89 │ ::79.170.71.49 │
└────────────┴─────────────────────┴───────────────────────────┴─────────────────────────┘
5 rows in set. Elapsed: 0.006 sec.
I know the IPv4 addresses are correct, and it's also properly showing IPv6 addresses. I just want to make sure I'm not missing anything glaring/obvious. Thanks.
Edited to add: Clickhouse server version 20.5.4 revision 54435
EDIT 2: Denis's suggestion below led me to come up with the solution:
else:
m = '::ffff:' + ipaddr
l = int(hexlify(inet_pton(AF_INET6, m)), 16)
return l.to_bytes(16, byteorder='big')
SELECT hex(IPv6StringToNum('::98.158.157.211'))
┌─hex(IPv6StringToNum('::98.158.157.211'))─┐
│ 000000000000000000000000629E9DD3 │
└──────────────────────────────────────────┘
SELECT hex(IPv6StringToNum('::ffff:98.158.157.211'))
┌─hex(IPv6StringToNum('::ffff:98.158.157.211'))─┐
│ 00000000000000000000FFFF629E9DD3 │
└───────────────────────────────────────────────┘
https://en.wikipedia.org/wiki/IPv6_address
For example, the IPv4-mapped IPv6 address ::ffff:c000:0280 is written as ::ffff:192.0.2.128, thus expressing clearly the original IPv4 address that was mapped to IPv6
https://www.ultratools.com/tools/ipv4toipv6
0:0:0:0:0:ffff:629e:9dd3
Converting IPv4 Address to a Hex IPv6 Address in Python

Is there a Tree Search function in Python to get all the parent for a given node based on Data, not by Node Id

Need to search a Tree based on Data.
Tried converting a Tree to Dict and search but did not work as expected. Requirement is to get all the related parent details of a node based on data.
tree = Tree()
tree.create_node(tag="Eatables",identifier="Food",data="Eatables") # root node
......
tree.create_node(tag="Apple",identifier="Apple1",parent="RF_01",data="Apple")
..............
tree.create_node(tag="Apple",identifier="Apple2",parent="AppJu1",data="Apple")
..............
tree.create_node(tag="Apple",identifier="Apple3",parent="Salad1",data="Apple")
............
tree.show()
Eatables
├── Ingredients
│ ├── Milk
│ ├── Sugar
│ └── Water
├── Processed_Food
│ ├── Apple_Juice
│ │ ├── Apple
│ │ └── Milk
│ └── Salad
│ ├── Apple
│ ├── Banana
│ ├── Beetroot
│ ├── Carrot
│ ├── JackFruit
│ └── Mango
├── Raw_Food
│ ├── Apple
│ ├── Banana
│ ├── Jack_Fruit
│ └── Mango
└── Vegetables
├── Beet_root
└── Carrot
Expected output is Node, Parent & grand_parent.
[ [Apple, Raw_food, Eatables],[Apple,Salad,Processed_Food],[Apple,Apple_Juice,Processed_Food] ]
By taking nodeid it is possible to get all the required data, fpointer and bpointer are very useful. Starting a search in a Tree data structure without nodeid is not an efficient way.

Resources