how to run code with %run inside a string in databricks - apache-spark

Hope all are doing well :D
I am trying to create a function with the path as a parameter.
Inside that function I would like to append that path to %run. And then run the notebook itself.
What did I do was:
def another_func(path):
nb = "%run" + str(path)
return nb
This return the everything in between "". I tried eval(), exec() but nothing works, it does return the following error: "invalid syntax (, line 1)".
I also tried instead of "nb = "%run" + str(path)", %run $path but it doesn´t work as well :(
Thank you so much in advance :)

I have tried same thing, but it worked
def another_func(path):
nb = "%run" + " " + str(path)
return nb
print(another_func("/sample/demo"))

The %run is a special directive and doesn't allow right now to parametrize the path. You can pass path to dbutils.notebook.run, but it will execute that notebook as a separate job, not including the functions & variables into the context of the current notebook.

Related

exec() not working when trying to execute a string containing the command "abs.__doc__"

I am trying to execute the command abs.__ doc__ inside the exec() function but for some reason it does not work.
function = input("Please enter the name of a function: ")
proper_string = str(function) + "." + "__doc__"
exec(proper_string)
Essentially, I am going through a series of exercises and one of them asks to provide a short description of the entered function using the __ doc__ attribute. I am trying with abs.__ doc__ but my command line comes empty. When I run python in the command line and type in abs.__ doc__ without anything else it works, but for some reason when I try to input it as a string into the exec() command I can't get any output. Any help would be greatly appreciated. (I have deliberately added spaces in this description concerning the attribute I am trying to use because I get bold type without any of the underscores showing.)
As a note, I do not think I have imported any libraries that could interfere, but these are the libraries that I have imported so far:
import sys
import datetime
from math import pi
My Python version is Python 3.10.4. My operating system is Windows 10.
abs.__doc__ is a string. You should use eval instead of exec to get the string.
Example:
function = input("Please enter the name of a function: ")
proper_string = str(function) + "." + "__doc__"
doc = eval(proper_string)
You can access it using globals():
def func():
"""Func"""
pass
mine = input("Please enter the name of a function: ")
print(globals()[mine].__doc__)
globals() return a dictionary that keeps track of all the module-level definitions. globals()[mine] is just trying to lookup for the name stored in mine; which is a function object if you assign mine to "func".
As for abs and int -- since these are builtins -- you can look it up directly using getattr(abs, "__doc__") or a more explicit: getattr(__builtins__, "abs").__doc__.
There are different ways to lookup for a python object corresponding to a given string; it's better not to use exec and eval unless really needed.

How to store windows system path in JMeter JSR223 sampler using groovy strings?

I am building a config in Jmeter where I specializing a system windows path to load CSV and write into CSV files. The path contains "" symbol.
There are some samplers with JSR223 PreProcessors and JSR223 Samplers. Language used is Groovy.
I know that I should screen symbols such this, but I am a bit lost at the moment because I do not understand why different options work sometimes and do not other times.
Initial path is "C:\user\jmeter\csv"
How I define this in jMeter JSR223:
def systemPath = "C:\\user\\jmeter\\csv\\"
Sometimes it works, but after I've changed some lines of code it won't!
Next time I've tried:
def systemPath = "C:\\\user\\\jmeter\\\csv\\\"
It worked for me for a while but after some code alterations after this line it won't work again.
I also 've read an article https://www.baeldung.com/groovy-strings and tried everything, but still I get this error every time:
javax.script.ScriptException: org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
Script343.groovy: 2: Unexpected input: '"' # line 2, column 16.
def systemPath = "C:\user\jmeter\csv\";
I want to store this system path reliably without errors and understand why I get this error.
I don't see any problems with this line:
def systemPath = "C:\\user\\jmeter\\csv\\"
this one is not good:
def systemPath = "C:\\\user\\\jmeter\\\csv\\\"
Your "work sometimes and do not other times" statement doesn't add any value unless you show jmeter.log preferably with debug level of verbosity for "working" and "not working" cases.
For the time being consider the following hints:
You can use forward slashes instead of escaping, i.e.
def systemPath = "C:/user/jmeter/csv/"
Or you can use File.separator property to get OS-specific "slash" value like:
def systemPath = "C:" + File.separator + "user" + File.separator + "jmeter" + File.separator + "csv" + File.separator
If you think that " character is a problem (it needs to be escaped as well by the way) you can declare Groovy strings using ' characters like:
def systemPath = 'C:\\user\\jmeter\\csv\\'

%s function Python 3

I want to build a function for following code:
PlayDirection_encoder = LabelEncoder()
train["PlayDirection"] = direction_encoder.fit_transform(train["PlayDirection"])
train.PlayDirection.unique()
That's my current function:
def object(category):
%s = Label_Encoder() % (category + "_encoder")
train[category] = %s.fit_transform(train[category]) %(category + "_encoder")
len(train.category.unique())
object("PlayDirection")
UsageError: Line magic function `%s` not found.
I am running my code on Kaggle's server.
Do you know how to solve this problem?
calling function an object is really really bad idea. It's a reserved word in Python, so it might break everything in your code.
what do you want to achieve? your problem is not clear

python - exec() line works in shell, but not while call in def

I try to get the related model for a foreignkey field :
for field in Model._meta.fields:
if "ForeignKey" in str(type(field)):
exec("related_model = Model." + field.name + ".get_query_set().model")
This works fine in shell, line per line.
But not when in a def:
def run(self):
for field in Model._meta.fields:
if "ForeignKey" in str(type(field)):
exec("related_model = Model." + field.name + ".get_query_set().model")
It seems that the variable 'related_model' is not created or taken in account further in the code.
Thank you in advance for any suggestions?
exec is a bad way to deal with this. Instead, use:
related_model = getattr(Model, field.name).get_query_set().model

Groovy : why this code doesn't work ? unable to resolve class

I have this test code:
def p = [:]
p.foo = [:]
p.foo.bar = 120
p.foo.bar - 3
(p.foo.bar) + 3
why on the last statement i get a compilation error : "unable to resolve class p.foo.bar "?
Thanks for the help
Groovy version 1.8.1
OK, I think I figured it out. I ran the AST browser against your sample script (using the GroovyConsole). It would only show an output at the Conversion phase. At this phase you can see how the script is converted. The key is that the last line is converted into this:
...
((1) as p.foo.bar)
This means that, apparently, it's trying to cast or convert the 1 into a class named p.foo.bar.
You can dig a little deeper and see that the parser is parsing the statement like this:
(p.foo.bar)(+1)
Which is the same as
(p.foo.bar)1
Therefore, the parser/compiler is seeing the + as a unary + operator. And that is why you are getting the error. (The way around it it to remove the parentheses, or swap the order of the arguments!)

Resources