I want to run this command ccloud create topic mytopic
using python
You can create topics using actual Python Kafka clients, so you shouldn't need this command.
But, assuming you still wanted to use it, you'd use subprocess module to run exactly that string, as commented
Related
I am using the flashtext library in a couple of UDFs. It works when I run it locally in Client mode, but once I try to run it in the Cloudera Workbench with several executors, I get an ModuleNotFoundError.
After some research I found that it is possible to add archives (and packages?) to a SparkSession when creating it, so I tried:
SparkSession.builder.config('spark.archives', 'flashtext-2.7-pyh9f0a1d_0.tar.gz')
but it didn't help, the same error remains.
According to Spark Configuration doc, there are other configs I could try, e.g. spark.submit.pyFiles, but I don't understand how these py-files to be added would have to look like.
Would it be enough to just create a pyton script with this content?
from flashtext import KeywordProcessor
Could you tell me the easiest way how I can install flashtext on every node?
Edit:
In the meantime, I figured that not only Flashtext was causing issues, but also every relative import from other scripts that I intended to use in a UDF. In order to fix it, I followed this article. I also took the source code from Flashtext and imported it to the main file without installing the actual library.
I think in order to point Spark executors to python modules extracted from your archive, you will need to add another config setting, that adds their location to PYTHONPATH. Something like this:
SparkSession.builder \
.config('spark.archives', 'flashtext-2.7-pyh9f0a1d_0.tar.gz#myUDFs') \
.config('spark.executorEnv.PYTHONPATH', './myUDFs')
Citing from the same link you have in the question:
spark.executorEnv.[EnvironmentVariableName]...Add the environment
variable specified by EnvironmentVariableName to the Executor process.
The user can specify multiple of these to set multiple environment
variables.
There are no environment details in your question (or I'm simply not familiar with Cloudera Workbench) but if you're trying to run Spark on YARN, you may need to use slightly different setting spark.yarn.dist.archives.
Also, please make sure that your driver log contains message confirming that an archive was actually uploaded, as in:
:
22/11/08 INFO yarn.Client: Uploading resource file:/absolute/path/to/your/archive.zip -> hdfs://nameservice/user/<your-user-id>/.sparkStaging/<application-id>/archive.zip
:
Does anybody knows, how we can run test cases on a data(function) sent by the user in Django, (as implemented by leetcode, codesignals and codewar)
How my function(solution) is tested against these test, how can I implement this functionality at backend using django, django rest framework (click on this link to see the image)
import subprocess
def test_submission():
compiled_code = "test.out"
test_input = open("path/to/test_input.txt")
submission_output = open("path/to/submission_output.txt")
cmd = [f"./{compiled_code}"]
subprocess.run(cmd, stdin=test_input, stdout=submission_output)
# At this point the output of executed code with test_input is stored in "path/to/submission_output.txt"
return
The test_program can be any file be it C, C++ or Python.
If you don't need test support, you should be able to run code easily by just using the official container images for the language and the Docker CLI.
and there is a tool too which was used by codewar and qualified,
and you can also see this clone of codewar too for better understanding, I think it's bit simplified and a practical implementation of this codewar CLI too,
and one last thing, if you are in python use can also use eval but the prob with using this, some malicious user can insert rogue script that could be harmful, so I think you should avoid using it,
personally I think, you should use the simple docker option with a bit of sandboxing, because in our case, It's mostly just a thin wrapper around Docker API that takes the submitted code, prepares the environment and executes them. It's so simple that the original PoC was just few lines of shell script and a tiny tool written in Go.
Hellow Guys
First of all, i'm using the windows 10, java_1.8.0_333, and Pentaho 9.3
I'm trying to many ways to run my Python code in Pentaho, but isn't work.
I try reinstall my Python using diferent version, try to change my java version, and try oldest version of Pentaho, but no way works.
If a ran my Python script using the PyCharm, it's works, but if i try to run the same script, or create a new Python script in Pentaho, does't works, and i receive this error.
Inside CPython is a basic script that declares the variable name and receives the string "Test"
I think this error is about somenthing in my computer, but i don't no why
Can anyone help me with this ?
I wish to create one NodeJS source file in a Jupyter notebook which is using the IJavascript kernel so that I can quickly debug my code. Once I have it working, I can then use the "Download As..." feature of Jupyter to save the notebook as a NodeJS script file.
I'd like to have the ability to selectively ignore / include code in the notebook source that will not execute when I run the generated NodeJS script file.
I have solved this problem for doing a similar thing for Python Jupyter notebooks because I can determine if the code is running in an interactive session (IPython [REPL]). I accomplished this by using this function in Python:
def is_interactive():
import __main__ as main
return not hasattr(main, '__file__')
(Thanks to Tell if Python is in interactive mode)
Is there a way to do a similar thing for NodeJS?
I don't know if this is the correct way but couldn't find anything else
basically if you
try {
const repl = __dirname
} catch (err) {
//code run if repl
}
it feels a little hacky but works ¯\_(ツ)_/¯
This may not help the OP in all cases, but could help others googling for this question. Sometimes it's enough to know if the script is running interactively or not (REPL and any program that is run from a shell).
In that case, you can check for whether standard output is a TTY:
process.stdout.isTTY
The fastest and most reliable route would just be to query the process arguments. From the NodeJS executable alone, there are two ways to launch the REPL. Either you do something like this without any script following the call to node.
node --experimental-modules ...
Or you force node into the REPL using interactive mode.
node -i ...
The option ending parameter added in v6.11.0 -- will never append arguments into the process.argv array unless it's executing in script mode; via FILE, -p, or -e. Any arguments meant for NodeJS will be filtered into the accompanying process.execArgv variable, so the only thing left in the process.argv array should be process.execPath. Under these circumstances, we can reduce the query to the solution below.
const isREPL = process.execArgv.includes("-i") || process.argv.length === 1;
console.log(isREPL ? "You're in the REPL" : "You're running a script m8");
This isn't the most robust method since any user can otherwise instantiate a REPL from an intiator script which your code could be ran by. For that I'm pretty sure you could use an artificial error to crawl the traceback and look for a REPL entry. Although I haven't the time to implement and ensure that solution at this time.
I need to run a service program, written in FO for abas-ERP continuous.
I heard about some already existing scripts for calling service programs from the shell. If that is possible I could simply use a cronjob for starting this script.
But I don't know exactly where to find a template for these shell scripts, which conditions have to be complied and if there are any restrictions.
For Example: Is it possible to call several FO-programs successively (this might be important relating to blocking licences)?
You can use edpinfosys.sh and execute infosystem TEXTZEIGEN per cronjob.
You could also use batchlg.sh
batchlg.sh 'FOP-Name' [ -PASSARGS ] [Parameter ...]