How to Identify This Pyspark Coding Issue - No viable alternative at input - azure

Trying to run the following code in Azure Synapse pyspark and receive the parsing error, it doesn't seem like Synapse accept the double brackets, anyone know how to fix it?
def curated_report(entity_name):
sqlstr ="WITH Participant_Name \
AS (SELECT \
CASEID, \
PARTICIPANTID, \
LASTWRITTEN, \
PARTICIPANT, \
FIRSTNAME, \
MIDDLENAME, \
LASTNAME \
FROM (SELECT \
ab.CASEID, \
ab.PARTICIPANTID, \
ab.DYNAMICDATATYPE, \
ab.DYNAMICEVIDENCEVALUE, \
ab.LASTWRITTEN \
FROM a.ev ab \
INNER JOIN (SELECT \
PARTICIPANTID, \
MAX(LASTWRITTEN) AS MAXDATE \
FROM a.bd \
where TYPE in ( 'PDC001' ) \
GROUP BY PARTICIPANTID) cd \
ON ab.PARTICIPANTID = cd.PARTICIPANTID \
AND ab.LASTWRITTEN = cd.MAXDATE \
GROUP BY ab.CASEID, \
ab.PARTICIPANTID, \
ab.DYNAMICDATATYPE, \
ab.DYNAMICEVIDENCEVALUE, \
ab.LASTWRITTEN) AS SOURCE \
PIVOT(max(DYNAMICEVIDENCEVALUE) \
FOR DYNAMICDATATYPE IN (PARTICIPANT, \
FIRSTNAME, \
MIDDLENAME,\
LASTNAME) \
)AS RESULT) \ <----*this line seems to be causing error*
SELECT* \
FROM PARTICIPANT_NAME"
df = spark.sql(sqlstr)
return df
*solved.

ParseException:
no viable alternative at input 'WITH Participant_Name AS (SELECT ...
Remove as RESULT from the query. For CTE table, it is sufficient to give with CTE table name.
Error Screenshot
I tried to repro this with similar script and got the same error.
In order to avoid this error, I removed the alias name as Result from the Query and it is executed successfully.
Corrected Code
sqlstr ="WITH Participant_Name \
AS (SELECT \
CASEID, \
PARTICIPANTID, \
LASTWRITTEN, \
PARTICIPANT, \
FIRSTNAME, \
MIDDLENAME, \
LASTNAME \
FROM (SELECT \
ab.CASEID, \
ab.PARTICIPANTID, \
ab.DYNAMICDATATYPE, \
ab.DYNAMICEVIDENCEVALUE, \
ab.LASTWRITTEN \
FROM enhanced.BDMCASEEVIDENCE ab \
INNER JOIN (SELECT \
PARTICIPANTID, \
MAX(LASTWRITTEN) AS MAXDATE \
FROM enhanced.BDMCASEEVIDENCE \
where EVIDENCETYPE in ( 'PDC0000258' ) \
GROUP BY PARTICIPANTID) cd \
ON ab.PARTICIPANTID = cd.PARTICIPANTID \
AND ab.LASTWRITTEN = cd.MAXDATE \
GROUP BY ab.CASEID, \
ab.PARTICIPANTID, \
ab.DYNAMICDATATYPE, \
ab.DYNAMICEVIDENCEVALUE, \
ab.LASTWRITTEN) AS SOURCE \
PIVOT(max(DYNAMICEVIDENCEVALUE) \
FOR DYNAMICDATATYPE IN ('PARTICIPANT', \
'FIRSTNAME', \
'MIDDLENAME',\
'LASTNAME') \
)) \
SELECT* \
FROM PARTICIPANT_NAME"
df = spark.sql(sqlstr)

Related

How to cross-compile a rust application for ARM which uses Rocket web-server and requires nightly toolchain using Yocto?

I want to compile myRustApp which uses Rocket webserver in my meta layer. The issu I'm having is that openembedded does not support nightly release of Rust, which is required to run Rocket web-server.
The alternative was to use meta-rust, but it got me no where, because if I understand correctly the meta layer only supports native builds.
So I ended up using meta-rust-bin layer with pre-built nightly toolchain.
I was able to build nightly by executing ./build-new-version.sh nightly under meta-rust-bin.
After all this my recipe build by bitbake returns an Error:
WARNING: /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross_1.59.0.bb: Exception during build_dependencies for TARGET_LLVM_FEATURES
WARNING: /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross_1.59.0.bb: Error during finalise of /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross_1.59.0.bb
ERROR: ExpansionError during parsing /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross_1.59.0.bb
Traceback (most recent call last):
File "Var <TARGET_LLVM_FEATURES>", line 1, in <module>
File "/home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-common.inc", line 117, in llvm_features(d=<bb.data_smart.DataSmart object at 0x7fe03b341f60>):
return ','.join(llvm_features_from_tune(d) +
> llvm_features_from_cc_arch(d) +
llvm_features_from_target_fpu(d))
File "/home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-common.inc", line 36, in llvm_features_from_tune(d=<bb.data_smart.DataSmart object at 0x7fe03b341f60>):
> if target_is_armv7(d):
f.append('+v7')
bb.data_smart.ExpansionError: Failure expanding variable TARGET_LLVM_FEATURES, expression was ${#llvm_features(d)} which triggered exception NameError: name 'target_is_armv7' is not defined
The variable dependency chain for the failure is: TARGET_LLVM_FEATURES
ERROR: Parsing halted due to errors, see error messages above
WARNING: /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross-canadian_1.59.0.bb: Exception during build_dependencies for TARGET_LLVM_FEATURES
WARNING: /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross-canadian_1.59.0.bb: Error during finalise of /home/jonas/project/sources/openembedded-core/meta/recipes-devtools/rust/rust-cross-canadian_1.59.0.bb
My question is:
What causes this error I'm getting? How can I fix it?
Did someone already tried to cross-compile Rust nightly apps? Are there any examples anywhere?
My Cargo.toml:
[package]
name = "MyRustApp"
version = "0.1.0"
edition = "2021"
# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[dependencies]
rocket = "0.4.10"
rocket_cors = "0.5.1"
serde = "1.0"
serde_json = "1.0"
serde_derive = "1.0"
[dependencies.rocket_contrib]
version = "0.4.10"
default-features = false
features = ["json"]
[profile.release]
strip = true
opt-level = "z"
lto = true
My recipe generated by cargo bitbake:
# Auto-Generated by cargo-bitbake 0.3.16
#
inherit cargo
# If this is git based prefer versioned ones if they exist
# DEFAULT_PREFERENCE = "-1"
# how to get strain-webserver could be as easy as but default to a git checkout:
# SRC_URI += "crate://crates.io/MyRustApp/0.1.0"
SRC_URI += "git://git#bitbucket.org/work/MyRustApp.git;protocol=ssh;nobranch=1;branch=main"
SRCREV = "xxx"
S = "${WORKDIR}/git"
CARGO_SRC_DIR = ""
PV:append = ".AUTOINC+d2562d3c92"
# please note if you have entries that do not begin with crate://
# you must change them to how that package can be fetched
SRC_URI += " \
crate://crates.io/aead/0.3.2 \
crate://crates.io/aes-gcm/0.8.0 \
crate://crates.io/aes-soft/0.6.4 \
crate://crates.io/aes/0.6.0 \
crate://crates.io/aesni/0.10.0 \
crate://crates.io/aho-corasick/0.7.20 \
crate://crates.io/atty/0.2.14 \
crate://crates.io/autocfg/1.1.0 \
crate://crates.io/base64/0.13.1 \
crate://crates.io/base64/0.9.3 \
crate://crates.io/bitflags/1.3.2 \
crate://crates.io/block-buffer/0.9.0 \
crate://crates.io/byteorder/1.4.3 \
crate://crates.io/cfg-if/0.1.10 \
crate://crates.io/cfg-if/1.0.0 \
crate://crates.io/cipher/0.2.5 \
crate://crates.io/cookie/0.11.5 \
crate://crates.io/cpufeatures/0.2.5 \
crate://crates.io/cpuid-bool/0.2.0 \
crate://crates.io/crypto-mac/0.10.1 \
crate://crates.io/ctr/0.6.0 \
crate://crates.io/devise/0.2.1 \
crate://crates.io/devise_codegen/0.2.1 \
crate://crates.io/devise_core/0.2.1 \
crate://crates.io/digest/0.9.0 \
crate://crates.io/filetime/0.2.18 \
crate://crates.io/form_urlencoded/1.1.0 \
crate://crates.io/fsevent-sys/2.0.1 \
crate://crates.io/fsevent/0.4.0 \
crate://crates.io/fuchsia-zircon-sys/0.3.3 \
crate://crates.io/fuchsia-zircon/0.3.3 \
crate://crates.io/generic-array/0.14.6 \
crate://crates.io/getrandom/0.2.8 \
crate://crates.io/ghash/0.3.1 \
crate://crates.io/glob/0.3.0 \
crate://crates.io/hashbrown/0.12.3 \
crate://crates.io/hermit-abi/0.1.19 \
crate://crates.io/hkdf/0.10.0 \
crate://crates.io/hmac/0.10.1 \
crate://crates.io/httparse/1.8.0 \
crate://crates.io/hyper/0.10.16 \
crate://crates.io/idna/0.1.5 \
crate://crates.io/idna/0.3.0 \
crate://crates.io/indexmap/1.9.2 \
crate://crates.io/inotify-sys/0.1.5 \
crate://crates.io/inotify/0.7.1 \
crate://crates.io/iovec/0.1.4 \
crate://crates.io/itoa/1.0.4 \
crate://crates.io/kernel32-sys/0.2.2 \
crate://crates.io/language-tags/0.2.2 \
crate://crates.io/lazycell/1.3.0 \
crate://crates.io/libc/0.2.137 \
crate://crates.io/log/0.3.9 \
crate://crates.io/log/0.4.17 \
crate://crates.io/matches/0.1.9 \
crate://crates.io/memchr/2.5.0 \
crate://crates.io/mime/0.2.6 \
crate://crates.io/mio-extras/2.0.6 \
crate://crates.io/mio/0.6.23 \
crate://crates.io/miow/0.2.2 \
crate://crates.io/net2/0.2.38 \
crate://crates.io/notify/4.0.17 \
crate://crates.io/num_cpus/1.14.0 \
crate://crates.io/opaque-debug/0.3.0 \
crate://crates.io/pear/0.1.5 \
crate://crates.io/pear_codegen/0.1.5 \
crate://crates.io/percent-encoding/1.0.1 \
crate://crates.io/percent-encoding/2.2.0 \
crate://crates.io/polyval/0.4.5 \
crate://crates.io/ppv-lite86/0.2.17 \
crate://crates.io/proc-macro2/0.4.30 \
crate://crates.io/proc-macro2/1.0.47 \
crate://crates.io/quote/0.6.13 \
crate://crates.io/quote/1.0.21 \
crate://crates.io/rand/0.8.5 \
crate://crates.io/rand_chacha/0.3.1 \
crate://crates.io/rand_core/0.6.4 \
crate://crates.io/redox_syscall/0.2.16 \
crate://crates.io/regex-syntax/0.6.28 \
crate://crates.io/regex/1.7.0 \
crate://crates.io/rocket/0.4.11 \
crate://crates.io/rocket_codegen/0.4.11 \
crate://crates.io/rocket_contrib/0.4.11 \
crate://crates.io/rocket_cors/0.5.2 \
crate://crates.io/rocket_http/0.4.11 \
crate://crates.io/ryu/1.0.11 \
crate://crates.io/safemem/0.3.3 \
crate://crates.io/same-file/1.0.6 \
crate://crates.io/serde/1.0.147 \
crate://crates.io/serde_derive/1.0.147 \
crate://crates.io/serde_json/1.0.89 \
crate://crates.io/sha2/0.9.9 \
crate://crates.io/slab/0.4.7 \
crate://crates.io/smallvec/1.10.0 \
crate://crates.io/state/0.4.2 \
crate://crates.io/subtle/2.4.1 \
crate://crates.io/syn/0.15.44 \
crate://crates.io/syn/1.0.103 \
crate://crates.io/time/0.1.44 \
crate://crates.io/tinyvec/1.6.0 \
crate://crates.io/tinyvec_macros/0.1.0 \
crate://crates.io/toml/0.4.10 \
crate://crates.io/traitobject/0.1.0 \
crate://crates.io/typeable/0.1.2 \
crate://crates.io/typenum/1.15.0 \
crate://crates.io/unicase/1.4.2 \
crate://crates.io/unicase/2.6.0 \
crate://crates.io/unicase_serde/0.1.0 \
crate://crates.io/unicode-bidi/0.3.8 \
crate://crates.io/unicode-ident/1.0.5 \
crate://crates.io/unicode-normalization/0.1.22 \
crate://crates.io/unicode-xid/0.1.0 \
crate://crates.io/universal-hash/0.4.1 \
crate://crates.io/url/1.7.2 \
crate://crates.io/url/2.3.1 \
crate://crates.io/version_check/0.1.5 \
crate://crates.io/version_check/0.9.4 \
crate://crates.io/walkdir/2.3.2 \
crate://crates.io/wasi/0.10.0+wasi-snapshot-preview1 \
crate://crates.io/wasi/0.11.0+wasi-snapshot-preview1 \
crate://crates.io/winapi-build/0.1.1 \
crate://crates.io/winapi-i686-pc-windows-gnu/0.4.0 \
crate://crates.io/winapi-util/0.1.5 \
crate://crates.io/winapi-x86_64-pc-windows-gnu/0.4.0 \
crate://crates.io/winapi/0.2.8 \
crate://crates.io/winapi/0.3.9 \
crate://crates.io/windows-sys/0.42.0 \
crate://crates.io/windows_aarch64_gnullvm/0.42.0 \
crate://crates.io/windows_aarch64_msvc/0.42.0 \
crate://crates.io/windows_i686_gnu/0.42.0 \
crate://crates.io/windows_i686_msvc/0.42.0 \
crate://crates.io/windows_x86_64_gnu/0.42.0 \
crate://crates.io/windows_x86_64_gnullvm/0.42.0 \
crate://crates.io/windows_x86_64_msvc/0.42.0 \
crate://crates.io/ws2_32-sys/0.2.1 \
crate://crates.io/yansi/0.5.1 \
"
# FIXME: update generateme with the real MD5 of the license file
LIC_FILES_CHKSUM = " \
"
SUMMARY = "Webserver with Rust"t"
LICENSE = "CLOSED"
# includes this file if it exists but does not fail
# this is useful for anything you may want to override from
# what cargo-bitbake generates.
include MyRustApp-${PV}.inc
include MyRustApp.inc

kubernetes config map - syntax error: unterminated quoted string

I am getting below error when trying to attach shell script as config map.
I am not sure what's the issue because script work without adding in config map
It shows error is on the line 58
Which is not even there.
Any help will be really appreciated.
apiVersion: v1
kind: ConfigMap
metadata:
name: {{ .Values.metadata.name }}-micro
data:
micro-integrator.sh: |
#!/bin/sh
# micro-integrator.sh
while [ "$status" = "$START_EXIT_STATUS" ]
do
$JAVACMD \
-Xbootclasspath/a:"$CARBON_XBOOTCLASSPATH" \
$JVM_MEM_OPTS \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath="$CARBON_HOME/repository/logs/heap-dump.hprof" \
$JAVA_OPTS \
-Dcom.sun.management.jmxremote \
-classpath "$CARBON_CLASSPATH" \
-Djava.io.tmpdir="$CARBON_HOME/tmp" \
-Dcatalina.base="$CARBON_HOME/wso2/lib/tomcat" \
-Dwso2.server.standalone=true \
-Dcarbon.registry.root=/ \
-Djava.command="$JAVACMD" \
-Dqpid.conf="/conf/advanced/" \
$JAVA_VER_BASED_OPTS \
-Dcarbon.home="$CARBON_HOME" \
-Dlogger.server.name="micro-integrator" \
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager \
-Dcarbon.config.dir.path="$CARBON_HOME/conf" \
-Dcarbon.repository.dir.path="$CARBON_HOME/repository" \
-Dcarbon.components.dir.path="$CARBON_HOME/wso2/components" \
-Dcarbon.dropins.dir.path="$CARBON_HOME/dropins" \
-Dcarbon.external.lib.dir.path="$CARBON_HOME/lib" \
-Dcarbon.patches.dir.path="$CARBON_HOME/patches" \
-Dcarbon.internal.lib.dir.path="$CARBON_HOME/wso2/lib" \
-Dcom.atomikos.icatch.hide_init_file_path=true \
-Dorg.apache.jasper.compiler.Parser.STRICT_QUOTE_ESCAPING=false \
-Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true \
-Dcom.sun.jndi.ldap.connect.pool.authentication=simple \
-Dcom.sun.jndi.ldap.connect.pool.timeout=3000 \
-Dorg.terracotta.quartz.skipUpdateCheck=true \
-Djava.security.egd=file:/dev/./urandom \
-Dfile.encoding=UTF8 \
-Djava.net.preferIPv4Stack=true \
-DNonRegistryMode=true \
-DNonUserCoreMode=true \
-Dcom.ibm.cacheLocalHost=true \
-Dcarbon.use.registry.repo=false \
-DworkerNode=false \
-Dorg.apache.cxf.io.CachedOutputStream.Threshold=104857600 \
-DavoidConfigHashRead=true \
-Dproperties.file.path=default \
-DenableReadinessProbe=true \
-DenableManagementApi=true \
$NODE_PARAMS \
-Dorg.apache.activemq.SERIALIZABLE_PACKAGES="*" \
org.wso2.micro.integrator.bootstrap.Bootstrap $*
status="$?"
done

How to create user for connect to database

ERROR :
[FATAL] [DBT-05509] Failed to connect to the specified database (cdb21).
CAUSE: OS Authentication might be disabled for this database (cdb21).
ACTION: Specify a valid sysdba user name and password to connect to the database.
First step:
./runInstaller -silent -responseFile /scratch/app/user/product/21.0.0/dbhome_1/install/response/db_install.rsp \
oracle.install.option=INSTALL_DB_SWONLY \
UNIX_GROUP_NAME=oinstall \
ORACLE_BASE=/scratch/app/user \
INVENTORY_LOCATION=/scratch/app/oraInventory \
SELECTED_LANGUAGES=en \
oracle.install.db.InstallEdition=EE \
oracle.install.db.isCustomInstall=false \
oracle.install.db.OSDBA_GROUP=oinstall \
oracle.install.db.OSBACKUPDBA_GROUP=oinstall \
oracle.install.db.OSDGDBA_GROUP=oinstall \
oracle.install.db.OSKMDBA_GROUP=oinstall \
oracle.install.db.OSRACDBA_GROUP=oinstall \
SECURITY_UPDATES_VIA_MYORACLESUPPORT=false \
DECLINE_SECURITY_UPDATES=true
Second step:
dbca -silent -createDatabase \
-templateName General_Purpose.dbc \
-gdbname cdb21 \
-sid cdb21 \
-responseFile NO_VALUE \
-characterSet AL32UTF8 \
-sysPassword Welcome1 \
-systemPassword Welcome1 \
-createAsContainerDatabase true \
-numberOfPDBs 1 \
-pdbName pdb21 \
-pdbAdminPassword Welcome1 \
-databaseType MULTIPURPOSE \
-memoryMgmtType auto_sga \
-totalMemory 4096 \
-storageType FS \
-datafileDestination /scratch/oradata/ \
-emConfiguration NONE \
-ignorePreReqs
Start The service using :
lsnrctl start
Then :
startup

How to overwrite specific partitions in spark/BigQuery/GCP

How to overwrite specific partitions in spark/BigQuery/GCP?
I user this Code,
df.write \
.format("bigquery") \
.mode("overwrite") \
.option("table",bq_path) \
.option("temporaryGcsBucket",GCS_BUCKET) \
.option("partitionField",'partition_date') \
.save()
but when batch is Executed, the existing data is deleted.
Please Refer this Image. -->>
enter image description here
You can do like that:
df.write.format('bigquery') \
.option("datePartition", "20220810").mode("overwrite") \
.option("partitionField", "processed_date") \
.option('table', "YOUR_TABLE") \
.save()

why run "python run_squad.py" doesn't work?

I want fine tune on squad with huggingface run_squad.py, but meet the following question:
1, when I use "--do_train" without "True" as following code, after 20 minutes runing,there is no models in output_dir:
!python run_squad.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--output_dir models/bert/ \
--data_dir data/squad \
--overwrite_output_dir \
--overwrite_cache \
--do_train \
--train_file train-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--do_eval \
--predict_file dev-v2.0.json \
--per_gpu_train_batch_size 2 \
--learning_rate 3e-5 \
--num_train_epochs 2.0 \
--max_seq_length 384 \
--doc_stride 128 \
--threads 10 \
--save_steps 5000
2, when I use "--do_train=True" as following code, the error message is "run_squad.py: error: argument --do_train: ignored explicit argument 'True'":
!python run_squad.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--output_dir models/bert/ \
--data_dir data/squad \
--overwrite_output_dir \
--overwrite_cache \
--do_train=True \
--train_file train-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--do_eval \
--predict_file dev-v2.0.json \
--per_gpu_train_batch_size 2 \
--learning_rate 3e-5 \
--num_train_epochs 2.0 \
--max_seq_length 384 \
--doc_stride 128 \
--threads 10 \
--save_steps 5000
3, when I use "--do_train True" as following code, the error message is "run_squad.py: error: unrecognized arguments: True":
!python run_squad.py \
--model_type bert \
--model_name_or_path bert-base-uncased \
--output_dir models/bert/ \
--data_dir data/squad \
--overwrite_output_dir \
--overwrite_cache \
--do_train True \
--train_file train-v2.0.json \
--version_2_with_negative \
--do_lower_case \
--do_eval \
--predict_file dev-v2.0.json \
--per_gpu_train_batch_size 2 \
--learning_rate 3e-5 \
--num_train_epochs 2.0 \
--max_seq_length 384 \
--doc_stride 128 \
--threads 10 \
--save_steps 5000
I run code in colab with GPU: Tesla P100-PCIE-16GB
Judging by the running time, I think the code didn't through training process, but I don't know how to set parameters in order to let training go.what should I do?

Resources