Trying to run Pyspark job on Google Cloud Dataproc.
Even after the installation of sparknlp I'm getting this error.
I'm trying to import like this:
from sparknlp.annotator import Lemmatizer, Stemmer, Tokenizer, Normalizer
And getting this error:
ModuleNotFoundError: No module named 'sparknlp'
Can you help me in finding the reason behind this?
Related
I need to use spark in transfer learning to train images ,the error is:
"nnot import name 'resnet50' from 'keras.applications' (/usr/local/lib/python3.7/dist-packages/keras/applications/init.py) "
i try to solve this question since one week, this one is coming from sparkdl, if you add to this file (sparkdl/transformers/keras_applications.py)
**
from tensorflow.keras.applications
**, it will be return normal, but this time you will see another error like
AttributeError: module 'tensorflow' has no attribute 'Session'
i tried on different IDE (Pycharm, Vs Code) but i got the same errors. there are different explications on Stackoverflow. but i'm totally confused now
I am trying to follow this tutorial on Google Cloud Platform,
https://github.com/GoogleCloudPlatform/ai-platform-samples/blob/master/notebooks/samples/tables/census_income_prediction/getting_started_notebook.ipynb, however, I am running into issues when I try to import the autoML module, specifically the below two lines
# AutoML library.
from google.cloud import automl_v1beta1 as automl
import google.cloud.automl_v1beta1.proto.data_types_pb2 as data_types
The first line works, but for the 2nd one, I get the error: ModuleNotFoundError: No module named 'google.cloud.automl_v1beta1.proto'. It seems for some reason there is no module called proto and I cannot figure out how to resolve this. There are a couple of posts regarding the issue of not being able to find module google.cloud. In my case I am able to import automl_v1beta1 from google.cloud but not proto.data_types_pb2 from google.cloud.automl_v1beta1
I think you can:
from google.cloud import automl_v1beta1 as automl
import google.cloud.automl_v1beta1.types as data_types
Or:
import google.cloud.automl_v1beta1 as automl
import google.cloud.automl_v1beta1.types as data_types
But (!) given the import errors, there may be other changes to the SDK in the code that follows.
Before reading my question I would like to clarify that running the program within the python shell will run without any issues. I need to run some code through jupyter notebook and only then does importing within jupyter notebook will result in a No Module found error. This issue is described further below.
I have a project with the following hierarchy:
-ProjectFolder
-test.ipynb
-modules
-Game.py
-Grid.py
I was able to successfully import both Game.py and Grid.py into my jupyter notebook test file by adding the two lines to import each python file like below:
test.ipynb
from modules import Game
from modules import Grid
Unfortunately for this project there is a hard requirement. I must import the Grid Class within the Game.py file. Because both Game.py and Grid.py reside in the same folder I thought I could simply add the line import Grid at the top of Game.py without any issues. I did this by adding the line importGrid:
Game.py
import Grid
...
class Grid(object):
// grid structure logic
With that line, my test.ipynb now imports only the Game class from Game.py like so:
test.ipynb
from modules import Game
...
class Game(object):
// game logic
When running test in jupyter I unfortunately get the error No module named 'Grid'. Could anyone please tell me what I am doing wrong in this case? I have tried several things like relative imports trying from .Grid import Grid but this didnt work either and had a module not found error as well"
Trying to create dummy data from a dataset.
Looked for hours to see why I'm getting this ImportError. I have Faker 2.0.0 installed.
import unicodecsv as csv
from faker import Faker
from collections import defaultdict
ImportError: cannot import name 'Faker' from 'faker' (unknown
location)
Receiving this error message still! I tried using solutions from other forum questions, to avail. Anyone have suggestions?
I found it. If you are dealing with the same issue look in your Scripts folder within your Python directory. You'll find an application named faker also. Rename it and you'll be good to go.
I am trying to replicate a deep learning project from https://medium.com/linagora-engineering/making-image-classification-simple-with-spark-deep-learning-f654a8b876b8 . I am working on spark version 1.6.3. I have installed keras and tensorflow. But everytime i try to import from sparkdl it throws an error. I am working on Pyspark. When I run this:-
from sparkdl import readImages
I get this error:-
File "C:\Users\HP\AppData\Local\Temp\spark-802a2258-3089-4ad7-b8cb-
6815cbbb019a\userFiles-c9514201-07fa-45f9-9fd8-
c8a3a0b4bf70\databricks_spark-deep-learning-0.1.0-spark2.1-
s_2.11.jar\sparkdl\transformers\keras_image.py", line 20, in <module>
ImportError: cannot import name 'TypeConverters'
Can someone pls help?
Its not a full fix, as i have yet to be able to import things from sparkdl in jupyter notebooks aswell, but!
readImages is a function in pyspark.ml.image package
so to import it you need to:
from pyspark.ml.image import ImageSchema
to use it:
imagesDF = ImageSchema.readImages("/path/to/imageFolder")
This will give you a dataframe of the images, with column "image"
You can add a label column as such:
labledImageDF = imagesDF.withColumn("label", lit(0))
but remember to import functions from pyspark.sql to use lit function
from pyspark.sql.functions import *
Hope this at least partially helps