I have deployed a lambda function which uses sparknlp, as a docker container. For working with sparknlp I need spark context. So, In my sparknlp code, I start with
sc = pyspark.SparkContext().getOrCreate()
I tested my lambda on local and it worked fine.
On aws I got this error :
java gateway process exited before sending its port number
even though JAVA_HOME was properly set.
I found out in the source code :
https://github.com/apache/spark/blob/master/python/pyspark/java_gateway.py
in the launch_gateway method that it tries to create a temporary file and if that file is not created it raises the above error. (line 105)
Lambda won't allow write access to file system, so the file cannot be created.
So I am trying to pass gateway_port and gateway_secret as environment variables.
I have kept the PYSPARK_GATEWAY_PORT=25333 which is the default value.
I am not able to figure out how to get the PYSPARK_GATEWAY_SECRET.
Which is why getting the error :
KeyError: 'PYSPARK_GATEWAY_SECRET'
I am running into a strange issue when using Google Translate API with a JSON authorization key. I can run the file without any issue directly in my editor of choice (VSCode). No errors.
But when I run the same file via terminal, the file executes up until after loading the Google Translate credentials from disk. The call then throws below error message.
Since this only ever happens when running the file from terminal, I find this bug hard to tackle.
The goal is to then have this file collect data and translate some of the fields using Google services, then store the data in a data base.
Below is the error message:
Error: 503 DNS resolution failed
Traceback (most recent call last):
File "/home/MY-USER-NAME/anaconda3/lib/python3.6/site-packages/google/api_core/grpc_helpers.py", line 57, in error_remapped_callable
File "/home/MY-USER-NAME/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 690, in __call__
File "/home/MY-USER-NAME/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 592, in _end_unary_response_blocking
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
status = StatusCode.UNAVAILABLE
details = "DNS resolution failed"
debug_error_string = "{"created":"#1584629013.091398712","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3934,"referenced_errors":[{"created":"#1584629013.091395769","description":"Resolver transient failure","file":"src/core/ext/filters/client_channel/resolving_lb_policy.cc","file_line":262,"referenced_errors":[{"created":"#1584629013.091394954","description":"DNS resolution failed","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/dns_resolver_ares.cc","file_line":370,"grpc_status":14,"referenced_errors":[{"created":"#1584629013.091389655","description":"C-ares status is not ARES_SUCCESS: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244,"referenced_errors":[{"created":"#1584629013.091380513","description":"C-ares status is not ARES_SUCCESS: Could not contact DNS servers","file":"src/core/ext/filters/client_channel/resolver/dns/c_ares/grpc_ares_wrapper.cc","file_line":244}]}]}]}]}"
>
This is the relevant part of my code:
Dedicated CRON file (I use this two-step setup in many other projects without any issue)
#! anaconda3/bin/python
import os
os.chdir(r'/home/MY-USER-NAME/path-to-project')
import code-file
Code file (simple .py script)
[...]
from google.oauth2 import service_account
from google.cloud import translate
key_path = 'path-to-file/credentials.json'
credentials = service_account.Credentials.from_service_account_file(
key_path, scopes=['https://www.googleapis.com/auth/cloud-platform'])
def translate_to_en(contents, credentials=None, verbose=False, use_pinyin=False):
[...]
client = translate.TranslationServiceClient(credentials=credentials)
parent = client.location_path(credentials.project_id, 'global')
[...]
response = client.translate_text(
parent=parent,
contents=[contents],
mime_type='text/plain', # mime types: text/plain, text/html
target_language_code='en-US')
[...]
[...]
for c in trans_cols:
df[f'{c}__trans'] = df[c].apply(lambda x: translate_to_en(contents=x, credentials=credentials, verbose=False))
[...]
If there is anyone with a good idea to solve this, your help is greatly appreciated.
It seems the issue is relevant to the grpc bug reported in github. For gRPC name resolution, you can follow this doc on github.
Hi I am working on TFS Migration form on-premises to VSTS online and also I am Upgrade the my TFS Server as a new Version.
Here I am Facing the problem is when I am running the validation commend in Command Prompt I am getting the below exception and I am passing the command is like below
TfsMigrator validate /collection:http://desktop-qvs5g7f:8080/tfs/DefaultCollection
when I am running the above command I am getting the below exception
The collection with url was not found: 'http://desktop-qvs5g7f:8080/tfs/DefaultCollection'
and with in the Log folder it creates one log description like in the below
TfsMigrator validate /collection:http://desktop-qvs5g7f:8080/tfs/DefaultCollection
Connection string not specified, trying to use the one from the Application Tier's web.config
Using the framework connection string Data Source=DESKTOP-QVS5G7F\SqlExpress;Initial Catalog=Tfs_Configuration;Integrated Security=True
Exception Message: The collection with url was not found: 'http://desktop-qvs5g7f:8080/tfs/DefaultCollection' (type ArgumentException)
Exception Stack Trace: at TfsMigrator.TfsMigratorCommand.CreateCollectionRequestContext(String collectionUrl)
at TfsMigrator.TfsMigratorCommandValidate.RunValidations(Boolean validateFiles)
at TfsMigrator.TfsMigratorCommandValidate.RunImpl()
at TfsMigrator.TfsMigratorCommand.Run()
For the above exception how to resolve that and for the validation Commands I am following the below URL as Reference is
https://www.visualstudio.com/en-us/articles/migration-import
I am using int-ftp:outbound-gateway sample given in GITHUB.
Its working for me. The problem I am facing is : If the file already exists in my local its not overwriting with the file from FTP. Its throwing the following error and the execution stops :
I want to overwrite the local copy... Any idea on how to handle this scenario ?
WARNING: failure occurred in gateway sendAndReceive
Throwable occurred: org.springframework.integration.MessagingException: Local file \TOPROCESS\UserLastLoginData2015-10-08.csv already exists
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.get(AbstractRemoteFileOutboundGateway.java:677)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway$2.doInSession(AbstractRemoteFileOutboundGateway.java:423)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway$2.doInSession(AbstractRemoteFileOutboundGateway.java:419)
at org.springframework.integration.file.remote.RemoteFileTemplate.execute(RemoteFileTemplate.java:295)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.doGet(AbstractRemoteFileOutboundGateway.java:419)
at org.springframework.integration.file.remote.gateway.AbstractRemoteFileOutboundGateway.handleRequestMessage(AbstractRemoteFileOutboundGateway.java:381)
at org.springframework.integration.handler.AbstractReplyProducingMessageHandler.handleMessageInternal(AbstractReplyProducingMessageHandler.java:142)
at org.springframework.integration.handler.AbstractMessageHandler.handleMessage(AbstractMessageHandler.java:73)
Starting with Spring Integration version 4.1.4, you can set mode="REPLACE" on the outbound gateway.
I'm getting the following error when attempting to run DotNetNuke 7.1 from IIS.
Object reference not set to an instance of an object.
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.NullReferenceException: Object reference not set to an instance of an object.
Source Error:
Line 572: //first call GetProviderPath - this insures that the Database is Initialised correctly
Line 573: //and also generates the appropriate error message if it cannot be initialised correctly
Line 574: string strMessage = DataProvider.Instance().GetProviderPath();
Line 575: //get current database version from DB
Line 576: if (!strMessage.StartsWith("ERROR:"))
I've tried running it from Visual Studio 2012 after downloading and extracting the source code to a folder, then running, but I get the same error (also, VS loads about 13 instances of it's built in webserver which can't be correct).
Clearly, there is something wrong with the database. From what I've read in the past, there should have been a start up configuration page (for configuring settings the first time you run the project).
I did look at the local version of IIS (running on Windows 8) and it created the site fine there, however, for some reason the internal webserver attempts to run (and the option to run on an external IIS is greyed out).
Anyone run into this problem with DNN Community edition? I've tried running as admin and setting permissions with no luck at all.
Any way to fix this?
Ok, the key is to delete the Database.mdf file completely.
Then create a new empty database of your choice in SQL Server (2008 or greater).
Create a new user account with db_owner access (as it must be able to create tables, etc).
Change the connection strings in the release.config and development.config to connect to the database.
DELETE the web.config file.
RENAME either config file to "web.config"
Set the default project to the web project in VS
set the default page to default.aspx
Run
I made the erroneous assumption that running the app would rename the config file for me (not sure why I assumed that).
SOLVED!