MS Azure Speech-to-text Python start_continuous_recognition does not stop at end of stream - speech-to-text

I'm using the MS Azure Speech-to-text service with Python.
My data input is a byte string, only a few seconds of audio. My expectation would be that the cloud service stops to process the audio when the end of the stream is finished and returns the recognized text. Instead it takes about 5 minutes until the recognized event is triggered.
speech_config = speechsdk.SpeechConfig(subscription=API_KEY,
region="westeurope",
speech_recognition_language='de-DE')
stream = PushAudioInputStream(stream_format=
AudioStreamFormat(samples_per_second=sample_rate, bits_per_sample=SAMPLE_WIDTH * 8,
compressed_stream_format=speechsdk.AudioStreamContainerFormat.FLAC))
audio_input = speechsdk.AudioConfig(stream=stream)
stream.write(data)
speech_recognizer = speechsdk.SpeechRecognizer(speech_config=speech_config, audio_config=audio_input)
speech_recognizer.start_continuous_recognition()
done = False
def stop_recognition(evt):
logger.debug("Stopped MS Azure recognition: %s", evt)
nonlocal done
done = True
def recognized(evt):
logger.info("Recognized MS Azure transcript: %s", evt)
nonlocal text
text += " " + evt.result.text
speech_recognizer.recognizing.connect(lambda evt: print('RECOGNIZING: {}'.format(evt)))
speech_recognizer.recognized.connect(lambda evt: print('RECOGNIZED: {}'.format(evt)))
speech_recognizer.session_started.connect(lambda evt: print('SESSION STARTED: {}'.format(evt)))
speech_recognizer.session_stopped.connect(lambda evt: print('SESSION STOPPED {}'.format(evt)))
speech_recognizer.canceled.connect(lambda evt: print('CANCELED {}'.format(evt)))
speech_recognizer.recognized.connect(recognized)
speech_recognizer.session_stopped.connect(stop_recognition)
speech_recognizer.canceled.connect(stop_recognition)
while not done:
time.sleep(.5)
speech_recognizer.stop_continuous_recognition()
Instead I see a delay of 5 minutes:
2022-11-13 23:58:19,504 - speech_processing.speech_recognition.speech_recognition - DEBUG - Sending 192000 bytes (6 sec) for recognition
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=50e5c478cdc34e0a8ced3867be493bc3, text="telefon", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=d1448833ac8f40ef9c1ebc4cae488bcd, text="telefonspeicher", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=cdf9f074c13b4a2c94960ec147db765c, text="telefon speichere", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=548133156bb44dc8ae08fd0848fa8ec5, text="telefon speichere als", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=c03970619f1e42278b2a2ef19ee4f1fe, text="telefon speichere als bärbel", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=ff5f6a18d1e4409cab2661582cb8a693, text="telefon speichere als bärbel 0", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=a3cb8f82c62b4235abc2fea2696342f8, text="telefon speichere als bärbel 03", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=cafc805031654aa4865a4fe1b742d1cd, text="telefon speichere als bärbel 038", reason=ResultReason.RecognizingSpeech))
RECOGNIZING: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=69782c9485244e3191846b924adb3807, text="telefon speichere als bärbel 0385", reason=ResultReason.RecognizingSpeech))
RECOGNIZED: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=9d92890d52d84b7f926a6977d6324ca1, text="Telefon speichere als Bärbel 0385.", reason=ResultReason.RecognizedSpeech))
2022-11-14 00:03:26,487 - speech_processing.speech_recognition.speech_recognition - INFO - Recognized MS Azure transcript: SpeechRecognitionEventArgs(session_id=2e4c92f4fed6498f8f5260199bdcc5d7, result=SpeechRecognitionResult(result_id=9d92890d52d84b7f926a6977d6324ca1, text="Telefon speichere als Bärbel 0385.", reason=ResultReason.RecognizedSpeech))


I found my error:
The stream must be closed:
stream.write(data)
stream.close()

Related

"No module named torch" on Heroku

I installed torch with this to Heroku:
https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-linux_x86_64.whl
But in Heroku, it says "No module named 'torch'.
My requirements.txt:
autopep8==1.4.3
certifi==2020.6.20
cffi==1.14.0
click==7.1.2
colorama==0.4.1
cryptography==2.9.2
Flask==1.1.2
Flask-Cors==3.0.8
Flask-Limiter==1.3.1
idna==2.10
isort==4.3.4
itsdangerous==1.1.0
Jinja2==2.11.2
lazy-object-proxy==1.3.1
limits==1.5.1
MarkupSafe==1.1.1
mccabe==0.6.1
numpy==1.18.5
pandas==1.1.0
pycodestyle==2.4.0
pycparser==2.20
PyJWT==1.7.1
python-dateutil==2.8.1
pytz==2020.1
six==1.15.0
https://download.pytorch.org/whl/cpu/torch-1.3.1%2Bcpu-cp36-cp36m-linux_x86_64.whl
Werkzeug==1.0.1
wincertstore==0.2

Spring Integration TCP Inbound Adapter client disconnect issue

We have created an IntegrationFlow for an inbound TCP server. Whenever we test this the integration is fine until the client disconnects, then we get a continuous flow of empty messages.
Spring Integration 5.2.5.RELEASE
Flow:
#Bean
public IntegrationFlow inboundFlow(MyService myService) {
return IntegrationFlows.from(
Tcp.inboundAdapter(
Tcp.netServer(12345)
.deserializer(TcpCodecs.lengthHeader4())))
.transform(Transformers.objectToString())
.log(INFO) // Logging only for testing
.handle(myService, "process") // Do some processing of the message
.get();
}
Test method:
public void sendPackets() throws Exception {
List<Path> files = getAllFilesSorted("/some/location");
try (Socket socket = new Socket("localhost", 12345)) {
log.info("local port: {}", socket.getLocalPort());
try (OutputStream outputStream = socket.getOutputStream()) {
for (int i = 0; i < 10; i++) {
Path path = files.get(i);
log.info("{} ({} of {})", path.toString(), i, files.size());
byte[] bytes = Files.readAllBytes(path);
outputStream.write(bytes);
outputStream.flush();
Thread.sleep(200);
}
}
}
}
Partial Log after test completes:
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=813ccf6b-9eb7-2ff3-0f8d-f2810966d8d7, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=1fd5c4c7-b2e0-6a89-7de6-894c8e42e242, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=78b852e8-0094-f8b9-8d05-37bc4912d096, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=4e4a7ee0-c66e-9501-9ef7-4d25143531b3, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=c90abe17-7036-94cc-af59-3b0cac5e75e4, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.963 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=425d2e2a-75c5-0184-06da-6d27eb546931, ip_hostname=localhost, timestamp=1588672852963}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=4c55d1d0-7421-1832-23ca-03744419c847, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=05b6fea6-8f76-5c74-e4cc-33bd8099d8e8, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=9b8fe16c-07f4-b64b-faec-794eb743559c, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=222725e8-19c6-62b6-8ba7-103892191c96, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=951403b6-18a9-ffe9-29b7-67b8c01ea311, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=2ac3002b-ccd1-9506-f3aa-60f4b63e26b8, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=dbbf12cf-0a27-407b-0a96-5b27ab0539db, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.965 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=3282d09e-9508-b053-3763-cc23d3c817a8, ip_hostname=localhost, timestamp=1588672852965}]
2020-05-05 11:00:52.966 INFO 31793 --- [pool-1-thread-2] o.s.integration.handler.LoggingHandler : GenericMessage [payload=, headers={ip_tcp_remotePort=59407, ip_connectionId=localhost:59407:44745:df23388b-b8b0-4250-b926-4199030b2ff6, ip_localInetAddress=/127.0.0.1, ip_address=127.0.0.1, id=549b2326-341f-19c0-db98-6b31da1901d8, ip_hostname=localhost, timestamp=1588672852966}]

There is nothing in the log about the client disconnecting. We would see socket close activity in the log.
The log implies we are receiving packets containing 0x00000000.
Use wireshark or similar to look at the activity on the wire.
EDIT
Your custom deserializer is causing the problem.
if (read < 0) {
return 0;
}
You need to throw a SoftEndOfStreamException when you detect a socket close.
This is explained in the documentation.
When the deserializer detects a closed input stream between messages, it must throw a SoftEndOfStreamException; this is a signal to the framework to indicate that the close was "normal". If the stream is closed while decoding a message, some other exception should be thrown instead.

jhipster 8080 blank page on windows

I am generating a jhipster project with yoeman on the windows terminal. Everytime i try to generate a new project, i get a blank page on localhost:8080 when i run mvnw and yarn start.
Github link : https://github.com/Fosio/JALPlanning.git
Here is the build i am using :
{
"generator-jhipster": {
"promptValues": {
"packageName": "com.jalarue.jalplanning",
"nativeLanguage": "fr"
},
"jhipsterVersion": "4.13.0",
"baseName": "JALPlanning",
"packageName": "com.jalarue.jalplanning",
"packageFolder": "com/jalarue/jalplanning",
"serverPort": "8080",
"authenticationType": "jwt",
"hibernateCache": "ehcache",
"clusteredHttpSession": false,
"websocket": false,
"databaseType": "sql",
"devDatabaseType": "h2Disk",
"prodDatabaseType": "mssql",
"searchEngine": "elasticsearch",
"messageBroker": false,
"serviceDiscoveryType": false,
"buildTool": "maven",
"enableSocialSignIn": false,
"enableSwaggerCodegen": false,
"jwtSecretKey": "73f9ef15682487a68491ce79f5c3352d0aa0aaab",
"clientFramework": "angularX",
"useSass": false,
"clientPackageManager": "yarn",
"applicationType": "monolith",
"testFrameworks": [
"gatling",
"cucumber",
"protractor"
],
"jhiPrefix": "jhi",
"enableTranslation": true,
"nativeLanguage": "fr",
"languages": [
"fr",
"en"
]
}
}
I have the following version of the requirement installed
node v9.3.0
yarn v1.3.2
Java 8
Here is the yarn start
Microsoft Windows [version 10.0.15063]
(c) 2017 Microsoft Corporation. Tous droits réservés.
C:\Users\Simon Frenette\Documents\GitHub\JALPlanning>yarn start
yarn run v1.3.2
$ yarn run webpack:dev
$ yarn run webpack-dev-server -- --config webpack/webpack.dev.js --progress --inline --hot --profile --port=9060 --watch-content-base
warning From Yarn 1.0 onwards, scripts don't require "--" for options to be forwarded. In a future version, any explicit "--" will be forwarded as-is to the scripts.
$ node --max_old_space_size=4096 node_modules/webpack-dev-server/bin/webpack-dev-server.js --config webpack/webpack.dev.js --progress --inline --hot --profile --port=9060 --watch-content-base
10% building modules 3/3 modules 0 active[HPM] Proxy created: [ '/api',
'/management',
'/swagger-resources',
'/v2/api-docs',
'/h2-console',
'/auth' ] -> http://127.0.0.1:8080
Project is running at http://localhost:9060/
webpack output is served from /
Content not from webpack is served from ./target/www 11% building modules 11/17 modules 6 acti
ve ...modules\style-loader\lib\addStyles.jsWarning: The 'no-unused-variable' rule requires type infomation.
[at-loader] Using typescript#2.5.3 from typescript and "tsconfig.json" from C:\Users\Simon Frenette\Documents\GitHub\JALPlanning/tsconfig.json.
11558ms building modules
31ms sealing
0ms optimizing
1ms basic module optimization
36ms module optimization
1ms advanced module optimization
28ms basic chunk optimization
0ms chunk optimization
0ms advanced chunk optimization
5ms module and chunk tree optimization
1ms chunk modules optimization
0ms advanced chunk modules optimization
73ms module reviving
11ms module order optimization
24ms module id optimization
5ms chunk reviving
1ms chunk order optimization
24ms chunk id optimization
37ms hashing
1ms module assets processing
3756ms chunk assets processing
204ms additional chunk assets processing
1ms recording
1ms additional asset processing
0ms chunk asset optimization
94% asset optimization
[at-loader] Checking started in a separate process...
[at-loader] Ok, 0.176 sec.
MergetJsonsWebpackPlugin compilation started...
MergetJsonsWebpackPlugin compilation completed... 299ms asset optimizatio
n
95% emittingMergetJsonsWebpackPlugin emit starts...
MergetJsonsWebpackPlugin emit completed... 304ms emittin
g
MergetJsonsWebpackPlugin emit starts...
MergetJsonsWebpackPlugin emit completed...
Hash: 9f78c5679d4cae109731
Version: webpack 3.10.0
Time: 16199ms
43 assets
[./node_modules/css-loader/index.js!./src/main/webapp/content/css/global.css] ./node_modules/css-loader!./src/main/webapp/content/css/global.css 5.09 kB {3} [built]
[] -> factory:269ms building:3900ms = 4169ms
[./node_modules/webpack-dev-server/client/index.js?http://localhost:9060] (webpack)-dev-server/client?http://localhost:9060 7.95 kB {1} [built]
[] -> factory:40ms building:29ms dependencies:497ms = 566ms
[./node_modules/webpack-dev-server/client/overlay.js] (webpack)-dev-server/client/overlay.js 3.73 kB {1} [built]
[] -> factory:24ms building:774ms = 798ms
[./node_modules/webpack-dev-server/client/socket.js] (webpack)-dev-server/client/socket.js 1.05 kB {1} [built]
[] -> factory:24ms building:769ms = 793ms
[./node_modules/webpack/hot ^\.\/log$] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {1} [built]
[] -> factory:14ms building:9ms dependencies:848ms = 871ms
[./node_modules/webpack/hot/dev-server.js] (webpack)/hot/dev-server.js 1.61 kB {1} [built]
[] -> factory:74ms building:14ms = 88ms
[./node_modules/webpack/hot/emitter.js] (webpack)/hot/emitter.js 77 bytes {1} [built]
[] -> factory:3ms building:769ms = 772ms
[./node_modules/webpack/hot/log-apply-result.js] (webpack)/hot/log-apply-result.js 1.31 kB {1} [built]
[] -> factory:853ms building:3534ms dependencies:0ms = 4387ms
[./src/main/webapp/app/app.main.ts] ./src/main/webapp/app/app.main.ts 590 bytes {2} [built]
[] -> factory:562ms building:3865ms = 4427ms
[./src/main/webapp/app/polyfills.ts] ./src/main/webapp/app/polyfills.ts 2.63 kB {1} [built]
[] -> factory:302ms building:4140ms = 4442ms
[./src/main/webapp/content/css/global.css] ./src/main/webapp/content/css/global.css 1.05 kB {3} [built]
[] -> factory:284ms building:15ms = 299ms
[0] multi (webpack)-dev-server/client?http://localhost:9060 webpack/hot/dev-server ./src/main/webapp/app/polyfills 52 bytes {1} [built]
factory:0ms building:2ms = 2ms
[1] multi (webpack)-dev-server/client?http://localhost:9060 webpack/hot/dev-server ./src/main/webapp/content/css/global.css 52 bytes {3} [built]
factory:0ms building:1ms dependencies:72ms = 73ms
[2] multi (webpack)-dev-server/client?http://localhost:9060 webpack/hot/dev-server ./src/main/webapp/app/app.main 52 bytes {2} [built]
factory:0ms building:0ms dependencies:72ms = 72ms
[./src/main/webapp/manifest.webapp] ./src/main/webapp/manifest.webapp 61 bytes {1} [built]
[] -> factory:383ms dependencies:2ms building:753ms = 1138ms
+ 1042 hidden modules
Child html-webpack-plugin for "index.html":
1 asset
[./node_modules/html-webpack-plugin/lib/loader.js!./src/main/webapp/index.html] ./node_modules/html-webpack-plugin/lib/loader.js!./src/main/webapp/index.html 883 bytes {0} [built]
factory:101ms building:854ms = 955ms
webpack: Compiled successfully.
Hash: 9f78c5679d4cae109731
Version: webpack 3.10.0
Time: 16588ms
43 assets
[./node_modules/css-loader/index.js!./src/main/webapp/content/css/global.css] ./node_modules/css-loader!./src/main/webapp/content/css/global.css 5.09 kB {3} [built]
[] -> factory:269ms building:3900ms = 4169ms
[./node_modules/webpack-dev-server/client/index.js?http://localhost:9060] (webpack)-dev-server/client?http://localhost:9060 7.95 kB {1} [built]
[] -> factory:40ms building:29ms dependencies:497ms = 566ms
[./node_modules/webpack-dev-server/client/overlay.js] (webpack)-dev-server/client/overlay.js 3.73 kB {1} [built]
[] -> factory:24ms building:774ms = 798ms
[./node_modules/webpack-dev-server/client/socket.js] (webpack)-dev-server/client/socket.js 1.05 kB {1} [built]
[] -> factory:24ms building:769ms = 793ms
[./node_modules/webpack/hot ^\.\/log$] (webpack)/hot nonrecursive ^\.\/log$ 170 bytes {1} [built]
[] -> factory:14ms building:9ms dependencies:848ms = 871ms
[./node_modules/webpack/hot/dev-server.js] (webpack)/hot/dev-server.js 1.61 kB {1} [built]
[] -> factory:74ms building:14ms = 88ms
[./node_modules/webpack/hot/emitter.js] (webpack)/hot/emitter.js 77 bytes {1} [built]
[] -> factory:3ms building:769ms = 772ms
[./node_modules/webpack/hot/log-apply-result.js] (webpack)/hot/log-apply-result.js 1.31 kB {1} [built]
[] -> factory:853ms building:3534ms dependencies:0ms = 4387ms
[./src/main/webapp/app/app.main.ts] ./src/main/webapp/app/app.main.ts 590 bytes {2} [built]
[] -> factory:562ms building:3865ms = 4427ms
[./src/main/webapp/app/polyfills.ts] ./src/main/webapp/app/polyfills.ts 2.63 kB {1} [built]
[] -> factory:302ms building:4140ms = 4442ms
[./src/main/webapp/content/css/global.css] ./src/main/webapp/content/css/global.css 1.05 kB {3} [built]
[] -> factory:284ms building:15ms = 299ms
[0] multi (webpack)-dev-server/client?http://localhost:9060 webpack/hot/dev-server ./src/main/webapp/app/polyfills 52 bytes {1} [built]
factory:0ms building:2ms = 2ms
[1] multi (webpack)-dev-server/client?http://localhost:9060 webpack/hot/dev-server ./src/main/webapp/content/css/global.css 52 bytes {3} [built]
factory:0ms building:1ms dependencies:72ms = 73ms
[2] multi (webpack)-dev-server/client?http://localhost:9060 webpack/hot/dev-server ./src/main/webapp/app/app.main 52 bytes {2} [built]
factory:0ms building:0ms dependencies:72ms = 72ms
[./src/main/webapp/manifest.webapp] ./src/main/webapp/manifest.webapp 61 bytes {1} [built]
[] -> factory:383ms dependencies:2ms building:753ms = 1138ms
+ 1042 hidden modules
Child html-webpack-plugin for "index.html":
1 asset
[./node_modules/html-webpack-plugin/lib/loader.js!./src/main/webapp/index.html] ./node_modules/html-webpack-plugin/lib/loader.js!./src/main/webapp/index.html 883 bytes {0} [built]
factory:101ms building:854ms = 955ms
webpack: Compiled successfully.
[Browsersync] Proxying: http://localhost:9060
[Browsersync] Access URLs:
--------------------------------------
Local: http://localhost:9000
External: http://192.168.0.111:9000
--------------------------------------
UI: http://localhost:3001
UI External: http://192.168.0.111:3001
--------------------------------------
Here is the mvnw
C:\Users\Simon Frenette\Documents\GitHub\JALPlanning>mvn
[INFO] Scanning for projects...
[INFO]
[INFO] ------------------------------------------------------------------------
[INFO] Building JAL Planning 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO]
[INFO] >>> spring-boot-maven-plugin:1.5.9.RELEASE:run (default-cli) > test-compile # jal-planning >>>
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (default-resources) # jal-planning ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 17 resources
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:resources (default-resources) # jal-planning ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 17 resources
[INFO]
[INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (enforce-versions) # jal-planning ---
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:copy-resources (docker-resources) # jal-planning ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO]
[INFO] --- jacoco-maven-plugin:0.7.9:prepare-agent (pre-unit-tests) # jal-planning ---
[INFO] argLine set to "-javaagent:C:\\Users\\Simon Frenette\\.m2\\repository\\org\\jacoco\\org.jacoco.agent\\0.7.9\\org.jacoco.agent-0.7.9-runtime.jar=destfile=C:\\Users\\Simon Frenette\\Documents\\GitHub\\JALPlanning\\target\\test-results\\coverage\\jacoco\\ja
coco.exec" -Djava.security.egd=file:/dev/./urandom -Xmx256m
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:compile (default-compile) # jal-planning ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 78 source files to C:\Users\Simon Frenette\Documents\GitHub\JALPlanning\target\classes
[INFO]
[INFO] --- maven-resources-plugin:3.0.1:testResources (default-testResources) # jal-planning ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 4 resources
[INFO] Copying 2 resources
[INFO]
[INFO] --- maven-compiler-plugin:3.7.0:testCompile (default-testCompile) # jal-planning ---
[INFO] Nothing to compile - all classes are up to date
[INFO]
[INFO] <<< spring-boot-maven-plugin:1.5.9.RELEASE:run (default-cli) < test-compile # jal-planning <<<
[INFO]
[INFO] --- spring-boot-maven-plugin:1.5.9.RELEASE:run (default-cli) # jal-planning ---
[INFO] Attaching agents: []
The Class-Path manifest attribute in C:\Users\Simon Frenette\.m2\repository\org\liquibase\liquibase-core\3.5.3\liquibase-core-3.5.3.jar referenced one or more files that do not exist: file:/C:/Users/Simon%20Frenette/.m2/repository/org/liquibase/liquibase-core/3
.5.3/lib/snakeyaml-1.13.jar
11:33:06.583 [main] DEBUG org.springframework.boot.devtools.settings.DevToolsSettings - Included patterns for restart : []
11:33:06.585 [main] DEBUG org.springframework.boot.devtools.settings.DevToolsSettings - Excluded patterns for restart : [/spring-boot-starter/target/classes/, /spring-boot-autoconfigure/target/classes/, /spring-boot-starter-[\w-]+/, /spring-boot/target/classes/
, /spring-boot-actuator/target/classes/, /spring-boot-devtools/target/classes/]
11:33:06.585 [main] DEBUG org.springframework.boot.devtools.restart.ChangeableUrls - Matching URLs for reloading : [file:/C:/Users/Simon%20Frenette/Documents/GitHub/JALPlanning/target/classes/]
██╗ ██╗ ██╗ ████████╗ ███████╗ ██████╗ ████████╗ ████████╗ ███████╗
██║ ██║ ██║ ╚══██╔══╝ ██╔═══██╗ ██╔════╝ ╚══██╔══╝ ██╔═════╝ ██╔═══██╗
██║ ████████║ ██║ ███████╔╝ ╚█████╗ ██║ ██████╗ ███████╔╝
██╗ ██║ ██╔═══██║ ██║ ██╔════╝ ╚═══██╗ ██║ ██╔═══╝ ██╔══██║
╚██████╔╝ ██║ ██║ ████████╗ ██║ ██████╔╝ ██║ ████████╗ ██║ ╚██╗
╚═════╝ ╚═╝ ╚═╝ ╚═══════╝ ╚═╝ ╚═════╝ ╚═╝ ╚═══════╝ ╚═╝ ╚═╝
:: JHipster ? :: Running Spring Boot 1.5.9.RELEASE ::
:: http://www.jhipster.tech ::
2017-12-17 11:33:07.438 INFO 6324 --- [ restartedMain] com.jalarue.jalplanning.JalPlanningApp : Starting JalPlanningApp on SimonFrenette with PID 6324 (started by Simon Frenette in C:\Users\Simon Frenette\Documents\GitHub\JALPlanning)
2017-12-17 11:33:07.439 DEBUG 6324 --- [ restartedMain] com.jalarue.jalplanning.JalPlanningApp : Running with Spring Boot v1.5.9.RELEASE, Spring v4.3.13.RELEASE
2017-12-17 11:33:07.439 INFO 6324 --- [ restartedMain] com.jalarue.jalplanning.JalPlanningApp : The following profiles are active: swagger,dev
2017-12-17 11:33:07.663 DEBUG 6324 --- [kground-preinit] org.jboss.logging : Logging Provider: org.jboss.logging.Slf4jLoggerProvider found via system property
2017-12-17 11:33:09.670 DEBUG 6324 --- [ restartedMain] c.j.j.config.AsyncConfiguration : Creating Async Task Executor
2017-12-17 11:33:10.365 DEBUG 6324 --- [ restartedMain] class org.ehcache.core.Ehcache-users : Initialize successful.
2017-12-17 11:33:10.374 DEBUG 6324 --- [ restartedMain] c.e.c.E.jalarue.jalplanning.domain.User : Initialize successful.
2017-12-17 11:33:10.377 DEBUG 6324 --- [ restartedMain] c.e.c.E.j.jalplanning.domain.Authority : Initialize successful.
2017-12-17 11:33:10.383 DEBUG 6324 --- [ restartedMain] c.e.c.E.j.j.domain.User.authorities : Initialize successful.
2017-12-17 11:33:10.612 DEBUG 6324 --- [ restartedMain] c.j.j.config.MetricsConfiguration : Registering JVM gauges
2017-12-17 11:33:10.627 DEBUG 6324 --- [ restartedMain] c.j.j.config.MetricsConfiguration : Monitoring the datasource
2017-12-17 11:33:10.627 DEBUG 6324 --- [ restartedMain] c.j.j.config.MetricsConfiguration : Initializing Metrics JMX reporting
2017-12-17 11:33:11.349 DEBUG 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Registering CORS filter
2017-12-17 11:33:11.474 INFO 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Web application configuration, using profiles: swagger
2017-12-17 11:33:11.475 DEBUG 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Initializing Metrics registries
2017-12-17 11:33:11.479 DEBUG 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Registering Metrics Filter
2017-12-17 11:33:11.480 DEBUG 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Registering Metrics Servlet
2017-12-17 11:33:11.483 DEBUG 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Initialize H2 console
2017-12-17 11:33:11.484 INFO 6324 --- [ restartedMain] c.j.jalplanning.config.WebConfigurer : Web application fully configured
2017-12-17 11:33:11.746 DEBUG 6324 --- [ restartedMain] c.j.j.config.DatabaseConfiguration : Configuring Liquibase
2017-12-17 11:33:11.761 WARN 6324 --- [ning-Executor-1] i.g.j.c.liquibase.AsyncSpringLiquibase : Starting Liquibase asynchronously, your database might not be ready at startup!
2017-12-17 11:33:12.772 DEBUG 6324 --- [ning-Executor-1] i.g.j.c.liquibase.AsyncSpringLiquibase : Liquibase has updated your database in 1010 ms
2017-12-17 11:33:19.682 DEBUG 6324 --- [ restartedMain] i.g.j.c.apidoc.SwaggerConfiguration : Starting Swagger
2017-12-17 11:33:19.689 DEBUG 6324 --- [ restartedMain] i.g.j.c.apidoc.SwaggerConfiguration : Started Swagger in 6 ms
2017-12-17 11:33:20.763 INFO 6324 --- [ restartedMain] com.jalarue.jalplanning.JalPlanningApp : Started JalPlanningApp in 14.159 seconds (JVM running for 14.612)
2017-12-17 11:33:20.764 INFO 6324 --- [ restartedMain] com.jalarue.jalplanning.JalPlanningApp :
----------------------------------------------------------
Application 'JALPlanning' is running! Access URLs:
Local: http://localhost:8080
External: http://192.168.0.111:8080
Profile(s): [swagger, dev]
----------------------------------------------------------
Any help would be appreciated
Regards
Simon

Logstash multiline exim logs

I need a help with task for parsing and combining exim_mainlog for ELK.
Issue is next:
My logstash multiline plugin won't collect lines of log file with unique message id into one event.
When I try to send 4 strings in right order it works good.
Order like this:
2017-04-10 00:00:30 1cxKsn-0001GB-2t CTAS=IN RefID= ( ISpam= IFlags=v=2.2 cv=Op4/823t c=1 sm=1 tr=0 a=6HVp5djceeYjte4jJb6Ryw==:17 a=AzvcPWV-tVgA:10 a=uHJYF-HtSykr7tHsIToA:9 a=CTTii-5M3Z-LMe4tr8cA:9 a=QEXdDO2ut3YA:10 a=pyshpDcKeHPZtuIe0Z8A:9 )
2017-04-10 00:00:30 1cxKsn-0001GB-2t <= email#domain.com H=m37s3-2-28db.ispgateway.com [176.221.47.15] P=smtp S=2567 id=201704092200.v39M0Qxr016654#m37s3-2-28db.ispgateway.com
2017-04-10 00:00:30 1cxKsn-0001GB-2t => info#domainx.com R=internal_gw T=remote_smtp H=192.168.1.11 [192.168.1.11] C="250 OK id=1cxKso-0002iK-Q7"
2017-04-10 00:00:30 1cxKsn-0001GB-2t Completed
If otder is right - everything works good.
But when between lines of the same event there are inserted other trash info it breaks down.
Actual logs are look like this:
2017-04-10 00:00:30 1cxKsn-0001GB-2t CTAS=IN RefID= ( ISpam= IFlags=v=2.2 cv=Op4/823t c=1 sm=1 tr=0 a=6HVp5djceeYjte4jJb6Ryw==:17 a=AzvcPWV-tVgA:10 a=uHJYF-HtSykr7tHsIToA:9 a=CTTii-5M3Z-LMe4tr8cA:9 a=QEXdDO2ut3YA:10 a=pyshpDcKeHPZtuIe0Z8A:9 )
2017-04-10 00:00:30 1cxKsn-0001GB-2t <= email#domain.com H=m37s3-2-28db.ispgateway.com [176.221.47.15] P=smtp S=2567 id=201704092200.v39M0Qxr016654#m37s3-2-28db.ispgateway.com
2017-04-10 00:00:30 1cxKsn-0001GB-2t => info#domainx.com R=internal_gw T=remote_smtp H=192.168.1.11 [192.168.1.11] C="250 OK id=1cxKso-0002iK-Q7"
2017-04-10 00:00:30 1cxKsn-0001GB-2t Completed
2017-04-10 00:00:30 fixed_login authenticator failed for (faYNpaLtF) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (lkLmh6Lk) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (dLKdHZ) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 H=mx4.rissoidupgrades.com [79.137.110.132] F=<rtcjrc-cmok892#rissoidupgrades.com> rejected RCPT <qfuohabte_p145#verim.de>: ICIR16 - unknown user
2017-04-10 00:00:30 unexpected disconnection while reading SMTP command from ([111.111.111.111]) [117.241.112.188] (error: Connection reset by peer)
2017-04-10 00:00:30 1cxKso-0001GQ-1R CTAS=IN RefID= ( ISpam=Confirmed IFlags=v=2.2 cv=Op4/823t c=1 sm=1 tr=0 a=LMNu0MzFDzFZvX0DaJwgIA==:17 a=AwJkFeBFn10A:10 a=AzvcPWV-tVgA:10 a=HFQ-CQzmNWWYERzML24A:9 )
2017-04-10 00:00:31 1cxKso-0001GQ-1R <= kd123456#abcdrfg.managed.com H=abcdrfg.managed.com [62.138.219.130] P=esmtp S=671 id=20170409220030.5BCED80909#ma60655.psmanaged.com
2017-04-10 00:00:30 fixed_login authenticator failed for (faYNpaLtF) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (lkLmh6Lk) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 fixed_login authenticator failed for (dLKdHZ) [192.168.24.24]: 535 Incorrect authentication data
2017-04-10 00:00:30 H=mx4.rissoidupgrades.com [79.137.110.132] F=<sdfsdg-sdfsd34#downgrades.com> rejected RCPT <sdfsdf_dsf343#varum.com>: ICIR16 - unknown user
2017-04-10 00:00:30 unexpected disconnection while reading SMTP command from ([117.241.112.188]) [117.241.112.188] (error: Connection reset by peer)
2017-04-10 00:00:31 1cxKso-0001GQ-1R => sarah#tele.com R=internal_gw T=remote_smtp H=192.168.1.11 [192.168.1.11] C="250 OK id=1cxKsp-0002iR-QJ"
2017-04-10 00:00:31 1cxKso-0001GQ-1R Completed
At the end I want to have two events with id 1cxKsn-0001GB-2t and 1cxKso-0001GQ-1R at my Kibana.
Here is my patterns:
EXIM_MSGID [0-9A-Za-z]{6}-[0-9A-Za-z]{6}-[0-9A-Za-z]{2}
EXIM_FLAGS (<=|[-=>*]>|[*]{2}|==)
EXIM_DATE %{YEAR:exim_year}-%{MONTHNUM:exim_month}-%{MONTHDAY:exim_day} %{TIME:exim_time}
EXIM_DATE_EMPTY %{YEAR}-%{MONTHNUM}-%{MONTHDAY} %{TIME}
EXIM_PID \[%{POSINT}\]
EXIM_QT ((\d+y)?(\d+w)?(\d+d)?(\d+h)?(\d+m)?(\d+s)?)
EXIM_EXCLUDE_TERMS (Message is frozen|(Start|End) queue run| Warning: | retry time not reached | no (IP address|host name) found for (IP address|host) | unexpected disconnection while reading SMTP command | no immediate delivery: |another process is handling this message)
EXIM_REMOTE_HOST (H=(%{NOTSPACE:remote_hostname} )?(\(%{NOTSPACE:remote_heloname}\) )?\[%{IP:remote_host}\])
EXIM_INTERFACE (I=\[%{IP:exim_interface}\](:%{NUMBER:exim_interface_port}))
EXIM_PROTOCOL (P=%{NOTSPACE:protocol})
EXIM_MSG_SIZE (S=%{NUMBER:exim_msg_size})
EXIM_HEADER_ID (id=%{NOTSPACE:exim_header_id})
EXIM_SUBJECT (T=%{QS:exim_subject})
NUM_EMAIL (%{HOSTNAME}\#%{HOSTNAME})
EXIM_RECEIVER (=>\s*%{EMAILADDRESS:receiver}(\s*<%{EMAILADDRESS:envelope_sndr}>)?|=>\s*%{NUM_EMAIL:receiver}(\s*<%{EMAILADDRESS:envelope_sndr}>)?)
EXIM_ROUTER (R=%{WORD:router})
EXIM_TRANSPORT (T=%{WORD:transport})
EXIM_REMOTE_SMTP_CONFIRM (C="%{GREEDYDATA:smtp_remote_response}")
EXIM_SPAM %{EXIM_DATE_EMPTY} %{EXIM_MSGID} CTAS=%{WORD:exim_spam_dest} RefID=(%{WORD:exim_refid})? \( (I|O)Spam=(%{WORD:exim_spam})? ((I|O)Virus=%{WORD:exim_virus} )?(I|O)Flags=(%{GREEDYDATA:exim_spam_flags})? cv=%{GREEDYDATA:exim_spam_other} \)
EXIM_LEFT %{EXIM_DATE_EMPTY} %{EXIM_MSGID} %{EXIM_FLAGS:exim_flags} %{GREEDYDATA:exim_email} (%{EXIM_REMOTE_HOST})? %{EXIM_PROTOCOL} (?:X=%{GREEDYDATA:exim_auth_details})?(?:A=%{GREEDYDATA:exim_authenticator})?(?:%{EXIM_MSG_SIZE:exim_mes_size})? (?:id=%{NUM_EMAIL:exim_uid})?
EXIM_RIGHT %{EXIM_DATE_EMPTY} %{EXIM_MSGID} %{EXIM_RECEIVER} %{EXIM_ROUTER} %{EXIM_TRANSPORT} %{EXIM_REMOTE_HOST} %{EXIM_REMOTE_SMTP_CONFIRM}
EXIM_SPAM_CHECK_ST %{EXIM_DATE} %{EXIM_MSGID:exim_msgid} Completed
Here is my filter.conf:
filter {
if [type] == "exim" {
multiline {
patterns_dir => "/etc/logstash/patterns.d"
pattern => "%{EXIM_DATE} %{EXIM_MSGID:msgid}"
what => "previous"
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
break_on_match => false
match => [ "message", "^%{EXIM_SPAM}" ]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
break_on_match => false
match => [ "message", "^%{EXIM_LEFT}" ]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
break_on_match => false
match => [ "message", "^%{EXIM_RIGHT}" ]
}
grok {
patterns_dir => "/etc/logstash/patterns.d"
break_on_match => false
match => [ "message", "^%{EXIM_SPAM_CHECK_ST}" ]
}
}
}

For cumulative collecting information about one event in multiline log you have some ways:
using "Aggregate filter plugin"
using "ElasticSearch filter plugin"
In first you need to collect all event info into one MessageId.
But this can be fraught with a lot of problems - some log lines havn't MessageId, andalso many Exim workers write own lines in mixed order.
In the second, and if you are using ElastisSearch to store event info, you can make any additional request for searching previously saved events and update their fileds.
Some example with ES like this https://gist.github.com/greem/6e02b57ff26eaacb01b2

spark-solr CDH job fails with java.lang.VerifyError at solrj.impl.HttpClientUtil

I'm running CDH 5.7.1 and submitting a Spark job that uses spark-solr 2.0.1 in yarn-cluster mode and the job is failing with the error below that arises from a (possibly) binary compatibility issue in org.apache.solr.client.solrj.impl.HttpClientUtil. The job runs fine in local mode allowing me to index a Spark dataframe to Solr 6.1.0.
I'm building the fat jar that is deployed to the cluster with the sbt-assembly plugin, see below for my build.sbt configuration.
I've tried various things to fix this such as specifying the --jars argument with spark-submit to provide the appropriate solr-solrj and httpclient jars with no luck, as well as numerous build.sbt changes to specify versions of solrj and httpclient explicitly, again with no luck.
Any insights or solutions would be greatly appreciated.
Spark job history error from yarn-cluster mode:
16/07/21 02:10:07 ERROR yarn.ApplicationMaster: User class threw exception: com.google.common.util.concurrent.ExecutionError: java.lang.VerifyError: Bad return type
Exception Details:
Location:
org/apache/solr/client/solrj/impl/HttpClientUtil.createClient(Lorg/apache/solr/common/params/SolrParams;Lorg/apache/http/conn/ClientConnectionManager;)Lorg/apache/http/impl/client/CloseableHttpClient; #58: areturn
Reason:
Type 'org/apache/http/impl/client/DefaultHttpClient' (current frame, stack[0]) is not assignable to 'org/apache/http/impl/client/CloseableHttpClient' (from method signature)
Current Frame:
bci: #58
flags: { }
locals: { 'org/apache/solr/common/params/SolrParams', 'org/apache/http/conn/ClientConnectionManager', 'org/apache/solr/common/params/ModifiableSolrParams', 'org/apache/http/impl/client/DefaultHttpClient' }
stack: { 'org/apache/http/impl/client/DefaultHttpClient' }
Bytecode:
0x0000000: bb00 0359 2ab7 0004 4db2 0005 b900 0601
0x0000010: 0099 001e b200 05bb 0007 59b7 0008 1209
0x0000020: b600 0a2c b600 0bb6 000c b900 0d02 002b
0x0000030: b800 104e 2d2c b800 0f2d b0
Stackmap Table:
append_frame(#47,Object[#143])
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2232)
at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
at com.lucidworks.spark.util.SolrSupport$.getCachedCloudClient(SolrSupport.scala:93)
at com.lucidworks.spark.util.SolrSupport$.getSolrBaseUrl(SolrSupport.scala:97)
at com.lucidworks.spark.util.SolrQuerySupport$.getUniqueKey(SolrQuerySupport.scala:82)
at com.lucidworks.spark.rdd.SolrRDD.<init>(SolrRDD.scala:32)
at com.lucidworks.spark.SolrRelation.<init>(SolrRelation.scala:63)
at solr.DefaultSource.createRelation(DefaultSource.scala:26)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222)
at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148)
at com.tempurer.intelligence.searchindexer.SolrIndexer3$.runJob(SolrIndexer3.scala:127)
at com.tempurer.intelligence.searchindexer.SolrIndexer3$.main(SolrIndexer3.scala:80)
at com.tempurer.intelligence.searchindexer.SolrIndexer3.main(SolrIndexer3.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:542)
Caused by: java.lang.VerifyError: Bad return type
I'm building the Spark app as a fat jar with the sbt-assembly plugin and here is my build.sbt file:
name := "search-indexer"
version := "0.1.0-SNAPSHOT"
scalaVersion := "2.10.6"
resolvers ++= Seq(
"Cloudera CDH 5.0" at "https://repository.cloudera.com/artifactory/cloudera-repos"
)
libraryDependencies ++= Seq(
"org.apache.hadoop" % "hadoop-common" % "2.6.0-cdh5.7.0" % "provided",
"org.apache.hadoop" % "hadoop-hdfs" % "2.6.0-cdh5.7.0" % "provided",
"org.apache.hive" % "hive-exec" % "1.1.0-cdh5.7.0",
"org.apache.spark" % "spark-core_2.10" % "1.6.0-cdh5.7.0" % "provided",
"org.apache.spark" % "spark-sql_2.10" % "1.6.0-cdh5.7.0" % "provided",
"org.apache.spark" % "spark-catalyst_2.10" % "1.6.0-cdh5.7.0" % "provided",
"org.apache.spark" % "spark-mllib_2.10" % "1.6.0-cdh5.7.0" % "provided",
"org.apache.spark" % "spark-graphx_2.10" % "1.6.0-cdh5.7.0" % "provided",
"org.apache.spark" % "spark-streaming_2.10" % "1.6.0-cdh5.7.0" % "provided",
"com.databricks" % "spark-avro_2.10" % "2.0.1",
"com.databricks" % "spark-csv_2.10" % "1.4.0",
"com.lucidworks.spark" % "spark-solr" % "2.0.1",
"com.fasterxml.jackson.core" % "jackson-core" % "2.8.0", // Solves runtime error: java.lang.NoSuchMethodError: com.fasterxml.jackson.core.JsonFactory.requiresPropertyOrdering()Z
"org.scalatest" % "scalatest_2.10" % "2.2.4" % "test"
)
// See: https://github.com/sbt/sbt-assembly
mergeStrategy in assembly <<= (mergeStrategy in assembly) { (old) =>
{
case PathList("META-INF", xs # _*) => MergeStrategy.discard
case x => MergeStrategy.first
}
}
More info, including the logged YARN context can be found in this github project ticket I opened: https://github.com/lucidworks/spark-solr/issues/75

Resources