I'm getting this error while trying to pass a file as input through my logstash configuration file. Can someone please help with proper steps on how to fix this
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], \"#\", \"input\", \"filter\", \"output\" at line 1, column 1 (byte 1)",
Related
I'm running a Dataflow job which I think should work, and is failing after 1.5 hrs with what looks like network errors. It works fine when run against a subset of the data.
The first trouble sign is a whole string of warnings like this:
Refusing to split <dataflow_worker.shuffle.GroupedShuffleRangeTracker object at 0x7f2bcb629950> at b'\xa4r\xa6\x85\x00\x01': proposed split position is out of range [b'\xa4^E\xd2\x00\x01', b'\xa4r\xa6\x85\x00\x01'). Position of last group processed was b'\xa4r\xa6\x84\x00\x01'.
Then there are four errors which seem to be about writing CSV files to GCS:
Error in _start_upload while inserting file gs://(redacted).csv: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py", line 565, in _start_upload self._client.objects.Insert(self._insert_request, upload=self._upload) File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 1156, in Insert upload=upload, upload_config=upload_config) File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 731, in _RunMethod return self.ProcessHttpResponse(method_config, http_response, request) File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 737, in ProcessHttpResponse self.__ProcessHttpResponse(method_config, http_response, request)) File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 604, in __ProcessHttpResponse http_response, method_config=method_config, request=request) apitools.base.py.exceptions.HttpError: HttpError accessing <https://www.googleapis.com/resumable/upload/storage/v1/b/(redacted).csv&uploadType=resumable&upload_id=(redacted)>: response: <{'content-type': 'text/plain; charset=utf-8', 'x-guploader-uploadid': '(redacted)', 'content-length': '0', 'date': 'Wed, 08 Jul 2020 22:17:28 GMT', 'server': 'UploadServer', 'status': '503'}>, content <>
Error in _start_upload while inserting file gs://(redacted).csv: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/gcsio.py", line 565, in _start_upload self._client.objects.Insert(self._insert_request, upload=self._upload) File "/usr/local/lib/python3.7/site-packages/apache_beam/io/gcp/internal/clients/storage/storage_v1_client.py", line 1156, in Insert upload=upload, upload_config=upload_config) File "/usr/local/lib/python3.7/site-packages/apitools/base/py/base_api.py", line 715, in _RunMethod http_request, client=self.client) File "/usr/local/lib/python3.7/site-packages/apitools/base/py/transfer.py", line 908, in InitializeUpload return self.StreamInChunks() File "/usr/local/lib/python3.7/site-packages/apitools/base/py/transfer.py", line 1020, in StreamInChunks additional_headers=additional_headers) File "/usr/local/lib/python3.7/site-packages/apitools/base/py/transfer.py", line 971, in __StreamMedia self.RefreshResumableUploadState() File "/usr/local/lib/python3.7/site-packages/apitools/base/py/transfer.py", line 873, in RefreshResumableUploadState self.stream.seek(self.progress) File "/usr/local/lib/python3.7/site-packages/apache_beam/io/filesystemio.py", line 301, in seek offset, whence, self.position, self.last_block_position)) NotImplementedError: offset: 0, whence: 0, position: 411, last: 411
The Dataflow job ID is 2020-07-07_13_08_31-7649894576933400587 -- if anyone from Google Cloud Support is able to look at this I'd be very grateful. Thanks very much.
P.S I asked a similar question last year (Dataflow job fails at BigQuery write with backend errors), the resolution was to use --experiments=use_beam_bq_sink -- I am already doing this.
You can safely ignore "Refusing to split" errors. This just means that the split position Dataflow service provided probably was received by the worker after that position was already read by the worker. Hence the worker has to ignore the split request.
Error "Error in _start_upload while inserting" seems more problematic and seems to be similar to https://issues.apache.org/jira/browse/BEAM-7014. I suspect this to be a rare flake though so I'm not sure if this was the reason for your job failure (the job only fails of the same workitem failed four times).
Can you contact Google Cloud support so that they can look into your job ?
I will mention this in the JIRA.
While i am working with open vino to convert .pb file into inference format using model optimizer, i am continuously facing "graph contains a cycle error".
And we have used TensorFlow Object Detection Models and SSD Inception V2 to generate .pb file.
[ ERROR ] -------------------------------------------------
[ ERROR ] ----------------- INTERNAL ERROR ----------------
[ ERROR ] Unexpected exception happened.
[ ERROR ] Please contact Model Optimizer developers and forward the following information:
[ ERROR ] Graph contains a cycle.
[ ERROR ] Traceback (most recent call last):
File "/opt/intel/computer_vision_sdk_2018.1.265/deployment_tools/model_optimizer/mo/main.py", line 222, in main
return driver(argv)
File "/opt/intel/computer_vision_sdk_2018.1.265/deployment_tools/model_optimizer/mo/main.py", line 190, in driver
mean_scale_values=mean_scale)
File "/opt/intel/computer_vision_sdk_2018.1.265/deployment_tools/model_optimizer/mo/pipeline/tf.py", line 141, in tf2nx
partial_infer(graph)
File "/opt/intel/computer_vision_sdk_2018.1.265/deployment_tools/model_optimizer/mo/middle/passes/infer.py", line 55, in partial_infer
nodes = nx.topological_sort(graph)
File "/opt/intel/computer_vision_sdk_2018.1.265/deployment_tools/model_optimizer/venv/lib64/python3.5/site-packages/networkx/algorithms/dag.py", line 157, in topological_sort
raise nx.NetworkXUnfeasible("Graph contains a cycle.")
networkx.exception.NetworkXUnfeasible: Graph contains a cycle.
[ ERROR ] ---------------- END OF BUG REPORT --------------
[ ERROR ] -------------------------------------------------
I am not familiar with model optimizer, but having cycles in TensorFlow graphs is totally normal. If you model has a tf.while_loop somewhere, its graph will have a cycle. My guess is that model_optimizer folks do not handle such models.
Execute the command with the following argument:
python3 mo_tf.py --input_model <MODEL_PATH> --tensorflow_use_custom_operations_config <OPENVINO_DIR>/deployment_tools/model_optimizer/extensions/front/tf/faster_rcnn_support.json
Then OpenVINO will successfully convert your model.
Check out this link for more info
I am using beats for file path in logstash and I want to merge multiline messages into a single event, I am using codec in my config file but it is not working, I am getting an error as below
Failed to execute action
{:action=>LogStash::PipelineAction::Create/pipeline_id:main,
:exception=>"LogStash::ConfigurationError", :message=>"Expected one of
, { at line 8, column 8 (byte 169) after # The # character at the beginning of a li e indicates a comment. Use\n# cooments to describe
your configuration.\ninput {\n beats{\n\tport => \"5044\"\n
}\n \n codec ",
I'm processing through Telegram history (txt file) and I need to extract & process quite complex (nested) multiline pattern.
Here's the whole pattern
Free_Trade_Calls__AltSignals:IOC/ BTC (bittrex)
BUY : 0.00164
SELL :
TARGET 1 : 0.00180
TARGET 2 : 0.00205
TARGET 3 : 0.00240
STOP LOS : 0.000120
2018-04-19 15:46:57 Free_Trade_Calls__AltSignals:TARGET
basically I am looking for a pattern starting with
Free_Trade_Calls__AltSignals: ^%(
and ending with a timestamp.
Inside that pattern (telegram message)
- exchange - in brackets in the 1st line
- extract value after BUY
- SELL values in a array of 3 SELL[3] : target 1-3
- STOP loss value (it can be either STOP, STOP LOSS, STOP LOS)....
I've found this Logstash grok multiline message but I am very new to logstash firend advised it to me. I was trying to parse this text in NodeJS but it really is pain in the ass and mad about it.
Thanks Rob :)
Since you need to grab values from each line, you don't need to use multi-line modifier. You can skip empty line with %{SPACE} character.
For your given log, this pattern can be used,
Free_Trade_Calls__AltSignals:.*\(%{WORD:exchange}\)\s*BUY\s*:\s*%{NUMBER:BUY}\s*SELL :\s*TARGET 1\s*:\s*%{NUMBER:TARGET_1}\s*TARGET 2\s*:\s*%{NUMBER:TARGET_2}\s*TARGET 3\s*:\s*%{NUMBER:TARGET_3}\s*.*:\s*%{NUMBER:StopLoss}
please note that \s* equals to %{SPACE}
It will output,
{
"exchange": [
[
"bittrex"
]
],
"BUY": [
[
"0.00164"
]
],
"BASE10NUM": [
[
"0.00164",
"0.00180",
"0.00205",
"0.00240",
"0.000120"
]
],
"TARGET_1": [
[
"0.00180"
]
],
"TARGET_2": [
[
"0.00205"
]
],
"TARGET_3": [
[
"0.00240"
]
],
"StopLoss": [
[
"0.000120"
]
]
}
I am trying to write efm for this:
file: app/assets/javascripts/topbar.js
line 17, col 3, Missing semicolon.
line 19, col 19, 'is_mobile' was used before it was defined.
line 21, col 1965, Expected '{' and instead saw 'check'.
line 25, col 18, 'onScroll' was used before it was defined.
file: app/assets/javascripts/trends.js
line 2, col 55, Missing semicolon.
line 6, col 27, 'trendTypeSelected' was used before it was defined.
line 7, col 32, Expected '===' and instead saw '=='.
JSHint check failed
but I think I am missing the concept of file on separate line. Can someone help me out with
this?
This is what I have so far and it does not work:
%-Pfile:\ %f,line\ %l\,\ col\ %c\,\ %m
Thanks!
Solution:
set efm=%-Pfile:\ %f,%*[\ ]line\ %l\\,\ col\ %c\\,\ %m,%-Q,%-GJSHint\ check\ failed
Vim provides %P, which pushes the parsed file (%f) onto the stack, see :help errorformat-separate-filename.
So for your error output, it would be something like %+Pfile: %f,...