I am getting this exception when getting my data with the Logstash JDBC input plugin:
error:
26413962
Sequel::InvalidValue
TZInfo::AmbiguousTime: 2017-11-05T01:30:00+00:00 is an ambiguous local time.
This is probably because I am already converting my time zone in my JDBC plugin with this parameter:
jdbc_default_timezone => "America/New_York"
Therefore 1:30am happened twice on November 5th, and I suspect Logstash doesn't know what to do and it fall in an infinite loop.
As a workaround, I removed the jdbc_default_timezone parameter and instead I convert my values in UTC in the select statement, like this:
DATEADD(hh, DATEDIFF(hh, GETDATE(), GETUTCDATE()), th.[DueDate]) as DueDate
But this workaround is annoying since I need to modify all of my logstash inputs date columns.
Is there a way to force it to pick any of the two possible times, or any more elegant way?
It seems that this is a known bug in the Logstash JDBC Input plugin, it is flagged as a P2 enhancement.
https://github.com/logstash-plugins/logstash-input-jdbc/issues/121
Meanwhile, the workaround is to convert to UTC all the date and timestamps in the SQL query, as described above in the question (MS SQL version), or like this for the Oracle version:
select from_tz(cast(<column> as timestamp), 'CET') at time zone ('EST') "#timestamp"
from <table>
where ...
We also need to remove the jdbc_default_timezone parameter in the input file and in the filter, if applicable.
Related
I am using logstashto read the postgres data using jdbc input plugin and push to elastic
The data is coming properly and things seems to be working fine just for a small problem i.e.
My logs table has a field requesttimestamp with datatype of timestamp. Since there is a historical data and also to ensure that the timelines are based on the data and not the time of run, I am trying to set the value of #timestamp with requettimestamp.
the filter configuration is as follows:
{
match => ["requesttimestamp", "yyyy-MM-dd HH:mm:ss.SSS"]
}
but while running it is tagging it as _dateparsefailure and using the system time as #timestamp
I also tried using the following format : ISO8601 but nothing seems to be working.
I am sure it is something simple but not able to put the finger to it.
I am using java8 and cassandra in my application.
The datatype of current_date in cassandra table is 'date'.
I am using entities to map to the table values. and the datatype in entity for the same field is com.datastax.driver.core.LocalDate.
When I am trying to retrieve a record
'Select * from table where current_date='2017-06-06';'
I am getting the following error'
Caused by: com.datastax.driver.core.exceptions.CodecNotFoundException: Codec
not found for requested operation:
['org.apache.cassandra.db.marshal.SimpleDateType' <->
com.datastax.driver.core.LocalDate]
I faced a similar error message while querying cassandra from Presto.
I needed to set to cassandra.protocol-version=V4 in cassandra.properties in Presto to resolve the problem in my case.
If you get this problem while using a java SDK application, check whether changing protocol version resolves the problem. In some cases, you have to write your own codec implementation.
By default, Java driver will map date type into com.datastax.driver.core.LocalDate Java type.
If you need to convert date to java.time.LocalDate, then you need to add extras to your project :
You can specify codec for given column only:
#Column(codec = LocalDateCodec.class) java.time.LocalDate current_date ;
If these two didnot work, please have a look into the code how you are creating the session,cluster etc to connect to database. Since date is a new addition to cassandra data type, Protocol version can also have an impact.
Update the version accordingly
I have the yml file which I used the "traslate" function to do lookup.
What was done is to translate a string like "superhost.com" to "found".
My problem is that if I were to add in more entries there entries will not be reflected.
For example
I add a "ultrahost.com" entry into the yml file while logstash is still running. Incoming logs with "ultrahost.com" will not be translated to "found". This will only work after I have restarted the logstash script.
There is a refresh_interval parameter to the translate plugin that can be used to specify how often to re-read the file. The default is 300 seconds (5 minutes). You can lower that to be whatever interval you think will satisfy how often that the file will be updated.
I am stuck on a problem and I am not sure what is the best way to solve it. I have a date column that I want to select and I want to fetch it as a string. Which is great, node-oracledb module has this option with fetchAsString mehotd. But it fetches the date like this for example 10-JAN-16 and I want to fetch it like this 10-01-2016. Is there a way to do that from the node-oracledb module, or I should modify the date after I get the result from the query?
UPDATE: I mean solution without to_char in the query and without query modifications
Check out this section of my series on Working with Dates in JavaScript, JSON, and Oracle Database:
https://dzone.com/articles/working-with-dates-using-the-nodejs-driver
The logon trigger shows an example of using alter session to set the default date format. Keep in mind that there is NLS_DATE_FORMAT, NLS_TIMESTAMP_FORMAT, NLS_TIMESTAMP_TZ_FORMAT.
I only show NLS_TIMESTAMP_TZ_FORMAT because I convert to that type in the examples that follow as I need to do some time zone conversion for the date format I'm using.
Another way to set the NLS parameters is to use environment variables of the same name. Note that this method will not work unless you set the NLS_LANG environment variable as well.
Doing a streaming insert into Google BigQuery, from a light Node.js app, using this package: https://www.npmjs.org/package/bigquery
I generated a timestamp on my server via this simple line of code:
jsonData['createdAt'] = new Date().getTime();
I then insert that into BigQuery, into a field with type 'timestamp'. There is no intermediate step (besides the Node package).
But many, although not all, of dates look waaaaaay off. For example:
46343-08-28 05:58:59 UTC
When that should say something like 11:45pm on 05-16-2014. However, some of my createdAt dates are correct, and I can't find a reason for the difference.
Any suggestions?
Without actually debugging the JS code, this seems to be an "off by a thousand" problem.
Check this out:
SELECT USEC_TO_TIMESTAMP(1400341611711851)
2014-05-17 15:46:51 UTC
SELECT USEC_TO_TIMESTAMP(1400341611711851*1000)
46345-01-22 13:01:51 UTC
SELECT MSEC_TO_TIMESTAMP(1400341611711851)
46345-01-22 13:01:51 UTC
SELECT MSEC_TO_TIMESTAMP(1400341611711851/1000)
2014-05-17 15:46:51 UTC
So to get a UNIX timestamp in seconds, divide the new Date().getTime() number by 1000.
I had a requirement where I am supposed to send Timestamp type to BigQuery table from NodeJS.
I did it as follows:
bigqueryClient.timestamp(new Date(Date.now()))
I hope this may help someone.
Use:
bigquery.datetime(new Date().toISOString())