I have a csv file which store up cpu usage. There is a field with date format like this "20150101-00:15:00". How can I change it to #timestamp in logstash as shown in kibana?
Use date filter on that field:
date {
match => [ "dateField" , "yyyyMMdd-HH:mm:ss"]
}
It will add the #timestamp field.
See documentation here: https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html
Related
my date which is in below format
"_messagetime" => "08/08/2022 22:18:17.254 +0530"
I am using date filter in my logstash
date {
match => ["_messagetime", "YYYY-MM-dd HH:mm:ss.SSS"]
}
but I am getting
"_dateparsefailure"
Can anyone plz suggest what might be wrong with my approach
The date filter must match the entire value of the field. It cannot just parse a prefix. Also, your date filter has YYYY-MM-dd, but your field has dd/MM/YYYY.
You can parse that field using
date { match => ["_messagetime", "dd/MM/YYYY HH:mm:ss.SSS Z"] }
to get "#timestamp" => 2022-08-08T16:48:17.254Z. Note the trailing Z in the value of [#timestamp] -- all timestamps in logstash are stored in Zulu / UTC timezone.
your error it's caused by the " +0530" string in the _messagetime field content.
To fix this, one option is :
Remove this string before the date plugin, you can do this with use of grok or dissect
For example :
filter {
grok {
match => { "_messagetime" => "%{DATESTAMP:newdate}%{DATA:trash}" }
}
}
Apply the same date plugin conf wich must work on new content now without " +0530" occurence
I have a Postgres table with a jsonb column containing UTC timestamp data in ISO format like the following:
{
"time": "2021-04-13T20:14:56Z"
}
The Django model for this table looks like:
class DateModel(models.Model):
values = models.JSONField(default=dict)
I need to query the table for all records with a timestamp on a certain date (ignoring time)
I'm looking for a solution similar to the following:
DateModel.objects.filter(values__time__date='2021-04-13')
The other solution I have found is to query for records with date greater than the previous day and less than the next one. This works but I am looking for a way to do it with a single query so the code would be more concise.
Any suggestions?
There's a couple of annotations you need to perform on the queryset to extract the time field and convert it to a datetime.
First you need to extract the time string by using django.contrib.postgres.fields.jsonb.KeyTextTransform
from django.contrib.postgres.fields.jsonb import KeyTextTransform
query = DateModel.objects.annotate(time_str=KeyTextTransform('time', 'values'))
Then you need to convert that string to a datetime using Cast
from django.db.models.functions import Cast
from django.db.models import DateTimeField
query = query.annotate(time=Cast('time_str', output_field=DateTimeField()))
Then you can filter by that annotation
query = query.filter(time__date='2021-04-13')
I have date field like this 1994/Jan In CSV .How to change it into date format.
What i am trying is this :
filter {mutate{convert=>["field_name","date"]}}
But its not working
Try this :
filter{
date{
match => [ "field_source","yyyy/MMM"]
target => "field_target"
}
}
Introduction:
In Azure Data Explorer there is a make-series-Operator which allow us to create series of specified aggregated values along specified axis.
Where is the problem:
The operator works good except the changes in timestamp format.
For example
let resolution = 1d;
let timeframe = 3d;
let start_ts = datetime_add('second', offset, ago(timeframe));
let end_ts = datetime_add('second', offset, now());
Table
| make-series max(value) default=0 on timestamp from start_ts to end_ts step resolution by col_1, col_2
Current results:
I got the result contains the timestamp in UTC like the following
"max_value": [
-2.69,
-2.79,
-2.69
],
"timestamp": [
"2020-03-29T18:01:08.0552135Z",
"2020-03-30T18:01:08.0552135Z",
"2020-03-31T18:01:08.0552135Z"
],
Expected result:
result should be like the following
"max_value": [
-2.69,
-2.79,
-2.69
],
"timestamp": [
"2020-03-29 18:01:08",
"2020-03-30 18:01:08",
"2020-03-31 18:01:08"
],
Question:
is there any way to change the datetime format which generated in make-series operation in kusto to be NOT in UTC format.
is there any way to change the datetime format which generated in make-series operation in kusto to be NOT in UTC format.
it's not clear what you define as "UTC Format". Kusto/ADX uses the ISO 8601 standard, and timestamps are always UTC. You can see that is used in your original message, e.g. 2020-03-29T18:01:08.0552135Z.
if, for whatever reason, you want to present datetime values in a different format, inside of a dynamic column (array or property bag), you could achieve that using mv-apply and format_datetime():
print arr = dynamic(
[
"2020-03-29T18:01:08.0552135Z",
"2020-03-30T18:01:08.0552135Z",
"2020-03-31T18:01:08.0552135Z"
])
| mv-apply arr on (
summarize make_list(format_datetime(todatetime(arr), "yyyy-MM-dd HH:mm:ss"))
)
I'm a bit of a noob with MongoDB, so would appreciate some help with figuring out the best solution/format/structure in storing some data.
Basically, the data that will be stored will be updated every second with a name, value and timestamp for a certain meter reading.
For example, one possibility is water level and temperature in a tank. The tank will have a name and then the level and temperature will be read and stored every second. Overall, there will be 100's of items (i.e. tanks), each with millions of timestamped values.
From what I've learnt so far (and please correct me if I'm wrong), there are a few options as how to structure the data:
A slightly RDMS approach:
This would consist of two collections, Items and Values
Items : {
_id : "id",
name : "name"
}
Values : {
_id : "id",
item_id : "item_id",
name : "name", // temp or level etc
value : "value",
timestamp : "timestamp"
}
The more document db denormalized method:
This method involves one collection of items each with an array of timestamped values
Items : {
_id : "id",
name : "name"
values : [{
name : "name", // temp or level etc
value : "value",
timestamp : "timestamp"
}]
}
A collection for each item
Save all the values in a collection named after that item.
ItemName : {
_id : "id",
name : "name", // temp or level etc
value : "value",
timestamp : "timestamp"
}
The majority of read queries will be to retrieve the timestamped values for a specified time period of an item (i.e. tank) and display in a graph. And for this, the first option makes more sense to me as I don't want to retrieve the millions of values when querying for a specific item.
Is it even possible to query for values between specific timestamps for option 2?
I will also need to query for a list of items, so maybe a combination of the first and third option with a collection for all the items and then a number of collections to store the values for each of those items?
Any feedback on this is greatly appreciated.
Don't use timestamp if you are not modifying the ObjectId.
As ObjectId itself has time stamp in it.
So you will be saving a lot of memory by it.
MongoDB Id Documentation
In case if you dont require the previous data then you can use update query in MongoDB to update the fields every second instead of storing.
If you want to store the updated data each time then instead of updating store it in flat structure.
{ "_id" : ObjectId("XXXXXX"),
"name" : "ItemName",
"value" : "ValueOfItem"
"created_at" : "timestamp"
}
Edit 1: Added timestamp as per the comments