I need to get the data from MongoDB between two given dates. The same mongo db query is working for ( yy-mm-dd hh:mm:ss.ms ) format but it is not working for ( dd-mm-yy hh:mm:ss) format.
Sample Data in DB
{
"name":"user1",
"Value":"Success",
"Date": "02-06-2020 00:00:00",
"Status":"available",
"Updated_on":"2021-01-09 00:00:00.0000"
}
Python:
start_date = "02-06-2020 00:00:00"
end_date = "11-06-2020 10:16:41"
data = list(db.collection.find({"Date":{"gte":start_date,"Slte":end_date},"Value":"Success"},{'_id':False,"Date":1,"name":1,"Value":1}))
print(data)
I need to get the data based on the "Date" field.
The problem is it is giving extra data than the start_date and end_date.
Example: if my start_date is "02-06-2020 00:00:00"and end_date is "11-06-2020 10:16:41", it is giving data from "02-04-2020 00:00:00" to "11-06-2020 10:16:41"
Any idea to achieve this and please explain why it is not taking dates correctly.
Related
I want to DELETE every record with a datetime (e.g. 2022-11-10T??????????) using a date (e.g. 2022-11-10) from a variable. For example:
last_Date = datetime.strftime(datetime.now() - timedelta(1), '%Y-%m-%d')
And use SQL like this:
cursor.execute("""
DELETE FROM alpaca_stock_prices_4H WHERE datetime = (?)
""", (last_Date))
The value of the variable last_Date is a formatted as YYYY-MM-DD date string but the column datetime in the table contains timestamps in the format YYYY-MM-DDT?????????? so you can't compare them with the = operator.
Use SQLite's strftime() or date() function to format the datetimes also to YYYY-MM-DD:
cursor.execute("""
DELETE FROM alpaca_stock_prices_4H WHERE strftime('%Y-%m-%d', datetime) = ?
""", (last_Date,))
or:
cursor.execute("""
DELETE FROM alpaca_stock_prices_4H WHERE date(datetime) = ?
""", (last_Date,))
If what you actually want is delete all yesterday's rows you can do it without passing a parameter, by using only SQL code:
cursor.execute("""
DELETE FROM alpaca_stock_prices_4H WHERE date(datetime) = date('now', '-1 day'))
""")
I have a MS SQL Server DateTime field, and Im trying to search all records that are in between a date range:
mySqlString = "select * from users where signupDate >=#from and signupdate <=#to"
The two variables containing the date range come with format MM/dd/yyyy (dataFrom and dataTo, so Im replacing #from and #to at the string as follows:
datefrom = new Date(dataFrom);
dateto = new Date(dataTo);
req.input('from', sql.DateTime, datefrom )
req.input('to', sql.DateTime, dateto )
But I do not get any result.
What's the best approach to get this working properly?
You can always use CONVERT to accommodate your SQL query to your input format. In your case its format 101: select convert(varchar, getdate(), 101) ---> mm/dd/yyyy
So your query should look like
where (signupdate >= CONVERT(date, #from, 101)) AND (signupdate <= CONVERT(date, #to, 101))
This way you won't worry about the time of the stored date
req.input('from', sql.Date, (dataFrom))
req.input('to', sql.Date, (dataTo))
Assuming you checked if dataFrom and dataTo have valid dates.
I have a Postgres table with a jsonb column containing UTC timestamp data in ISO format like the following:
{
"time": "2021-04-13T20:14:56Z"
}
The Django model for this table looks like:
class DateModel(models.Model):
values = models.JSONField(default=dict)
I need to query the table for all records with a timestamp on a certain date (ignoring time)
I'm looking for a solution similar to the following:
DateModel.objects.filter(values__time__date='2021-04-13')
The other solution I have found is to query for records with date greater than the previous day and less than the next one. This works but I am looking for a way to do it with a single query so the code would be more concise.
Any suggestions?
There's a couple of annotations you need to perform on the queryset to extract the time field and convert it to a datetime.
First you need to extract the time string by using django.contrib.postgres.fields.jsonb.KeyTextTransform
from django.contrib.postgres.fields.jsonb import KeyTextTransform
query = DateModel.objects.annotate(time_str=KeyTextTransform('time', 'values'))
Then you need to convert that string to a datetime using Cast
from django.db.models.functions import Cast
from django.db.models import DateTimeField
query = query.annotate(time=Cast('time_str', output_field=DateTimeField()))
Then you can filter by that annotation
query = query.filter(time__date='2021-04-13')
I am trying to filter the data from mongo using python code and available date format in mongo is quite different hence resulting zero records. I am trying to convert the date format but still it did not work.
One of the value from date field:
2018-06-28 21:27:31.132Z
I have connected to DB and using below code that returns zero records even though there are more than 1000 records available in DB.
I have tried by formatting as below
import datetime
date_time_str_st = '2018-03-07 23:22:29'
date_time_obj_st = datetime.datetime.strptime(date_time_str_st, '%Y-%m-%d %H:%M:%S')
date_time_str_en = '2018-03-08 00:07:44'
date_time_obj_en = datetime.datetime.strptime(date_time_str_en, '%Y-%m-%d %H:%M:%S')
foramtdt1 = date_time_obj_st.strftime("%Y-%m-%d %H:%M:%S.%fZ")
foramtdt2 = date_time_obj_en.strftime("%Y-%m-%d %H:%M:%S.%fZ")
pipeline = [{'$match':{'$and':[{'date':{'$gte': {'$date': '2018-03-07 23:22:29.683Z' }}},{'date':{'$lt': {'$date': '2018-03-08 00:07:44.629Z' }}}]}}]
Read_data = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri",connectionstring).option("pipeline",pipeline).load()
display(Read_data)
Also tried using direct filter
pipeline = [{'$match':{'$and':[{'date':{'$gte': '2018-03-07 23:22:29.683Z'}},{'date':{'$lt': '2018-03-08 00:07:44.629Z' }}]}}]
Readdata = spark.read.format("com.mongodb.spark.sql.DefaultSource").option("uri",connectionstring).option("pipeline",pipeline).load()
display(Readdata)
0 records are filtering. I believe i am not properly converting required timestamp format. Can anyone help me on this?
At first - I am a beginner with mongodb. So i have next probleb. I am using such a model as below with mongoengine:
class Stats(Document):
Name = StringField(max_length=250)
timestamp = LongField(default=mktime(datetime.now().timetuple()))
count = IntField()
<some other fields>
What exactly I want is to filter by the name (it's clear) and use aggregation operation sum over field count. But I want to count the sum of records grouped by hours/days/months.
As example, if we have records with such timestamps [1532970603, 1532972103, 153293600, 1532974500], then 1-2 form first group, and 3-4 form second group.
And that is where I have stuck. I have some ideas about grouping by every n records, or by dividing timestamp on 3600
(1 hour = 3600 seconds), but how to make it with mongoengine. Or even how to insert some expressions with python in a pipeline?
I will very appreciate any help.
I would recommend to use ISO date format and store complete date in timestamp. Here is your model
class Stats(Document):
Name = Document.StringField(max_length=250)
timestamp = Document.DateTime(default=datetime.utcnow()) //ISO time format recommended
count = Document.FloatField()
meta = {'strict': False}
Now you can aggregate them accordingly.
Stats.objects.aggregate(
{
'$group': {
'_id': {'year': {$year: '$timestamp'},
'month': {$month: '$timestamp'},
'day' : {$dayOfMonth: '$timestamp'},
'hour': {'$hour: '$timestamp'},
}
}
}
)