I try to collect cloud-front-logs from S3's bucket and put those it into database.
Date time of logs in these files are really problem.
Is it logged in the time of Standard Time? or the time of x-edge-location?
If I want to fix this to Japan's Standard Time should I calculate by x-edge-location?
I have one more question.
Do logs delay when those written on S3 bucket??
If I observe my s3bucket by using "s3cmd ls s3://mys3bucket/".
Log's count changes within 2 hours.
https://forums.aws.amazon.com/thread.jspa?threadID=30346
The date and hour are specified according to the GMT time zone.
I found this answer.
http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html
CloudFront saves log files within 24 hours after receiving the corresponding requests.
Related
I am using https://dev.flurry.com/metrics/data-download, yesterday I tapped New Request, selected a date range of yesterday, add a filter of one parameter. I saved. Download Status = Processing, 24 hours later, Download Status is still Processing.
Is there an issue with Data Download? I so wish Event Logs still existed.
If this is not the right place to ask this, could someone please tell me where I should post this?
None
None
24 hours and still processing.
While large reports can take several hours to process, if a report is taking longer than expected, email support#flurry.com.
So I have database of users which have a reminderTime field which currently is just a string which looks like that 07:00 which is a UTC time.
In the future I'll have a multiple strings inside reminderTime which will correspond to at which time the user should receive a notification.
So imagine you logged into an app, set a multiple reminders like so 07:00, 15:00, 23:30 and sent it to server. The server will save those inside a database and run a task and send a notification at 07:00 then at 15:00 and so on. So later a user decided that he will no longer wants to receive notifications at 15:00 or change it to 15:30 and we should adapt to that.
And each user has a timezone, but I guess since reminderTime is already in UTC I can just create a task without looking at timezone.
Currently I have a reminderTime as number and after client sends me a 07:00 I convert it to seconds, but as I understand I can change that and stick to string.
All my tasks are running with Bull queue library and Redis. So as I understood the best scalable approach is to take reminderTime and just create notifications for each day at a given time and just run the task, the only problem is that should I save them to my database or add a task to a queue in Bull. The same will be for multiple times.
I don't understand how should I change already created tasks inside Bull so that the time will be different and so on.
Maybe I could just create like a 1000 records at which time user should receive a notification inside my database. Then I create a repeatable job which will run like every 5 minutes and take all of the notifications which should be send in the next couple of hours and then add them to a Bull queue and mark it that it was sent.
So basically you get the idea, maybe it could be done a little bit better.
Unless you have really a lot of users, you could simply create a schedule-like table in your DB, which is simply a list of user_id | notify_at records. Then, run a periodic task every 1-5 minutes, which compares current time and selects all the records, where notify_at is less than the current time.
Add the flag notified, if you want to send notifications more than once a day to ignore ones that was already sent. There is no need to create thousands of records for every day, you can just reset that flag once a day, e.g. at 00:00 AM.
It's ok that your users wont recieve their notifications all at the same time, there could be little delays.
The solution you suggested is pretty much fine :)
Currently, I'm working on a project that requires a window of time to be selected that is used as a valid window to trigger an event within. This window is selected by the user as a start time (24 hour time), end time (24 hour time), and a timezone. My goal is to then be able to convert these times into UTC based on the offset from the provided timezone and save into MySQL.
The main problem is I have set up the entire flow to deal with time-only data types from the mobile app all the way back to the MySQL database. I have been trying to figure out a solution that won't require changing all those data types to include date and time which would require changes in many parts of the project.
Can I make this calculation without dealing with the date? I don't believe I can as timezone offsets range from -12:00 to +14:00 which would push some windows to the next or previous days when turned into UTC.
Is the correct approach to add in the date component and then continue to update it as time progresses? I also want to ensure daylight savings doesn't create errors.
Ultimately I would like the best approach to take so if I have to change a lot now I'd rather do that then deal with a headache later. Any thoughts would be greatly appreciated!
I want to make schedules that depend on entries of a database to schedule cron jobs. Like if there's an entry in database with a timestamp 2:00 PM, 3rd of Apr, I want to send a mail to users on 2nd of Apr. I also want to send notifications at 1:55 PM 3rd of Apr.
So, this means I have to look into the database, find the entries after the current times tamp, see if they suit the criteria for notification (like 5 minutes to time stamp or 1 day to time stamp) and send the notification or mail. I'm only worried that every one minute seems like too much overload. Are the AWS web workers built for this sort of thing?
Any suggestions on how this can be accomplished?
I don't think crontab will be the best choice but if you're familiar with it, it's fine.
First you should estimate how frequently your entries are created. If, let's say, only a couple of hundred a day. My suggestion is to create the crontab job right after the entry is created. But if more than a hundred a minutes, pooling will be fine.
But there are also side effects, like canceling or updating the cron job .
I think it's better to use a proper MQ.
I've searched for several examples to analyze IIS logs using the Log Parser, taking time into account... For example, this query that shows the number of hits per hour:
SELECT
QUANTIZE(TO_LOCALTIME(TO_TIMESTAMP(date, time)), 3600) AS Hour,
COUNT(*) AS Hits
FROM D:\Logs\*.log
Group By Hour
However I cannot understand why use "TO_LOCALTIME"... Also, if there is a time difference (and a difference in results while using "TO_LOCALTIME" or not), how is that?... Thank you!
All IIS uses UTC for all times in its logs regardless of the time zone of the server, so to get your local time, you can use TO_LOCALTIME.
I guess if you are fine with UTC, you don't need to use TO_LOCALTIME.