I'm writing a program that receives data in the form of GPRMC NMEA sentences and outputs it as specified by GTFS-realtime.
GTFS-realtime wants a "timestamp" value, and describes it as such:
Moment at which the vehicle's position was measured. In POSIX time (i.e., number of seconds since January 1st 1970 00:00:00 UTC).
Here's a complete GPRMC sentence, with the relevant fields bolded:
$GPRMC,123519,A,4807.038,N,01131.000,E,022.4,084.4,230394,003.1,W*6A
The above tells me that the reading was taken at 12:35:19 on 23/03/94.
According to http://www.epochconverter.com/, this date and time is represented as 767104519 in POSIX time.
Using Erlang, what is the best way to go about deriving a POSIX timestamp from a GPRMC NMEA sentence?
Related
I'm currently working on a multi-lang online dictionary for Japanese words and kanji. My current problem is to generate furigana for kanji-compounds within expression, sentences and words. I have the kana and kanji reading (separated) available in each case, but I don't get a reliable algorithm to work, which generates the readings for each kanji-compound based on the kana reading.
I don't need the exact reading for each kanji, which is clearly impossible based on the data I have, but it should be possible to determine the readings for all kanji-compounds since I have the full sentence/word/expression in kana.
I have:
kanji = 私は学生です
kana = わたしはがくせいです
I want to automatically assign
私 to わたし
and
学生 to がくせい.
I tried to iterate over the kanji string and check if the chars 'change' between kana and kanji and looked up until this position in the kana string. This approach works for all sentences where no kanji is followed by a hiragana syllable which is the same as the reading of the kanji ends with.
Another Idea of mine was to replace all hiragana-compounds from the kanji string in the kana, and take the left kana compounds as readings for the kanji. This clearly doesn't work in each case.
How can I write such an algorithm, which works in every case?
The standard way to do this is to use a Part-of-Speech and Morphological Analyzer like MeCab.
It will split up the sentence into a bunch of tokens, and use a dictionary to generate the reading.
If you feed the CLI with your example sentence, it will look like this:
私 名詞,代名詞,一般,*,*,*,私,ワタシ,ワタシ
は 助詞,係助詞,*,*,*,*,は,ハ,ワ
学生 名詞,一般,*,*,*,*,学生,ガクセイ,ガクセイ
です 助動詞,*,*,*,特殊・デス,基本形,です,デス,デス
EOS
The next-to-last column is the reading (in katakana), and the last one is the pronunciation.
For choosing which dictionary to use, check out this article.
MeCab has Python bindings (and probably for many other programming languages).
IMPORTANT NOTE: It will NOT always produce the correct readings. There are two reasons for this:
The tokenization may be incorrect
A word can have different readings depending on the context, whereas MeCab always uses a single reading for each word
I have a given time string 14:00 and want to subtract 5 Minutes.
How can this be done in bash?
Gives me a datetime object: date -d "14:00" +'%H:%M' -> 14:00
I tried subtract: date -d "14:00 - 5min" +'%H:%M'
-> Gives 21:01. But Why?
Desired result if of course: 13:55.
The following works:
$ date -d "14:00 5 minutes ago" +'%H:%M'
13:55
I can't explain why though.
From the info page for gnu date:
Our units of temporal measurement, from seconds on up to months,
are so complicated, asymmetrical and disjunctive so as to make
coherent mental reckoning in time all but impossible. Indeed, had
some tyrannical god contrived to enslave our minds to time, to make
it all but impossible for us to escape subjection to sodden
routines and unpleasant surprises, he could hardly have done better
than handing down our present system. It is like a set of
trapezoidal building blocks, with no vertical or horizontal
surfaces, like a language in which the simplest thought demands
ornate constructions, useless particles and lengthy
circumlocutions. Unlike the more successful patterns of language
and science, which enable us to face experience boldly or at least
level-headedly, our system of temporal calculation silently and
persistently encourages our terror of time.
—Robert Grudin, ‘Time and the Art of Living’.
.
and, more relevantly:
The time may alternatively be followed by a time zone correction,
expressed as ‘SHHMM’, where S is ‘+’ or ‘-’, HH is a number of zone
hours and MM is a number of zone minutes
Your attempt to perform arithmetic is failing, and date is trying to parse the expression "- 5min" as a time zone correction.
In Erlang, i can get the Unix Epoc Timestamp by the code below:
{MegaSecs, Secs, MicroSecs} = now().
UnixTime = MegaSecs * 1000000 + Secs.
OR (as suggested by some sites)
calendar:datetime_to_gregorian_seconds(calendar:universal_time())-719528*24*3600.
However that conversion has occured for the result of erlang:now(). What if i have a given date time value in the format: "YYYY-MM-DD HH:MI:SS" e.g. "2012-12-30 15:13:40"
The time being in 24 hour format. This website gives the complete picture of what i need to be able to do in erlang.
How can i interchangeably convert from human readable formats to unix epoch timestamps and vice versa "at will", in my code. Such that if in my programs i store the timestamps as Unix Epochs, at the time of reading them, i can retrieve the human readable format automatically from the unix timestamp value or viceversa.
You can use this lib: https://github.com/selectel/tempo
tempo is a library for parsing and formatting dates in Erlang. It provides a clean and nice interface to libc's strptime and strftime functions, which are unfortunately missing from Erlang's standard library.
As well as tempo there's qdate:
https://github.com/choptastic/qdate
Would be interested to hear comparisons :)
How does mongodb apply comparison operators and sorting on _ids? Does it do it by the timestamp portion of the _id? Also, does it make a difference if the objectId was generated on the client or server?
If so, would paging be reliable on this field? e.g. _id: { $gte: last_idOnPage }
Looking at the documentation about ObjectId() you can see that _id is a hexadecimal string which represents 12-byte number which consists of:
4-byte value representing the seconds since the Unix epoch,
3-byte machine identifier,
2-byte process id, and
3-byte counter, starting with a random value.
Therefore partially you are correct: it uses timestamp to sort values as well. But other parts are also used. Because this string represents a number, mongo just compares numbers to find which one is bigger.
Regarding your second question (does it makes a difference was _id generated by application layer or by database): it does not make any difference. Mongo still compares only numbers.
Timestamp is the first portion of BSON::ObjectId value. So basically yes, it first sorts by timestamp and then by other parts.
How do you convert epoch time (unix timestamp) to standard time in D? Is there a way to customize the format?
You really should separate questions out, not ask two completely different questions at the same time.
How do you convert epoch time (unix timestamp) to standard time in D?
If you need to convert from unix time to SysTime's "std time," then you use unixTimeToStdTime:
assert(unixTimeToStdTime(0) == (Date(1970, 1, 1) - Date.init).total!"hnsecs");
So, if you do SysTime(unixTimeToStdTime(0)), you'll get a SysTime in your local time zone at the point in time when it was midnight, January 1st 1970 in UTC (i.e. the epoch time). Unlike the other SysTime constructors, it does not treat the time it's given as being in the timezone that it's given. Rather, it just sets its stdTime to the given value, and it's timezone to the given value. So, to construct an identical SysTime with the other constructors, you'd do something like
auto epochInLocalTime = SysTime(Date(1970, 1, 1), UTC()).toLocalTime();
If you want to convert in the opposite direction, stdTimeToUnixTime will convert a std time to a unix time. However, SysTime has toUnixTime on it, making it so that you generally don't need stdTimeToUnixTime.
time_t epoch = epochInLocalTime.toUnixTime();
However, one thing to be aware of is the fact that std.datetime truly deals with unix time - time_t is always considered to be in UTC. The reason that this matters is that for some inexplicable reason, Windows applies the local time's DST to time_t so that the UTC offset never changes, which means that it's wrong for a good chunk of the year. It works with all of the Microsoft's functions which use time_t, because they expect that nonsense, but you'll have interoperability problems if you try and use a Windows time_t with another box or with a library like std.datetime which actually uses time_t correctly.
Is there a way to customize the format?
You mean that you're looking for a way to provide a user-defined format for a string representing the time (like C's strftime)? std.datetime doesn't have that yet. The API still needs to be designed for it. Some discussion has taken place, but it hasn't been settled on yet. So, it'll happen eventually, but it could be a while.
In the interim, you have toISOString, toISOExtString, and toSimpleString, which use the ISO format, the ISO extended format, and Boost's simple string format respectively. In general, I'd suggest using toISOExtString, because it's both easily read by humans and standard. It's also generally best to put it in UTC (e.g. sysTime.toUTC()) when communicating with other computers (as opposed to printing it out for humans), because then the time zone is part of it, unlike with LocalTime, which doesn't append the time zone.
If you haven't read this article on std.datetime yet, then I suggest that you do, since it should give you a good overview of the module and how to use it.
I'm not familiar with D, but according to std.datetime, you could use these steps
long unixTimeToStdTime(time_t)
struct SysTime(long stdTime)
SysTime.dayOfGregorianCal()
struct Date(int day)
Date.toISOExtString()