I am currently working on a React.js full stack application with Express back-end. I had a question regarding a design decision for the API calls. I have 3 APIs at the moment
GET /airports/
{
"total_count":269,
"items":[
{
"airport_code":"ABJ",
"city":"ABJ",
"country":"CI",
"name":"Port Bouet Airport",
"city_name":"Abidjan",
"country_name":"Cote d'Ivoire",
"lat":5.261390209,
"lon":-3.926290035,
"alt":21,
"utc_offset":0.0
},
{
"airport_code":"ABV",
"city":"ABV",
"country":"NG",
"name":"Nnamdi Azikiwe International Airport",
"city_name":"Abuja",
"country_name":"Nigeria",
"lat":9.006790161,
"lon":7.263169765,
"alt":1123,
"utc_offset":1.0
},
........
]
}
GET /airports/{airport_code}
GET /flights/
{
"total_count": 898,
"items": [
{
"flight_number": "ZG6304",
"aircraft_registration": "ZGAJG",
"departure_airport": "BAH",
"arrival_airport": "LHR",
"scheduled_departure_time": "2020-01-01T20:50:00",
"scheduled_takeoff_time": "2020-01-01T21:00:00",
"scheduled_landing_time": "2020-01-02T03:00:00",
"scheduled_arrival_time": "2020-01-02T03:10:00"
},
{
"flight_number": "ZG6311",
"aircraft_registration": "ZGAJH",
"departure_airport": "CDG",
"arrival_airport": "FRA",
"scheduled_departure_time": "2020-01-01T06:45:00",
"scheduled_takeoff_time": "2020-01-01T06:55:00",
"scheduled_landing_time": "2020-01-01T07:50:00",
"scheduled_arrival_time": "2020-01-01T08:00:00"
},
........
]
}
I am working on building an airport arrivals and departures web application using the above data. My idea was to try and combine the data of /fligts/ and /airports/ API call based on departure_airport and arrival_airport to be able to have more information inside a single array such as information about the city_name, lat, long etc. to visualize the data. I wanted to know a good approach for solving this issue keeping in mind the computational overhead of filtering and merging large sets of data. I looked into using RxJS but I have not worked with it before to be sure if it would provide a good solution
I recommend to convert the airports array to an object. After you can access the airports by keys.
const airports = {
total_count: 269,
items: [
{
airport_code: 'ABJ',
city: 'ABJ',
country: 'CI',
name: 'Port Bouet Airport',
city_name: 'Abidjan',
country_name: "Cote d'Ivoire",
lat: 5.261390209,
lon: -3.926290035,
alt: 21,
utc_offset: 0.0,
},
{
airport_code: 'ABV',
city: 'ABV',
country: 'NG',
name: 'Nnamdi Azikiwe International Airport',
city_name: 'Abuja',
country_name: 'Nigeria',
lat: 9.006790161,
lon: 7.263169765,
alt: 1123,
utc_offset: 1.0,
},
],
};
const mappedAirports = airports.items.reduce(
(result, airport) =>
(result = { ...result, [airport.airport_code]: airport }),
{}
);
console.log(mappedAirports);
Output:
{"ABJ":{"airport_code":"ABJ","city":"ABJ","country":"CI","name":"Port Bouet Airport","city_name":"Abidjan","country_name":"Cote d'Ivoire","lat":5.261390209,"lon":-3.926290035,"alt":21,"utc_offset":0},"ABV":{"airport_code":"ABV","city":"ABV","country":"NG","name":"Nnamdi Azikiwe International Airport","city_name":"Abuja","country_name":"Nigeria","lat":9.006790161,"lon":7.263169765,"alt":1123,"utc_offset":1}}
Related
Imaging I have an array of objects, available before the aggregate query:
const groupBy = [
{
realm: 1,
latest_timestamp: 1318874398, //Date.now() values, usually different to each other
item_id: 1234, //always the same
},
{
realm: 2,
latest_timestamp: 1312467986, //actually it's $max timestamp field from the collection
item_id: 1234,
},
{
realm: ..., //there are many of them
latest_timestamp: ...,
item_id: 1234,
},
{
realm: 10,
latest_timestamp: 1318874398, //but sometimes then can be the same
item_id: 1234,
},
]
And collection (example set available on MongoPlayground) with the following schema:
{
realm: Number,
timestamp: Number,
item_id: Number,
field: Number, //any other useless fields in this case
}
My problem is, how to $group the values from the collection via the aggregation framework by using the already available set of data (from groupBy) ?
What have been tried already.
Okay, let skip crap ideas, like:
for (const element of groupBy) {
//array of `find` queries
}
My current working aggregation query is something like that:
//first stage
{
$match: {
"item": 1234
"realm" [1,2,3,4...,10]
}
},
{
$group: {
_id: {
realm: '$realm',
},
latest_timestamp: {
$max: '$timestamp',
},
data: {
$push: '$$ROOT',
},
},
},
{
$unwind: '$data',
},
{
$addFields: {
'data.latest_timestamp': {
$cond: {
if: {
$eq: ['$data.timestamp', '$latest_timestamp'],
},
then: '$latest_timestamp',
else: '$$REMOVE',
},
},
},
},
{
$replaceRoot: {
newRoot: '$data',
},
},
//At last, after this stages I can do useful job
but I found it a bit obsolete, and I already heard that using [.mapReduce][1] could solve my problem a bit faster, than this query. (But official docs doesn't sound promising about it) Does it true?
As for now, I am using 4 or 5 stages, before start working with useful (for me) documents.
Recent update:
I have checked the $facet stage and I found it curious for this certain case. Probably it will help me out.
For what it's worth:
After receiving documents after the necessary stages I am building a representative cluster chart, that you may also know as a heatmap
After that I was iterating each document (or array of objects) one-by-one to find their correct x and y coordinated in place which should be:
[
{
x: x (number, actual $price),
y: y (number, actual $realm),
value: price * quantity,
quantity: sum_of_quantity_on_price_level
}
]
As for now, it's old awful code with for...loop inside each other, but in the future, I will be using $facet => $bucket operators for that kind of job.
So, I have found an answer to my question in another, but relevant way.
I was thinking about using $facet operator and to be honest, it's still an option, but using it, as below is a bad practice.
//building $facet query before aggregation
const ObjectQuery = {}
for (const realm of realms) {
Object.assign(ObjectQuery, { `${realm.name}` : [ ... ] }
}
//mongoose query here
aggregation([{
$facet: ObjectQuery
},
...
])
So, I have chosen a $project stage and $switch operator to filter results, such as $groups do.
Also, using MapReduce could also solve this problem, but for some reason, the official Mongo docs recommends to avoid using it, and choose aggregation: $group and $merge operators instead.
How do I return the array of newly inserted documents? I want to perform a bulk insert from Java.
INSERT {
text: #text0,
createdAt: #createdAt0
} IN messages
LET result0 = { id: NEW._key, text: NEW.text, createdAt: NEW.createdAt }
INSERT {
text: #text1,
createdAt: #createdAt1
} IN messages
LET result1 = { id: NEW._key, text: NEW.text, createdAt: NEW.createdAt }
RETURN [result0, result1]
Is there a better way to collect the results from each insert, other than defining a new variable to keep the results in it?
This query should do what you want.
FOR d IN [
{text:#text0, createdAt: #createdAt0}, {text:#text1, createdAt: #createdAt1}
]
INSERT d INTO abc
RETURN {
id: NEW._id,
text: d.text,
createdAt: d.createdAt
}
An example response from the AQL is:
[
{
"id": "abc/21971344",
"text": "apple",
"createdAt": 1584107091
},
{
"id": "abc/21971345",
"text": "banana",
"createdAt": 1584108473
}
]
So long as the end user doesn't craft the AQL it should be fine, especially since you're using parameters.
If you ever used the product beyond testing, I'd definitely recommend looking at Foxx, as it gives you a layer of abstraction, hosted (relatively) within the database, which at the least stops consumers from having to execute AQL queries at all, rather they just communicate via REST endpoints.
PS. This query format also works:
FOR d IN #messages
INSERT d INTO abc
RETURN {
id: NEW._id,
text: d.text,
createdAt: d.createdAt
}
Where #messages is
"messages": [
{
"text": "apple",
"createdAt": 1584107091
},
{
"text": "banana",
"createdAt": 1584108473
}
]
EDIT 2: The following schema (provided by a colleague) works. I removed the quotation marks from the schema in the examples from Microsoft, but that still didn't work. I'm not sure what the issue is. I leave the question open in case someone else wants to provide an answer, but I've got it working.
const card = {
contentType: 'application/vnd.microsoft.card.adaptive',
content: {
$schema: 'http://adaptivecards.io/schemas/adaptive-card.json',
type: 'AdaptiveCard',
version: '1.0',
{
type: 'Input.Text',
placeholder: 'Name',
style: 'text',
maxLength: 50,
id: 'defaultInput'
},
actions: [
{
type: 'Action.Submit',
title: 'Siguiente',
data: {} // will be populated with form input values
}
]
}
};
I'm trying to make a form in my MS Bot using Adaptive Cards. I took the sample form from the MS site (https://blog.botframework.com/2019/07/02/using-adaptive-cards-with-the-microsoft-bot-framework/) but get the following error
The error seems to be thinking that my action type is Action.openUrl but I don't see that in my code, which is below. Any help much appreciated. Using Microsoft Bot Framework 3, Node 12.13.0.
function askPolicyNumber(session) {
const card = {
'$schema': 'https://adaptivecards.io/schemas/adaptive-card.json',
'type': 'AdaptiveCard',
'version': '1.1',
'body': [
{
'type': 'Input.Text',
'id': 'id_text'
},
{
'type': 'Input.Number',
'id': 'id_number'
}
],
'actions': [
{
'type': 'Action.messageBack',
'title': 'Submit',
'data': {
'prop1': true,
'prop2': []
}
}
]
};
const msg = new builder.Message(session).attachments([card]);
return session.send(msg);
}
EDIT:
It seems that no matter what I set the action to it keeps thinking it's an openUrl action. In fact, if I set it to openUrl and give it a url property, it works fine.
I looked at this page -- https://learn.microsoft.com/en-us/microsoftteams/platform/task-modules-and-cards/cards/cards-actions#adaptive-cards-actions -- and followed the instructions there for 'Adaptive Cards with messageBack action', but it didn't change anything
"actions": [
{
"type": "Action.Submit",
"title": "Click me for messageBack",
"data": {
"msteams": {
"type": "messageBack",
"displayText": "I clicked this button",
"text": "text to bots",
"value": "{\"bfKey\": \"bfVal\", \"conflictKey\": \"from value\"}"
}
}
}
]
}
There are a lot of problems with what you're doing. It is recommended that everyone use Bot Builder v4 instead of v3. The main problem that your colleague solved was that you were trying to use an Adaptive Card object as though it was an Attachment object.
The blog post you linked to explains that Adaptive Cards must follow the Adaptive Cards schema. There is no Action.messageBack in the Adaptive Cards schema. Please continue referring to the documentation for more information.
Using node-recurly, I can create a subscription object and pass it to recurly.subscriptions.create call:
const subscription = {
plan_code: plan.code,
currency: 'USD',
account: {
account_code: activationCode,
first_name: billingInfo.first_name,
last_name: billingInfo.last_name,
email: billingInfo.email,
billing_info: {
token_id: paymentToken,
},
},
};
I would also like to add subscription_add_ons property, which, looking at the documentation, supposed to be an array of add-ons. I tried passing it like this:
subscription_add_ons: [
{
add_on_code: shippingMethod.servicelevel_token,
unit_amount_in_cents: parseFloat(shippingMethod.amount) * 100,
},
],
The server returned an error:
Tag <subscription_add_ons> must consist only of sub-tags named
<subscription_add_on>
I attempted this:
subscription_add_ons: [
{
subscription_add_on: {
add_on_code: shippingMethod.servicelevel_token,
unit_amount_in_cents: parseFloat(shippingMethod.amount) * 100,
},
},
],
Got back this error:
What's the proper format to pass subscription add on in this scenario?
The proper format is:
subscription_add_ons: {
subscription_add_on: [{
add_on_code: shippingMethod.servicelevel_token,
unit_amount_in_cents: parseFloat(shippingMethod.amount) * 100,
}],
},
I ended up doing this which works whether you have 1 add-on or multiple add-ons. subscription_add_ons is an array which can contain 1 or more subscription add ons. I then send over the details (along with other info) in the subscription update call. This is similar to what you attempted in your original post so I'm not sure why that didn't work for you.
details.subscription_add_ons = [
{ subscription_add_on: {add_on_code: "stream", quantity: 3} },
{ subscription_add_on: {add_on_code: "hold", quantity: 2} }
];
I am working on a NodeJS application.
I am new to redis, I just installed it yesterday, but I'd like to be able publish this data and subscribe to it from another process.
Suppose I have the following data:
var Exchanges = [
{
_id: 'tsx',
name: 'Toronto Stock Exchange',
data: {
instrument: [
{
symbol: 'MBT'
markPrice: 0,
},
{
symbol: 'ACQ'
markPrice: 0,
}
],
orderBooks: [
{
symbol: 'MBT',
bids: [],
asks: [],
},
{
symbol: 'ACQ',
bids: [],
asks: [],
}
],
trades: [
{
timestamp: "2014-11-06T20:53:00.000Z",
symbol: "MBT",
side: "Buy",
size: 0,
price: 352.80,
},
{
timestamp: "2014-11-06T20:53:00.000Z"
symbol: "ACQ",
side: "Sell",
size: 0,
price: 382.90,
}
],
},
},
{
_id: 'nyse',
name: 'New York Stock Exchange',
data: {
instrument: [
{
symbol: 'IBM'
markPrice: 0,
},
{
symbol: 'WMT'
markPrice: 0,
}
],
orderBooks: [
/* Orderbook Data Here */
],
trades: [
/* Trades Data Here */
],
},
}
];
I am saving this with something like:
exchange.websocket_conn.on('message', function (updateData) {
// Use 'updateData' (a diff) to update exchange.data object.
// ...
// Then
redisClient.hmset(exchange._id.toString(), exchange.data);
redisClient.publish(exchange._id.toString(), exchange.data);
});
This works and does publish the data, however I've been reading about 'PSUBSCRIBE' and I'm wondering if this can be broken down a bit further:
I'd like to be able to do something like:
someOtherRedisClient.subscribe('tsx');
// Receive All Data from the Exchange Whenever Anything Changes.
someOtherRedisClient.subscribe('tsx.instrument');
// Receive 'Instrument' array of All Instruments on Exchange when any Instrument Changes.
someOtherRedisClient.subscribe(tsx.instrument:MBT');
// Get Back Only the 'MBT' Instrument Whenever It Changes.
Can the 'Pattern Subscribe' function be used to achieve this?
Thanks,
I'd break down that one big JSON into many JSON's, one for each type of content, e.g.
Level 1 (e.g. last trade price, best bid/ask)
Order book
Trades
and create a seperate topic for each, e.g.
mktdata:tsx:level1:MBT would have the market price for MBT on the TSX exchange
mktdata:tsx:orderbook:MBT would be the order book for MBT on the TSX exchange
mktdata:tsx:trades:MBT could be all the trades but more likely, due to volume, would be better used as a notification to the client to make a seperate query to get the last N trades required from a list
You don't say how many instruments you're writing into Redis, or how many different client applications are consuming the data, but assuming you've not got a huge number of instruments you could indeed use PSUBSCRIBE to get all orderbook updates across the exchange, etc. Given a list of symbols you can also subscribe a long list of channels (e.g. mktdata:tsx:level1:MBT mktdata:tsx:orderbook:MBT mktdata:tsx:level1:ACQ), which can run to tens/hundreds without problem.