Convert data from spreadsheets to nested json - node.js

I'm using mongodb in my project. And I'll import about 20,000 products to the database in the end. So, I tried to write a script to convert the data from the spreadsheet to json, and then upload them to the mongodb. But many fields were missing.
I'm trying to figure out how to layout the spreadsheet so it would contain nested data. But I didn't find any resources to do so but only this package:
https://www.npmjs.com/package/spread-sheet-to-nested-json
But it has one problem, it will always contain "title" and "children", not the actual name of the field.
This is my product json:
[
{
"sku": "ADX112",
"name": {
"en": "Multi-Mat Gallery Frames",
"ar": "لوحة بإطار"
},
"brand": "Dummy brand",
"description": {
"en": "Metal frame in a Black powder-coated finish. Tempered glass. 2 removable, acid-free paper mats included with each frame. Can be hung vertically and horizontally. D-rings included. 5x7 and 8x10 frames include easel backs. Sold individually. Made in China.",
"ar": "إطار اسود. صنع في الصين."
},
"tags": [
"art",
"frame",
"لوحة",
"إطار"
],
"colors": [
"#000000"
],
"dimensions": [
"5x7",
"8x10"
],
"units_in_stock": {
"5x7": 5,
"8x10": 7
},
"thumbnail": "https://via.placeholder.com/150",
"images": [
"https://via.placeholder.com/150",
"https://via.placeholder.com/150"
],
"unit_size": {
"en": [
"individual",
"set of 3"
],
"ar": [
"فردي",
"مجموعة من 3"
]
},
"unit_price": 2000,
"discount": 19,
"category_id": "631f3ca65b2310473b978ab5",
"subCategories_ids": [
"631f3ca65b2310473b978ab5",
"631f3ca65b2310473b978ab5"
],
"featured": false
}
]
How can I layout a spreadsheet so it would be a template for future imports?

Related

Shopware 6 API: Update an existing product without stock or price

It is very common that the stock entries of a product are updated in the shop by a different ERP system than the system that updates the product information (number, properties, etc.). Shopware 6 does not seem to support this method and throws the error seen below.
I do understand that for the initial creation of a product there are some required fields like productNumber or stock but when updating an already existing product, it should be totally fine to leave those values out, so that they can be updated by an external system.
This worked in Shopware 5 but not in Shopware 6. Does anyone know a workaround?
(I am thinking about sending the existing value from Shopware because then it is technically not updated but that is my last resort.)
Array
(
[code] => c1051bb4-d103-4f74-8988-acbcafc7fdc3
[status] => 400
[detail] => This value should not be blank.
[template] => This value should not be blank.
[meta] => Array
(
[parameters] => Array
(
[{{ value }}] => null
)
)
[source] => Array
(
[pointer] => /0/stock
)
)
Full example for a request payload
{
"newData": {
"action": "upsert",
"entity": "product",
"payload": [
{
"id": "206c59a3339383101655aae7598e328c",
"language": "default",
"taxId": "6460303d84264f36858d1fe9e8c2f60f",
"name": "SLT 95 Nano Crystal",
"active": true,
"visibilities": [
{
"id": "d43840cff1b2bcc741a9a83ebb5b3c16",
"salesChannelId": "e8d8fd2337dd42e0a86b47ea68739824",
"visibility": 30
}
],
"categories": [
{
"id": "66ca2d80dc532d4d6659b4430e6954a8"
}
],
"description": "A not so much very short text. Another short text.",
"price": [
{
"net": 9999.99,
"gross": 9999.99,
"linked": true,
"currencyId": "b7d2554b0ce847cd82f3ac9bd1c0dfca"
}
],
"productNumber": "P_10254",
"crossSellings": [
{
"id": "b3e3171397d30794171877fed2329d96",
"name": "Similar Products",
"assignedProducts": [],
"type": "productList",
"active": true,
"sortBy": "name",
"sortDirection": "ASC",
"limit": 24,
"position": 1
}
]
}
]
}
}
Short example for a request payload
"payload": [
{
"id": "206c59a3339383101655aae7598e328c",
"language": "default",
"taxId": "6460303d84264f36858d1fe9e8c2f60f",
"name": "SLT 95 Nano Crystal",
"active": true,
...
}
]
I simply do not send the stock for a normal product upsert in hopes that Shopware uses the already existing stock.
With the version "v6.4.4.1 Stable Version" Shopware introduced a concept change which allows you to update existing products without having to send all required fields.
This solves this issue and is very useful if for example the stocks shall be maintained by hand and not be controlled by the API calls.

How to correctly use LUIS ML-features?

I just stumbled over the new "ML-features" in LUIS and I am not sure if I really understand how to use them correctly. The documentation seems very abstract and vague to me:
https://learn.microsoft.com/de-de/azure/cognitive-services/luis/luis-concept-feature
Besides a good general explanation a solution for the following example would be very welcome:
Example
Intent: OpenABox
Sample utterances: "open the green box", "open the azure box".
Entity: ColorEntity (no prebuilt entity).
The color should understand "green", "blue", "azure" and "olive", where "olive" should be regarded as synonym to "green" and "azure" to "blue".
Solution Proposal
I assume you would have to
Add an intent
Add a list-entity, that lists all colors and assigns their synonyms?
Add a phrase list, that again lists some, but maybe not all, colors, without respect to their meaning?
Make the ML-feature global?
Mark the values as interchangable?
Add a ML-entity, and assign the list entity as well as the phrase list as features?
Make the list-entity-feature required?
Add sample utterances and mark the entities with the list-entity? Or with the ML-entity? Or both?
Add the ML-Entity as feature to the intent? Or the phrase list? Or the list-entity? Or none at all?
Is it correct, that there is no way to confirm the correct resolution of "olive" to its canonical form "green" using the test panel? So I have to use the API to test this?
The Model
This model has been created as described above. It seems to do its job. But is this really the optimal way to do it? There seems to be a lot of redundancy in there.
{
"luis_schema_version": "7.0.0",
"intents": [
{
"name": "None",
"features": []
},
{
"name": "OpenABox",
"features": [
{
"modelName": "ColorMLEntity",
"isRequired": false
}
]
}
],
"entities": [
{
"name": "ColorMLEntity",
"children": [],
"roles": [],
"features": [
{
"featureName": "ColorPhraseList",
"isRequired": false
},
{
"modelName": "ColorListEntity",
"isRequired": true
}
]
}
],
"hierarchicals": [],
"composites": [],
"closedLists": [
{
"name": "ColorListEntity",
"subLists": [
{
"canonicalForm": "green",
"list": [
"olive"
]
},
{
"canonicalForm": "blue",
"list": [
"azure"
]
}
],
"roles": []
}
],
"prebuiltEntities": [],
"utterances": [
{
"text": "open the azure box",
"intent": "OpenABox",
"entities": [
{
"entity": "ColorMLEntity",
"startPos": 9,
"endPos": 13,
"children": []
}
]
},
{
"text": "open the green box",
"intent": "OpenABox",
"entities": [
{
"entity": "ColorMLEntity",
"startPos": 9,
"endPos": 13,
"children": []
}
]
}
],
"versionId": "0.1",
"name": "ColorTest",
"desc": "",
"culture": "en-us",
"tokenizerVersion": "1.0.0",
"patternAnyEntities": [],
"regex_entities": [],
"phraselists": [
{
"name": "ColorPhraseList",
"mode": true,
"words": "green,blue,azure,olive",
"activated": true,
"enabledForAllModels": false
}
],
"regex_features": [],
"patterns": [],
"settings": []
}
Features are supposed to be signals relevant to an intent, or an entity.
So for this example,
Create an ML entity "ColorEntity",
Label the utterances
Add ColorEntity as a feature for the intent
Then you can add a feature to ColorEntity, either a list entity or phrase list, no need for both.

Get a workable URL from Python Get request

I'm scraping a JS loaded website using requests. In order to do so, I go to inspect website, network console and look for the XHR calls to know where is the website calling for the data and how. Process would be as follows
Go to the link https://www.888sport.es/futbol/#/event/1006276426 in Chrome. Once that is loaded, you can click on many items with an unique ID. After doing so, a pop up window with information appears. In the XHR call I mentioned above you get a direct link to get this information as follows:
import requests
url='https://eu-offering.kambicdn.org/offering/v2018/888es/betoffer/outcome.json?lang=es_ES&market=ES&client_id=2&channel_id=1&ncid=1586874367958&id=2740660278'
#ncid is the date in timestamp format, and id is the unique id of the node clicked
response=requests.get(url=url,headers=headers)
Problem is, this isn't user friendly and require python. If I put this last url in the Chrome driver, I get the information but in plain text, and I can't interact with it. Is there any way to get a workable link from the request so that manually inserting it in a Chrome driver it loads that pop up window directly, as a regular website?
You've to make the requests as .json() so you receive a json dict, which you can access it with keys.
import requests
import json
def main(url):
r = requests.get(url).json()
print(r.keys())
hview = json.dumps(r, indent=4)
print(hview) # here to see it in nice view.
main("https://eu-offering.kambicdn.org/offering/v2018/888es/betoffer/outcome.json?lang=es_ES&market=ES&client_id=2&channel_id=1&ncid=1586874367958&id=2740660278")
Output:
dict_keys(['betOffers', 'events', 'prePacks'])
{
"betOffers": [
{
"id": 2210856430,
"closed": "2020-04-17T14:30:00Z",
"criterion": {
"id": 1001159858,
"label": "Final del partido",
"englishLabel": "Full Time",
"order": [],
"occurrenceType": "GOALS",
"lifetime": "FULL_TIME"
},
"betOfferType": {
"id": 2,
"name": "Partido",
"englishName": "Match"
},
"eventId": 1006276426,
"outcomes": [
{
"id": 2740660278,
"label": "1",
"englishLabel": "1",
"odds": 1150,
"participant": "FC Lokomotiv Gomel",
"type": "OT_ONE",
"betOfferId": 2210856430,
"changedDate": "2020-04-14T09:11:55Z",
"participantId": 1003789012,
"oddsFractional": "1/7",
"oddsAmerican": "-670",
"status": "OPEN",
"cashOutStatus": "ENABLED"
},
{
"id": 2740660284,
"label": "X",
"englishLabel": "X",
"odds": 6750,
"type": "OT_CROSS",
"betOfferId": 2210856430,
"changedDate": "2020-04-14T09:11:55Z",
"oddsFractional": "23/4",
"oddsAmerican": "575",
"status": "OPEN",
"cashOutStatus": "ENABLED"
},
{
"id": 2740660286,
"label": "2",
"englishLabel": "2",
"odds": 11000,
"participant": "Khimik Svetlogorsk",
"type": "OT_TWO",
"betOfferId": 2210856430,
"changedDate": "2020-04-14T09:11:55Z",
"participantId": 1001024009,
"oddsFractional": "10/1",
"oddsAmerican": "1000",
"status": "OPEN",
"cashOutStatus": "ENABLED"
}
],
"tags": [
"OFFERED_PREMATCH",
"MAIN"
],
"cashOutStatus": "ENABLED"
}
],
"events": [
{
"id": 1006276426,
"name": "FC Lokomotiv Gomel - Khimik Svetlogorsk",
"nameDelimiter": "-",
"englishName": "FC Lokomotiv Gomel - Khimik Svetlogorsk",
"homeName": "FC Lokomotiv Gomel",
"awayName": "Khimik Svetlogorsk",
"start": "2020-04-17T14:30:00Z",
"group": "1\u00aa Divisi\u00f3n",
"groupId": 2000053499,
"path": [
{
"id": 1000093190,
"name": "F\u00fatbol",
"englishName": "Football",
"termKey": "football"
},
{
"id": 2000051379,
"name": "Bielorrusa",
"englishName": "Belarus",
"termKey": "belarus"
},
{
"id": 2000053499,
"name": "1\u00aa Divisi\u00f3n",
"englishName": "1st Division",
"termKey": "1st_division"
}
],
"nonLiveBoCount": 6,
"sport": "FOOTBALL",
"tags": [
"MATCH"
],
"state": "NOT_STARTED",
"groupSortOrder": 1999999000000000000
}
],
"prePacks": []
}

Parsing nested objects Python 3.4

I am trying to extract user reviews linked to a business from the google places API.
I am using the requests library :
import requests
r = requests.get('https://maps.googleapis.com/maps/api/place/details/json?placeid=ChIJlc_6_jM4DW0RQUUtaQj2_lk&key=AIzaSyBuS4meH_HW3FO1cpUaCm6jbqzRCWe7mjc')
json_data_dic = r.json()
print(json_data_dic)
So I get the json object converted to a python object for me to parse and extract my user ratings & reviews .
I get a "lump" of text back (see below). As someone new to python/coding, all I see is a tangle of nested dictionaries and lists. In situations where there is nesting- how do I refer to an items such as 'rating', or the 'text' of the review?
Any guidance would be appreciated.
{
"html_attributions": [],
"result": {
"reviews": [
{
"relative_time_description": "in the last week",
"profile_photo_url": "//lh6.googleusercontent.com/-06-3qCU8jEg/AAAAAAAAAAI/AAAAAAAAACc/a1z-ga9rOhs/photo.jpg",
"rating": 1,
"time": 1481050128,
"author_name": "Tass Wilson",
"text": "Worse company I've had the pleasure of dealing with. They don't follow through on what they say, their technical support team knows jack all. They are unreliable and lie. The only good part about their entire team is the sales team and thats to get you in the door, signed up and committed to a 12 month contract so they can then screw you over many times without taking you out to dinner first\n\nI would literally rather go back to using smoke signals and frikkin carrier pigeons then use Orcon again as an internet provider",
"aspects": [
{
"rating": 0,
"type": "overall"
}
],
"language": "en",
"author_url": "https://www.google.com/maps/contrib/116566965301711692941/reviews"
},
{
"relative_time_description": "3 weeks ago",
"rating": 1,
"time": 1479258972,
"author_name": "Anne-Marie Petersen",
"text": "I have experienced nothing but pain with them - from the start to (almost the end) in fact so I could skip the 5 days without internet I have had my other provider set up my fibre and just cut the loss of the extra month as the final bad money I will ever have to pay them. I called them to ask why I hadn't been offered an upgrade to fibre - I was told I wasn't eligible for fibre due to bad credit rating. I flipped out. Namely because I have a good credit score - it's something I check regularly. So I said well how do you know - they gave me the number of the company they use so I could call them. I hung up, called the number - it was the YELLOW PAGES number. I call back, I get given the same number (same person answered) I am seeing RED by this point, so I say just give me the name of the company. I find the number myself - I then call them only to be told they don't even work with Orcon. Then the guy offers to do a quick scan of the system to see if my name is in then. Doesn't even appear. Round and round the mulberry bush - I called another company and finally have had my fibre installed and everything ago. I still have no idea how to use the extra remote they've given me but the internet is fabulous. Oh - and I also got sick of every time something was wrong it was always MY fault even though I knew they would go offline and fix something. I used to WORK at a telco company let me tell you I get the system. Finally I have to send the modem back but I've already been advised to take it into their head office and take a photo of myself handing it in because they have on numerous occasions said to people that they've never received the modem even though they had.... why the hell are they still even a company????\n\nUPDATE>> got sent two cancellation notices - neither of which were for accounts that I had power over apparently. Have taken to twitter to have a public forum so they can't get me on not returning my modem.",
"aspects": [
{
"rating": 0,
"type": "overall"
}
],
"language": "en",
"author_url": "https://www.google.com/maps/contrib/113533966859873380749/reviews"
},
{
"relative_time_description": "4 months ago",
"profile_photo_url": "//lh4.googleusercontent.com/-lAEJbEHaIoE/AAAAAAAAAAI/AAAAAAAAEFo/IATRvjK2Oak/photo.jpg",
"rating": 1,
"time": 1469481312,
"author_name": "Keith Rankine",
"text": "Everything works well until you try to cancel your account. Do not be fooled into thinking you cannot give them notice to cancel within your contract term. They will try everything in their power to squeeze an extra month from you. \nI had a 12 month contract and informed them of my wish to cancel on the anniversary on that sign up date. All their emails were carefully worded to imply that this could not be done. If read carefully, and you argue, they will agree to it.",
"aspects": [
{
"rating": 0,
"type": "overall"
}
],
"language": "en",
"author_url": "https://www.google.com/maps/contrib/115729563512218272075/reviews"
},
{
"relative_time_description": "a month ago",
"rating": 1,
"time": 1476082876,
"author_name": "Wayne Furlong",
"text": "Completely useless. Dishonest, lazy and downright incompetent. Corporate bullies. I'm so much happier with Bigpipe.",
"aspects": [
{
"rating": 0,
"type": "overall"
}
],
"language": "en",
"author_url": "https://www.google.com/maps/contrib/113527219809275504416/reviews"
},
{
"relative_time_description": "3 months ago",
"rating": 1,
"time": 1471292986,
"author_name": "Shaun b",
"text": "Recently upgraded to \"unlimited\" Fibre with Orcon. Most mornings (5-9) we have a limited wired or wireless connection. Too often (as is the case this morning) we have no internet (so while at home we have to use phone data). This is on Wellington's cbd area. Their customer service is such that a reply could take upwards of 4 weeks. I intend to change provider.",
"aspects": [
{
"rating": 0,
"type": "overall"
}
],
"language": "en",
"author_url": "https://www.google.com/maps/contrib/101110905108291593535/reviews"
}
],
"utc_offset": 780,
"adr_address": "<span class=\"street-address\">1 The Strand</span>, <span class=\"extended-address\">Takapuna</span>, <span class=\"locality\">Auckland</span> <span class=\"postal-code\">0622</span>, <span class=\"country-name\">New Zealand</span>",
"photos": [
{
"html_attributions": [
"Orcon"
],
"height": 877,
"photo_reference": "CoQBdwAAAO0RRplNcUkeQUxtJLTNk3uAOTadHfKZQ8g2NMa6XLRmGX2oKdUHItfnKZP0CG2WwIj198PwzfDRJpZIw4M1wSENCEOD9mFjITSwWTMjHkw1PzHb9teT6vuuROxcCdH-fwCYp0tkeBc75R8RHb2drPbTk-NN_5q88jkJTfNwdZQDEhB-25Az9550mGd00B-zK-LRGhQpTusm33tZBFXA1952txiuAUsgQA",
"width": 878
}
],
"id": "a7c161a7081101d8897c2dd2fb41fa94b812b050",
"scope": "GOOGLE",
"vicinity": "1 The Strand, Takapuna, Auckland",
"international_phone_number": "+64 800 131 415",
"url": "https://maps.google.com/?cid=6484891029444838721",
"types": [
"point_of_interest",
"establishment"
],
"name": "Orcon",
"rating": 1.8,
"geometry": {
"viewport": {
"northeast": {
"lat": -36.7889585,
"lng": 174.77310355
},
"southwest": {
"lat": -36.7890697,
"lng": 174.77302975
}
},
"location": {
"lat": -36.78901820000001,
"lng": 174.7730694
}
},
"place_id": "ChIJlc_6_jM4DW0RQUUtaQj2_lk",
"formatted_address": "1 The Strand, Takapuna, Auckland 0622, New Zealand",
"reference": "CmRRAAAAD0loSIVYDAuRKLbv5Cp6ZM_jxKHbzJ7EOrDLakY1PAlmq5YDTJ82A4qzWje0ILFv3lsEdaUpCtkuVHuOxXW6so5yqxDSfkgEnXbzd84jtfItuxis7Izu-y87vwkD7JO4EhBZB6aIdHSchBT6_USM5B5VGhTTRgmnDKndDt6amWnPkXw-57-Pww",
"icon": "https://maps.gstatic.com/mapfiles/place_api/icons/generic_business-71.png",
"website": "https://www.orcon.net.nz/",
"formatted_phone_number": "0800 131 415",
"address_components": [
{
"short_name": "1",
"types": [
"street_number"
],
"long_name": "1"
},
{
"short_name": "The Strand",
"types": [
"route"
],
"long_name": "The Strand"
},
{
"short_name": "Takapuna",
"types": [
"sublocality_level_1",
"sublocality",
"political"
],
"long_name": "Takapuna"
},
{
"short_name": "Auckland",
"types": [
"locality",
"political"
],
"long_name": "Auckland"
},
{
"short_name": "Auckland",
"types": [
"administrative_area_level_1",
"political"
],
"long_name": "Auckland"
},
{
"short_name": "NZ",
"types": [
"country",
"political"
],
"long_name": "New Zealand"
},
{
"short_name": "0622",
"types": [
"postal_code"
],
"long_name": "0622"
}
]
},
"status": "OK"
}
json_data_dic.get("result").get("reviews") or json_data_dic['result']['reviews'] gives you the list of reviews
json_data_dic.get("result").get("reviews")[0].get("text") returns the text of the first review
If you need to get each review:
for review in json_data_dic.get("result").get("reviews"):
print review.get("text")
In general, use .get(KEY) or [KEY] access a dictionary item by the key and use
[INDEX] access an item in a list by the index (starting from 0)

Marklogic 8 Node.js API - How can I scope a search on a property child of root?

[updated 17:15 on 28/09]
I'm manipulating json data of type:
[
{
"id": 1,
"title": "Sun",
"seeAlso": [
{
"id": 2,
"title": "Rain"
},
{
"id": 3,
"title": "Cloud"
}
]
},
{
"id": 2,
"title": "Rain",
"seeAlso": [
{
"id": 3,
"title": "Cloud"
}
]
},
{
"id": 3,
"title": "Cloud",
"seeAlso": [
{
"id": 1,
"title": "Sun"
}
]
},
];
After inclusion in the database, a node.js search using
db.documents.query(
q.where(
q.collection('test films'),
q.value('title','Sun')
).withOptions({categories: 'none'})
)
.result( function(results) {
console.log(JSON.stringify(results, null,2));
});
will return both the film titled 'Sun' and the films which have a seeAlso/title property (forgive the xpath syntax) = 'Sun'.
I need to find 1/ films with title = 'Sun' 2/ films with seeAlso/title = 'Sun'.
I tried a container query using q.scope() with no success; I don't find how to scope the root object node (first case) and for the second case,
q.where(q.scope(q.property('seeAlso'), q.value('title','Sun')))
returns as first result an item which matches all text inside the root object node
{
"index": 1,
"uri": "/1.json",
"path": "fn:doc(\"/1.json\")",
"score": 137216,
"confidence": 0.6202662,
"fitness": 0.6701325,
"href": "/v1/documents?uri=%2F1.json&database=Documents",
"mimetype": "application/json",
"format": "json",
"matches": [
{
"path": "fn:doc(\"/1.json\")/object-node()",
"match-text": [
"Sun Rain Cloud"
]
}
]
},
which seems crazy.
Any idea about how doing such searches on denormalized json data?
Laurent:
XPaths on JSON are supported by MarkLogic.
In particular, you might consider setting up a path range index to match /title at the root:
http://docs.marklogic.com/guide/admin/range_index#id_54948
Scoped property matching required either filtering or indexed positions to be accurate. An alternative is to set up another path range index on /seeAlso/title
For the match issue it would be useful to know the MarkLogic version and to see the entire query.
Hoping that helps,

Resources