I am using node.js utility of google cloud transfer api to transfer data from s3 bucket to my Google Cloud bucket using
#1
storagetransfer.transferJobs.create(request, function (err, response)
And once it is completed i want to proceed with with some functionality.
To check the status of transfer Job there is an api
#2
storagetransfer.transferOperations.get(request, function(err, response)
which takes
var request = {
// The name of the operation resource.
name: 'transferOperations/my-transfer-operation',
auth: authClient,
};
as parameter.
I cant seem to find what is the value that need to be input'ed in place of "my-transfer-operation"
The response received from "storagetransfer.transferJobs.create" has a data object with {name: 'transferJobs/2469904309496821948'}, but this value is not working for "my-transfer-operation" value.
I need to get value of "my-transfer-operation" which can be used a parameter for getting status of my transfer job using #2 api.
source : https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferOperations/get
You can use transferOperations.list, which you can test here, putting in the request name : 'transferOperations' and filter : "{"project_id" : "<your-project-id>"}".
The response will contain an array of operations with a structure starting like
{
"operations": [
{
"name": "transferOperations/transferJobs-<id-from-transferJobName>-<another-id>",
"metadata": {
...
"status": "<some-status>", //e.g. IN_PROGRESS
...
},
...
and that long name, containing the id you were trying to use, is what you need to input to transferOperations.get. You can also obtain the status directly from that response, as you see.
Related
I have a Angular JS application, from this application I send data to my AWS API endpoint
/**
* Bulk sync with master
*/
async syncDataWithMaster(): Promise<AxiosResponse<any> | void> {
{
axios.defaults.headers.post.Authorization = token;
const url = this.endpoint;
return axios.post(url, compressed, {
onUploadProgress: progressEvent => {
console.log('uploading')
},
onDownloadProgress: progressEvent => {
console.log('downloading')
},
}).then((response) => {
if (response.data.status == 'success') {
return response;
} else {
throw new Error('Could not authenticate user');
}
});
} catch (e) {
}
return;
}
the api gateway triggers my Lambda function (NodeJS) with the data it received:
exports.handler = async (event) => {
const localData = JSON.parse(event.body);
/**
Here get data from master and compare with local data and send back any new data
**/
const response = {
statusCode: 200,
body: JSON.stringify(newData),
};
return response;
};
the lambda function will call the database and get the master data for a user (not shown in the example) and then this data is compare using various logic with the local data and it determines if we need to send any new rows back to the local device to be store/ updated. (Before anyone asks, the nature of the application needs full data)
This principle works great for 90% of my users. However some users have fairly large amounts of data the current maximum being around 17mb of data.
So my question is is it possible to stream the data to and from the lambda function? So stream the data to the function, process and stream back? So that it is not affected by payload limits from AWS?
Or is it possible to somehow, begin sending data to the function as a stream and then as data becomes available it starts streaming data back at the same time?
(Data is JSON format)
I am wondering what alternatives to this solution (as it need to be fairly quick as well max 30sec)
(One other idea I had was for certain data above a certain size, frist client saves to s3 using signed url. The calls the api gateway for lambda. Lambda gets the saved file and compare to master. New data to be returned saved to s3 if over certain size. Then signed url returned to client. Client downloads the new data and processes) - However I am not sure if this of cost effective and it sounds live execution time may be long
Thanks for any help, been trying to figure this out for a while now
Im integrating the Zoom API with my Firebase app, and to do so I'm relying on Zooms pre-made webhooks to run an HTTP function on my Firebase for various events like "meeting started" and "meeting ended". Zoom API reference: https://marketplace.zoom.us/docs/guides/webhooks
This is the Google Cloud function that the Zoom API is calling:
exports.zoomTestA = functions.https.onCall((req, res) => {
console.log(req);
let data = req.body;
var xmlData = data.toString();
console.log(xmlData);
});
Here is the payload sent by Zoom:
{
"event": "meeting.ended",
"payload": {
"account_id": "LTf-KjgUTR2df-knT8BVEw",
"object": {
"duration": 0,
"start_time": "2019-05-07T14:02:51Z",
"timezone": "",
"topic": "Alexander Zoom Meeting",
"id": "864370042",
"type": 1,
"uuid": "2h/SWVrrQMu7fcbpLUly3g==",
"host_id": "Ty6ykNolSU2k1N4oc0NRcQ"
}
}
This causes this error to appear in my Google Cloud console:
Request body is missing data. { event: 'meeting.ended',
payload:
{ account_id: 'LTf-KjgUTR2df-knT8BVEw',
object:
{ duration: 0,
start_time: '2019-04-30T14:23:44Z',
timezone: '',
topic: 'Alexander\'s Zoom Meeting',
id: '837578313',
type: 1,
uuid: 'WotbHO3RRpSviETStKEGYA==',
host_id: 'Ty6ykNolSU2k1N4oc0NRcQ' } } }
The request body that Zoom sends is not wrapped in in a "data: {}" tag as required by Google Cloud functions. I've found solutions to this problem if you can control the payload here: Dart json.encode is not encoding as needed by Firebase Function .
My problem is that I am unable to alter the request that the Zoom API sends. Is there any way I can still accept the request in my Google Cloud function? Or is there any way to alter the format of the request sent by Zoom? I'm unable to find references for either.
One potential solution would be to set up another server that receives the request by Zoom, format it to Google Cloud functions specifications, and then pass it on to my Google Cloud function. I would however like to avoid stepping out of the Google Cloud eco-system.
Is this problem solvable on the Google Cloud platform?
So I figured it out. In Firebase / Google Cloud functions you can either make HTTP-functions with
functions.https.onCall((req, res) => {
var data = req.body;
and
functions.https.onRequest((req, res) => {
var data = req.body;
The difference seems to be that onCall is made for being used within the Firebase/ Google Cloud functions environment. However if you wan external functions you need to use onRequest as this does not require specific formatting of the payload.
Using onRequest instead solved all my problems.
I'm using the Microsoft Graph Client for Node.js to fetch email messages. I'm only interested in the headers. The code is working, but recently I got some errors in fetching internetMessageHeaders via the API.
For some messages, the API doesn't return any data for this field (and the code wasn't checking for this, hence the errors).
I tried to manually run the queries using POSTMAN. Using ?$select=internetMessageHeaders as a query param to only fetch the headers. If I remove the query param, I can fetch the message normally (body, etc.).
I haven't been able to find anywhere in the docs why this would happen.
Any ideas on what it's happening?
Edit: Here is the node.js code I am using
const MicrosoftGraphClient = require("#microsoft/microsoft-graph-client").Client
async function fetchEmailMessageHeaders(id, credentials) {
let client = MicrosoftGraphClient.init({
authProvider: callback => callback(null, credentials.access_token)
});
let req = client.api(id).select("internetMessageHeaders");
let message = await req.get();
return message.internetMessageHeaders;
}
Sample output (value of message var):
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('156751349d3cc68b')/messages(internetMessageHeaders)/$entity",
"#odata.etag": "W/\"CQAAABYAAABcr9US8aH2RIGaOGQZwDg3AAKyfdpk\"",
"id": "AQMkADAwATM0MDAAMS1hZWIxLThjZmYALTAwAi0wMAoARgAAA3wcQrEUgfhHoZ0BD2jmyXYHAFyv1RLxofZEgZo4ZBnAODcAAAIBDAAAAFyv1RLxofZEgZo4ZBnAODcAArJaPEcAAAA=",
"internetMessageHeaders": [
{
"name": "Received",
"value": "from BY2NAM01HT225.eop-nam01.prod.protection.outlook.com (2603:10a6:803:118::39) by VI1PR0301MB2221.eurprd03.prod.outlook.com with HTTPS via VE1PR03CA0050.EURPRD03.PROD.OUTLOOK.COM; Wed, 10 Apr 2019 11:41:55 +0000"
}
...
]
}
The problem is that for some emails, the message does not contain internetMessageHeaders.
I am using an inbox subscription to receive events for new email messages from Outlook. When an event is received I use the API to retrieve the message and extract headers from it.
API reference here
Edit2:
Here is an example JSON output of an event that fails to also retrieve headers:
{
"#odata.context": "https://graph.microsoft.com/v1.0/$metadata#users('53e07bf4-bb6a-4a82-a724-37dadfb1cf11')/messages(internetMessageHeaders)/$entity",
"#odata.etag": "W/\"CQAAABQAAADdeU+v2VzbRpZMSJGral7kAAXJkA==\"",
"id": "AAMkADhmMmVmM2NjLWFhNGMtNDBlYy04NzBkLTg5MmU2OWI0ODU4MgBGAAAAAAALg6E-e6CxRLEbZfrhFaDBBwAzAAMgKF1iTp242t34rFy5AAAAAAAOAAB5l_O62bUUTqTLk6KYrl4sAAGrJcjfAAA="
}
It seems internetMessageHeaders are not set when an email is replied to using outlook.office365.com. I'm also not able to see message headers when inspecting the message using Outlook on Mac.
I am currently using Swagger (2.0) on Node.js. My YAML file validates correctly and the requests are going through. My particular call has a request body using the following format -
{
"postalCode": "invalidpostalcode",
"phone": "invalidnumber",
"website": "invalidwebsite"
}
I would like to validate each body variable before continuing to the next piece of code. To accomplish validating the data I would like to use validator.js.
Now my basic validation code looks something like that -
if(!validator.isURL(req.body.website)){
var tmp = {
"message": req.__('error.url', {
field: req.body.website
})
}
res.status(400).json(tmp);
return;
}
if(!validator.isPostalCode(req.body.postalCode,'any')){
var tmp = {
"message": req.__('error.postalCode', {
field: req.body.postalCode
})
}
res.status(400).json(tmp);
return;
}
Is there there any better way to validate data sent in through the API that would require less code? For example by using a function to control the majority of the logic?
while this is one example of the endpoint I will have some in the future that will require upwards of 20 body parameters.
I have tried create a function that will return the response code. However, that returns an error Cannot set headers after they are sent to the client.
I'm having difficulty in updating an existing metafield with Shopify API. Each time I receive an error, advising me that the variant already exists... so it must be thinking I'm trying to create a new one (and not update).
I thought this may be an issue with 'put' and 'post' - so changed my method to put, however the error persists. I've hardwired in all my variables to make it easier to test.
I'm working with Cloudinary. I'm using https://github.com/sinechris/shopify-node-api with Express.js
app.post('/upload', function(req, res){
// upload page... assume we have uploaded our image - but have hard-wired a local file in for now
cloudinary.uploader.upload('/Users/Rob/Pictures/testimg.jpg', function(savedImg) {
var imageURL = savedImg.url;
console.log(imageURL)
},
{
public_id: "testimg"
});
// the saved image is returned - so we add it to our updateMetafieldData json object
var updateMetafieldData = {
"variant": {
"id": '253818949',
"metafields": [
{
"key": "variant_image_0",
"value": 'testimg', // whatever the public id of our image is.
"value_type": "string",
"namespace": "variant_image"
}
]
}
}
// and post the result to shopify - then redirect to /getvariants
Shopify.put('/admin/variants/253818949.json', updateMetafieldData, function(data){
// res.redirect('/getvariants')
});
});
I actually created Shopify Node API and was just now happened upon this months later but thought I'd answer for anyone else coming along.
Take a look at the shopify API here: https://docs.shopify.com/api/metafield#update
You can update the metafield directly by performing a PUT request against the metafield resource instead of the variant like so:
/admin/metafields/#{id}.json
You would of course need to know the ID of the metafield first so that would require a call to the variant first or you could simply store the id in your local database for reference.