I am trying to access google drive using the node client. This will run on a server in a background process without user involvement. In preparation, I have done the following:
Created a service account
Enabled Drive API access on the account whose drive I am accessing
Shared a particular folder in the drive with the service account (called MyFolder below).
I am able to successfully authenticate as the service account and list files inside the directory. However, I am not able to download any of the files. When I try, I apparently get a 403 error. It's kind of buried in the error message but that appears to be the issue. Here is my code:
const fs = require('fs');
const { google } = require('googleapis');
const auth = require('./service-creds.json');
(async () => {
let jwtClient = new google.auth.JWT(auth.client_email, null,
auth.private_key, ['https://www.googleapis.com/auth/drive']);
try {
const tokens = await jwtClient.authorize();
let drive = google.drive('v3');
const res1 = await drive.files.list({
auth: jwtClient, q: `name = 'MyFolder'`
});
const folder = res1.data.files[0];
const res2 = await drive.files.list({
auth: jwtClient,
q: `'${folder.id}' in parents`
});
// print out all files under MyFolder
res2.data.files.forEach(f => console.log(f.name, f.id));
const dest = fs.createWriteStream('./myfile.csv');
const file = res2.data.files[0];
const response = await drive.files.export({
fileId: file.id,
mimeType: file.mimeType,
auth: jwtClient
}, {
responseType: 'stream'
});
response.data.on('error', err => {
console.log(err);
}).on('end', () => {
console.log('done');
}).pipe(dest);
}
catch (err) {
console.log('The API returned an error: ', err);
}
})();
Here is part of the resulting error:
The API returned an error:
... at Gaxios.<anonymous> (/api-test/node_modules/gaxios/build/src/gaxios.js:73:27)
Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]:
{ body:
Gunzip {
_readableState: [Object],
readable: true,
domain: null,
_events: [Object],
_eventsCount: 7,
_maxListeners: undefined,
_writableState: [Object],
writable: true,
allowHalfOpen: true,
_transformState: [Object],
bytesRead: 0,
_opts: [Object],
_chunkSize: 16384,
_flushFlag: 2,
_finishFlushFlag: 2,
_scheduledFlushFlag: 0,
_handle: [Object],
_hadError: false,
_buffer: <Buffer 00 00 00 00 00 00 00 00 34 00 00 00 00 00 00 00 ... >,
_offset: 0,
_level: -1,
_strategy: 0 },
disturbed: false,
error: null },
[Symbol(Response internals)]:
{ url: 'https://www.googleapis.com/drive/v3/files/123abc123abc/export?mimeType=text%2Fplain',
status: 403,
statusText: 'Forbidden',
headers: Headers { [Symbol(map)]: [Object] },
counter: 0 } }
I have not been able to find anything in the error that states why the 403 is being thrown. It appears to be zipped up but I have not been able to successfully unzip any part of it.
You want to download a file of text/plain from Google Drive.
If my understanding is correct, how about this modification?
Modification points:
I think that the reason of your issue is to download the file of the mimeType of text/plain using the files.export method of Drive API.
When Google Docs (Spreadsheet, Document, Slides and so on) files are downloaded, you can do it by the files.export method of Drive API.
When you want to download the files except for Google Docs, please use the files.get method.
When I tried to download the file of text/plain using the files.export method, I could confirm that the same error occurs.
In order to reflect above points, please modify as follows.
Modified script:
From:
const response = await drive.files.export({
fileId: file.id,
mimeType: file.mimeType,
auth: jwtClient
}, {
responseType: 'stream'
});
To:
const response = await drive.files.get({
fileId: file.id,
alt: "media",
auth: jwtClient
}, {
responseType: 'stream'
});
Reference:
Download files
Related
I'm using twitter-lite library and I want to twit an image with text.
I'm able to twit only text, but My goal is twit with text and image.
At the moment I'm trying to use a static image like:
const fs = require("fs")
const imageData = fs.readFileSync("./public/images/watermark_update_idea.png")
const mediaUploadResponse = await client.post('statuses/upload', {
media:imageData,
}).catch(err => {
console.error(err);
});
But I get that error:
{
_headers: Headers {
[Symbol(map)]: [Object: null prototype] {
'cache-control': [Array],
connection: [Array],
'content-encoding': [Array],
'content-length': [Array],
'content-type': [Array],
date: [Array],
server: [Array],
'set-cookie': [Array],
'strict-transport-security': [Array],
'x-connection-hash': [Array],
'x-response-time': [Array],
'x-tsa-request-body-time': [Array]
}
},
errors: [ { message: 'Sorry, that page does not exist', code: 34 } ]
}
My main goal is to twit with buffer, But I get the same error.
I tried to replicate this test but without lucky.
Potentially two issues here:
Use different subdomains for uploading an image and for posting a tweet:
const Twitter = require("twitter-lite");
const client = Twitter({/* auth data */});
const upload = Twitter({/* auth data */, subdomain: "upload"});
You have to convert your image content to a base64 string:
const imageData = fs.readFileSync("./public/images/watermark_update_idea.png");
const { media_id_string } = await upload.post('media/upload', {
media: imageData.toString("base64")
});
Want to upload XLSX or CSV file to azure. So I have created an API
http://localhost:3000/upload
Post Request using post man
This the code
app.post('/upload',(req,res)=>{
var form = new formidable.IncomingForm();
form.parse(req, async(err,fields,files)=> {
const blobServiceClient = await BlobServiceClient.fromConnectionString(AZURE_STORAGE_CONNECTION_STRING);
const containerClient = await blobServiceClient.getContainerClient('mycon');
if(!containerClient.exists()){
const createContainerResponse = await containerClient.create();
}
const blobName = files.file.name;
const blockBlobClient = containerClient.getBlockBlobClient(blobName);
const uploadBlobResponse = await blockBlobClient.upload(files,files.file.size);
console.log(files);
res.send({message:files.file.size + "uploaded"})
})
})
When I did console.log(files) in above code snippet
{
file: File {
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
size: 18,
path: 'C:\\Users\\bakhil\\AppData\\Local\\Temp\\upload_8531745c7d9ae14f105d50c775256e00',
name: 'Book1.csv',
type: 'application/vnd.ms-excel',
hash: null,
lastModifiedDate: 2020-04-01T10:03:55.885Z,
_writeStream: WriteStream {
_writableState: [WritableState],
writable: false,
_events: [Object: null prototype] {},
_eventsCount: 0,
_maxListeners: undefined,
path: 'C:\\Users\\bakhil\\AppData\\Local\\Temp\\upload_8531745c7d9ae14f105d50c775256e00',
fd: null,
flags: 'w',
mode: 438,
start: undefined,
autoClose: true,
pos: undefined,
bytesWritten: 18,
closed: false
}
}
}
When I am triggering the endpoint from Postman, facing the following issue:
UnhandledPromiseRejectionWarning: Error: body must be a string, Blob, ArrayBuffer, ArrayBufferView, or a function returning NodeJS.ReadableStream.
I have no clue about the error. Please help me to resolve it.
Thanks in advance.
Please try by changing the following line of code:
const uploadBlobResponse = await blockBlobClient.upload(files,files.file.size);
to
const uploadBlobResponse = await blockBlobClient.uploadFile(files.file.path);
I got a little problem with the nodejs fetch module and more particularly with the JSON parse. When I want to parse the stock variable, he tells me that the size is equal to 0 ? But my file is not empty and the path is good.
The code is really simple but I don't know why this error append and I spend too much time on this.
Someone know why I get this error and how I can resolve it ?
here the code of my js file :
const fetch = require('node-fetch');
const keyword='test';
const url='http://localhost:8888/test.json';
fetch(url).then((stock) => {
console.log(stock);
const jsonFile = JSON.parse(stock);
const newCategory = jsonFile[test];
console.log(newCategory);
}).catch((e)=>{console.log(e)});
And the error in my terminal with the first console.log() :
Response {
size: 0,
timeout: 0,
[Symbol(Body internals)]:
{ body:
PassThrough {
_readableState: [ReadableState],
readable: true,
_events: [Object],
_eventsCount: 2,
_maxListeners: undefined,
_writableState: [WritableState],
writable: false,
allowHalfOpen: true,
_transformState: [Object] },
disturbed: false,
error: null },
[Symbol(Response internals)]:
{ url: 'http://localhost:8888/test.json',
status: 200,
statusText: 'OK',
headers: Headers { [Symbol(map)]: [Object] },
counter: 0 } }
SyntaxError: Unexpected token o in JSON at position 1
at JSON.parse (<anonymous>)
at fetch.then (/Users/me/Desktop/test_json/index.js:11:30)
at process._tickCallback (internal/process/next_tick.js:68:7)
The fetch api returns a Body object
you can call Body.json()
Using async/await:
const body = await fetch(url);
const json = await body.json();
console.log(json);
Using promises:
fetch(url).then((stock) => {
return stock.json()
}).then(json => {
console.log(json);
});
JSON.parse:
fetch(url).then((stock) => {
return stock.text()
}).then(text => {
console.log(JSON.parse(text));
});
You need to pass the body in json parse like bellow,
JSON.parse(stock.text())
Or you can directly use json data without parsing just using stock.json() function
I know absolutely nothing about SOAP lol, But a vital part of my software requires I use it for a particular webservice. The documentation for the webservice was written for .net so it makes it even harder for me to understand what I need to do here. On top of all that they require authentication.
For the connecting I do not need to authorize so I am able to retreive the describe function result. They are as follows:
I20151214-09:20:20.381(-8)? Getting inside soap client creation method
I20151214-09:20:20.722(-8)? Exception while invoking method 'createSoapClient' TypeError: Cannot call method 'describe' of undefined
I20151214-09:20:20.723(-8)? at Object.Soap.createClient (packages/zardak_soap/packages/zardak_soap.js:37:1)
I20151214-09:20:20.724(-8)? at [object Object].Meteor.methods.createSoapClient (controllers/server/testFiles.js:21:1)
I20151214-09:20:20.724(-8)? at maybeAuditArgumentChecks (livedata_server.js:1698:12)
I20151214-09:20:20.725(-8)? at livedata_server.js:708:19
I20151214-09:20:20.725(-8)? at [object Object]._.extend.withValue (packages/meteor/packages/meteor.js:1013:1)
I20151214-09:20:20.726(-8)? at livedata_server.js:706:40
I20151214-09:20:20.726(-8)? at [object Object]._.extend.withValue (packages/meteor/packages/meteor.js:1013:1)
I20151214-09:20:20.726(-8)? at livedata_server.js:704:46
I20151214-09:20:20.727(-8)? at tryCallTwo (C:\Users\Media Center\AppData\Local\.meteor\packages\promise\0.5.1\npm\node_modules\meteor-promise\node_modules\promise\lib\core.js:45:5)
I20151214-09:20:20.727(-8)? at doResolve (C:\Users\Media Center\AppData\Local\.meteor\packages\promise\0.5.1\npm\node_modules\meteor-promise\node_modules\promise\lib\core.js:171:13)
I20151214-09:20:21.996(-8)? Getting inside the return of the create client
I20151214-09:20:22.007(-8)? { PRIMEStandardV1_1:
I20151214-09:20:22.008(-8)? { PRIMEStandardV1_1Soap:
I20151214-09:20:22.009(-8)? { RunTrip: [Object],
I20151214-09:20:22.009(-8)? ReverseGeocode: [Object],
I20151214-09:20:22.010(-8)? FindLocationsInRadius: [Object],
I20151214-09:20:22.010(-8)? FindLocationsOnRoute: [Object],
I20151214-09:20:22.010(-8)? FindLocationsInState: [Object],
I20151214-09:20:22.011(-8)? GetAverageDieselPriceInState: [Object],
I20151214-09:20:22.012(-8)? TestRadiusGeofence: [Object],
I20151214-09:20:22.012(-8)? TestRouteGeofence: [Object],
I20151214-09:20:22.013(-8)? RunSimpleTrip: [Object],
I20151214-09:20:22.013(-8)? Geocode: [Object],
I20151214-09:20:22.014(-8)? GetTodaysUSDieselAverage: [Object],
I20151214-09:20:22.014(-8)? GetTodaysCanadianDieselAverage: [Object],
I20151214-09:20:22.015(-8)? GetTripDistance: [Object],
I20151214-09:20:22.016(-8)? ValidateLocation: [Object] },
I20151214-09:20:22.017(-8)? PRIMEStandardV1_1Soap12:
I20151214-09:20:22.017(-8)? { RunTrip: [Object],
I20151214-09:20:22.018(-8)? ReverseGeocode: [Object],
I20151214-09:20:22.019(-8)? FindLocationsInRadius: [Object],
I20151214-09:20:22.021(-8)? FindLocationsOnRoute: [Object],
I20151214-09:20:22.021(-8)? FindLocationsInState: [Object],
I20151214-09:20:22.022(-8)? GetAverageDieselPriceInState: [Object],
I20151214-09:20:22.022(-8)? TestRadiusGeofence: [Object],
I20151214-09:20:22.023(-8)? TestRouteGeofence: [Object],
I20151214-09:20:22.023(-8)? RunSimpleTrip: [Object],
I20151214-09:20:22.024(-8)? Geocode: [Object],
I20151214-09:20:22.025(-8)? GetTodaysUSDieselAverage: [Object],
I20151214-09:20:22.025(-8)? GetTodaysCanadianDieselAverage: [Object],
I20151214-09:20:22.026(-8)? GetTripDistance: [Object],
I20151214-09:20:22.026(-8)? ValidateLocation: [Object] } } }
caseless:
I20151216-11:53:14.658(-8)? { dict:
I20151216-11:53:14.658(-8)? { 'cache-control': 'private',
I20151216-11:53:14.659(-8)? 'content-type': 'text/xml; charset=utf- 8',
I20151216-11:53:14.659(-8)? server: 'Microsoft-IIS/7.0',
I20151216-11:53:14.660(-8)? 'x-aspnet-version': '4.0.30319',
I20151216-11:53:14.660(-8)? 'x-powered-by': 'ASP.NET',
I20151216-11:53:14.661(-8)? date: 'Wed, 16 Dec 2015 19:40:29 GMT',
I20151216-11:53:14.661(-8)? connection: 'close',
I20151216-11:53:14.662(-8)? 'content-length': '441' } },
I20151216-11:53:14.662(-8)? pipe: [Function],
I20151216-11:53:14.663(-8)? addListener: [Function: addListener],
I20151216-11:53:14.664(-8)? on: [Function: addListener],
I20151216-11:53:14.665(-8)? pause: [Function],
I20151216-11:53:14.665(-8)? resume: [Function],
I20151216-11:53:14.666(-8)? read: [Function],
I20151216-11:53:14.666(-8)? body: 'soap:ServerServer was unable to process request. ---> Object reference not set to an instance of an object.' }
I20151216-11:53:16.716(-8)? Error: [object Object]
I20151216-11:53:16.722(-8)? { Envelope: { Body: { Fault: [Object] } } }
I20151216-11:53:16.723(-8)? undefined
As you can see I am able to connect. Now the part that is trowing me off is to actually call one of these functions. Below is the code I am using to try to call the "RunSimpleTrip". However when I console log the Result it is a huge jumble of messages that end up running the buffer out on my cmd window and I can only see back a little ways none of it making sense.
var url = 'http://prime.promiles.com/Webservices/v1_1/PRIMEStandardV1_1.asmx?wsdl';
var simpleTrip = {
AvoidTollRoads: false,
BorderOpen: true,
RoutingMethod: "PRACTICAL",
TripLegs: [{LocationText: "77611"},
{LocationText: "90210"}]
}
Soap.createClient(url, function(err, client) {
console.log(client.describe());
client.setSecurity(new Soap.BasicAuthSecurity('hoperd', 'mailaaron', 'bkkyt'));
client.PRIMEStandardV1_1.PRIMEStandardV1_1Soap.RunSimpleTrip(simpleTrip, function(err, result, raw, soapHeader) {
//console.log("Result: ");
console.log(result);
console.log("Error: " + err.root);
console.log(err.root);
console.log(soapHeader);
// result is a javascript object
// raw is the raw response
// soapHeader is the response soap header as a javascript object
})
});
From the API's documentation this is how they call the same function using .net
PRIMEEnterpriseV1 PRIME = new PRIMEEnterpriseV1();
//Authorization Credentials
Credentials c = new Credentials();
c.Username = "MyUsername;
c.Password = "MyPassword";
c.CompanyCode ="MyCompanyCode";
SimpleTrip st = new SimpleTrip();
st.AvoidTollRoads = false;
st.BorderOpen = true;
st.RoutingMethod = com.promiles.PRIME.Enterprise.RouteMethod.PRACTICAL;
TripLeg[] Legs = new TripLeg[2];
//Origin
TripLeg t = new TripLeg();
t.LocationText = "77611";
Legs[0] = t;
//Destination
t = new TripLeg();
t.LocationText = "90210";
Legs[1] = t;
st.TripLegs = Legs;
//Call Function
SimpleTrip rt = PRIME.RunSimpleTrip(c, st);
I am hoping someone our there has a clue to this mystery for me or can point me in the right direction as to how to properly connect this this. Any and all help will be greatly appreciated.
So after much trial and error I was able to figure this out. The below code works to call the SimpleTrip and return a proper response from the server. My TripLegs arg still isn't 100% correct to what the SOAP is looking for but the code and the way to call it is.
var url = 'http://prime.promiles.com/Webservices/v1_1/PRIMEStandardV1_1.asmx?wsdl';
var credentials = { Username: "xxxxx",
Password: "xxxxxx",
CompanyCode: "xxxxx"
};
var simpleTrip = {
AvoidTollRoads: false,
BorderOpen: true,
RoutingMethod: "PRACTICAL",
VehicleType: "Tractor2AxleTrailer2Axle",
TripLegs: [{Location: {LocationText: "77611"}},
{Location: {LocationText: "90210"}}]
}
args = {c: credentials, BasicTrip: simpleTrip};
Soap.createClient(url, function(err, client) {
console.log(err);
console.log(simpleTrip);
client.RunSimpleTrip(args, function(err, result, raw, soapHeader) {
console.log(result);
//console.log(err.root);
console.log(err.root.Envelope);
})
});
What's wrong here?
res.render('/somepage', {user:req.session.user})
It leads to Converting circular structure to JSON errors, (results in session element that has a circular user reference.)
exports.home = function (req, res) {
var entityFactory = new require('../lib/entity-factory.js').EntityFactory();
entityFactory.get_job_task_lists({
callback : function (err, job_task_lists) {
res.render('home.jade', {
locals:{
title: 'Logged in.',
user:req.session.user, // does not work
job_task_lists:job_task_lists || []
}
});
}
});
};
I added some logging in node_modules/express/node_modules/connect/lib/middleware/session/memory.js
MemoryStore.prototype.set = function(sid, sess, fn){
var self = this;
process.nextTick(function(){
console.log(sess); //this is giving the output listed
self.sessions[sid] = JSON.stringify(sess);
...
This is what I expect the session to look like, in terms of structure:
{ lastAccess: 1330979534026,
cookie:
{ path: '/',
httpOnly: true,
_expires: Tue, 06 Mar 2012 00:32:14 GMT,
originalMaxAge: 14399999 },
user: // this is the object I added to the session
{ id: 1,
username: 'admin',
password: '8e3f8d3a98481a9073d2ab69f93ce73b',
creation_date: Mon, 05 Mar 2012 18:08:55 GMT } }
But here's what I find:
{ lastAccess: 1330979534079, // new session
cookie:
{ path: '/',
httpOnly: true,
_expires: Tue, 06 Mar 2012 00:32:14 GMT,
originalMaxAge: 14399999 },
user: // but here it is again, except now it's a mashup,
// containing members it shouldn't have, like locals,
// and, well, everything but the first 4 properties
{ id: 1,
username: 'admin',
password: '8e3f8d3a98481a9073d2ab69f93ce73b',
creation_date: '2012-03-05T18:08:55.701Z',
locals:
{ title: 'Logged in.',
user: [Circular], //and now it's circular
job_task_lists: [Object] },
title: 'Logged in.',
user: [Circular],
job_task_lists: [ [Object], [Object], [Object], getById: [Function] ],
attempts: [ '/home/dan/development/aqp/views/home.jade' ],
scope: {},
parentView: undefined,
root: '/home/dan/development/aqp/views',
defaultEngine: 'jade',
settings:
{ env: 'development',
hints: true,
views: '/home/dan/development/aqp/views',
'view engine': 'jade' },
app:
{ stack: [Object],
connections: 6,
allowHalfOpen: true,
_handle: [Object],
_events: [Object],
httpAllowHalfOpen: false,
cache: [Object],
settings: [Object],
redirects: {},
isCallbacks: {},
_locals: [Object],
dynamicViewHelpers: {},
errorHandlers: [],
route: '/',
routes: [Object],
router: [Getter],
__usedRouter: true },
partial: [Function],
hint: true,
filename: '/home/dan/development/aqp/views/home.jade',
layout: false,
isPartial: true } }
node.js:201
throw e; // process.nextTick error, or 'error' event on first tick
^
TypeError: Converting circular structure to JSON
at Object.stringify (native)
at Array.0 (/home/dan/development/aqp/node_modules/express/node_modules/connect/lib/middleware/session/memory.js:77:31)
at EventEmitter._tickCallback (node.js:192:40)
See how the user object is nested?
Note that this time I did not send values in explicitly with 'locals' but it ended up in one (thats the source of the circular reference.
It looks like the session is being used to transfer objects to the view.
Here's my only middleware (it only reads from the session):
function requiresAuthentication(req, res, next){
if (req.session.user){
next();
} else {
next(new Error('Unauthorized. Please log in with a valid account.'))
}
}
and the only time I modify the req.session is in this route:
app.post('/home', function (req,res,next) {
var auth = require('./lib/authentication');
auth.authenticate_user(req.body.user, function (user) {
if (user){
req.session.user = user;
console.log('authenticated');
res.redirect(req.body.redir || '/home');
//next();
} else {
console.log('not authenticated');
res.render('logins/new.jade', {title: 'Login Failed', redir:''})
}
});
});
I don't have much else going on in my application yet, as it's still quite young. I know I'm not mangling the session anywhere myself; I checked.
I did some more testing, and it appears this is only an issue when I then try to use the local variable on a page. For instance, here is my view home.jade
div(data-role="page")
div(data-role="header")
a(href='/logout', data-icon='delete', data-ajax="false") Log out
h1= title
a(href='/account', data-icon='info', data-ajax="false") Account
!= partial('user', user)
each jtl in job_task_lists
div(id=jtl.name, class = 'draggable_item', style='border:2px solid black;')
#{jtl.name} - #{jtl.description}
a(data-icon='plus')
div(data-role="footer")
h3 footer
script(src="/javascripts/home.js")
If I comment out the user partial, it renders, else I get this Converting circular structure to JSON issue.
UPDATE
So after hooking up eclipse and the v8 debugger, I have been stepping through the code and I know where the mashup of session and user objects is occurring,
in node_modules/connect/lib/middleware/session/session.js
utils.union ends up mashing the members of the user object into the session, causing the circular reference. I'm just not sure why (admittedly probably my code)
This was a problem with session data being modified in a view.
After much digging, I found that it was a bug in the way partials are handled in 2.5.8. I submitted an issue, and subsequently a patch. (in case anyone needs this info at a future date) as npm is still serving up Express 2.5.8 AFAIK.
Thanks for your help #freakish and #Ryan