I have the following headers:
X-NewRelic-ID: VQACUF9bCBABU1JQAAQPU1I=
X-Device-Descriptor-Id: YWY1Zjk5OGI5N2U1ODg0Zg
User-Agent: Sahibinden-Android/4.5.2 (671; Android 5.1.1; Android Android SDK built for x86)
x-api-key: e91092ad5ea2e030c201ce9ac4373f6b565a7842
x-client-profile: Generic_v2.1
x-device-id: d4dc022224a74bc7964cb4d449cd32c2
x-timestamp: 1591117620862
x-api-hash: DF7AB8944EDB3140BCB2A3EA9BC95F70C3200DB7
adid: b165e39f-b0ee-4b22-aee5-df3675136c2c
Content-Type: application/json; charset=utf-8
X-J3PopQvX-e: a;grA22x5m7f3tr-0iQLVuq4BLsOW3uMVgYrreUpRM5M9sSufn0I0K5QO-LrGIm1gY8IHM_GtH_7uQ-cRTU-eV9762ivhZAOpU0I2ZC_dii-H-LDjjBUj8dI1uMmlx8sJjURVN1NWhJuf7iYYkopysE4rI5Ku0OAq-vv8po6yl7vF6KA4c1JEoyeJHX0WImy3c4SV30gzj4fLe1HfYlsA1Qa3ea7pl6kE4fXnYZ0q9DbWcxT_xUBj2Hx7GPYqqdOy8LK2eVQtj_qIyBqlUMu_B1WBteBo8JpEDsCN6hVZVVOmCvzBFVrJgh3rXd-bz3J6-TAgFOgIlbKEob6SrxPFvmrjrjFz2CESgBYUe2YBkwzOMYiNfRLfYKRF9bifV53JcX25D6ycfE5wp72kRvUDsF049QUgUyEvAOIdZcOTbETQtoAOIYtvClrsZQ8lI8VuKpHaqm3z0Cz59jNvXDYSIyVGQ43BbFHYJmvuVDK5hms8YQe1AU4z-kWFEZbR1GI2SD3e1ivl5a8YQNj3YLId6HeHy1BZZ4vrMfKhRmW6RFz3E8--TDu3r9J_RyYOh7_UhJ6-KW1FQ4mnoAD-1ObhBgw7VvefrijzY6iQKbnVhR2GVs0eAMv8RvON81vgIE7eZvqv4TFbNfpFzbNDYE4JboRsrroyElH9kbRkh6xc8Y_dlXZnZqHZH8Z55f1rBAQjeci9YzxGUJ4ssJE2vlniRkUSf1IZEwpxDdeUmKjk95EmdNIuC3NapbQbeRumyoe7XBCN3GOJ860o1wfeVE_u6aPpwtWAU7Huc2wY4-0M8V-WraljaUYt3iH2qVV9K-GryALZujOmjl6lLZTFBEFqijc8=;oh90F-L4vdHrRPBFFSIfspu1OBIaoliYHzKnk2hKNKM=
X-J3PopQvX-a: 0JMyqk3jVhB_ktkVhvJjiwUXW1jwIGUwnf4AuOufJI0WLr3Urn4m1ANxyELGawEfHAh3vvz7CcPSjuByQZVxMm-jqYz7WjSseSmlZGLnkSqTaTTYN7Rfz7aZTf2fSHc-F5zaQdRGr3t_KDw17Wu1CTjo8QGWUizaHYjkkclkI2q9R7McZmd_K2dpbBGOFwuVeuH4BoTabg7m9PxQ5PoqFykgD2dp=1a8_-iWK85xHc7N0iC7Kr0CmVECcipNabR42f=D15vshVt0cxZaZZd9hFHoY_b9RrDR5AjRvUTznXmJalncdoR0Uf9oP8DEv=uwgpWYp2O1szeXLHhrH9x9uutcDvyvbikmfl4G3qMa1b9TFvkWQCU7xEKN0qE5MTF2JOkJP0K-tRspZTm1dV7hFISNrU-nNJnDr0NI7l8-97eot5jkcTL=I=_7NDicD=8LlPbyWHeWBts3RJBQndMnW9p=J2Kth5rDIrHzfxsodpT0PsEHYY--EGX7-ijux_Xy4BTzYfYIc1g5teDskpD11nxM1Z=ioeOo0Ig-ocuf9TlHQPZcGru8koECDU8AKu_eptN_B50=Molip5MRHnD994ZKDjzv3N2I85rGzEZIjAT33_VSKNcqdsPSkDzUNwYolSc272M3=mAMtA2j3vWPoTuBdxkNZ3BIlBiE2k5sjiAG8upeDiVhAOzQB8BeO2LmP4VHH88Ng5brkoHeX3ncLD4WkIw0R-9TAou3zNmsClfJ_0dszt24ISM-H0GNDwuKKFm92JswKvWQOrrRGAYZODRJCzQAZQgjn=oxnjxiznUegsD84foGvwAxZyZZq8yMGNuEUk0pePeQjxWmZznzAAolQer1JvM0iHGjRJwwfXnzTF71H8GElk-sPFYDOzje11ozUnOwgom-o5FYemTpj8aR4uHffTrQa1KwXK_KMkeuJY0Q2mXNXJo49CSbO=lqWL1fNxi5OTZsYU7q2jXCSStaNR1DuTymMBJO3IFBeNFZDysgaSFooy_4H8Oi4G-eXNkuTE3sT-m-ZRleF8aVbTuhWo_ZgWuQDHAgELj2wCeGHnmh1_AeewO4wM55Le-YSC2hGHtg2qXoruUCvlQ9XJ=pcS7PLEMfEGbpbxvOaPnHAxiMc1Ahq54ZIzo-tuxDBlrrBO9FMq0a__z_CJxD0hSgWBaF2nnrrKS4ijOOhP8s1T7nPEdEDQcbVO9Q_wNd8CVzU5yH-28cCf-ojGBemEziprBB=5uzby0OJsmUTkwNXJkiF=LEalaVMS=XMnJpPWIAhfBT7OGGB4OUMk=onoMa2GJnL4Wv4SsshAiEamEV7QKp53TIkQEXXpRXa3wvq0sjiXEH-0CGivZDiS2KK-ZgcB0d-_GMD4jifXUkEaw9xmMyNkAQ51HxxgRaqO1Io_2xVF=cz3xUIFN5dS044xhn2V2neeqHd31QL_xBAMvQd9En5rPmyHpy4tJGgYXqzOfznN87b_4xab=My0BkP8JC-=CduTE5yTzbZXpb3uAlhiVv=UvE1tCBoFGJgcqV_AJt_b80Re-2Tw0Jc-ELyVoNvFrVm4-eRFDNGYlRo0V_IRYaY3ImGsG4lyCjYRaTEFF1pI5=8e539paXrGCSQJlZ5qxcAh_454XM-frw_D=s_TiZLRYeFLdcvy1jgEbkKGEcj7DbuFxNu9_YEuto2PhPTnttYRB0XW3u-wYIfq0sEdkLVes3aEUgLg7UhVjJxKESVXZvoW2jRXgbNM0p0QnDtvC9lBz-z1U9=7UkFPLXAqG_DabIY920oEnlHpHXgDL9ifX5anNuNSvr25-BxXYRgxRnG9hah4eXAkc2PcBbqif5HA0nNxXvAzZ5j7ib=h-jqI8I84aDjehZTNxjIH=Z5gsYkTWVYO5U=cvyJ4Qe8gCJYzfDirhfR-I2lg7A5mvGJb15_tAjXLOnwI1874pJvhOUqfuD8eJpITuynP9LsqiRpEU1nw7X1s_Ray9SHU3EcGkr935P33m5CBvrNzASXSr=N7mzu-WxVjevBK9MrUNHNRfoXJfunTwGwPCYZEluL9YpTLDSZtrd470yVyZK-PHFo_TV8y=ETtA-IDysdrWrE9R-5DEQpkKpGJKGieNDHptrNMsVODRm2y_RGEzI_j-qmC8mQir2GDqTZKRWqmUzVkgCsIGviKYK5_h9JSERGlvrzkWIESKdX_skM-s1n0Q=Vm4bCmFz1hSBy3k2Dx0f8myZdLWR2E0InNHTjw5rmWELfr_=ax4l5EYJirE7sO3Wm9-eimauhNYen5KNC0PeGhJ7BLCu0eamN_PXge=r_G_HlF3xXhPA8FfkxCjV3v4fWCbtGBVIQGES0jQR0sGPmxHuZsQQ9BZuGhh9ykSYce-OmeOCu7xojRXh7PJx1AVwmhrm1jE48CTYSUfegx7p-CyeWKRkGOZ_gafCibxwnJ9mslopc0WviVvbbwBKrNuZrZlWXQrnzi8ml=oq35l4xWs0Nb7VZOYezEiUgoayg7pFm8eLV02GiU7lrpQMqF_pl04dXo3QAEwDW_YgIrLCtBoVM92dckSfScDL12nZREI3Yd-V3dkN23vFr2cFXbBKoHK2uVDTqUyzMxDgqHkXxPu3rzp9g0FSA95TgKke0ikBsX7KrdNBc9cQNgj2AouCXwIzzAExo2pa2wqJk7zXna3ObWHbgk1dpiJ=dk_Gbr3LOJ-CeTWGANr=u1Cnyo-weSvvLGMlQTJjWOedNDCtduC193Bfeqn2brGiKSUV3rx09ywipxjJVjGW=eJ_k0Cex7bIq_RNZJHso1qzQ-R=IR59XtaSyM9BLwKNJ4tOiCpMltCJyWOd0yJnYcgZSUCiLBINMh-e7F_Si-e_1c3VH5UqyENO8YvR59-IYl4qUQs5NpIEZc5Hl0lImENjL_7WsBB7jue4Ty0JPxXrYo3ME_P0Xc0UIEJKN3CQCzKl9a5ZmmMTrHyJHNNW0FSrMsAgUXRtDH_MuExGdqcEjgr-8hhoS2kAV_JER1FupukcsGi0JLegpLEYF9GLDnUoDCDrPRP5tQb21gQUpEdvozNmXdVNRCukPDJeRSSYr4iCM5=-3UzYPSDwkb=_7Ea=WvgXvKtIeKbYcF5ZPk8OU-rq_ysU2Slfaf7fgHHR4Aif2cx_ZTekL=0W5LQL3AZv2QBh-VjjNri
X-J3PopQvX-b: -1o1lvb
X-J3PopQvX-c: AEC_kwhyAQAA125LdUCpBmDn_gaT36bu5R1v7tLmaWD2Y_AhYfLAeVsrDVYa
X-J3PopQvX-d: o_2
X-J3PopQvX-f: AyT1lAhyAQAA645HTYkhB4gLDM5zokBx1RSq1NMZ34yfPEFY650x2JWUiytPAWf_BDWcuH01wH8AAEB3AAAAAA==
X-J3PopQvX-z: p
Host: api.sahibinden.com
Connection: Keep-Alive
Accept-Encoding: gzip
With each request I want to be able to regenerate following headers:
X-J3PopQvX-a
X-J3PopQvX-b
X-J3PopQvX-c
X-J3PopQvX-d
X-J3PopQvX-f
X-J3PopQvX-z
I know of XMLHttpRequest but Im not sure how to use it in this case. I am new to node.js so a bit clueless
Edit:
I am able to send my headers using this : try {
let res = await httpClient.get(url, {
headers: headers,
params: query,
});
But I want to change j3pop headers everytime I send them. Need to change them manually. Is there any way for that?
Edit1:
This code here generates new headers:
function() {
'use strict';
console.log("inside");
var afterReadyCbCalled = false;
var originalHeaders = ["X-Origin-DC", "gytp","Cache-Control", "max-age=0","X-Forwarded-For", "103.255.4.53","X-Client-SrcPort", "45244","Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8","Accept-Language", "en-US,en;q=0.5","X-Forwarded-Proto", "https","X-TLS-Version", "771","Upgrade-Insecure-Requests", "1","ISTL-REFERER", "https://www.sahibinden.com/",];
var originalBody = "";
function afterReadyCb() {
if (afterReadyCbCalled) return;
afterReadyCbCalled = true;
var xhr = new XMLHttpRequest();
xhr.onload = function() {
var isValid = xhr.getResponseHeader("ISTL-INFINITE-LOOP");
if (isValid != null && isValid != '') return;
var a = xhr.getResponseHeader("ISTL-REDIRECT-TO");
if (a != null && a != '') {
location.replace(a);
} else {
if (window.history != null && typeof history.replaceState === 'function') {
var responseURL = xhr.responseURL != null ? xhr.responseURL : xhr.getResponseHeader("ISTL-RESPONSE-URL");
if (responseURL != null && responseURL != '') {
history.replaceState(null, '', responseURL);
}
}
document.open();
document.write(xhr.responseText);
document.close();
}
};
xhr.open("get", location.href, true);
for (var i = 0; i < originalHeaders.length; i += 2) {
var headerName = originalHeaders[i];
try {
xhr.setRequestHeader(headerName, originalHeaders[i + 1]);
} catch (e) {}
}
xhr.setRequestHeader("ISTL-INFINITE-LOOP", '1');
xhr.send(originalBody);
var evt = document.createEvent('Event');
evt.initEvent('QLpZFJdHv', true, true);
dispatchEvent(evt);
}
addEventListener('afterReady', afterReadyCb, false);
setTimeout(afterReadyCb, 400);
}());
Code is extracted from website by simply saving the html. Here is the link to website : https://www.sahibinden.com/
Here a sample code:
function doScrapRequest (postData, onSuccessCallback, onErrorCallback) {
var options = {
hostname: "your-host-name",
port: 443,
path: "/path-you-want-to-post-to",
method: 'POST',
headers: {
'X-J3PopQvX-a': '0JMyqk3j...',
'X-J3PopQvX-b': '-1o1lvb'
'X-J3PopQvX-c': 'AEC_kwhy...'
'X-J3PopQvX-d': 'o_2'
'X-J3PopQvX-f': 'AyT1lA...'
'X-J3PopQvX-z': 'p'
}
};
var req = https.request(options, function (res) {
res.setEncoding('utf8');
var body = '';
res.on('data', function (chunk) {
body = body + chunk;
});
res.on('end',function(){
console.log("Body :" + body);
if (res.statusCode != 200) {
onErrorCallback("Api call failed with response code " + res.statusCode);
} else {
// If you are expecting application/json you can do JSON.parse in the body variable.
onSuccessCallback(body);
}
});
});
req.on('error', function (e) {
console.log("Error : " + e.message);
onErrorCallback(e);
});
// write data to request body
req.write(postData);
req.end();
}
Related
While authenticating with google i m getting this error:
TypeError: crypto.createCredentials is not a function
at exports.OAuth2._request (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\oauth\lib\oauth2.js:33:22)
at exports.OAuth2.getOAuthAccessToken (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\oauth\lib\oauth2.js:130:8)
at loaded (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\passport-oauth2\lib\strategy.js:173:20)
at NullStore.verify (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\passport-oauth2\lib\state\null.js:9:3)
at Strategy.OAuth2Strategy.authenticate (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\passport-oauth2\lib\strategy.js:219:26)
at attempt (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\passport\lib\middleware\authenticate.js:366:16)
at authenticate (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\passport\lib\middleware\authenticate.js:367:7)
at Layer.handle [as handle_request] (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\layer.js:95:5)
at next (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\route.js:137:13)
at Route.dispatch (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\route.js:112:3)
at Layer.handle [as handle_request] (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\layer.js:95:5)
at D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\index.js:281:22
at Function.process_params (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\index.js:335:12)
at next (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\index.js:275:10)
at Function.handle (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\index.js:174:3)
at router (D:\WEB_D_CODING_NINJAS\Back-end\6.major\node_modules\express\lib\router\index.js:47:12)
faced this doubt myself, took a lot of effort but finally, I figured it out, this is a problem because of your file in node modules read the error and reach here
node_modules\oauth\lib\oauth2.js:33 , on line 33 you will see that crypto.createCredentails command, I don't know why this file gets installed but change the code inside it
make it this one
var querystring= require('querystring'),
crypto= require('crypto'),
https= require('https'),
http= require('http'),
URL= require('url'),
OAuthUtils= require('./_utils');
exports.OAuth2= function(clientId, clientSecret, baseSite, authorizePath, accessTokenPath, customHeaders) {
this._clientId= clientId;
this._clientSecret= clientSecret;
this._baseSite= baseSite;
this._authorizeUrl= authorizePath || "/oauth/authorize";
this._accessTokenUrl= accessTokenPath || "/oauth/access_token";
this._accessTokenName= "access_token";
this._authMethod= "Bearer";
this._customHeaders = customHeaders || {};
this._useAuthorizationHeaderForGET= false;
//our agent
this._agent = undefined;
};
// Allows you to set an agent to use instead of the default HTTP or
// HTTPS agents. Useful when dealing with your own certificates.
exports.OAuth2.prototype.setAgent = function(agent) {
this._agent = agent;
};
// This 'hack' method is required for sites that don't use
// 'access_token' as the name of the access token (for requests).
// ( http://tools.ietf.org/html/draft-ietf-oauth-v2-16#section-7 )
// it isn't clear what the correct value should be atm, so allowing
// for specific (temporary?) override for now.
exports.OAuth2.prototype.setAccessTokenName= function ( name ) {
this._accessTokenName= name;
}
// Sets the authorization method for Authorization header.
// e.g. Authorization: Bearer <token> # "Bearer" is the authorization method.
exports.OAuth2.prototype.setAuthMethod = function ( authMethod ) {
this._authMethod = authMethod;
};
// If you use the OAuth2 exposed 'get' method (and don't construct your own _request call )
// this will specify whether to use an 'Authorize' header instead of passing the access_token as a query parameter
exports.OAuth2.prototype.useAuthorizationHeaderforGET = function(useIt) {
this._useAuthorizationHeaderForGET= useIt;
}
exports.OAuth2.prototype._getAccessTokenUrl= function() {
return this._baseSite + this._accessTokenUrl; /* + "?" + querystring.stringify(params); */
}
// Build the authorization header. In particular, build the part after the colon.
// e.g. Authorization: Bearer <token> # Build "Bearer <token>"
exports.OAuth2.prototype.buildAuthHeader= function(token) {
return this._authMethod + ' ' + token;
};
exports.OAuth2.prototype._chooseHttpLibrary= function( parsedUrl ) {
var http_library= https;
// As this is OAUth2, we *assume* https unless told explicitly otherwise.
if( parsedUrl.protocol != "https:" ) {
http_library= http;
}
return http_library;
};
exports.OAuth2.prototype._request= function(method, url, headers, post_body, access_token, callback) {
var parsedUrl= URL.parse( url, true );
if( parsedUrl.protocol == "https:" && !parsedUrl.port ) {
parsedUrl.port= 443;
}
var http_library= this._chooseHttpLibrary( parsedUrl );
var realHeaders= {};
for( var key in this._customHeaders ) {
realHeaders[key]= this._customHeaders[key];
}
if( headers ) {
for(var key in headers) {
realHeaders[key] = headers[key];
}
}
realHeaders['Host']= parsedUrl.host;
if (!realHeaders['User-Agent']) {
realHeaders['User-Agent'] = 'Node-oauth';
}
if( post_body ) {
if ( Buffer.isBuffer(post_body) ) {
realHeaders["Content-Length"]= post_body.length;
} else {
realHeaders["Content-Length"]= Buffer.byteLength(post_body);
}
} else {
realHeaders["Content-length"]= 0;
}
if( access_token && !('Authorization' in realHeaders)) {
if( ! parsedUrl.query ) parsedUrl.query= {};
parsedUrl.query[this._accessTokenName]= access_token;
}
var queryStr= querystring.stringify(parsedUrl.query);
if( queryStr ) queryStr= "?" + queryStr;
var options = {
host:parsedUrl.hostname,
port: parsedUrl.port,
path: parsedUrl.pathname + queryStr,
method: method,
headers: realHeaders
};
this._executeRequest( http_library, options, post_body, callback );
}
exports.OAuth2.prototype._executeRequest= function( http_library, options, post_body, callback ) {
// Some hosts *cough* google appear to close the connection early / send no content-length header
// allow this behaviour.
var allowEarlyClose= OAuthUtils.isAnEarlyCloseHost(options.host);
var callbackCalled= false;
function passBackControl( response, result ) {
if(!callbackCalled) {
callbackCalled=true;
if( !(response.statusCode >= 200 && response.statusCode <= 299) && (response.statusCode != 301) && (response.statusCode != 302) ) {
callback({ statusCode: response.statusCode, data: result });
} else {
callback(null, result, response);
}
}
}
var result= "";
//set the agent on the request options
if (this._agent) {
options.agent = this._agent;
}
var request = http_library.request(options);
request.on('response', function (response) {
response.on("data", function (chunk) {
result+= chunk
});
response.on("close", function (err) {
if( allowEarlyClose ) {
passBackControl( response, result );
}
});
response.addListener("end", function () {
passBackControl( response, result );
});
});
request.on('error', function(e) {
callbackCalled= true;
callback(e);
});
if( (options.method == 'POST' || options.method == 'PUT') && post_body ) {
request.write(post_body);
}
request.end();
}
exports.OAuth2.prototype.getAuthorizeUrl= function( params ) {
var params= params || {};
params['client_id'] = this._clientId;
return this._baseSite + this._authorizeUrl + "?" + querystring.stringify(params);
}
exports.OAuth2.prototype.getOAuthAccessToken= function(code, params, callback) {
var params= params || {};
params['client_id'] = this._clientId;
params['client_secret'] = this._clientSecret;
var codeParam = (params.grant_type === 'refresh_token') ? 'refresh_token' : 'code';
params[codeParam]= code;
var post_data= querystring.stringify( params );
var post_headers= {
'Content-Type': 'application/x-www-form-urlencoded'
};
this._request("POST", this._getAccessTokenUrl(), post_headers, post_data, null, function(error, data, response) {
if( error ) callback(error);
else {
var results;
try {
// As of http://tools.ietf.org/html/draft-ietf-oauth-v2-07
// responses should be in JSON
results= JSON.parse( data );
}
catch(e) {
// .... However both Facebook + Github currently use rev05 of the spec
// and neither seem to specify a content-type correctly in their response headers :(
// clients of these services will suffer a *minor* performance cost of the exception
// being thrown
results= querystring.parse( data );
}
var access_token= results["access_token"];
var refresh_token= results["refresh_token"];
delete results["refresh_token"];
callback(null, access_token, refresh_token, results); // callback results =-=
}
});
}
// Deprecated
exports.OAuth2.prototype.getProtectedResource= function(url, access_token, callback) {
this._request("GET", url, {}, "", access_token, callback );
}
exports.OAuth2.prototype.get= function(url, access_token, callback) {
if( this._useAuthorizationHeaderForGET ) {
var headers= {'Authorization': this.buildAuthHeader(access_token) }
access_token= null;
}
else {
headers= {};
}
this._request("GET", url, headers, "", access_token, callback );
}
after changing this file, google OAuth will work just fine
I using that API to create resumable Upload in Drive.
It worked but i can't receive response data from Promise
Here is my code:
var fs = require('fs');
var request = require('request');
var mime = require('mime');
var _ = require("underscore");
class resumableUpload {
constructor(credentials,token,options={}){
this.credentials = credentials; // object from json credentials
this.token = token; // object from token credentials
this.new_token = {};
this.expiry_token_time = 0;
this.options = Object.assign({
size:256*1024,
retry:-1
},options)
}
refresh_token(){
return new Promise((ok,faild)=>{
var now = new Date().getTime();
if(now>this.expiry_token_time){
request.post("https://www.googleapis.com/oauth2/v4/token",{
form:{
refresh_token:this.token.refresh_token,
client_id:this.credentials.client_id,
client_secret:this.credentials.client_secret,
redirect_uri:this.credentials.redirect_uris[0],
grant_type:'refresh_token'
}
},(err,httpResponse,body)=>{
if(err) faild(err);
this.new_token = JSON.parse(body);
this.expiry_token_time = now + this.new_token.expires_in*1000;
ok(this.new_token.access_token);
})
}else{
ok(this.new_token.access_token);
}
})
}
/*
Upload Resumable
file:{
name:'name of file',
path:'path/to/file'
}
*/
upload(file){
return new Promise((ok,faild)=>{
this.refresh_token().then((access_token)=>{
console.log(access_token);
this.file_len = fs.statSync(file.path).size;
var form = {};
if(!_.isEmpty(file.name)) form.name = file.name;
if(!_.isEmpty(file.parents)) form.parents = [file.parents];
request.post({
uri:'https://www.googleapis.com/upload/drive/v3/files?uploadType=resumable',
headers:{
'Authorization':`Bearer ${access_token}`,
'Content-Type':'application/json; charset=UTF-8',
'X-Upload-Content-Type':mime.getType(file.path),
'X-Upload-Content-Length': this.file_len,
'Content-Length':new Buffer(JSON.stringify(form)).length
},
body:JSON.stringify(form)
},(err,res,body)=>{
if(err || _.isEmpty(res.headers.location)){
if ((this.retry > 0) || (this.retry <= -1)) {
this.retry--;
return this.upload(file); // retry
} else {
//console.log(body);
faild("Over retry");
}
}else{
var location = res.headers.location;
//console.log("URL UPLOAD: ",location);
this.sendUpload(location,file).then((data)=>{
console.log("UPLOAD DONE!!!");
ok(data);
}).catch((err)=>{
console.log("ERROR: ",err);
faild(err);
});
}
})
}).catch(faild)
})
}
sendUpload(api,file,start=0){
return new Promise((ok,faild)=>{
this.refresh_token().then((access_token)=>{
var end = parseInt(start+this.options.size-1);
if(start>=this.file_len) ok("DONE");
if(end>=this.file_len) end = this.file_len - 1;
var headers = {
'Authorization':`Bearer ${access_token}`,
'Content-Length': parseInt(end - start + 1),
'Content-Range':`bytes ${start}-${end}/${this.file_len}`
}
var uploadPipe = fs.createReadStream(file.path, {
start: start,
end: end
})
uploadPipe.pipe(request.put({uri:api,headers:headers}).on('response',(res)=>{
if(res.statusCode == 308){
//console.log(res.headers);
var new_start = parseInt(res.headers.range.split('-')[1]);
console.log("Percent: "+(new_start/this.file_len*100).toFixed(2)+" %");
return this.sendUpload(api,file,new_start+1);
}else if(res.statusCode == 200 || res.statusCode == 201){
console.log("percent: 100 %");
//ok({status:"OK"});
}
}).on('error',(err)=>{
if ((this.retry > 0) || (this.retry <= -1)) {
this.retry--;
console.log("Retry Again");
return this.sendUpload(api,file,start);
}else{
faild("Over retry");
}
}).on('data',(data)=>{
if(!_.isEmpty(data)){
console.log(data.toString());
ok(data.toString());
}
}))
}).catch(faild)
})
}
}
module.exports = resumableUpload;
In that code - sendUpload function, I split my file to many chunk and upload it in to drive.
when I upload the last chunk, server will response statusCode is 200 or 201.
It console.log "percent: 100 %" and received data from that response.
My data of response like it:
{
"kind": "drive#file",
"id": "xxxxxxxxxxxxxxxxxxxxxxxxx",
"name": "test-video",
"mimeType": "video/mp4"
}
But, the Resolve (ok function) not call, cause in upload function, i can't receive it in then function. it mean i can't see "UPLOAD DONE!!!" in screen.
Resloved: Thanks for Roamer-1888
In Pipe, return a promise not work. I change return this.sendUpload(...) to this.sendUpload(...).then(ok,faild) and it worked.
I've got a nodejs script that read data in excels files and send it to a nodejs API with 'request'. this script works fine with a small amount of data. But when i test with an Excel with 60.000 lines, it breaks with error:
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
function fillWithFile(file, path, header, fileConfig) {
let workbook = XLSX.readFile(path + '/' + file); // read file
for (var sheetIterator = 0; sheetIterator < workbook.SheetNames.length; sheetIterator++) { // for each tab in excel file
let worksheetJson = XLSX.utils.sheet_to_json(workbook.Sheets[workbook.SheetNames[sheetIterator]]); // transforme one tab of excel to JSON
let config = fileConfig[workbook.SheetNames[sheetIterator]]; // get the config for the current tab
let datasPromises = getPromises(worksheetJson, config, header);
Promise.all(datasPromises).then(function (data) {
var dataString = '{"data":' + JSON.stringify(data) + ',"options":{"purgeBefore":false}}';
let options = {
url: API_URL + '/' + config.service + 's/import',
method: 'POST',
headers: header,
body: dataString
}
request.post(options, function (err, res, body) {
if (err) throw err;
console.log('Server responded with:', body);
});
});
}
}
function getPromises(worksheetJson, config, header) {
let datasPromises = [];
let promises = [];
for (let lineIterator = 0; lineIterator < worksheetJson.length; lineIterator++) { // for each line
datasPromises.push(new Promise(function (resolve, reject) {
for (let key in config.columns) { // for each column in config
if (config.columns[key].service !== undefined) { // if service exist we need to get id of the object in this service.
promises.push(new Promise(function (resolve, reject) {
findChildren(worksheetJson[lineIterator], config.columns[key], header)
.then(function (res) {
resolve({ key: key, data: res.id });
});
}));
}
else {
promises.push(new Promise(function (resolve, reject) {
resolve({ key: key, data: worksheetJson[lineIterator][config.columns[key]] });
}));
}
}
let tmpObj = {};
Promise.all(promises).then(function (values) {
for (var i = 0; i < values.length; i++) {
tmpObj[values[i].key] = values[i].data;
}
resolve(tmpObj);
});
}));
}
return datasPromises;
}
function findChildren(sheetData, config, header) { // get children with get request
let dataObj = {};
let searchParams;
for (let key in config.columns) {
dataObj[key] = sheetData[config.columns[key]];
if (searchParams === undefined) searchParams = key + '=' + sheetData[config.columns[key]];
else searchParams += '&' + key + '=' + sheetData[config.columns[key]];
}
var headers = {
'Authorization': header.Authorization,
'Accept': 'application/json, text/plain, */*',
};
var options = {
url: API_URL + '/' + config.service + 's?' + searchParams,
headers: headers
};
return new Promise(function (resolve, reject) {
request(options, function (error, response, body) {
if (!error && response.statusCode == 200) {
try {
resolve(JSON.parse(body));
} catch (e) {
reject(e);
}
}
else {
return reject(error);
}
});
});
}
The script use a huge amount of memory and he ends up to crash ...
Anyone have an idea how i can solve this ?
I think, i have to find a way to force promises to resolve instantly for avoid the memory overflow.
thanks for any help.
I think the problem is you don't control parallelism and when everything runs 'at once' - it causes memory issues.
I've tried to rewrite the code using async/await - effectively all operations are now serial:
async function fillWithFile(file, path, header, fileConfig) {
let workbook = XLSX.readFile(path + '/' + file); // read file
for (var sheetIterator = 0; sheetIterator < workbook.SheetNames.length; sheetIterator++) { // for each tab in excel file
const worksheetJson = XLSX.utils.sheet_to_json(workbook.Sheets[workbook.SheetNames[sheetIterator]]); // transforme one tab of excel to JSON
const config = fileConfig[workbook.SheetNames[sheetIterator]]; // get the config for the current tab
const data = await getData(worksheetJson, config, header);
const dataString = '{"data":' + JSON.stringify(data) + ',"options":{"purgeBefore":false}}';
const options = {
url: API_URL + '/' + config.service + 's/import',
method: 'POST',
headers: header,
body: dataString
};
await new Promise((resolve, reject) => {
request.post(options, function (err, res, body) {
if (err) {
reject(err);
return;
}
console.log('Server responded with:', body);
resolve(body);
})
});
}
}
async function getData(worksheetJson, config, header) {
const data = [];
for (let lineIterator = 0; lineIterator < worksheetJson.length; lineIterator++) { // for each line
const values = {};
for (let key in config.columns) { // for each column in config
if (config.columns[key].service !== undefined) { // if service exist we need to get id of the object in this service.
const res = await findChildren(worksheetJson[lineIterator], config.columns[key], header);
values[key] = res.id;
}
else {
values[key] = worksheetJson[lineIterator][config.columns[key]];
}
}
data.push(values);
}
return data;
}
function findChildren(sheetData, config, header) { // get children with get request
let dataObj = {};
let searchParams;
for (let key in config.columns) {
dataObj[key] = sheetData[config.columns[key]];
if (searchParams === undefined) searchParams = key + '=' + sheetData[config.columns[key]];
else searchParams += '&' + key + '=' + sheetData[config.columns[key]];
}
var headers = {
'Authorization': header.Authorization,
'Accept': 'application/json, text/plain, */*',
};
var options = {
url: API_URL + '/' + config.service + 's?' + searchParams,
headers: headers
};
return new Promise(function (resolve, reject) {
request(options, function (error, response, body) {
if (!error && response.statusCode === 200) {
try {
resolve(JSON.parse(body));
} catch (e) {
reject(e);
}
}
else {
return reject(error);
}
});
});
}
You now run import like this:
fillWithFile(..args).then(() => console.log('done'))
Using a lambda calling an external API....Populating the POST with a JSON of an xml file.
API call: https:serverName/api/SyncPersonnelViaAwsApi/SapEaiCall
Hits the API function and returns the correct message 'Not latest version of file, update not performed'
However the lambda says it has failed.
Response:
{
"errorMessage": "\"{'response' : 'Not latest version of file, update not performed'}\""
}
this is all the data that is given in the logs...but this is the correct message postback...does anyone have any idea why it is still flagging as a fail?
(code below)
//// POST api/<controller>
public string SapEaiCall([FromBody]string xmlFile)
{
string responseMsg = "Failed Import Active Directory User";
if (string.IsNullOrEmpty(xmlFile))
{
responseMsg = "XML file is NULL";
}
if (responseMsg != "XML file is NULL")
{
xmlFile = RemoveFirstAndLastQuotes(xmlFile);
if (!IsNewestVersionOfXMLFile(xmlFile))
{
responseMsg = "Not latest version of file, update not performed";
}
else
{
Business.PersonnelReplicate personnelReplicate = BusinessLogic.SynchronisePersonnel.BuildFromDataContractXml<Business.PersonnelReplicate>(xmlFile);
bool result = Service.Personnel.SynchroniseCache(personnelReplicate);
if (result)
{
responseMsg = "Success Import Sap Cache User";
}
}
}
return "{\"response\" : \" " + responseMsg + " \" , \"isNewActiveDirectoryUser\" : \" false \"}";
}
(lambda:)
var querystring = require('querystring');
var https = require('https');
var fs = require('fs');
exports.handler = function(event, context) {
const post_data = JSON.stringify('="xmlData"');
// An object of options to indicate where to post to
var post_options = {
host: 'ServerName',
protocol: 'https:',
// port: '443',
path: '/api/SyncPersonnelViaAwsApi/SapEaiCall',
method: 'POST',
json:post_data,
headers: {
'Content-Type': 'application/json',
'Content-Length': post_data.length
}
};
process.env.NODE_TLS_REJECT_UNAUTHORIZED = "0";
var post_request = https.request(post_options, function(res) {
var body = "";
res.on('data', function(chunk) {
//chunk = '456';
body += chunk;
});
res.on('end', function() {
context.done(body);
});
res.on('error', function(e) {
context.fail('error:' + e.message);
});
});
// post the data
post_request.write(post_data);
post_request.end();
console.log("posted data " +post_data);
};
context.done() takes two parameters. The first one is the error object and the second is response object on success. Change
res.on('end', function() {
context.done(body);
});
to
res.on('end', function() {
context.done(null, body);
});
Thanks for looking into the code.
Here I am fetching some data using feed parser and taking out id's in navcodes array variable and wants to use these Id to make http call.Please find code below.
function processNavCode(){
var mfId = [53];
var preTitle = '';
var navCodes = [];
mfId.forEach(function(id){
var query = "http://portal.xyz.com/Rss.aspx?mf="+id;
feed(query, function(err, feeds) {
if (err) {
throw err;
}
feeds.forEach(function(feed){
var link = feed.link;
var title = feed.title;
var navCode = link.substr(link.length - 6);
if(title.split('-')[0].trim() != preTitle){
preTitle = title;
counter ++;
}
if(parseInt(navCode) != '')
navCodes.push = parseInt(navCode);
});
});
async.eachSeries(navCodes,insertbulkMFValues,function(){
console.log('I am done');
});
// insertbulkMFValues(navCode);
//Directly call insertbulkMFValues function
});
}
I have also tried to call the insertbulkMFValues directly as commented now but due to async nature of nodejs, I am getting the error of either 'Socket hang up' or 'read ECONNRESET'. I checked and used async but not able to work with that also.
var insertbulkMFValues =function(navCode,callback){
var options = {
host: 'www.quandl.com',
path: '/api/v3/datasets/AMFI/'+navCode+'.json?api_key=123456789&start_date=2013-08-30',
method: 'GET',
headers: {
'Content-Type': 'application/json'
}
}
var req1 = https.request(options, function(response) {
var body = '';
response.setEncoding('utf8');
response.on('data', function(chunk) {
body += chunk;
});
response.on('end', function() {
if(typeof body === "string") {
var json = JSON.parse(body);
}
var mfData = json.dataset.data;
var schemeId = json.dataset.dataset_code;
var schemeName = json.dataset.name;
var isinCode = json.dataset.description;
var valueData=[];
for (var k = 0; k < mfData.length; k++) {
var myDate = new Date(mfData[k][0]);
valueData.push({
date:myDate,
NAV:parseFloat(mfData[k][1]).toFixed(2)
});
}
var query = { "navCode": schemeId };
var newData = {
createdDate: Date.now(),
navCode: schemeId,
schemeCode:count,
schemeName:schemeName,
ISINCode:isinCode,
values:valueData
};
MHistory.findOneAndUpdate(query, newData , {upsert:true}, function(err, doc){
if(err)
console.log('Errorr');
else
console.log('Success');
});
});
});
req1.on('error', function(e) {
console.log('problem with request: ' + e.message);
callback(true);
});
req1.end();
}
Thanks in advance..
J
You can directly call insertbulkMFValues for each navCode like:
if(parseInt(navCode) != '') {
insertbulkMFValues(navCode, function () {
console.log('something'}
});
}
Anything that you intend to do must be within the callback of the asynchronous function.
One option for you is to use the waterfall or parallel method of the async library to retrieve all feeds for each id and then invoke
async.eachSeries(navCodesAccumulated,insertbulkMFValues,function(){
console.log('I am done');
});
within the final result callback using the codes obtained.