I am kind of asking about best practice here, since I've looked everywhere but can't find any. I'm creating a Solr-based application using NodeJS, particularly solr-client to connect to the Solr. My application is supossed to be able to receive files in multiple kind of formats (from json, xml, csv, etc, which is possible in Solr) and I am using NodeJS for that.
On terminal I can use command like bin/solr post -c core-name path_to_file to post any kind of format readable by Solr, but I have some trouble doing that in Javascript. The following code just doesn't work.
client.add(thisData, function(err, obj){
if(err){
console.log("Failed add file");
console.log(err.stack);
// throw err;
}else{
// console.log(obj);
console.log("Successful");
}
});
client.commit(function(err, res){
if(err){
console.log(err);
}
if(res){
console.log(res);
}
});
As a note, thisData is a variable containing the file content that I have obtained using fileSystem. I don't understand why I can post any kind of file format from terminal, but can't do the same from programming thing. Is it not allowed in Solr?
Some friends suggested me to convert all the file contents (from xls, csv, json, etc) to json then send the json to Solr. It works, but I realize that it's so much works. I have to use a lot of nodejs packages to convert those files and I don't find them really efficient. I have been looking for the best way to do this on github but seems like I found nothing to reference to. Can someone give me some hints? I am sorry if I am not being really clear, I will explain more if you need me to.
Related
How do I work with node modules?
tldr; How do I look at a node module I've installed and know where to go and what I'm looking for
If I use npm i googleapis for example, it downloads the node module for Googles APIs but how do I browse the module and work out what's useful for me?
To try and eliminate any ambiguity from the question, I'll use this use case.
I'm developing a Discord bot and I want to add statistics to one of
the commands. Here is the supplied code from Google:
<script src="https://apis.google.com/js/api.js"></script>
<script>
/**
* Sample JavaScript code for youtube.channels.list
* See instructions for running APIs Explorer code samples locally:
* https://developers.google.com/explorer-help/code-samples#javascript
*/
function authenticate() {
return gapi.auth2.getAuthInstance()
.signIn({scope: "https://www.googleapis.com/auth/youtube.readonly"})
.then(function() { console.log("Sign-in successful"); },
function(err) { console.error("Error signing in", err); });
}
function loadClient() {
gapi.client.setApiKey("YOUR_API_KEY");
return gapi.client.load("https://www.googleapis.com/discovery/v1/apis/youtube/v3/rest")
.then(function() { console.log("GAPI client loaded for API"); },
function(err) { console.error("Error loading GAPI client for API", err); });
}
// Make sure the client is loaded and sign-in is complete before calling this method.
function execute() {
return gapi.client.youtube.channels.list({
"part": [
"snippet,contentDetails,statistics"
],
"id": [
"UC_x5XG1OV2P6uZZ5FSM9Ttw"
]
})
.then(function(response) {
// Handle the results here (response.result has the parsed body).
console.log("Response", response);
},
function(err) { console.error("Execute error", err); });
}
gapi.load("client:auth2", function() {
gapi.auth2.init({client_id: "YOUR_CLIENT_ID"});
});
</script>
<button onclick="authenticate().then(loadClient)">authorize and load</button>
<button onclick="execute()">execute</button>
Now Google offers a supported library which means I can replace the external script tags and import from the node package, specifically the parts I need.
So I'll need to import or require whatever gives me access to things like:
gapi.auth2.getAuthInstance
gapi.client.setApiKey
gapi.client.youtube.channels.list
For someone who is new to nodejs, instead of copy and pasting from every piece of documentation and hoping it works, how do I comfortably look at a node package and find the things I need and can use?
Edit #1
I think my use of the google apis case threw off the direction of the question and changed the scope of what I asked so I'm going to correct the best I can.
The assumption should be made that there is no documentation on the package whether it's so poorly written, doesn't exist at all or the documentation is down for an extended period of time during a time sensitive development.
At that point, is there any possible way to look at the node_modules folder, the specific package that needs to be worked with and work out what's going on? Is there any way to look at the structure of a package and recognise "well most likely what I need is in this folder or file"
That's what documentation is for.
When someone writes an API for a package; to make things clearer for the consumer, he should document the exported functions well enough.
The best way to get the documentations for node packages is to search for the package at www.npmjs.com .
For google apis you can go to that page here to see the "get started" and some examples. And you can go here to see the full detailed APIs of that package.
Answering Edit #1
Well, in that case, it could be a difficult task, depends on the how structured and organized the package is.
Since we are talking about nodejs, you should look for the package.json file and search for the path of the main file "main": "<PATH HERE>".
Then, you can go to that main file and try to locate what exactly is being exported. You can search for module.exports or the export keyword.
Everything that is explicitly exported is intended to be used as an API.
I'm not familiar with any other way other than go deeper in the package's files and identify what exactly is being exported.
I'm developing a bot, which has a suggestion feature.
You can type !suggest <suggestion> and then the bot outputs an embed with the suggestion in a channel - that works and I like it, but I also want it to log it into either a .txt file or a .json file, but I cant figure out how to do that.
Can anybody help me?
like Thecave3 said you should prob read docs and learn nodejs before you use a library like discord.js
https://nodejs.org/api/fs.html
https://nodejs.org/api/fs.html#fs_fs_writefile_file_data_options_callback
but anyways this would be the code
fs.writeFile(path, JSON.stringify(object), (err) => {
if(err) console.error(err);
});
I am a PHP developer trying her hands on node.js. The introductory books about it and online tutorials are great and are helping me come to speed with this language
however I find the official site nodejs.org documentation hard to drill through for somewhat reasons I don't understand, may be I am just spoiled by php.net.
For example many asynchronous functions take a callback function with arguments and docs document that fact but the types of arguments these callbacks take are barely (not) documented
See below
fs.readFile('./template.html', function(err, data) {
if (err) {
console.error(err);
res.end('Server Error');
}
else {
var tmpl = data.toString();
}
});
Here the data argument appears to be an object with toString method and that's all I know about it.
Old langange users can you please point or guide me on how to get the most out of this nice langange, it can be anything such as how to read the docs.
Thank you.
The docs for node.js seem clear to me.
The callback is passed two arguments (err, data), where data is the contents of the file.
I am saving files as Buffers in my mongo database (using mongoose, nodejs, electron). For now, I'm keeping it simple with text-only files. I read a file in using
fs.readFile(filePath, function(err, data) {
if (err) {console.log(err);}
typeof callback == "function" && callback(data);
});
Then I create a new file in my database using the data variable. And, now I have something that looks like BinData(0,"SGVsbG8gV29ybGQK") stored in my mongodb. All is fine so far.
Now, what if I wanted to display that file in the UI? In this case, in Electron? I think there are two steps.
Step 1 The first is bringing this variable out of the DB and into the front-end. FYI: The model is called File and the variable that stores the file contents is called content.
So, I've tried File.content.toString() which gives me Object {type: "Buffer", data: Array[7]} which is not the string I'm expecting. What is happening here? I read here that this should work.
Step 2 Display this file. Now, since I'm only using text files right now, I can just display the string I get once Step 1 is working. But, is there a good way to do this for more complex files? Images and GIFs and such?
You should save the file mime.
And then set response.header MIME
response.setHeader("Content-Type", "image/jpeg");
response.write(binary)
response.end()
No Matter how much I've searched and changed and toyed over the last 24 hours, I simply can not find the right combination of settings that allows Node.js to upload multiple files.
My setup is quite simple - I have a form interface that posts multipart content (files) to an endpoint.. lets call it: /theendpoint
and this end point is supposed to parse the multiple files. However, during the parsing, there are various events that need to be called once the file is uploaded.
I'm currently using the express.bodyParser({ uploadDir:'/tmp', keepextensions:true, defer:true}); in the app configuration.
Using the following method, I am trying to parse the file, but the problem is
Only 2 files will begin uploading, and will not complete (ie. the progress bar hangs near the end without fully completing).
The other files to be uploaded by the form (item 3+) do not even begin to upload to the server.
It seems to be some sort of asynchronus holdup, however I can't properly interpret the problem. Some of the code used at the upload endpoint are as follows:
// This applies to /theendpoint route. Using Express.
exports.theendpoint = function(req,res){
console.log(req.files);
fs.readfile(uploadPath, function(err,data){
if(err) throw err;
fs.writeFile(newFilePath, data, function(err){
// Series of checks and definitions
// Database connection
// Conditional executions
fs.unlink(req.files.file.path, function(err){
if(err) throw err;
console.log('Deleted');
});
});
});
};
Obviously I've left out some of the code here. If anyone can help - is this structure workable?
You should know that items in the commented section.. ie DB connection etc. are asynchronus tasks.
after
fs.unlink(req.files.file.path, function(err){
if(err) throw err;
console.log('Deleted');
});
add
res.redirect("back");
and it works!