I have a django application that I use to visualize data. The data itself is in an Amazon S3 bucket, and the plotting of the data is done using bokeh. One of the applications however, uses d3 instead. And while I managed to configure the bokeh apps to work properly, I don't know how to do that with the d3 app. It has to read data either directly from S3 or locally (If I download it within django views before rendering the web page) and plot it. But whatever I try I get a 404 Not Found error.
I'm learning as I go, so I don't know what I'm doing wrong. Going through SO, I found this question which gives an example of downloading a file from S3 using Node.js, but I am already running a django server so I don't know if that works. I should also mention that the files to be read are quite large (several megabytes). So, to summarize, my question is:
Is using Node.js my only solution here and can it be done without having both Nodejs and django running at the same time? I'm worried that this might be too complex for me to set up. Or better yet, what would be a recommended solution in my case? I am almost done with the whole project but, unfortunately, I've gotten stuck pretty bad here.
Thank you to anyone willing to help or offer advice.
Related
This is unfamiliar ground for me and I've been poking around with resources trying to find a suitable solution. But essentially, I have a website setup that I want to allow users to upload MP3/FLAC files. Then I want to take those files and send them to a Google Cloud Bucket. The second part seems easy enough, plenty of NodeJS tutorials regarding that.
Since I'm pretty in the dark with how this is done, would I need to "upload" a file on my frontend and then hit my node-express api backend with some sort of fs. solution that looks up the file on my machine? If so, how could that be consistent between users, what if their directory structure is different on their machines?
Anyway, kinda shooting in the dark here. Would love to have some advice regarding this.
It's not really feasible for a backend to "reach into" a frontend machine to pull files from it. The client needs to provide the data directly to the backend.
Mostly commonly, Firebase client libraries are used to directly upload contents from a client machine to a storage bucket. If you don't do that, you'll need to create your own backend API that clients can invoke to send data.
I need to write own video hosting with player on client side.
My required approaches:
The user can upload video to hosting
The user can watch any video from hosting
I don't ask to write me solution, I am asking for help where should I start from to lean about it? Which technologies or frameworks should I learn for my task to realize it using python?
P.S. Each detail will be very useful, especially some links to articles because I couldn't find by myself not knowing accurately what do I need to search.
Added
Now, I think to store videos in the file system directly and use postgresql to store additional information about videos and users. Of course, large services use Hadoop, BigTable and etc but for my task so solution will be enough I think.
When the user uploaded a new video, my server saves it into a temporary directory and puts in the processing queue. Small programs takes new videos one by one, generates thumbnails and decrease a quality of videos and moves it to the base storage. Is it a good idea?
But I still can't get how make a video streaming
Ok So I dont want to encourage the behavior of people thinking SO is a codewriting service, But this is a truly legitimate answer. So first of all, you want to choose a language. Currently I'd recommend the use of javascript and Node.js (Java needs to die). However, IDK Node as well as I know Python. Python is an all purpose language yadda yadda yadda blah blah blah. Whats important in this case is your framework of choice (or library). The libraries that allow you to make websites in python (or make it easier to do so) are very interesting. There are several but my favorite is Python Flask. Python flask is actually very similar to Node.js + Express.js. Use this link to get started. Take a few days to learn the insides of this framework. VERY MODULAR, VERY POWERFUL. Using basing logic and database knowledge, one can easily accomplish a simple file upload and authentication-using, web service. However I Know of 2 really good guides that will help you with the streaming of the videos. I mean yes, You don't really need to know this. You could potentially load the requested video using a <video> tag, but streaming is a much MUCH
more favorable solution. Take some time to learn about video streaming and compression, and after you think about it, check out these links: AUDIO STREAMING GIST and MIGUEL GRINBERG FLASK VIDEO STREAMING BLOG POST
Good luck with flask and
Pro tip: learn about http(s) and the get and post methods
You would never imagine how many times I struggled with a bad request error or a method not allowed because I didn't do my research
Hi I am very interested in machine learning.
familiar with keras and tensorflow and know how to train model and save it.
I want to build node.js(loopback) web application that provides machine learning API.
I'm quite new to web application and don't know how to utilize python code in the node app.
To be specific, I'm trying to make deep style application like this site for practice purpose.
For codes that is open to public in github. most starred implementation is in torch and
I am currently looking (TF).
It has script work as "main.py" that consists of
1) loading pretrained VGG model(about 500Mb) to memory.
2) and using each framework's function, it create output photo inside folder
So I want to run this with GPU instance in AWS
problem is, I don't know how to call "main.py" inside javascript.
Q1 How did many machine learning based services achieved this??
did they implement their server in python like django to load trained model and inference from it?
But I want to integrate in node.js especially loopback. how can I achieve this in a most efficient and common way??
Q2 Is calling "main.py" efficient whenever there's request to server?
I mean, If the script is called everytime request is made, It loads 500Mb object to memory all the time and that's so inefficient.
and as far as I know AWS GPU instance can utilize 4 GPU at a time. I wonder how can I utilize all of it with most efficient and common way...
Thank you for reading!
I'm too frustrated with myself... I really want to know method deploy my model to server
and thank you in advance
You can communicate between NodeJS and e.g. Django with a REST interface. Deploy a Django server with the REST framework and call an url from NodeJS. This way, you can trigger the python code. You can also let the python code call your NodeJS with another REST interface on the javascript side if that is needed.
What are the pros and cons of these two approaches to store and retrieve JSON data in Meteor?
Storing and retrieving are must be done in chunks.
We used on our server GridFS for a while before switching to AWS-S3 a few weeks ago. I can't really say what went wrong back then but every once in a while, an uploaded file would cause some error on the server and crash it randomly, despite the fact we enforced size and mime limitations. Also, when trying to retrieve and visualise those files, the server would start screaming and some crashed while uploading files would just be pointing to thewebsite.url/uploaded-docs/null. There is this note on the README:
NOTE: This package is under active development right now (2014-3-31).
It has bugs and the API may continue to change. Please help test it
and fix bugs, but don't use in production yet.
Because of that, we now use AWS and despite the fact it's not so free, we didn't encounter any other crash on file upload from our users. But as I said, the package note is pretty old and we couldn't really understand if the issue was on our end or on GridFS. I'd be interested if any other have some insights on this matter.
We have a Heroku NodeJS application and have been unable to run our data migrations successfully. We're using MongoDB and continuously encountering problems with legacy data sets from old data models not being converted properly.
I'm tempted to start building and managing a library of scripts, but don't think this is an ideal solution.
In a short past life with Ruby, I remember the rake commands being incredibly easy. Ideally, I'd love to use something like that. Does anyone know the right way to do this?
Any help is appreciated!