I am using python pyramid and find out if one try to access self.request, the whole 2G file will be loaded to a temporary folder automatically. Instead, I would like to lazy load the file and pass the file pointer to another method where it can be uploaded to other server. Then I can save the time to load to disk twice.
Or Are there any recommended library(other than waitress) I can move to?
Related
I'm creating a local data store in node js and it will create a json file in a specific location
I want that file to be used by only one client at any given time if another client tries to access the same file then it should throw error, everything happens locally in the computer
Sounds like you want to implement a lock. If you don't know what it is I'd recommend reading about it.
If it's just a pet project, create an object that's responsible for opening and closing a file. It can have some internal state that indicates if file is currently used.
I have a config file in an S3 bucket that needs to be modified by adding the response from ec2.describe_subnets and sent to an api endpoint. Is there a way to automate this process using Lambda to get the file from S3 without having to save it to a temp directory?
Is there a way to automate this process using Lambda to get the file from S3 without having to save it to a temp directory?
If you're asking about modifying the contents of the S3 Object, then the answer is no, it's not possible because S3 doesn't support that kind of operation. The only thing you can do is overwrite the entire object (ie, not just parts of it).
If you're asking about overwriting the S3 Object with new contents, then yes, it is possible to do it "without having to save it to a temp directory" if you do it in memory, for example.
Download the object from S3 without writing it to storage, make the changes in memory, and re-upload it to S3. If the file size is too big to fully store in memory, you can do it in a streaming fashion, too (i.e., initiate the download, for each chunk you download you make the necessary changes, and upload the modified chunk with a multipart upload, clear up the memory, repeat, etc).
As a final note, do keep in mind that S3 supports only eventual consistency for updates. This means that after you update the object, subsequent reads may still download the previous version. If whatever is consuming the file cannot properly deal with that, you'll probably need a different approach (i.e., don't overwrite, but write a new object with a new key, and send that new key to the consumer; or just use storage system that does support strong consistency, such as DynamoDB).
I have an angular application that has a feature to render the pdf in the client side with the help of ng2-pdfviewer, that reads the file from the local directory from the client side.
Now I have an upload option which uploads the pdf files to the server and storing it in local uploads directory in the server side. Now I want to read the physical pdf file available in the server instead of downloading as blob and saving.
Does anybody have any suggestions on how to do this?
ng2-pdf-viewer accepts both an UInt8Array (that fits your former approach on reading from a file utilising FileReader, for instance) or a string (that can contain a remote URL) on its src parameter.
So, as far as you have the complete URL for your target PDF, you can provide it as the source for rendering.
Every search result says something about storing the images in the file system but store the paths in the database, but I'm not sure exactly what "file system" means. Would that mean you have something like:
/public (assets)
/js
/css
/img
/app (frontend)
/server (backend)
and you'd upload directly to that /public/img directory?
I remember trying something like that in the past with a Node.js app hosted on Heroku, and it wouldn't let me. I had to set up Amazon S3 and upload the images THERE, which leads to my confusion.
Is using something like Amazon S3 the usual practice or do people upload directly to the /img directory (assuming this is the "file system"?) and it just happened to be the case that Heroku doesn't allow this but other hosts do?
I'd characterize the pattern as "store the data in a blob storage service, store a pointer in your database". The uploaded file is the "blob" - once it has left the user's computer and filesystem, is it really a file anymore? :) On the server, a file system can store that "blob". S3 can store that blob. In the first case, you are storing a path. In the second case, you are storing the URL to the S3 object. A database could even store that blob (not at all recommended, though...)
In any case, the question to ask is: "what happens when I need two app servers to support my traffic?". Wherever that blob goes, both app servers need access to it.
In a data center under your control, there are many ways to share a filesystem across servers - network attached storage (NFS- or SMB-mounted volumes), or storage area networks (iSCSI, Fibre Channel). With more limited network/hardware configuration options in cloud-based Infrastructure/Platform-as-a-Service providers, the de facto standard is S3 because it is inexpensive, reliable, easy to use, and can completely offload serving the file from your servers.
For Heroku, though, you don't have much control over the file system. And, know that the file system for each of your dynos is "ephemeral" - it goes away when the dyno restarts. Which will happen when your app goes idle, or every 24 hours, whichever comes first. So that forces the choice a little.
Final point - S3 comes with the ancillary benefit of taking the burden of serving the blob off of your servers. You can also store files directly to S3 from the browser, without routing it through your app (see https://devcenter.heroku.com/articles/s3-upload-node). The benefit in both cases is that those downloads/uploads can take up lots of your application's precious time for stuff that's pretty rote.
Uploading directly to a host file system is generally not a best practice. This is one reason services like S3 are so popular.
If you're using the host file system and ever need more than one instance of a server, the file systems will grow out of sync. Imagine one user uploads 'foo.jpg' to server A (A/app/uploads) and another uploads 'bar.jpg' to server B (B/app/uploads). When either of these images is later requested, the request has a 50% chance of failing, depending on whether the load balancer routes the request to server A or server B.
There are several ancillary benefits to avoiding the host filesystem. For instance, you can set the filesystem serving your app to read-only for increased security. Files are a form of state, and stateless web servers allow you to do things like blow away one instance and deploy another instance to take over its work.
You might find this of help:
https://codeforgeek.com/2014/11/file-uploads-using-node-js/
I used multer in my node.js server file to handle uploading from the front end. Basically I had an html form that would submit the image to the server file, where it would be handled by multer. This actually led it to be saved in the file system (to answer your question concretely, yes, this was to something like the /img directory right in your project file structure). My application is running on heroku, and this feature works on there as well. However, I would not recommending using the file system to store your image like this (I doubt you will have enough space for a large amount of images/files) - using AWS storage or a DB would be better.
I want to upload images from the client to the server. The client must see a list of all images he or she has and see the image itself (a thumbnail or something like that).
I saw people using two methods (generically speaking)
1- Upload image and save the binaries to MongoDB
2- Upload an image and move it to a folder, save the path somewhere (the classic method, and the one I implemented so far)
What are the pros and cons of each method and how can I retrieve the data and show it in a template in each case (getting the path and writing to the src attribute of and img tag and sending the binaries) ?
Problems found so far: when I request foo.jpg (localhost:3000/uploads/foo.jpg) that I uploaded and the server moved to a known folder, my router (iron router) fails to find how to deal with the request.
1- Upload image and save the binaries to MongoDB
Either you limit the file size to 16MB and use only basic mongoDB, either you use gridFS and can store anything (no size limit). There are several pro-cons of using this method, but IMHO it is much better than storing on the file system :
Files don't touch your file system, they are piped to you database
You get back all the benefits of mongo and you can scale up without worries
Files are chunked and you can only send a specific byte range (useful for streaming or download resuming)
Files are accessed like any other mongo document, so you can use the allow/deny function, pub/sub, etc.
2- Upload an image and move it to a folder, save the path somewhere
(the classic method, and the one I implemented so far)
In this case, either you store everything in your public folder and make everything publicly accessible using the files names + paths, either you use dedicated asset delivery system, such as an ngix server. Either way, you will be using something less secure and maintenable than the first option.
This being said, have a look at the file collection package. It is much simpler than collection-fs and will offer you everything you are looking for out of the box (including a file api, gridFS storing, resumable uploads, and many other things).
Problems found so far: when I request foo.jpg
(localhost:3000/uploads/foo.jpg) that I uploaded and the server moved
to a known folder, my router (iron router) fails to find how to deal
with the request.
Do you know this path leads to your root folder public/uploads/foo.jpg directory? If you put it there, you should be able to request it.