I'm making an service that will allow users to post files to my web server which will then copy that file (after a few checks) to the image server. The main way of communicating between my web server and my image server will be scp. However, I also want to maintain user filenames, so it would look like this:
User posts their file to web server
Web server checks if the file is supported
Web server checks if file is under file size limit
Web server says OK and tries to send to image server
Web server runs ("scp " + filepath + " root#imageServer:~/images")
Image server receives the file and is ready to send the file to user on request (the folder is public and will be served by nginx)
the dangerous part here is the scp command. I'm not an expert on security, but is there a way that this command can get hijacked the same way a database can get SQL injection? What if somebody named their file to be malicious. Is there a way to safely join the filename to the script? To safely "escape" the command?
I'm using express (node.js) for the web server. Is there another way to send files from the web server to a simple Ubuntu install without unix commands or writing up a REST api for the image server? Is there is, then I might not need to "escape" at all
Btw, the reason why I'm choosing to have the image server and the web server separate is because I want to scale the application in the future. For example, if there were 10 web servers and no central image server, then it would be impossible to retrieve files if the file isn't on the web server you request from.
You can run an external command without a shell (and therefore without issues with shell metacharacters) using child_process.spawn (or other methods in child_process). (Obviously, you must not specify the shell option as anything other than the default false.)
That lets you not worry about metacharacters in the filepath, but it seems to me that there are plenty of other issues with letting the user provide a filepath name to be used as such on a live filesystem. Personally, I'd autogenerate safe, short names and keep the correspondence from user name to filesystem name in a database somewhere.
Related
To render on threejs, we need some images(jpg/png) and , jsons(uv data). All these files are stored in respective folders and the files visible for clients to look at.
I use django/python to start a local server, python code is compiled to .pyc & js code is obfuscated. But the folder structure is accessible for Casual Users. In threejs, we use tex_loader and json_loader functions to which the file paths are given as inputs. Was looking at ways of securing the behind the scenes work.
Happened to read about custom binary formats, but that felt like a lot of work.
or giving access to files only for certain process starting through django/web browser?
Are there any available easy to deploy solutions to protect our IP ?
An option would be to only serve the files to authenticated users. This could be achieved by having an endpoint on your backend like:
api/assets/data.json
and the controller in the backend would receive the file name(data.json), the code could check if the user requesting the endpoint is authenticated and if so read the file from the file system(my-private-folder/assets/data.json) and return it as file with correct mime-type to the browser.
So basically I have a SQLite3 Database that I am using for a Discord Bot. Is it possible for me to access my database file to check what's inside while the application is hosted and live?
I've considered that it is probably possible to do this from the command line within my terminal that I use to access the virtual host, but I do not know where to begin/what to input.
Ideally I would want to be able to view the file inside the DB Browser for SQLite program, for visual purposes, but for starters I want to know if its possible to see the live-updated database remotely, as my application is hosted on a virtual host/machine.
You can install phpliteadmin (https://www.phpliteadmin.org/) on your server to view your database.
Otherwise your can download your database file and view it in a client (Example of online client: https://extendsclass.com/sqlite-browser.html)
I need to allow users to select directories locally (similar to the html file selector), with the path info uploaded to node.
I appreciate that sounds pointless, but it's to create a config file on the server that will match the client's requirements. So the user can select, say 'c:\documents' and 'd:\data\stuff' and have those paths passed as strings via http POST to the node (express) process.
How might I achieve this?
EDIT More info:
The data is for a client app, that works with the SaaS I'm working on. So the user (of which there will be many - I hope!) can make all their changes in one place, rather than having to configure settings in the portal, and then settings in the client GUI.
When the client app runs, it will pull the config (ie the directories it needs to process) from the server.
Every search result says something about storing the images in the file system but store the paths in the database, but I'm not sure exactly what "file system" means. Would that mean you have something like:
/public (assets)
/js
/css
/img
/app (frontend)
/server (backend)
and you'd upload directly to that /public/img directory?
I remember trying something like that in the past with a Node.js app hosted on Heroku, and it wouldn't let me. I had to set up Amazon S3 and upload the images THERE, which leads to my confusion.
Is using something like Amazon S3 the usual practice or do people upload directly to the /img directory (assuming this is the "file system"?) and it just happened to be the case that Heroku doesn't allow this but other hosts do?
I'd characterize the pattern as "store the data in a blob storage service, store a pointer in your database". The uploaded file is the "blob" - once it has left the user's computer and filesystem, is it really a file anymore? :) On the server, a file system can store that "blob". S3 can store that blob. In the first case, you are storing a path. In the second case, you are storing the URL to the S3 object. A database could even store that blob (not at all recommended, though...)
In any case, the question to ask is: "what happens when I need two app servers to support my traffic?". Wherever that blob goes, both app servers need access to it.
In a data center under your control, there are many ways to share a filesystem across servers - network attached storage (NFS- or SMB-mounted volumes), or storage area networks (iSCSI, Fibre Channel). With more limited network/hardware configuration options in cloud-based Infrastructure/Platform-as-a-Service providers, the de facto standard is S3 because it is inexpensive, reliable, easy to use, and can completely offload serving the file from your servers.
For Heroku, though, you don't have much control over the file system. And, know that the file system for each of your dynos is "ephemeral" - it goes away when the dyno restarts. Which will happen when your app goes idle, or every 24 hours, whichever comes first. So that forces the choice a little.
Final point - S3 comes with the ancillary benefit of taking the burden of serving the blob off of your servers. You can also store files directly to S3 from the browser, without routing it through your app (see https://devcenter.heroku.com/articles/s3-upload-node). The benefit in both cases is that those downloads/uploads can take up lots of your application's precious time for stuff that's pretty rote.
Uploading directly to a host file system is generally not a best practice. This is one reason services like S3 are so popular.
If you're using the host file system and ever need more than one instance of a server, the file systems will grow out of sync. Imagine one user uploads 'foo.jpg' to server A (A/app/uploads) and another uploads 'bar.jpg' to server B (B/app/uploads). When either of these images is later requested, the request has a 50% chance of failing, depending on whether the load balancer routes the request to server A or server B.
There are several ancillary benefits to avoiding the host filesystem. For instance, you can set the filesystem serving your app to read-only for increased security. Files are a form of state, and stateless web servers allow you to do things like blow away one instance and deploy another instance to take over its work.
You might find this of help:
https://codeforgeek.com/2014/11/file-uploads-using-node-js/
I used multer in my node.js server file to handle uploading from the front end. Basically I had an html form that would submit the image to the server file, where it would be handled by multer. This actually led it to be saved in the file system (to answer your question concretely, yes, this was to something like the /img directory right in your project file structure). My application is running on heroku, and this feature works on there as well. However, I would not recommending using the file system to store your image like this (I doubt you will have enough space for a large amount of images/files) - using AWS storage or a DB would be better.
I would like to know how to download data from server without knowing the server, or server must not contain any log of my downloading, Is there any way ??
Please Help.
In linux OR in any of the system, if you want to download something from server, you send a request to the server and the server sends some response back. Now,obviously every server at least keeps a log of what all actions are being implemented and what all responses are being generated. At the very minimal, all user history will be flushed at server in minimal server but,even then there is a possible chance that it'd keep a generated log of it's clients actions,although it depends more on the server ...
So, to be more clear, something like the data/log will be stored at the server about the downloading process SO THAT it keeps a track on your download and better serves you as a client.
There is nothing so wget command specific about it. From the linux.about.com's page entry about wget,
wget is a free utility for non-interactive download of files from the
Web. It supports HTTP, HTTPS, and FTP protocols, as well as retrieval
through HTTP proxies.
wget is non-interactive, meaning that it can work in the background,
while the user is not logged on. This allows you to start a retrieval
and disconnect from the system, letting wget finish the work. By
contrast, most of the Web browsers require constant user's presence,
which can be a great hindrance when transferring a lot of data.
wget can follow links in HTML pages and create local versions of
remote web sites, fully recreating the directory structure of the
original site. This is sometimes referred to as ``recursive
downloading.'' While doing that, wget respects the Robot Exclusion
Standard (/robots.txt). wget can be instructed to convert the links in
downloaded HTML files to the local files for offline viewing.