I have a script that writes files to disk using fs createWriteStream.
What I am trying to achieve now is write those files to a shared network drive.
With a directory like so - //hostname/scratch/reece
I am running this script on windows, but this application will sit on ubnutu/rhel when I deploy it.
This is a crucial part of this script so any suggestions on how I can write to a network drive would be great.
The same would go for reading from a network drive and sending that back via HTTP.
Keeping in mind there would likely be hundreds of thousands of requests to write to this drive through my nodejs api, so I would like to avoid relying on background processes to handle the file transfer.
Any ideas on approach?
You will have to connect the drive to your server using a technology appropriate for that particular OS (may be different on Ubuntu vs. Windows). You can then address that server through whatever OS mount tech it uses.
In Windows, you can use either a drive letter or a UNC path. On Ubuntu, perhaps a mounted network file system volume.
This is one case where you aren't likely to make the exact same setup work on Windows vs. Ubuntu. If you put the appropriate root path name into a config file, the rest of your code can probably be identical. Beyond this, it isn't clear what you're asking.
Related
The problem is that I need to share files between 2 programs, but I don't want that those files are accessible by the user of the computer and other programs than these 2. So the flow of the files are like this: Program A (which I will code myself) recieves a file from the internet and puts somewhere on the computer. Then Program A calls Program B (which I didn't code and can't change). Program B reads the downloaded file and does some things with it and produces another file which Program B puts also somewhere on the computer. Then Program A reads that file and uploads it to the internet.
What I have found
I thought that maybe Windows Sandbox was interesting, but the problem with Windows Sandbox is that it's only available to windows 10 pro and windows 11, and that it is virtualised, and performance is quite important for Program B... So any virtualised software is not very usable, unless it is close to native performance.
For Linux, I found FreeBSD jails. But this seems more focussed on keeping the applications in the jail prohibited to access files outside the jail than to prohibit the programs outside the jail from reading and writing to files in the jail. So actually I need the opposite...
Another interesting concept was to keep the files stored in RAM like mmap in Linux, but since I can't change Program B, I don't know how to implement that. Is there some kind of container application that encapsulates the IO of Program B and redirects it to a file in RAM?
Does anyone have some suggestions? Thanks!
You can't really prevent the user/owner of the computer from reading the file if you are storing it on their disk. You can try to make it more difficult to access the content (which is what DRM does) but ultimately you the user can always bypass your controls given sufficient motivation and resources. Even if you store the files purely in RAM, a user with administrative permissions can dump your program's memory, and extract the files from there.
First of all I am a beginner with node.js.
In node.js when I use functions such as fs.writeFile(); the file is created and is visible in my repository. But when this same process is done on a cloud such as heroku no file is visible in the repository(cloned via git). I know the file is being made because I am able to read it but I cannot view it. Why is this??? Plus how can I view the file?
I had the same issue, and found out that Heroku and other cloud services generally prefer that you don't write in their file system; everything you write/save will be store in "ephemeral filesystem", it's like a ghost file system really.
Usually you would want to use Amazon S3 or reddis for json files etc, and other bigger ones like mp3.
I think it will work if you rent a remote server, like ECS, with a linux system, and a mounted storage space, then this might work.
I'd like to ideally be able to do
echo "1" > file
and that file is being monitored or maybe the file itself is an app that just forwards all contents that's about to be written to another server or source.
Any idea how I can do this or where I should start looking? I tried a few searches online but came up short
Use a network file system like NFS or smbfs or any other network enabled FS. You may consider using FUSE (Filesystem in USErspace) if you don't want to use root account to be able to perform mounts. FUSE also has a bunch of "modules" providing access for different "file-like" network protocols, like sshfs, ftpfs, webdav etc
I have two scenarios.
Scenario 1: Machine A contains 1000 documents as folders. This folder of machine A is mounted in machine B. I process documents within these folders in machine B and store the output result in mounted path in machine B.
Scenario 2: The documents in machine A is directly copied into machine B and processed
Scenario 2 is much faster than Scenario 1. I could guess its because there is no data transfer happening over the network between 2 machines. Is there a way I can use mounting and still achieve better performance?
Did you try enabling a cache? - for NFS: https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Storage_Administration_Guide/fscachenfs.html - CIFS should have caching enabled by default (unless you disabled it)
The other option would be to use something like Windows’ offline files, which copies files and folders between client and server in the background, so you don’t need to deal with it. The only thing I’ve found for linux is OFS.
But the performance depends on the size of the files and if you read them randomly or sequentially. For instance when I am encoding videos, I access the file right away via the network from my NFS, because it takes as much time as it would take to read and write the file. This way no additional time is “wasted” on the encoding, as the application can encode the stream which is coming from the network.
So for large files you might want to change the algorithms to a sequential read, on the other hand small files which are copied within seconds, could be also synced between server and client using rsync, bittorrent sync, dropbox or one of the other hundreds of tools. And this is actually quite commonly done.
I bought a new server and I want to move all the data (directories, sub directories, users, passwords, ..etc) from my old server to it.
Is there a way to do that?
Thanks,
Do you have physical access to both servers? If so you can use the dd command to make a clone of the disk from the old server to the disk that is going into the new server.
In order to do this though, both hard drives have to be installed in one of the servers.
You can also use netcat and dd to clone a disk over a network.
for the directories and files, use a FTP client from your server, if it allows you to, if not, just download all the content to your computer and upload it to the new server.
For the users and passwords, i guess they are in a Database, connect to the database using SSH, telnet, or MysqlAdmin or any RMDB client system and export a dump file, then log in to the new server's SQL system and import that dump file.
Anyway you should give more details of both servers anyway so we can help you, for example, are they Shared hosting or dedicated machine? and what kind of access do you have to them, also, their operative system would help people to reply you accurately
In principle, yes.
If the hardware is similar (= just more RAM, disk space but same CPU architecture and no special graphics card drivers), you might be able to copy every file and then install the boot loader once more (the boot loader config usually changes when the hard disk size changes).
Or you can create a list of all services that you use, determine which config files each one uses and then just copy those. Ideally, you shouldn't copy them but compare the old and the new versions and merge them.
The most work intensive way is to use a tool like puppet. In a nutshell, puppet allows to create install scripts for services (along with all the configuration that you need). So if you need to install a service again (new hardware, second server), you just tell puppet to do it. On the plus side, your whole installation will be documented, too. If you ever wonder why something is the way it is, you can look into the puppet files.
Of course, this approach takes a lot of time and discipline, so it might not be worth it in your case. Apply common sense.