I am pretty new to MapR and I have a task about creating a MapR volume. I used this command
maprcli volume create -name test-volume -path /MyCluster/apps/application_logs/node1
to create the volume but I get this error:
ERROR (2) - Failed to mount test-volume, parent of mount dir /MyCluster/apps/application_logs/node1 does not exist, No such file or directory
Before running the command I created this path '/MyCluster/apps/application_logs/node1' using mkdir :
Does anyone know why I get this error and if someone can help me with fixing it?
Thank you.
When you say;
I created this path '/MyCluster/apps/application_logs/node1' using mkdir
I assume that you have created it in local file system. The path parameter needs to be created in MapR file system.
To do that, you can either create the dir by interacting with MapR file system directly as follows;
hadoop fs -mkdir -p <path>
or you can mount MapR fs to local file system so that you can interact with it from mount point; /mapr/<cluster-name>/
From the doc;
Parameter: path
Default Value: No default value
Possible Values: Any valid path.
Description: The path at which to mount the volume. The path must be relative to / and cannot be in the form of a global namespace path (for example, /mapr/<cluster-name>/).
Related
I am trying to merge two files from my HDFS to a folder on my local machine's desktop. The command that I am using is:
hadoop fs -getmerge -nl /user/hadoop/folder_name/ /Desktop/test_files/finalfile.csv
But that returns the following error:
getmerge: Mkdirs failed to create file:/Desktop/test_files (exists=false, cwd=file:/home/hadoop)
Does anyone know why this might be? I couldn't find much of anything else in my search.
You need to create the local folder /Desktop/test_files/
I have a Node-RED app running in a docker container, with the aim to periodically read contents of a directory where .csv files are constantly updated and new .csv files are sometimes added. The point is to read new entries periodically, parse data, and send it onward.
I have not utilized the numerous 'contrib' nodes, as I have enabled the NodeJS 'fs' module and played with it. Additionally the built-in 'file' and 'file in' Node-RED modules are useful when reading the .csv files' contents, so that is not an issue.
The problem comes with the new .csv files being added into the directory where all the .csv files are. I want be able to read all the file names and subsequently read all the .csv files.
I have mounted the .csv file directory into the docker container, and when testing whether I'm able to read the file names, weird things happen. Even though the files are visible in the container (viewed using docker exec -it CONTAINER /bin/bash) a piece of code containing fs.readdir does not list the files. When I try the fs.readdir too see the contents of /data directory, which is mounted into the container, it lists the contents like 10 % of the time (injecting a timestamp into the node to run it)
As you can see from the image, the contents of the directorty in question are not listed on every execution of the node. The contents of the mounted directory containing the .csv files are never listed upon running this node with the correct path as parameter.
The operating system is CentOS 7, where I am not a sudoer. I have managed to make it so that none of the mounted files or directories are owned by root, so they are owned by user node-red within the container. I managed to pull this directory file listing through on my ubuntu where I am a sudoer, but as none of the stuff is root-owned there either, I am not sure if that is the problem. I have a feeling this might be an operating system -relating thing.
Notes:
All relevant files and directories have permissions rwxr-xr-x
I have tried to mount the .csv files containing directory under /data directory, and as its own directory directly under root as /files
I am able to read the file contents with the Node-RED file nodes, just not the directories. Reading static file names is not enough as the directory contents keep changing
I have enabled NodeJS 'fs' module from the settings.js file which is mounted into the container
The Node-RED node (in image) does not output any errors (I tried this by adding an error return to the function in the image)
I have tried to run the Node-RED container as root user and without defining the user
I am running the Node-RED container using docker-compose
I hope this was not too much text or too unclear, I just wanted to make sure at least most of the stuff I have tried would be written here. If someone has some insight on the workings of Node-RED under docker and using the NodeJS fs module, it would be most appreciated :)
The core Watch node should do all of this for you, no need to write function nodes.
If you want walk subdirectories make sure you tick the right box in the config.
From the Sidebar docs for the watch node:
The full filename of the file that actually changed is put into
msg.payload and msg.filename, while a stringified version of the watch
list is returned in msg.topic.
msg.file contains just the short filename of the file that changed.
msg.type has the type of thing changed, usually file or directory,
while msg.size holds the file size in bytes.
To answer my question of why Node-RED was unable to read directory contents most of the time, it was because of using the asynchronous fs.readdir module. When I switched to using the synchronous version fs.readdirSync, Node-RED was able to read directory contents without problems.
I am having an issue trying to use Capistrano to deploy an application that requires having several Amazon EFS bind mounts inside of the deployment (current) folder.
I have a directory on the webserver in the root called /webroot inside of it is where all of our code currently is along with about 7 folders (bind mounts) that are shared across three nodes.
Inside of my deploy.rb I have the following line set :deploy_to, "/webroot/testingCap" in which Capistrano is deploying the code into the symlinked folder current. This is great but now when it gets to the step of symlinking the bind mount directories for example:/webroot/uploads it throws an error:
rm -rf /webroot/uploads
rm: cannot remove '/webroot/uploads'
Device or resource busy
I am not sure why it is trying to forcefully remove that directory? I thought it was supposed to just symlink to the directory.
My linked_dirs part looks like this inside of deploy.rb:
append :linked_dirs, "/webroot/uploads"
What am I doing wrong?
:linked_dirs only works with relative paths and always uses Capistrano's shared directory.
When you add e.g. "foo" to :linked_dirs, Capistrano will create a symlink within your deployed app. If anything already exists there, it will delete it first (that is why you are seeing the rm -rf).
The destination of that link will always be to the same name in Capistrano's shared directory. So the chain of events will be like this:
rm -rf /webroot/testingCap/current/foo
ln -s /webroot/testingCap/shared/foo /webroot/testingCap/current/foo
Thus if you look inside current, you will a link that points
foo -> /webroot/testingCap/shared/foo
Notice that the path relative to current is identical to the path relative to shared. This is how :linked_dirs works and you can't change it.
For example, if your app expects to store uploads in public/uploads, you will need the exact same relative path to exist inside shared in order for the link to be established. In other words, the link will point like this:
/webroot/testingCap/current/public/uploads -> /webroot/testingCap/shared/public/uploads
In your case, I suspect you can get this to work, but you'll need to make sure that your mount points are located exactly where Capistrano expects them to be.
server is running on centOS6.5 with ext4 file system
We have a problem with create a new dir in specific folder (for eg. /var/www/html/img/) when we are trying use mkdir,cp,scp, from another server,create with winSCP, and php script it's type "mk dir cant create folder 'name' invalid argument
when we was trying create something "upper" (for eg. /var/www/html) everything was created correct
have somebody some idea how to fix this problem?
I run my build stored in linux server path eg user/aaditya/builds/build1/bin/ i have written an api which takes linux mount path as input
for example API- takeimage,..//xyz.bmp
when i run it capture the image and stored in build1 folder but when i execute takeimage,..//..//build2/ , no image is stored in build2 folder
where path to build2 is user/aaditya/builds/build2/ how to mount user/aaditya/builds/build2/
because API uses the linux system call to write the file
In your two examples you are running your API like this:
takeimage ..//zyx.bmp
takeimage ..//..//build2/
I am guessing you run this from user/aaditya/builds/build1/bin/
In your first example you are providing your API the name of a file to save your image in. In your second example you are only providing a directory path. If the Build2 directory does not exist it could be producing an error.