I am following the instructions to configure hadoop-2.0.0 cluster for installing Impala. In hdfs-site.xml, I add two properties "dfs.client.read.shortcircuit" and "dfs.domain.socket.path" (/var/lib/hadoop-hdfs/dn_socket).
But when I start the Hadoop cluster by start-dfs.sh, it fails to start datanodes. The log in datanode says that "failed to stat a path component: '/var/lib/hadoop-hdfs'". Then I create /var/lib/hadoop-hdfs manually, and start Hadoop cluster again. It fails again and log says that it's the permission problem about that directory. OK, fine. I change the owner of hadoop-hdfs from root to ubuntu (ubuntu is the machine username). Now it finally works normally.
I am just confused. Am I doing in the right way? Do we really need to create /var/lib/hadoop-hdfs by ourselves and change the permission or the owner of that directory? Or I missed some configuration setting?
I was running into similar problems using Cloudera Manager. It was an issue of trying to run in 'single-user mode' instead of using root. I think you are doing something similar with user ubuntu. Is this a clean install or are you upgrading / did you have a failed install last time?
I'm guessing you sudo-ed somewhere you should have run something as 'ubuntu'.
If you can make it work by manually setting permissions, go for it. I have a feeling there are lots of other files owned by root that should be owned byubuntu lurking about in your system.
Anecdotally, if there is no critical data in the server, I have found it is easier to very thoroughly remove any and all files from the old install and then reinstall fresh.
I was facing a similar issue with starting the datanodes. Then, I came across this link https://github.com/cloudera/Impala/wiki/Build-prerequisites, where it states that we need to create the /var/lib/hadoop-hdfs manually and set the appropriate permissions. This has also fixed my problem.
Make certain directory /var/lib/hadoop-hdfs/present is OK.
Related
My flume.log file was getting enormous so I foolishly deleted it. It's absolute path was
/etc/apache-flume-1.8.0-bin/logs/flume.log
I then recreated it (ie: touch flume.log) in the same directory. However nothing gets appended to it anymore after running a Pyspark program. It belongs to root, and I am running the pyspark code as root.
I also checked /etc/apache-flume-1.8.0-bin/conf/log4j.properties and ensured the path was correct (which it is). I also tried changing the path to my project directory to no avail.
I tried deleting the file (to see if it would be automatically created), and I tried creating an empty file first as well. Still no error logs.
Is there a way to fix this?
I am on Ubuntu 18.04, Apache Flume 1.8.0, Pyspark 2.2.2
I ended up having to uninstall Flume, and then reinstall Flume. Unfortunately I could not find any other way to get it working.
I managed to install STS 3.8.2 on Ubuntu 16.04 - with a lot of hacking experiments. I have it working, but I am not happy with my solution.
Here is what I had to do:
Extracted the tar file into /opt/sts-bundle.
If you put it anywhere else, like /opt/sts, the TC server fails to start from STS.
With files in /opt/sts-bundle, TC server still fails to start from STS - permission errors. To get it to work you need to futz around with permissions of the pivotal-c-server subdirectories, essentially you need to open it up your group (the same one running STS) (security hole ?).
A local install in your own ~/sts-bundle fails on "files not found" while attempting to backup - all the conf files. It still looks in /opt/sts-bundle for all these config files (just to copy them to /backup). You can change the top directory of the server in STS server properties - but it still looks in /opt/sts-bundle. Seems hard-coded - don't know where. So you have to create all the config files in the conf directory in the tree rooted at /opt/sts-bundle ("touch" works - creating empty files). TC Server still fails to start with a "failed to clean" error - with no clue from the detailed message what files are being "cleaned".
I tried creating a non-privileged user "tcserver" per suggestion from the Pivotal TC Server docs. I installed to /opt/sts-bundle, while logged in as tcserver (with sudo privileges). That fails when I am using STS as a regular developer that is not "tcserver". Could not figure out how to tell TC server to run under a different user than the one that started STS.
The solution I have working and I am not happy with, starts by extracting the tar.gz file into /opt/sts-bundle, as it wants. Then changing owner and group of sts-bundle to my id and my group (same ones that are used in STS UI). I am not happy with that. It seems wrong to put things in /opt that are owned by a single developer.
I am new to Linux, and I still have some Windows habits that need to be unlearned.
The question is: how do I get the clean solution (installing using a "tcserver" user in the global /opt directory) to work for developers who are not "tcserver"? How should the tcserver user be related to the developers (same group?).
Am I making this problem harder than it should be? What am I missing?
I'm not sure this what you want, but I don't install the STS bundles in some kind of shared directory as a special user at all. I just install it in my user.home dir, as myself, and launch it from there.
It is very unsophisticated. I just download the tar.gz file, unpack it in my home dir and then launch it from a trivial bash script which looks something like this:
#!/bin/bash
/home/kdvolder/Applications/sts-bundle/sts-*/STS
That script is on my PATH. So I can just type 'STS' in a terminal and STS will start.
I don't have to do anything else and it works.
If you are trying to somehow install this so that several different users can run a shared installation then this isn't a good setup. But I think for your own personal laptop or desktop which only you are using, this simple setup is perfectly fine.
For a shared-user env, unfortunately, I don't know how to help you. It could be complicated to sort out all the permissions issues etc because Eclipse is a complicated beast w.r.t to installation of plugins etc.
I have spent numerous hours on an issue that has left me puzzled. I am attempting to install Drupal on Linux Redhat using apache, but it will not allow me to pass step 3 due to the fact that sites/default/files is not writable.
I have followed the instructions on Drupal's site, in their install.txt file as well as the instructions of others who have had the same error with no success.
I have granted permissions access all different ways root:root 777, root:apache 777, I have verified that apache is the user running the apache process and I am still stuck.
Note: I was able to complete the install on windows.
Any new ideas?
Okay, so after following directions from both official and non-official web sources, the one thing that was never instructed to do or try was to reboot the application AFTER making permission changes to the files directory. I tried it, and this solved the issue.
This is weird because I've never had to reboot an OS after making permission changes on a directory. Additionally I did restart httpd after each change thinking that would be sufficient. Hopefully this can help anyone else running Redhat 7 with the same issue.
Thanks, TH
I solved this problem by changing the security context of the directory "sites".
My Drupal core files are in: /var/www/html/drupal
Then I applied the command:
chcon -R -t httpd_sys_content_rw_t /var/www/html/drupal/sites/
I am currently learning Jenkins and how to utilize continuous integration. I am having an issue where all of my data/config files are reset after rebooting my PC. Has anyone had similar issues or am I missing something?
Sincerely, I didn't have this problem it's somehow strange because Jenkins, as I know, stores all the configurations in config.xml files in the installation directory. Not sure if it'll help you, but after a restart, if your data doesn't appear to be indexed in Jenkins, go to "Configure" and there you'll find "Reload configuration from disk".
What I didn't understood from your question:
after a restart of Jenkins+ PC, you data+settings don't appear in Jenkins GUI? Or they are also missing from the config.xml file from the installation directory. For example, if you create a user and a job, after a restart this settings are missing only from the GUI or also from config.xml & jobs directory.
How do you run Jenkins? You start Jenkins from Eclipse, you have it installed on your PC.
Does Jenkins have permissions to create/edit files in the installation directory? Be aware that after install, Jenkins creates a default username "JENKINS" and will try to edit files and create directories with that username on your PC.
I've been stuck on a problem for two days now where the software I'm trying to install will not proceed unless I make a separate user which is non-root.
Keep in mind I'm a big linux noob and not very experienced with the OS.
I make a user called "smrtanalysis" in a group called "smrtanalysis".
I put him in the sudoers file.
I made a folder called smrtanalysis in my home/nick/ directory
I downloaded the software from the PacBio website and put the .run files into this directory I noted above.
I used chmod 777 and chown (to user smrtanalysis) on the directory
noted above, and the .run file
I logged into smrtanalysis user by su smrtanalysis, password, and typed
./smrtanalyis-2.2.0.133377.run
the file runs, but then aborts with the following error message:
We recommend running this script as a designated SMRT Analysis user
(e.g. smrtanalysis) who will own all smrtpipe jobs and daemon
processes.
Current user is 'root' (primary group: root)
Installing as 'root' is currently not supported Switch to the desired
user and restart the install. Aborting installation...
Here is the install documentation:
https://github.com/PacificBiosciences/SMRT-Analysis/wiki/SMRT-Analysis-Software-Installation-v2.2.0
It seems pretty straightforward but I can't seem to get it working. If you guys look at the install docs, you'll probably be able to tell me what I'm doing wrong.
Thanks for any help!
Regards,
Nick
change
SMRT_ROOT=/opt/smrtanalysis
on
SMRT_ROOT=/home/nick/smrtanalysis
the rest should be easy.
Be very careful installing binaries from internet, make sure you're confident in the source.
Just don't use root for that, you ran the script as root by accident.
(update: pacbio team can help from the github page at https://github.com/PacificBiosciences/SMRT-Analysis/issues as well.)