This is my .modulepath file, the last two lines are the paths where I have my modules mounted from another hard drive. Even though I added these lines module avail command does not fetch me any of the modules in those folders. If anyone could help it would be of great help.
#
# #(#)$Id: 38aa24cc33a5f54a93781d63005a084f74418022 $
# Module version 3.2.10
# init/.modulespath. Generated from .modulespath.in by configure.
#
# Modulepath initial setup
# ========================
#
# This file defines the initial setup for the module files search path.
# Comments may be added anywhere, which begin on # and continue until the
# end of the line
# Each line containing a single path will be added to the MODULEPATH
# environment variable. You may add as many as you want - just
# limited by the maximum variable size of your shell.
#
/etc/environment-modules/modules
#/usr/share/modules/versions # location of version files
#/usr/Modules/$MODULE_VERSION/modulefiles # Module pkg modulefiles (if versioning)
#/usr/Modules/modulefiles # Module pkg modulefiles (if no versioning)
#/usr/share/modules/modulefiles # General module files
#/usr/Modules/3.2.10/your_contribs # Edit for your requirements
/opt/apps/modulefiles/Core
/opt/apps/modulefiles/Compiler
I have even tried using module use /opt/apps/modulesfiles/Core
user#user-N501VW:~$ module use /opt/apps/modulefiles/Core
user#user-N501VW:~$ $MODULEPATH
bash: /opt/apps/modulefiles/Core:/etc/environment-modules/modules:/usr/share/modules/versions:/usr/Modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles: No such file or directory
akhila#akhila-N501VW:~$ module avail
------------------------------------------------------------- /usr/share/modules/versions --------------------------------------------------------------
3.2.10
------------------------------------------------------------ /usr/share/modules/modulefiles ------------------------------------------------------------
dot module-git module-info modules null use.own
Even though your specific modulepaths are correctly set in MODULEPATH environment variable, module avail does not return any modulefiles for these directories. It means module has not found a file in these directories that is a modulefile.
So I suggest you to:
check if your mountpoint is correctly mounted
verify that the files in the directories are modulefiles compatible with the module command you use (on the module command version you use, the content of a modulefile should start with the #%Module magic cookie)
Related
I have different logs that are written to our moutend nfs share that i need to send to our syslog-server (graylog) they are located outside /var/log folder.
So i add some extra conf in /etc/rsyslog.d/
For this example i have two files with following config:
atlassian-application-confluence-log.conf
module(load="imfile")
module(load="imklog")
$MaxMessageSize 50k
global(workDirectory="/atlassian/test/confluence/logs")
# This is the main application log file
input(type="imfile"
File="/atlassian/test/confluence/logs/atlassian-confluence.log"
Tag="atlassian"
PersistStateInterval="200"
)
# This file contains entries related to the search index.
input(type="imfile"
File="/atlassian/test/confluence/logs/atlassian-confluence-index.log"
Tag="atlassian"
PersistStateInterval="200"
)
# Send to Graylog
action(type="omfwd" target="log-server-company.com" port="5140")
# if you want to keep a local copy of the logs.
action(type="omfile" File="/var/log/rsyslog.log" template="RSYSLOG_TraditionalFileFormat")
atlassian-application-jira-log.conf
module(load="imfile")
module(load="imklog")
$MaxMessageSize 50k
global(workDirectory="/atlassian/test/jira/log")
# Contains logging for most of Jira, including logs that aren’t specifically written elsewhere
input(type="imfile"
File="/atlassian/test/jira/log/atlassian-jira.log"
Tag="atlassian"
PersistStateInterval="200"
)
# Send to Graylog
action(type="omfwd" target="log-server-company.com" port="5140")
# if you want to keep a local copy of the logs.
action(type="omfile" File="/var/log/rsyslog.log" template="RSYSLOG_TraditionalFileFormat")
So to my problem.
When i check the Rsyslogd configuration with following command:
rsyslogd -N1 -f /etc/rsyslog.d/atlassian-application-confluence-log.conf
It says it is valid.
When i restart the rsyslog service i get the following errors:
module 'imfile' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
module 'imklog' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
error during parsing file /etc/rsyslog.d/atlassian-tomcat-confluence-log.conf, on or before line 6: parameter 'workdirectory' specified more than once - one instance is ignored. Fix config [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2207]>
error during parsing file /etc/rsyslog.d/atlassian-tomcat-confluence-log.conf, on or before line 6: parameter 'workDirectory' not known -- typo in config file? [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2207]
module 'imfile' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
module 'imklog' already in this config, cannot be added [v8.2102.0-10.el8 try https://www.rsyslog.com/e/2221 ]
[origin software="rsyslogd" swVersion="8.2102.0-10.el8" x-pid="379288" x-info="https://www.rsyslog.com"] start
imjournal: journal files changed, reloading... [v8.2102.0-10.el8 try https://www.rsyslog.com/e/0 ]
How can i get rid of the warnings?
I have already tried to put the two modules in /etc/rsyslog.conf
I get following errors from that config:
parameter 'PersistStateInterval' not known
parameter 'Tag' not known
parameter 'File' not known
If there are multiple configuration files, they are processed in ascending sort order of the file name (numerically/alphabetically), See: $IncludeConfig.
Therefore you don't have to include any configuration parameters (modules, work directories, rulesets etc.) multiple times. You can include them once in the config which is loaded first.
As far as I'm aware I'm using best practices to define paths (using raw strings) and how I go about joining them (using os.path.join()), e.g.
import os
fdir = r'C:\Code\...\samples'
fpath = os.path.join(fdir, 'fname.ext')
and doing so has not caused me any problems when running my code within a Python or command shell. If I print fpath to the console I get consistent use of \s in the path:
C:\Code...\samples\fname.ext
But when I run a Docker containerized version of the code and run the image I get the error:
FileNotFoundError: [Errno 2] No such file or directory:
'C:\Code\...\samples/fname.ext'
I don't understand why os.path.join() has used a / to join fdir and fname.ext when the rest of the path included \\. It doesn't do this when I run the code outside of the container.
I have tried using os.path.normpath():
fpath = os.path.join(fdir, 'fname.ext')
fpath = os.path.normpath(fpath)
as discussed here, and os.sep.join():
fpath = os.sep.join([fdir, 'fname.ext'])
as covered here, and Path().joinpath():
from pathlib import Path
fpath = Path(fdir).joinpath('fname.ext')
as well as Path() / 'path_to_add':
fpath = Path(fdir) / 'fname.ext'
as discussed here, but in every case I end up with the same result using os.path.join().
Can someone please help me to understand what is going on and how to create consistent paths that will work whether I run the code in Python in a Windows environment, or in a Docker container?
Update Nov. 16:
In trying to keep my question brief I think I've left out details that are crucial. Apologies to those who have kindly taken the time to offer suggestions based on my incomplete description of the problem.
My code needs to import/export files from/to directories that are defined within a user-specified configuration file.
So the configuration file has a section of code where the user defines variables and paths, e.g.
samplesDir = r"path-to-samples-directory"
The variables are stored in a dictionary of dictionaris and stored as a .json.
At the start of the code the user defines the key that selects the dictionary of interest so that at various parts in my code when a file needs to be imported/exported, the paths are at hand.
So back to my example, samplesDir is stored in the configuration dictionary, cfgDict, so all I need to do is append the file name:
sampleFpath = os.path.join(sampleDir, sampleFname)
and sampleFname is determined based on other variables.
Because of the dynamic nature of the variables (including directory paths and file paths), I think it rules out the use of static path defined in a .yml with Docker Compose.
Update Nov. 18:
It may help to include a few more details and some screenshots.
The above screenshot shows the file and folder structure of the src directory containing the source code, the main app.py script for command-line use, the Dockerfile, etc.
The configs folder contains JSON files that includes variables, paths to directories and files. The user can create configuration files either by copying an existing one and modifying the entries, or configuration files can be generated by calling config.py.
Within config.py I have pre-set variables and paths, so that the directory path to the configuration files (configs), sample files (sample_DROs) and others (e.g. fiducials) are all within src.
I don't anticipate any reason why the user would want to store the config files anywhere else, nor do I expect them to want to use different sample files (or move them elsewhere). However, they will undoubtedly create their own fiducials and may decide not to store them in the fiducials directory (i.e. somewhere not within the src directory).
Likewise I have pre-set the download directory (based on the parameters stored within the configuration files, files are fetched from a server and downloaded) to be the default Downloads directory:
rootDownloadDir = os.path.join(Path.home(), "Downloads", "xnat_downloads")
Those files are later imported, processed, and the outputs are (by default) exported into sub-directories within rootDownloadDir.
Within Dockerfile I set the working directory of the container to be that of the source code and copy all of the contents of src (with the exception of some directories defined in .dockerignore):
WORKDIR C:/Code/WP1.3_multiple_modalities/src
...
COPY . .
so that the structure of the container mimics that of WORKDIR:
Hence I have allowed for flexibility in import/export directories, and they are by default a combination of paths within and outside of the src directory. And so, the code executed within the container will need to access files both within and outside of src.
That said, I don't know what rootDownloadDir will look like when os.path.join(Path.home(), "Downloads", "xnat_downloads") is run within the container.
This has got me thinking - Is it bad practice to set the download directory outside of src?
Returning to the original error:
the sample file is in the container:
From the actual behavior I can suppose that the container is based on Unix-like image. Path separator is / in such systems.
To build an environment-independent path which works inside and outside of the container you need the following steps:
Mounting of host folder to container directory.
Environment variable inside and outside the container.
I can show an example of how this is achievable via docker-compose tool and its configuration file docker-compose.yml:
# docker-compose.yml file
version: '3'
services:
<service_name>: # your service name here
image: <image_name> # name of image your container is built on
environment:
- SAMPLES_PATH=/samples
volumes:
- C:\Code\somepath\samples:/samples
In your python code you can use the following structure:
import os
fdir = os.getenv('SAMPLES_PATH', r'C:\Code\...\samples')
fpath = os.path.join(fdir, 'fname.ext')
I have the following code;
#!/usr/bin/env python3
import yaml
with open('config.yml', 'r') as config_file:
config = yaml.load(config_file)
The file is called __init__.py which is in the directory ~/bin/myprogram/myprogram/ and in the same directory, I have a file called config.yml
My symlink is as follows;
user$ ls -la /usr/local/bin/
lrwxr-xr-x 1 user admin 55 27 Nov 13:25 myprogram -> /Users/user/bin/myprogram/myprogram/__init__.py
Every time I run myprogram, I get the error FileNotFoundError: [Errno 2] No such file or directory: 'config.yml'. I believe this is because the config.yml is not in /usr/local/bin/. What is the best way to work around this issue?
You can use __file__ to access the location of the __init__.py file when executing code in that file. It returns the full path, but care has to be taken as it may be the .pyc (or .pyo) version. Since you are using Python3 I would use the pathlib module:
import yaml
from pathlib import Path
my_path = Path(__file__).resolve() # resolve to get rid of any symlinks
config_path = my_path.parent / 'config.yaml'
with config_path.open() as config_file:
config = yaml.safe_load(config_file)
Please note:
If you have to use PyYAML, use safe_load(), even PyYAML's own documentation indicates .load() can be unsafe. It almost never necessary to use that. And in the unlikely event that safe_load() cannot load your config, e.g. if it has !!python/... tags, you should explicitly add register the classes that you actually need to the SafeLoader).
Since September 2006 the recommended extension for YAML files has been .yaml
I like to configure syslog. It seems that are more than one way to set up syslog. I am asking for the common way/steps to do that.
I have several use cases. To simplify I like to ask how to configure syslog to write an infinity long log file in /var/log/.
Following steps:
1.) configure what messages
1.1) create own "syslog.conf" (define /var/log/myLog)
1.2) append it to "recipes-core/busybox"
2.) configure how to log
??
I found two possible places to do that:
#meta-poky
-> "meta-poky/recipes-core/busybox/busybox/poky-tiny/defconfig"
#
# System Logging Utilities
#
CONFIG_SYSLOGD=y
CONFIG_FEATURE_ROTATE_LOGFILE=y
CONFIG_FEATURE_REMOTE_LOG=y
CONFIG_FEATURE_SYSLOGD_DUP=y
CONFIG_FEATURE_SYSLOGD_CFG=y
CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256
CONFIG_FEATURE_IPC_SYSLOG=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=16
CONFIG_LOGREAD=y
CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y
CONFIG_KLOGD=y
CONFIG_FEATURE_KLOGD_KLOGCTL=y
CONFIG_LOGGER=y
add/change:
"CONFIG_FEATURE_ROTATE_LOGFILE=n" by adding that line to meta-mylayer/conf/layer.conf"
etc.
???
# "/etc/syslog-startup.conf"
# This configuration file is used by the busybox syslog init script,
# /etc/init.d/syslog[.busybox] to set syslog configuration at start time.
DESTINATION=file # log destinations (buffer file remote)
LOGFILE=/var/log/messages # where to log (file)
REMOTE=loghost:514 # where to log (syslog remote)
REDUCE=no # reduce-size logging
DROPDUPLICATES=no # whether to drop duplicate log entries
#ROTATESIZE=0 # rotate log if grown beyond X [kByte]
#ROTATEGENS=3 # keep X generations of rotated logs
BUFFERSIZE=64 # size of circular buffer [kByte]
FOREGROUND=no # run in foreground (don't use!)
#LOGLEVEL=5 # local log level (between 1 and 8)
In the systemV init script "/etc/init.d/syslog.bussybox" the file "/etc/syslog-startup.con" is read and used for configuration.
System behaviour:
When running my system, the log wraps when the logfile reaches 200kBytes. One logfile + one log-rotate file is generated.
Any ideas how to archive that syslog writes an infinite long log-file?
I am working on the Yocto krogoth branch + meta-atmel / meta_openembedded (# krogoth too).
By checking the sources of syslog and busybox I found a possible solution. This solution shows how to configure syslog to log in two logs with max 10MByte:
1.) get valid syslog build config
1.1) download busybox -> git/busybox
1.2) build busybox via bitbake -> bitbake busybox
1.3) copy defconfig file to downloaded busybox -> cp /defconfig git/busybox/
1.4) make menueconfig
1.5) goto "System Logging Utilities"
1.6) deselect klogd because it can colide with printk
1.7) save to "defconfig"
#
# System Logging Utilities
#
# CONFIG_KLOGD is not set
# CONFIG_FEATURE_KLOGD_KLOGCTL is not set
CONFIG_LOGGER=y
CONFIG_LOGREAD=y
CONFIG_FEATURE_LOGREAD_REDUCED_LOCKING=y
CONFIG_SYSLOGD=y
CONFIG_FEATURE_ROTATE_LOGFILE=y
CONFIG_FEATURE_REMOTE_LOG=y
CONFIG_FEATURE_SYSLOGD_DUP=y
CONFIG_FEATURE_SYSLOGD_CFG=y
CONFIG_FEATURE_SYSLOGD_READ_BUFFER_SIZE=256
CONFIG_FEATURE_IPC_SYSLOG=y
CONFIG_FEATURE_IPC_SYSLOG_BUFFER_SIZE=64
CONFIG_FEATURE_KMSG_SYSLOG=y
2.) setup your log config
Create "syslog.conf" and enter the rules:
This is an example:
#
# /etc/syslog.conf Configuration file for busybox's syslogd utility
#
kern.notice /var/log/messages
#
# my software messages
#
user.err /var/log/mySWError
user.* /var/log/mySWFull
local0.* /var/log/mySWFull
local0.err /var/log/mySWError
#
#this prevents from logging to default log file (-O FILE or /var/log/messages)
#
*.* /dev/null
3.) modify configuration for the busybox syslog deamon
This example logs to files which are limited to 10 MBytes. If "ROTATESIZE" is not set syslog set the log filesize automatic to 200 kBytes. The content of "syslog-startup.conf" looks like:
# This configuration file is used by the busybox syslog init script,
# /etc/init.d/syslog[.busybox] to set syslog configuration at start time.
DESTINATION=file # log destinations (buffer file remote)
#LOGFILE=/var/log/messages # where to log (file)
REMOTE=loghost:514 # where to log (syslog remote)
REDUCE=no # reduce-size logging
DROPDUPLICATES=no # whether to drop duplicate log entries
ROTATESIZE=10000 # rotate log if grown beyond X [kByte]
#ROTATEGENS=3 # keep X generations of rotated logs
BUFFERSIZE=64 # size of circular buffer [kByte]
FOREGROUND=no # run in foreground (don't use!)
#LOGLEVEL=5 # local log level (between 1 and 8)
4.) get the configuration into yocto build
4.1) create following directory structure in your own layer(meta-custom):
meta-custom/recipes-core/
meta-custom/recipes-core/busybox/
meta-custom/recipes-core/busybox/busybox
4.2) copy into "meta-custom/recipes-core/busybox/busybox":
defconfig
syslog.conf
syslog-startup.conf
4.3) create in "meta-custom/recipes-core/busybox/" "busybox_1.24.1.bbappend". If you using an older/newer version of busybox you need to change the "1.24.1" number to yours. You can find your version in "/poky/meta/recipes-core/busybox/"
Add this two lines to this file:
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}:"
FILESEXTRAPATHS_prepend := "${THISDIR}/${PN}/poky-tiny:"
5.) build your custom defined syslog
bitbake busybox
and transfer it into the rootfs of the image
bitbake core-image-minimal
Now it should work!
Adding to the Stefan Jaritz comment, you can skip the download step by just issuing
bitbake busybox -c devshell
Then running make menuconfig and getting the new config file from it. Note that this will use Yocto's defconfig by default, so you don't need to do worry about "what's this default config".
I am very new to linux environment and I have an issue with loading a module in linux. The problem is I am trying to point an already loaded module to the one present in my local directory. For this I have changed all the path that would point the module to my local directory and also updated the MODULEPATH with my local directory path.
This is the command I am using for changing the MODULEPATH
export MODULEPATH=~/local/directory:${MODULEPATH}
Though the question I have is completely different, how to give a version number to my module?
I have a moudle file which looks like this -
#%Module
proc ModulesHelp { } {
puts stderr "Sets up users' environment to use XXXX. XXXX is the ..\n.. It is a piece of middleware developed by XXXX to \nhelp track XXX usage. \n\nIf you encounter any problems while linking or while executing aprun, please unload this module."
}
conflict totalview
setenv XXXX_SELECT_ON 0
setenv XXXX_SELECT_USERS kagrawa1
setenv XXXX_SELECT_OFF_USERS ""
setenv XXXX_VERBOSE 0
setenv XXXX_ON 1
setenv XXXX_PATH /d/home/kpilagr/
prepend-path PATH /nics/d/home/kagrawa1/altd/bin