I'd like to use ELK to analyze and visualize our GxP Logs, created by our stoneold LIMS system.
At least the system runs on SLES but the whole logging structure is some kind of a mess.
I try to give you an impression:
Main_Dir
| Log Dir
| Large number of sub dirs with a lot of files in them of which some may be of interest later
| Archive Dir
| [some dirs which I'm not interested in]
| gpYYMM <-- subdirs created automatically each month: YY = Year ; MM = Month
| gpDD.log <-- log file created automatically each day.
| [more dirs which I'm not interested in]
Important: Each medical examination, that I need to track, is completely logged in the gpDD.log file that represents the date of the order entry. The duration of the complete examination varies between minutes (if no material is available), several hours or days (e.g. 48h for a Covid-19 examination) or even several weeks for a microbiological sample. Example: All information about a Covid-19 sample, that reached us on December 30th is logged in ../gp2012/gp30.log even if the examination was made on January 4th and the validation / creation of report was finished on January 5th.
Could you please provide me some guidance of the right beat to use ( I guess either logbeat or filebeat) and how to implement the log transfer?
Logstash file input:
input {
file {
path => "/Main Dir/Archive Dir/gp*/gp*.log"
}
}
Filebeat input:
- type: log
paths:
- /Main Dir/Archive Dir/gp*/gp*.log
In both cases the path is possible, however if you need further processing of the lines, I would suggest using at least Logstash as a passthrough (using beats input if you do not want to install Logstash on the source itself, which can be understood)
I have a log file which logs network activities. I want to view the log file, but I want to see matched pattern. I mean I want to see the content of my choice. The log file is in this format: Nov 7 12:00:00 ......... How can I view content of a specific date or specific time. I mean if I want to see only 3:00 to 5:00 on Nov 7 how can I use the tail command to do that?
There are multiple ways to do what you want. One of them using grep is
grep '^Nov 7 0[3-5]:' network.log | less
I have a PowerShell script that uses log4Net for the management of logs. The logs are written in log files and in MS SQL database. The script is using multi-threading with run spaces.
The issue is multiple threads are managing several objects and logging lots of data on different objects at the same time. And I need to regroup logs by object. An example will help me to explain myself better! ^^
log file line 1 OBJECT 1.ACTION 1
log file line 2 OBJECT 1.ACTION 2
log file line 3 OBJECT 1.ACTION 3
log file line 4 object2.action1
log file line 5 object3.action1
log file line 6 OBJECT 1.ACTION 4
log file line 7 object2.action2
log file line 8 OBJECT 1.ACTION 5
…
To manage this intercalated logging issue, I planned to log in memory, for example in a table, and at the end of the treatment of the objects; block other threads using mutex and write all logs with a foreach loop.
Main {
Treat object {
Action1 -> Logs +=log1
Action2 -> Logs +=log2
…
}
System.Threading.Mutex WaitOne()
For each ($log in $Logs) {
Write in log file
Write in SQL DB
}
System.Threading.Mutex ReleaseMutex()
}
I would like to know if there is any better solution to manage intercalated logging issue with multiple run spaces please.
Log4Net can perhaps natively manage this; stocks all logs in memory and “commits” the writes only when I type a command? Or some other solutions without using Mutex?
One way would be to use one logger per group, and if you don't know the number of groups in advance you could just create the loggers dynamically.
If you prefer to work with one logger, the best practice usually is to log things as they happen and do the grouping afterwards, for example when the logs are displayed.
It's trivial to do in SQL by adding a column for the grouping criteria, or for a text file you could use the unix sort command.
First of all i'm a beginner in bash scripting. I usually code in Java but this certain task requires me to create some bash scripts in Linux. FYI i've already made a working script but I think its not efficient enough because of the large files I'm dealing with.
The problem is simple I have 2 logs that I need to compare and make some correction on one of the logs... ill call it logA and logB. This 2 logs contains different format here is an example:
01/04/2015 06:48:59|9691427842139609|601113090635|PR|30.00|8.3|1.7| <- log A
17978712 2015-04-01 06:48:44 601113090635 SUCCESS DealerERecharge.<-log B
17978714 2015-04-01 06:48:49 601113090635 SUCCESS DealerERecharge.<-log B
As you can see there is a gap in time stamp. The actual logs that will match with log A is the one with the ID 17978714 because it is the closest time from it. The highest time gap I've seen is 1 minute. I cant use the RANGE logic because if there are more than one line on log B that is within the 1 minute range then all of the line will show in my regenerated log.
The script I made contains a for loop which iterate the timestamp of log A until it hits something in log B (The first one it hits is the closest)
Inside the for loop I have this line of code which makes the loop slow.
LINEOUTPUT=$(less $File2 | grep "Validation 1" | grep "Validation 2" | grep "Timestamp From Log A")
I've read some sample using SED but the problem is I have 2 more validation to consider before matching it with the time stamp.
The validation works as a filter to narrow down the exact match for log A and B.
Additional Info: I tried doing some benchmark test for the script I made by performing some loop. One thing I've noticed is that even though I only use 1 pipe for that script the loop tick is still slow.
I'm trying to create a log file at the start of my program.
I need to check if a /log directory exists if it doesn't create the directory then move on to creating the log file.
Well I tried to use os.Mkdir (as well as os.MkdirAll), but no matter what value I put into the second parameter I get a locked out folder with no permissions. What value should this be in order to get a read / write for user folder? I thought it would be 0x700 but it doesn't seem to work.
Thanks!
You can use octal notation directly:
os.Mkdir("dirname", 0700)
Permission Bits
+-----+---+--------------------------+
| rwx | 7 | Read, write and execute |
| rw- | 6 | Read, write |
| r-x | 5 | Read, and execute |
| r-- | 4 | Read, |
| -wx | 3 | Write and execute |
| -w- | 2 | Write |
| --x | 1 | Execute |
| --- | 0 | no permissions |
+------------------------------------+
+------------+------+-------+
| Permission | Octal| Field |
+------------+------+-------+
| rwx------ | 0700 | User |
| ---rwx--- | 0070 | Group |
| ------rwx | 0007 | Other |
+------------+------+-------+
A Unix Permission Primer
Common Permission Usages
0755 Commonly used on web servers. The owner can read, write, execute. Everyone else can read and execute but not modify the file.
0777 Everyone can read write and execute. On a web server, it is not advisable to use ‘777’ permission for your files and folders, as it allows anyone to add malicious code to your server.
0644 Only the owner can read and write. Everyone else can only read. No one can execute the file.
0655 Only the owner can read and write, but not execute the file. Everyone else can read and execute, but cannot modify the file.
www.maketecheasier.com/file-permissions-what-does-chmod-777-means/
Directory Permissions on Linux
When applying permissions to directories on Linux, the permission bits have different meanings than on regular files. (source)
Read bit The user can read the file names contained in the directory.
Write bit The user can {add,rename,delete} files names IF the execute bit is set too.
Execute bit The user can enter the directory and access the files inside.
https://unix.stackexchange.com/a/21252
Permissions Calculator
A handy permissions calculator.
#Daniel's statement in his answer is not really correct, and also it talks about a decimal number and then uses an octal one, as #SashaCrofter correctly pointed out in his comment.
In reality, it doesn't matter what form your permission value is in as long as it represents sensible Unix permissions.
Since permission bits on POSIX file systems come in triples of bits — three bits for owner, group and others access, plus three bits of modifiers (such as sticky bits), — it's customary to use octal numbers to represent permissions as each digit in an octal number represents a three-bit value.
Hence, when you use 0700 in Go code, the leading 0 is stripped and is only there to tell the parser it sees an octal number literal, and the following three letters stand for the owner, group and others permissions, in this order. Should you, say, want to also set the group sticky bit as well as making the file system object group-readable and executable, you'd specify 02750 and so on.
Note that the actual permissions the file system object acquires is further modulated by the active umask of the process which creates the object.
To get more grip on these topics, it's best to read the chmod manual pages and general literature on Unix-like operating systems.
You can reset the umask to 0. I would call this as the first thing in my main file
syscall.Umask(0)
Example
_ = os.MkdirAll("/tmp/dirs/1", 0664)
syscall.Umask(0)
_ = os.MkdirAll("/tmp/dirs/2", 0664)
Result
/tmp/dirs$ stat -c '%A %a %n' *
drw-r--r-- 644 1
drw-rw-r-- 664 2
Besides the other answers, remember that on Unix and Linux style operating systems, all programs run with a umask setting. The umask, which in many cases defaults to 022 or sometimes 002, is the set of permissions that the system will automatically remove from file and directory creation requests.
What this means is that most programs–there are several exceptions to this rule—should use mode 0666 for creating files and mode 0777 for creating directories. The user's configuration, recorded in the running process, says which of these permissions to take away. If the user's setting is 022, and we create a file with mode 0666, the actual setting we get is rw-r--r--: read and write for the user, read-only for the group, and read-only for others.
If a user wishes to extend writability to their group, they need only set their umask to 2: now they take away write permission for others, but leave it for their group. New files are now created with mode rw-rw-r--. The program does not change: it still uses 0666 for its mode. But the files are created with mode 0664.
Similarly, if you call os.Mkdir or os.MkdirAll with 0777, the umask will take away the unwanted permissions, leaving you with the right permissions.
But I mentioned that there are exceptions. These include programs that make copies of sensitive information meant only for the user: these should generally use mode 0700 for directories and 0600 for files. They may include long-running servers that act as a system user rather than any one individual ... although those servers could be run with a correct umask, in which case, 0777 or 0666 is fine.
You must apply some judgment here. Programs that are especially security-conscious, such as ssh or similar, may wish to use limited permissions, and may even want to check (with os.Lstat or similar) that permissions are appropriately tight on important directories.
(Note that the umask does not apply to os.Chmod calls. Here you choose the mode directly.)
One way to make sure that you're setting the kind of permissions you want, without figuring out the complex calculations in octal, is to use the very convenient FileMode constants in package os:
https://golang.org/pkg/os/#FileMode
I usually use os.ModePerm (which is actually coded as 0777) for fully permissive directories, such as those required for caches or temporary files, but your mileage may vary. To set the additional bits (sticky, etc.), which, as #kostix has noted, has to deal with the issue of octal representation of flags in Go, you can always use something like:
if err := os.MkdirAll("my/tmp/dir", os.ModeSticky|os.ModePerm); err != nil {
... handle error ...
}
Go playground
As always, it's worth mentioning again that these permissions are 'filtered' by whatever umask has been set.