I have a shared folder in my server that contain some files. i read these file with filebeat(version 5.0) and send them to logstash for processing. when i run this process, the number of files that i received in logstash output are lower than the actual number of files in shared folder.
I understand that if there isn't \r\n at the end of file, filebeat didn't recognize that file finish and concatenate next file to this file and send these two files as one event to logstash.
for example file 1 is send as one event
file 1:
> - 23L/AKEXXXX/XXX/80/X-23R/AKEXXXX/XXX/80/X\r\n
> - 41L/AKEXXXX/XXX/80/X-41R/AKEXXXX/XXX/80/X\r\n
> - 42L/AKEXXXX/XXX/80/X-42R/AKEXXXX/XXX/80/X\r\n
> - 43L/AKEXXXX/XXX/330/C2-43R/AKEXXXX/XXX/683/BF0\r\n
> - SI\r\n
> - ;\r\n
> - \r\n
but file 2 is not recognize as one file and it concatenate to next file that is ok
file 2:
- 23L/AKEXXXX/XXX/80/X-23R/AKEXXXX/XXX/80/X\r\n
- 41L/AKEXXXX/XXX/80/X-41R/AKEXXXX/XXX/80/X\r\n
- 42L/AKEXXXX/XXX/80/X-42R/AKEXXXX/XXX/80/X\r\n
- 43L/AKEXXXX/XXX/330/C2-43R/AKEXXXX/XXX/683/BF0\r\n
- SI\r\n
- ;\r\n
i play with all filebeat configuration options, but it doesn't help. do you have an idea how can handle and resolved this problem?
Related
I'll start by mentioning that I've no knowledge in Python but read online that it could help me with my situation.
I'd like to do a few things using (I believe?) a Python script.
I have a bunch of .yml files that I want to transfer the contents into one main .yml file (let's call it Main.yml). However, I'd also like to be able to take the name of each individual .yml and add it before it's content into Main.yml as "##Name". If possible, the script would look like each file in a directory, instead of having to list every .yml file I want it to look for (my directory in question only contains .yml files). Not sure if I need to specify, but just in case: I want to append the contents of all files into Main.yml & keep the indentation (spacing). P.S. I'm on Windows
Example of what I want:
File: Apes.yml
Contents:
Documentation:
"Apes":
year: 2009
img: 'link'
After running the script, my Main.yml would like like:
##Apes.yml
Documentation:
"Apes":
year: 2009
img: 'link'
I'm just starting out in Python too so this was a great opportunity to see if my newly learned skills work!
I think you want to use the os.walk function to go through all of the files and folders in the directory.
This code should work - it assumes your files are stored in a folder called "Folder" which is a subfolder of where your Python script is stored
# This ensures that you have the correct library available
import os
# Open a new file to write to
output_file = open('output.txt','w+')
# This starts the 'walk' through the directory
for folder , sub_folders , files in os.walk("Folder"):
# For each file...
for f in files:
# create the current path using the folder variable plus the file variable
current_path = folder+"\\"+f
# write the filename & path to the current open file
output_file.write(current_path)
# Open the file to read the contents
current_file = open(current_path, 'r')
# read each line one at a time and then write them to your file
for line in current_file:
output_file.write(line)
# close the file
current_file.close()
#close your output file
output_file.close()
I need to write a python script to read through files in a directory, retrieve the header record (which contains date)? I need to compare the date in the header record of each file with current date and if the difference is greater than 30 days. I need to delete such files.
I managed to come up with below code but not sure how to proceed since I am new to Python.
Example:
Sample file in the directory (/tmp/ah): abcdedfgh1234.123456
Header record : FILE-edidc40: 20200602-123539 46082 /tmp/ah/srcfile
I have below code for the list of files in the current directory. I need to pass the python equivalent of below actions on unix files
head -1 file|cut -c 15-22
Output: 20200206 (to compare with current date and if older than 30) then delete file(using rm command).
import os
def files in os.listdir(path):
for files in os.listdir(path):
if os.path.isfile(os.path.join(path,file)):
yield file
for file in files(".") : # prints the list of files
I'm trying to follow this tutorial in abinit: https://docs.abinit.org/tutorial/paral_images/
When trying to run abinit for any of the tstring files, no output file is produced. For instance, I copy the files tstring_01.in and tstring.files into the subdirectory work_paral_string, edit the tstring.files with the appropriate file names, and run the command mpirun -n 20 abinit < tstring.files > log 2> err. No error message is shown but also no output is produced. (The expected output file would be tstring_01.out.)
Any suggestions?
Currently I have the following cron.file formula
date > system_cron:
cron.file:
- name: salt://crons/cron_jobs
- source_hash: "md5sum=895dcbbddd27bfa77056ef8c8340549a"
- user: security
But this updates the crontab each time the highstate is run event though the cron_jobs file has not changed and hence the state is the same.
Is there a way to stop creating temp crontab file each time highstate is run when using cron.file
I found that this happens when you have blank lines at the end of the file, or if you have dos line endings instead of unix line endings. Once this has been changed, the file will only be updated when it has changed.
Please note: as you have placed the file in "salt://", you don't need source_hash. This is only required for remote (i.e. http) files.
On OS X, there is an API to create a "bookmark" for a given file on disk, which will track its target even if it is moved or renamed. An OS X application could then serialise the bookmark, store it by whatever means it wants, and at some later date—perhaps after being quit and re-launched—deserialise the bookmark and resolve the file path from it.
Does anything comparable exist for Linux?
http://en.wikipedia.org/wiki/Inotify
I guess that inotify is what you want.
With inotify, you can monitor following events for a file:
IN_ACCESS - read of the file
IN_MODIFY - last modification
IN_ATTRIB - attributes of file change
IN_OPEN - open of file
IN_CLOSE_WRITE - sent when a file opened for writing is closed
IN_CLOSE_NOWRITE - sent when a file opened not for writing is closed
IN_MOVED_FROM and IN_MOVED_TO - when the file is moved or renamed
IN_DELETE - a file/directory deleted
IN_CREATE - a file in a watched directory is created
IN_DELETE_SELF - file monitored is deleted