I am implementing the ftp server on linux box (fedora 11, vsftpd). Everything works good so far but i need Ftp server log files to contain transfer rate information.
At the moment when I use "get " or " put" command from client end , I got following message on ftp client
example
ftp: 18 bytes received in 0.00seconds 18000.00kbytes/sec.
Is there is any way , I can get same message on ftp server side?
Below is the sample of my xferlog file
Tue Oct 23 01:28:52 2012 1 10.65.112.55 1 /home/test/testfile b _ o r test ftp 0 * c
Tue Oct 23 01:32:46 2012 1 10.65.112.55 18 /home/test/uploadServer b _ i r test ftp 0 * c
Tue Oct 23 01:50:23 2012 1 192.168.10.27 1 /home/test/testfile a _ o r test ftp 0 * c
Tue Oct 23 01:50:36 2012 1 192.168.10.27 19 /home/test/test a _ i r test ftp 0 * c
I really appreciate everyone's help here.
Well I have solved that.
I have included
*dual_log_enable= yes* in the vsftpd.conf file and as a result it created the new log file as /var/log/vsftpd.log and it contains all the information that I need.
Related
I'm trying to run background task processing, redis and rabbitMQ work in separate docker containers
#dramatiq.actor(store_results=True)
def count_words(url):
try:
response = requests.get(url)
count = len(response.text.split(" "))
print(f"There are {count} words at {url!r}.")
except requests.exceptions.MissingSchema:
print(f"Message dropped due to invalid url: {url!r}")
result_backend = RedisBackend(host="172.17.0.2", port=6379)
result_broker = RabbitmqBroker(host="172.17.0.5", port=5672)
result_broker.add_middleware(Results(backend=result_backend))
dramatiq.set_broker(result_broker)
message = count_words.send('https://github.com/Bogdanp/dramatiq')
print(message.get_result(block=True))
RabbitMQ:
{"queue_name":"default","actor_name":"count_words","args":["https://github.com/Bogdanp/dramatiq"],"kwargs":{},"options":{},"message_id":"8e10b6ef-dfef-47dc-9f28-c6e07493efe4","message_timestamp":1608877514655}
Redis
1:C 22 Dec 2020 13:38:15.415 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
1:M 22 Dec 2020 13:38:15.417 * Running mode=standalone, port=6379.
1:M 22 Dec 2020 13:38:15.417 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 22 Dec 2020 13:38:15.417 # Server initialized
1:M 22 Dec 2020 13:38:15.417 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
1:M 22 Dec 2020 13:38:15.417 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
1:M 25 Dec 2020 10:08:12.274 * Background saving terminated with success
1:M 26 Dec 2020 19:23:59.445 * 1 changes in 3600 seconds. Saving...
1:M 26 Dec 2020 19:23:59.660 * Background saving started by pid 24
24:C 26 Dec 2020 19:23:59.890 * DB saved on disk
24:C 26 Dec 2020 19:23:59.905 * RDB: 4 MB of memory used by copy-on-write
1:M 26 Dec 2020 19:23:59.961 * Background saving terminated with success
Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/dramatiq/message.py", line 147, in get_result
return backend.get_result(self, block=block, timeout=timeout)
File "/usr/local/lib/python3.6/dist-packages/dramatiq/results/backends/redis.py", line 81, in get_result
raise ResultTimeout(message)
dramatiq.results.errors.ResultTimeout: count_words('https://github.com/Bogdanp/dramatiq')
I have a string from which I have to remove the timestamp and punctuation. And I have to remove all the digits also but responseCode value
has to be kept as is for example 400 in this case. And wherever 400 comes, it should not be removed. And I want to remove all the url's
and file name ending with tar.gz.
mystr="sun aug 19 13:02:09 2018 I_am.98189: hello please connect to the local host:8080
sun aug 19 13:02:10 2018 hey.94289: hello not able to find the file
sun aug 19 13:02:10 2018 I_am.94289: Base url for file_transfer is: abc/vd/filename.tar.gz
mon aug 19 13:02:10 2018 how_94289: $var1={
'responseCode' = '400',
'responseDate' = 'Sun, 19 Aug 2018 13:02:08 ET',
'responseContent' = 'ABC' }
mon aug 20 13:02:10 2018 hello!94289: Error performing action, failed with error code [400]
"
Expected result:
"I_am hello please connect to the local host
hello not able to find the file
Base url for file_transfer
var1
responseCode = 400
responseDate
responseContent = ABC
Error performing action, failed with error code 400
"
My Solution to remove punctuation:
punctuations = '''!=()-[]{};:'"\,<>.?##$%^&*_~'''
no_punct = ""
for char in mystr:
if char not in punctuations:
no_punct = no_punct + char
# display the unpunctuated string
print(no_punct)
Maybe:
patterns = [r"\w{3} \w{3} \d{2} \d{2}:\d{2}:\d{2} \d{4}\s*", #sun aug 19 13:02:10 2018
r"\w{3}, \d{2} \w{3} \d{4} \d{2}:\d{2}:\d{2} \w{2}\s*", #Sun, 19 Aug 2018 13:02:08 ET
r":\s*([\da-zA_Z]+\/)+([a-zA-Z0-9\.]+)", #URL
r"([a-zA-Z_!]+)[\.!_]\d+:\s*", #word[._!]number:>=0space
r":\d+",
"[/':,${}\[\]]" #punctuations
]
s = mystr
for p in patterns:
s = re.sub(p,'', s)
s = s.strip()
print(s)
Output:
hello please connect to the local host
hello not able to find the file
Base url for file_transfer is
var1=
responseCode = 400
responseDate =
responseContent = ABC
Error performing action failed with error code 400
I have installed the gitlab-omnibus version in my centos.It seems that everything is OK.But when a newer want to sign up, he/she must recieve the comfirm enalbe which send from gitlab.I have config the /etc/gitlab/gitlab.rb according to the gitlab's office document.
And I check my var/log/email,I found the message is as follow:
Apr 22 15:05:39 iZ23syflhhzZ postfix/cleanup[5899]: 9789EC3907: message-id=<201504220705.t3M75dkU005893#iZ23syflhhzZ>
Apr 22 15:05:39 iZ23syflhhzZ sendmail[5893]: t3M75dkU005893: to=postmaster, delay=00:00:00, xdelay=00:00:00, mailer=relay, pri=32576, relay=[127.0.0.1] [127.0.0.1], dsn=2.0.0, stat=Sent (Ok: queued as 9789EC3907)
Apr 22 15:05:39 iZ23syflhhzZ postfix/qmgr[5724]: 9789EC3907: from=<>, size=4301, nrcpt=1 (queue active)
Apr 22 15:05:39 iZ23syflhhzZ postfix/smtpd[5896]: disconnect from localhost[127.0.0.1]
Apr 22 15:05:39 iZ23syflhhzZ postfix/cleanup[5899]: A6D63C38FD: message-id=<201504220705.t3M75dkU005893#iZ23syflhhzZ>
Apr 22 15:05:39 iZ23syflhhzZ postfix/qmgr[5724]: A6D63C38FD: from=<>, size=4456, nrcpt=1 (queue active)
Apr 22 15:05:39 iZ23syflhhzZ postfix/local[5900]: 9789EC3907: to=<postmaster#iZ23syflhhzZ.localdomain>, orig_to=<postmaster#iZ23syflhhzZ>, relay=local, delay=0.1, delays=0.05/0.04/0/0.01, dsn=2.0.0, status=sent (forwarded as A6D63C38FD)
Apr 22 15:05:39 iZ23syflhhzZ postfix/qmgr[5724]: 9789EC3907: removed
Apr 22 15:05:40 iZ23syflhhzZ postfix/smtp[5902]: A6D63C38FD: to=<hiufan#qq.com>, orig_to=<postmaster#iZ23syflhhzZ>, relay=mx3.qq.com[113.108.16.62]:25, delay=0.73, delays=0.01/0.03/0.12/0.57, dsn=5.0.0, status=bounced (host mx3.qq.com[113.108.16.62] said: 550 Mail content denied. http://service.mail.qq.com/cgi-bin/help?subtype=1&&id=20022&&no=1000726 (in reply to end of DATA command))
Apr 22 15:05:40 iZ23syflhhzZ postfix/qmgr[5724]: A6D63C38FD: removed
It is very confused that I am not setting the email called hiufan#qq.com,I just setting the 066337#sina.com as the default sender email.
Here is my setting of /etc/gitlab/gitlab.rb:
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.sina.com"
gitlab_rails['smtp_port'] = 25
gitlab_rails['smtp_user_name'] = "066337#sina.com"
gitlab_rails['smtp_password'] = "xxxxxx"
gitlab_rails['smtp_domain'] = "sina.com"
gitlab_rails['smtp_authentication'] = "login"
gitlab_rails['smtp_enable_starttls_auto'] = true
gitlab_rails['smtp_tls'] = false
gitlab_rails['gitlab_email_from'] = "066337#sina.com"
user["git_user_email"] = "066337#sina.com"
Could anyone tell me why this happened?Thank you very much.
Look at the second last line of the log file
to=<hiufan#qq.com>, orig_to=<postmaster#iZ23syflhhzZ>, relay=mx3.qq.com[113.108.16.62]:25, delay=0.73, delays=0.01/0.03/0.12/0.57, dsn=5.0.0, status=bounced (host mx3.qq.com[113.108.16.62] said: 550 Mail content denied. (in reply to end of DATA command))
If hiufan#qq.com is the email address of the user registering, that
indicates that the message was sent by the gitlab machine but rejected by the receiving mail server (at qq.com):
status=bounced (host mx3.qq.com[113.108.16.62] said: 550 Mail content denied
Looking at the given link (using google translate)
http://service.mail.qq.com/cgi-bin/help?subtype=1&&id=20022&&no=1000726
it seems that it was rejected as it is identified as spam/bulk email. (I guess that if you do read chinese the error message makes more sense)
If so, I don't think there is much you can do at the gitlab side. You can perhaps try to edit the confirmation message template, but it is and automatically generated email and there is always a risk of a spam filter identifying it as such.
Gitlab confirmation emails can easily be identified as spam. And QQMail is really powerful in identifying and rejecting spam.
My suggestion would be changing the config of Gitlab, to stop needing confirmation email for registration.
I have a Jenkins post build Groovy script running out of the "Post build task plugin". From the same plugin, immediately before running the Groovy script, I check for the existence of the file and its size. The log shows:
09:14:53 -rw-r--r-- 1 aaa users 978243 Nov 4 08:53 /jk/workspace/xxxx/output/delta.txt
09:14:53 cppcheck.groovy: Checking build result: SUCCESS
09:14:53 cppcheck.groovy: workspace = /jk/workspace/xxxx
09:14:53 cppcheck.groovy: delta = /jk/workspace/xxxx/output/delta.txt
09:14:53 cppcheck.groovy: delta.txt length = 0
The groovy script is as follows:
import hudson.model.*
def build = Thread.currentThread().executable
def result = build.getResult()
println("cppcheck.groovy: Checking build result: " + result.toString())
if (result.isBetterOrEqualTo(hudson.model.Result.SUCCESS)) {
def workspace = build.getEnvVars()["WORKSPACE"]
def delta = workspace + "/output/delta.txt"
println("cppcheck.groovy: workspace = " + workspace)
println("cppcheck.groovy: delta = " + delta)
def f = new File(delta)
println("cppcheck.groovy: delta.txt length = " + f.length())
if (f.length() > 0) {
build.setResult(hudson.model.Result.UNSTABLE)
}
}
What am I doing wrong here?
Update: There seems to be some scepticism that the file exists and that there is some sort of race condition. To put your minds at rest, let's rule that out. I have modified the build to execute the same ls -l command after it runs the groovy script, to prove the file does exist and that this problem is ultimately Groovy not being able to open the file. I also added the file exists() check to the above Groovy script, which as I suspected it would, reports the file doesn't exist. I don't dispute that Groovy thinks the file doesn't exist. What I am trying to work out is why?
10:31:39 [xxxx] $ /bin/sh -xe /tmp/hudson8964729240493636268.sh
10:31:39 + ls -l /jk/workspace/xxxx/output/delta.txt
10:31:39 -rw-r--r-- 1 aaa users 978243 Nov 4 08:53 /jk/workspace/xxxx/output/delta.txt
10:31:40 cppcheck.groovy: Checking build result: SUCCESS
10:31:40 cppcheck.groovy: workspace = /jk/workspace/xxxx
10:31:40 cppcheck.groovy: delta = /jk/workspace/xxxx/output/delta.txt
10:31:40 cppcheck.groovy: delta.txt length = 0
10:31:40 cppcheck.groovy: delta.txt exists = false
10:31:40 [xxxx] $ /bin/sh -xe /tmp/hudson8007562636561507409.sh
10:31:40 + ls -l /jk/workspace/xxxx/output/delta.txt
10:31:40 -rw-r--r-- 1 aaa users 978243 Nov 4 08:53 /jk/workspace/xxxx/output/delta.txt
Also, notice the timestamp on said file, is still 08:53 when it was created.
I suspected that the Groovy script was running on the build master as opposed to the build node that this particular build was running on. I added some debug to print the hostname for which the Groovy script was running and sure enough it wasn't the same host that the shell variant of the script was running.
I am porting an old mathematical model (between 1995 to 2000) to a current linux machine. For this, I adapted all makefiles as shown:
FORTRAN = gfortran # f90 -f77 -ftrap=%none
OPTS = -O -u -lgfortran -g -fconvert="big-endian" # -O -u
NOOPT =
LOADER = gfortran #f90
LOADOPTS = #-lf77compat
and:
SYSFFLAGS = -O0 -u -g -fconvert="big-endian" # -f77=input
SYSCFLAGS = -DX_WCHAR
SYSLDFLAGS =
SYSCPPFLAGS = -DSYS_UNIX -DCODE_ASCII -DCODE_IEEE # -DSYS_Sun
SYSAUTODBL = -fdefault-real-8 #-r8
SYSDEBUG = -g
SYSCHECK = -C
LINKOPT =
CPPOPT =
SHELL = /bin/sh
CC = cc
FC = gfortran # f90
LD = gfortran # f90
AR = ar vru
RM = rm -f
CP = cp
MV = mv -f
LN = ln -s
So I replaced all the outdated compilers/options to be able to compile the code. After that, It generates the binaries with no error. Note that all option behind the # symbol where the original options in the Makefiles.
So, when running the program, it is not possible to read the sample data. IMO those files were created as unformatted sequence mode on a Sun machine. The following hex dump belongs to the file that I need to read.
0000000: 0000 0400 2020 2020 2020 2020 2020 2020 ....
0000010: 3930 3130 7465 7374 2d63 3031 2020 2020 9010test-c01
0000020: 2020 2020 4741 5520 2020 2020 2020 2020 GAU
0000030: 2020 2020 2020 2020 2020 2020 2020 2020
0000040: 2020 2020 2020 2020 2020 2020 2020 2020
0000050: 2020 2020 2020 2020 2020 2020 2020 2020
...
...
0000390: 2020 2020 2020 2020 2020 2020 2020 2020
00003a0: 2020 2020 2020 2020 2020 2020 2020 2020
00003b0: 2020 2020 3139 3936 3037 3232 2032 3030 19960722 200
00003c0: 3434 3920 4147 434d 352e 3420 2020 2020 449 AGCM5.4
00003d0: 2020 2020 3230 3030 3036 3134 2031 3230 20000614 120
00003e0: 3831 3720 6869 726f 2020 2020 2020 2020 817 hiro
00003f0: 2020 2020 2020 2020 2020 2020 2020 2034 4
0000400: 3039 3630 0000 0400 0002 8000 bef7 21f3 0960..........!.
0000410: bf3c 55ab bf7a 8f71 bf99 e26a bfb2 db4e .<U..z.q...j...N
0000420: bfc7 425f bfd6 64b1 bfdf d44f bfe3 6a43 ..B_..d....O..jC
After analyzing the code, it is able to read until the line 0000410. It is not possible to continue after the mark 0002 8000. The source code shown below is actually reading this file.
...
* [INPUT]
INTEGER IFILE
CHARACTER HITEM *(*) !! name for identify
CHARACTER HDFMT *(*) !! data format
*
* [ENTRY INPUT]
REAL * 8 TIME1 !! time
REAL * 8 TIME2 !! time
REAL*8 DMIN
REAL*8 DMAX
REAL*8 DIVS
REAL*8 DIVL
INTEGER ISTYPE
INTEGER JFILE !! output file No.
INTEGER IMAXD
INTEGER JMAXD
*
* [WORK]
REAL * 8 DDATA ( NGDWRK )
REAL * 4 SDATA ( NGDWRK )
*
* [INTERNAL WORK]
INTEGER I, J, K, IJK, IJKNUM, IERR
...
...
READ ( IFILE, IOSTAT=IEOD ) HEAD
...
...
...
DO 2150 IJK = 1, IJKNUM
READ ( IFILE, END=2150 ) SDATA(IJK)
WRITE (6,*) ' IGTIO::GTZZRD: iteration=', IJK, SDATA(IJK)
2150 CONTINUE
In order to easily debug the loop I replaced for the one above. The original one is implicit.
READ ( IFILE, IOSTAT=IEOD)
& (SDATA(IJK), IJK=1, IJKNUM )
And the output for the loop is:
IGTIO::GTZZRD: iteration= 1 -0.48268089
IGTIO::GTZZRD: iteration= 2 1.35631564E-19
IGTIO::GTZZRD: iteration= 3 -0.48142704
IGTIO::GTZZRD: iteration= 4 1.35631564E-19
IGTIO::GTZZRD: iteration= 5 244.25270
IGTIO::GTZZRD: iteration= 6 1.35631564E-19
IGTIO::GTZZRD: iteration= 7 983.87988
IGTIO::GTZZRD: iteration= 8 1.35631564E-19
IGTIO::GTZZRD: iteration= 9 1.59284362E-04
IGTIO::GTZZRD: iteration= 10 1.35631564E-19
IGTIO::GTZZRD: iteration= 11 0.0000000
---error here---
I am definately lost on this, so any help is appreciated.
Here is whats going on - first this This is definitely a Big Enfian file.
The first 4 bytes
00000400
are the big end 4 byte integer 1024, which is the length of your first record.
which agrees with the length of HEAD (per comment)
Now note that 00000400 is repeated at byte position 1024+4 exactly (hex dump line 400) as you should expect for a fortran unformatted file...so far so good.
Now the next 4 bytes
0002 8000
begin the second record. (edit correcting mistake) This is 163840 (2*16^4+8*16^3) You should find that repeated at position 1024+8+163840+4 in the hex dump. (should be line 028400 i think..)
Here is the problem: in your code you are reading that 160 kilobyte record into a single 4 byte variable, then moving on to the next record. My guess you are seeing that alternating 10^-19 because every other record is of type character.
In unformatted fortran you must read a whole record in one shot - try reading the entire array (without the loop..)
READ ( IFILE )SDATA
assuming sdata is dimensioned to hold 160 kb of course. (eg. real*4 (40960) )
The answer to your problem is in the edit that I missed. Thanks also to george's arithmetic---which I hadn't bothered to do.
We can safely say that the record headers are correct, and that you can solve your problem with the endian conversion.
So, the problem is: reads with an implied-do loop are not equivalent to reads inside a do loop.
That is: read(unit) (i(j), j=1,5) is not the same as
do j=1,5
read(unit) i(j)
end do
In the first, reading of the five values is from one record, in the second each is read from a distinct record.
You should, then, revert your change. If you want to do the same diagnostics, however, you can do something like
READ ( IFILE, IOSTAT=IEOD) (SDATA(IJK), IJK=1, IJKNUM )
WRITE (6, '("IGTIO::GTZZRD: iteration='", I0.0, F12.8)') (IJK, SDATA(IJK), IJK=1,IJKNUM)
Out of this scope, for some reason a deep method in charge of reading the file is called in the 1100 loop. That was causing to read the file more times than necessary. Find below the fixed code:
* 1100 CONTINUE
CALL GDREDX !! read data
O ( GDATA , IEOD ,
O HITEMD, HTITL , HUNIT , HDSET ,
O TIME , TDUR , KLEVS ,
I IFILE , HITEM , HDFMT ,
I IMAXD , JMAXD ,
I IDIMD , JDIMD , KDIMD )
IF ( IEOD .EQ. 0 ) THEN
WRITE (6,*) ' IRWGD.F::GDRDTS: TSEL0=', TSEL0
WRITE (6,*) ' IRWGD.F::GDRDTS: TSEL1=', TSEL1
WRITE (6,*) ' IRWGD.F::GDRDTS: TIME=', TIME
* IF ( ((TSEL0.GE.0).AND.(TIME.LT.TSEL0))
* & .OR.((TSEL1.GE.0).AND.(TIME.GT.TSEL1)) ) THEN
* GOTO 1100
* ENDIF
ENDIF
*
RETURN
END
That mislead me in order to figure out what was going on.