For example... Say I am running a script with Forever https://www.npmjs.com/package/forever for a week while recording a log file of the node applications output.
If I say included colors would it make that filesize bigger? Dealing with crazy sizes logs 5gb+ with the colors on. So curious if I could shave even 10mb without it?
{
pass: [0,255,0],
fail: [255,0,0],
info: [0,255,255],
warning: [255,127,80]
}
You're storing more characters to colorize log output, so yes, you will increase the log size (more data == more data). For example, check out these source lines from chalk tests:
it('should style string', function () {
// Notice all the extra characters
assert.equal(chalk.underline('foo'), '\u001b[4mfoo\u001b[24m');
assert.equal(chalk.red('foo'), '\u001b[31mfoo\u001b[39m');
assert.equal(chalk.bgRed('foo'), '\u001b[41mfoo\u001b[49m');
});
If you absolutely need the colors for readability, so be it. But if you do w/o you can shave off some space, but there's no guarantee it'll be in the order of 10MB :)
Another thing to note is that depending on where you're reading the logs, the color may or may not come through properly. I've run into this when looking at some raw logs on AWS. The colorized portions were pretty mangled.
Related
I started using fmt for printing recently. I really like the lib, fast, easy to use. But when I completed my conversion, there are ways that my program can run that will render with a bunch of additional newlines. It's not every case, so this will get a bit deep.
What I have is a compiler and a build manager. The build manager (picture Ninja, although this is a custom tool) launches compile processes, buffers the output, and prints it all at once. Both programs have been converted to use fmt. The key function being called is fmt::vprint(stream, format, args). When the build manager prints directly, things are fine. But when I'm reading the child process output, any \n in the data has been prefixed with \r. Windows Terminal will render that fine, but some shells (such as the Visual Studio output window) do not, and will show a bunch of extra newlines.
fmt is open source so I was able to hack on it a bunch and see what is different between what it did and what my program was doing originally. The crux is this:
namespace detail {
FMT_FUNC void print(std::FILE* f, string_view text) {
#ifdef _WIN32
auto fd = _fileno(f);
if (_isatty(fd)) {
detail::utf8_to_utf16 u16(string_view(text.data(), text.size()));
auto written = detail::dword();
if (detail::WriteConsoleW(reinterpret_cast<void*>(_get_osfhandle(fd)),
u16.c_str(), static_cast<uint32_t>(u16.size()),
&written, nullptr)) {
return;
}
// Fallback to fwrite on failure. It can happen if the output has been
// redirected to NUL.
}
#endif
detail::fwrite_fully(text.data(), 1, text.size(), f);
}
} // namespace detail
As a child process, the _isatty() function will come back with false, so we fall back to the fwrite() function, and that triggers the \r escaping. In my original program, I have an fwrite() fallback as well, but it only picks up if GetStdHandle(STD_OUTPUT_HANDLE) returns nullptr. In the child process case, there is still a console we can WriteFile() to.
The other side-effect I see happening is if I use the fmt way of injecting color, eg:
fmt::print(fmt::emphasis::bold | fg(fmt::color::red), "Elapsed time: {0:.2f} seconds", 1.23);
Again Windows Terminal renders it correctly, but in Visual Studio's output window this turns into a soup of garbage. The native way of doing it -- SetConsoleTextAttribute(console, FOREGROUND_RED | FOREGROUND_GREEN | FOREGROUND_INTENSITY);-- does not trigger that problem.
I tried hacking up the fmt source to be more like my original console printing code. The key difference was the _isatty() function. I suspect that's too broad of a question for the cases where console printing might fail.
\r is added because the file is opened in text mode. You could try (re)opening in binary mode or ignore \r on the read side.
I am using container engine, and my pods are hosted there.
I am trying to fetch logs, using log api :
http://localhost:8000/api/v1/namespaces/app-test/pods/designer-0/log?tailLines=100&sinceTime=2017-09-17T10:47:58Z
if i used both the query params separately, it works and show the proper result, but if i am using it simultaneously only the top 100 logs are returning, the sinceTime param is get ignored.
my scenario is, i need a log from a specific time, in a chunk like, 100 lines, 100 lines.. like this.
I am not sure, whether it is a bug, or it is not implemented.
I found this from the api reference manual
https://kubernetes.io/docs/api-reference/v1.6/
tailLines - If set, the number of lines from the end of the logs to
show. If not specified, logs are shown from the creation of the
container or sinceSeconds or sinceTime
So, that means if you specify tailLines, it start from the end. I dont see any option explicitly mentioned other than limitBytes. But you will have to play around with it as it does not guarantee number of lines.
tailLines=X tells the server to start that many lines from the end
sinceTime tells the server to start from the specified time
the options are mutually exclusive
Thanks All,
I have later on recognized that, it is not ignoring the sinceTime, as the TailLines intended functionality is return the lines from the last.
So, if i mentioned the sinceTime= 10 PM yesterday, it will return the records from that time..And if also tailLines, is mentioned, so it will return the recent logs from that chunk.
So, it was working as expected. I need to play with LimitBytes for getting the logs in chunk, from that time, Instead of full logs.
I don't find a way to check the free space available in a device using Haxe, Openfl, Lime or another library.
I would like to avoid download data that will exceed the size recommended for an app in each device.
What do you do to check that?
Try creating a file of that size! Then either delete it or reopen and write (not append) over its contents.
I don't know whether all platforms Haxe supports will work fine with this trick, but this algorithm is reported to work in many places and languages (I personally tested it in Ruby and saw the same suggestion for C++/.NET). To check whether X bytes of disk space are available:
open a new file for writing
seek X-1 bytes from the beginning
write a byte of data (whatever you want, 0, 42...)
close the file (probably unrelated to the task at hand, but don't forget to do that anyway)
If there's insufficient disk space, you'll likely get an exception at some point in this algorithm. You'll have to find out what errors to expect and process them properly.
Using ihx I've found this is working and requires nothing but Haxe Standard Library:
haxe interactive shell v0.3.4
type "help" for help
>> import sys.io.*;
>> var f = File.write('loca', true)
sys.io.FileOutput : { __f => #abstract }
>> f.seek(39999, FileSeek.SeekBegin)
Void : null
>> f.writeByte(0)
Void : null
>> f.close()
Void : null
After these manipulations, I had a file named loca of exactly 40000 bytes in my working directory.
By the way, be careful when doing things like these in ihx since it re-runs the entire session with the last entered line appended each time.
Ongoing experimentation:
However, when there's insufficient disk space, it may not fail with errors. In this case you'll have to check the real size with sys.FileSystem.stat(path).size. And don't forget to delete the file if there's not enough space.
I've got question about program architecture.
Say you've got 100 different log files with different formats and you need to parse and put that info into an SQL database.
My view of it is like:
use general config file like:
program1->name1("apache",/var/log/apache.log) (modulename,path to logfile1)
program2->name2("exim",/var/log/exim.log) (modulename,path to logfile2)
....
sqldb->configuration
use something like a module (1 file per program) type1.module (regexp, logstructure(somevariables), sql(tables and functions))
fork or thread processes (don't know what is better on Linux now) for different programs.
So question is, is my view of this correct? I should use one module per program (web/MTA/iptablat)
or there is some better way? I think some regexps would be the same, like date/time/ip/url. What to do with that? Or what have I missed?
example: mta exim4 mainlog
2011-04-28 13:16:24 1QFOGm-0005nQ-Ig
<= exim#mydomain.org.ua** H=localhost
(exim.mydomain.org.ua)
[127.0.0.1]:51127 I=[127.0.0.1]:465
P=esmtpsa
X=TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32
CV=no A=plain_server:spam S=763
id=1303985784.4db93e788cb5c#mydomain.org.ua T="test" from
<exim#exim.mydomain.org.ua> for
test#domain.ua
everything that is bold is already parsed and will be putted into sqldb.incoming table. now im having structure in perl to hold every parsed variable like $exim->{timstamp} or $exim->{host}->{ip}
my program will do something like tail -f /file and parse it line by line
Flexability: let say i want to add supprot to apache server (just timestamp userip and file downloaded). all i need to know what logfile to parse, what regexp shoud be and what sql structure should be. So im planning to have this like a module. just fork or thread main process with parameters(logfile,filetype). Maybe further i would add some options what not to parse (maybe some log level is low and you just dont see mutch there)
I would do it like this:
Create a config file that is formatted like this: appname:logpath:logformatname
Create a collection of Perl class that inherit from a base parser class.
Write a script which loads the config file and then loops over its contents, passing each iteration to its appropriate handler object.
If you want an example of steps 1 and 2, we have one on our project. See MT::FileMgr and MT::FileMgr::* here.
The log-monitoring tool wots could do a lot of the heavy lifting for you here. It runs as a daemon, watching as many log files as you could want, running any combination of perl regexes over them and executing something when matches are found.
I would be inclined to modify wots itself (which its licence freely allows) to support a database write method - have a look at its existing handle_* methods.
Most of the hard work has already been done for you, and you can tackle the interesting bits.
I think File::Tail is a nice fit.
You can make an array of File::Tail objects and poll them with select like this:
while (1) {
($nfound,$timeleft,#pending)=
File::Tail::select(undef,undef,undef,$timeout,#files);
unless ($nfound) {
# timeout - do something else here, if you need to
} else {
foreach (#pending) {
# here you can handle log messages depending on filename
print $_->{"input"}." (".localtime(time).") ".$_->read;
}
(from perl File::Tail doc)
On writing to the display with:
::TextOutW( pDC->m_hDC, x, y, &Out, 1 );
It only shows on the screen after every 15 calls (15 characters).
For debugging purposes only, I would like to see the new character on the display after each call. I have tried ::flushall() and a few other things but no change.
TIA
GDI function calls are accumulated and called in batches for performance reasons.
You can call GdiFlush after the TextOut call to perform the drawing immediately. Alternatively, call GdiSetBatchLimit(1) before outputting the text to disable batching completely.
::flushall() is for iostreams, so it won't affect Windows screen output at all. I've never tried it, but based on the docs, I believe GDIFlush() might be what you want. You should also be able to use GDISetBatchLimit(1); to force each call to run immediately upon being called.