I'm looking for something similar to BLPOP, but instead of element I want to get them all run running over them in a loop.
It means that I want to get all the records of redis collection, and truncate it.
Consider using a LUA script to do the LRANGE+DEL atomically.
Or use RENAME to move the list to a temporary key which you will use to process the data.
RENAME yourlist temp-list
LRANGE temp-list 0 -1
... process the list
DEL temp-list
Related
I want to just see the total number of keys available in Azure Redis cache that matches the given pattern. I tried the following command it is showing the count after displaying all the keys (which caused server load), But I need only the count.
>SCAN 0 COUNT 10000000 MATCH "{UID}*"
Except command SCAN, the command KEYS pattern can return the same result as your current command SCAN 0 COUNT 10000000 MATCH "{UID}*".
However, for your real needs to get the number of keys matching a pattern, there is an issue add COUNT command from the Redis offical GitHub repo which had answered by the author antirez for you, as the content below.
Hi, KEYS is only intended for debugging since it is O(N) and performs a full keyspace scan. COUNT has the same problem but without the excuse of being useful for debugging... (since you can simply use redis-cli keys ... | grep ...). So feature not accepted. Thanks for your interest.
So you can not directly get the count of KEYS pattern, but there are some possible solutions for you.
Count the keys return from command KEYS pattern in your programming language for the small number of keys with a pattern, like doing redis-cli KEYS "{UID}*" | wc -l on the host server of Redis.
To use the command EVAL script numkeys key \[key ...\] arg \[arg ...\] to run a Lua script to count the keys with pattern, there are two scripts you can try.
2.1. Script 1
return #redis.call("keys", "{UID}*")
2.2. Script 2
return table.getn(redis.call('keys', ARGV[1]))
The completed command in redis-cli is EVAL "return table.getn(redis.call('keys', ARGV[1]))" 0 {UID}*
I would like to extract a range of keys from either leveldb or redis. For example i have the following key structure;
group:1/member:1
group:1/member:1/log:1
group:1/member:1/log:2
group:1/member:1/log:3
group:1/member:1/log:4
group:1/member:2
group:1/member:2/log:1
group:1/member:2/log:2
group:1/member:3
group:1/member:3/log:1
I would like to get all members(members:1, members:2, members:3) but i do not want their log entries to be included in results(there may be thousands of logs). What is the best approach to achieving this using a KV store like redis or leveldb?
For LevebDB, you can use the leveldb::Iterator to iterate the key space, and only keep the keys that match your pattern.
For Redis, you can use the SCAN command to scan the key space with a pattern.
I've created Oracle trigger which execute external python file through DBMS_SCHEDULER.RUN_JOB() but it executes python file first then insert row into table.I want exactly opposite operation.
CREATE OR REPLACE TRIGGER sample
AFTER INSERT ON client
BEGIN
EXEC DBMS_SCHEDULER.RUN_JOB("JOB CONTAN PYTHON FILE");
END;
Tell me right way to do this
There's a difference between the row(s) being inserted into the table and those rows being visible to another session. Until the data is committed then those inserted rows cannot be seen by any other transaction. If your python code tries to connect to the database to look at those rows, it won't see them.
Equally, your transaction can't report back to the client (in your case SQL Developer) that the insert succeeded until the trigger has completed. In this case it needs to wait until the python call has completed before returning.
Generally triggers are considered 'bad practice', though they do have some good applications. Having a session wait on an external task is also something to avoid. I'd recommend you rethink your approach to whatever you are trying to achieve.
How do you know that?
Think!
you have a table
there's a trigger on that table
trigger fires when you insert data into the table and ...
... calls the Python script
So, how can that script run before inserting a row, if exactly that action - a row being inserted - triggers & runs the Python script?
Unless you prove me wrong, everything should be OK.
My development instance of Accumulo became quite messy with a lot of tables created for testing.
I would like to bulk delete a large number of tables.
Is there a way to do it other than deleting the entire instance?
BTW - If it's of any relevance, this instance is just a single machine "cluster".
In the Accumulo shell, you can specify a regular expression for table names to delete by using the -p option of the deletetable command.
I would have commented on original answer, but I lack the reputation (first contribution right here).
It would have been helpful to provide a legal regex example.
The Accumulo shell can only escape certain characters. In particular it will not escape brackets []. If you want to remove every table starting with the string "mytable", the otherwise legal regex commands have the following warning/error.
user#instance> deletetable -p mytable[.]*
2016-02-18 10:21:04,704 [shell.Shell] WARN : No tables found that match your criteria
user#instance> deletetable -p mytable[\w]*
2016-02-18 10:21:49,041 [shell.Shell] ERROR: org.apache.accumulo.core.util.BadArgumentException: can only escape single quotes, double quotes, the space character, the backslash, and hex input near index 19
deletetable -p mytable[\w]*
A working shell command would be:
user#instance> deletetable -p mytable.*
There is not currently (as of version 1.7.0) a way to bulk delete many tables in a single call.
Table deletion is actually done in an asynchronous way. The client submits a request to delete the table, and that table will be deleted at some point in the near future. The problem is that after the call to delete the table is performed, the client then waits until the table is deleted. This blocking is entirely artificial and unnecessary, but unfortunately that's how it currently works.
Because each individual table deletion appears to block, a simple loop over the table names to delete them serially is not going to finish quickly. Instead, you should use a thread pool, and issue delete table requests in parallel.
A bulk delete table command would be very useful, though. As an open source project, a feature request on their issue tracker would be most welcome, and any contributions to implement it, even more so.
I've got question about program architecture.
Say you've got 100 different log files with different formats and you need to parse and put that info into an SQL database.
My view of it is like:
use general config file like:
program1->name1("apache",/var/log/apache.log) (modulename,path to logfile1)
program2->name2("exim",/var/log/exim.log) (modulename,path to logfile2)
....
sqldb->configuration
use something like a module (1 file per program) type1.module (regexp, logstructure(somevariables), sql(tables and functions))
fork or thread processes (don't know what is better on Linux now) for different programs.
So question is, is my view of this correct? I should use one module per program (web/MTA/iptablat)
or there is some better way? I think some regexps would be the same, like date/time/ip/url. What to do with that? Or what have I missed?
example: mta exim4 mainlog
2011-04-28 13:16:24 1QFOGm-0005nQ-Ig
<= exim#mydomain.org.ua** H=localhost
(exim.mydomain.org.ua)
[127.0.0.1]:51127 I=[127.0.0.1]:465
P=esmtpsa
X=TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32
CV=no A=plain_server:spam S=763
id=1303985784.4db93e788cb5c#mydomain.org.ua T="test" from
<exim#exim.mydomain.org.ua> for
test#domain.ua
everything that is bold is already parsed and will be putted into sqldb.incoming table. now im having structure in perl to hold every parsed variable like $exim->{timstamp} or $exim->{host}->{ip}
my program will do something like tail -f /file and parse it line by line
Flexability: let say i want to add supprot to apache server (just timestamp userip and file downloaded). all i need to know what logfile to parse, what regexp shoud be and what sql structure should be. So im planning to have this like a module. just fork or thread main process with parameters(logfile,filetype). Maybe further i would add some options what not to parse (maybe some log level is low and you just dont see mutch there)
I would do it like this:
Create a config file that is formatted like this: appname:logpath:logformatname
Create a collection of Perl class that inherit from a base parser class.
Write a script which loads the config file and then loops over its contents, passing each iteration to its appropriate handler object.
If you want an example of steps 1 and 2, we have one on our project. See MT::FileMgr and MT::FileMgr::* here.
The log-monitoring tool wots could do a lot of the heavy lifting for you here. It runs as a daemon, watching as many log files as you could want, running any combination of perl regexes over them and executing something when matches are found.
I would be inclined to modify wots itself (which its licence freely allows) to support a database write method - have a look at its existing handle_* methods.
Most of the hard work has already been done for you, and you can tackle the interesting bits.
I think File::Tail is a nice fit.
You can make an array of File::Tail objects and poll them with select like this:
while (1) {
($nfound,$timeleft,#pending)=
File::Tail::select(undef,undef,undef,$timeout,#files);
unless ($nfound) {
# timeout - do something else here, if you need to
} else {
foreach (#pending) {
# here you can handle log messages depending on filename
print $_->{"input"}." (".localtime(time).") ".$_->read;
}
(from perl File::Tail doc)