This is probably more about producers. There is a need to define appropriate ACL for each resource in use by clients if allow.everyone.if.no.acl.found=false is about to be set, which should be desired state anyway, so, how can one monitor this to find out what is actually needed, without developer's help?
I guess currently there is no better answer to that question than this:
Reading data from _transaction_state topic in Kafka 0.11.0.1
To improve it a bit, let us post the link to the relevant class, namely this:
https://github.com/a0x8o/kafka/blob/master/core/src/main/scala/kafka/coordinator/transaction/TransactionLog.scala
If I compare my result of proposed kafka-console-consumer command with TransactionLogMessageFormatter applied to it, to the implementation presented there in readTxnRecordValue function, I see that I miss some TransactionMetadata, my record looks something like this:
txn_id::TransactionMetadata(transactionalId=txn_id, producerId=440243, producerEpoch=14094, txnTimeoutMs=600000, state=Empty, pendingState=None, topicPartitions=Set(), txnStartTimestamp=-1, txnLastUpdateTimestamp=1613483758593)
So, no lastProducerId and lastProducerEpoch, probably due to the older version that I am running (kafka_2.13-2.6.0)?
Related
I am using Python 3.8.10 and fabric 2.7.0.
I have a Connection to a remote host. I am executing a command such as follows:
resObj = connection.run("cat /usr/bin/binaryFile")
So in theory the bytes of /usr/bin/binaryFile are getting pumped into stdout but I can not figure out what wizardry is required to get them out of resObj.stdout and written into a file locally that would have a matching checksum (as in, get all the bytes out of stdout). For starters, len(resObj.stdout) !== binaryFile.size. Visually glancing at what is present in resObj.stdout and comparing to what is in /usr/bin/binaryFile via hexdump or similar makes them seem about similar, but something is going wrong.
May the record show, I am aware that this particular example would be better accomplished with...
connection.get('/usr/bin/binaryFile')
The point though is that I'd like to be able to get arbitrary binary data out of stdout.
Any help would be greatly appreciated!!!
I eventually gave up on doing this using the fabric library and reverted to straight up paramiko. People give paramiko a hard time for being "too low level" but the truth is that it offers a higher level API which is pretty intuitive to use. I ended up with something like this:
with SSHClient() as client:
client.set_missing_host_key_policy(AutoAddPolicy())
client.connect(hostname, **connectKwargs)
stdin, stdout, stderr = client.exec_command("cat /usr/bin/binaryFile")
In this setup, I can get the raw bytes via stdout.read() (or similarly, stderr.read()).
To do other things that fabric exposes, like put and get it is easy enough to do:
# client from above
with client.open_sftp() as sftpClient:
sftpClient.put(...)
sftpClient.get(...)
also was able to get the exit code per this SO answer by doing:
# stdout from above
stdout.channel.recv_exit_status()
The docs for recv_exit_status list a few gotchas that are worth being aware of too. https://docs.paramiko.org/en/latest/api/channel.html#paramiko.channel.Channel.recv_exit_status .
Moral of the story for me is that fabric ends up feeling like an over abstraction while Paramiko has an easy to use higher level API and also the low level primitives when appropriate.
I'm more of a HW engineer who's currently trying to use Python at work. What I want to accomplish via Python is read the CAN-FD output from the DUT and use it for monitoring purposes in the measurement setup. But, I think I didn't get the correct result. Because it shows the same message(id) even there so much more. Based on my understanding from other examples, this should shows the stream of all the messages since there was no filters. Is there anyone who can help me solve this issue or have the similar experience?
import can
def _get_message(msg):
return msg
bus = can.interface.Bus(bustype='vector',app_name ='app', channel=1, bitrate=500000)
buffer = can.BufferedReader()
can.Notifier(bus,[_get_message,buffer])
while True:
msgs = bus.recv(None)
print(msgs)
You don't say what you expected to see on your bus, but I assume there are a lot of different messages on it.
Simplify your problem to start with - set up a bus which has only a single message on it at a low rate. You might have to write some code for that, or you might get some tools which can send periodic messages with counters in for example. Make sure you can capture that correctly first. Then add a second message at a faster rate.
You will proably learn a selection of useful things from these exercises which mean that when you go back to your full-system you have more success, or at least have more ideas on what to debug :)
First of all, thanks for the answers and comments. the root cause was the incorrect HW configuration for the interface. I thought if HW configuration is wrong in the first place, there will be no output at all. But it turned out it is not. After proper configuration, I could see the output stream I expected.
TL;DR
I'm browsing through a number of solutions on npm and github looking for something that would allow me to read and write to the same file in two different places at the same time. So far I'm having trouble actually finding anything like this. Is there a module of some sort that will allow that?
Background
In essence my requirement is that in a large file I need to, in the following order:
read
transform
write
Ideally the usage would be something like:
const fd = fs.open(file, "r+");
const read = createReadStreamSomehowFrom(fd);
const write = createWriteStreamSomehowFrom(fd);
read
.pipe(new Transform(transform() {...}))
.pipe(write);
I could do that with standard fs.create[Read/Write]Stream but there's no way to control the flow of both streams and if my write position goes beyond read position then I'm reading something I just wrote...
The use case is the same as perl -p -i -e, read and write to the same file (meaning the same inode) asynchronously and replace the contents without loading everything into memory.
I would expect this a real world use case, yet all implementations I found actually load the whole file into memory and then save it. Am I missing a known module here or is there a need to actually write something like this?
Hmm... a tough one it seems. :)
So here's for the record - I found no such module and actually discussed this with some people responsible for a nice in-file replacing module. Seeing no way to solve this I decided to write it from scratch and here it is:
signicode/rw-stream repo on github
rw-stream at npm
The module works on a simple principle that no byte can be written until it has been consumed in the readable stream and it's fairly simple underneath (couple fs.read/write ops with keeping eye on the point of read and write).
If you find this useful then I'm happy. :)
Using uClinux we have one of two flash devices installed, a 1GB flash or a 2GB flash.
The only way I can think of solving this is to somehow get the device ID - which is down the in the device driver code, for me that is in:
drivers/mtd/devices/m25p80.c
I have been using the command mtdinfo (which comes from mtdutils binaries, derived from mtdinfo.c/h). There is various information stored in here about the flash partitions including flash type 'nor' eraseblock size '65536', etc. But nothing that I can identify the chip with.
Its not very clear to me how I can get information from "driver-land" into "user-land". I am looking at trying to extend the mtdinfo command to print more information but there are many layers...
What is the best way to achieve this?
At the moment, I have found no easy way to do this without code changes. However I have found an easy code change (probably a bit of a hack) that allows me to get the information I need:
In the relevant file (in my case drivers/mtd/devices/m25p80.c) you can call one of the following:
dev_err("...");
dev_alert("...");
dev_warn("...");
dev_notice("...");
_dev_info("...");
Which are defined in include/Linux/device.h, so they are part of the Linux driver interface so you can use them from any driver.
I found that the dev_err() and devalert() both get printed out "on screen" during run time. However all of these device messages can be found in /var/log/messages. Since I added messages in the format: dev_notice("JEDEC id %06x\n", jedecid);, I could find the device ID with the following command:
cat /var/log/messages | grep -i jedec
Obviously using dev_err() ordev_alert() is not quite right! - but dev_notice() or even _dev_info() seem more appropriate.
Not yet marking this as the answer since it requires code changes - still hoping for a better solution if anyone knows of one...
Update
Although the above "solution" works, its a bit crappy - certainly will do the job and good enough for mucking around. But I decided that if I am making code changes I may as well do it properly. So I have now implemented changes to add an interface in sysfs such that you can get the flash id with the following command:
cat /sys/class/m25p80/m25p80_dev0/device_id
The main function calls required for this are (in this order):
alloc_chrdev_region(...)
class_create(...)
device_create(...)
sysfs_create_group(...)
This should give enough of a hint for anyone wanting to do the same, though I can expand on that answer if anyone wants it.
Goal
I would like to stitch up a GNU GPL licensed Knot Resolver module either in C or in CGO that would examine the client's query and the corresponding resolved answer with the goal of querying an external API offering a knowledge base of malware infected hostnames and ip addresses (e.g. GNU AGPL v3 IntelMQ).
If there is a match with the resolved A's (AAAA's) IP address it is to be logged, likewise a match with the queried hostname should be logged or (optionally) it could result in sending the client an IP address of a sinkhole instead of the resolved one.
Means
I studied the layers and I came to the conclusion that the phase I'm interested in is consume. I don't want to affect the resolution process, I just want to step in at the last moment and check the results and possibly modify them.
I ventured to register the a consume function
with
static knot_layer_api_t _layer = {
.consume = &consume,
};
but I'm not sure it is the right place to do the deed.
Furthermore, I also looked into module hints.c, especially its query method
and module stats.c for its _to_wire function usage.
Question(s)
Phase (Layer?)
When is the right time to step in and read/write the answer to the query before it's send to the client? Am I at the right spot in consume layer?
Answer sections
If the following attempt at getting the resolved IP address gives me the Name Server's address:
char addr_str[INET6_ADDRSTRLEN];
memset(addr_str, 0, sizeof(addr_str));
const struct sockaddr *src = &(req->answer->sections);
inet_ntop(qry->ns.addr[0].ip.sa_family, kr_inaddr(src), addr_str, sizeof(addr_str));
DEBUG_MSG(NULL, "ADDR: %s\n", addr_str);
how do I get the resolved (A, AAAA) IP address for the query's hostname? I would like to iterate over A/AAAA IP addresses and CNAMEs in the answer and look at the IP addresses they were resolved to.
Modifying the answer
If the module setting demands it, I would like to be able to "ditch" the resolved answer and provide a new one comprising an A record pointed at a sinkhole.
How do I prepare the record so as it could be translated from char* to Knot's wire format and the proper structure in the right context at the right phase?
I guess it might go along functions such as knot_rrset_init and knot_rrset_add_rdata, but I wasn't able to arrive at any successful result.
THX for pointers and suggestions.
If you want to step in the last moment when the response is finalised but not yet sent to the requestor, the right place is finish. You can do it in consume as well, but you'll be overwriting responses from authoritative servers here, not the assembled response to requestor (which means DNSSEC validator is likely to stop your rewritten answers).
Disclaimer: Go interface is rough and requires a lot of CGO code to access internal structures. You'd be probably better suited by a LuaJIT module, there is another module doing something similar that you may take as an example, it also has wrappers for creating records from text etc. If you still want to do it, that's awesome and improvements to Go interface are welcome, read on.
What you need to do is roughly this (as CGO).
That will walk you through RR sets in the packet (C.knot_rrset_t),
where you can match type (rr.type) and contents (rr.rdata).
Contents is stored in DNS wire format, for address records it is the address in network byte order, e.g. {0x7f, 0, 0, 1}.
You will have to compare that to address/subnet you're looking for - example in C code.
When you find a match, you want to clear the whole packet and insert sinkhole record (you cannot selectively remove records, because the packet is append-only for performance reasons). This is relatively easy as there is a helper for that. Here's code in LuaJIT from policy module, you'd have to rewrite it in Go, using all functions mentioned above and using A/AAAA sinkhole record instead of SOA. Good luck!