I am new to redis and created an index and am attempting to ingest Zeek Logging Data, create an index for multiple fields, and then search fields in that index. For the life of me, I cannot get any values to return when searching for the #orig_ip name or using JSON.GET to retried any id.* related fields.
UPDATE: I figured this out after more troubleshooting and am updating here to help anyone else struggling with this problem.
Here is my WRONG code for creating the index:
# Options for index creation
index_def = IndexDefinition(
index_type=IndexType.JSON,
prefix = ['uid:'],
score = 0.5,
score_field = 'doc_score'
)
# Schema definition
schema = (
TagField('$.orig_l2_addr', as_name='orig_mac'),
TagField('$.id.orig_h', as_name='orig_ip'), #Wrong field path
TagField('$.id.resp_h', as_name='resp_ip'), #Wrong field path
NumericField('$.orig_bytes', as_name='orig_bytes'),
NumericField('$.resp_bytes', as_name='resp_bytes'),
NumericField('$.ts', as_name='timestamp')
)
r.ft('py_conn_idx').create_index(schema, definition = index_def)
Here is the result I kept getting with the above WRONG schema (no results)
search_result4 = r.ft('py_conn_idx').search(Query('#orig_ip:{192\.168\.210\.27}'))
Results for "#orig_ip:{192\.168\.210\.27}":
0
UPDATE: Working schema definition:
So it turns out even though Zeek is only using the . in field names vice using it to create an object, but the . in the field names was the culprit in my query failures. I needed to access the fields for the index as follows:
# Schema definition
schema = (
TagField('$.orig_l2_addr', as_name='orig_mac'),
TagField('$.["id.orig_h"]', as_name='orig_ip'), #Fixed field reference
TagField('$.["id.resp_h"]', as_name='resp_ip'), #Fixed field reference
NumericField('$.orig_bytes', as_name='orig_bytes'),
NumericField('$.resp_bytes', as_name='resp_bytes'),
NumericField('$.ts', as_name='timestamp')
)
After recreating the index with this schema, I get results with my query:
Results for "#orig_ip:{192\.168\.210\.27}":
Document {'id': 'uid:CPvYfTI4Zb1Afp2l5',....
Thanks to this stackoverflow question for finally walking me to the cause of my troubles: How to get objects value if its name contains dots?
Putting this answer here so this question gets marked as having one. See the updated question/code above!
Related
pymongo: 3.12.0
mongoengine: 0.23.1
I have a document:
class Logs(Document):
reference_id = StringField(default=None)
data = DictField(default=None)
In data field, i have a list failed_stories. This can have hundreds of elements and I want to perform pagination on it. So, i write this query as:
start_idx = 0
page_size = 10
reference_id = 'asdfg345678'
Logs.objects(reference_id=reference_id).fields(slice__data__failed_stories=[start_idx, page_size])
With this, i get one document in which all field are None except the dociment id (_id).
The following query results in document with correct data in document fields.
Logs.objects(reference_id=reference_id).get()
Is there any issue with the way I am writing this?
Note: I would like to do this with mongoengine only, if possible.
I would like to write small module that inherit sale.order.line and adding reserved lot on sale order line tree view.
under the sale.order.line module, there is the field name move_ids (stock.move , one2many) fields. I would like to create move_ids.move_line_ids.lot_id
So I tried the following code:
lot_name = fields.Char(related="move_ids.move_line_ids.lot_id", string="String")
but no luck and saw internal server error.
jooze
You can not do that, type of related field lot_name is inconsistent with lot_id.
Try to use a computed Many2many field to get all lot ids related to the current record.
Edit:
class SaleOrderLine(models.Model):
_inherit = 'sale.order.line'
lot_ids = fields.Many2many('stock.production.lot', compute='_get_lot_ids')
#api.one
#api.depends('move_ids.move_line_ids.lot_id')
def _get_lot_ids(self):
# returns the union of all lots, with duplicates removed
self.lot_ids = self.mapped('move_ids.move_line_ids.lot_id')
Consider the following mapping for a document in ES.
class MyDoc(elasticseach_dsl.Document):
id_info = Object(IdInfo)
class IdInfo(elasticseach_dsl.InnerDoc):
id = Keyword()
type = Keyword()
Using elasticsearch-dsl, there are 2 ways of retrieving a document (that I am interested in):
Using MyDoc.search().query().execute(), that yields Hit objects
Using MyDoc.get(), that yields a MyDoc object
Here is the issue I am experiencing:
When I retrieve the same document from ES, and that document is missing, for example, the type field, I get different behaviours:
When using search(): doc being a Hit object, accessing doc.type raises a KeyError
When using get(): doc being a MyDoc object, accessing doc.type simply returns None
To workaround this discrepancy, I would like to convert a Hit instance to a MyDoc instance, so that I can always use the doc.type syntax without any errors being raised.
How can I do that?
Alternatively, is there a way that I could access Hit instances with the same behaviour as MyDoc instances?
dict_hit = hit.to_dict()
doc = YourDocument(**dict_hit)
doc.property1 # you can access the property here
I know it is a bit awkward and annoying, it used to work with versions below 6.
I found a workaround, if you take the dictionary coming out from elasticsearch response you can then ask the document class to interpret it like the following.
query = MyDoc.search()
response = query.execute()
my_doc = MyDoc.from_es(response.hits.hits[0])
We were facing this situation. In our case, is was due to the index name in the Index subclass to configure Document indices. Our model looked more or les like this:
class MyDoc(Document):
my_field = Keyword()
class Index:
name = "my-doc-v1-*"
This way, when querying for documents in indexes that match that name (for example "my-doc-v1-2022-07"), hits are automatically instantianted as MyDoc objects.
Now we have started to generate 'v2' indices, named like "my-doc-v2--000001", and then hits were not being populated as MyDoc objects.
For that to happen, we had to change Index.name to my-doc-*. That way, documents from both 'v1' and 'v2' indices are always populated automatically by the library, since they match the Index.name expression.
What is wrong with this query?
const db = firebase.firestore()
const query = db.doc(this.props.user.uid).collection('statements').orderBy('uploadedOn', 'desc').limit(50)
I get the following error:
Uncaught Error: Invalid document reference. Document references must have an even number of segments, but FrMd6Wqch8XJm32HihF14tl6Wui2 has 1
at new FirestoreError (index.cjs.js:346)
at Function.DocumentReference.forPath (index.cjs.js:15563)
at Firestore.doc (index.cjs.js:15368)
at UploadStatementPresentation.componentWillMount (UploadStatementPage.jsx:61)
at UploadStatementPresentation.componentWillMount (createPrototypeProxy.js:44)
at callComponentWillMount (react-dom.development.js:6872)
at mountClassInstance (react-dom.development.js:6968)
at updateClassComponent (react-dom.development.js:8337)
at beginWork (react-dom.development.js:8982)
at performUnitOfWork (react-dom.development.js:11814)
Since you haven't described what exactly you're trying to query, I'll just point out that all documents must be in a collection, without exception. So, if you say this:
db.doc(this.props.user.uid)
Firestore assumes that the string you're passing to doc() contains both the collection and document id separated by a slash. But this seems to be highly unlikely in your case. You need to determine which collection the uid is in, and use that first when you build the reference to the collection you want to query. Assuming that you do have a statements subcollection in the uid document, and that some other collection contains the uid document, you'll have to specify the full path like this:
db.collection('that-other-collection').doc(this.props.user.uid).collection('statements')
Of course, only you know the actual structure of your data.
If you want to get a collection of documents with querying, you don’t have to specify a document id. Below code should work in this case.
const query = db.collection('statements').orderBy('uploadedOn', 'desc').limit(50)
Or if you want to get the document, you can pass the document id to doc() method. In that case, the code should be.
const query = db.collection('statements').doc(this.props.user.uid)
For more details about querying firestorm data: https://firebase.google.com/docs/firestore/query-data/get-data?authuser=0
For others having this issue, make sure that no document reference has an empty string.
I had this issue when using a get method with uid input as below and forgot to check if uid is empty
private fun getFullRef(uid: String): CollectionReference {
return ref.document(uid).collection(FireContact.SUB_PATH)
}
With Django/Haystack/SOLR, I'd like to be able to restrict the result of a search to those records within a particular range of django_ids. Getting these IDs is not a problem, but trying to filter by them produces some unexpected effects. The code looks like this (extraneous code trimmed for clarity):
def view_results(request,arg):
# django_ids list is first calculated using arg...
sqs = SearchQuerySet().facet('example_facet') # STEP_1
sqs = sqs.filter(django_id__in=django_ids) # STEP_2
view = search_view_factory(
view_class=SearchView,
template='search/search-results.html',
searchqueryset=sqs,
form_class=FacetedSearchForm
)
return view(request)
At the point marked STEP_1 I get all the database records. At STEP_2 the records are successfully narrowed down to the number I'd expect for that list of django_ids. The problem comes when the search results are displayed in cases where the user has specified a search term in the form. Rather than returning all records from STEP_2 which match the term, I get all records from STEP_2 plus all from STEP_1 which match the term.
Presumably, therefore, I need to override one/some of the methods in for SearchView in haystack/views.py, but what? Can anyone suggest a means of achieving what is required here?
After a bit more thought, I found a way around this. In the code above, the problem was occurring in the view = search_view_factory... line, so I needed to create my own SearchView class and override the get_results(self) method in order to apply the filtering after the search has been run with the user's search terms. The result is code along these lines:
class MySearchView(SearchView):
def get_results(self):
search = self.form.search()
# The ID I need for the database search is at the end of the URL,
# but this may have some search parameters on and need cleaning up.
view_id = self.request.path.split("/")[-1]
view_query = MyView.objects.filter(id=view_id.split("&")[0])
# At this point the django_ids of the required objects can be found.
if len(view_query) > 0:
view_item = view_query.__getitem__(0)
django_ids = []
for thing in view_item.things.all():
django_ids.append(thing.id)
search = search.filter_and(django_id__in=django_ids)
return search
Using search.filter_and rather than search.filter at the end was another thing which turned out to be essential, but which didn't do what I needed when the filtering was being performed before getting to the SearchView.