I am basically analyzing Twitter search results i.e activities on tweets (like, retweet, replies) to abstract unique interactors/users and comparing their profile picture with the expected PFP image. If the user profile image matches the comparison threshold, It is added to a collection.
Twitter search:
const searchResult: any = await axios.get(twitterBaseUrl + `/tweets/search/recent`, {
params: {
'query': searchKey,
'max_results': '10',
'tweet.fields': 'author_id,public_metrics,conversation_id'
},
headers: {
'Authorization': 'Bearer ' + twitterBearerToken,
}
});
find unique users interacting on tweets:
const uniqueUsers: any = [...new Map(users.map((user: any) => [user["id"], user])).values()];
Compare images:
for (const user of uniqueUsers) {
const compatibility = await commons.twitterService.compareImage(
pfpImage,
user.profile_image_url.replace('_normal', '')
);
// Compatibility threshold = 90%
if (compatibility > 90) {
usersWithDBPFP.push(user)
}
}
The problem with this approach is, it is hitting API rate limits and the accumulated result is way less than expected also I don't think this is a scalable solution.
In the meantime, I have found this awesome tool nftinspect which was quickly picking up and collecting Twitter profiles, I want to achieve a similar result, yet not clear in which direction I should go. Can you guys help me with this or suggest me any alternative ways, if you are comfortable? I would really appreciate that. Thank you :)
Related
I'm working on contentful to create blog posts.
I have created a field named with category with dropdown data, like the below image.
I have created more blogs for each categories (Ex: game has 5 blogs, Tour has 10 blogs and etc).
I want to show the list of all categories with content counts,
Is there any possible to get all of the categories with content count? ( I can get it by getting all blogs using this query
const res = ContentfulService.getEntries({ content_type: 'blog'})
then I grouped with category, but to get the category only, I don't want to get all of the blogs.)
Please let me know if there is a solution.
Thanks
The only way to do this through the API would be to make a request for each category and look at the total property of the response and that would be less efficient than what you're already suggesting.
https://cdn.contentful.com/spaces/{space_id}/environments/{environment_id}/entries?access_token={access_token}&content_type={content_type}&fields.category[in]={categoryValue}
I have achieved that using graphQL,
const query =
{
categoryCollection
{
items{
linkedFrom {
blogCollection {
total
}
}
name:category
sys {
id
}
}
}
}
let categories = await fetch(https://graphql.contentful.com/content/v1/spaces/${spaceId}, {
method: 'POST',
headers: {
'Content-Type': 'application/json',
// Authenticate the request
Authorization: Bearer ${accessToken},
},
// send the GraphQL query
body: JSON.stringify({ query }),
})
I want to implement .test file for the following function:
import Fuse from 'fuse.js';
import data from '../dal/data';
async function search(searchedData) {
// In case searched name is null
if (!searchedData) {
return [];
}
// load data
const dataload = await data();
// set options to filter data
const options = {
includeMatches: true,
// Search in `name'
keys: ['name'],
};
const fuse = new Fuse(dataload, options);
const matchedList = fuse.search(searchedData)
.map((element) => element.item);
return { matchedList };
}
export default search;
How to do for Fuse.js, shall I implement a mock data?
Fuse.js is library that provides fuzzy search based on dynamic configurations pass to it. There are different library which are available such as fuzzy-search.
Now, the question is how you can mock similar library as fuzzy search as per the comment. This is a good example of how you can start building your own fuzzy search engine, but to do so, you need to understand what fuzzy search is and how it works.
Now, if you want to know how Fuse.js works with any mock data. I can add some code which might help you to integrate with your existing project.
Mock Data
// it could be of any type, I am using Array of objects
const mockData = [{
name : 'Jason Durham',
age : 28,
address : 'Newyork city, New York, USA'
}, {
name : 'Jameson Durham',
age : 28,
address : 'Cleveland, Ohio, USA'
},
{
name : 'Tintan Groofs',
age : 28,
address : 'Ohio city, Ohio, USA'
}]
const options = {
includeScore: true, // the o/p will include the ranking based on weights
keys: [
{
name: 'name',
weight: 0.3 // what weightage you want this key in your search result
},
{
name: 'address',
weight: 0.7 // what weightage you want this key in your search result
}
]
}
// Create a new instance of Fuse
const fuse = new Fuse(mockData, options)
// Now search for 'Man'
const result = fuse.search('nwyork');
console.log(result);
You can check fuse.js documentation to check how to advance your search and configurable.
You can test the fuzzy search results using a test framework provided that you know the expected values, but it will contain a very broad set of o/p based on your configuration. As of now Fuse.js doesn't provide direct way in their documentation to check for it, but you can create a method based on the ranking you get and create a threshold value for it. This is the way we are testing our search results using Fuse.js, all the results below this threshold is discarded.
I've been working on a news site where users can post articles and leave comments. Part of the comments is to designate an emotion to the comment (one of 5 options: Amazed, Happy, Neutral, Sad, Angry).
What i want to do with this is make a 'top 5' of articles with the highest emotion count for a specific emotion. And sure, i know on a forum the emotion is always 'angry' so i should just work with that ;D , but this is part of a learning experience and i'd like to know how to do this.
So for other top5's ( 'hot discussion' and 'hot news' ) i have introduced counters in my Article Schema, that get incremented either when a comment is made (for 'hotDiscussion'), or when a user designates something as truly 'hot news' ( 'like button functionality' ). Then i simply search for the Articles and sort by these counters...
BUT, for this specific case, where there are 5 options, how do i sort ?!? ... at first i thought i would just implement the counters when a certain emotion is picked (pull down menu). But then all the counters are 'on the Article' as a single entity and i don't see how you would sort by all of them to get a comprehensive list. So then i though to put them in an object or array... something like:
Article {
title: String,
author: user...
comments: []
emotions: [ or {
Amazed: amazedCount,
Happy: happyCount,
Neutral: neutralCount,
Sad: sadCount,
Angry: angryCount
} or ]
Then, maybe i can find all the articles and sort by emotions (somehow?) and then skim off the top 5 from the list. But here again i run into the 5 options thing. And i have no clue how to sort the emotions? to then perhaps within the results, sort by the first (highest) count. To then skim of the top5.
So looking around here somewhat hints at me going for something like this
Article {
...
...
emotions: [
{name: Amazed, value: amazedCount},
{name: Happy, value: happyCount},
{name: Neutral, value: neutralCount},
{name: Sad, value: sadCount},
{name: Angry, value: angryCount}
]
And i like this direction somewhat, due to the clear keys 'name' & 'value'.
But here i run into the same issue, aka. lack of knowledge ... Because:
how to sort on a second order value, as i would be sorting on value?
will the same article just show up multiple times in the result? (i'dd like to prevent that if at all possible, a very busy mixed emotion thread should not show up twice over a less busy but clearly 'happy' thread in the top5 ).
So hopefully somebody can help me, as i can see this type of searching valuable in various types of situations, not just for some learning environment type 'forum' top5.
For those needing a more useful 'use case'
warehouse{
name: String,
... : ...
parts: [
{name: part1, value: Count}
{name: part2, value: Count}
{etc }
]
}
Generating a list of warehouses based upon the count of parts (where double mention of a certain warehouse isn't a problem); Or say a list of Car Dealers with the highest count for available brands or models (to get input for redistribution f/e). Or a list of online shops with a multi value rating system (f/e: delivery speed, customer service, price of products, etc.)
Well, it turns out this is actually fairly simple ... as i was most the way there. Just needed to put the sort criteria in between '...' /doh !
! THIS CODE IS NOT COMPLETE, as in Copy/Paste you are done, but the Code below does show you most the places where you need to 'do' things to make this work (/show routes not included, aka the pages the user sees and can input stuff on)
// In the Schemas:
Article {
...
...
emotions: [ {emotion: String, eCount: Number} ]
}
Comment{
emotion: String; //to save the emotion of the user on the comment
}
// Upon initializing an Article:
router.post("/route", function(req, res){
let etcetera: etc,
let theEmotions = [ // in your usecase you may populate these elsewhere
{ emotion: "Amazed", eCount: 0 },
{ emotion: "Happy", eCount: 0 },
{ emotion: "Neutral", eCount: 0 },
{ emotion: "Sad", eCount: 0 },
{ emotion: "Angry", eCount: 0 }
];
let newArticle = {
etc: etc,
emotions: theEmotions
};
Article.create(newArticle, function(err, newArticle){
*stuff here*
});
});
// when a comment gets made, it updates the scores based upon the emotion selected
//REST :: CREATE --> actually creates a new comment
router.post("/", function(req, res){
Article.findById(req.params.id, function(err, foundArticle){
if(err){
console.log("ERROR # finding article to POST the COMMENT");
res.redirect('/articles');
} else {
Comment.create(req.body.comment, function(err, createdComment){
if(err){
console.log("ERROR # CREATing new COMMENT");
} else {
switch(createdComment.emotion){
case "Amazed":
foundArticle.emotions[0].eCount += 1;
break;
case "Happy":
foundArticle.emotions[1].eCount += 1;
break;
case "Neutral":
foundArticle.emotions[2].eCount += 1;
break;
case "Sad":
foundArticle.emotions[3].eCount += 1;
break;
case "Angry":
foundArticle.emotions[4].eCount += 1;
break;
default:
console.log("ERROR # Emotion Switch in comments route");
}
//add username and id to comment
createdComment.user.id = req.user._id;
createdComment.user.username = req.user.username;
createdComment.save();
foundArticle.comments.push(createdComment);
foundArticle.save();
console.log("SUCCESS # CREATing new COMMENT");
res.redirect('/articles/' + foundArticle._id);
}
});
}
});
});
//Where one wants to sort: ... note the '...' around 'emotions.eCount'
router.get("/specificRoute, function(req, res){
Article.find({}).sort({'emotions.eCount': -1}).exec(function(err, hotEmotions){
*stuff happens here*
});
});
Hopefully this will be of some help to someone somewhere in the future :)
I have a store table that has a column lat_long which stores each store latitude/longitude as geometry point in sequelize. I want to be searching for stores based on proximity of the latitude/longitude from the request body from the front end. Have tried to exploit many related topics(answer) concerning this in Stackoverflow but what have learned is that the query can be written like this.
const location = sequelize.literal(`ST_GeomFromText('POINT(${lng} ${lat})', 4326)`);
User.findAll({
attributes: [[sequelize.fn('ST_Distance_Sphere', sequelize.literal('geolocation'), location),'distance']],
order: 'distance',
limit: 10,
logging: console.log
})
.then(function(instance){
console.log(instance);
})
Have tried it and it works fine but the challenge am having is that; in this example and almost all the sample I saw sofar they made use of the attribute which return only distance but I want to return all the fields in the table. Then I tried removing the attribute then the response I got was not searched based on the geometry(distance) and I also tried to use where but am getting an error.
Shown below is my latest query which is returning error please how can I go about this? thank you.
export const getNearestStoreWithCategory = async (lat, long) => {
try {
const location = sequelize.literal(`ST_GeomFromText('POINT(${lat} ${long})')`);
const distance = sequelize.fn('ST_Distance_Sphere', sequelize.literal('lat_long'), location)
const storeArr = await Store.findAll({
order: distance,
where: sequelize.where(distance),
limit: 20
})
return storeArr
} catch (error) {
return error
}
}
From the doc at https://sequelize.org/v4/manual/tutorial/querying.html, I'm guessing what you are missing in the sample code you had is including the distance to the fields to be returned as the response from the db rather that only computing the distance and returning;
const location = sequelize.literal(`ST_GeomFromText('POINT(${lng} ${lat})', 4326)`);
User.findAll({
attributes: {
include: [[sequelize.fn('ST_Distance_Sphere', sequelize.literal('geolocation'), location),'distance']]
},
order: 'distance',
limit: 10,
logging: console.log
})
.then(function(instance){
console.log(instance);
})
This should give a query like;
SELECT id, OTHER_FIELDS_OF_THIS_TABLE, ST_Distance_Sphere(geolocation) AS distance ...
Now, this is untested code, but you could give it a try and comment if it helps or not.
Can anyone please suggest me how to implement pagination based on date? For example, users will push some data daily and they may want to see how many data records they've pushed on a certain day by paginating. They may not push the exact amount of data every day, so what is the most scalable way to implement this? Here is typical pagination that I implemented
router.get('/:page?', (req, res, next) => {
const perPage = 5;
let page = req.params.page || 1;
Upload.find({ user: req.user._id })
.populate('user text')
.skip((perPage * page) - perPage)
.limit(perPage)
.exec((err, uploads) => {
Upload.count().exec((err, count) => {
if (err) return next(err)
res.render('record', {
title: 'Records',
data: uploads,
current: page,
pages: Math.ceil(count / perPage)
});
})
})
});
well, so in this case we can go initially with the find query only.
later you can implement with aggregation.
so the idea is,
in each page we will show the data of each day, doesnot matter how many are there.
[surely not a good idea when you have a huge amount of data for that day or against your query]
here is what i have implemented for you:
router.get('/:page?', (req, res, next) => {
const page = req.params.page ? req.params.page : new Date().getDate();
const currentYear = new Date().getFullYear();
const currentMonth =new Date().getMonth();
Upload.find({
"createdAt": {
"$gte": new Date(currentYear, currentMonth, page),
"$lt": new Date(currentYear, currentMonth, page)}
})
.populate('user text')
.exec((err, uploads) => {
if (err) return next(err)
//uploads contains the result of date `page` of `currentMonth` month and year `currentYear`
res.render('record', {
title: 'Records',
data: uploads,
resultDay:page
});
})
});
few things to remember, in here, we cant use limit & skip as we are trying to query for a day.
so in this above if you pass the page param as 1 it will give the result of 1st day of the current month and current year.
in here we dont know how many page will be there, as because we dont know for which day how many docs are there in advance.
so, its expected in frontend you will show Next button and call this api. with the +1 value of current page value.
and accordingly you can show the previous button too.
to get count of all the docs of a sigle day you can pass the cindition part, which we are already using inside .find() in .count().
Hope this helps,
let me know if i've missed anything.