I have around 10000+ rows(Listitems) in a list I want to query.
I want to iterate each of the item in a timerJob - but I cant take all of them at a time
since : Object Model Override - NO, ListView threshold - 1000 - in the FARM level and i we cant change this.
What is the way for me to iterate all the 10000+ (like a batch) ??
You should use a ContentIterator. This will allow you to iterate over the contents of very large lists without triggering an SPQueryThrottledException.
For example:
SPList list = SPContext.Current.Web.Lists["MyList"];
// Build query using an iterator-defined WHERE clause
string query = string.Format("<Where><Eq><FieldRef Name='MyFieldName'/><Value Type='Text'>MyFieldValue</Value></Eq></Where>{0}", ContentIterator.ItemEnumerationOrderByNVPField);
// Instantiate iterator and use it to process the list
ContentIterator iterator = new ContentIterator();
iterator.ProcessListItems(list, query, ProcessListItem, ProcessListItemError);
Then you'd define your ProcessListItem and ProcessListItemError thusly:
static void ProcessListItem(SPListItem item) {
Console.WriteLine("ProcessListItem: Name {0}", item.Title);
}
static bool ProcessListItemError(SPListItem item, Exception e) {
Console.WriteLine("ERROR: message {0}", e.Message);
return true;
}
I'd also recommend you review Microsoft's Best Practices with SharePoint Server articles, in particular "Writing Efficient Code In SharePoint Server", which further discusses properly writing queries.
Related
How can a search context for website like stack overflow be modelled following Domain driven design?
Let's say in my domain, I have three type of entities as Questions, Answers, Tags of Questions.
I have to model a search context in a way that it accepts a search string and return matching questions, answers and tags.
I want to understand that, In search context, is search going to be just a ddd-service which will perform search or can there be some entities, aggregates etc.
It can be assumed that matching algorithm is simple sql like query.
You need to implement the search in a Query Service.
CQRS fits really good with DDD because when you want to update the model you use commands, such as in your case:
AnswerToQuestionCommand, PostNewQuestionCommand, etc.
These commands are sent to your application service, which updates the entity, which in turn send a DomainEvent, which is intercepted in the hexagonal architecture and updates the search index. You can do it in the hexagonal architecture: by calling a service dedicated to maintain the index in the same transaction. You consider an entity persisted if the index has also been updated for instance. I would go this way.
Here is a sample with Spring #TransactionalEventListener
#Component
public class ProjectRenamedListener {
#Value("${spring.jpa.properties.hibernate.search.default.indexBase:target}")
private String indexBase;
private final Logger logger = LoggerFactory.getLogger(ProjectRenamedListener.class);
#TransactionalEventListener
public void projectRenamed(final ProjectRenamed event) {
try (final StandardAnalyzer analyzer = new StandardAnalyzer();
final Directory directory = NIOFSDirectory.open(Paths.get(indexBase, ProjectData.class.getName()));
final IndexWriter writer = new IndexWriter(directory, new IndexWriterConfig(analyzer))){
final Document document = new Document();
document.add(new StringField("project_id", event.projectId().identity(), Field.Store.YES));
document.add(new StringField("tenant_id", event.tenant().identity(), Field.Store.YES));
document.add(new TextField("name", event.name(), Field.Store.YES));
writer.updateDocument(new Term("project_id", event.projectId().identity()), document);
} catch (IOException e) {
logger.warn("Unable to update index for project name.", e.getMessage());
}
}
}
When you need to query your Questions, Answers, etc. you go through a query service. You ask the index, and load the entities that you want to return to your client.
The beauty of this is that it does not have to be the same object when you modify it and query it. It sounds a bit wierd when you start because of the code duplication but on the long run its clearly superior solution as you have no coupling between reading / query operation which occurs a lot and should be fast, and writing operation which should not occur a lot and does ont have to be specially super fast.
I would suggest the approach of Vaughn Vernon on query services (https://github.com/VaughnVernon/IDDD_Samples), I used lucene on my part for a query service, here is some code:
#PreAuthorize("hasAuthority('Administrator')")
public Page<ProjectData> searchProjectsData(
final String tenantId,
final String queryText,
final Pageable pageable) {
if (!StringUtils.isEmpty(tenantId)) {
return this.searchProjectsDataOfTenant(tenantId, queryText, pageable);
}
final StandardAnalyzer analyzer = new StandardAnalyzer();
final QueryBuilder builder = new QueryBuilder(analyzer);
final Query query = builder.createPhraseQuery("name", queryText);
try {
final IndexReader reader = DirectoryReader.openIfChanged(this.directoryReader);
final IndexSearcher searcher = new IndexSearcher(reader);
final TopDocs documents = searcher.search(query, pageable.getPageSize());
return this.fetchProjectsData(searcher, documents, pageable);
} catch (IOException e) {
throw new RuntimeException(String.format("Unable to search project for query: %s", queryText), e);
}
}
You probably don't need the whole array of DDD tactical patterns for a Search bounded context, no. Except if you're planning on storing search preferences or creating search templates, maybe - even then, it seems very CRUDish.
Designing how the search context indexes or accesses the data from other BC's can be a more complex matter, though.
I have a simple query as follows "select * from USERS". I also use Pageable to enable pagination.
This query may have optional predicates based on the given parameters being null or not.
For example if "code" parameter is given and not null, then the query becomes
"select * from USERS where code = :code";
As far as I know I cannot implement this using #Query annotation. I can implement a custom repository and use EntityManager to create a dynamic query.
However, I am not sure how I can integrate "Pageable" with that to get back paginated results.
How can I achieve this?
This is very easy to do in Spring Data using QueryDSL (as alternative to the criteria API). It is supported out of the box with the following method of QueryDSLPredicateExecutor where you can just pass null as the Predicate if no restrictions are to be applied:
Page<T> findAll(com.mysema.query.types.Predicate predicate,
Pageable pageable)
Using QueryDSL may not be an option for you however if you look at the following series of tutorials you might get some ideas.
http://www.petrikainulainen.net/programming/spring-framework/spring-data-jpa-tutorial-part-nine-conclusions/
The scenario you have is actually discussed by the author in the comments to part 9 of his guide.
Getting page results for querydsl queries is somehow complicated since you need two queries: one for the total number of entries, and one for the list of entries you need in the page.
You could use the following superclass:
public class QueryDslSupport<E, Q extends EntityPathBase<E>> extends QueryDslRepositorySupport {
public QueryDslSupport(Class<E> clazz) {
super(clazz);
}
protected Page<E> readPage(JPAQuery query, Q qEntity, Pageable pageable) {
if (pageable == null) {
return readPage(query, qEntity, new QPageRequest(0, Integer.MAX_VALUE));
}
long total = query.clone(super.getEntityManager()).count(); // need to clone to have a second query, otherwise all items would be in the list
JPQLQuery pagedQuery = getQuerydsl().applyPagination(pageable, query);
List<E> content = total > pageable.getOffset() ? pagedQuery.list(qEntity) : Collections.<E> emptyList();
return new PageImpl<>(content, pageable, total);
}
}
You have to use querydsl and build your where depending on not null parameter for example
BooleanBuilder where = new BooleanBuilder();
...
if(code != null){
where.and(YOURENTITY.code.eq(code));
}
and after execute the query
JPAQuery query = new JPAQuery(entityManager).from(..)
.leftJoin( .. )
...
.where(where)
and use your own page
MaPage<YOURENTITY> page = new MaPage<YOURENTITY>();
page.number = pageNumber+1;
page.content = query.offset(pageNumber*pageSize).limit(pageSize).list(...);
page.totalResult = query.count();
I create MyPage like that
public class MaPage<T> {
public List<T> content;
public int number;
public Long totalResult;
public Long totalPages;
...
}
it works but if in your query you got a fetch then you gonna have this warning
nov. 21, 2014 6:48:54 AM org.hibernate.hql.internal.ast.QueryTranslatorImpl list
WARN: HHH000104: firstResult/maxResults specified with collection fetch; applying in memory!
and it will slow down your request So the solution is to get ride of the fetch and define a #BatchSize(size=10) and use Hibernate.initialize(....) to fetch data in collections and other object type.
Display data from related entities to avoid the lazy initialization exception with setting up #BatchSize
How to execute a JPAQuery with pagination using Spring Data and QueryDSL
The information here is obsolete. Have your Repository implement the QueryDslPredicateExecutor and paging comes for free.
I'm using a function to allow query composition from Web UI and I would to implement paging functionality which it will be available for dataBound controls such as ObjectDataSource, gridView, etc:
public class MyClass<TEntity> where TEntity : class
{
FakeEntities xxx = new FakeEntities();
public IEnumerable<TEntity> Get(Func<IQueryable<TEntity>, IQueryable<TEntity>> queryExpression)
{
var query = xxx.Set<TEntity>();
return queryExpression(query).ToList();
}
public int Count()
{
// What Can I return?
}
}
// **** USAGE ****
MyClass<User> u = new MyClass<User>();
var all = u.Get(p => p.Where(z => z.Account == "Smith").OrderBy(order => order.IdOther).Skip(1).Take(2));
The above query use Take and Skip function, so can I get real count of my entities? Obviously I must return Query Count without modifying filter expression.
I found this solution: Get count of an IQueryable<T>
However I get targetInvocationException with inner message {"This method supports the LINQ to Entities infrastructure and is not intended to be used directly from your code."}
I know my request could be freak-abnormal, because best practice should to impose to move "presentation needs" to some wrap class and that's is what I'll do. So I don't need anymore to get Count entities on my business logic class.
That's just UI concern only.
Thank you the same.
I use a function to get access to a configuration document:
private Document lookupDoc(String key1) {
try {
Session sess = ExtLibUtil.getCurrentSession();
Database wDb = sess.getDatabase(sess.getServerName(), this.dbname1);
View wView = wDb.getView(this.viewname1);
Document wDoc = wView.getDocumentByKey(key1, true);
this.debug("Got a doc for key: [" + key1 + "]");
return wDoc;
} catch (NotesException ne) {
if (this.DispLookupErrors)
ne.printStackTrace();
this.lastErrorMsg = ne.text;
this.debug(this.lastErrorMsg, "error");
}
return null;
}
In another method I use this function to get the document:
Document wDoc = this.lookupDoc(key1);
if (wdoc != null) {
// do things with the document
wdoc.recycle();
}
Should I be recycling the Database and View objects when I recycle the Document object? Or should those be recycled before the function returns the Document?
The best practice is to recycle all Domino objects during the scope within which they are created. However, recycling any object automatically recycles all objects "beneath" it. Hence, in your example method, you can't recycle wDb, because that would cause wDoc to be recycled as well, so you'd be returning a recycled Document handle.
So if you want to make sure that you're not leaking memory, it's best to recycle objects in reverse order (e.g., document first, then view, then database). This tends to require structuring your methods such that you do whatever you need to/with a Domino object inside whatever method obtains the handle on it.
For instance, I'm assuming the reason you defined a method to get a configuration document is so that you can pull the value of configuration settings from it. So, instead of a method to return the document, perhaps it would be better to define a method to return an item value:
private Object lookupItemValue(String configKey, itemName) {
Object result = null;
Database wDb = null;
View wView = null;
Document wDoc = null;
try {
Session sess = ExtLibUtil.getCurrentSession();
wDb = sess.getDatabase(sess.getServerName(), this.dbname1);
wView = wDb.getView(this.viewname1);
wDoc = wView.getDocumentByKey(configKey, true);
this.debug("Got a doc for key: [" + configKey + "]");
result = wDoc.getItemValue(itemName);
} catch (NotesException ne) {
if (this.DispLookupErrors)
ne.printStackTrace();
this.lastErrorMsg = ne.text;
this.debug(this.lastErrorMsg, "error");
} finally {
incinerate(wDoc, wView, wDb);
}
return result;
}
There are a few things about the above that merit an explanation:
Normally in Java, we declare variables at first use, not Table of Contents style. But with Domino objects, it's best to revert to TOC so that, whether or not an exception was thrown, we can try to recycle them when we're done... hence the use of finally.
The return Object (which should be an item value, not the document itself) is also declared in the TOC, so we can return that Object at the end of the method - again, whether or not an exception was encountered (if there was an exception, presumably it will still be null).
This example calls a utility method that allows us to pass all Domino objects to a single method call for recycling.
Here's the code of that utility method:
private void incinerate(Object... dominoObjects) {
for (Object dominoObject : dominoObjects) {
if (null != dominoObject) {
if (dominoObject instanceof Base) {
try {
((Base)dominoObject).recycle();
} catch (NotesException recycleSucks) {
// optionally log exception
}
}
}
}
}
It's private, as I'm assuming you'll just define it in the same bean, but lately I tend to define this as a public static method of a Util class, allowing me to follow this same pattern from pretty much anywhere.
One final note: if you'll be retrieving numerous item values from a config document, obviously it would be expensive to establish a new database, view, and document handle for every item value you want to return. So I'd recommend overriding this method to accept a List<String> (or String[ ]) of item names and return a Map<String, Object> of the resulting values. That way you can establish a single handle for the database, view, and document, retrieve all the values you need, then recycle the Domino objects before actually making use of the item values returned.
Here's an idea i'm experimenting with. Tim's answer is excellent however for me I really needed the document for other purposes so I've tried this..
Document doc = null;
View view = null;
try {
Database database = ExtLibUtil.getCurrentSessionAsSigner().getCurrentDatabase();
view = database.getView("LocationsByLocationCode");
doc = view.getDocumentByKey(code, true);
//need to get this via the db object directly so we can safely recycle the view
String id = doc.getNoteID();
doc.recycle();
doc = database.getDocumentByID(id);
} catch (Exception e) {
log.error(e);
} finally {
JSFUtils.incinerate(view);
}
return doc;
You then have to make sure you're recycling the doc object safely in whatever method calls this one
I have some temporary documents which exist for a while as config docs then no longer needed, so get deleted. This is kind of enforced by an existing Notes client application: they have to exist to keep that happy.
I've written a class which has a HashMap of Java Date, String and Doubles with the item name as the key. So now I have a serializable representation of the document, plus the original doc noteID so that it can be found quickly and amended/deleted when it's not needed anymore.
That means the config doc can be collected, a standard routine creates a map of all items for the Java representation, taking into account the item type. The doc object can then be recycled right away.
The return object is the Java class representation of the document, with getValue(String name) and setValue(String name, val) where val can be Double String or Date. NB: this structure has no need for rich text or attachments, so it's kept to simple field values.
It works well although if the config doc has lots of Items, it could mean holding a lot of info in memory unnecessarily. Not so in my particular case though.
The point is: the Java object is now serializable so it can remain in memory, and as Tim's brilliant reply suggests, the document can be recycled right away.
I am running a sharepoint caml query where I want to check that a field on an item is equal to one of many values. I do this dynamically and may wish to check against many hundreds of values.
I found that when performing a query with 780 OR elements I got an error related to server memory. Obviously this is variable across environments, but I am looking for some guidelines suggesting a maximum query length to which I should cap.
Thanks!
How about using ContentIterator?
http://community.zevenseas.com/Blogs/Robin/Lists/Posts/Post.aspx?ID=122
It supports recursion for walking a tree of items and acting on them in some way.
This code does a "publish all" on the style library files whose "FeatureId" properties match a particular value:
SPList styleLibrary = rootWeb.Lists.TryGetList("Style Library");
SPFolder folder = styleLibrary.RootFolder;
ContentIterator ci = new ContentIterator();
ci.ProcessFilesInFolder(
styleLibrary,
folder,
true,
new ContentIterator.FileProcessor((SPFile f) =>
{
// Check the FeatureId property the file's been "stamped" with
if (f.Properties.ContainsKey("FeatureId"))
{
if (String.Equals(f.Properties["FeatureId"] as string, featureId, StringComparison.InvariantCultureIgnoreCase))
{
if (f.Level == SPFileLevel.Checkout)
f.CheckIn(String.Empty, SPCheckinType.MajorCheckIn);
if (f.Level == SPFileLevel.Draft)
f.Publish("");
}
}
}),
new ContentIterator.FileProcessorErrorCallout((SPFile f, Exception Ex) =>
{
//Define the action I need to do if an error occur
return false;
}));
You could get all folders by SPList.Folders, iterate over the folders and filter it by whatever...