I have a couple of questions surrounding the NotesView.update functionality.
I currently have a single Notes database on my domino server that pulls information from a few other databases via a sync agent into itself making it the primary data source for my DTG mobile app.
My Notes sync agent removes / hard deletes documents from my primary Notes Database that Domino To Go syncs with.
Are the documents in my DTG view on the device automatically removed too when they are no longer in the Server database during a NotesView.update or do they live on in the DTG view oblivious to the fact that their parent has been deleted?
The reason I ask is that I want to institute the syncModifiedOnly parameter and the documentation describes needing to add a $dtgIsRemoved column to the end of my Notes view to speed things up.
Also describes needing to add a formula to the column in order to make the DTG view understand that I want to remove the underlying DTG document.
Having said that, does that mean that:
I shouldn't be Removing the NotesDocuments in my sync agent or from the NotesDatabase.
Leave them in the Notes view and create a formula in the $dtgIsRemoved column that will return "1" if I want the DTG document removed during a NotesView.update?
Originally I thought that deleting a Notes Document on the server database would automatically remove the document from DTG during a NotesView.update without user intervention in the cleanupCallback.
A little unclear as to how the $dtgIsRemoved can help me speed up the update.
Ultimately I want to speed up my view.update.
I have 5 views that are coupled in callbacks and view.updates and displaying a progressbar.
Still though it is very inconsistent when run on my iOS device, total doc count is around 73000 and sometimes takes 60 seconds and other times 180 - 200.
Displaying the information is fast and searching lightning fast with SQL, it's just the NotesView.update that is an issue for me at this stage.
This is how I think it works:
NotesView.update - runs through all entries in the DTG view
for each doc in the DTG view,
check the $dtgIsRemoved column, if = "1", remove the doc from the DTG view
get the corresponding NotesDocument in the Parent NotesDatabase
if no handle then mark the DTG doc for deletion
otherwise check the syncModifiedOnly parameter
if set to true then look at the last sync date parameter
check the last modified date of the NotesDocument is >= last sync
if true then sync all fields / columns.
At end of sync, run the cleanup process to actually remove the docs that were tagged for deletion OR weren't found in the parent NotesDatabase.
Still failing to see the point of the $dtgIsRemoved column if the NotesView.update removes documents from the DTG view that aren't in the source database view. Removing them from the source view would happen in the Notes view selection formula or in a straight NotesDocument.Remove in a sync agent like mine.
TB
Tamir,
I'm in a hurry right now but I want to give a quick answer right now:
Yes, documents in the app will be removed if they do not longer exist on the Domino side.
The $dtgisremoved column is optional, but can speed things up if used together with the syncModifiedOnly option, otherwise it has no effect.
SyncModifiedOnly gives a date to the DTG XPage which creates the JSON data, and the XPage only creates JSON data for documents that has been changed since this date. So this can speed up the sync greatly if only few documents change in a couple of days.
SyncModifiedOnly looks after the "modified" date of the Notes documents. But you can add a column (type datetime) in your Domino view which the SyncModifiedOnly logic should use. I believe the DTG option is syncModifedOnlyColumn - but I'm not sure right now.
If you use SyncModifiedOnly, but you don't have a $dtgisremoved column, the DTG XPage needs to put the UNID of EVERY document in the view in the JSON data in order to be able to detect which one has been deleted. That means a bigger JSON dataset, more data to transfer and more data to compute in the app. But if you hard delete the documents in the Domino database, you cannot create a $dtgisremoved column.
I hope this already helps!
Related
Sorry this isn't a specific coding question, it is more of a design concept.
What is the usage case for programmatically adding and removing fields to Notes Forms e.g. NotesDocument.RemoveItem(), ie why would you add and remove fields in the background?
For many years I have designed my forms with the fields layed out on the form which are required and then hide and show as required.
By adding dynamically you can't position them and frustratingly removing them or deleting they still appear the Database Fields in Domino Designer, getting rid of them is a bit a a black art, but that's another story.
I must be missing a trick or a basic design concept. Any thoughts on best practice would be appreciated.
Many thanks.
Yes, you are missing the difference between "Fields" and "Items". A field is a design element that you can place anywhere on your form. You define how it looks, what content it contains, what datatype it is, etc.
When creating a document with the form the value of the FIELD is stored in an ITEM in the resulting NotesDocument.
This item is totally decoupled from the field that created it. If you were to change the field in the form from text to number or move it around or make a names- field of it, the item in the existing documents would never change unless you open the documents and save them in frontend or use any LotusScript or Formula Code to recalculate the document in backend.
Very often items are added programmatically to documents to fulfill different purposes: Calculate values to be displayed in views, calculate values that are import for the workflow but not for the user, etc.
Complex applications often consist of a lot more items than there are fields in the several forms.
Back to your question: Removing an item from a document simply removes the value that was created by the field in the form. When reopening the document, the item will be repopulated, either by default value or whatever....
Usually you would use this to remove items that you no longer need (and probably already removed from the form).
As soon as you removed all references to a field / item everywhere in design and documents, you can finally get rid of it completely by compacting the database.
An item is distinct from a field in Notes. The form is purely a UI concept, the item is what the data is stored in.
Manipulating data in the backend can be used for a number of reasons. One such use case is the setting of a flag when a date on the form has expired.
Say you want a view showing all documents that have expired. Your rules dictate that documents are considered as "Expired" after 7 days. You could create a view with a formula that shows all document whose date is 7 days older than today:
SELECT Date < #Adjust(#Today; 0; 0; -7; 0; 0; 0);
This view will ALWAYS be out of date and will constantly be updated by the server as it re-evaluates #Today.
Now, a better way would be to create an agent that runs daily that sets an item on the document to indicate that it has expired e.g.
#SetField("Expired"; 1);
The view formula would then be
SELECT Expired = 1
The view would only need to update daily and you have a much faster view because of it.
RemoveItem is used to get rid of data no longer needed e.g. FaxNumber.
There are many use cases for RemoveItem. Here's one that comes up frequently.
You have a database and an agent that processes documents in that database. Every time it runs, the agent replaces the value of a bunch of items. There are a variety of error conditions that can cause it to abort processing a document early, but you're a smart programmer and you've accounted for that with on error traps. When you hit one, you log an error message, save your document, and then either abort your agent or go on to processing the next document.
But at this point, some of the items that the agent normally updates have values saved from this run, and some of them have values saved from a previous run. This might be bad. This might be confusing for someone who is looking at the item values and trying to figure out what's going on. This might even cause validation errors on the form.
So how do you avoid this? At the very beginning of your agent, you call a cleanup sub that finds and removes all the items that the agent is going to update. Now you have a clean slate, and if your agent hits that error condition, it can save whatever it can save without any concern about whether it is leaving things in an inconsistent state. Of course, in cases where you are doing this to avoid validation errors, your validation formulas will have to be smart enough to be checking #IsAvailable for dependent items, but that's a good practice anyhow.
I created a view to retrieve all document UNID in lotus notes. The view selection formula is select#all. The total no.of documents in nsf is 1200 but the view shows 1198 documents what about the other 2 documents. As per my analysis there is no deleted or profile documents present. How to retrieve those two documents please advice. Thanks
Every view in a Notes Database has an option called "Organizing response documents in a hierarchy" that is enabled by default.
That means that documents that do not have a parent document are not shown in these views. In addition reponses might be "invisible" in the view as they are shown "indented" but collapsed without a visible indicator to expand the hierarchy and show the missing documents.
Disabling this view property makes the documents visible again. They might be "real" responses or probably Save / Replication conflicts that occur if one document is saved in two different places at the same time (these places might even be within the same database replica).
Another possibility for "invisible" documents are documents that are protected by reader fields. They are counted to the document count of the database but not visible to you due to your name missing in the reader field.
I have a notes database and recently we are having an issue across random documents. Users gets error "The linked document cannot be found in the view" when they try to open a document from a view. I am not sure how to reproduce this issue.
I tried the below options to reproduce it, but no luck:
Attaching a document link to one of the Rich text field and deleting the link's target.
Attaching a file as a link and deleting the file in the target.
I am not seeing any pattern with the documents having the issue. The documents accessed few days ago, which was not accessed for more than 2 years also becomes corrupt like this.
Other details:
The Field info tab in Document properties dialog box is empty
There is no scheduled agent running overnight affecting this documents
No change was done to the database recently
We recently migrated to Notes 11.0 from 9.0
Since we have a daily backup of the database, we restore the document from one of the restored copy in which it was working fine.
A guess: did you try re-indexing the view? Shift-F9?
It could also be in the form, or even the view.
What if ...
you open the same document from a different view?
you remove some, many or all fields from the form, do you still see the error?
I think those documents have actually been deleted, and two things make me think that:
You say in a comment that rebuilding the view makes the documents disappear. I'd expect this only if the documents no longer exist.
I have in the past had LotusScript code doing view lookups which produced documents that I knew to be deleted, but the NotesDocument.IsDeleted property was False and the NotesDocument.Items property was Empty (i.e. no fields in the document). This is consistent with your saying the Field info tab is empty.
I'd check if any user action or code could delete documents. You say there are no scheduled agents, but maybe some user action could run code which deletes documents, or some users with the Delete privilege in the database ACL could be directly deleting documents using the Delete key on the keyboard.
Also check the database properties -> Info tab -> User Detail. This should show if there are recent deletions, but the record is probably limited to the last week or less.
I am doing "traditional" lotus notes programming (same since R5) and need to implement linking between 2 document types (forms) residing in different databases.
Document of type (A) in database (A) can reference several documents of type (B) in database (B).
And document (B) should also display its relationships with document (A), as document (B) can be related to different documents (A).
We have Many to Many relationship.
At the moment it is implemented on one side only (one to many):
Form of Document (A) contains embedded view of special
"link" documents residing in database A. This link documents are created by lotusScript when user selects documents from database (B). When user clicks on an item in this embedded view, it opens document (B).
Client wants to be able to edit this relationship on any side, so that if he edits it in form (A), form (B) is updated.
Form (B) is supposed to have the same kind of embedded view or a list of associated documents of type (A)
What is the best way to implement it?
Client's infrastracture is Lotus Domino 8.5.2 + Lotus Notes 8.5.2, so theoretically, composite applications approach may be an option.
The reason why I ask this question is that as far as I understand there is no good way in Notes to embed a view from another database.
The requirement is that the database should be present on workspace to be displayed in some sort of dodgy list.
It would be great to be able to specify target database for embedded view by server and replicaID, but instead we have a weird list of random workspace databases.
The main problem is that Notes wasn't designed to handle relationships like that between databases (nor anything besides parent child relationships for that matter). So the solution will have to be a creative one.
A couple of (off-the wall, potentially awful) ideas come to mind. One is to store the references in the documents themselves, and update them whenever the document is saved. That could all be done in LotusScript, and would require searching through the other database's documents to update their references.
Upside is that the performance when reading the documents would be excellent. There'd be no issues while reading Database A if Database B was unavailable. It keeps data local to each database. The downsides include the likelihood of save conflicts and the danger that references could get out of sync if documents aren't "saved" but instead are updated via agents, etc.
Another thought is to use agents to manage the links on a scheduled basis. If you don't need real-time up-to-date references, you could run an agent that scans Database B and updates the references in Database A. With this method you could choose either to update the Database A documents themselves - or - as it sounds like you've already done, create a set of link documents that show up in an embedded view. The latter eliminates the save conflict problem.
One more idea is to hide any references when you open a document in Database A, but provide a button to "show" or "update" references. When you click that button, it fires off LotusScript to search Database B and build a list on the fly. This would probably work quickly with less than 10,000 documents. That function could update the link documents you store on the same database which feed the embedded view.
Hope this helps!
What is the best way to implement it?
As you mention creating a composite application may allow you to do this, but would be restricted to windows rather then design level in the form.
eg.
[Window A] --- trigger ---> [Window B]
If you are not familiar with this system I did a tutorial which explains the basics.
http://www-10.lotus.com/ldd/compappwiki.nsf/dx/ibm-my-first-wire
Although the tutorial calls back to the same database, it is easy enough to point to a different one.
Personally I'd do it through XPages. I personally find it much easier to implement then through classic style notes design/comp apps. It will also allow you to display design elements within the same screen area.
As you've already heard, Lotus Notes has no referential integrity constraints built-in, you have to do it yourself.
I wouldn't be relying on document links as they're geared around UNID's which can change if you cut and paste the same document, thus losing the link. Try this,
1/ Create an "ID" field on each document. You can populate it by using #Unique in a computed field to generate an ID, and save that to the documents in both databases. You can create an agent to do this in lotusScript (LS), or formula. (Consider using the evaluate statement if doing in LS)
2/ Create a lookup view in each database that lists the documents by the new ID (don't forget to set the "sort order" of the ID column.
3/ Using an action button that can be configured for both databases, you can create a LS function that will open the the opposing databases view and return the ID field. (NotesUIWorkspace.pickliststrings would be the simplest way to pick the documents, otherwise you could build a dialogbox. Store the list of results in a field called "linkedID" as multi-value list.
4/ There may be more info that you want to store like document title or author, so you'll need to then get a handle to those documents using getdocumentbykey and then interrogating the fields you'll need to display information on screen.
5/ You can then also add a new field on the target documents you're referring to, call it "referrerID", which is a list of documents that reference the current document. This will maintain the two-way relationship.
The field that stores links must be a multi-valued field, otherwise it gets quite cumbersome to loop through list of linked document ID's and manage them.
This approach uses a static key so you can copy databases around without losing the relationships between documents the user has invested time in producing. You can (and probably will) lose those relationships if you rely on document universal ID's (described well in the #documentUniqueID documentation), if you cut and paste the document, or copy the database somewhere else they become new documents despite copying the same fields, and will be assigned a new universal ID, any document links for the old document will be invalid.
If the information you're displaying from the other database changes, you'll need to be able to refresh that data regularly, so consider writing a scheduled agent that can do the look up and refresh the relevant fields.
If the user intends to un-link or change the relationships between documents, then you'll need to add functions that loop through the key fields and keep the lists consistent with what the user is doing. So, like I said, Lotus Notes' flat data structure requires you manage all integrity constraints yourself.
If you want to get a little fancy you can use embedded views as they do support references from another database on the same server. Some tips about handling it in LotusScript here. And use an additional view that categorises your data by the referring ID. Embedded views are ok, as long as the view they're based on is not too big, otherwise it may affect the performance of the form that it is embedded into.
I'm currently writing an application that moves Notes documents between databases based on the amount of days that have elapsed from the creation/modified/last accessed dates. I would just like to get ideas on a simple and convenient way to create documents with specific dates, without having to change the time on the Domino server, so that I could test out my application.
The best way I found so far was to create a local replica and change the system clock to the date I want. Unfortunately there are problems associated with this method. It does not work on the modified date - I'm not sure how it is getting the modified date information when the location is set to Island (Disconnected) - and it also changes the modified and last accessed dates when the documents are replicated to the server replica.
Someone suggested trying to create a DXL of the document, modify the date time in the DXL file, then import it back into the database as a Notes document; but that does not work. It just takes on the date-time that it was created.
Can anyone offer any other suggestions?
You can set the created date for a document by setting the UNID (which is fundamentally a struct of timestamps, although the actual implementation has changed in recent versions). Accessed and modified times, though, would be unsettable from within the Notes/Domino environment, since the changes you make would be overwritten by the process of saving the changes. If you have a flair for adventure and a need to run with scissors, you could make the changes in the database file itself either programmatically from an external application, or manually with a hex editor. (Editing the binary will work -- folks have been using hex editors to clear the "hide design" flag safely for years. Keep in mind that signed docs will blow up badly, and that you need to ensure that local encryption is off for the database file.)
There's actually a very simple way to spoof the creation date/time: just add a field called $Created with whatever date/time you want. This is alluded to in the Notes C API header file nsfdata.h:
Time/dates associated with notes:
OID.Note Can be Timedate when the note was created
(but not guaranteed to be - look for $CREATED
item first for note creation time)
Obtained by NSFNoteGetInfo(_NOTE_OID) or
OID in SEARCH_MATCH.
Unfortunately, there's no analogous technique for spoofing the mod or access dates. At least none that's ever been documented, as far as I know.
I imagine given how dependent Lotus Notes is on timestamps (for replication, mainly), there isn't an API call that allows you to change the modified, created, or last access dates of a note. (More on the internals of Lotus Notes can be found here.)
I dug around the Notes C API documentation, and found only one mention on how to get/set information in the note's header, including the modified date. However, the documentation states that when you try to update that note (i.e. write it to disk), the last modified date will be overwritten with the date/time it is written to disk.
As an alternative, I would suggest creating your own set of date items within the documents that only you control, for example MyCreated, MyModified, and MyAccessed, and reference those in your code that moves documents based on dates. You would then be able to change these dates as easily as changing any other document item (via agents, forms, etc.)
For MyCreated, create a hidden calculated form field with the formula of #CREATED or #NOW. Set the type to computed when composed.
For MyModified, create a hidden calculated form field with the formula #NOW, and set the type to computed.
MyAccessed gets a bit tricky. If you can do without it, I suggest you live work with just the MyCreated and MyModified. If you need it, you should be able to manage it by setting a field value within the QueryOpen or PostOpen events. Problems occur if your users have only read access to a document - the code to update the MyAccessed field won't be able to store that value.
Hope this helps!