Partial row update
Hi Will
Our project based on dbsight has been accepted and we just started development. Has the document number and size will be quite large and retrieval for indexing will be costly (we'll fetch and decrypt documents live from an encrypted file vault), we'd like to anticipate performance problems as early as possible.
In the file vault, access rights to documents are strictly restricted based on user name. So, in order to avoid filtering dbsight results sets in the client application (fetched through a web service) we plan to include a metadata column containing the user lists (and maybe groups too if necessary), so we can specify the username in the query and avoid to present documents whose access would be later denied to the user (and also avoid showing confidential extracts/summaries to wrong users). That would greatly simplify the client application code too versus a "live filtering and repaging" step.
Does this sounds a good idea with dbsight ?
However that means that whenever the user list changes for a document, we must change the modification date in the master dbsight table (actually, a view) so it can be reindexed. We thus fear that user additions/deletion to frequently used groups ("almost everybody group") would trigger a large index update and massive file vault access (as de documents would need to be decrypted again even if the document itself has not been modified).
So here is my question: is there a way or an applicable trick to only update partial indexes (rows) (In fact I don't even know if lucene itself supports this) and could this be used in dbsigh ? I'm thinking to solutions like:
- using multiple modification times in the main query and/or in subqueries ?
- using multiple indexes if we can run a query across indexes ?
- returning a special constant in the document blob that would tell dbsight it can keep the previously indexed datas for this row (seems the most easy way to go) ?
- any other idea or advice ?
thank you
kind regards
Gaetan