February 26, 2016

Better SharePoint Records Management with Analytical Search

5 minute read

Since we've been working in the SharePoint records management industry for so many years we have the good fortune of knowing some of the best SharePoint ECM experts in the world so for this article we tapped SharePoint MVP Steve Curran, Distinguished Engineer at KnowledgeLake. You can read more about Steve on his blog.

Many companies use SharePoint to store their records along with third party tools to manage them. The “File Plan” is the first step in the process of SharePoint records management. Here a records manager will identify criteria and rules to classify what constitutes a record. This criteria can consist of metadata, location (scope), or business activity. The majority of the time the metadata is used to search for documents that qualify as possible records. This qualification is limited to how well the search technology can identify records using metadata. SharePoint and other repositories make it difficult to search for records because of the structure of the metadata. In this post I will explain why SharePoint and other repositories limit your ability to identify records. I will also explain why using analytical search can make for more accurate records classification.

The Limitation of Key Values

You can store your documents in SharePoint along with metadata. The metadata is limited to columns with values very similar to storing things in a structured SQL table. In order to find records in SharePoint you must search using these columns of data. The columns are used to create “managed properties” that you use to search. A typical search in SharePoint may look like this:

Title:Agreement AND Created:10/25/2014

A SharePoint search uses the managed property name + colon + value to search. You can combine numerous managed properties and values with either an AND or an OR to search for records. This type of searching for classification may be good enough in many cases. However, in cases where you are searching for “Vital” records (records containing information essential to the survival of an organization in the event of a disaster) the use of key values may not be enough. For example some companies may consider certain accounting or tax documents vital. Unless you specifically index the document with a key and value that can be used to search for these type of documents, then it is likely not to be identified. Using key value searching will result in many non-vital records being classified as vital.

Documents are more than Keys and Values

Most business documents have some inner structure that organizes the data contained in them. For example an invoice will have the account number, name and address and line item detail. If part of the classification is to find invoices that contain certain products then the search would have to be against the line item detail contained in the invoice. SharePoint and other repositories that only support key and value metadata storage would not be able to do this type of search. The actual structure of the data within the document would have to be stored and indexed. This could be accomplished using JSON (Java Script Object Notation) to store the metadata. Below is an example of an invoice in JSON format.

   "vendorname": "Chemex Container",
   "items": [
         "productdesc": "50 gal cannister",
         "productid": 1256,
         "productuom": "ea",
         "quantity": 12,
         "price": 25
         "productdesc": "25 gal drum",
         "productid": 1257,
         "productuom": "ea",
         "quantity": 12,
         "price": 10
   "discountamt": 5,
   "discountdate": "2014-02-28T00:00:00",
   "vendor": 1600,
   "duedate": "2014-03-31T00:00:00",
   "invoicetotal": 420,
   "invoicenumber": 2569

This invoice data is complex, however the structure and meaning of the data has been retained in the JSON.

Document Oriented Searching

For SharePoint records management, the key to more accurate record classification is using document oriented searching. Document oriented searching indexes, searches, sorts, and filters documents on the whole object not just on key value pairs. This is a fundamentally different way of thinking about data and is one of the reasons document oriented search can perform complex searches. Using analytical search technologies record classification can leverage searching on any type of metadata structure. Retaining the inner metadata structure of documents like invoices, tax forms, resumes, applications, and contracts can make record classification more exact, thus reducing the costs of manually identification, retention, compliance and storage.

Records Management and Document Oriented Searching

Document oriented searching can enhance many parts of records management. The ability to index and search any type of document structure can enable better identification of records based on business function. For example, analytical search could discern between invoices and contracts based on the structure of the metadata rather than just keys and values. Analytical search has a much richer syntax for finding related documents so you can build better case and subject file classifications.

Receive News Updates As Soon As They Happen