In SharePoint search the most of time people annoying that actual content and search content not in Sync. So the search administrator keeps hitting head on the wall and putting same excuse in front of stakeholders “please wait for the next incremental crawl, most of the time :)”. As we know we already have two content crawling methods first is “Full Crawl” and second is “Incremental crawl”.

 

Disadvantage of the “Full Crawl” and “Incremental Crawl” as both can’t run in a parallel i.e. the content change during the crawl, it required next incremental crawl.
So what is new in continues crawl?
The content source that using continues crawl that run in parallel. The default waiting time is 15 min. So the default wait time can change via PowerShell , no UI for that. Now the content is up to date most of the time. This crawler only for SharePoint content source, so the job of the SharePoint administrator need to identify those content which are keep updating on the regular interval & also comes under the part of search need to be comes under “Continues crawl “category. The “Continuous Crawl” is a type of crawl that aims to maintain the index as current as possible. So the following are list of crawl are available in SharePoint 2013 search architecture.
  1. Run By User
    • Full Crawl
    • Incremental Crawl
    • Continues crawl
  2. Run By system (automated crawl)
    • Incremental Crawl (clean-up)
  1. Run by User: The content source created by user/Administrator and it is trigger/ scheduled by the user.
    • Full Crawl: 
      • Crawl full items
      • Can be scheduled
      • Can be stop and paused
      • When required
        • Change content access account
        • Added new manage properties
        • Content enrichment web service codes change/modified.
        • Add new IFilter
    • Incremental Crawl:
      • Crawl last modified content
      • Can be scheduled
      • Can be stop and paused
      • When required
        • Crawl last modified content
    • Continues Crawl
      • Index as current as possible.
      • Cannot be scheduled
      • Cannot be stop and paused (Once started, a “Continuous Crawl” can’t be paused or stopped, you can just disable it.)
      • When required
        • Content frequently changed (Multiple instance can be run in parallel).
        • Only for SharePoint Content Source
        • E-commerce site in crass site publishing mode.
  2. Run by System: The crawl run automatically by the timer job.
    • Clean-Up continues crawl (Microsoft definition): A continuous crawl does not process or retry items that return errors more than three times. A “clean-up” incremental crawl automatically runs every four hours for content sources that have continuous crawl enabled to re-crawl any items that repeatedly return errors. This incremental crawl will try to crawl the item again and then will postpone retries if the error persists.

**********************************************************************************************************

SharePoint 2013: Continuous Crawl and the Difference Between Incremental and Continuous Crawl

With the new version of SharePoint a new type of crawl appeared in 2013 named « Continuous Crawl ».  For Old schools like me on SharePoint 2010 we had 2 crawls available and configurable on our Search Service Application.

  • Full : Crawl all content,
  • Incremental : As the name is says, it crawls content has been modified since the last crawl.

The disadvantage of these crawls, is that once launched, you are not able to launch a second in parallel (on the same content source), and therefore the content changed in the meantime we will need to wait until the current crawl is finished (crawl and another) to be integrated into the index, and therefore to be found via search. An example :

  • A incremental crawl named ALFA is started and will last 50 take minutes,
  • After 10 minutes of crawling a new document has been added, so we need a second incremental crawl named BETA to get the document in the index.
  • This item will have to wait at least 40 minutes to be integrated into the index.

 

So, we can’t keep an updated index with the latest changes, because latency is invited in each crawling process. It is possible that in most of cases this operation is suitable and favorable for your clients, but for those who want to search their content immediately or after their integration into SharePoint there is now a new solution in SharePoint: “Continuous Crawl“.

 

The Continuous Crawl  So resuming: The “Continuous Crawl” is a type of crawl that aims to maintain the index as current as possible.

His operation is simple: once activated, it will launch the crawl at regular intervals. The major difference with incremental crawl is that the crawl can run in parallel, and do not expect that the crawl is completed prior to launch.

Important Points:

  • “Continuous Crawl” is only available for sources of content type “SharePoint Sites”
  •  By default, a new crawl is run every 15 minutes, but the SharePoint administrator can change this interval using the PowerShell cmdlet Set-SPEnterpriseSearchCrawlContentSource  ,
  • Once started, a “Continuous Crawl” can’t be paused or stopped, you can just disable it.

If we take our example above with “Continuous Crawl”:

  •  Our ALFA crawl starts and will take at least 50 minutes,
  •  After 10 minutes of crawling an item already crawl is hereby amended, and requires a new crawl.
  •  Crawl “BETA” is launched,
  •  The crawl “BETA” starts in (15-10) minutes,
  •  Therefore this item will not need to wait 5 minutes (instead of 50 minutes) to be integrated into the index.

 

 

1- How to Enable it?

In Central Administration, click on your search service application, and then in the menu on the “Content Sources”

 

Clique on « New Content Source » at the menu

 

Chose « SharePoint Sites »

  

Select « Enable Continuous Crawls »

  

 

  • The content source has been created so we can see his status on « Crawling Continuous »

 

2 – How to disable it?

  •  From the content source page, chose the option “Enable Incremental Crawls” option. This will disable the continuous crawl.
  •  Save changes.

 

3 – How to see if it works ?

Click on your service application search then “Crawl Log” in the section “Diagnostics”.

 

Select your Content Source and click on « View crawl history »

Or via PowerShell Execute the followoing cmdlets  $SearchSA = « Search Service» Get-SPEnterpriseSearchCrawlContentSource -SearchApplication $SearchSA | select *

 

Impact on our Servers

The impact of a “Continuous Crawl” is the same as an incremental crawl. At the parallel execution of crawls, the “Continuous Crawl” within the parameters defined in the “Crawler Impact Rule” which controls the maximum number of requests that can be executed by the server (default 8).

Advertisements

SPQuery List Joins

Posted: January 3, 2014 in SharePoint
Tags: ,
SharePoint 2010 Step by Step SPQuery List joins Using CAML
I am using the example following example lists
  1. CustomerCity List
  2. Customer
List Columns
CustomerCity columns are:
Tilte
Single Line text
Customer columns are:
Title
Single Line text
City
Lookup type of CustomerCity
Dummy Data used
For CustomerCity
Title
City1
City2
City3
For Customer
Title
City
Customer1
City1
Customer2
City1
Customer3
City2
Customer4
City2
customer5
City2
Customer6
<NULL>
Using the Join in SharePoint 2010 List Using SPQuery Class.
We need to set the three most important properties for that.
  1. Joins
  2. ProjectedFields
  3. ViewFields
SharePoint 2010 adds Join To CAML SPQuery.Joins
Types of joins
  1. Inner
  2. Left
Requested Lookup columns
Projections allow inclusion of fields joined lists
SPQeury.ProjectedFields
Joins: Each join is represented by a Join element child of the Joins element. Only inner and left outer joins are permitted. Moreover, the field in the primary list must be a Lookup type field that looks up to the field in the foreign list. There can be joins to multiple lists, multiple joins to the same list, and chains of joins. If a given list is the foreign list in more than one join, it must have distinct aliases assigned to it by the ListAliasattributes of the Join elements representing the joins.
Note: Multiple Lines of text, Choice type columns are not supported in ProjectedFields.
private void button1_Click(object sender, EventArgs e)
        {
            string siteUrl = “http://home&#8221;;
            SPWeb _web = new SPSite(siteUrl).OpenWeb();
            var items =_web.Lists[“Customer”].GetItems(GetQuery());
            foreach (SPListItem item in items)
            {
                MessageBox.Show(string.Format(“{0}—-{1}”, item[“Title”], item[“CityTitle”]));
            }
        }
        private SPQuery GetQuery()
        {
            SPQuery _query = new SPQuery();
            _query.Query = “”;
            _query.Joins = @”<Join Type=’INNER’ ListAlias=’City’>
                          <!–List Name: CustomerCity–>
                          <Eq>
                            <FieldRef Name=’City’ RefType=’ID’ />
                            <FieldRef List=’City’ Name=’ID’ />
                          </Eq>
                        </Join>”;
            _query.ProjectedFields = @”<Field Name=’CityTitle’ Type=’Lookup’ List=’City’ ShowField=’Title’ />
                                    <Field Name=’CityContentTypeId’ Type=’Lookup’ List=’City’ ShowField=’ContentTypeId’ />”;
            _query.ViewFields = @” <FieldRef Name=’Title’ />
                                     <FieldRef Name=’CityTitle’ />”;
            return _query;
        }
The above “Query” property of the SPQuery class I left blank you can enter the condition according to your requirement.

SharePoint lists are, by and large, similar to database tables in structure and behavior. Like Tables, Lists support different data types and can have triggers (event receivers) that fire on events such as creating, updating or deleting items. Also, lists can be configured to filter, sort or group items based on column values.

Also, SharePoint lists support the concept of view like in database and can be queried by CAML. The columns in the list can be indexed for better performance while querying.

In SharePoint 2010, the lists are much more enhanced to incorporate important aspects of RDBMS like :

  1.  Referential Integrity
  2.  Unique Keys
  3.  Joins

1.  Referential Integrity with improved lookup columns

In previous versions of SharePoint (2003, 2007), the  lists lacked an important aspect of RDBMS – Referential integrity- that is a measure of data consistency in a database. If you have a lookup between two lists and you want to have Referential Integrity, previous SP versions won’t provide it. The lookup columns could be used just to get their data from other list columns.

Now, SharePoint 2010 provides the mechanism to enforce Referential integrity in our data model. Also, it can block or cascade your deletes between lists automatically.

Just like foreign key constraints in RDBMS, SharePoint 2010 allows you to configure restrict delete and cascade delete rules on lookup column relationships.

• Cascade delete : When we delete a list item, SharePoint automatically delete all items that have references to that item by look up column. It can be used to enforce parent-child relationships.

• Restrict delete : With this rule, SharePoint don’t allow you to delete a list item that is referenced by a lookup column in another list. It can be used to enforce peer-to-peer relationships.

[Note: SharePoint is not going to be a replacement for SQL Server with this functionality. If you have a very complex data model, SQL Server is an obvious choice]

An additional improvement on lookup columns is that now we can retrieve more columns(Projected Columns) from the related list.

Configuring the Delete Behavior on a Lookup Field 

We can configure list relationship rules through the user interface when we create look up column.

Look Up field

We can also programmatically configure delete behavior in a feature receiver class. Since the list relationship is specific to individual list instances, we cannot configure it declaratively in column definitions or content types.Programmatically setting the delete behavior can be useful if you use features to deploy the lists and columns.

SPFieldLookup is class that represents lookup columns. This class has a RelationshipDeleteBehaviorenum property that enables us to specify the delete behavior for the column.It can be set to one of the following SPRelationshipDeleteBehavior enumeration values:

  • SPRelationshipDeleteBehavior.Cascade. Setting this value deletes all items if the lookup column refers to an item that no longer exists in the related list.
  •  SPRelationshipDeleteBehavior.Restrict. Setting this value prohibit the users from deleting items in the related list if those items are referred to in the lookup column values.

For example, we can use below methods in a feature receiver class to restrict or cascade delete.

private void RestrictDelete(SPWeb web)

{

SPList list = web.Lists[“MyList”];

SPField field = list.Fields[“TestNameLookUp”];

SPFieldLookup fieldLookup = (SPFieldLookup)field;

//For unique values of column in items

fieldLookup.EnforceUniqueValues = true;

//For indexing column

fieldLookup.Indexed = true;

fieldLookup.RelationshipDeleteBehavior = SPRelationshipDeleteBehavior.Restrict;

fieldLookup.Update();

}

private void CascadeDelete(SPWeb web)

{

SPList list = web.Lists[“MyList”];

SPField field = list.Fields[“TestNameLookUp”];

SPFieldLookup fieldLookup = (SPFieldLookup)field;

//For unique values of column in items

fieldLookup.EnforceUniqueValues = true;

//For indexing column

fieldLookup.Indexed = true;

fieldLookup.RelationshipDeleteBehavior = SPRelationshipDeleteBehavior.Cascade;

fieldLookup.Update();

}

Programmatically Find Lookups and Related Lists

We can use the GetRelatedFields method to return a SPRelatedFieldCollection collection. We can retrieve properties, such as the LookupList that the column  is related to,  the relationship behavior when some item is deleted from the list and also some other useful information.

Below is the example

private void GetRelatedColumnsAndListsInfo(SPWeb web)

{

SPList list = web.Lists[“MyCustomersList”];

//Get related columns

SPRelatedFieldCollection relatedFields = list.GetRelatedFields();

foreach (SPRelatedField relatedField in relatedFields)

{

//Get Lookup list

SPList relatedList = relatedField.LookupList;

Console.WriteLine(relatedField.ListId + ” ” +   relatedField.FieldId);

Console.WriteLine(“List Name: ” + relatedList.Title + ” Relationship Behavior: ” + relatedField.RelationshipDeleteBehavior.ToString());

}

}

Controlling the cascade delete limits and time outs

At web appilcation level, We can have control over the Cascade deletes  usingSPWebApplication.CascadeDeleteMaximumItemLimit andSPWebApplication.CascadeDeleteTimeoutMultiplier.

  • CascadeDeleteMaximumItemLimit allows you to specify the maximum  number of cascaded items that can be deleted. By default, this value is 1000 items.
  •  CascadeDeleteTimeoutMultiplier  allows you to specify the timeout(in secs)  for the operation. The default value is 120 seconds.

private void SetCascadeDeletes(SPWebApplication webApp)

{

//sets max cascade delete limit to 2000

webApp.CascadeDeleteMaximumItemLimit = 2000;

//sets the time out to 5 mins

webApp.CascadeDeleteTimeoutMultiplier = 300;

webApp.Update();

}

2.  Unique Keys (Columns)

SharePoint 2010  lists have the ability to ensure uniqueness for the values in your columns. As the unique column  is guaranteed to have a unique value, We can  it as an  index to make look ups faster.

LookUp1

3. List Joins

SharePoint lists support joins like database lists.  SharePoint can perform left and inner joins but not right joins.

  • By inner join, we can combine the values from the liststables.
  • By left join or left outer join , anything that appears in the left listtable  will be returned in the result set even if it does not exist in the right listtable.

We can perform a join between two lists on a lookup field by setting the Joins property on our SPQueryobject with the join we want to perform.

Apart from the Joins property, we must also specify a value for the ProjectedFields property. This property gets other required columns from the lookup list. We can alias the column by using the Name attribute and specify the column name in the ShowField attribute. After getting results, we have to use theSPFieldLookupValue  to display the values for projected columns.

In the below example, we are joining on the Customers list, where the Customers.customer = Orders.Customer

private void PerformListJoin(SPWeb web)         {

SPList OrderList = web.Lists[“Orders”];

SPQuery CustomerQuery = new SPQuery();

CustomerQuery.Joins =             ” <Join Type=’INNER’ ListAlias=’Customers’> ” +             ” <Eq> ” +             ” <FieldRef Name=’Customer’ RefType=’Id’ /> ” +             ” <FieldRef List=’Customers’ Name=’ID’ /> ” +             ” </Eq> ” +             ” </Join> “;

StringBuilder ProjectedFields = new StringBuilder();

ProjectedFields.Append(” <Field Name=’CustomerTitle’  Type=’Lookup’ List=’Customers’ ShowField=’Title’ /> “);

ProjectedFields.Append(” <Field Name=’CustomerAddress’Type=’Lookup’ List=’Customers’ ShowField=’CustomerNum’ /> “);

CustomerQuery.ProjectedFields = ProjectedFields.ToString();

SPListItemCollection Results = OrderList.GetItems(CustomerQuery);

foreach (SPListItem Result in Results)             {

SPFieldLookupValue CustomerTitle = new SPFieldLookupValue(Result[“CustomerTitle”].ToString());

SPFieldLookupValue CustomerAddress = new SPFieldLookupValue(Result[“CustomerAddress”].ToString());

Console.WriteLine(Result.Title + ” ” + CustomerTitle.LookupValue + ”  ” + CustomerAddress.LookupValue);

}

}

Introducing OAUTH

Posted: January 3, 2014 in SharePoint
Tags:

OAuth is a standard way for Service providers and Consumers(Apps for example) to handle authentication. Also, the OAuth authorization framework enables a third-party application to obtain limited access to a service hosted by Service provider. By using OAuth, you can allow the user to access a service (Facebook wall for example) stored at one site(Service Provider) with another application without having to store or manage credentials at the Application side.

There are 3 main Actors in an OAuth transaction: the User, the Service Consumer ( which is generally an APP), and the Service Provider.  This gang of three are often said to form the OAuth Love Triangle.

OAuth-Lovetriangle

Service: We all understand the idea of Service. A Service can be a Web Service or API that is consumed by client applications

Service consumer : Any application that consumes a service. Now tell me what’s is the most common breed of application now-a-days that consume the Services?….guess… APPs! Yes, Apps are the most common Service Consumers now-a-days.

Service Provider: It’s the one who hosts the service and where the service runs. The service generally requires authentication so that only trusted consumers can access the restricted functionality or data offered by the service.

Now,  imagine as a developer, you have developed an enterprise application that offer users to play some interesting games on their Mobile devices. For that, you created a Windows 8 or iOS App (or any App) whose main function is to entertain user via some puzzle games on his mobile device.

Now, assuming that your Game APP implemented OAuth for FaceBook, it can initiate authentication using OAuth.

Below are the steps(and the technical conversation) that will occur at high level  among all parties:

1.  A user wants Service Consumer (App) to access his protected resource lying with Service Provider (Facebook) :

User: “Hey App, I enjoyed the game but now I would like you to be able to share my game score on my Facebook timeline.”

Game App: “No problem! Let me ask for permission from Facebook. ”

2.  The service consumer asks for request token from service provider.

Game App: “My user wants me to post to his wall. Please share a request token.”

Facebook: “Your request is answered. Take the token and associated secret.”

Using the secret,  the service provider is able to verify the future requests by consumer (if its coming from the valid Service Consumer).

3.  The user is redirected to the service provider where user approve the service consumer to act on his behalf.

Game App: “Hey user,  I’m redirecting you to Facebook so you can approve me for the actions you want. Take the token with you which I got  from Service Provider(Facebook).”

User: “Ok”

4.   The user sees a form presented by the Service provider which lists all the actions the Service Consumer can take on user’s behalf. If user thinks its all OK, he approves.

Facebook: “Welcome User, do you want to authorize the Game App for all the A, B, and C actions?”

 User: “Yes, I approve”

Facebook: “OK, the request token is approved for the Game App”

5.   The service consumer obtains an access token in lieu of request token from service provider

Game App: “Facebook, Can you provide an access token for this approved request token?”

Facebook: “Sure. Take the access token and secret.”

6.  The service consumer preserve the access token and secret information for later use.This information can be saved along with user account with service consumer.

Game App : Hey User, now you can share the score on your wall as long as you keep me authorized with Facebook.

User: Great! But please note I will revoke your permission on my account with Facebook anytime I wish.

7.   The service consumer accesses the protected resource (of user) on behalf of user.

 Game App: “My user wants to share score on his Facebook wall.  Here is the access token for the action”

 Facebook: “Your access token looks valid. Your request can be carried on!”

OAuth is adopted by many Service Providers like Facebook, Twitter,Google, Yahoo and all the Consumers that consume the service from them.

The initial version of OAuth is 1.0 which is  still being used by some software companies. The second and latest version , OAuth 2.0, is created to simplify development while still providing app authentication and specific authorization flows for web apps, desktop applications, and mobile devices.