.NET

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure Cosmos DB–Multi Master

October 8, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Azure, Azure CLI, Azure Cosmos DB, CosmosDB, Data Consistancy, Data Integrity, Microsoft, Multi-master, Performance, Reliability, Resilliancy, Scalability, Scale Up No comments

During the Ignite 2018, Microsoft has announced the general availability of Multi-Master feature being introduced to Azure Cosmos DB to provide more control into data redundancy and elastic scalability for your data from different regions with multiple writes and read instances.

What is Multi-Master essentially?

Multi-master is a capability that provided as part of Cosmos DB, that would provide you multiple write regions and provides an option to handle conflict resolution automatically through different options provided by the platform. Most of the major scenarios you would encounter the conflict can be resolved with these simple configurations.

A sample diagram depicting a use case of load balanced web app writing to respective regional master:-

image

With multi-master, Azure Cosmos DB delivers a single digit millisecond write latency at the 99th percentile anywhere in the world, and now offers 99.999 percent write availability (in addition to 99.999 percent read availability) backed by the industry-leading SLAs.

image

Wow! That’s an amazing performance Cosmos DB guarantees to provide so that your mission-critical systems will have zero downtime, if they start using Cosmos DB.

 

How to Enabled Multi-Master support in your Cosmos DB solutions?

Currently multi-master can only be enabled for new Cosmos DB instances using “Enable Multi-Master” option in Azure Portal or through PowerShell or ARM templates or through SDK.

These options are detailed below with necessary examples:

1.) Azure Portal – Enable Multi-region writes and Enable geo-redundancy

image

2.) Azure CLI 
Set the “enable-multiple-write-locations” parameter to “true”

az cosmosdb create \
   –-name "thingx-cosmosdb-dev" \
   --resource-group "consmosify-dev" \
   --default-consistency-level "Session" \
   --enable-automatic-failover "true" \
   --locations "EastUS=0" "WestUS=1" \
   --enable-multiple-write-locations true \

3.) AzureRM PowerShell
In AzureRM PowerShell cmdlet – Set enableMultipleWriteLocations parameter to “true”

$locations = @(@{"locationName"="East US"; "failoverPriority"=0},
             @{"locationName"="West US"; "failoverPriority"=1})

$iprangefilter = ""

$consistencyPolicy = @{"defaultConsistencyLevel"="Session";
                       "maxIntervalInSeconds"= "10";
                       "maxStalenessPrefix"="200"}

$CosmosDBProperties = @{"databaseAccountOfferType"="Standard";
                        "locations"=$locations;
                        "consistencyPolicy"=$consistencyPolicy;
                        "ipRangeFilter"=$iprangefilter;
                        "enableMultipleWriteLocations"="true"}

New-AzureRmResource -ResourceType "Microsoft.DocumentDb/databaseAccounts" `
  -ApiVersion "2015-04-08" `
  -ResourceGroupName "consmosify-dev" `
  -Location "East US" `
  -Name "thingx-cosmosdb-dev" `
  -Properties $CosmosDBProperties

4.) Through CosmosDB SDK
Setting connection policy in DocumentDBClient and set UseMultipleWriteLocations to true.

ConnectionPolicy policy = new ConnectionPolicy
{
   ConnectionMode = ConnectionMode.Direct,
   ConnectionProtocol = Protocol.Tcp,
   UseMultipleWriteLocations = true,
};
policy.PreferredLocations.Add("East US");
policy.PreferredLocations.Add("West US");
policy.PreferredLocations.Add("West Europe");
policy.PreferredLocations.Add("North Europe");
policy.PreferredLocations.Add("Southeast Asia");
policy.PreferredLocations.Add("Japan East");
policy.PreferredLocations.Add("Japan West");

Azure Cosmos DB multi-master configuration is the game changes that really makes it a true global scale database with automatic conflict resolution capabilities for data synchronization and consistancy.

In my later sessions I will write examples to cover how conflict resolutions can be configured and used in realtime scenarios.

Useful Refs:

Azure Cosmos DB – 429 Too Many Requests

October 6, 2018 .NET, Azure, CosmosDB, Document DB, Microsoft, Performance, Reliability, Resilliancy, Scalability, Visual Studio 2017, VisualStudio, VS2017 No comments

Recently while I was doing Performance Testing in one of the APIs interacting with Cosmos DB, I encountered a problem as Azure Cosmos DB API’s started returning Http Code 429.  Http Status Code 429 indicates that too many request been received or request rate is very large. This error would happen when we have concurrent users trying to write or read from same cosmos db collection.

Following diagram covers the architecture of the performance test I am performing:

image

Based on analysis it found out to be the Throttling happening from Azure Cosmos DB, as we make requests that may use more than provisioned Request Units(RU) per second. We were using default Cosmos DB configuration for a fixed collection of 1000 RU’s per second which is sufficient enough for a 500 reads and 100 writes for a 1 kb file. You can refer more about Request Units from Azure Docs.

image

 

 

 

Solution(s):

1. Now first logical step we can do is to get rid off this error by increasing the Throughput for the collection.  I am going to increase to 10000 RU/s maximum allocatable for a Storage Capacity: Fixed.   This should ideally improve the Throughput for 250 or more virtual users hitting.

image

2. Second logical step is to improve the code: Improve the connection parameters in the Document DB SDK –> DocumentDbClient. For this I referred to the Microsoft Docs: Performance tips for Azure Cosmos DB and .NET

Providing optimum values to the following Properties in RetryOption class   to be passed as parameter to Connection Policy.

image

 

In my case I provided a value of 30 to give ultimate results:

new RetryOptions() { MaxRetryAttemptsOnThrottledRequests = 30, MaxRetryWaitTimeInSeconds = 30  }

That should resolve most of the 429 issues when dealing with Cosmos DB SDK

NDepend–VSTS/Azure DevOps Integration–Part 01

September 30, 2018 .NET, .NET Core, .NET Framework, Azure DevOps, Best Practices, Code Analysis, Code Quality, Dynamic Analysis, Emerging Technologies, Microsoft, Static Analysis, Tools No comments

In my previous article I wrote an introductory about NDepend and how it will be useful for Agile Team to ensure code quality.

In that article we found how we can use NDepend in a developer machine. Now with this article we will familiarize ourselves in using NDepend in your build automation pipeline in your VSTS/Azure DevOps Build Agent.

There are two types of integration possible for NDepend:

  1. Directly using NDepend Package Extension from VSTS Marketplace
  2. Manual Integration using NDepend Command Line Tool. (This would provide you more control over licensing by setting up the license in your own OnPrem VSTS Build Agent.

For the interest of this article I will cover the use of VSTS Package Extension and using NDepend Build Task in VSTS Build Pipeline.

Installation of NDepend Extension for VSTS/Azure DevOps :

1.) Got to Azure DevOps Market Place:  https://marketplace.visualstudio.com/items?itemName=ndepend.ndependextension

image

2.) Click on Get to Install this extension in to your AzureDevOps account and follow the steps. For the demo purpose I am starting with 30 day free trial, otherwise you can go ahead and buy the full license.

image

image

image

3.) Now when you get back to Azure DevOps project, you can see the NDepend side menu enabled, this is where you would see the report summary of your project.

image

Integration NDepend into Azure DevOps Pipeline :

1.) Select “NDepend Task” and add in to Pipeline

image

image

Note:

  • You can choose to stop the build when at least one quality gate fails.
  • You also need to specify the NDepend project file customized for your project, otherwise NDepend will use their default project file configuration.  Having your own NDepend project file will provide you more control over the policies for the scan.

Queue a new Build and wait for Build to complete. Now you can see the BuildArtifacts includes all NDepend report file.

image

Now you go back to NDepend menu from Left side menu item in Summary Tab. This will provide you detailed view of Technical Debt in your project.

image

image

image

image

image

In the next article I will cover the manual integration steps.

New Microsoft Azure Certifications

September 16, 2018 Azure, Azure SDK, Azure Tools, Certification, Emerging Technologies, MCP, Microsoft, Microsoft Learning, Windows Azure Development No comments

Microsoft has recently announced new certification exam tracks for Azure Administrators, Developers and Architects. Here are the line ups that should help you move your career with right certifications. 

The three new Microsoft Azure Certifications are:

  • Microsoft Certified Azure Developer
  • Microsoft Certified Azure Administrator
  • Microsoft Certified Azure Architect

These certifications would essentially split the previous MCSA/MCSE: Cloud Platform and Infrastructure track and introduce new exams for individual certification track. 

So far I only have limited information available about all the exam numbers for each individual track, as recently Microsoft has made BETA exams available for Microsoft Certified Azure Administrator track. 

These exams are still in BETA, would commence general availability in coming months.  Will keep you posted about newer exams for other tracks as we get to know more. 

References: https://www.microsoft.com/en-us/learning/exam-list.aspx 

70-777 – Azure Cosmos DB Certification Exam (BETA)

August 18, 2018 .NET, Azure, Azure Cosmos DB, Certification, CosmosDB, MCP, Microsoft, Microsoft Specialist No comments

70-777: Implementing Microsoft Azure Cosmos DB Solutions”  is the new Microsoft certification in the town for any Developer or Architect who leverages Azure Cosmos DB. If you understand how Cosmos DB database can be configured and consumed using platform agnostic Cosmos DB API, this certification will help you prove your expertise.

To read more about the exam 70-777 visit: https://www.microsoft.com/en-us/learning/exam-70-777.aspx  ( the link will also provide you guidance on the topics covered by this exam)

As on today, 70-777 BETA exam enrollment started, has around 300 beta seats for this exam, and you must register and take the exam before October,1 2018. Hurry if you are passionate enough to learn and try out this certification exam, please register now before you quote is full.

Steps to Utilize this Offer:

Source: Born to Learn Community