VisualStudio

Azure Cosmos DB – Programatically Connect to a preferred location using the SQL API

May 29, 2018 .NET, Azure, CosmosDB, Microsoft, VisualStudio, Windows, Windows Azure Development No comments ,

Cosmos Db is a multi-region scallable, globally-distributed database solution as part of Microsoft Azure Platform.  With a button click, Azure Cosmos DB enables you to elastically and independently scale throughput and storage across any number of Azure’s geographic regions. It offers throughput, latency, availability, and consistency guarantees with comprehensive service level agreements (SLAs),  that no other database service can offer. [REF]

What is multi-region scalability or global distribution ?

What it means is that once you select this option, and underlying platform will ensure that your main database is replicated across other global regions you have defined.

So when a customer/application requests the data from a certain geo location:

  1. Cosmos Db will serve the data from nearest available regional copy to provide low latency in accessing the database.  Inorder to achieve it is recommended to deploy both the application and Azure Cosmos DB in the regions that correspond.
  2. Incase that nearest available region is not defined, it would serve from nearest available or main copy. This could be East US or West US depending on your deployment decisions.
  3. As BCDR(Business Continuity and Disaster Recovery) plan, Incase main copy is not available, it would faillover to serve the requests from any backup region.  

Benefits?

  • Ensured AVAILABILITY @ 99.99% – Azure Cosmos DB offers low latency reads and writes at the 99th percentile worldwide.
  • Faster READS: It ensures that all reads are served from the closest (local) region.  To serve a read request, the quorum local to the region in which the read is issued is used.
  • Reliable WRITES: The same applies to writes. A write is acknowledged only after a majority of replicas have durably committed the write locally but without being gated on remote replicas to acknowledge the writes.

PS: The replication protocol of Azure Cosmos DB operates under the assumption that the read and write quorums are always local to the region where the request has been issued.

How to turn on – Cosmos Db and multi-region replication?

In CosmosDb instance settings select Replicate data globally page, then select the regions to add or remove by clicking regions in the map.

Azure Cosmos DB enables you to configure the regions (associated with the database) for “read”, “write” or “read/write” regions.

image

image 

image

Then configure Manual/Automatic failover options as well. image I would cover this in later articles.

All that said, you are in good hands of Azure Platform as a  Cosmos Db customer or user. 

NB: For the purpose of this article, I have configured my instance to run different regions with write region as East US and read region as West Europe,North Europe and West US.

image

Programatically Connect to a preferred location using the SQL API:

Now coming to the context of this blog, as a application developer some times you would like to programatically control the access to these regions while using Cosmos Db .NET SQL API. 

In CosmosDb.NET SDK version 1.8 and later, there is the ConnectionPolicy parameter for the DocumentClient constructor has a property called Microsoft.Azure.Documents.ConnectionPolicy.PreferredLocations

  • All reads will be sent to the first available region in the PreferredLocations list. If the request fails, the client will fail down the list to the next region, and so on.
  • SDK will automatically send all writes to the current write region.
  • SDK will only attempt to read from the regions specified in PreferredLocations.
  • For example: If you have 4 read regions defined in your cosmos Db instance and you only have 2 regions defined in PreferredLocations in connectionPolicy, requests from other two regions would never be served from SDK.

NB: The client application can verify the current write endpoint and read endpoint chosen by the SDK by checking two properties, WriteEndpoint and ReadEndpoint. **SDK version 1.8+.

Following code snippet would make it easiter to implement:

 
   //Setting read region selection preference. 
   connectionPolicy.PreferredLocations.Add(LocationNames.EastUS); // applications first preference
   connectionPolicy.PreferredLocations.Add(LocationNames.WestEurope); // applications second preference

Full Source Code: https://github.com/AzureContrib/CosmosDB-DotNet-Quickstart-Preferred-Location 

References:

Azure Cosmos DB – Connection Policy – Setting Connection Mode and Connection Protocol

May 13, 2018 .NET, Azure, CosmosDB, Microsoft, PaaS, VisualStudio, Windows, Windows Azure Development No comments , , ,

Recently I have been trying multiple ways to optimize CosmosDb SQL.NET SDK integration calls from my web application that sits within a VNET.

After carefully analyzing different options available within Cosmos Db SQL API’s have realized there are different aspects we could optimize in achieving minimal turn around time. In this article I am going to discuss about one such useful find, that is to use Cosmos Db SQL SDK connection policy to use diferent networking options to improve the latency between web application and cosmos db API calls.

Connection Policy:

Performance of an client application has important implication based on – how SQL .NET SDK  connects to Azure Cosmos DB , because of expected client-side latency due to networking conditions. There are two key configuration settings available for configuring client Connection Policy – the connection mode and the connection protocol.

There are two connection mode options provides by Cosmos Db SQL.NET SDK:

  • Gateway Mode(which is default): This mode is the default option being used and works with all Cosmos DB SDK versions.  Since it is only accessible over HTTPS/TCP, it is more secure and best choice for applications that run on a constrained secure corporate network. If you are using the .NET Framework version of the CosmosDb SQL.NET SDK, then proably this is the only connection mode that would work for you. 

  • Connection Protocol – TCP:  443 is the CosmosDb port, 10255 is the MongoDB API port.   
  • Connection Protocol – HTTPS: Default 443
  • Direct Mode:  This is a new mode which will work only on .NET Standard 2.0 onwards. It provides you an ability to choose between TCP or HTTPS more efficiently.  Only caveat is that you would need .NET Standard 2.0 as target framework for your client application.
    • Connection Protocol – TCP: TCP would be more faster when client and db are in same VNET.  Since TCP within the same network would be more faster, you would be amazed by the latency improvements by your client application. It would respond faster to you cosmos Db requests.  NB In TCP mode apart from 443 and 10255 mentioned in Gateway more, we also need to ensure  port range between 10000 and 20000 is open in your firewall configuration,  because Azure Cosmos DB uses dynamic TCP ports.
    • Connection Protocol – HTTPS: Since client application and cosmosDb are in same network limits, you could see that HTTPS option is also a reliable, secure and faster access channel for you, but not highly performing as TCP.

    A simplified diagram below :

    image

    Sample Code:

     string cosmosDbEndpoint = new Uri("https://mycosmosDbinstance.documents.net");
     string authKey ="cosmosDb-apiKey";
     DocumentClient client = new DocumentClient(cosmosDbEndpoint, authKey,
     new ConnectionPolicy
     {
        ConnectionMode = ConnectionMode.Direct,
        ConnectionProtocol = Protocol.Tcp
     });
     

    Refer more :

    You can find the completed sample here: AzureContrib/CosmosDB-DotNet-Quickstart-With-ConnectionPolicy

    General Availability of Azure Database Services for MYSQL and PostgreSQL

    March 23, 2018 Azure, Cloud Computing, Cloud Services, Data Services, Emerging Technologies, Microsoft, Tech Newz, VisualStudio, VS2017 No comments

    It has been a while I have written something on my blog. I thought of getting started again with a good news that Microsoft Azure team has announced the general availability of Azure Database Services for MySQL and PostgreSQL. In my earlier posts, I have provided some oversight into Preview Availability of these services as part of the Azure cloud. Now that it is generally available, customers should be able to utilize these services for their general purpose or enterprise level database requirements in Azure Cloud.

    You may read about it more on Microsoft announcement blog Announcing general availability of Azure database services for MySQL and PostgreSQL  by Tobias Ternstrom Principal Group Program Manager, Azure Data

    Getting Started local development with Azure Cosmos DB services – Part 2

    May 29, 2017 .NET, .NET Core 1.0, .NET Core 1.0.1, .NET Framework, ASP.NET, Azure, Azure SDK Tools, Azure Tools, Cloud Computing, CodeSnippets, CosmosDB, Document DB, Microsoft, PaaS, SaaS, Visual Studio 2015, Visual Studio 2015 Update 3, Visual Studio 2017, VisualStudio, VS2015, VS2017, Windows, Windows 10, Windows Azure Development, Windowz Azure No comments

    In my previous article we discussed about setting local development environment using Cosmos DB Emulator for Windows. With this part 2 of the article, we will cover developing, debugging and integration related aspects of using Cosmos DB Emulator.

    Developing with Cosmos DB Emulator

    Once you have Cosmos DB emulator installed and running on your machine, you can use any supported Cosmos DB SDK or Cosmos DB REST API to interact with emulator. This process is same as you are using a Cosmos DB cloud service.

    Cosmos DB Emulator also provides a build-in visual explorer through which you can view,create and edit collections and documents.

    image

    Before you integrate Cosmos DB SDK or Cosmos DB REST API you would need to generate master key for authentication. Unlike cloud service, Cosmos DB emulator only support single fixed account and master key.  You would not be able to communicate with Emulator without this master key.

    Default Master Key:

    Account name: localhost:<port>

    Account key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==

    PS: This key is only to be used in Emulator. You cannot use the same key for Production(Cosmos DB Cloud Service).

    Furthermore, if you want to set your own key. You can go to command line references and run DocumentDB.Emulator.exe with sufficient command switch to set your own key. Remember it should meet the key security requirements. See command-line tool reference for more information.

    The Azure Cosmos DB Emulator is installed by default to the C:\Program Files\Azure Cosmos DB Emulator  or C:\Program Files\DocumentDB Emulator  directory.

    Once you have account name and key, you are good to go with development and debugging using Azure Cosmos DB emulator.

    Let us start looking at how to use CosmosDB SDK. Once you add Cosmos DB SDK for .NET from NUGET sources. You would need to import the following namespaces to reference necessary classes.

     using Microsoft.Azure.Documents;
       
     using Microsoft.Azure.Documents.Client;
       
     using Microsoft.Azure.Documents.Linq;
    

    Simple code to establish connection:

    // Connect to the Azure Cosmos DB Emulator running locally use DocumentClient class in : 
    DocumentClient client = new DocumentClient(
        new Uri("https://localhost:8081"), 
        "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==");
    

    In the above code block we are directly embedding endpoint, key in the source code.But as a suggested approch keeping in mind to easily point to production service would be maintain the key in Web.config appSettings.

       <add value="https://localhost:8081/" key="endpoint"/>
        <add value="C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" key="authKey"/>
     

    Add NuGet reference to Microsoft.Azure.DocumentDB  (always use the latest version of the library)

    image

    For the ease of this article, I am going to use the existing ToDoList sample from DocumentDB Samples provided by Microsoft. You can originally find the same source from C:\Program Files\DocumentDB Emulator\Packages\DataExplorer\quickstart.

    image

    Copy and Unzip DocumentDB-Quickstart-DotNet.zip and open todo.sln in Visual Studio 2017 and your solution structure will look like below:

    image

    Now run the application in your Visual Studio.

    1. You will see an initial screen:

    image

    2. Click on Create New:

    image

    3. New record will be added to your Azure Cosmos DB Emulator:

    image

    4. To verify in Cosmos DB emulator now open Cosmos DB explorer, click on Collections and Select ToDoList

    image

    5.Expand Documents and select item with id:da305da3-c1dc-4e34-94d9-fd7f82d26c58

    image

    Hope this article was helpful for you with initial development.  Share your feedback through comments and share this to your friends and colleagues.

    Useful Links:

    Getting Started local development with Azure Cosmos DB services – Part 1

    May 20, 2017 .NET, Azure, Azure SDK, Azure SDK Tools, Azure Tools, Cloud Computing, Computing, CosmosDB, Data Services, Document DB, Emerging Technologies, KnowledgeBase, Microsoft, PaaS, Visual Studio 2013, Visual Studio 2015, Visual Studio 2017, VS2013, VS2015, VS2017, Windows 10, Windows Azure Development, Windows Server 2012 R2, Windows Server 2016, Windowz Azure 1 comment , ,

    Azure Cosmos DB is a multi-API, multi-model highly scalable NoSQL database services from Microsoft Azure platform. In order to develop an application consuming Azure Cosmos DB requires an azure live subscription or emulator in your local machine.

    The Azure Cosmos DB Emulator provides a local development/test environment for Azure Cosmos DB development purposes. Using Azure Cosmos DB Emulator, you can develop and test your application locally, without needing an azure subscription or without subscription costs.

    With this article I am going to take you through necessary steps and requirements to set up your local environment.

    1. Pre-Requisites:

    Azure Cosmos DB emulator has the following software and hardware requirements:

    • Software requirements
      • Windows Server 2012 R2, Windows Server 2016, or Windows 10
    • Minimum Hardware requirements
      • 2 GB RAM
      • 10 GB available hard disk space

    2. Installation:

    • Download Azure Cosmos DB Emulator   (DocumentDb.Install.msi)   ** do not get confused by the name. Azure Cosmos DB is a super set of Document DB, and the DocumentDb emulator they tweaked a bit to support Cosmos Db.
    • Install DocumentDb.Install.msi

    Additionally Azure CosmosDB emulator can be run on Docker for Windows. After installing Docker for Windows, you can pull the Emulator image from Docker Hub.

    docker pull microsoft/azure-documentdb-emulator

    imageimageimage

    3. Start/Launch Azure Cosmos DB Emulator:

    image image

    After some time you can see the emulator started. When the Azure Cosmos DB emulator launches it will automatically open the Azure Cosmos DB Data Explorer in your browser.

    The address will appear as https://localhost:8081/_explorer/index.html

    Incase you have closed browser and later would like to open the explorer again, you can open the Data Explorer by right clicking on the taskbar menu.

    image

    image

    Now you can write some sample app to try it, or download already created sample applications from Microsoft depending on the preferred platform of your choice.

    4. Limitation of Azure Cosmos DB Emulator: (or Differences between Azure Cosmos DB Emulator vs Real Cosmos DB Cloud Service)

    Since the Azure Cosmos DB Emulator provides an emulated environment running on a local developer workstation, there are some fundamental differences between the emulator and an Azure Cosmos DB account in the cloud:

    The following table is also helpful in determining when to use Cosmos DB Emulator and when direct cloud service. Depending on the choice of requirement, you would need to use associated services efficiently.


    Cosmos DB Emulator Cosmos DB Cloud Service
    Supports only a single fixed account and a well-known master key. Key regeneration is not possible. Supports multiple accounts and different master keys. You can regenerate keys any time from Azure Portal.
    Non scalable Highly scalable
    Does not support larger data sets Support for large data sets
    Does not simulate consistency levels Different Consistency levels available
    Does not simulate multi-region replication Configurable as part of the platform, as needed basis.
    Does not support quota override feature Supports document size limit increases, increased partitioned collection storage etc.
    Might not support most recent changes to Cosmos Db platform Most recent platform update will be available.

    Hope this article was helpful for your initial start. If you would need to understand further on Azure Cosmos DB development follow the links. I will be writing further insights in later sessions.

    Azure in China

    May 17, 2017 .NET, AppFabric, Azure, Azure In China, Azure IoT Suite, Cloud Computing, Cloud Services, IaaS, Intelligent Cloud, Intelligent Edge, IoT Hub, Media Services, Microsoft, Mobile Services, PaaS, SQL Azure, SQL Server, VisualStudio, Windowz Azure 1 comment

    Microsoft Azure presence in China is always a question when there is a need for any customer to deploy azure applications specifically for Chinese Regional customers.

    Recently I had an interaction with a Microsoft Certified Trainer, who carelessly said Azure only uses Chinese partner based environment is only for serving CDN (Content Delivery Network) needs, partial knowledge is difficult to convince. Being a Certified Trainer, he should have known these. As I recollect my earlier experience with azure, I was aware that Microsoft has partnered with 21Vianet , to setup a Chinese instance of Azure with most possible IaaS, PaaS and SaaS offerings are available to Chinese customers. At the same time data would reside within Chinese governments data privacy and storage regulations.

    Similarly other competitor cloud providers such as IBM also partnered with 21Vianet to setup their BlueMix instances in China.

    Microsoft Azure in China

    General availability of Azure in China through 21vianet partnership has been announced in March 26th 2014. These Azure Services in Chinese region will be operated and sold by 21vianet.

    Quoting to Doug Hauger GM- National Cloud Programs, Microsoft

    Windows Azure, operated by 21Vianet, is based on the same technology as Microsoft’s global Windows Azure service and offers a reliable, flexible and value-based service to bring Microsoft’s latest cloud technologies and benefits to Chinese customers and propel forward technological development in China as a whole.

    Why Partnership?

    As per Chinese Government(PRC) Cyber Security Law and Governance policies there is a clear restriction for a foreign entrant to obtain license and provide services to Chinese customers.  All the data relating to Chinese customers should reside within Chinese Data Centres, and foreign companies would have to pass on the operational ownership to a Chinese company in order to operate within PRC(Peoples Republic of China).

    In Additional to that on November’2016 new amendments attached to new PRC Cyber Security Law (effective 1st June 2017).

    As per DLA PIPER

    The new law will come into force on 1 June 2017 and has significant implications for the data privacy and cybersecurity practices of both Chinese companies and international organisations doing business in China.

    The new PRC Cybersecurity Law intends to combat online fraud and protect China against Internet security risks. In short, it imposes new security and data protection obligations on “network operators”; puts restrictions on transfers of data outside China by “key information infrastructure operators”; and introduces new restrictions on critical network and cybersecurity products.  [Read more from Source]

    Where is Azure in China?

    You can access Azure in China instance through a separate portal https://www.azure.cn. Azure Portal is available at https://portal.azure.cn/. Pricing information is available on http://www.azure.cn/pricing/pia/  (click on English to get an English version).

    Below are two data centre regions Azure is available in China.

    image

    Services Offered in Azure China Instances:

    Not all all services of Azure including Mobile Services, Analytics and IoT Suite are available for Chinese regions. I will try to cover a specific list of services that are available for China region.Azure China is a few versions behind regular Azure and not all services are supported. Some of the services supported are:

    • Compute – Virtual Machines (For IaaS Services)
    • Compute – Cloud Service (For PaaS Services)
    • Compute – Web Site
    • Data Services – Storage (For Blobs, tables and Queues)
    • Data Services – SQL Database
    • Data Service – HDInsight
    • SQL Database – SQL Import/ExportNetworks – Virtual Network (To communicate between cloud services via Local networks)
    • Service Bus
    • Active Directory
    • Access Control Service – which has its own UI and namespace
    • Caching – only dedicated cache is supported
    • Mobile Service
    • Media Service
    • CDN
    • Traffic Manager
    • Azure IoT Suite
    • Notification Center
    • Event Center
    • Service Fabric
    • Batch Service
    • Flow Analytics, Power BI
    • Document DB, SQL Database, SQL Data Warehouse, MySQL
    • and many more

    image

    A list of supported and not supported services can be found at (This is in Chinese, and it is a working document, use google translate to take you through all steps necessary for you to understand services offered in Azure China) http://www.windowsazure.cn/documentation/articles/developerdifferences/#readyonwacn

    This also includes all the endpoints that are required for developers to integrate/develop applications targeting China region.

    Hope this information was helpful to you. I would assume it would be helpful for some who are specifically looking forward for providing technology solutions for Chinese customers through Azure Cloud.

    Sources: