Cloud Computing

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure Database for MariaDB: Public Preview

October 4, 2018 Azure Database for MariaDB, Managed Services, MariaDB, OpenSource No comments

During Ignite 2018, Microsoft has announced the availability of Maria DB support in Azure Database services. Today it has been opened for Public Preview for all Azure customers.

mariadbhero

What is MariaDB?

MariaDB is a community-developed fork of the MySQL relational database management system intended to remain free under the GNU GPL.Development is led by some of the original developers of MySQL, who forked it due to concerns over its acquisition by Oracle Corporation.Wikipedia

Azure Database for MariaDB: Public Preview Availability

The Azure Database for MariaDB service is now available in preview. It offers an enterprise-ready, fully managed database service that based on the Community Edition of MariaDB.

The service features open-source compatibility, built-in high availability, dynamic scaling, and flexible pricing. Customers can lift and shift to the cloud and use languages and frameworks of their choice, leveraging the power of MariaDB running on Azure.

To learn more about the service, view the service page, pricing, and documentation.

You can create a MariaDB server by using the Azure portal or Azure CLI.

More References:

Azure Cosmos DB – Connection Policy – Setting Connection Mode and Connection Protocol

May 13, 2018 .NET, Azure, CosmosDB, Microsoft, PaaS, VisualStudio, Windows, Windows Azure Development No comments , , ,

Recently I have been trying multiple ways to optimize CosmosDb SQL.NET SDK integration calls from my web application that sits within a VNET.

After carefully analyzing different options available within Cosmos Db SQL API’s have realized there are different aspects we could optimize in achieving minimal turn around time. In this article I am going to discuss about one such useful find, that is to use Cosmos Db SQL SDK connection policy to use diferent networking options to improve the latency between web application and cosmos db API calls.

Connection Policy:

Performance of an client application has important implication based on – how SQL .NET SDK  connects to Azure Cosmos DB , because of expected client-side latency due to networking conditions. There are two key configuration settings available for configuring client Connection Policy – the connection mode and the connection protocol.

There are two connection mode options provides by Cosmos Db SQL.NET SDK:

  • Gateway Mode(which is default): This mode is the default option being used and works with all Cosmos DB SDK versions.  Since it is only accessible over HTTPS/TCP, it is more secure and best choice for applications that run on a constrained secure corporate network. If you are using the .NET Framework version of the CosmosDb SQL.NET SDK, then proably this is the only connection mode that would work for you. 

  • Connection Protocol – TCP:  443 is the CosmosDb port, 10255 is the MongoDB API port.   
  • Connection Protocol – HTTPS: Default 443
  • Direct Mode:  This is a new mode which will work only on .NET Standard 2.0 onwards. It provides you an ability to choose between TCP or HTTPS more efficiently.  Only caveat is that you would need .NET Standard 2.0 as target framework for your client application.
    • Connection Protocol – TCP: TCP would be more faster when client and db are in same VNET.  Since TCP within the same network would be more faster, you would be amazed by the latency improvements by your client application. It would respond faster to you cosmos Db requests.  NB In TCP mode apart from 443 and 10255 mentioned in Gateway more, we also need to ensure  port range between 10000 and 20000 is open in your firewall configuration,  because Azure Cosmos DB uses dynamic TCP ports.
    • Connection Protocol – HTTPS: Since client application and cosmosDb are in same network limits, you could see that HTTPS option is also a reliable, secure and faster access channel for you, but not highly performing as TCP.

    A simplified diagram below :

    image

    Sample Code:

     string cosmosDbEndpoint = new Uri("https://mycosmosDbinstance.documents.net");
     string authKey ="cosmosDb-apiKey";
     DocumentClient client = new DocumentClient(cosmosDbEndpoint, authKey,
     new ConnectionPolicy
     {
        ConnectionMode = ConnectionMode.Direct,
        ConnectionProtocol = Protocol.Tcp
     });
     

    Refer more :

    You can find the completed sample here: AzureContrib/CosmosDB-DotNet-Quickstart-With-ConnectionPolicy

    Introduction to Kubernetes

    April 22, 2018 Cloud Computing, Cloud Native Computing Foundation, Computing, Emerging Technologies, Google Cloud, IaaS, OpenSource, PaaS, Platforms No comments

    What is Kubernetes?

    Kubernetes (a.k.a K8s) is an open-source system for automating deployment, scaling and management of containerized applications that was originally designed by Google and now maintained by the Cloud Native Computing Foundation.

    What Kubernetes can do?
    Kubernetes has a number of features in cloud computing world, it can be thought as a :

    • A container platform
    • A microservices platform
    • A portable cloud platform and a lot more

    Kubernetes defines a set of building blocks (“primitives”) which collectively provide mechanisms for deploying, maintaining, and scaling applications. The components which make up Kubernetes are designed to be loosely coupled and extensible so that it can meet a wide variety of different workloads. The extensibility is provided in large part by the Kubernetes API, which is used by internal components as well as extensions and containers running on Kubernetes.

    If you are interested  to know more, learn more about Kubernates  through Official tutorials:

    Some useful online training is:

    Kubernetes vs Service Fabric

    April 13, 2018 Application Virtualization, Azure, Emerging Technologies, Kubernates, Orchestrator, OS Virtualization, PaaS, Service Fabric, Virtual Machines, Virtualization No comments

    What is the difference between Kubernates and Service Fabric?

    It is a common question today among most of the business stakeholders, infrastructure specialists, and information technology architects.

     

     

     

     

     

     

     

     

     

    To answer in simpler words, quoting from this Reddit log :

    • Kubernetes manage/orchestrate containers and applications within. 
    • ServiceFabric is a framework for microservices based on one of three models; stateful, stateless, actor. Service Fabric provides a framework for creating micro services, runtime for managing distributed instances, and also provides the ‘fabric’ that holds everything together.

    A detailed comparison quoting from an MSDN blog  from here:

    Azure Container Service: If you are looking to deploy your application in Linux environment and are comfortable with an orchestrator such as Swarm, Kubernetes or DC/OS, use ACS. A typical 3 tier application (such as a web front end, a caching layer, a API layer and a database layer) can be easily container-ized with 1 single dockerfile (or docker-compose file). It can be continuously decomposed into smaller services gradually. This approach provides an immediate benefit of portability of such an application. Containers is Open technology and there is great community support around containers.

    Azure Service Fabric: If an application must have its state saved locally, then use Service Fabric. It is also a good choice if you are looking to deploy the application in Windows server ecosystem(Linux support is in the works as well!). Refer to common workloads on Service Fabric for more discussion on applications that can benefit from Service Fabric. Biggest benefit is that Service Fabric applications can run on-premise, on Azure or even in other cloud platforms also.

    What’s Azure Container Service (ACS/AKS)

    April 12, 2018 Application Virtualization, Azure, Azure Container Service, Cloud Computing, Cloud Services, Computing, Containers, Docker, Emerging Technologies, IaaS, Kubernates, Microsoft, OpenSource, Orchestrator, OS Virtualization, PaaS, Virtual Machines, Virtualization, Windows Azure Development No comments

    I will start with history: Sometime around 2016, Microsoft launched an IaaS service called Azure Container Service a.k.an ACS serves as a bridge between Azure Ecosystem and existing container ecosystem being used widely by the developer community around the world.

    kubernates_azureIt helps as a gateway for infrastructure engineers and developers to manage underlying infrastructure such as Virtual Machines, Storage, Network Load Balancing services individually than the application itself.  The application developer doesn’t have to worry about planet-scale of the application, instead, a container orchestrator can manage the scale up and scale down of your application environment based on peaks and downs of your application usage.

    It offers an option to select from 3 major container orchestrators available today such as DC/OS, Swarm, Docker, and Kubernates.   ACS along with your choice of container orchestrators works efficiently with different container ecosystems to enable the promise of application virtualization.

    To make it simpler, ACS is your Super Glue to gel your Azure infrastructure and your container orchestrator together. Means you will be able to make your fully managed container cluster in a matter of minutes with Azure.

    ACS is for making your microservices dream come true, by providing individual services scale according to the demand and automatically reduce the scale, if usage is low. You don’t have to worry, ACS and your container orchestrator will take care of you.

    If you are a beginner to container-based infrastructure for your applications, you don’t have to take the pain at all of setting up Kubernates on your own, instead, ACS will simplify your implementation with a couple of easier click thru’s and your container infrastructure is ready to be fully managed by you. As simple as that.

    What is Azure Container Kubernates Service (AKS) then?

    As I am writing today, Microsoft has a new fully managed PaaS service called as Azure Container Service (AKS) or Managed Kubernates, meaning that Kubernates would be your default fully managed container orchestrator, if you choose Azure Container Service. But you would be able to deploy other open-source container orchestrators if you prefer to choose to have your own unmanaged Kubernates, Docker or DC/OS and then add your specific management and monitoring tools.

    This service is currently available in PUBLIC PREVIEW, you can get started from here

    Means though it is a fully managed service, you still have the option to manage it your own using your preferred set of tools and orchestrators.

    Charging Model

    Whether you manage your AKS service with your own set of tools and orchestrator or you use Fully Managed Kubernates, you only need to pay for resources you consume. No need to worry about per-cluster charges like other providers.

    Useful References: