Migration for generation users began in October 2017 and took approximately two months. A process monitors that RU usage scaling allocated RUs up or down as needed.
“Using the change feed chip library gave control in ensuring all of SLAs were fulfilled, & rdquo; explains Kaduk to us. “For example, with Azure Works, a function can fail or spin-and-wait while it retries. We could ’t afford to spin-and-wait, so we used the change feed chip library to implement a queue that retries periodically and, if still unsuccessful after a day or two, sends the request to a ‘dead letter collection’ for inspection. We also implemented observation –such as fast requests are processed, which nodes are processing them, and projected work for every partition.
Developers used the Command and Query Responsibility Segregation (CQRS) pattern along with the event sourcing pattern to execute separate write and read paths, interfaces, and data units, each tailored for their applicable tasks. “After CQRS is used using the Event Sourcing routine, the shop of occasions is the compose model, and is still the official source of data capturing what has happened or changed, what was the intention, and who had been the originator,” explains Kaduk. “All this is saved on one JSON record for every changed domain –user, person, and category. Views that are optimized for querying are provided by the model that is read and are stored in a second, smaller JSON documents. This is enabled by the Azure Cosmos DB file format and the capacity to store different types of files with different data structures. ” Find more information on using Event Sourcing with CQRS.

Change feed also provided a foundation for implementing background post-processing, such as trades that span the data of more than one user. The case of John from sending messages to him preventing Sally is a fantastic example. The machine takes the command from user John to obstruct user Sally, upon which the request is supported and dispatched to the suitable handler, which stores the function history and updates the query able data for user John. A postprocessor accountable for cross-partition trades monitors the feed, copying the information that John blocked Sally into the information for Sally (which likely resides in a different partition) as a reverse block. This information can be used for specifying the relationship between peers. (More info on this pattern can be seen in the article, “Life beyond Distributed Transactions: an Apostate’s Opinion.

Custom shift feed processing

Cross-partition trades and integration with other providers

To facilitate the migration of consumer information from SQL Server to Azure Cosmos DB, developers wrote a service that iterated over all the user information in the old PCS support to:

Skype found that the best match the globally distributed NoSQL database service from Microsoft, in Azure Cosmos DB. It gave Skype everything necessary for its People Core Service (PCS), such as turnkey global distribution and flexible scaling of throughput and storage, making it an ideal foundation for distributed programs like Skype which require extremely low latency at international scale.

The alternative

Putting data closer to users

  • Query the information in SQL Server and change it in the new data models for Azure Cosmos DB.
  • Add the data into Azure Cosmos DB and mark the consumer ’s address book as mastered from the new database.

To meet the solution’s audit history requirements, developers implemented an occasion sourcing with catch state pattern. Rather than storing only the present condition of the information in a domainname, this pattern utilizes an append-only store to record the complete collection of actions taken on the data (the “occasion sourcing” portion of this pattern), together with the mutated state (i.e. the “catch state”-RRB-. The append-only shop functions as the method of record and may be used to materialize domain items. It also provides consistency and maintains background that may enable compensating actions and audit trails.
In building the new PCS service, Skype developers executed a micro-services, event-driven architecture according to change feed service at Azure Cosmos DB. Change feed works by “listening” to a Azure Cosmos DB container for any changes and outputting a set of files which were altered, in the order in. The changes are persisted, may be processed incrementally and asynchronously, and the output can be distributed across one or more customers for parallel processing. (Change Feed in Azure Cosmos DB is allowed by default for all accounts, and it doesn’t incur any extra expenses. It’s possible to use provisioned RU/s to see in the feed, exactly like any other operation in Azure Cosmos DB.)

To make the entire process seamless to users, developers also implemented a proxy service which assessed the migration position in the search table to get a user and routed requests to the proper data store, old or new. After all users were migrated, the search table the older PCS service, along with the temporary proxy service were removed from production.
&ldquo an event-driven structure uses Kafka, Event Hub, or some other event supply,” clarifies Kaduk. “But with Azure Cosmos DB, change feed supplied a occasion origin that simplified our structure. ”

Migration of user information

Migration for production flowchart
Azure Cosmos DB flowchart
However, it later turned out that there was in each of these 3 geographies a existence sufficient to meet with all SLAs.

  • Consistency level: In establishing geo-replication, the group chose session consistency from one of the five consistency degrees supported by Azure Cosmos DB.
  • Partitioning: Skype picked UserID since the partition crucial , so ensuring that data for every user would reside on the exact same physical partition. A physical partition size of 20GB was used rather than the default option 10GB size because the larger number enabled more effective allocation and utilization of petition units per second (RU/s)–a measure of pre-allocated, guaranteed database throughput. (With Azure Cosmos DB, every collection must have a partition key, which functions as a logical partition for the data and provides Azure Cosmos DB using a natural border for transparently distributing it internally, across physical walls )
  • Event-driven structure based on Azure Cosmos DB alter feed

    Note: Opinions in italics/parenthesis are the writer 's.
    Some early selections made by the team

    Separate read and write paths and data models for optimum performance

    This is a three-part blog article show about how organizations are using Azure Cosmos DB to meet real world requirements, and the difference it’therefore creating to them. In part 1, we researched the challenges Skype faced that led them to take action. In this informative article (part 2 of 3)we examine how Skype employed Azure Cosmos DB to update its own backend infrastructure.
    Instead of using Azure Works to handle change feed processing, the development team chose to implement its own change feed processing using the Azure Cosmos DB alter feed processor library–exactly the exact same code used internally by Azure Functions. This gave programmers more control over switch feed processing, including the capability to implement retrying over observation , dead-letter event support, and queues.

    Continue on to part 3, that covers the outcomes leading to Skype’s execution of Azure Cosmos DB.

    Initial design decisions

    Programmers used change feed conversation, and to encourage integration with other providers, such as telling, graph search. By all running switch feed processors, one of which is responsible for publishing a notification to outside event customers, such as Azure Event Hub the event is received on background.