tag:blogger.com,1999:blog-13040666569936954432024-03-16T01:09:35.702+00:00Paul Done's Technical BlogA blog about enterprise software development and deploymentPaul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.comBlogger60125tag:blogger.com,1999:blog-1304066656993695443.post-29745771918771659712023-09-17T17:46:00.003+01:002023-09-17T17:55:29.950+01:00New Paper Version Of Practical MongoDB Aggregations Book Now Available<span id="docs-internal-guid-2e1eb2b7-7fff-5709-d3b0-41e49f6d414a"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">Just over 2 years ago, I <a href="https://pauldone.blogspot.com/2021/05/mongodb-aggregations-book.html">self-published</a> the <a href="https://www.practical-mongodb-aggregations.com/">Practical MongoDB Aggregations eBoo</a>k, which is also referenced by parts of the <a href="https://www.mongodb.com/docs/manual/">MongoDB Manual</a>. </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">Now, there is a MongoDB Inc. officially endorsed paper and electronic version of the book, published by <a href="https://www.packtpub.com/">Packt</a>. The Packt version of the book includes extra information on some topics and two additional example chapters.</span></p><br /><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiay037prx2t1jn4Vnh7NGtURaVs-lvvo7WYjwgC8WOGCA1RS6-nR9h8v3Az0WEsRqmsPptJfHc_Sv1_na4l3AyDccBphbydzT9NGkJvFSqlEvyVxOcesIJpsuZvgV2FiJVIz_V_AHLpfMbzFUkxPUb-KCYd1HkUuWaxW5ma8pQq5Rdw26G9uW52hWhyMw/s633/packt_book_front_cover.png" style="margin-left: 1em; margin-right: 1em;"><img alt="Practical MongoDB Aggregations book front cover" border="0" data-original-height="633" data-original-width="520" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiay037prx2t1jn4Vnh7NGtURaVs-lvvo7WYjwgC8WOGCA1RS6-nR9h8v3Az0WEsRqmsPptJfHc_Sv1_na4l3AyDccBphbydzT9NGkJvFSqlEvyVxOcesIJpsuZvgV2FiJVIz_V_AHLpfMbzFUkxPUb-KCYd1HkUuWaxW5ma8pQq5Rdw26G9uW52hWhyMw/w329-h400/packt_book_front_cover.png" width="329" /></a></div><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><br /></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">You can purchase the book from the </span><a href="https://www.packtpub.com/product/practical-mongodb-aggregations/9781835080641" style="text-decoration-line: none;"><span face="Arial, sans-serif" style="color: #1155cc; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space-collapse: preserve;">Packt</span></a><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;"> website, </span><a href="https://www.amazon.com/Practical-MongoDB-Aggregations-developing-aggregation-ebook/dp/B0CGVKYGPT" style="text-decoration-line: none;"><span face="Arial, sans-serif" style="color: #1155cc; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space-collapse: preserve;">Amazon</span></a><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">, or other book retailers.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">This Practical MongoDB Aggregations book helps you unlock the full potential of the MongoDB aggregation framework, including the latest features of MongoDB 7.0. It arms you with practical, easy-to-digest principles and approaches for increasing your effectiveness in developing aggregation pipelines, supported by examples for building pipelines </span><span face="Arial, sans-serif" style="font-size: 11pt; white-space-collapse: preserve;">to solve complex data manipulation and analytical tasks.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">This book is tailored to developers, architects, data analysts, data engineers, and data scientists with some familiarity with the aggregation framework (it’s not for aggregation beginners). It starts by explaining the framework’s architecture and then shows you how to build pipelines optimized for productivity and scale. Given the critical role arrays play in MongoDB’s document model, the book delves into best practices for optimally manipulating arrays. The latter part of the book equips you with examples to solve common data processing challenges so you can apply the lessons you’ve learned to practical situations.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;">What You Will Learn</span></p><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-wrap: wrap; vertical-align: baseline;">Develop dynamic aggregation pipelines tailored to changing business requirements</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-wrap: wrap; vertical-align: baseline;">Eliminate the performance penalties of processing data externally by filtering, grouping, and calculating aggregated values directly within the database</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-wrap: wrap; vertical-align: baseline;">Master essential techniques to optimize aggregation pipelines for rapid data processing</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-wrap: wrap; vertical-align: baseline;">Achieve optimal efficiency for applying aggregations to vast datasets with effective sharding strategies</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-wrap: wrap; vertical-align: baseline;">Employ MongoDB expressions to transform data and arrays for deeper insights</span></p></li><li aria-level="1" dir="ltr" style="font-family: Arial, sans-serif; font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; text-wrap: wrap; vertical-align: baseline;">Secure your data access and distribution with the help of aggregation pipelines</span></p></li></ul><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span face="Arial, sans-serif" style="font-size: 11pt; font-variant-alternates: normal; font-variant-east-asian: normal; font-variant-numeric: normal; font-variant-position: normal; vertical-align: baseline; white-space-collapse: preserve;"> I hope you enjoy the book!</span></p></span>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-81625609130350133972021-12-06T11:54:00.025+00:002023-04-20T13:55:56.788+01:00Achieving At Least An Order Of Magnitude Aggregation Performance Improvement By Scaling & Parallelism<p><span style="font-family: Arial; font-size: 20pt; white-space: pre-wrap;">Introduction</span></p><span id="docs-internal-guid-519bcd6d-7fff-3c89-aa8d-d5c40c13f377"><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">When I started my MongoDB Inc career 8 years ago, the 'bootcamp' project topic I chose for self-learning was to investigate how to speed up aggregations via parallelism. Specifically, I investigated the benefits of splitting an aggregation into parts, each running against a subset of the data concurrently. At the time, my study yielded a positive outcome by reducing the response time of a "full collection scan" style aggregation (see my original </span><a href="https://pauldone.blogspot.com/2014/03/mongoparallelaggregation.html" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">unsharded</span></a><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> and </span><a href="https://pauldone.blogspot.com/2014/04/mongoparallelaggsharded.html" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">sharded</span></a><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> results and write-ups). However, in hindsight, running everything inside a single laptop dulled the impact. In reality, I was probably hitting host machine resource contention (e.g. CPU, RAM, Storage IOPS limits) in a way that wouldn't occur in a real distributed environment.</span><span style="font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I thought I’d take the opportunity to revisit this topic:</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">“</span><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Can I improve the performance of a 'full-table-scan' type of aggregation workload by splitting the aggregation into parallel jobs, each operating on a subset of the data?</span><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">”</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">This time, I chose to test against a database containing all the movies ever catalogued to calculate the average movie rating across all movies. </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">I decided to use a remote MongoDB cluster deployment for the test environment, with separate host machines for each replica/shard. Just because it is so easy to rapidly create a production-like environment, I used </span><a href="https://www.mongodb.com/atlas/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">MongoDB Atlas</span></a><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> to provision and host the database cluster. I’d expect to see similar results if I was to run the tests on equivalent hardware for any self-managed version of MongoDB.</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The executed tests analyse how the aggregation workload completion time changes when adding more hardware (e.g. CPUs), more shards plus more parallelisation of the aggregation's pipeline, in various combinations.</span></p><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="color: black; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline;">Data</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">I created a collection of 100 million ‘movie’ documents by duplicating records from the smaller </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">sample_mflix</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> database sourced from the Atlas </span><a href="https://docs.atlas.mongodb.com/sample-data/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">sample data set</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">. Approximately ⅓ of the documents in the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">movies</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> collection have a field called </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> which provides an aggregated movies rating across many reviews collated by the </span><a href="https://en.wikipedia.org/wiki/Metacritic" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">Metacritic website</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">.</span></p><br /><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Below is an example of a movie document, from the collection, for one of my favourite films, </span><a href="https://en.wikipedia.org/wiki/Drive_(2011_film)" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">Drive</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhs2oImq0rSkO6nIPRUMQC4zU-s0JZUuAONvu3krT-VajgAr068gGLkrNdk2P0QuvBX_9W6vjVuT8kmkdiHGa0pMe0yOYdl0F0wPeYdtqA5RmUQ8Qd1WmALji3__ofF8vtPc9wmBOe745xJntNuGZiHl7YYUiKGErUEFebglzCiIoIr-HBjk3zGV-_3=s736" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="588" data-original-width="736" height="512" src="https://blogger.googleusercontent.com/img/a/AVvXsEhs2oImq0rSkO6nIPRUMQC4zU-s0JZUuAONvu3krT-VajgAr068gGLkrNdk2P0QuvBX_9W6vjVuT8kmkdiHGa0pMe0yOYdl0F0wPeYdtqA5RmUQ8Qd1WmALji3__ofF8vtPc9wmBOe745xJntNuGZiHl7YYUiKGErUEFebglzCiIoIr-HBjk3zGV-_3=w640-h512" width="640" /></a></div></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="color: black; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline;">Aggregation</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">I wanted to compute the average </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> score across every movie in the database collection. Using the MongoDB Shell (</span><a href="https://docs.mongodb.com/mongodb-shell/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">mongosh</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">), I am able to execute the following aggregation pipeline to calculate the average rating across all movies:</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">use sample_mflix;</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">pipeline = [</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> {</span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$group"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: {</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"_id"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">""</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"average_rating"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: {</span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$avg"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$metacritic"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">},</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> }},</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">];</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">db.movies.aggregate(pipeline);</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">[{_id: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">''</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">, average_rating: </span><span style="color: #bf9000; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">59.387109125717934</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">}]</span></p></span></span></div><div><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><h1 dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt; white-space: pre-wrap;"><span style="color: black; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline;">Adding Parallelism</span></h1><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">I then wrote a small Python test utility called </span><a href="https://github.com/pkdone/mongo-parallel-agg" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">Mongo Parallel Agg</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> to achieve parallelism and test the outcome quickly. This utility analyses the movie data set, divides the data set into sections (e.g. 8 parts), and then spawns multiple sub-processes. Each sub-process runs in parallel, targeting one of the subsections of data.</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">MongoDB has a handy tool for working out the approximate “natural split points” for a given data set, which is the </span><a href="https://docs.mongodb.com/manual/reference/operator/aggregation/bucketAuto/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">$bucketAuto</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> aggregation operator. The </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> app uses </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$bucketAuto</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> to perform an operation similar to the following to understand the shape of the movie data set and identify its subsections (in this case asking for 8 approximately balanced subsections of the titles of movies):</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">pipeline = [</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> {</span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$bucketAuto"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: {</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> <span style="color: #38761d; font-size: 14.6667px;">"$group"</span>: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$title"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"buckets"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #bf9000; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">8</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> }},</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> {</span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$group"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: {</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"_id"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">""</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"splitPoints"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: {</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$push"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$_id.min"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> },</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> }},</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> {</span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"$unset"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: [</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"_id"</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> ]},</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">];</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">db.movies.aggregate(pipeline);</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">[{</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> splitPoints: [</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'!Women Art Revolution'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'Boycott'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'Exotica'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'Ishqiya'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'Mr Perfect'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'Salesman'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">'The Counterfeiters'</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">,</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">"The Strange History of Don't Ask, Don't Tell"</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> ]</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">}]</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">As you can see from these results, it's important to analyse the spread of values in a collection to determine its natural divisions. A naïve approach would be to manually divide all the movie titles by their initial letter in the English alphabet (e.g. A-Z). With 26 letters, if you need 8 subsections, you might naïvely split at every 3</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-size: 0.6em; vertical-align: super;">rd</span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> or 4</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-size: 0.6em; vertical-align: super;">th</span></span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> letter (</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">26 ÷ 8 = 3.25</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">). You would come up with subsections such as "A-C", "S-U", etc. However, as you can tell from the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$bucketAuto</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> output above, each subsection would not be evenly balanced for the number of documents it covers. Many more movies begin with the letter "T" than other letters due to titles like "</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">The …</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">". Using uneven subsections of documents for parallel aggregations results in some sub-processes taking far longer than others, prolonging the overall response time of the entire aggregation workload.</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">The other main 'trick' the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> app pulls is to use the identified subsections information to produce multiple aggregation pipelines, one for each sub-process, targeting just a subset of the collection. The aggregation excerpt below provides an example of the pipeline dynamically generated by </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> to target a subsection of data to help with calculating the average rating:</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">pipeline = [</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$match"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"title"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$gte"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"Boycott"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">, </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$lt"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"Exotica"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">}}</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> },</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$group"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"_id"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">""</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">,</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"total"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$sum"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$metacritic"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">},</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"count"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$sum"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$cond"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"if"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: {</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$eq"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: [</span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"$metacritic"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">, </span><span style="background-color: transparent; color: #bf9000; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">null</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">]},</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"then"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: </span><span style="background-color: transparent; color: #bf9000; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">0</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">,</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span><span style="background-color: transparent; color: #38761d; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">"else"</span><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">: </span><span style="background-color: transparent; color: #bf9000; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">1</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> }}},</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> }},</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">];</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;"> </span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt;"><span style="background-color: transparent; color: black; font-family: 'Roboto Mono',monospace; font-size: 11pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline; white-space: pre;">db.movies.aggregate(pipeline);</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-left: 36pt; margin-top: 0pt; white-space: pre-wrap;"><span id="docs-internal-guid-8b54df4b-7fff-2782-055c-4924b0a4e904"><br /><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">[{ </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">_id</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">''</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">, </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">total</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #bf9000; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">732084631</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">, </span><span style="color: #38761d; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">count</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">: </span><span style="color: #bf9000; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">12497379</span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">}]</span></span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">As you can see, the pipeline uses a </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$match</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> stage to target a subsection of the data for analysis. The other main change is that the pipeline no longer uses the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$avg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> operator to calculate an average. Instead, the pipeline includes two computed fields, one for the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">total</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> and one for the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">count</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">, which each use the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$sum</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> operator. This change is required because, mathematically, calculating an average of averages will yield an invalid result. In this solution, the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> Python code performs the "last-mile" average computation by summing the totals produced by each sub-process. It then divides this grand total by the sum of all the counts calculated by each sub-process to determine the average. Also, you'll notice that the pipeline must handle ignoring documents for the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">count</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> field if the field (</span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">) doesn't exist in a document (the previously used </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$avg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> operator automatically did this and so there was no need for a check).</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">To optimise each sub-process pipeline, the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> app first ensures a compound index exists for </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">title</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> & </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">. This enables the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$match</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> part of the pipeline to target the index. It also enables the aggregation </span><a href="https://docs.mongodb.com/manual/core/query-optimization/#covered-query" style="text-decoration-line: none;"><span style="color: #1155cc; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">to be covered for increased efficiency</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> because the only other field analysed (</span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">) belongs to the same index.</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 14.6667px; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="color: #0e101a; font-family: Arial;">Purely for convenience, the </span></span><span style="color: #0e101a; font-family: Roboto Mono, monospace;"><span style="font-size: 14.6667px; white-space: pre-wrap;">mongo-parallel-agg</span></span><span style="color: #0e101a; font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"> demo app performs these two actions every time it executes before running the main aggregation workload against the collection (i.e. </span><span style="color: #0e101a; font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">performing </span><span style="color: #0e101a; font-family: Roboto Mono, monospace;"><span style="font-size: 14.6667px; white-space: pre-wrap;">$bucketAuto</span></span><span style="color: #0e101a; font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;"> analysis and </span><span style="color: #0e101a; font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">ensuring an index exists</span><span style="color: #0e101a; font-family: Arial; font-size: 14.6667px; white-space: pre-wrap;">). In a real production system, both these actions would be performed once or infrequently and not every time the aggregation workload runs. For this reason, the app doesn't start its aggregation execution timer until after it completes these two actions.</span></p><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="font-family: Arial; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline; white-space: pre-wrap;">Results</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: #0e101a; font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">The following table shows the execution times, in seconds, for an aggregation computing the average movie rating when run against different host environment topologies with varying levels of parallelism:
</span></span></p></span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></span></div><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg74rlYSQHgYrMXNVekSlhu6qXaZ-WPn-f5uNBDP9gizuuvk2rfuE4dFBN6kbngrnrYlRw_Wf4-URDTo1Y21NQj_VfrhXDNGUcT5J5H9C0Sd16__v6ewB7uV093yDUg0MMH5Q6iIZPnyd5KibJJF1jejhTil_XETnEzS4prOdLne0iUNKGqbMHnF47h=s1373" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="370" data-original-width="1373" height="172" src="https://blogger.googleusercontent.com/img/a/AVvXsEg74rlYSQHgYrMXNVekSlhu6qXaZ-WPn-f5uNBDP9gizuuvk2rfuE4dFBN6kbngrnrYlRw_Wf4-URDTo1Y21NQj_VfrhXDNGUcT5J5H9C0Sd16__v6ewB7uV093yDUg0MMH5Q6iIZPnyd5KibJJF1jejhTil_XETnEzS4prOdLne0iUNKGqbMHnF47h=w640-h172" width="640" /></a></div><div style="text-align: center;"><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span id="docs-internal-guid-ff66060b-7fff-de4c-df47-f3d2fe334458"><span style="color: black; font-size: 10pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Per replica host specification: Intel Xeon processor with a maximum speed of 3.1 GHz, 512 TB storage with 3000 non-provisioned IOPS</span></span></span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></span></div><div><span style="font-family: Arial; font-size: 20pt; white-space: pre-wrap;">Observations</span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Scalability</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The results show the solution is able to achieve scalability in multiple individual dimensions:</span></p></li><ul style="margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px;"><li aria-level="2" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: circle; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Vertical Scaling</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The execution time is reduced by adding more vCPUs and making no other changes. Note, not evident in the displayed results is the effect of adding more RAM. In some cases, the addition of RAM does actually have an impact. See the later bullet-point titled "</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">Increasing RAM For Analytics May Help</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">" for more detail.</span></p></li><li aria-level="2" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: circle; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Horizontal Scaling</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The execution time is reduced by adding more shards and making no other changes. </span></p></li><li aria-level="2" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: circle; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Parallel-processing Scaling</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The execution time is reduced by splitting the aggregation into parallel sub-processes, each acting on a subset of records and making no other changes. </span></p></li></ul><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Combined Scaling</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. Overall, by combining the benefits of all three scaling dimensions, the solution manifests two orders of magnitude of reduction in execution time - from 908 seconds (over 15 minutes) down to 9 seconds.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Non-Linear Scaling</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The scaling exhibited isn't linear, but I wouldn't expect this because </span><a href="https://en.wikipedia.org/wiki/MapReduce" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">map-reduce style workloads</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">, such as calculating averages, will never scale 100% linearly. Such workloads must serialise parts of the computation to accumulate partial results together in one place and sum them together.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Unexpected Degree Of Speed-Up From 1 to 2 Sub-Processes</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. In each situation where there is a transition from a single aggregation process to two parallel sub-processes, there is at least a 6x speed-up. This difference is far more significant than the typical "best-case" linear (2x) speed-up that could have realistically been hoped for. This puzzled me for quite a while, but I eventually realised why this occurs. See the later section titled "</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">The Slow Single-Threaded Result Conundrum</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">" for the reason why.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Optimal Sub-Process to CPU Mapping</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The optimal number of sub-processes to execute for this particular workload appears to be roughly 1 to 2 times the total vCPUs available. For example, in the first result row in the table (</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">M40, 1 shard</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">), for a total of 4 vCPUs, 8 sub-processes yields the quickest result. Another example is visible in the penultimate table row (</span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">M60, 2 shards</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">). Here, the solution yields the quickest result for a total of 32 vCPUs when running either 32 or 64 sub-processes. Overall, we can infer that at some point, between 1 and 2 sub-processes per vCPU, the benefits of multiprocessing are outweighed by the overhead of facilitating so many processes.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Increasing RAM For Analytics Can Help</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. The table's results for the M60 two-shards and four-shards configurations, for the single sub-process tests, do not capture the full picture. In both cases, the response time significantly decreases (not shown in the table) when running the test configuration for the second time with the same aggregation pipeline. For the </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">M60 two-shard single-process</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"> test (with a combined RAM total of 128GB), the response time was reduced from 557 seconds to 438 seconds. For the </span><span style="font-size: 11pt; font-style: italic; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">M60 four-shard single-process test </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">(with a combined RAM total of 256GB), the response time was reduced from 340 seconds to 102 seconds. This latency drop occurs because, for these configurations, the size of RAM available across the shards is approaching the size of the full collection. Consequently, a significant portion of the data is fetched directly from RAM rather than from disk, because the data is already present in memory following the first test run. The remaining four smaller test configurations exhibited no noticeable difference in execution times between first and second runs.</span></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline; white-space: pre-wrap;">Increasing Storage IOPS Was Not Tested</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;">. I employed the tests to analyse the effects of scaling parallel processing factors such as the number of CPU cores, shards and sub-processes. I expect the results to be even better when increasing the storage IOPS allocated to the host machines. This increased storage bandwidth should enable an aggregation to pull the scanned collection data from storage quicker. However, this aspect wasn't under consideration here and wasn’t tested, and so I've left it as an exercise for the reader.</span></p></li></ul></span></span></div></span><span><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><h1 dir="ltr" style="line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt;"><span style="color: black; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline;">The Slow Single-Threaded Result Conundrum</span></h1><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">So why is there a jump down of at least 6x lower latency when going from a single process to two parallel sub-processes to calculate the average movie rating for all movies?</span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Having run all the tests, I investigated deeper and realised why this phenomenon occurred. I'd initially optimised the </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> app when splitting an aggregation into multiple pipelines, to each performing a </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$match</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> for a subset of data. I’d realised that each pipeline was no longer performing a "full collection scan" and would benefit from an index on the </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">title</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> field being used by the </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$match</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> filter. Additionally, I’d realised that the only other field used by these "split aggregations" was the </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> field from which the average is calculated. Therefore, rather than using a simple index on </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">title</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">, I’d employed a compound index on </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">title</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> & </span><span style="color: black; font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="color: black; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> to cover the query part of the aggregation. Consequently, each aggregation sub-process was able to fully leverage an index and didn’t need to scan each full document (so full documents were not pulled from disk). Also, the index is far smaller than the full collection's size, and therefore, the database can rapidly serve the index’s data from RAM. </span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">During the tests, I’d not considered whether I could apply some of these same benefits when just running the entire aggregation pipeline as a single process. It was only later that I realised I could also optimise the original “full-collection” aggregation with a small “hack”, inspired by how I’d originally optimised the divided pipelines for sub-processing. Upon testing this hack for just the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">M60 two-shard</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> configuration (single process), I obtained the following results:</span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; text-align: center;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;">557 seconds </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">(mostly on disk) </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;"> → 448 seconds </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">(significantly in RAM) </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 700; vertical-align: baseline;"> → 92 seconds </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">(with the pipeline hack)</span></span></p><br /><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">My hack involved refactoring the aggregation pipeline that the application generates for the single-threaded processing option, as shown below:</span></p></span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgzNePS9DJ4sWTIEL7wqjopspFyCJAGb-eSwkuxIHIZTKAZo3jeE7FZyy4yFq280tNJYgCuZQugRuGWg5iuMwdI838zrlxZQ-1KHsFEUUAvO_Vf2Yv4-7N-2KGV17yRsV6EbiTo5a7-HXH1ppMbzObyRr0sOvIQpf4Kr1xwzabg3IL6ATJGEvA00aJb=s1655" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="398" data-original-width="1655" height="154" src="https://blogger.googleusercontent.com/img/a/AVvXsEgzNePS9DJ4sWTIEL7wqjopspFyCJAGb-eSwkuxIHIZTKAZo3jeE7FZyy4yFq280tNJYgCuZQugRuGWg5iuMwdI838zrlxZQ-1KHsFEUUAvO_Vf2Yv4-7N-2KGV17yRsV6EbiTo5a7-HXH1ppMbzObyRr0sOvIQpf4Kr1xwzabg3IL6ATJGEvA00aJb=w640-h154" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div></span></div><div><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><span style="font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 14.6667px;">Using the “filter documents greater than MinKey” hack, I saw that the aggregation performs an index scan rather than a full-collection scan. The runtime invariably pulls this [far smaller] index from RAM rather than disk. The aggregation is covered, locating all the required </span><span style="font-family: "Roboto Mono", monospace; font-size: 14.6667px;">title</span><span style="font-size: 14.6667px;"> and </span><span style="font-family: "Roboto Mono", monospace; font-size: 14.6667px;">metacritic</span><span style="font-size: 14.6667px;"> values from the index with no need to examine each underlying document in the collection.</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><br /></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Interestingly, before including my hack, I first tried defining just a simple index on </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">metacritic</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> and using a </span><a href="https://docs.mongodb.com/manual/reference/operator/meta/hint/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">hint</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> to force the “full aggregation” to use this index for a covered query. However, this didn’t yield any performance benefits and it was even slower.</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Unfortunately, I’d run out of time to re-test all the deployment scenarios with this improved single-process pipeline. Therefore, the results table reflects the original results before this final improvement. I have since folded the code refactoring into the <a href="https://github.com/pkdone/mongo-parallel-agg/">Github codebase</a> for the </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">mongo-parallel-agg</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> project so that others can leverage this optimisation in the future.</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">Also, upon reflection, instead of using the “match documents greater than MinKey” filter in a new </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$match</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> stage, I suspect I could probably use something like the </span><a href="https://docs.mongodb.com/manual/reference/operator/query/exists/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">$exists</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> operator instead in the </span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">new </span><span style="font-family: "Roboto Mono", monospace; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">$match</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> stage</span><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">. My guess is this would also ensure the aggregation pipeline leverages the compound index, and as a covered query, rather than performing a full collection scan. Additionally, it may also be the case that using a </span><a href="https://docs.mongodb.com/manual/core/index-sparse/" style="font-size: 11pt; text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">sparse index</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> or </span><a href="https://docs.mongodb.com/manual/core/index-partial/" style="font-size: 11pt; text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">compound index</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"> provides further improvement for all the aggregations executed, whether parallelised or not (to be determined). I will leave these two elements for the reader to test.</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></p><p dir="ltr" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><i><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 14.6667px;"><b>EDIT: 7-Dec-2021: </b></span></span><span style="color: #0e101a; font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">Prompted by a colleague, Chris Harris, I tried employing the hint mechanism again since publishing this blog post. This time, using a hint did work, and the single-threaded aggregation leveraged an index as a covered query. On the previous occasion I'd tried this, I suspect I'd introduced a typo when referencing the index from the hint. In conclusion, there appears to be a few different options for inducing an index scan and covered query to occur. For the background on why you explicitly have to force an index scan to be used, rather than a collection scan when there is no </span></span><span style="color: #0e101a; font-family: "Roboto Mono", monospace; font-size: 14.6667px; white-space: pre-wrap;">find()</span><span style="color: #0e101a; font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;">/</span></span><span style="color: #0e101a; font-family: "Roboto Mono", monospace; font-size: 14.6667px; white-space: pre-wrap;">$match</span><span style="color: #0e101a; font-family: Arial;"><span style="font-size: 14.6667px; white-space: pre-wrap;"> filter defined, review the <a href="https://jira.mongodb.org/browse/SERVER-20066">server ticket 20066</a>.</span></span></i></p><h1 dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 6pt; margin-top: 20pt; white-space: pre-wrap;"><span style="color: black; font-size: 20pt; font-variant-east-asian: normal; font-variant-numeric: normal; font-weight: 400; vertical-align: baseline;">Summary</span></h1><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">I've shown here that, for a specific aggregation, I can reduce the time taken to calculate the average rating across 100 million movies on "mid-range" hardware by two orders of magnitude. I achieved this by scaling vertically (adding more CPUs), scaling horizontally (adding more shards), and by increasing the number of processes run in parallel. Also, some pipeline tweaking and index tuning helped. </span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">These findings don't guarantee that every type of aggregation workload will scale to the same degree, or even at all when applying similar configuration changes. The impact will depend on the nature of the aggregation pipeline.</span></p><br /><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">I'd be remiss not to recommend that before trying the scaling optimisations outlined here, you should first ensure you have optimised your aggregation pipeline more generally. Scaling comes with an increased infrastructure cost which can often be avoidable. Also, the act of running more sub-processes for each aggregation could drain computation power you’d previously “allocated” to other types of workloads running against the database. My book, </span><a href="https://www.practical-mongodb-aggregations.com/" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline;">Practical MongoDB Aggregations</span></a><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;">, and specifically the following sections, outlines some of the techniques to use to optimise your aggregations without applying scaling changes:</span></p><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></p><ul style="color: #0e101a; font-family: Arial; font-size: 11pt; margin-bottom: 0px; margin-top: 0px; padding-inline-start: 48px; white-space: pre-wrap;"><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://www.practical-mongodb-aggregations.com/guides/performance.html" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Pipeline Performance Considerations</span></a></p></li><li aria-level="1" dir="ltr" style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; list-style-type: disc; vertical-align: baseline; white-space: pre;"><p dir="ltr" role="presentation" style="line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt;"><a href="https://www.practical-mongodb-aggregations.com/guides/sharding.html#performance-tips-for-sharded-aggregations" style="text-decoration-line: none;"><span style="color: #4a6ee0; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; text-decoration-line: underline; text-decoration-skip-ink: none; vertical-align: baseline; white-space: pre-wrap;">Performance Tips For Sharded Aggregations</span></a></p></li></ul><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"></span></p><div style="color: #0e101a; font-family: Arial; font-size: 11pt;"><span style="white-space: pre;"><br /></span></div><p dir="ltr" style="color: #0e101a; font-family: Arial; font-size: 11pt; line-height: 1.38; margin-bottom: 0pt; margin-top: 0pt; white-space: pre-wrap;"><span style="font-size: 14.6667px;">Lastly, is the obvious question about whether I believe it is possible for the movie average rating calculation to take <b>less than one second to complete for 100 million movies</b>? The answer is absolutely yes, with sufficiently increased CPUs, RAM, storage IOPS, shards and parallel sub-processes. However, I suspect it may be a while before I find out for sure because the required uplift in hardware is likely to be cost-prohibitive for me at least.</span></p><div style="color: #0e101a; font-family: Arial; font-size: 11pt;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></div><div style="color: #0e101a; font-family: Arial; font-size: 11pt;"><br /></div></span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><br /></span></span></div><div><span style="color: #0e101a; font-family: Arial; font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline; white-space: pre-wrap;"><span style="font-size: 11pt; font-variant-east-asian: normal; font-variant-numeric: normal; vertical-align: baseline;"><i style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px; white-space: normal;">Song for today: Carol by </i></span></span><span face="Arial, Tahoma, Helvetica, FreeSans, sans-serif" style="color: #444444;"><span style="font-size: 13px;"><i><a href="https://en.wikipedia.org/wiki/The_Peep_Tempel">The Peep Tempel</a></i></span></span></div></span>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com1tag:blogger.com,1999:blog-1304066656993695443.post-76834284397061884192021-05-15T11:04:00.002+01:002021-12-06T12:36:18.519+00:00New MongoDB Aggregations book is out<p>My book, <b>Practical MongoDB Aggregations</b>, was published this week.</p><p>The book is available electronically for free for anyone to use at: <a href="https://www.practical-mongodb-aggregations.com">https://www.practical-mongodb-aggregations.com</a></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgAcovevAgXr5U8599BeNNyW5CLOu1iBCjjN0LkpzZoGm-m58P0ZRkE0w91x6xunzHkf8tgTJo_6hlvfZAhEcY2ycszMVL6_FPG3HobhfhV3ezVDUkOsrnQYzBX3Hg426rbwaPRi0cR70/s1500/cover.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1500" data-original-width="1000" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjgAcovevAgXr5U8599BeNNyW5CLOu1iBCjjN0LkpzZoGm-m58P0ZRkE0w91x6xunzHkf8tgTJo_6hlvfZAhEcY2ycszMVL6_FPG3HobhfhV3ezVDUkOsrnQYzBX3Hg426rbwaPRi0cR70/w266-h400/cover.png" width="266" /></a></div><br /><p>This book is intended for developers, architects, data analysts, data engineers, and data scientists. It aims to improve your productivity and effectiveness when building aggregation pipelines and help you understand how to optimise their pipelines.</p><p>The book is split into two key parts:</p><p></p><ol style="text-align: left;"><li>A set of tips and principles to help you get the most out of aggregations.</li><li>A bunch of example aggregation pipelines for solving common data manipulation challenges, which you can easily copy and try for yourself.</li></ol><p></p><p>I hope readers get some good value from it!</p><p><br /></p><p><i>Song for today: Earthmover by <a href="https://en.wikipedia.org/wiki/Have_a_Nice_Life">Have a Nice Life</a></i></p><p><br /></p>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-71393737906931748552021-02-27T10:15:00.007+00:002021-02-27T20:50:34.383+00:00MongoDB Reversible Data Masking Example Pattern<h3 style="text-align: left;">Introduction</h3><p>In a <a href="https://pauldone.blogspot.com/2021/02/mongdb-data-masking.html">previous blog post</a> I explored how to apply one-way non-reversible data masking on a data-set in MongoDB. Here I will explore why, in some cases, there can be a need for reversible data masking, where, with appropriate privileges, each original record's data can be deduced from the masked version of the record. I will also explore how this can be implemented in MongoDB, using something I call the <b>idempotent masked-id generator </b>pattern.</p><p>To accompany this blog post, I have provided a GitHub project which shows the Mongo Shell commands used for implementing the reversible data masking pattern outlined here. Please keep referring to the project’s README as you read through this blog post, at:</p><p></p><ul><li><a href="https://github.com/pkdone/mongo-data-mask-reversible">https://github.com/pkdone/mongo-data-mask-reversible</a></li></ul><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">Why Reversible Data Masks?</h3><p>In some situations a department in an organisation, that masters and owns a set of data, may need to provide copies of the whole or subset of the data to a different department or even a different partner organisation. If the data-set contains sensitive data, like personally identifiable information (PII) for example, the 'data-owning' organisation will first need to perform data masking on the values of the sensitive fields of each record in the data-set. This redaction of fields will often be one-way (irreversible) preventing the other department or partner organisation from being able to reverse engineer the content of the masked data-set, to retrieve the original sensitive fields. </p><p>Now consider the example where the main data-owning organisation is collecting results of 'tests'. The results could be related to medical tests where the data-owning organisation is a hospital for example. Or the results could be related to academic tests where the data-owning organisation is a school for example. Let's assume that the main organisation needs to provide data to a partner organisation for specialist analysis, to identify individuals with concerning test results patterns. However there needs to be assurance that each individual's sensitive details or real identity is not shared with the partner organisation.</p><p>How can the partner organisation report back to the main organisation flagging individuals for concern and recommended follow-up without actually having access to those real identities? </p><p>One solution is for the redacted data-set that is provided to the partner organisation, to carry an obfuscated but reversible unique identity field as a substitute for the real identity, in addition to containing other irreversibly redacted sensitive fields. With this solution, it would not be possible for the partner organisation to reverse engineer the substituted unique identity, to a real social security number, national insurance number or national student identifier, for example. However, it would be possible for the main data-owning organisation to convert the substituted unique id back to the real identity, if subsequently required. </p><p>The diagram below outlines the relationship between the data-owning organisation, the partner organisations and the data masked data-sets shared between them. </p><p></p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQuEGQC9rSY0AoYEK3dlW6GwVbHy3vDqnKbBN9yY-Yj4WTBtsGBUmokTGKBsBEkEl_mCboh028ogAhgPX-L6QOM5JGHKv7V8f3mc0-Bh4ZX2cOfBk67GkI7mXkUS4m-h0NJiuM9hL1eMc/s701/MongoDB+Reversible+Data+Masking+Examples+-+DIAGRAMS.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="420" data-original-width="701" height="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQuEGQC9rSY0AoYEK3dlW6GwVbHy3vDqnKbBN9yY-Yj4WTBtsGBUmokTGKBsBEkEl_mCboh028ogAhgPX-L6QOM5JGHKv7V8f3mc0-Bh4ZX2cOfBk67GkI7mXkUS4m-h0NJiuM9hL1eMc/w640-h384/MongoDB+Reversible+Data+Masking+Examples+-+DIAGRAMS.png" width="640" /></a></div><p>A partner organisation can flag an individual of concern back to the main organisation, without ever being able to deduce the real life person who the substituted unique ID maps to.</p><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">How To Achieve Reversibility With MongoDB?</h3><p>To enable a substituted unique ID to be correlated back to the original real ID of a person, the main data-owning organisation needs to be able to generate the substitute unique IDs in the first place, and maintain a list of mappings between the two, as shown in the diagram below.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisRXChTTc4TrNl63M8QJL7DnWngDvFuuLbhMvIiYLz2H8_7LE43G7VOyP5q6KvoimEL-SFa2A4jkuhpa0H2Q_Pn-_xX2LS1hor00W__OTizPUv138qm8pNYxCwZlNKSLaXDs44DUVTviM/s444/do.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="423" data-original-width="444" height="381" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEisRXChTTc4TrNl63M8QJL7DnWngDvFuuLbhMvIiYLz2H8_7LE43G7VOyP5q6KvoimEL-SFa2A4jkuhpa0H2Q_Pn-_xX2LS1hor00W__OTizPUv138qm8pNYxCwZlNKSLaXDs44DUVTviM/w400-h381/do.png" width="400" /></a></div><p>The stored mappings list needs to be protected in such a way that only specific staff in the data-owning organisation, with specific approved roles, have access to it. This prevents the rest of the organisation and partner organisations from accessing the mappings to be able to reverse engineer masked identifies back to real identities.</p><p>Essentially, the overall process of masking and 'unmasking' data with MongoDB, as shown in the <a href="https://github.com/pkdone/mongo-data-mask-reversible">GitHub project</a> accompanying this blog post, is composed of three different key aggregation pipelines:</p><p></p><ol style="text-align: left;"><li><b>Generation of Unique ID Mappings</b>. A pipeline for the data-owning organisation to generate the new unique anonymised substitute IDs for each person appearing in a test result, into a new mappings collection using the <i>idempotent masked-id generator</i> pattern</li><li><b>Creation of the Reversible Masked Data-Set</b>. A pipeline for the data-owning organisation to generate a masked version of the test results, where each person's id has been replaced with the substitute ID (an anonymous but reversible ID); additionally some other fields will be filtered out (e.g. national id, last name) or obfuscated with partly randomised values (e.g data of birth).</li><li><b>Reverse Engineer Masked Data-Set Back To Real Identities</b>. An optional pipeline, if./when required, for the data-owning organisation to be able to take the potentially modified partial masked data-set back from the partner organisation, and, using the mappings collection, reverse engineer the original identities and other sensitive fields. </li></ol><p></p><p></p><p>The screenshot below captures an example of the outcome of steps 1 and 2 of the process outlined above.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLyAzuXYxwiYy8BSCMUp5-Wy84idyB2sAhIVQUG7JIsV-Lps-W4kwBZNq0x5VmTHOPq3l7fODNRGoCUnsrQbfTEhj4jZoCNhixDvKqMsqWquzsth0ltzn2Rb0DynpAi-nwNONyuCUJpx4/s1067/.datapic.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="899" data-original-width="1067" height="540" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLyAzuXYxwiYy8BSCMUp5-Wy84idyB2sAhIVQUG7JIsV-Lps-W4kwBZNq0x5VmTHOPq3l7fODNRGoCUnsrQbfTEhj4jZoCNhixDvKqMsqWquzsth0ltzn2Rb0DynpAi-nwNONyuCUJpx4/w640-h540/.datapic.png" width="640" /></a></div><br /><p>Here, each person's ID has been replaced with a reversible substitute unique ID. Additionally, the date of birth field ('dob') has been obfuscated (shown with the red underline) and some other sensitive fields have been filtered out.</p><p>I will now explore how each of the three outlined process steps is achieved in MongoDB, in the following three sub-sections.</p><p><br /></p><div style="text-align: left;"><b>1. Generation of Unique ID Mappings</b></div><p>As per the companion <a href="https://github.com/pkdone/mongo-data-mask-reversible">GitHub project</a>, the list of original ID to substitute ID mappings is stored in a MongoDB collection with very strict <a href="https://docs.mongodb.com/manual/core/authorization/">RBAC</a> controls applied. An example record in this collection might look like the one shown in the screenshot below.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBSL0my53k_4m4Q_VlrJCl1Qa4pJEUCEsTzTjNL8cVxAjbJopA7DZOguw2VIjbWoL9oBMt1GnJQdNlJlHsi9k1Y1SOyB8bwPHiYeewNSDsShengUktQt3r-bUYxTLcFJXeVBkF4CBimSk/s568/mapping.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="211" data-original-width="568" height="238" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBSL0my53k_4m4Q_VlrJCl1Qa4pJEUCEsTzTjNL8cVxAjbJopA7DZOguw2VIjbWoL9oBMt1GnJQdNlJlHsi9k1Y1SOyB8bwPHiYeewNSDsShengUktQt3r-bUYxTLcFJXeVBkF4CBimSk/w640-h238/mapping.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><p>Here the collection is called <b>masked_id_mappings</b>, where each record's field '<i>_id</i>' contains a newly generated substitute ID, based on a generated <a href="https://en.wikipedia.org/wiki/Universally_unique_identifier">universally unique identifiers</a> (UUIDs). The field '<i>original_id</i>' contains the real identifier of the person or entity in the same format it was in, in the original data-set. For convenience, two date related attributes are included in each record. The '<i>date_generated</i>' field is generally useful for tracking when the mapping was created (e.g. for reporting), and the '<i>date_expired</i>' is associated with a <a href="https://docs.mongodb.com/manual/core/index-ttl/">time-to-live</a> (TTL) index to enable the mapping to be automatically deleted by the database, after a period of time (3 years out, in this example).</p><p>The remaining field, '<i>data_purpose_id</i>' is worthy of a little more detailed discussion. Let's say the same data-set needs to be provided to multiple 3rd parties, for different purposes. It makes sense to mask each copy of the data differently, with different unique IDs for the same original IDs. This can help prevent the risk of any potential future correlation of records between unrelated parties or consumers. Essentially when a mapping record is created, in addition to providing the original ID, a data purpose 'label' must be provided. A unique substitute ID is generated for a given source identity, per data use/purpose. For one specific data consumer purpose, the same substituted unique ID will be re-used for the same reoccurring original ID, However, a different substituted unique ID will be generated and used for an original ID when the purpose and consumer requesting a masked data-set is different.</p><p>To populate the <b>masked_id_mappings</b> collection, an aggregation pipeline (called '<i>maskedIdGeneratprPipeline</i>') is run against each source collection (e.g. against both the '<i>persons</i>' collection and the '<i>tests</i>' collection). This aggregation pipeline implements he <b>idempotent masked-id generator</b> pattern. Essentially, this pattern involves taking each source collection, picking out the original unique id and then placing a record of this, with a newly generated UUID it is mapped to (plus the other metadata such as <i>data_purpose_id</i>, <i>date_generated</i>, <i>date_expired</i>), into the <i>masked_id_mappings</i> collection. The approach is idempotent in that the creation of each new mapping record is only fulfilled if a mapping doesn't already exist for the combination of the original unique id and data purpose. When further collections in the data-set are run through the same aggregation pipeline, if some records from these other collections have the same original id and data purpose as one of the records that already exists in the <i>masked_id_mappings</i> collection, a new record with a new UUID will not be inserted. This ensures that, per data purpose, the same original unique id is always mapped to the same substitute UUID, regardless of how often it appears in various collections. This <i>idempotent masked-id generator</i> process is illustrated in the diagram below. </p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiToXOgZcfXEd4CVXMFzqRE8yEbxf3WCdJ51tV9aI-7s7TwNJWqGhBa-5HeRG4HCnJEmm-RfrHEbhpood7ZbdQ9oodLG-prrb6ACcRhXBlfrURtLPsuVmhbvVnksxMAfcYwseB7szrSHUY/s951/rev.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="190" data-original-width="951" height="128" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiToXOgZcfXEd4CVXMFzqRE8yEbxf3WCdJ51tV9aI-7s7TwNJWqGhBa-5HeRG4HCnJEmm-RfrHEbhpood7ZbdQ9oodLG-prrb6ACcRhXBlfrURtLPsuVmhbvVnksxMAfcYwseB7szrSHUY/w640-h128/rev.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><div class="separator" style="clear: both; text-align: center;"><br /></div>The same aggregation pipeline is run multiple times, once against each source collection which belongs to the source data-set. Even if the source data-set is ever added to in the future, the aggregation can be re-run against the same data-sets, over and over again, without any duplicates or other negative consequences, and with only the additions being acted upon. The pipeline is so generic that it can also be run against other previously unseen collections which have completely different shapes but where each contains an original unique ID in one of its field.<div><br /><div><br /><div><div style="text-align: left;"><b>2. Creation of the Reversible Masked Data-Set</b></div><p>Once the mappings have been generated for a source data-set, it is time to actually generate a new masked set of records from the original data-set. Again, this is achieved by running an aggregation pipeline, once per different source collection in the source data-set. The diagram below illustrates how this process works. The aggregation pipeline takes the original ID fields from the source collection, then performs a lookup on the mappings collection (including the specific data purpose) to grab the previously generated substitute unique ID. The pipeline then replaces the original IDs with the substitute IDs, in the outputted data masked collection</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKEsrmjV7mU1MDeC7n4uE71lJ6trrGczwHXCBlLCPijXjyxsAzgAVK5yIG95aJAqkzGxJjckdFqLfltaTB6aNjBYs2NQjhZWDpMaX5bjFWnbNt-FqWcU4o8zkslueAEI4U3Rv3mDUzNeU/s902/phase2.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="323" data-original-width="902" height="230" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiKEsrmjV7mU1MDeC7n4uE71lJ6trrGczwHXCBlLCPijXjyxsAzgAVK5yIG95aJAqkzGxJjckdFqLfltaTB6aNjBYs2NQjhZWDpMaX5bjFWnbNt-FqWcU4o8zkslueAEI4U3Rv3mDUzNeU/w640-h230/phase2.png" width="640" /></a></div><div class="separator" style="clear: both; text-align: center;"><br /></div><p>The remaining part of the aggregation pipeline is less generic and must contain rules distinct to the source data-set it operations on. The latter part of the pipeline contains specific data masking actions to apply to specific sensitive fields in the specific data-set.</p><p>The generated masked data-set collections can then be <a href="https://docs.mongodb.com/database-tools/mongoexport/">exported</a> ready to be shipped to the consuming business unit or 3rd party organisation, who can then <a href="https://docs.mongodb.com/database-tools/mongoimport/">import</a> the masked data-set into their own local database infrastructure, ready to perform their own analysis on.</p><p><br /></p><div style="text-align: left;"><b>3. Reverse Engineer Masked Data-Set Back To Real Identities</b></div><p>In the example 'test results' scenario, the partner organisation may need to subsequently report back to the main organisation flagging individuals for concern and recommended follow-up. They can achieve this by providing the substituted identities back to the owning organisation, with the additional information outlining why the specific individuals have been flagged. The <a href="https://github.com/pkdone/mongo-data-mask-reversible">GitHub project</a> accompanying this blog post shows an example of performing this reversal, where some of the '<i>tests</i>' collection records in the masked data-set have been marked with the following new attribute by the 3rd party organisation:</p></div><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px;"><div><pre style="border-radius: 6px; box-sizing: border-box; color: #24292e; font-size: 13.6px; line-height: 1.45; margin-bottom: 0px; margin-top: 0px; overflow-wrap: normal; overflow: auto; padding: 16px; text-align: left; word-break: normal;"><span style="font-family: Roboto Mono;"><b><span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'flag'</span> : <span class="pl-s" color="var(--color-prettylights-syntax-string)" style="box-sizing: border-box;">'INTERVENTION-REQUIRED'</span></b></span></pre></div></blockquote><div><p>The GitHub project then shows how the masked and now flagged data-set, if passed back to the original data-owning organisation, can then have a '<i>reverse</i>' aggregation pipeline executed on it by the original organisation. This '<i>reverse</i>' aggregation pipeline looks up the mappings collection again, but this time to retrieve the original ID using the substitute unique IDs provided in the input (plus the data purpose). This results in a reverse engineered view of the data with real identities flagged, thus enabling the original data-owning organisation to schedule follow-ups with the identified people.</p><h3 style="text-align: left;"><br /></h3><h3 style="text-align: left;">Summary</h3><p>In this blog post I have explored why it is sometimes necessary to apply data masking to a data-set, for the masked data-set to be subsequently distributed to another business unit or organisation, but where the original identities can be safely reversed engineered. This is achieved with the appropriate strong data access controls and privileges in place, for access by specific users in the original data-owning organisation only, if the need arises. As a result, no sensitive data is ever exposed to the lesser trusted parties. </p><p><br /></p><p><i>Song for today: Lullaby by <a href="https://en.wikipedia.org/wiki/Low_(band)">Low</a></i></p></div></div></div>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-24115291707612112022021-02-10T12:40:00.027+00:002021-02-27T20:55:44.682+00:00MongoDB Data Masking Examples<h3 style="text-align: left;">Introduction</h3><p><a href="https://en.wikipedia.org/wiki/Data_masking">Data Masking</a> is a well established approach to protecting sensitive data in a database yet allowing the data to still be usable. There are a number of reasons why organisations need to employ data masking, with two of the most common being:</p><p></p><ol style="text-align: left;"><li>To obtain a recent copy of data from the production system for use in a test environment, thus enabling tests to be conducted on realistic data which better represents the business, but with the most sensitive parts redacted. Invariably, there is an increased level of reliability in the results, when tests are applied on real data, instead of using synthetically generated data.</li><li>To provide an API for consumers to access live production data, but where the consumer’s role or security level means they are not permitted to see values of all the fields in each record. Instead the subset of protected fields are represented by obfuscated values that still carry meaning (e.g the fact that a field existed or that a date field’s value has an approximate rather than random date).</li></ol><p></p><p>In this blog post I will show how MongoDB’s powerful <a href="https://docs.mongodb.com/manual/aggregation/">aggregation pipelines</a> can be used to efficiently mask data belonging to a MongoDB collection, with various examples of obfuscating/redacting fields. The power of MongoDB’s aggregation pipelines doesn’t necessarily come from any inherent ease of use, compared to say SQL. Indeed its learning curve is not insignificant (although it is far easier to learn and use than say building raw map-reduce code to run against a large data set in an Hadoop cluster, for example). The real power of aggregation pipelines comes from the fact that once a pipeline is defined, a single command can be issued to a MongoDB cluster to then be applied to massive data sets. The cluster might contain a database of billions or more of records, and the aggregation pipeline will be automatically optimised and executed, including transparently processing subsets of the data in parallel across each shard, to reduce the completion time.</p><p>To accompany this blog post, I have provided a GitHub project which shows the Mongo Shell commands used for defining the example aggregation pipeline and masking actions that I will outline here. Please keep referring to the project’s README as you read through this blog post, at:</p><p></p><ul style="text-align: left;"><li><a href="https://github.com/pkdone/mongo-data-masking">https://github.com/pkdone/mongo-data-masking</a></li></ul><p></p><p>The approach that I will describe provides examples of applying <b>irreversible</b> and <b>non-deterministic</b> data masking actions on fields. That is to say, as a result of applying the outlined data masking techniques, it would be extremely difficult, and in most cases impossible, for a user to reverse engineer or derive the original values of the masked fields.</p><p><i><span style="font-size: x-small;">[EDIT: 27-Feb-2021: For reversible data masking, see the <a href="https://pauldone.blogspot.com/2021/02/mongdb-reversible-data-masking.html">part 2 blog post on this topic</a>]</span></i></p><p><i><br /></i></p><h3 style="text-align: left;">Sample Payments Data & Data Masking Examples</h3><p>For these examples I will simulate a ‘payments system’ database containing a collection of ‘card payments’ records. Such a database might be used by a bank, a payment provider, a retailer or an eCommerce vendor, for example, to accumulate payments history. The example data structures here are intentionally over-simplified for clarity. In the screenshot below you can see an example of 2 sample payments records, in their original state, shown on the left hand side. On the right hand side of the screenshot you can see the result of applying the data masking transformation actions from the <a href="https://github.com/pkdone/mongo-data-masking">GitHub project</a>, and which I will describe in more detail further below.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0XmsGgueH7X4jLTUGOPtMcgbG2vf9tRZfapxqlIUDEwZq56SJ1LIh7CLTXdv-4NLTeLBdjxr1_h7sy6wb_M94xUqdaT2qFtQTWgbAKgwCj55RP-BVfSDVsTbR2FFQkd3C4Iw8o5Z3C2w/s1191/.datapic.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="827" data-original-width="1191" height="445" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi0XmsGgueH7X4jLTUGOPtMcgbG2vf9tRZfapxqlIUDEwZq56SJ1LIh7CLTXdv-4NLTeLBdjxr1_h7sy6wb_M94xUqdaT2qFtQTWgbAKgwCj55RP-BVfSDVsTbR2FFQkd3C4Iw8o5Z3C2w/w640-h445/.datapic.png" width="640" /></a></div><br /><p>For the example project, 10 different fields have data masking actions applied. Below is a summary of each masking action, the field it is applied to and the type of change applied. You can view and cross reference these to the 10 corresponding aggregation pipeline operations provided in the <a href="https://github.com/pkdone/mongo-data-masking">companion GitHub project</a>:</p><p></p><ol style="text-align: left;"><li>Replace the <b>card’s security code</b> with a different random set of 3 digits <b><span style="color: red;">(e.g. 133 → 472)</span></b></li><li>For the <b>card number</b>, obfuscate the first 12 digits <span style="color: red;"> <b>(e.g. 1234567890123456 → XXXXXXXXXXXX3456)</b></span></li><li>For the <b>card’s listed owner name</b>, obfuscate the first part of the name, resulting in only the last name being retained <b><span style="color: red;">(e.g. 'Mrs. Jane A. Doe' → 'Mx. Xxx Doe')</span></b></li><li>For the <b>payment transaction’s recorded date-time</b>, obfuscate the value by adding or subtracting a random amount of time to the current value, by up to one hour maximum <b><span style="color: red;">(e.g. 00:23:45 on 01-Sep-2019 → 23:59:33 on 31-Aug-2019)</span></b></li><li>Replace the <b>payment settlement's recorded date-time</b> by taking an arbitrary fixed time (e.g. 29-Apr-2020) and adding a random amount of time to that, up to one year maximum <b><span style="color: red;">(e.g. 07:48:55 on 15-Dec-2018 → 21:07:34 on 08-Jun-2020)</span></b></li><li>Replace the <b>payment card expiry date</b> with the current date (e.g. 10-Feb-2021) + a random amount of days of up to one year maximum <b><span style="color: red;">(e.g. 31-Mar-2020 → 31-Nov-2021)</span></b></li><li>For the <b>transaction’s payment amount</b>, obfuscate the value by adding or subtracting a random percent to its current value, by up to 10% maximum <b><span style="color: red;">(e.g. 49.99 → 45.51)</span></b></li><li>Replace the <b>payment’s ‘reported’ status</b> boolean value with a new randomly generated true or false value where there is a 50:50 chance of the result being either value <b><span style="color: red;">(e.g. false → true, false → false)</span></b></li><li>Replace the <b>payment transaction’s ID</b> (which is composed of 16 hexadecimal digits), with a new 16 hexadecimal digit value based on an ‘MD5 hash’ of the original ID <b><span style="color: red;">(note, do not regard this as 'cryptographically safe')</span></b></li><li>For the <b>extra customer info sub-document</b> composed of 3 fields, only retain this sub-document and its fields if the value of its ‘category’ field is not equal to the text ‘SENSITIVE’ <b><span style="color: red;">(i.e. redact out a sub-section of the document where the customer is marked as ‘sensitive’)</span></b></li></ol><p></p><h3 style="text-align: left;">Exposing Masked Data To Consumers</h3><p>Once this pipeline has been built, it can be used in one of four ways in MongoDB depending on your specific needs (again the <a href="https://github.com/pkdone/mongo-data-masking">companion GitHub project</a> provides more specific details on how each of these four ways are achieved in MongoDB):</p><p></p><ol style="text-align: left;"><li><b>DATA MASKED AGGREGATION ON DEMAND</b>. Enable the aggregation pipeline to be executed on demand (by calling <span style="font-family: Roboto Mono; font-size: x-small;">db.aggregate(pipeline)</span> from a trusted mid-tier application, where the application would have rights to see the non-obfuscated data too. As a result the trusted mid-tier would need to be relied on to return just the result of the ‘data masking aggregation’ to consumers, and to never expose the underlying unmasked data, in the database, in any other way.</li><li><b>DATA MASKED READ-ONLY VIEW</b>. Create a view (e.g. <span style="font-family: Roboto Mono; font-size: x-small;">payments_redacted_view</span>) based on the aggregation pipeline and then, using <a href="https://docs.mongodb.com/manual/core/authorization/">MongoDB’s Role Based Access Control (RBAC) capabilities</a>, only allow consumers to have access to the view, with no permissions to access the underlying source collection, thus ensuring consumers can only access the ‘data masked’ results. Consuming applications can even use a ‘find’ operation, with a ‘filter’ and ‘projection’, when querying the view, to reduce down further the fields and records they want to see.</li><li><b>DATA MASKED COPY OF ORIGINAL DATA</b>. Execute the aggregation pipeline using an additional <span style="font-family: Roboto Mono; font-size: x-small;">$merge</span> pipeline stage, to produce a brand new collection (e.g. <span style="font-family: Roboto Mono; font-size: x-small;">payments_redacted</span>), which contains only the data masked version of the original collection. The original collection can then either be locked down as not visible to the consumer, using <a href="https://docs.mongodb.com/manual/core/authorization/">RBAC</a>, or can even just be deleted, if it is no longer required. The latter option is often applicable if you are producing a test environment with a masked version of recent production data, where the original raw data is no longer needed.</li><li><b>DATA MASKED OVERWRITTEN ORIGINAL DATA</b>. Execute the aggregation pipeline using an additional <span style="font-family: Roboto Mono; font-size: x-small;">$merge</span> pipeline stage, to overwrite records in the existing collection with modified versions of the corresponding records. The resulting updated collection will now only contain the masked version of the data. Again, this may be applicable in the case where you are producing a test environment with a masked version of recent production data, where the original collection is no longer needed.</li></ol><p></p><h3 style="text-align: left;">Summary</h3><p>In this post and the <a href="https://github.com/pkdone/mongo-data-masking">companion GitHub project</a>, I’ve shown how common irreversible obfuscation patterns can be effectively defined and then applied to mask sensitive data in a MongoDB collection. The example shows only a couple of records being redacted. However, the real power comes from being able to apply the same aggregation pipeline, unchanged, to a collection containing a massive data set, often running across multiple shards, which will automatically be parallelised, to reduce turnaround times for provisioning masked data sets.</p><p><br /></p><p><i>Song for today: The Black Crow by <a href="https://en.wikipedia.org/wiki/Jason_Molina">Songs: Ohia (Jason Molina)</a></i></p>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com2tag:blogger.com,1999:blog-1304066656993695443.post-34165576260813740712020-11-24T14:49:00.011+00:002020-11-25T09:25:45.971+00:00Is Querying A MongoDB View Optimised?<p><a href="https://docs.mongodb.com/manual/core/views/">Views in MongoDB</a> appear to database users like read-only collections, ready to be queried in the same way normal collections are. A View is defined by an <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline/">Aggregation pipeline</a> and when a query is issued on a View, using <a href="https://docs.mongodb.com/manual/reference/method/db.collection.find/"><i>find()</i></a>, there is the potential for the execution of the View to be optimised by MongoDB in the same way as MongoDB would <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline-optimization/">optimise any aggregation pipeline</a> that is executed.</p><p>In reality, most applications will not issue a <i>find()</i> without specifying a query filter as an argument. This begs the question: <b>When issuing a <i>find()</i> with a query filter against a View (backed by an aggregation pipeline), how is the combination optimised, and can indexes be leveraged effectively?</b></p><p>In the rest of this post, I will explore this further and answer this question.</p><p><br /></p><h1 style="text-align: left;"><span style="font-weight: normal;">Source Collection Data</span></h1><p>The data I am using for the investigation is a <i>music</i> based data-set sourced from the <a href="https://www.discogs.com/">Discogs website</a>, imported from Discog's XML data dump, using an <a href="https://github.com/pkdone/import-large-xml-into-mdb">XML MongoDB import utility</a>.</p><p>The resulting <i>releases</i> collection, representing the albums and singles released by all artists, has over 1.5 million documents in it. I've defined various obvious indexes for the collection in anticipation of wanting to run finds and aggregations efficiently against it. Below is a screenshot showing some of the data in this collection, illustrating each document's typical shape...</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIJGaq0UdIYXBowbs8W95ljtzZFKDLoxml8HJytxZmHpuxGzH9FZsIVqP9XkVCpa_aMmraTqRoUraweREKqVQ5o1yPD6QKS1m-cHp7TbHNk9g9gQ1vT1pzVVPzwgXRgBZX69y6QvRXxVA/s731/data.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="706" data-original-width="731" height="618" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhIJGaq0UdIYXBowbs8W95ljtzZFKDLoxml8HJytxZmHpuxGzH9FZsIVqP9XkVCpa_aMmraTqRoUraweREKqVQ5o1yPD6QKS1m-cHp7TbHNk9g9gQ1vT1pzVVPzwgXRgBZX69y6QvRXxVA/w640-h618/data.png" width="640" /></a></div><div><br /></div>As you can see, the <i>releases</i> collection contains fields for the artist, the title of the release, the year of the release and the music genres & styles associated with the release.<div><br /></div><div>Let's now look at using two different Views, with different degrees of complexity, against this same collection, to see if and how these Views are optimised at runtime, when a <i>find()</i> is issued...</div><div><br /></div><div><br /></div><div><h1 style="text-align: left;"><span style="font-weight: 400;">Using A View Which Filters Out Some Records & Fields</span></h1><p>So let's create a View which only shows music released since the start of the year 2000, concatenates the array of one or more styles into a new 'style' string field and then excludes the 'styles' and '_id' fields from the result.</p><p><span style="font-family: Roboto Mono;"> var pipeline = [</span></p><p><span style="font-family: Roboto Mono;"> {$match: {'year': {'$gte': 2000}}},</span></p><p><span style="font-family: Roboto Mono;"> {$set: {'style': {</span></p><p><span style="font-family: Roboto Mono;"> $reduce: {</span></p><p><span style="font-family: Roboto Mono;"> input: '$styles',</span></p><p><span style="font-family: Roboto Mono;"> initialValue: '',</span></p><p><span style="font-family: Roboto Mono;"> in: {$concat: ['$$value', '$$this', '. ']}</span></p><p><span style="font-family: Roboto Mono;"> }</span></p><p><span style="font-family: Roboto Mono;"> }}},</span></p><p><span style="font-family: Roboto Mono;"> {$unset: ['styles']},</span></p><p><span style="font-family: Roboto Mono;"> {$unset: ['_id']},</span></p><p><span style="font-family: Roboto Mono;"> ];</span></p><p><span style="font-family: "Roboto Mono";"><span> db.createView('<b>millennium_releases_view</b>', 'releases', pipeline);</span></span></p><p>Below is an example of the shape of result documents, when the View is queried for a specific artist:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiww0UtiBAQLhoUENFvU-p1bIQXN3ckKxFseiXdivTdemyOSvGV-uFS8kFzVk1ypbvgMzZAy3_1UgHdM35p8gxTr8fJLSxMDlT6kwWLWY7PrUQ1SOF8yTgK5fyzokckxj1CaPnxohOwVFY/s763/v1.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="497" data-original-width="763" height="416" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiww0UtiBAQLhoUENFvU-p1bIQXN3ckKxFseiXdivTdemyOSvGV-uFS8kFzVk1ypbvgMzZAy3_1UgHdM35p8gxTr8fJLSxMDlT6kwWLWY7PrUQ1SOF8yTgK5fyzokckxj1CaPnxohOwVFY/w640-h416/v1.png" width="640" /></a></div><p>If I ask MongoDB to provide the explain plan for an 'empty' query on the View, using the following command...</p><p><span style="font-family: Roboto Mono;"> db.millennium_releases_view.find().explain();</span></p><p>...the resulting explain plan shows the database runs the following steps in the order shown:</p><p></p><ol style="text-align: left;"><li><b>MATCH</b> using <b>INDEX SCAN</b> hitting an index for the <i>Year</i> field</li><li><b>SET</b> new <i>Style</i> string field to concatenate values from existing <i>Styles</i> array field</li><li><b>UNSET</b> <i>Styles</i> array field</li><li><b>UNSET</b> <i>_id</i> field</li></ol><p></p><p>It's good to see here that the '<i>year greater than or equal</i>' clause in the aggregation pipeline defined for the View is being run as the first step and is targeting an index to avoid a 'full table scan'. However, what happens when I include a query filter when issuing a find() against the View, to only show releases for a specific artist?</p><p><span style="font-family: Roboto Mono;"> db.millennium_releases_view.find({<span style="background-color: #fcff01;">'artist': 'Fugazi'</span>}).explain();</span></p><p>This time the resulting explain plan shows the following steps executed:</p><p></p><ol style="text-align: left;"><li><b>MATCH</b> using <b>INDEX SCAN</b> hitting a compound index composed of both the <i><b style="background-color: #fcff01;">Artist</b></i> & <i>Year</i> fields</li><li><b>SET</b> new <i>Style</i> string field to concatenate values from existing <i>Styles</i> array field</li><li><b>UNSET</b> <i>Styles</i> array field</li><li><b>UNSET</b> <i>_id</i> field</li></ol><p></p><p>This is great news, because when I am specifying a query filter for the <i>find()</i> on this View, the optimiser is converting the regular <i>find()</i> filter syntax into an aggregation <i>match</i> expression and pushing it to the existing <i>$match</i> stage at the <u>start</u> of the pipeline. As a result, the optimum compound index of (<i>artist, year</i>) is being used, to entirely satisfy the <i>find's</i> 'artist=Fugazi' expression combined with the View's 'year>=2000' expression,.</p><p><b>Does this mean a <i>find()</i> with a query filter will always be pushed to the top of the View's aggregation pipeline, at runtime?</b></p><p>Well actually, <b>no</b>. Let's see why, in this second example...</p><p><br /></p><div><h1 style="text-align: left;"><span style="font-weight: 400;">Using A View Which Rolls Up Some Data</span></h1><p>This time let's create a View which groups releases (albums & singles) for each artist by the style associated with the release. For example, if an artist has five albums categorised with the style 'Stoner Rock' and 7 albums categorised by 'Post Rock', the resulting View will contain 2 documents for the artist, one for each of the two styles. This is the command for creating this View:</p><p><span style="font-family: Roboto Mono;"> var pipeline = [</span></p><p><span style="font-family: Roboto Mono;"> {$unwind: {path: '$styles'}}, </span></p><p><span style="font-family: Roboto Mono;"> {$group: {</span></p><p><span style="font-family: Roboto Mono;"> _id: {artist: '$artist', style: '$styles'}, </span></p><p><span style="font-family: Roboto Mono;"> titles: {'$push': '$title'},</span></p><p><span style="font-family: Roboto Mono;"> }}, </span></p><p><span style="font-family: Roboto Mono;"> {$set: {'artist': '$_id.artist'}},</span></p><p><span style="font-family: Roboto Mono;"> {$set: {'style': '$_id.style'}}, </span></p><p><span style="font-family: Roboto Mono;"> {$unset: ['_id']},</span></p><p><span style="font-family: Roboto Mono;"> ];</span></p><p><span style="font-family: Roboto Mono;"> db.createView('<b>styles_view</b>', 'releases', pipeline);</span></p><p>Below is an example of the shape of result documents from querying this new second View, for a specific artist:</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCJ6La6683thLbXD_MqIcJaa5xAFRXxdHVaIsQf765t7pKtEJ9D4xLwIJUyU2yJIbiI2uTomuxIVKA89fIDn1gmM0xYnhT500z6xwSVQ0Yv9N5siChJkwPJwbLRF30rPSTTUwDpk2AJSE/s832/v2.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="832" data-original-width="724" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCJ6La6683thLbXD_MqIcJaa5xAFRXxdHVaIsQf765t7pKtEJ9D4xLwIJUyU2yJIbiI2uTomuxIVKA89fIDn1gmM0xYnhT500z6xwSVQ0Yv9N5siChJkwPJwbLRF30rPSTTUwDpk2AJSE/w556-h640/v2.png" width="556" /></a></div><p>If I ask MongoDB to provide the explain plan for an 'empty' query on this View, using the following command...</p><p><span style="font-family: Roboto Mono;"> db.styles_view.find().explain();</span></p><p></p><p></p><ol></ol><p></p><p style="-webkit-text-stroke-width: 0px; color: black; font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-decoration-thickness: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">...the resulting explain plan shows the database runs the following steps in the order shown:</p></div><p></p><ol style="text-align: left;"><li><b>COLLECTION_SCAN</b> with <b>PROJECTION</b> of <i>Artist</i>, <i>Styles</i> & <i>Title</i> fields only</li><li><b>UNWIND</b> of <i>Styles</i> array field producing a record for each array element</li><li><b>GROUP</b> on <i>Artist + Style</i> fields, adding each associated release title to a new <i>Titles</i> array field</li><li><b>SET</b> <i>Artist</i> string field to the first of element of the group's id</li><li><b>SET</b> <i>Style</i> string field to the second of element of the group's id</li><li><b>UNSET</b> <i>_id</i> field which was created by the group stage</li></ol><p></p><p>As expected here, because the aggregation pipeline defined for the View does not contain a <i>$match</i>, the first step will result in a 'full table scan', where all the documents in the collection are inspected, and then the required fields only, are projected out.</p><p>What happens this time when I include a query filter for the <i>find()</i> run against the View, to only show results for a specific artist, using the following command to explain?</p><p><span><span style="font-family: Roboto Mono;"> db.</span><span style="font-family: "Roboto Mono";">styles_view</span><span style="font-family: Roboto Mono;">.find({<span style="background-color: #fcff01;">'artist': 'Fugazi'</span>}).explain();</span></span></p><p></p><p></p><ol></ol><p></p><p style="-webkit-text-stroke-width: 0px; color: black; font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: left; text-decoration-color: initial; text-decoration-style: initial; text-decoration-thickness: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">This time, the resulting explain plan shows the following ordered steps executed:</p><p></p><ol style="text-align: left;"><li><b>COLLECTION_SCAN</b> with <b>PROJECTION</b> of <i>Artist</i>, <i>Styles</i> & <i>Title</i> fields only</li><li><b>UNWIND</b> of <i>Styles</i> array field producing a record for each array element</li><li><b>GROUP</b> on <i>Artist + Style</i> fields, adding each associated release title to a new <i>Titles</i> array field</li><li><b>SET</b> <i>Artist</i> string field to the first of element of the group's id</li><li><span style="background-color: #fcff01;"><b>MATCH</b> on <i>Artist</i> filed (no index used)</span></li><li><b>SET</b> <i>Style</i> string field to the second of element of the group's id</li><li><b>UNSET</b> <i>_id</i> field which was created by the group stage</li></ol><p></p><p>Here the new <i>$match</i> generated by MongoDB to capture the <i>find()</i> expression run against the View, is included in the executed aggregation pipeline, but the <i>$match</i> cannot be pushed all the way up to the first step of the pipeline. This is to be expected...</p><p>Essentially what happens when a <i>find()</i> with filter is run on a View is as follows. The filter expression is initially placed in a new <i>$match</i> stage appended to the <u>end</u> of the aggregation pipeline. Then the <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline-optimization/">normal aggregation pipeline runtime optimiser</a> kicks in and attempts to move the newly added <i>$match</i> step as near to the top of the pipeline as possible. However, the <i>$group</i> stage (and related <i>$set</i> on <i>artist</i>, in this case), acts as a barrier. The <i>$group</i> operator stage completely changes the shape of documents and effectively drops any existing fields that preceded it. The optimiser has no way of knowing that a filter on an <i>artist</i> field, being applied to the outcome of a View, is definitively referring to a field called <i>artist</i> that existed in the original source collection used by the View. Instead, for all it knows, the expression on <i>artist</i> could be referring to some other intermediate aggregation pipeline field of similar name. In the example above, even if we don't use <i>$set</i> in the View's pipeline to set a new field called <i>artist</i>, the new $match expression is still blocked by <i>$group</i> and so is only executed straight after <i>$group</i> (a scenario which I also tested).</p><p>So even though I only want to see the results for one artist which relates to only a few 10s of documents in the database, the <i>find()</i> which applies a filter on the View, will result in the total data set of 1.5 million documents being 'full table scanned', adding considerable latency to the response.</p><p>If I wasn't querying the View and instead running my own hand-crafted aggregation pipeline directly against the source collection, to achieve the same functional outcome, my pipeline could be composed of the following stages where I explicitly include the match on <i>artist</i> as the first stage:</p><p><span style="font-family: Roboto Mono;"> var pipeline = [</span></p><p><span style="background-color: #fcff01; font-family: Roboto Mono;"> {$match: {'artist': 'Fugazi'}},</span></p><p><span style="font-family: Roboto Mono;"> {$unwind: {path: '$styles'}}, </span></p><p><span style="font-family: Roboto Mono;"> {$group: {</span></p><p><span style="font-family: Roboto Mono;"> _id: {artist: '$artist', style: '$styles'}, </span></p><p><span style="font-family: Roboto Mono;"> titles: {'$push': '$title'},</span></p><p><span style="font-family: Roboto Mono;"> }}, </span></p><p><span style="font-family: Roboto Mono;"> {$set: {'artist': '$_id.artist'}},</span></p><p><span style="font-family: Roboto Mono;"> {$set: {'style': '$_id.style'}}, </span></p><p><span style="font-family: Roboto Mono;"> {$unset: ['_id']},</span></p><p><span style="font-family: Roboto Mono;"> ];</span></p><p><span style="font-family: Roboto Mono;"> db.releases.aggregate(pipeline);</span></p><p>Then when I ask for the explain plan...</p><p><span style="font-family: Roboto Mono;"> db.releases.explain().aggregate(pipeline);</span></p><p></p><p></p><ol></ol><p></p><p>...I see that the following steps are executed:</p><p></p><ol style="text-align: left;"><li><span style="background-color: #fcff01;"><b>MATCH</b> using <b>INDEX_SCAN</b> on the <i>Artist</i> field</span> with <b>PROJECTION</b> of <i>Artist</i>, <i>Styles</i> & <i>Title</i> fields only</li><li><b>UNWIND</b> of <i>Styles</i> array field producing a record for each array element</li><li><b>GROUP</b> on <i>Artist + Style</i> fields, adding each release title to a new <i>Titles</i> array field</li><li><b>SET</b> <i>Artist</i> string field from the first of element in the group id</li><li><b>SET</b> <i>Style</i> string field from the second of element in the group id</li><li><b>UNSET</b> <i>_id</i> field which was created by the group stage</li></ol><p></p><p>This time an index will be leveraged so that only the few 10s of records, corresponding to the desired artist, are retrieved, ready for unwinding and grouping. The aggregation does not attempt to grab 1.5 million records. This is only possible because, as the developer of the aggregation pipeline logic, I have extra knowledge which the MongoDB runtime does not have. Specifically, I know that the <i>$match</i> on the <i>artist</i> field should actually be applied to the field named <i>artist</i> in the View's source collection and not to the result of the <i>$group</i> stage.</p><p><br /></p><h1 style="text-align: left;"><span style="font-weight: 400;">Wrapping Up</span></h1><p>What these findings show for Views is that at runtime, when MongoDB receives a <i>find()</i> containing query filter expressions, these expressions are dynamically appended to the end of the View's aggregation pipeline, before the resulting composite pipeline is executed. Then, as is the case if you are just issuing a regular aggregation against a normal collection, MongoDB's <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline-optimization/">aggregation pipeline runtime optimiser</a> attempts to re-order the pipeline on the fly, without changing its functional behaviour, to be more efficient. These runtime optimisations include attempting to push any <i>$match</i> stages as near to the start of the pipeline as possible, to help promote maximum use of indexes when executed. However, stages like the <i>$group</i> stage, which completely transform the shape of documents, mean that the optimiser cannot move a <i>$match</i> ahead of such stages, without risking changing the functional behaviour and ultimately the resulting output. </p><p>In practice, where Views are used to filter a subset of records and/or a subset of fields. the system should be able to fully optimise the <i>find()</i> run against the collection, pushing query filter expressions to the first step of the executed aggregation pipeline, to best leverage indexes. Only in places where there is a loss of fidelity (e.g when using a <i>$group</i> stage), will it be the case that the <i>find()</i> query filter cannot be placed earlier in the pipeline being executed against the View. </p><p><br /></p><p><br /></p><p><i>Song for today: Runaway Return by <a href="https://en.wikipedia.org/wiki/Fugazi">Fugazi</a></i></p></div>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-84020965052429613622020-10-04T15:31:00.009+01:002020-12-24T10:28:45.690+00:00Rust & MongoDB - Perfect Bedfellows<p>I've been learning <b>Rust</b> over the last month or so and I'm really enjoying it. It's a really elegant and flexible programming language despite being the most strongly typed and compile-time strict programming language I've ever used (bearing in mind I used to be a professional C & C++ developer way back in the day). </p><p>I'd recently read the really good and commonly referenced blog post <a href="https://blog.logrocket.com/creating-a-rest-api-in-rust-with-warp/"><b>Creating a REST API in Rust with warp</b></a>, which shows how to create a simple example <b>Groceries stock management REST API service</b>, and which uses an in-memory <i>HashMap</i> as its backing store. As part of my learning I thought I'd have a go at <i>porting</i> this to use MongoDB as its data store instead, using the fairly new <a href="https://docs.rs/mongodb/"><b>MongoDB Rust Driver</b></a>.</p><p>It turns out that this was really easy to do, also due to how well engineered the new MongoDB Rust Driver turned out to be, with its rich yet easy to use API. </p><p>You can see my resulting MongoDB version of this sample Groceries application, in the Github project <a href="https://github.com/pkdone/rust-groceries-mongo-api"><b>rust-groceries-mongo-api</b></a> I created. Check out that project link to view the source code showing how MongoDB was integrated with for the Groceries REST API and how to test the application using a REST client.</p><div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvHjTcTSMwQgMYBQN1e6KpbI9RBf_ao5pcM5IhrjGjKzDfbLr155DyWM5Ihiw3hOxwtgrZY4ZwCmIPevnR7wwMeYI5M6VRQeXrjV8yHQacMM0CXcoadp0ttuZOvSBxdS_lixvUNf0uhRQ/s930/groceries.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="779" data-original-width="930" height="536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjvHjTcTSMwQgMYBQN1e6KpbI9RBf_ao5pcM5IhrjGjKzDfbLr155DyWM5Ihiw3hOxwtgrZY4ZwCmIPevnR7wwMeYI5M6VRQeXrjV8yHQacMM0CXcoadp0ttuZOvSBxdS_lixvUNf0uhRQ/w640-h536/groceries.png" width="640" /></a></div><p>What was even more surprising was how easy it was to integrate MongoDB's flexible data model with a programming language as strict as Rust, and I encountered no friction between the two at all. In fact, this was even easier to achieve by leveraging the option of using the driver team's additional contribution of <b>BSON</b> translation to the open source <a href="https://serde.rs/"><b>Rust Serde</b></a> framework, which makes it easy to serialize/deserialize Rust data structures to/from other formats (e.g. JSON, Avro and now BSON).</p><p>I plan to blog again in the future, in more detail, about how to combine Rust's strict typing and MongoDB's flexible schema, especially when the data model and consuming microservices inevitably change over time. <b><i>[UPDATE 09-Dec-2020: I have now blogged on this at MongoDB DevHub, see: <a href="https://developer.mongodb.com/article/six-principles-building-robust-flexible-shared-data-applications">The Six Principles for Building Robust Yet Flexible Shared Data Applications</a>]</i></b></p><p><br /></p><p><i style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px;">Song for today: </i><span face="Arial, Tahoma, Helvetica, FreeSans, sans-serif" style="color: #444444;"><span style="font-size: 13px;"><i>Dissolution</i></span></span><i style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px;"> by <a href="https://en.wikipedia.org/wiki/Cloud_Nothings" style="color: #4d469c; text-decoration-line: none;">Cloud Nothings</a></i></p>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0London, UK51.5073509 -0.127758323.197117063821153 -35.284008299999996 79.817584736178844 35.028491700000004tag:blogger.com,1999:blog-1304066656993695443.post-54762499146417739752020-05-03T10:30:00.004+01:002020-09-07T11:42:13.380+01:00Converting Gnarly Date Strings to Proper Date Types Using a MongoDB Aggregation Pipeline<b><span style="font-size: medium;"><br /></span></b>
<b><span style="font-size: large;">
Introduction</span></b><br />
I recently received some example bank payments data in a CSV file which had been exported from a relational database with that database's default export settings. After using <a href="https://docs.mongodb.com/manual/reference/program/mongoimport/">mongoimport</a> to import this data 'as-is', into a MongoDB database, I noticed that there was a particularly gnarly date string field in each record. For example:<br />
<ul>
<li><b>23-NOV-20 22.57.36.827000000</b></li>
</ul>
<div>
Why do I say gnarly? Well if you lived through <a href="https://en.wikipedia.org/wiki/Year_2000_problem"><b>Y2K</b></a> you should be horrified by the 'year' field shown above. How would you know from the data, without any context, <b>what century this applies to?</b> Is it 1920? Is it 2020? Is it 2120? There's no way of knowing from just the exported data alone. Also, there is no indication of <b>which time zone this applies to</b>. Is it British Summer Time? Is it Eastern Daylight Time? Who knows? Also the month element appears to be an abbreviation of a month expressed in a specific spoken language. <b>Which spoken language?</b></div>
<div>
<br /></div>
<div>
I needed to get this into a proper Date type in MongoDB so I could then easily index it, perform date range queries natively, perform sort by date natively, etc.. My usual tool of choice for this is MongoDB's Aggregation pipeline to generate a new collection from the existing collection with the 'date' string fields converted to proper date type fields. To perform the string to date conversion, the usual operator of choice to use is <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/dateFromString/">$dateFromString</a> (introduced in MongoDB 3.6). </div>
<div>
<br /></div>
<div>
However, $dateFromString [rightly] expects an input string which isn't missing crucial date related text, indicating things like the century or timezone. Also, the $dateFromString operator contains no format specifiers to indicate that the text 'NOV' maps to the 11<i>th</i> month of a year in a specific spoken language.</div>
<div>
<br /></div>
<div>
Therefore, armed with the extra context of knowing this exported data refers to dates in the 21st century (the '2000s') with a <a href="https://en.wikipedia.org/wiki/Coordinated_Universal_Time">UTC</a> 'time zone' and in the English language (only inferred by asking the owner of the data), I had to perform some additional string manipulation in the aggregation pipeline before using $dateFromString to generate a true and accurate date type. The rest of this blog post shows how I achieved this for date strings like '23-NOV-20 22.57.36.827000000'.</div>
<div>
<br /></div>
<b><br /></b>
<b><span style="font-size: large;">
Converting Incomplete Date Strings to Date Types Example</span></b><br />
<div>
<br />
In the Mongo Shell targeting a running MongoDB test database, run the following code to insert 12 sample 'payment' records, with example 'bad date string' fields for testing each month of a sample year.</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">use test;</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db.rawpayments.insert([</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '010101', 'pymntdate': '01-JAN-20 01.01.01.123000000', 'amount': 1.01},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '020202', 'pymntdate': '02-FEB-20 02.02.02.456000000', 'amount': 2.02},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '030303', 'pymntdate': '03-MAR-20 03.03.03.789000000', 'amount': 3.03},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '040404', 'pymntdate': '04-APR-20 04.04.04.012000000', 'amount': 4.04},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '050505', 'pymntdate': '05-MAY-20 05.05.05.345000000', 'amount': 5.05},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '060606', 'pymntdate': '06-JUN-20 06.06.06.678000000', 'amount': 6.06},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '070707', 'pymntdate': '07-JUL-20 07.07.07.901000000', 'amount': 7.07},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '080808', 'pymntdate': '08-AUG-20 08.08.08.234000000', 'amount': 8.08},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '090909', 'pymntdate': '09-SEP-20 09.09.09.567000000', 'amount': 9.09},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '101010', 'pymntdate': '10-OCT-20 10.10.10.890000000', 'amount': 10.10},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '111111', 'pymntdate': '11-NOV-20 11.11.11.111000000', 'amount': 11.11},</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> {'account_id': '121212', 'pymntdate': '12-DEC-20 12.12.12.999000000', 'amount': 12.12}</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">]);</span></div>
</div>
<div>
<br /></div>
<div>
Then execute the following Aggregation pipeline to copy the contents of the 'rawpayments' collection, populated above, into a new collection named 'payments', but with the 'pymntdate' field values converted from string types to date types.</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db.rawpayments.aggregate([</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$set: {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pymntdate: {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> $dateFromString: {format: '%d-%m-%Y %H.%M.%S.%L', dateString:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$concat: [</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$substrCP: ['$pymntdate', 0, 3]}, // USE FIRST 3 CHARS IN DATE STRING</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$switch: {branches: [ // REPLACE MONTH 3 CHARS IN DATE STRING WITH 2 DIGIT MONTH</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'JAN']}, then: '01'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'FEB']}, then: '02'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'MAR']}, then: '03'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'APR']}, then: '04'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'MAY']}, then: '05'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'JUN']}, then: '06'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'JUL']}, then: '07'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'AUG']}, then: '08'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'SEP']}, then: '09'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'OCT']}, then: '10'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'NOV']}, then: '11'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {case: {$eq: [{$substrCP: ['$pymntdate', 3, 3]}, 'DEC']}, then: '12'},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ], default: 'ERROR'}},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> '-20', // ADD HYPEHN + HARDCODED CENTURY 2 DIGITS</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$substrCP: ['$pymntdate', 7, 15]} // USE REMAINING PART OF DATE STRING UP UNTIL THE 3 MILLISECOND DIGITS (IGNORE REMAINING 6 NANOSECOND CHARS)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }, </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$out: 'payments'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">]);</span><span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
<div>
<br /></div>
<div>
In this pipeline, the string <b>'23-NOV-20 22.57.36.827000000'</b> will be converted to <b>'ISODate("2020-11-23T22:57:36.827Z")' </b>by concatenating the following four elements of text together before passing it to the $dateFromString operator to convert to a date:</div>
<div>
<ol>
<li><b>'23-'</b> <i>(from the input string)</i></li>
<li><b>'11'</b> <i>(replacing 'NOV')</i></li>
<li><b>'-20'</b> <i>(hard-coded hyphen + century)</i></li>
<li><b>'20 22.57.36.827'</b> <i>(the rest of input string apart from last 6 nanosecond digits)</i></li>
</ol>
</div>
<div>
<i>Note</i>: A $set stage is used in this pipeline, which is a type of stage first introduced in MongoDB 4.2. <b>$set</b> is an <b>alias</b> for <b>$addFields</b>, so if using an earlier version of MongoDB, replace $set with $addFields in the pipeline.<br />
<br />
To see what the converted records look like, containing new date types, query the new collection:</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db.payments.find({}, {_id:0});</span></div>
<div>
<br /></div>
<div>
Which will show the following results:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "010101", "pymntdate" : ISODate("2020-01-01T01:01:01.123Z"), "amount" : 1.01 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "020202", "pymntdate" : ISODate("2020-02-02T02:02:02.456Z"), "amount" : 2.02 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "030303", "pymntdate" : ISODate("2020-03-03T03:03:03.789Z"), "amount" : 3.03 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "040404", "pymntdate" : ISODate("2020-04-04T04:04:04.012Z"), "amount" : 4.04 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "050505", "pymntdate" : ISODate("2020-05-05T05:05:05.345Z"), "amount" : 5.05 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "060606", "pymntdate" : ISODate("2020-06-06T06:06:06.678Z"), "amount" : 6.06 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "070707", "pymntdate" : ISODate("2020-07-07T07:07:07.901Z"), "amount" : 7.07 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "080808", "pymntdate" : ISODate("2020-08-08T08:08:08.234Z"), "amount" : 8.08 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "090909", "pymntdate" : ISODate("2020-09-09T09:09:09.567Z"), "amount" : 9.09 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "101010", "pymntdate" : ISODate("2020-10-10T10:10:10.890Z"), "amount" : 10.1 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "111111", "pymntdate" : ISODate("2020-11-11T11:11:11.111Z"), "amount" : 11.11 }</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">{ "account_id" : "121212", "pymntdate" : ISODate("2020-12-12T12:12:12.999Z"), "amount" : 12.12 }</span></div>
</div>
<div>
<br />
<br /></div>
<div>
<br /></div>
<div>
<i>Song for today: For Everything by <a href="https://en.wikipedia.org/wiki/The_Murder_Capital">The Murder Capital</a></i></div>
Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-34993725657342468522019-12-29T23:14:00.003+00:002020-11-10T15:56:50.322+00:00Running MongoDB on ChromeOS (via Crostini)In my <a href="http://pauldone.blogspot.com/2019/12/my-notes-chromeos-crostini.html">previous post</a> I explored Linux application support in ChromeOS and Chromebooks (a.k.a. Crostini). Of course I was bound to try running MongoDB in this environment, which I found to work really well (for development purposes). Here's my notes on running a MongoDB database and tools on a Chromebook with Linux (beta) enabled:<br />
<ul>
<li>In ChromeOS, launch the Terminal app (which opens a Shell inside the 'Penguin' Linux container inside the 'Termina' Linux VM)</li>
<li>Run the following commands which are documented in the <a href="https://docs.mongodb.com/manual/tutorial/install-mongodb-enterprise-on-debian/">MongoDB Manual page</a> on installing MongoDB Enterprise on Debian (following the manual's tab instructions titled “Debian 9 "Stretch”):</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">wget -qO - https://www.mongodb.org/static/pgp/server-4.2.asc | sudo apt-key add -</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">echo "deb http://repo.mongodb.com/apt/debian stretch/mongodb-enterprise/4.2 main" | sudo tee /etc/apt/sources.list.d/mongodb-enterprise.list</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo apt-get update</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo apt-get install -y mongodb-enterprise</span><br />
<ul>
<li>Start a MongoDB database instance running:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">mkdir ~/data</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">mongod --dbpath ~/data</span><br />
<ul>
<li>Launch a second Terminal window and then run the Mongo Shell against this database and perform a quick database insert and query test:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">db.mycoll.insert({a:1})</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">db.mycoll.find()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">db.mycoll.drop()</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">exit</span><br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGIesQVnjFuIvGmrdFvUtPx3K9kxwVXcSQCi7g1urtVft8D1gTTu5izAS8ADbRf67U2dhfQmMNPHFuIN4a8MUWH2ecUrFkXxdz8zZfAIdx181G1BmslEDDvrb51wdxvPuI3UxOfRXD3vk/s1600/Screenshot+2019-12-29+at+4.26.30+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGIesQVnjFuIvGmrdFvUtPx3K9kxwVXcSQCi7g1urtVft8D1gTTu5izAS8ADbRf67U2dhfQmMNPHFuIN4a8MUWH2ecUrFkXxdz8zZfAIdx181G1BmslEDDvrb51wdxvPuI3UxOfRXD3vk/s640/Screenshot+2019-12-29+at+4.26.30+PM.png" width="640" /></a></div>
<div>
<ul>
<li>Install Python 3 and the PIP Python package manager (using Anaconda) and then install the MongoDB Python driver (PyMongo):</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">wget https://repo.anaconda.com/archive/Anaconda3-2019.10-Linux-x86_64.sh</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">bash Anaconda3-*-Linux-x86_64.sh</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">source ~/.bashrc</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">python --version</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">pip --version</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">pip install --user pymongo</span><br />
<ul>
<li>Test PyMongo by running a small ‘payments data generator’ Python script pulled down from a GitHub repository (this should insert records into the MongoDB local database’s “fs.payments” collection; after letting it run for a minute, continuously inserting new records, press Ctrl-C to stop it):</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">git clone https://github.com/pkdone/PaymentsWriteReadConcerns.git</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">cd PaymentsWriteReadConcerns/</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">./payments-records-loader.py -p 1</span><br />
<ul>
<li>Download MongoDB Compass (use the Ubuntu 64-bit 14.04+ version), install and run it against the 'localhost' MongoDB database and inspect the contents of the “fs.payments” collection:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">wget https://downloads.mongodb.com/compass/mongodb-compass_1.20.4_amd64.deb</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">sudo apt install ./mongodb-compass_*_amd64.deb</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">mongodb-compass</span></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipf_xoHorWOfcg6765Dx-l4oE6Gy_f2qJpcm16yVbiE7ao3AGGLaRJwrzM90-yJH2KmhvK7e4mS6pUt5qvpdhd9G69miOYp_ZUBTi50I6d_H8x9KnX7Oa8V7pXOA7EKYwxIXGNQKs59lc/s1600/Screenshot+2019-12-29+at+4.10.49+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEipf_xoHorWOfcg6765Dx-l4oE6Gy_f2qJpcm16yVbiE7ao3AGGLaRJwrzM90-yJH2KmhvK7e4mS6pUt5qvpdhd9G69miOYp_ZUBTi50I6d_H8x9KnX7Oa8V7pXOA7EKYwxIXGNQKs59lc/s640/Screenshot+2019-12-29+at+4.10.49+PM.png" width="640" /></a></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<i>Song for today: Sun. Tears. Red by <a href="https://en.wikipedia.org/wiki/Jambinai">Jambinai</a></i></div>
<div>
<br /></div>
Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com1tag:blogger.com,1999:blog-1304066656993695443.post-2422369463652932552019-12-29T21:56:00.003+00:002020-11-10T15:57:40.049+00:00My Notes on Linux Application Support in ChromeOS (a.k.a. Crostini)These are my own rough notes from spending a few days studying Chrome OS and its Linux app support on a HP Chromebook 14* I got for free (retails for about £150) when I recently purchased a Google Pixel 4 Android mobile phone. I thought I’d share the notes in case they are of use to others. I’m sure there needs to be some corrections, so feedback is welcome.<br />
<div style="text-align: right;">
<span style="font-size: xx-small;">* released: 2019, model: db0003na, codename: careena, board: grunt</span></div>
<br />
Some references to other articles that I used to bootstrap my knowledge:<br />
<ul>
<li><a href="https://chromium.googlesource.com/chromiumos/docs/+/master/containers_and_vms.md">Running Custom Containers Under Chrome OS</a> from the Chromium OS Docs</li>
<li>A <a href="https://support.google.com/chromebook/thread/1319823?hl=en">useful set of answers provided to a query</a> on the Chromebook Community Help site</li>
<li>An article called <a href="https://blog.simos.info/a-closer-look-at-chrome-os-using-lxd-to-run-linux-gui-apps-project-crostini/">A closer look at Chrome OS using LXD to run Linux GUI apps (Project Crostini</a>)</li>
</ul>
<br />
Below are some screenshots showing the ChromeOS Settings section where “Linux (beta)” (a.k.a. Crostini) can be enabled and the Linux apps that are then installed by default when (essentially just the GNOME Help application and the Terminal application, from which many other Linux apps can subsequently be installed):<br />
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEOTUTxsYs3MXZP5d9pbOTDDPM3lFoK0qH0fM3LKurfc9CH9Gf6yyQthGqb_2WqA4pvArIJ-K7shCAq3aa9y6n1GjN96F7YC0wBGiaW0Ex4lfF2lCtnntQZ7FgwWL9IgY12jXk6fbCE7U/s1600/Screenshot+2019-12-29+at+5.37.33+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEOTUTxsYs3MXZP5d9pbOTDDPM3lFoK0qH0fM3LKurfc9CH9Gf6yyQthGqb_2WqA4pvArIJ-K7shCAq3aa9y6n1GjN96F7YC0wBGiaW0Ex4lfF2lCtnntQZ7FgwWL9IgY12jXk6fbCE7U/s640/Screenshot+2019-12-29+at+5.37.33+PM.png" width="640" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcZa8siUhYBJ610yPPwF5_y4YwVg2qQxO0uI6JCgSk0w7ccG2AAcEiLQjlsEdwBJaC3MMYX4IcfFC4XU9V9W-htFoYpxYVFVWbHGHtP7QbWU5uaVx3Fgpf4Ngu1zCrlJ1c3iZ4aljgzVg/s1600/Screenshot+2019-12-29+at+5.06.37+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhcZa8siUhYBJ610yPPwF5_y4YwVg2qQxO0uI6JCgSk0w7ccG2AAcEiLQjlsEdwBJaC3MMYX4IcfFC4XU9V9W-htFoYpxYVFVWbHGHtP7QbWU5uaVx3Fgpf4Ngu1zCrlJ1c3iZ4aljgzVg/s640/Screenshot+2019-12-29+at+5.06.37+PM.png" width="640" /></a></div>
<div>
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrZc3pYAxTqYjPEu2NxZ0f8wAuMCcRBNjdMA5xn3eY5vqS4tNxALrZmgeAuRYu0ayDRD1OP61ZimKcWeegBWzuRyeOJhid_-toTf5-8VmzaanVMU63xwuJmZCUT7-vG0g9h2WbtOpvFGI/s1600/Screenshot+2019-12-29+at+5.13.17+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgrZc3pYAxTqYjPEu2NxZ0f8wAuMCcRBNjdMA5xn3eY5vqS4tNxALrZmgeAuRYu0ayDRD1OP61ZimKcWeegBWzuRyeOJhid_-toTf5-8VmzaanVMU63xwuJmZCUT7-vG0g9h2WbtOpvFGI/s640/Screenshot+2019-12-29+at+5.13.17+PM.png" width="640" /></a></div>
<div>
<br /></div>
<div>
Here is a diagram I put together to attempt to capture the architecture of Crostini in ChromeOS as I understand it (the rest of this document digs into the details behind some of these layers):</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5vljikDXJcQh0aGRlWN0PBMRRUM0qUzxEoM4sPLErNAy12cxpOFZ8PvlSdRwUNPbu1AVJ0mhuArrzEgZhIPv3RT2BILHUQkl2dYyE7b1vklHqReR4PZbHT3OkgvZEGwgbQAcDNPf5XiM/s1600/Crostini+Arch.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="540" data-original-width="960" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5vljikDXJcQh0aGRlWN0PBMRRUM0qUzxEoM4sPLErNAy12cxpOFZ8PvlSdRwUNPbu1AVJ0mhuArrzEgZhIPv3RT2BILHUQkl2dYyE7b1vklHqReR4PZbHT3OkgvZEGwgbQAcDNPf5XiM/s640/Crostini+Arch.png" width="640" /></a></div>
<div>
<h3>
ChromeOS & Crostini</h3>
<div>
<ul>
<li>Under the covers, ChromeOS is based on Gentoo and the Portage package manager</li>
<li>crosh (ChromeOS Developer Shell) is the pluggable command line shell/terminal for ChromeOS (in the Chrome browser, enter Ctrl-Alt-T to launch crosh inside a browser tab)</li>
<li>Crostini is the term for Linux application support in ChromeOS which manages the specific Linux VM and then the specific Linux container inside it, managing the lifecycle of when to launch them, mounting the filesystem to show the container’s files in the ChromeOS Files app, etc.. Crostini provides easy to use Linux application support integrated directly into the running ChromeOS desktop, rather than, for example, needing to dual boot or having to run a separate Linux VM and needing to explicitly switch, via the desktop, between ChromeOS and the Linux VM.</li>
<li>ChromeOS also has a Developer mode (verification is disabled when the OS boots) which is a special mode built into all Chromebooks to allow users and developers to access the code behind the Chrome Operating System and load their own builds of ChromeOS. This mode also allows users to install and run another Linux system like Ubuntu instead of ChromeOS (i.e. dual boot), but still have ChromeOS available to boot into too</li>
<li>As an alternative to Crostini, in addition to the dual-boot option, developer mode can also be used for Crouton which is a set of scripts that bundle up a chroot generator/environment to run both ChromeOS and Ubuntu at the same time. Here a Linux OS runs alongside ChromeOS, so users can switch between the ChromeOS desktop and Linux desktops via a keyboard shortcut. This gives users the ability to take advantage of both environments without needing to reboot. Unlike with virtualisation, a second OS is not being booted and instead the guest OS is running using the Chromium OS system. As a result any performance penalty is reduced because everything is run natively, and RAM is not being wasted to boot two OSes at the same time. Note, Crostini is different than this Crouton capability, as it enables the Linux shell and apps to be brought into the platform in verified (non-developer) mode with seamless user interface desktop integration and multi-layered security, in a supported way.</li>
<li>To use Crostini, from the ChromeOS Settings select ‘Linux (Beta)’ and choose to enable it, which, behind the scenes, will download and configure a specific Linux VM containing a specific Linux Container (see the next sections for more details) and it adds a launcher group to the ChromeOS desktop called ‘Linux Apps’. This launcher group includes a launcher to run a Linux shell/terminal application, called Terminal, which is displayed in the ChromeOS desktop but is connected directly inside the container</li>
</ul>
<br />
<ul>
</ul>
</div>
</div>
<div>
<h3>
Crostini Linux VM Layer</h3>
<div>
<ul>
<li>crosvm (ChromeOS Virtual Machine Monitor) is a custom virtual machine manager written in Rust that runs guest VMs via Linux's KVM hypervisor virtualisation layer and manages the low-level virtual I/O device communication (Amazon’s Firecracker is a fork of crosvm)</li>
<li>A specific VM is used to run a container rather than ChromeOS running a container directly, for security reasons because containers do not provide sufficient security isolation on their own. With the two layers, an adversary has to exploit crosvm via its limited interactions with the guest, in addition to the container, and the VM itself is heavily sandboxed.</li>
<li>The VM (and its container) are tied to a ChromeOS login session and as soon as a user logs out, all programs are shut down/killed by design (all user data lives in the user’s encrypted home to ensure nothing is leaked when a user log out). The VM, container and their data are persisted across user sessions and are kept in the same per-user encrypted storage as the rest of the browser's data.</li>
<li>KVM generally (rather than Crostini specifically) can execute multiple virtual machines running unmodified Linux or Windows images. Each virtual machine has private virtualised hardware: a network card, disk, graphics adapter, etc. The kernel component of KVM is included in mainline Linux codebase and the userspace component of KVM is included in mainline QEMU codebase</li>
<li>Termina is the VM launched by crosvm and is based on a ChromeOS (CrOS) image with a stripped-down ChromeOS Linux kernel and userland tools. The main goal is to just boot up Termina as quickly as possible, as a secure sandbox, and start running containers.</li>
<li>Currently, other custom VMs (other Linux variants, Windows, etc) cannot be run and only instances of the Termina VM image can be booted, although multiple VM instances can be run simultaneously based on the Termina image</li>
<li>vmc is the crosh command line utility to manually manage custom VM instances via Concierge (the ChromeOS daemon that manages VM/container life cycles)</li>
<li>To view the registered VM(s) from crosh (Ctrl-Alt-T), which may or may not be running, run:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vmc list</span><br />
<ul>
<li>To launch the Termina VM as a VM instance called ‘termina’ and open a shell directly in the VM, run:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vmc start termina</span><br />
<ul>
<li>With the above command, the default container in the VM will not be started automatically. However, instead, if from the ChromeOS desktop, a Linux Shell (Terminal) or other Linux App is launched (or the ‘Linux files’ app, Files , is launched) the Termina VM is automatically launched and the default container it owns is also automatically started</li>
<li>If the Termina VM is already running, to connect to it via a shell, run:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vsh termina</span><br />
<ul>
<li>If the ‘vmc start’ command is run with a different VM name, a new VM of that name will be created, launched and its shell entered from the existing terminal command line. This will use the same Termina image, and when running, ‘vmc list’ with list both VMs (the new instance doesn’t have any containers defined in it by default, ready to run, unlike the main Termina VM)</li>
<li>To stop the main Termina VM, run:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vmc stop termina</span></div>
</div>
<div>
<br />
<br /></div>
<div>
<h3>
Crostini Container Layer</h3>
<ul>
<li>The Termina VM only supports running containers using the “Linux Containers” (LXC) technology at the moment and doesn’t support Docker or other container technologies</li>
<li>The default container instance launched via Termina is called Penguin and is based on Debian 9 with some custom packages</li>
<li>Containers are run inside a VM rather than programs running directly in the VM to help keep VM startup times low, to help improve security sandboxing by providing a stateless immutable VM image and to allow the container, its applications and their dependencies to be maintained independently from the VM, which otherwise may have contradicting dependecy requirements</li>
<li>LXC, generally, works in the vanilla Linux kernel requiring no additional patches to be applied to the kernel source and uses various kernel features to contain processes including kernel namespaces (ipc, uts, mount, pid, network and user), Apparmor and SELinux profiles, Seccomp policies, chroots (using pivot_root), CGroups (control groups). LXCFS provides the userspace (FUSE) filesystem providing overlay files for cpuinfo, meminfo, stat and uptime plus a cgroupfs compatible tree allowing unprivileged writes.</li>
<li>LXD is a higher level container framework, which Crostini uses and LXD uses its own specific image formats and also provides the ability to manage containers remotely. Although LXD uses LXC under the covers, it is based on more than just LXC. The Termina VM is configured to run the LXD daemon. Confusingly, the command line tool for controlling LXD is called ‘lxc’ (the ‘LXD Client). If users are using LXD commands to manage containers, they should avoid using any commands that start with ‘lxc-’ as these are lower level LXC commands. Users should avoid mixing and matching the use of both sets of commands in the same system. Crostini uses LXD to launch the Penguin container and LXD is configured to only allow unprivileged containers to be run, for added security. Therefore with Crostini, users should not use the lower level ‘lxc-’ commands because these can’t manage the LXD derived containers that Crostini uses. By default, LXD comes with 3 remote repositories providing images: 1) ubuntu: (for stable Ubuntu images), 2) ubuntu-daily: (for daily Ubuntu images), and 3) images: (for other distros)</li>
<li>In the Termina VM, the full LXC/LXD capabilities are provided, and remote images for many types of distros can be used to spawn multiple containers, in addition to the main Penguin container (these are not tested or certified though so may or may not work correctly)</li>
<li>Sommelier (a Wayland proxy compositor provides seamless X forwarding integration for content, input events, clipboard data, etc... between Linux apps and the ChromeOS desktop) and Garcon (a daemon for passing requests between the container and ChromeOS) binaries are bind-mounted into the main Penguin container. The Penguin container’s systemd is automatically configured to start these daemons. The libraries for these daemons are already present in the Penguin container LXD image used for Penguin (‘google:debian/stretch’). Other LXD containers launched in the VM don't seem to be enabled for their X based GUI apps to be displayed in the ChromeOS desktop, even if they use the special ‘google:debian/stretch’ LXD container image as it seems Crostini won’t attempt to integrate with this at runtime. Note: Some online articles imply it may be possible to get X-forwarding working from multiple containers.</li>
<li>In the Penguin container (which users can access directly, via the Terminal app launcher in the ChomeOS desktop), users can query the IP address of the container which is accessible from ChromeOS and can then run crosh (Ctrl-Alt-T) in ChromeOS and ping the IP address of the container directly. Users can also SSH from the ChromeOS desktop to the Penguin container using Google’s official SSH client that can be installed in Chrome via Chrome Web Store</li>
<li>If other containers are launched and then Google’s official SSH client is installed in ChromeOS (install ‘Secure Shell Extension’ via the Chrome Web Store), users can then define SFTP mount-points to other non-Penguin containers and the files in these containers will automatically appear in the Files app too </li>
<li>From the Termina VM, users can use the standard LXD lxc command line tool to list containers and then to see if the Penguin container is running, by running:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc list<br />lxc info penguin | grep "Status: "</span><br />
<ul>
<li>To check the logs for the Penguin container, run:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc info --show-log penguin</span><br />
<ul>
<li>To open a command line shell as root in the running container (note, the Terminal app has a different identity for connecting to the Penguin container, which is a non-root user), run</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc exec penguin -- /bin/bash</span><br />
<ul>
<li>Within the Penguin container you can run GUI apps which automatically display in the main ChromeOS user interface. For example to install the GEdit text editor Linux application run the following (which also adds a launcher for GEdit in the ChromeOS desktop ‘Linux Apps’ launcher group):</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">sudo apt install gedit</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"><br /></span></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirwVgC37VNBHOGNY5RrwjpeM0pv_nzOTHremG0BFp8i3Wj5dqUjBjDQ3xga1jFG4YIucDxLQrzF-VNsnlqj4KoGjDmiyT-V94vqUwtgcMaFw7vCLP6l6nCSjskhkpIbsVSZ4y45RLVwoo/s1600/Screenshot+2019-12-29+at+5.03.39+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirwVgC37VNBHOGNY5RrwjpeM0pv_nzOTHremG0BFp8i3Wj5dqUjBjDQ3xga1jFG4YIucDxLQrzF-VNsnlqj4KoGjDmiyT-V94vqUwtgcMaFw7vCLP6l6nCSjskhkpIbsVSZ4y45RLVwoo/s640/Screenshot+2019-12-29+at+5.03.39+PM.png" width="640" /></a></div>
<div>
<ul>
<li>It is even possible to install and run a new Google Chrome browser installation from the Linux container, by running the following (which also adds a launcher for this Linux version of Chrome in the ChromeOS desktop ‘Linux Apps’ launcher group):</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">sudo apt install ./google-chrome-stable_current_amd64.deb</span></div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcG1bZPGPI9bK2qWqI3Lzq2Ctk701SGxdR2dNaIOscfUkvOfy8r6TyiPyuOXROEGml-CJSmErit-NyIGaHn44RQFjegOuRF6q5MNGb8pYzRWUXwzBAyTHksXQY0M0kNxyJHJSw3aAHbWs/s1600/Screenshot+2019-12-29+at+5.01.09+PM.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="768" data-original-width="1366" height="358" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjcG1bZPGPI9bK2qWqI3Lzq2Ctk701SGxdR2dNaIOscfUkvOfy8r6TyiPyuOXROEGml-CJSmErit-NyIGaHn44RQFjegOuRF6q5MNGb8pYzRWUXwzBAyTHksXQY0M0kNxyJHJSw3aAHbWs/s640/Screenshot+2019-12-29+at+5.01.09+PM.png" width="640" /></a></div>
</div>
<div>
<ul>
<li>From crosh (Ctrl-Alt-T), it is also possible to start the main container in the main VM (if not already started) and then connect a shell directly to the main container in the main VM, by running</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vmc container termina penguin</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vsh termina penguin</span><br />
<br />
<br />
<h3>
Playing with Custom Containers</h3>
<ul>
<li>First of all launch crosh (Ctrl-Alt-T), and connect a shell to the Termina VM:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vsh termina</span><br />
<ul>
<li>Import Google’s own image repository into LXD to include the special Debian image used by Penguin:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc remote list<br />lxc remote add google https://storage.googleapis.com/cros-containers --protocol=simplestreams<br />lxc remote list<br />lxc image list google:<br />lxc image info google:debian/stretch</span><br />
<ul>
<li>Launch and test a container using Google’s special Debian 9 image:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc launch google:debian/stretch mycrosdebiancontainer<br />lxc list<br />lxc exec mycrosdebiancontainer -- /bin/bash<br />cat /etc/*elease*<br />apt update && apt upgrade -y<br />exit</span><br />
<ul>
<li>Launch and test a container using a standard Ubuntu 18.04 image:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc launch ubuntu:18.04 myubuntucontainer<br />lxc list<br />lxc exec myubuntucontainer -- /bin/bash<br />cat /etc/*elease*<br />apt update && apt upgrade -y<br />exit</span><br />
<ul>
<li>Launch and test a container using a standard Centos 7 image:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc launch images:centos/7 mycentoscontainer<br />lxc list<br />lxc exec mycentoscontainer -- /bin/bash<br />cat /etc/*elease*<br />yum -y update<br />exit</span><br />
<ul>
<li>If the Chromebook is rebooted and the Termina VM restarted, these 3 containers still exist as they are persisted, but they will be in a stopped state. When the containers are then manually restarted they will still have the same settings, files and modifications that were made before they were stopped. To start a stopped container run (example shown for one of the containers):</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">lxc start myubuntucontainer</span><br />
<ul>
<li>None of the containers launched above seem to enable GUI apps (e.g. GEdit) to be forwarded automatically to the ChromeOS desktop. Even though the ‘google:debian/stretch’ based container has the relevant X forwarding libraries bundled, it doesn't seem to be automatically integrated with at runtime by the Crostini framework to enable X forwarding</li>
<li>Another way to launch a new container is to use one of the following commands, although, again, neither seem to automatically configure X-forwarding, even though they use the ‘google:debian/stretch’ image. It seems that only the Penguin container specifically is beiung managed by Crostini and has X forwarding configured (the first command below should be launched from ChromeOS crosh, the second command which is deprecated performs the same action but should be run from inside the Termina VM:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">vmc container termina mycontainer<br />run_container.sh --container_name=mycontainer --user=jdoe --shell</span><br />
<ul>
<li>Note, this may throw a timeout error similar to below, but the containers do seem to be created ok:</li>
</ul>
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;">Error: routine at frontends/vmc.rs:397 `container_create(vm_name,user_id_hash,container_name,image_server,image_alias)` failed: timeout while waiting for signal</span><br />
<br />
<br />
<i>Song for today: The Desert Song, No.2 - live by <a href="https://en.wikipedia.org/wiki/Sophia_(British_band)">Sophia</a></i></div>
Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-74796940077025730762019-12-19T14:50:00.007+00:002020-12-20T09:59:23.323+00:00Some Tips for Diagnosing Client Connection Issues for MongoDB Atlas<h3>
</h3>
<h3>
Introduction</h3>
<br />
<b><i> [UPDATE 07-Sep-2020: I've now written an executable binary tool you can run which performs the equivalent of the checks in this blog post to diagnose connectivity issues to Atas or any other type of MongoDB deployment, downloadable from <a href="https://github.com/pkdone/mongo-connection-check">here</a>]</i></b>
<br />
<br />
By default, for recent MongoDB drivers and client tools, <a href="https://www.mongodb.com/cloud/atlas">MongoDB Atlas</a> advertises the exposed URL for a deployed database cluster using a <i>service name</i> which maps to a set of <a href="https://en.wikipedia.org/wiki/SRV_record">DNS SRV records</a> to provide an initial connection <b>seed list</b>. This results in a much more 'human digestible' URL, but more importantly, increases deployment flexibility and the ability for underlying database server hosts to migrate over time, without needing to subsequently reconfigure clients.<br />
<br />
For example, an Atlas Cluster may be referenced in a connection string by:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> testcluster-abcd.mongodb.net</span><br />
<br />
...as an alternative to the full connection endpoint list:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> testcluster-shard-00-00-abcd.mongodb.net:27017,testcluster-shard-00-01-abcd.mongodb.net:27017,testcluster-shard-00-02-abcd.mongodb.net:27017/test?replicaSet=TestCluster-shard-0</span><br />
<br />
It is worth noting though, whichever approach is used (explicitly defining all endpoints in the connection string or having it discovered via the DNS SRV service name), the connection URL seed list is only ever used for bootstrapping a client application to the database cluster, when the client first starts or when it later needs to restart. On start-up, the client uses the connection seed list to attempt to attach to any member of the cluster, and in fact, all but one of the endpoints could be incorrect and a successful cluster connection will still be achieved. Once the initial connection is made, the true cluster member endpoint list is dynamically and continuously shared between the cluster and the client at runtime. This enables the client to continue operating against the database even if the members of the database cluster change locations or identities over time. For example, after a year of a database cluster and application continuously running, there could be the need to increase database capacity by dynamically rotating the database hosts to new higher processing capacity machines. This all happens dynamically and the already running client application automatically becomes aware and leverages the new hosts without downtime and without needing to consult the connection string again. If the client application restarts though, it will need to read the updated connection string to be able to bootstrap a connection back up to the database cluster.<br />
<br />
In the rest of this post we will explore some of the ways initial client connectivity issues can be diagnosed and resolved when using DNS SRV based connection URLs. For reference, <a href="https://twitter.com/jdrumgoole">Joe Drumgoole</a> provides a <a href="https://www.mongodb.com/blog/post/mongodb-3-6-here-to-SRV-you-with-easier-replica-set-connections">great explanation</a> about how DNS SRV records work more generally, and how MongoDB drivers and tools can leverage these.<br />
<br />
<h3>
Naive Connectivity Diagnosis</h3>
<br />
If you are having connection problems with Atlas when using the SRV service name based URL, be weary of drawing the wrong conclusions regarding the cause of the connection problem...<br />
<br />
For example, lets say you can't connect an application to a cluster with the Atlas advertised URL of '<i>mongodb+srv://testcluster-abcd.mongodb.net</i>' from your laptop. You may be tempted to try to debug the connection problem by running some of the following commands from your laptop:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ ping testcluster-abcd.mongodb.net</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ping: testcluster-abcd.mongodb.net: Name or service not known</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ nc -zv -w 5 testcluster-abcd.mongodb.net 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nc: getaddrinfo for host "testcluster-abcd.mongodb.net" port 27017: Name or service not known</span><br />
<br />
Neither of these work even if you actually do have Atlas connectivity configured correctly. This is because "<i>testcluster-abcd.mongodb.net</i>" is not the DNS name of a specific host endpoint. It is actually used by the MongoDB drivers and tools to dynamically lookup the DNS SRV records which have been populated for a <i>service</i> called '<i>testcluster-abcd.mongodb.net</i>'.<br />
<br />
<h3>
Useful Connectivity Diagnosis</h3>
<br />
As documented in the <a href="https://github.com/mongodb/specifications/blob/master/source/initial-dns-seedlist-discovery/initial-dns-seedlist-discovery.rst">MongoDB Drivers specification document</a> and the <a href="https://docs.mongodb.com/manual/reference/connection-string/#dns-seedlist-connection-format">MongoDB Manual</a>, a DNS SRV query is performed by the drivers/tools by prepending the text '<i><b>_mongodb._tcp.</b></i>' to the service name. Therefore, to lookup the list of real endpoints for the Atlas cluster from your laptop using the DNS <i>nslookup</i> tool, you should run:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ nslookup -q=SRV _mongodb._tcp.testcluster-abcd.mongodb.net</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Server:<span style="white-space: pre;"> </span>127.0.0.53</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Address:<span style="white-space: pre;"> </span>127.0.0.53#53</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Non-authoritative answer:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">_mongodb._tcp.testcluster-abcd.mongodb.net<span style="white-space: pre;"> </span>service = 0 0 27017 testcluster-shard-00-02-abcd.mongodb.net.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">_mongodb._tcp.testcluster-abcd.mongodb.net<span style="white-space: pre;"> </span>service = 0 0 27017 testcluster-shard-00-01-abcd.mongodb.net.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">_mongodb._tcp.testcluster-abcd.mongodb.net<span style="white-space: pre;"> </span>service = 0 0 27017 testcluster-shard-00-00-abcd.mongodb.net.</span><br />
<br />
You can see that in this case that the database service name maps to 3 endpoints (i.e. the hosts of the 3 replica set members). You can then lookup the actual IP address of any one of these endpoints if you desire:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ nslookup testcluster-shard-00-00-abcd.mongodb.net</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Server:<span style="white-space: pre;"> </span>127.0.0.53</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Address:<span style="white-space: pre;"> </span>127.0.0.53#53</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Non-authoritative answer:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">testcluster-shard-00-00-abcd.mongodb.net<span style="white-space: pre;"> </span>canonical name = ec2-35-178-15-240.eu-west-2.compute.amazonaws.com.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Name:<span style="white-space: pre;"> </span>ec2-35-178-15-240.eu-west-2.compute.amazonaws.com</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Address: 35.178.14.238</span><br />
<br />
So to now debug your connectivity issue further you can use <i>ping</i> but this time by specifying one of the underlying host server endpoints for the database cluster:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ ping -c 3 testcluster-shard-00-00-abcd.mongodb.net</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">PING ec2-35-178-15-240.eu-west-2.compute.amazonaws.com (35.178.14.238) 56(84) bytes of data.</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">64 bytes from ec2-35-178-15-240.eu-west-2.compute.amazonaws.com (35.178.14.238): icmp_seq=1 ttl=51 time=10.2 ms</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">64 bytes from ec2-35-178-15-240.eu-west-2.compute.amazonaws.com (35.178.14.238): icmp_seq=2 ttl=51 time=9.73 ms</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">64 bytes from ec2-35-178-15-240.eu-west-2.compute.amazonaws.com (35.178.14.238): icmp_seq=3 ttl=51 time=11.7 ms</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">--- ec2-35-178-15-240.eu-west-2.compute.amazonaws.com ping statistics ---</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">3 packets transmitted, 3 received, 0% packet loss, time 2002ms</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">rtt min/avg/max/mdev = 9.739/10.586/11.735/0.850 ms</span><br />
<br />
If this is successful it still doesn't necessarily mean that you can connect to the database service. The next thing to try is to see if you can actually open a socket connection to the <i>mongod</i> (or <i>mongos</i>) daemon process running on one of the endpoints, which you can achieve from your laptop using the <i>netcat</i> utility:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ nc -zv -w 5 testcluster-shard-00-00-abcd.mongodb.net 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">nc: connect to testcluster-shard-00-00-abcd.mongodb.net port 27017 (tcp) timed out: Operation now in progress</span><br />
<br />
If this doesn't connect but you are able to ping the endpoint host (as is the case in this example), it probably indicates that the IP address of your client laptop has not been added to the <a href="https://docs.atlas.mongodb.com/security-whitelist/">Atlas project's access list</a>, which is easy to remedy via the Atlas Console:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUTAuo7TIEeG3uQ0suJxrggHYBkUtHf_jtUQvPxX79SMfZ02QK3SvoTb2FD2UiL93uenxkL0eedyu2QYfRJ5oKFj3Er1GQ_hx2kFGGDcReiHAp5A215ieuuYHxbB1yPAStb5A9TBmDL30/s1600/wl.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="833" data-original-width="1438" height="370" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiUTAuo7TIEeG3uQ0suJxrggHYBkUtHf_jtUQvPxX79SMfZ02QK3SvoTb2FD2UiL93uenxkL0eedyu2QYfRJ5oKFj3Er1GQ_hx2kFGGDcReiHAp5A215ieuuYHxbB1yPAStb5A9TBmDL30/s640/wl.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
Once your laptop has been added to the access list, running <i>netcat</i> again should demonstrate that a socket connection can now be successfully made:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ nc -zv -w 5 testcluster-shard-00-00-abcd.mongodb.net 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Connection to testcluster-shard-00-00-abcd.mongodb.net 27017 port [tcp/*] succeeded!</span><br />
<br />
If this connects, then it is advisable to move on to trying to connect to the database via the Mongo Shell.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnw6j3bS0BdZ3651jzgWEfdLCqql34SNFQJgaMz8HSTVUI3Iv321aMeYbF6VkcUd_5fuRS8DkNjjfhyK5AFPasGzrH1n4trbuzzOJXB7SBjdkQXsXzG0OaKqcJW2wf5M2M1I26YmESk6U/s1600/shell.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1274" data-original-width="1600" height="505" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhnw6j3bS0BdZ3651jzgWEfdLCqql34SNFQJgaMz8HSTVUI3Iv321aMeYbF6VkcUd_5fuRS8DkNjjfhyK5AFPasGzrH1n4trbuzzOJXB7SBjdkQXsXzG0OaKqcJW2wf5M2M1I26YmESk6U/s640/shell.png" width="640" /></a></div>
<br />
In this example screenshot, the Atlas console suggests the following Mongo Shell command line to use to connect:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> mongo "mongodb+srv://testcluster-abcd.mongodb.net/test" --username main_user</span><br />
<br />
With this connection string, some of you may be thinking how does the Shell know to connect to Atlas over SSL/TLS, what replica-set name it should request and what authentication source database it should specify to locate the user's credentials?<br />
<br />
Well, in addition to querying the DNS SRV records for the service, when dynamically constructing the initial bootstrap URL for the cluster, the MongoDB drivers/tools also lookup a <a href="https://en.wikipedia.org/wiki/TXT_record">DNS TXT record</a> for the service which Atlas also populates for the deployed cluster. This TXT record contains the set of connection options, to be added as parameters to the dynamically constructed connecting string (e.g. '<i>ssl=true&replicaSet=TestCluster-shard-0&authSource=admin</i>'). You can view what these parameter settings are for a particular Atlas cluster, yourself, by running the following DNS query:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ nslookup -q=TXT testcluster-abcd.mongodb.net</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Server:<span style="white-space: pre;"> </span>127.0.0.53</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Address:<span style="white-space: pre;"> </span>127.0.0.53#53</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">Non-authoritative answer:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">testcluster-abcd.mongodb.net text = "authSource=admin&replicaSet=TestCluster-shard-0"</span><br />
<br />
Note, the default behaviour for MongoDB drivers/tools using a '<i>mongodb+srv</i>' based URL is defined as to enable SSL/TLS for the connection. As a result, '<i>ssl=true</i>' doesn't have to be included in the DNS TXT record, as shown in the example above, because the drivers/tools will automatically add this parameter to the connection string on the fly.<br />
<br />
<h3>
Summary</h3>
<br />
There's other potential causes of MongoDB Atlas connectivity issues that aren't covered in this post, but hopefully the tips highlighted here will help some of you, especially if you are diagnosing problems when using DNS SRV based service names in the connection URLs you use.<br />
<br />
<br />
<i>Song for today: Lose the Baby by <a href="https://en.wikipedia.org/wiki/Tropical_Fuck_Storm">Tropical Fuck Storm</a></i><br />
<br />Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com5tag:blogger.com,1999:blog-1304066656993695443.post-7539807974899631322019-05-11T21:30:00.003+01:002023-05-09T19:11:24.086+01:00Running a Mongo Shell Script From Within A Larger Bash Script<div><i>[EDIT May 2023: The post below was written for the legacy 'mongo' shell but has since been tested with the modern 'mongosh' shell, which behaves the same with no issues.]</i></div><div><br /></div><div>
If you have a Bash script that amongst other things needs to execute a set of multiple Mongo Shell commands together, there are a number of approaches that can be taken. This blog post contains nothing revelatory, but hopefully at least captures examples of these approaches in one single place for easy future reference. There are many situations where this is required, for example:</div>
<div>
<div>
<ul>
<li>From within a Docker container image’s <a href="https://docs.docker.com/engine/reference/builder/#entrypoint">Entrypoint</a>, running a Bash script which includes a section of Mongo Shell JavaScript code to configure a MongoDB replica-set, using <a href="https://docs.mongodb.com/manual/reference/method/rs.initiate/">rs.initiate()</a> and associated commands.</li>
<li>From within a Continuous Integration process, running a Bash script which installs a MongoDB environment in a host Operating System (OS) and then populates the new MongoDB database with some sample data, using a set of Mongo Shell <a href="https://docs.mongodb.com/manual/crud/">CRUD</a> commands</li>
<li>From within a host system’s monitoring Bash script, which, in addition to gathering some host OS metrics, invokes a set of MongoDB’s server <a href="https://docs.mongodb.com/manual/reference/method/db.serverStatus/">status</a> and <a href="https://docs.mongodb.com/manual/reference/method/db.stats/">statistics</a> commands to also capture database metrics.</li>
</ul>
</div>
<div>
The rest of this blog post shows some of the different approaches that can be taken to execute a block of Mongo Shell JavaScript code from within a larger Bash script. In these specific examples a trivial block of JavaScript code will insert 2 records into a ‘persons’ database collection, then query and print both the records belonging to the collection and then remove the 2 records from the collection.</div>
<div>
<br /></div>
<div>
It is worth noting that there is a <a href="https://docs.mongodb.com/manual/tutorial/write-scripts-for-the-mongo-shell/">difference in some of Mongo Shell’s behaviour</a> when running a block of JavaScript code in the Mongo Shell’s <b>Scripted mode</b> rather than its <b>Interactive mode</b>, including the inability to run the Shell Helper commands (e.g. unable to utilise <span style="font-family: "courier new" , "courier" , monospace;">use db</span>, <span style="font-family: "courier new" , "courier" , monospace;">show collections</span>, etc.).<br />
<br /></div>
</div>
<div>
<br /></div>
<div>
<h3>
1. EXTERNAL SCRIPT FILE</h3>
<div>
<br />
This option requires executing a separate file which contains the block of JavaScript code. First create a new JavaScript file called <span style="font-family: "courier new" , "courier" , monospace;">test.js</span> with the following content:</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db = db.getSiblingDB('testdb');</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db.persons.insertOne({'firstname': 'Sarah', 'lastname': 'Smith'});</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db.persons.insertOne({'firstname': 'John', 'lastname': 'Jones'});</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db.persons.find({}, {'_id': 0, 'firstname': 1}).forEach(printjson);</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">print(db.persons.remove({}));</span></div>
<div>
<br /></div>
<div>
Then create, make executable, and run a new Bash <span style="font-family: "courier new" , "courier" , monospace;">.sh</span> script file with the following content (this will run the Mongo Shell in <b>Scripted</b> mode):</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">#!/bin/bash</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some Bash script work first"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mongo --quiet ./test.js</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some more Bash script work afterwards"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
</div>
<div>
<br /></div>
<div>
<h3>
2. SINGLE-LINE EVAL SCRIPT</h3>
<div>
<br />
This option involves executing the Mongo Shell with its <span style="font-family: "courier new" , "courier" , monospace;">eval</span> option, passing in a single line containing each of the JavaScript commands separated by a semicolon. Create, make executable, and run a new Bash <span style="font-family: "courier new" , "courier" , monospace;">.sh</span> script file with the following content (this will run the Mongo Shell in <b>Scripted</b> mode):</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">#!/bin/bash</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some Bash script work first"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mongo --quiet --eval "db = db.getSiblingDB('testdb'); db.persons.insertOne({'firstname': 'Sarah', 'lastname': 'Smith'}); db.persons.insertOne({'firstname': 'John', 'lastname': 'Jones'}); db.persons.find({}, {'_id': 0, 'firstname': 1}).forEach(printjson); print(db.persons.remove({}));"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some more Bash script work afterwards"</span></div>
<div>
<br /></div>
<div>
<i>Note</i>: Depending on your desktop resolution, your browser may show the Mongo Shell command wrapping onto multiple lines. However, it is actually just a single line, which can be proved by copying the line into a text editor which has its ‘text wrapping’ feature disabled.<br />
<br /></div>
</div>
<div>
<br /></div>
<div>
<h3>
3. MULTI-LINE EVAL SCRIPT</h3>
<div>
<br />
This option involves executing the Mongo Shell with its <span style="font-family: "courier new" , "courier" , monospace;">eval</span> option, passing in a block of multiple lines of JavaScript code, where the start and end of the code block are delimited by single or double quotes. Create, make executable, and run a new Bash <span style="font-family: "courier new" , "courier" , monospace;">.sh</span> script file with the following content (this will run the Mongo Shell in <b>Scripted</b> mode):</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">#!/bin/bash</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some Bash script work first"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mongo --quiet --eval "</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db = db.getSiblingDB('testdb');</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db.persons.insertOne({'firstname': 'Sarah', 'lastname': 'Smith'});</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db.persons.insertOne({'firstname': 'John', 'lastname': 'Jones'});</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db.persons.find({}, {'_id': 0, 'firstname': 1}).forEach(printjson);</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> print(db.persons.remove({}));</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some more Bash script work afterwards"</span></div>
<div>
<br /></div>
<div>
<i>Note</i>: Care has to be taken to ensure that any quotes used within the JavaScript code block are single-quotes, if the Mongo Shell’s <span style="font-family: "courier new" , "courier" , monospace;">eval</span> delimiters are double-quotes, or vice versa.<br />
<br /></div>
</div>
<div>
<br /></div>
<div>
<h3>
4. MULTI-LINE SCRIPT WITH HERE-DOC</h3>
<div>
<br />
This option involves redirecting the content of a block of JavaScript multi-line code into the standard input (‘stdin’) stream of the Mongo Shell program, using a <a href="http://tldp.org/LDP/abs/html/here-docs.html">Bash Here-Document</a>. Create, make executable, and run a new Bash <span style="font-family: "courier new" , "courier" , monospace;">.sh</span> script file with the following content (unlike the other approaches this will run the Mongo Shell in <b>Interactive</b> mode):</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">#!/bin/bash</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some Bash script work first"</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">mongo --quiet <<EOF</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> show dbs;</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db = db.getSiblingDB("testdb");</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db.persons.insertOne({'firstname': 'Sarah', 'lastname': 'Smith'});</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db.persons.insertOne({'firstname': 'John', 'lastname': 'Jones'});</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> db.persons.find({}, {'_id': 0, 'firstname': 1}).forEach(printjson);</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;"> print(db.persons.remove({}));</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">EOF</span></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">echo "Doing some more Bash script work afterwards"</span></div>
<div>
<br /></div>
<div>
In this case, because the Mongo Shell is run in Interactive mode, the output of the script will be more verbose. Also, by virtue of running in Interactive mode, the <a href="https://docs.mongodb.com/manual/tutorial/write-scripts-for-the-mongo-shell/#differences-between-interactive-and-scripted-mongo">Shell Helpers</a> commands can now be used within the JavaScript code. The block of code above contains the additional line <span style="font-family: "courier new" , "courier" , monospace;">show dbs;</span> as the first line, to illustrate this. However, don’t take this example as a recommendation to use Shell Helpers in your scripts. Generally you should avoid using Shell Helpers in any of your Mongo Shell scripts, regardless of which approach you use.</div>
<div>
<br /></div>
<div>
Also, because the Mongo Shell <span style="font-family: "courier new" , "courier" , monospace;">eval</span> option is not being used, the JavaScript code can contain a mix of both single and double quotes, as illustrated by the modified line of code <span style="font-family: "courier new" , "courier" , monospace;">db = db.getSiblingDB("testdb");</span> shown above, which utilises double-quotes.</div>
</div>
<div>
<br />
<div style="-webkit-text-stroke-width: 0px; color: black; font-family: "Times New Roman"; font-size: medium; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
</div>
<br />
<h3 style="-webkit-text-stroke-width: 0px; color: black; font-family: "Times New Roman"; font-style: normal; font-variant-caps: normal; font-variant-ligatures: normal; letter-spacing: normal; orphans: 2; text-align: start; text-decoration-color: initial; text-decoration-style: initial; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px;">
Another Observation</h3>
</div>
<div>
<br />
It is worth noting that for all of these four methods, apart from the <i>External Script File</i> method, you can reference <a href="http://tldp.org/HOWTO/Bash-Prog-Intro-HOWTO-5.html">Bash environment variables</a> inline within the Mongo Shell JavaScript code (as long as double-quotes deliminate the code for the <i>eval</i> methods, rather than single-quotes). For example, from a Bash terminal if you have set a variable with the name of the database to write to...</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">export DBNAME=testdb</span></div>
</div>
<div>
<br /></div>
<div>
... you can then use the value of this environment variable from within the inline Mongo Shell JavaScript...</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">db = db.getSiblingDB('${DBNAME}');</span><br />
<br /></div>
<div>
...to factor out the database name. At face value this may not seem particularly powerful until you realise that many <i>build</i> frameworks (e.g. Docker Compose, Ansible, etc.) allow you to declare environment variables within configuration settings before invoking Bash scripts, to factor out environment specific settings.<br />
<br />
One bit of caution though, if you are using the MongoDB query operators, they include an ampersand in the syntax (e.g. '&gt', '&exists') which will need to be escaped in these scripts (e.g. '\&gt', '\&exists'). Otherwise Bash will treat each ampersand as a special control character which, in this case, will likely result in being replaced with some empty text.<br />
<br />
<br />
<h3>
Summary</h3>
<div>
<br />
The following table summarises the main differences between the four approaches to running a JavaScript block of code with the Mongo Shell, from within a larger Bash script:</div>
</div>
<div>
<br /></div>
<div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpB1aEzAmGKGq0NpPYqeDwwyPLKN5D9b2WafEHT3ZIU2ssEA50yXFhyBcjh5UyIrlgCGhodTUyV8Mozf1su9kJn3oAAF9VqJ2N0M6N_sLEsHsb0-O8aOKw-HVmRu8xikBFte4CupvwJ-0/s1600/tab.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="753" data-original-width="1600" height="300" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhpB1aEzAmGKGq0NpPYqeDwwyPLKN5D9b2WafEHT3ZIU2ssEA50yXFhyBcjh5UyIrlgCGhodTUyV8Mozf1su9kJn3oAAF9VqJ2N0M6N_sLEsHsb0-O8aOKw-HVmRu8xikBFte4CupvwJ-0/s640/tab.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
</div>
<div>
<br />
<br />
<i>Song for today: D. Feathers by <a href="https://en.wikipedia.org/wiki/Bettie_Serveert">Bettie Serveert</a></i><br />
<br /></div>
Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com2tag:blogger.com,1999:blog-1304066656993695443.post-652083507729201322018-04-13T12:15:00.002+01:002020-04-19T21:55:19.439+01:00MongoDB Graph Query Example, Inspired by Designing Data-Intensive Applications Book<h3>
<span style="font-size: large;">
Introduction</span></h3>
<br />
People who have worked with me recently are probably bored by me raving about how good this book is: <a href="https://dataintensive.net/">Designing Data-Intensive Applications</a> by <a href="https://twitter.com/martinkl">Martin Kleppmann</a> (O'Reilly, 2016). Suffice to say, if you are in IT and have any sort of interest in databases and/or data-driven applications, you should read this book. You will be richly rewarded for the effort.<br />
<br />
In the second chapter of the book ('Data Models and Query Languages'), Martin has a section called 'Graph Like Data Models' which explores 'graph use cases' where many-to-many relationships are typically modelled with tree-like structures, with indeterminate numbers of inter-connections. The book section shows how a specific 'graph problem' can be solved by using a dedicated graph database technology with associated query language (<a href="https://www.opencypher.org/">Cypher</a>) and by using an ordinary relational database with associated query language (<a href="https://en.wikipedia.org/wiki/SQL">SQL</a>). One thing that quickly becomes evident, when reading this section of the book, is how difficult it is in a relational database to model complex many-to-many relationships. This may come as a surprise to some people. However, this is consistent with something I've subconsciously learnt over 20 years of using relational databases, which is, relationships ≠ relations, in the world of <a href="https://en.wikipedia.org/wiki/Relational_database_management_system">RDBMS</a>.<br />
<br />
The graph scenario illustrated in the book shows an example of two people, Lucy and Alain, who are married to each other, who are born in different places and who now live together in a third place. For clarity, I've included the diagram from the book, below, to best illustrate the scenario (annotated with the book's details, in red, for reference).<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMyiG7BFXdJQ6loAmX-7T1FItE6D-UwiCG5lD7AOVBD236YARwhvkBP-SYK1YbKmaq2bOBc2TGzpD1cDhRGrbfFQzQu-kiwLXRUpOfSsaDO5_dv-Ec-90JwpYYdltDAjW0FG1K2MsF-Ys/s1600/ddia_book.png" imageanchor="1"><img border="0" data-original-height="605" data-original-width="1074" height="360" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMyiG7BFXdJQ6loAmX-7T1FItE6D-UwiCG5lD7AOVBD236YARwhvkBP-SYK1YbKmaq2bOBc2TGzpD1cDhRGrbfFQzQu-kiwLXRUpOfSsaDO5_dv-Ec-90JwpYYdltDAjW0FG1K2MsF-Ys/s640/ddia_book.png" width="640" /></a></div>
<br />
<br />
Throughout the book, numerous types of databases and data-stores are illustrated, compared and contrasted, including MongoDB in many places. However the book's section on graph models doesn't show how MongoDB can be used to solve the example graph scenario. Therefore, I thought I take this task on myself. Essentially, the premise is that there is a data-set of many people, with data on the place each person was born in and the place each person now lives in. Of course, any given place may be within a larger named place, which may in turn be within a larger named place, and so on, as illustrated in the diagram above. In the rest of this blog post I show one way that such data structures and relationships can be modelled in MongoDB and then leveraged by MongoDB's graph query capabilities (specifically using the <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/graphLookup/">graph lookup</a> feature of MongoDB's <a href="https://docs.mongodb.com/manual/core/aggregation-pipeline/">Aggregation Framework</a>). What will be demonstrated is how to efficiently answer the exam question posed by the book, namely: '<b><i>Find People Who Emigrated From US To Europe</i></b>'.<br />
<br />
<br />
<h3>
<span style="font-size: large;">
Solving The Book's Graph Challenge With MongoDB</span></h3>
<br />
To demonstrate the use of MongoDB's Aggregation 'graph lookup' capability to answer the question '<i>Find People Who Emigrated From US To Europe</i>', I've created the following two MongoDB collections, populated with data:<br />
<ol>
<li><b>'persons' collection</b>. Contains around one million randomly generated person records, where each person has 'born_in' and 'lives_in' attributes, which each reference a 'starting' place record in the places collection.</li>
<li><b>'places' collection</b>. Contains hierarchical geographical places data, with the graph structure of: <i>SUBDIVISIONS-->COUNTRIES-->SUBREGIONS-->CONTINENTS</i>. Note: The granularity and hierarchy of the data-set is slightly different than illustrated in the book, due to the sources of geographical data I had available to cobble together.</li>
</ol>
Similar to the book's example, amongst the many 'persons' records stored in MongoDB data-set, are the following two records relating to 'Lucy' and 'Alain'.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: '<b>Lucy Smith</b>', born_in: 'Idaho', lives_in: 'England'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: '<b>Alain Chirac</b>', born_in: 'Bourgogne-Franche-Comte', lives_in: 'England'}</span><br />
<div>
<br /></div>
Below is an excerpt of some of the records from the 'places' collection, which illustrates how a place record may refer to another place record, via its 'part_of' attribute.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">{name: '<b>England</b>', type: 'subdivision', part_of: 'United Kingdom of Great Britain and Northern Ireland'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">..</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{name: '<b>United Kingdom of Great Britain and Northern Ireland</b>', type: 'country', part_of: 'Northern Europe'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">..</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{name: '<b>Northern Europe</b>', type: 'subregion', part_of: 'Europe'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">..</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{name: '<b>Europe</b>', type: 'continent', part_of: ''}</span><br />
<div>
<br /></div>
If you want to access this data yourself and load it into the two MongoDB database collections, I've created JSON exports of both collections and made these available in a <a href="https://github.com/pkdone/GraphPersonsAndPlaces">GitHub project</a> (see the project's README for more details on how to load the data into MongoDB and then how to actually run the example's 'graph lookup' aggregation pipeline).<br />
<br />
The MongoDB aggregation pipeline I created, to process the data across these two collections and to answer the question '<b><i>Find People Who Emigrated From US To Europe'</i></b>, has the following stages:<br />
<ol>
<li><b>$graphLookup:</b> For every record in the 'persons' collection, using the person's '<b>born_in</b>' attribute, locate the matching record in the 'places' collection and then walk the chain of ancestor place records building up a hierarchy of 'born in' place names.</li>
<li><b>$match:</b> Only keep 'persons' records, where the 'born in' hierarchy of discovered place names includes '<b>United States of America</b>'.</li>
<li><b>$graphLookup:</b> For each of these remaining 'persons' records, using each person's '<b>lives_in</b>' attribute, locate the matching record in the 'places' collection and then walk the chain of ancestor place records building up a hierarchy of 'lives in' place names.</li>
<li><b>$match:</b> Only keep around the remaining 'persons' records, where the 'lives in' hierarchy of discovered place names includes '<b>Europe</b>'.</li>
<li><b>$project:</b> For the resulting records to be returned, just show the attributes 'fullname', 'born_in' and 'lives_in'.</li>
</ol>
<br />
The actual MongoDB Aggregation Pipeline for this is:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">db.persons.aggregate([</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$graphLookup: {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> from: 'places',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> startWith: '$born_in',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> connectFromField: 'part_of',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> connectToField: 'name',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> as: 'born_hierarchy'</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$match: {'born_hierarchy.name': <b><span style="color: red;">born</span></b>}},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$graphLookup: {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> from: 'places',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> startWith: '$lives_in',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> connectFromField: 'part_of',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> connectToField: 'name',</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> as: 'lives_hierarchy'</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$match: {'lives_hierarchy.name': <b><span style="color: red;">lives</span></b>}},</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> {$project: {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> _id: 0,</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> fullname: 1, </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> born_in: 1, </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> lives_in: 1, </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> }}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">])</span><br />
<br />
When this aggregation is executed, after first declaring values for the variables highlighted in red...<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">var <b><span style="color: red;">born</span></b> = 'United States of America', <b><span style="color: red;">lives</span></b> = 'Europe'</span><br />
<br />
...the following is an excerpt of the output that is returned by the aggregation:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: '<b>Lucy Smith</b>', born_in: 'Idaho', lives_in: 'England'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: 'Bobby Mc470', born_in: 'Illinois', lives_in: 'La Massana'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: 'Sandy Mc1529', born_in: 'Mississippi', lives_in: 'Karbinci'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: 'Mandy Mc2131', born_in: 'Tennessee', lives_in: 'Budapest'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: 'Gordon Mc2472', born_in: 'Texas', lives_in: 'Tyumenskaya oblast'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: 'Gertrude Mc2869', born_in: 'United States of America', lives_in: 'Planken'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">{fullname: 'Simon Mc3087', born_in: 'Indiana', lives_in: 'Ribnica'}</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">..</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">..</span><br />
<br />
On my laptop, using the data-set of a million person records, the aggregation takes about 45 seconds to complete. However, if I first define the index...<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">db.places.createIndex({name: 1})</span><br />
<br />
...and then run the aggregation, it only takes around 2 seconds to execute. This shows just how efficiently the 'graphLookup' capability is able to walk a graph of relationships, by leveraging an appropriate index.<br />
<br />
<br />
<h3>
<span style="font-size: large;">
Summary</span></h3>
<br />
I've shown the expressiveness and power of MongoDB's aggregation framework, combined with 'graphLookup' pipeline stages, to perform a query of a graph of relationships across many records. A 'graphLookup' stage is efficient as it avoids the need to develop client application logic to programmatically navigate each hop of a graph of relationships, and thus avoids the network round trip latency that a client, traversing each hop, would otherwise incur. The 'graphLookup' stage can and should leverage an index, to enable the 'tree-walk' process to be even more efficient.<br />
<br />
Although MongoDB may not be as rich in terms of the number of graph processing primitives it provides, compared with 'dedicated' graph databases, it possesses some key advantages for 'graph' use cases:<br />
<ol>
<li><i><b>Business Critical Applications</b></i>. MongoDB is designed for, and invariably deployed as a realtime operational database, with built-in high availability and enterprise security capabilities to support realtime business critical uses. Dedicated graph databases tend to be built for 'back-office' and 'offline' analytical uses, with less focus on high availability and security. If there is a need to leverage a database to respond to graph queries in realtime for applications sensitive to latency, availability and security, MongoDB is likely to be a great fit.</li>
<li><b><i>Cost of Ownership & Timeliness of Insight.</i></b> Often, there may be requirements to satisfy <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a> random realtime operations on individual data records and satisfy graph-related analysis of the data-set as a whole. Traditionally, this would require an ecosystem containing two types of database, an operational database and a graph analytical database. A set of <a href="https://en.wikipedia.org/wiki/Extract,_transform,_load">ETL</a> processes would then need to be developed to keep the duplicated data synchronised between the two databases. By combining both roles in a single MongoDB distributed database, with appropriate <a href="https://docs.mongodb.com/manual/core/workload-isolation/">workload isolation</a>, the financial cost of this complexity can be greatly reduced, due to a far simpler deployment. Additionally, and as a consequence, there will be no lag that arises when keeping one copy of data in one system, up to date with the other copy of the data in another system. Rather than operating on stale data, the graph analytical workloads operate on current data to provide more accurate business insight.</li>
</ol>
<br />
<br />
<i>Song for today: Cosmonauts by <a href="https://en.wikipedia.org/wiki/Quicksand_(American_band)">Quicksand</a></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com3tag:blogger.com,1999:blog-1304066656993695443.post-50693007086414504392018-02-04T09:33:00.000+00:002020-07-04T13:54:54.858+01:00Run MongoDB Aggregation Facets In Parallel For Faster Insight<h2>
Introduction</h2>
MongoDB version 3.4 introduced a new Aggregation stage, <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/facet/">$facet</a>, to enable developers to "<i>create multi-faceted aggregations which characterize data across multiple dimensions, or facets, within a single aggregation stage</i>". For example, you may run a clothes retail website and use this aggregation capability to characterise the choices across a set of filtered products, by the following facets, simultaneously:<br />
<ol>
<li>Size (e.g. S, M, L,)</li>
<li>Full-price vs On-offer</li>
<li>Brand (e.g. Nike, Adidas)</li>
<li>Average Rating (e.g. 1 - 5 stars)</li>
</ol>
In this blog post, I explore a way in which the response times for faceted aggregation workloads can be reduced, by leveraging parallel processing.<br />
<br />
<h2>
Parallelising Aggregated Facets</h2>
If an aggregation pipeline declares the use of the<i> $facet</i> stage, it defines multiple facets where each facet is a "sub-pipeline" containing a series actions specific to its facet. When a faceted aggregation is executed, the result of the aggregation will contain the combined output of all the facet's sub-pipelines. Below is an example of the structure of a "faceted" aggregation pipeline.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifyc-yn2wpJy2FTcqKk2hYALg6mAZrITRPBuCYruhFo9uVH_K-lT4nB8Ab056xi7ft8i8qF0exhuryI21YBneeKjvoc5VZmqnU35XXAP7IR9yRPvn_yK05AnzwzKVevLHgz9oQyxYOrC8/s1600/bg.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="500" data-original-width="929" height="172" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEifyc-yn2wpJy2FTcqKk2hYALg6mAZrITRPBuCYruhFo9uVH_K-lT4nB8Ab056xi7ft8i8qF0exhuryI21YBneeKjvoc5VZmqnU35XXAP7IR9yRPvn_yK05AnzwzKVevLHgz9oQyxYOrC8/s320/bg.png" width="320" /></a></div>
In this example, there are two facets or dimensions, each containing a sub-pipeline. Each sub-pipeline is essentially a regular aggregation pipeline, with just a <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/facet/#behavior">small handful of restrictions</a> on what it can contain. Notable amongst these restrictions is the fact that the sub-pipeline cannot contain a <i>$facet</i> stage. Therefore you can't use this to go infinite levels deep!<br />
<br />
The ability to define an aggregation containing different facets is not just useful for responding to online user interactions, in realtime. It is also useful for activities such as running a business's "internal reporting" workloads, where a report may need to analyse a full data set, and then summarise the data in different dimensions.<br />
<div>
<br /></div>
A data set that I've been playing around with recently, is the publically available "MOT UK Annual Vehicle Test Result Data". An <a href="https://en.wikipedia.org/wiki/MOT_test">MOT</a> is a UK annual safety check on a vehicle, and is mandatory for all cars over 3 years old. The UK government makes the <a href="https://data.gov.uk/dataset/anonymised_mot_test">data available to download</a>, in anonymised form, for anyone to consume, through its <a href="http://data.gov.uk/">data.gov.uk</a> platform. It's a rich data set, providing a lot of insight into the characteristics of cars that UK residents have been driving over the last ten years or so. As a result, it's a good data set for me to use to explore faceted aggregations.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgP0n8oVoBQlZfi0gAKSiTj5_4AP5vGSOuyAn_Zil3ixxhODTKSdMnMHN0Y3zSBYp_ZrZg_UbdUcyfuoNQPBAsy_DPsDmL7L9yvDP2DhDbgnRghI6OzKG6KtRP0bIi0knNCpOFr67Uixy4/s1600/mot.png" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" data-original-height="1068" data-original-width="1600" height="425" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgP0n8oVoBQlZfi0gAKSiTj5_4AP5vGSOuyAn_Zil3ixxhODTKSdMnMHN0Y3zSBYp_ZrZg_UbdUcyfuoNQPBAsy_DPsDmL7L9yvDP2DhDbgnRghI6OzKG6KtRP0bIi0knNCpOFr67Uixy4/s640/mot.png" width="640" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">2014-2016 MOT car data loaded into MongoDB - displayed in MongoDB Compass</td></tr>
</tbody></table>
<br />
To analyse the car data, I created a <a href="https://github.com/">GitHub</a> project at <a href="https://github.com/pkdone/mongo-uk-car-data">mongo-uk-car-data</a>. This contains some Python scripts to load the data from the MOT data CSV files into MongoDB, and to perform various analytics on the data set using MongoDB's <a href="https://docs.mongodb.com/manual/aggregation/">Aggregation Framework</a>. One of the Python scripts I created, <a href="https://github.com/pkdone/mongo-uk-car-data/blob/master/mdb-mot-agg-cars-facets.py">mdb-mot-agg-cars-facets.py</a>, uses a <i>$facet</i> stage to aggregate together summary information, in the following three different dimensions:<br />
<ol>
<li>Analyse the different car makes/brands (e.g. Ford, Vauxhall) and categorise them into a range of "buckets", based on how many different unique models each car make has.</li>
<li>Summarise the amount of tested cars that fall into each fuel type category (e.g. Petrol, Diesel, Electric). <i>Note</i>: "Petrol" is equivalent to "Gas" for my American friends, I believe.</li>
<li>List the top 5 car makes/brands from the car tests, showing how many cars there are for each car make, plus each car make's most popular and least popular models.</li>
</ol>
The following shows the result of the aggregation when run against the data set for years 2014-16 (a data set of approximately 113 million records).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5ZI_ph8gbPiK8BiZf3ec-fa1-LdOQg-RNg2g9tmYKRUCwGcxbJYuCIk7JaXMPbeDVozi_j_L4WVcd2UXaA_mjubrblbyQsKaW1ldXvJ_XJqH35wNfPA3BmggNBMHyOqwkqWPx8YdvZN8/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="1600" data-original-width="948" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5ZI_ph8gbPiK8BiZf3ec-fa1-LdOQg-RNg2g9tmYKRUCwGcxbJYuCIk7JaXMPbeDVozi_j_L4WVcd2UXaA_mjubrblbyQsKaW1ldXvJ_XJqH35wNfPA3BmggNBMHyOqwkqWPx8YdvZN8/s640/1.png" width="377" /></a></div>
<br />
When I ran this test on my Linux Laptop (hosting both a <i>mongod</i> server and the test Python script), the aggregation was <b>completed in about 5:20 minutes</b>. In my test Python client code, a faceted pipeline is constructed, which uses the <a href="https://api.mongodb.com/python/current/">PyMongo Driver</a> to send the <a href="https://docs.mongodb.com/manual/reference/method/db.collection.aggregate/">aggregation command</a> and pipeline payload to the MongoDB database. Significantly, the database's Aggregation framework processes each facet's sub-pipeline serially. Therefore, for example, if the first facet takes 5 minutes to process, the second takes 2 minutes and the third facet takes 10 minutes, the client application will only receive a full response in just over 17 minutes.<br />
<br />
It occurred to me that there was a way to potentially speed up the execution time of this analytics job. At the point of invoking <span style="background-color: white; color: #24292e; font-family: "courier new" , "courier" , monospace; white-space: pre;">collection.aggregate(pipeline)</span><i> </i>in the <a href="https://github.com/pkdone/mongo-uk-car-data/blob/master/mdb-mot-agg-cars-facets.py">mdb-mot-agg-cars-facets.py</a> script, a custom function could be invoked instead, that internally breaks the pipeline up into separate pipelines, one for each facet. The function could then send each facet as a separate aggregation command, in parallel, to the MongoDB database to be processed, before merging the results into one, and returning it. Functionally, the behaviour of this code and the content of the response would be identical, but I hoped the response time would be significantly less. So I replaced the line of code that directly invoked the MongoDB aggregation command, <span style="background-color: white; color: #24292e; font-family: "courier new" , "courier" , monospace; white-space: pre;">collection.aggregate(pipeline)</span>, with a call to my new function <span style="font-family: "courier new" , "courier" , monospace;"><span style="background-color: white; color: #24292e; white-space: pre;">aggregate_facets_in_parallel(collection, pipeline)</span></span>, instead. The implementation of this function can be seen in the Python file <a href="https://github.com/pkdone/mongo-uk-car-data/blob/master/parallel_facets_agg.py">parallel_facets_agg.py</a>. The function uses Python's <i>multiprocessing.pool.ThreadPool</i> library to send each facet's sub-pipeline in a separate client thread and waits for all parallel aggregations to complete before returning the combined result.<br />
<br />
This time, when I ran the test Python script against the same data set, I received the exact same result, but in a <b>time of just 3:30 minutes</b> (versus the original time of 5:20 minutes). This is not a bad speed up! :-D<br />
<br />
<h2>
Some Observations</h2>
Some people may look at this and ask why, given that there were 3 facets, the aggregation didn't respond in just one third of the original time (i.e. in around 1:47 minutes). Well, there are many reasons, including:<br />
<ol>
<li>This would assume each separate facet sub-pipeline takes the same amount of time to execute, which is highly unlikely. The different facet sub-pipelines will each have different complexities and processing requirements. The overall response time cannot be any faster than the slowest of the 3 facet sub-pipelines.</li>
<li>Just because the client code spawns 3 "concurrent" threads, it doesn't mean that these 3 threads are actually running completely in parallel. For example, my laptop has 2 CPU cores, which would be a cause of some resource contention. There will of course be many other potential causes of resource contention, such as multiple threads competing to retrieve different data from the same hard drive.</li>
<li>For my simple tests, the client test Python script is running on the same machine as the MongoDB database, and thus will consume some of the compute capacity (albeit, for these tests, it will mostly just be blocking and waiting).</li>
<li>In most real world cases (but not for my simple tests here), there may also be other workloads being processed by the MongoDB database simultaneously, consuming significant portions of the shared compute resources.</li>
</ol>
The other question people may ask is, if this is so simple, why doesn't the MongoDB server implement such parallelism itself for processing the different sections of an aggregation <i>$facet</i> stage. There are at least two reasons why this is not the case in MongoDB:<br />
<ol>
<li>My test scenario places some restrictions on the aggregation pipeline as a whole. Specifically, the top level pipeline must only contain one stage (the <i>$facet</i> stage) and my custom function throws an exception if this is not the case. This is fine and quite common where the workload is an analytical workload that needs to "full table scan" most or all of a data set. However, in the original retail example at the top of the post, the likelihood is that there would need to be a <a href="https://docs.mongodb.com/manual/reference/operator/aggregation/match/">$match</a> stage, before the <i>$facet</i> stage, to first restrict the multi-faceted clothes classifications based on a filter that the user has entered (e.g. product name contains "Black Trainers"). Thus, it may well be more efficient to perform the <i>$match</i> just once, to reduce the set of data to work with, before having this data passed on to a <i>$facet</i> stage. The workaround would be to duplicate the <i>$match</i> as the first stage of each of the <i>$facet</i> sub-pipelines, which could well turn out to be slower, as the same work would be repeated.</li>
<li>For the most part, MongoDB's runtime architecture does not attempt to divide and process an individual client request's <a href="https://en.wikipedia.org/wiki/Create,_read,_update_and_delete">CRUD</a> operations into parallel chunks, and instead processes the elements of an individual request serially. One reason why this is a good thing, is that typically a MongoDB database will be processing many requests in parallel from many clients. If one particular request was allowed to dominate the system's resources, by being parallelised "server-side" for a "burst of time", this may adversely affect other requests and cause the database to exhibit inconsistent performance as a whole. MongoDB's architecture generally encourages a fairer share of resources, spread across all clients' requests. For this reason, you may want to carefully consider how much you use the "client-side parallelism" tip in this blog post, in order to avoid abusing this "fair share" trust.</li>
</ol>
<div>
<br /></div>
<h2>
Summary</h2>
I've shown an example in this blog post of how multi-faceted MongoDB aggregations can be sped up by encouraging parallelism, from the client application's perspective. The choice of Python to implement this was fairly arbitrary. I could have implemented it in any programming languages that there is a <a href="https://docs.mongodb.com/ecosystem/drivers/">MongoDB Driver</a> for, using the appropriate multi-threading libraries for that language. The parallelism benefits discussed in this post are really aimed at analytics type workloads that need to process a whole data-set to produce multi-faceted insight. This is in contrast to the sorts of workloads that would first match a far smaller subset of records, against an index, before then aggregating on the small data subset.<br />
<div>
<br />
<br /></div>
<br />
<i>Song for today: Bella Muerte by <a href="https://en.wikipedia.org/wiki/This_Patch_of_Sky">The Patch of Sky</a></i><br />
<br />Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com1tag:blogger.com,1999:blog-1304066656993695443.post-15893046143996173872017-07-13T10:23:00.000+01:002020-06-24T10:39:24.283+01:00Deploying a MongoDB Sharded Cluster using Kubernetes StatefulSets on GKE<i>[Part 4 in a series of posts about running MongoDB on Kubernetes, with the Google </i><i>Kubernetes</i><i> Engine (GKE). For this post, a <b>newer</b> GitHub project <a href="https://github.com/pkdone/gke-mongodb-shards-demo">gke-mongodb-shards-demo</a> has been created to provide an example of a scripted deployment for a Sharded cluster specifically. This gke-mongodb-shards-demo project also incorporates the conclusions from the earlier posts in the series. Also see: <a href="http://k8smongodb.net/">http://k8smongodb.net/</a>]</i><br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Introduction</span></h2>
In the previous posts of my blog series (<a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">1</a>, <a href="http://pauldone.blogspot.co.uk/2017/06/mongodb-kubernetes-production-settings.html">2</a>, <a href="http://pauldone.blogspot.co.uk/2017/06/enterprise-mongodb-on-kubernetes.html">3</a>), I focused on deploying a MongoDB Replica Set in GKE's Kubernetes environment. A MongoDB <a href="https://docs.mongodb.com/manual/replication/">Replica Set</a> provides data redundancy and high availability, and is the basic building block for any mission critical deployment of MongoDB. In this post, I now focus on the deployment of a MongoDB Sharded Cluster, within the GKE Kubernetes environment. A <a href="https://docs.mongodb.com/manual/sharding/">Sharded Cluster</a> enables the database to be scaled out over time, to meet increasing throughput and data volume demands. Even for a Sharded Cluster, the recommendations from my previous posts are still applicable. This is because each Shard is a Replica Set, to ensure that the deployment exhibits high availability, in addition to scalability.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Deployment Process</span></h2>
My <a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">first blog post</a> on MongoDB & Kubernetes observed that, although Kubernetes is a powerful tool for provisioning and orchestrating sets of related containers (both stateless and now stateful), it is not a solution that caters for every required type of orchestration task. These tasks are invariably technology specific and need to operate below or above the “containers layer”. The example I gave in my <a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">first post</a>, concerning the correct management of a MongoDB Replica Set's configuration, clearly demonstrates this point.<br />
<br />
In the modern <a href="https://en.wikipedia.org/wiki/Infrastructure_as_Code">Infrastructure as Code</a> paradigm, before orchestrating containers using something like Kubernetes, other tools are first required to provision infrastructure/<a href="https://en.wikipedia.org/wiki/Cloud_computing#Infrastructure_as_a_service_.28IaaS.29">IaaS</a> artefacts such as Compute, Storage and Networking. You can see a clear example of this in the <a href="https://github.com/pkdone/gke-mongodb-demo/blob/master/scripts/generate.sh">provisioning script used in my first post</a>, showing non-Kubernetes commands ("<i>gcloud</i>"), which are specific to the Google's Compute Platform (GCP), being used first, to provision storage disks. Once containers have been provisioned by a tool like Kubernetes, higher level configuration tasks, such as data loading, system user identity provisioning, secure network modification for service exposure and many other "final bootstrap" tasks, will also often need to be scripted.<br />
<br />
With the requirement here to deploy a MongoDB Sharded Cluster, the distinction between container orchestration tasks, lower level infrastructure/IaaS provisioning tasks and higher level technology-specific orchestration tasks, becomes even more apparent...<br />
<br />
For a MongoDB Sharded Cluster on GKE, the following categories of tasks must be implemented:<br />
<br />
<b>Infrastructure Level </b>(using Google's "<b>gcloud</b>" tool)<br />
<ol>
<li>Create 3 VM instances</li>
<li>Create storage disks of various sizes, for containers to attach to</li>
</ol>
<b>Container Level </b> (using Kubernetes' "<b>kubectl</b>" tool)<br />
<ol>
<li>Provision 3 "<a href="https://github.com/kubernetes/contrib/tree/master/startup-script">startup-script</a>" containers using a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">DaemonSet</a>, to enable the XFS filesystem to be used and to disable <a href="https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/">Huge Pages</a></li>
<li>Provision 3 "mongod" containers using a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSet</a>, ready to be used as members of the <a href="https://docs.mongodb.com/manual/core/sharded-cluster-config-servers/">Config Server Replica Set</a> to host the <a href="https://docs.mongodb.com/manual/reference/config-database/">ConfigDB</a></li>
<li>Provision 3 separate Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSets</a>, one per Shard, where each StatefulSet is composed of 3 "mongod" containers ready to be used as members of the Shard's Replica Set</li>
<li>Provision 2 "mongos" containers using a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a>, ready to be used for managing and routing client access to the Sharded database</li>
</ol>
<b>Database Level</b> (using MongoDB's "<b>mongo shell</b>" tool)<br />
<ol>
<li>For the Config Servers, run the <a href="https://docs.mongodb.com/manual/reference/method/rs.initiate/">initialisation command to form a Replica Set</a></li>
<li>For each of the 3 Shards (composed of 3 mongod processes), run the <a href="https://docs.mongodb.com/manual/reference/method/rs.initiate/">initialisation command to form a Replica Set</a></li>
<li>Connecting to one of the Mongos instances, run the <a href="https://docs.mongodb.com/manual/reference/method/sh.addShard/">addShard command</a>, 3 times, once for each of the Shards, to enable the Sharded Cluster to be fully assembled</li>
<li>Connecting to one of the Mongos instances, under <a href="https://docs.mongodb.com/manual/core/security-users/#localhost-exception">localhost exception</a> conditions, create a database administrator user to apply to the cluster as a whole</li>
</ol>
<br />
The quantities of resources highlighted above are specific to my example deployment, but the types and order of provisioning steps apply regardless of deployment size.<br />
<br />
In the accompanying <a href="https://github.com/pkdone/gke-mongodb-shards-demo">example project</a>, for brevity and clarity, I use a <a href="https://github.com/pkdone/gke-mongodb-shards-demo/blob/master/scripts/generate.sh">simple Bash shell script</a> to wire together these three different categories of tasks. In reality, most organisations would use more specialised automation software, such as <a href="https://en.wikipedia.org/wiki/Ansible_(software)">Ansible</a>, <a href="https://en.wikipedia.org/wiki/Puppet_(software)">Puppet</a>, or <a href="https://en.wikipedia.org/wiki/Chef_(software)">Chef</a>, for example, to glue all such steps together.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Kubernetes Controllers & Pods</span></h2>
The Kubernetes StatefulSet definitions for the MongoDB Config Server "mongod" containers and the Shard member "mongod" containers hardly differ from those described in my <a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">first blog post</a>.<br />
<br />
Below is an excerpt of the StatefulSet definition for each <b>Config Server mongod container</b> (shard specific addition highlighted in bold):<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-configdb-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> image: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "mongod"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--port"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "27017"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--bind_ip"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "0.0.0.0"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--wiredTigerCacheSizeGB"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "0.25"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--configsvr"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--replSet"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "ConfigDBRepSet"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--auth"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--clusterAuthMode"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "/etc/secrets-volume/internal-auth-mongodb-keyfile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--setParameter"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "authenticationMechanisms=SCRAM-SHA-1"</span><br />
<br />
Below is an excerpt of the StatefulSet definition for each <b>Shard member mongod container</b> (shard specific addition highlighted in bold):<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-shard1-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> image: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "mongod"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--port"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "27017"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--bind_ip"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "0.0.0.0"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--wiredTigerCacheSizeGB"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "0.25"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--shardsvr"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--replSet"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "Shard1RepSet"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--auth"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--clusterAuthMode"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "/etc/secrets-volume/internal-auth-mongodb-keyfile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--setParameter"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "authenticationMechanisms=SCRAM-SHA-1"</span><br />
<br />
For the Shard's container definition, the name of the specific Shard's Replica Set is declared. This Shard definition will result in 3 mongod replica containers being created for the Shard Replica Set, called "Shard1RepSet". Two additional and similar StatefulSet resources also have to be defined, to represent the second Shard ("Shard2RepSet") and third Shard ("Shard3RepSet") too.<br />
<br />
To provision the Mongos Routers, a StatefulSet is not used. This is because neither persistent storage nor a fixed network hostname are required. Mongos Routers are stateless and, to a degree, ephemeral. Instead, a Kubernetes <a href="https://kubernetes.io/docs/concepts/workloads/controllers/deployment/">Deployment</a> resource is defined, which is the Kubernetes preferred approach for stateless services. Below is the Deployment definition for the <b>Router mongos container</b>:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: apps/v1beta1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">kind: Deployment</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: mongos</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> replicas: <b>2</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> template:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> volumes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: secrets-volume</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> secret:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> secretName: shared-bootstrap-data</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> defaultMode: 256</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongos-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> image: <b>mongo</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "numactl"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--interleave=all"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "<b>mongos</b>"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--port"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "27017"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--bind_ip"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "0.0.0.0"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "<b>--configdb</b>"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "<b>ConfigDBRepSet/mongod-configdb-0.mongodb-configdb-service.default.svc.cluster.local:27017,mongod-configdb-1.mongodb-configdb-service.default.svc.cluster.local:27017,mongod-configdb-2.mongodb-configdb-service.default.svc.cluster.local:27017</b>"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--clusterAuthMode"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "/etc/secrets-volume/internal-auth-mongodb-keyfile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--setParameter"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "authenticationMechanisms=SCRAM-SHA-1"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ports:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - containerPort: 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> volumeMounts:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: secrets-volume</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readOnly: true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> mountPath: /etc/secrets-volume</span><br />
<div>
<br /></div>
The actual structure of this resource definition is not a radical departure from the definition of a StatefulSet. A different command has been declared ("mongos", rather than "mongod"), but the same base container image has been referenced (<a href="https://hub.docker.com/_/mongo/">mongo image from Docker Hub</a>). For the "mongos" container, it is still important to enable authentication and to reference the generated cluster key file. Specific to the "mongos" parameter list is "<i>--configdb</i>", to specify the URL of the "ConfigDB" (the 3-node Config Server Replica Set). This URL will be connected to by each instance of the mongos router, to enable discovery of the Shards in the cluster and to determine which Shard holds specific ranges of the stored data. Due to the fact that the "mongod" containers, used to host the "ConfigDB", are deployed as a StatefulSet, their hostnames remain constant. As a result, a fixed URL can be defined in the "mongos" container resource definition, as shown above.<br />
<br />
Once the <a href="https://github.com/pkdone/gke-mongodb-shards-demo/blob/master/scripts/generate.sh">"generate" shell script</a> in the <a href="https://github.com/pkdone/gke-mongodb-shards-demo">example project</a> has been executed, and all the Infrastructure, Kubernetes and Database provisioning steps have successfully completed, 17 different <a href="https://kubernetes.io/docs/concepts/workloads/pods/pod/">Pods</a> will be running, each hosting one container. Below is a screenshot showing the results from running the "kubectl" command, to list all the running Kubernetes Pods:<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgQLxkuvL9wgNMB-YIEubkUioNcPlqv1E3BkyROBBBM73Ag48YVOZIXgf0amn_7DYfo8m6M7H6V110EROkZdBODXaDtqNF2bx6YCBewc2jPLQrHam5ohIvJfSJfJ_VbCoHjnZXT_aGCQE/s1600/k1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="827" data-original-width="1263" height="418" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgQLxkuvL9wgNMB-YIEubkUioNcPlqv1E3BkyROBBBM73Ag48YVOZIXgf0amn_7DYfo8m6M7H6V110EROkZdBODXaDtqNF2bx6YCBewc2jPLQrHam5ohIvJfSJfJ_VbCoHjnZXT_aGCQE/s640/k1.png" width="640" /></a></div>
This tallies up with what Kubernetes was asked to be deployed, namely:<br />
<ul>
<li>3x DaemonSet startup-script containers (one per host machine)</li>
<li>3x Config Server mongod containers</li>
<li>3x Shards, each composed of 3 replica mongod containers</li>
<li>2x Router mongos containers</li>
</ul>
Only the "mongod" containers for the Config Servers and the Shard Replicas have fixed and deterministic names, because these were deployed as Kubernetes StatefulSets. The names of the other containers reflect the fact that they are regarded as stateless, disposable and trivial to re-create, on-demand, by Kubernetes.<br />
<br />
With the full deployment generated and running, it is a straight forward process to connect to the sharded cluster to check its status. The screenshot below shows the use of the "kubectl" command to open a Bash shell connected to the first "mongos" container. From the Bash shell, the "mongo shell" has been opened, connecting to the local "mongos" process running in the same container. Before the command to view the status of the Sharded cluster has been run, the database has first been authenticated with, using the "admin" user.<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlh8v0GlzRgj0PV0yhzsh1x7gPZTp_8DW7HjFTj7ae5Jzepv0O-Y2U2D4wLViltTwL1jp_cLpvpM7GUBHtAYJ6fmIv1Mp5fkn5ijO2XubYEO6KK7L9ErxDozInW8sHBkLBT63rhri6_FU/s1600/k2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="992" data-original-width="1600" height="396" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhlh8v0GlzRgj0PV0yhzsh1x7gPZTp_8DW7HjFTj7ae5Jzepv0O-Y2U2D4wLViltTwL1jp_cLpvpM7GUBHtAYJ6fmIv1Mp5fkn5ijO2XubYEO6KK7L9ErxDozInW8sHBkLBT63rhri6_FU/s640/k2.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
The output of the status command shows that the URLs of all three Shards that have been defined (each is a MongoDB Replica Set). Again, these URLs remain fixed by virtue of the use of Kubernetes StatefulSets for the "mongod" containers that compose each Shard.<br />
<br />
<b style="background-color: white; color: #444444; font-family: Arial, Tahoma, Helvetica, FreeSans, sans-serif; font-size: 13px;">UPDATE 02-Jan-2018: </b><i style="background-color: white;"><span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13px;">Since writing this blog post, I realised that because the mongos routers ideally require stable hostnames, to be easily referenceable from the app tier, the mongos router containers should also be declared and deployed as a Kubernetes StatefulSet and Service, rather than a Kubernetes Deployment. The </span></span></i><span style="color: #444444; font-family: "arial" , "tahoma" , "helvetica" , "freesans" , sans-serif;"><span style="font-size: 13px;"><i>GitHub project <a href="https://github.com/pkdone/gke-mongodb-shards-demo">gke-mongodb-shards-demo</a>, associated with this blog post, has been changed to reflect this.</i></span></span><br />
<h2>
<span style="font-size: x-large;"><br /></span></h2>
<h2>
<span style="font-size: x-large;">Summary</span></h2>
In this blog post I’ve shown how to deploy a MongoDB Sharded Cluster using Kubernetes with the Google Kubernetes Engine. I've mainly focused on the high level considerations for such deployments, rather than listing every specific resource and step required (for that level of detail, please view the <a href="https://github.com/pkdone/gke-mongodb-shards-demo">accompanying GitHub project</a>). What this study does reinforce, is that a single container orchestration framework (Kubernetes in this case) does not cater for every step required to provision and deploy a highly available and scalable database, such as MongoDB. I believe that this would be true for any non-trivial and mission critical distributed application or set of services. However, I don't see this is a bad thing. Personally, I value flexibility and choice, and having the ability to use the right tool for the right job. In my opinion, a container orchestration framework that tries to cater for things beyond its obvious remit would be the wrong framework. A framework that attempts to be all things to all people, would end up diluting its own value, down to the lowest common denominator. At the very least, it would become far too prescriptive and restrictive. I feel that Kubernetes, in its current state, strikes a suitable and measured balance, and with its StatefulSets capability, provides a good home for MongoDB deployments.<br />
<br />
<br />
<i>Song for today: Three Days by <a href="https://en.wikipedia.org/wiki/Jane%27s_Addiction">Jane's Addiction</a></i><br />
<br />Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com12tag:blogger.com,1999:blog-1304066656993695443.post-39217342561913053182017-06-30T22:58:00.000+01:002020-02-13T19:59:52.982+00:00Using the Enterprise Version of MongoDB on GKE Kubernetes<i>[Part 3 in a series of posts about running MongoDB on Kubernetes, with the Google </i><i>Kubernetes</i><i> Engine (GKE). See the GitHub project <a href="https://github.com/pkdone/gke-mongodb-demo">gke-mongodb-demo</a> for an example scripted deployment of MongoDB to GKE, that you can easily try yourself. The gke-mongodb-demo project combines the conclusions from all the posts in this series so far. </i><i>Also see: <a href="http://k8smongodb.net/">http://k8smongodb.net/</a>]</i><br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Introduction</span></h2>
In the previous two posts of my blog series (<a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">1</a>, <a href="http://pauldone.blogspot.co.uk/2017/06/mongodb-kubernetes-production-settings.html">2</a>) about running MongoDB on GKE's Kubernetes environment, I showed how to ensure a MongoDB Replica Set is secure by default, resilient to system failures and how to ensure various best practice "production" environment settings are in place. In those examples, the community version of the MongoDB binaries were used. In this blog post I show how the enterprise version of MongoDB can be utilised, instead.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Referencing a Docker Image for Use in Kubernetes </span></h2>
For the earlier two blog post examples, a pre-built "mongo" Docker image was "pulled" by Kubernetes, from Docker Hub's <a href="https://hub.docker.com/_/mongo/">"official" MongoDB repository</a>. Below is an excerpt of the Kubernetes StatefulSet definition, that shows how this <a href="https://kubernetes.io/docs/concepts/containers/images/">image was referenced</a> (highlighted in bold):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>image: mongo</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "mongod"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<br />
By default, if no additional metadata is provided, Kubernetes will look in the <a href="https://hub.docker.com/explore/">Docker Hub repository</a> for the image with the given name. Other repositories such as Google's Container Registry, AWS's EC2 Container Registry, Azure's Container Registry, or any private repository, can be used instead. <br />
<br />
It is worth clarifying what is meant by the "official" MongoDB repository. This is a set of images that are "official" from Docker Hub's perspective because Docker Hub manages how the images are composed and built. They are not, however, official releases from MongoDB Inc.'s perspective. When the Docker Hub project builds an image, in addition to sourcing the "mongod" binary from MongoDB Inc's website, other components like the underlying Debian OS, plus various custom scripts, are generated into the image too.<br />
<br />
At the time of this blog post, Docker Hub currently provides images for MongoDB community versions 3.0, 3.2, 3.4 and 3.5 (unstable). The Docker Hub repository only contains images for the <a href="https://www.mongodb.com/download-center#community">community version</a> of MongoDB and not the <a href="https://www.mongodb.com/download-center#enterprise">enterprise version</a>.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Building a Docker Image Using the MongoDB Enterprise binaries </span></h2>
You can build a Docker image to run "mongod" in any way you want, using your own custom <a href="https://docs.docker.com/engine/reference/builder/">Dockerfile</a> to define how the image should be generated. The Docker manual even provides a tutorial for creating a custom Docker image specifically for MongoDB, called <a href="https://docs.docker.com/engine/examples/mongodb/">Dockerize MongoDB</a>. That process can be followed as the basis for building an image which pulls down and uses the enterprise version of MongoDB, rather than the community version.<br />
<br />
However, to generate an image containing the enterprise MongoDB version, it isn't actually necessary to create your own custom Dockerfile. This is because, a few weeks ago, I created a pull request to add support for <a href="https://github.com/docker-library/mongo/pull/185">building MongoDB enterprise based images</a>, as part of the normal <a href="https://github.com/docker-library/mongo">"mongo" GitHub project</a> that is used to generate the "official" Docker Hub "mongo" images. The GitHub project owner accepted and merged this enhancement a few days later. My enhancement essentially allows the project's Dockerfile, when used by Docker's build tool, to instruct that the enterprise MongoDB binaries should be downloaded into the image, rather than the community ones. This is achieved by providing some "build arguments" to the Docker build process, which I will detail further below. This doesn't mean the <a href="https://hub.docker.com/_/mongo/">Docker Hub's "official repository" for MongoDB</a> now contains pre-generated "enterprise mongod" images ready to use. It just means you can use the source Github project directly, without any changes to the project's Dockerfile, to generate your own image, containing the enterprise binary.<br />
<br />
To build the Docker "mongo" image, that pulls in the enterprise version of MongoDB, follow these steps:<br />
<br />
<b>1.</b> <a href="https://docs.docker.com/engine/installation/">Install Docker</a> locally on your own workstation/laptop.<br />
<br class="Apple-interchange-newline" />
<b>2. </b>Download the source files for the <a href="https://github.com/docker-library/mongo">Docker Hub "mongo" project</a> (on the project page, click the "Clone or download" green button, then click the "Download zip" link and once downloaded, unpack into a new local folder).<br />
<br class="Apple-interchange-newline" />
<b>3. </b>From a command line shell, in the project folder, change directory to the sub-folder of the major version you want to build (e.g. "3.4"), and run:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;"> $ docker build -t pkdone/mongo-ent:3.4 --build-arg MONGO_PACKAGE=mongodb-enterprise --build-arg MONGO_REPO=repo.mongodb.com .</span><br />
<br />
This tells Docker to build an image using the Dockerfile in the current directory (".") with the resulting image <a href="https://docs.docker.com/engine/reference/commandline/images/">name</a> being "<i>pkdone/mongo-ent</i>" and <a href="https://docs.docker.com/engine/reference/commandline/tag/">tag</a> being "<i>:3.4</i>". By convention, the image name is prefixed by the author's username, which in my case is "<i>pkdone</i>" (obviously this should be replaced by a different prefix, for whoever follows these steps).<br />
<br />
The two new "<i>--build-arg</i>" parameters, "<i>MONGO_PACKAGE</i>" and "<i>MONGO_REP</i>" are passed to the <a href="https://github.com/docker-library/mongo/blob/master/3.4/Dockerfile">"mongo" Dockerfile</a>. The version of the Dockerfile with my enhancements uses these two parameters to locate where to download the specific type of MongoDB binary from. In this case, the values specified mean that the enterprise binary is pulled down into the generated image. If no "build args" are specified, the community version of MongoDB is used, by default.<br />
<br />
<b>Important note:</b> When running the "docker build" command above, because the enterprise version of MongoDB will be downloaded, it will mean you are implicitly accepting MongoDB Inc's associated commercial licence terms.<br />
<br />
Once the image is generated, you can also quickly test the image by running it in a Docker container on your local machine (as just a "mongod" single instance):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ docker run -d --name mongo -t pkdone/mongo-ent:3.4</span><br />
<br />
To be sure this is running properly, connect to the container using a shell, check if the "mongod" process is running and check that the Mongo Shell can connect to the containerised "mongod" process.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ docker exec -it mongo bash</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ ps -aux</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo</span><br />
<br />
The output of the Shell should include the prompt "<i>MongoDB Enterprise ></i>" which shows that the database is using the enterprise version of MongoDB. Exit out of the Mongo Shell, exit out of the container and then from the local machine, run the command to view the "mongod" container's logged output (with example results shown):<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ docker logs mongo | grep enterprise</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">2017-07-01T12:08:42.177+0000 I CONTROL [initandlisten] modules: enterprise</span><br />
<br />
Again this result should demonstrate that the enterprise version of MongoDB has been used.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Using the Generated Enterprise Mongod Image from the Kubernetes Project</span></h2>
The easiest way to use the "mongod" container image that has just been created, from GKE Kubernetes, is to first register it with Docker Hub, using the following steps:<br />
<br />
<b>1.</b> Create a new free account on <a href="https://hub.docker.com/">Docker Hub</a>.<br />
<br class="Apple-interchange-newline" />
<b>2. </b>Run the following commands to associate your local workstation environment with your new Docker Hub account, to list the built image registered on your local machine, and to push this newly generated image to your remote Docker Hub account:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;"> $ docker login</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> $ docker images</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> $ docker push pkdone/mongo-ent:3.4 </span><br />
<br />
<b>3.</b> Once the image has finished uploading, in a browser, return to the <a href="https://hub.docker.com/">Docker Hub</a> site and log-in to see the list of your registered repository instances. The newly pushed image should now be listed there.<br />
<br />
Now back in the GKE Kubernetes project, for the "mongod" Service/StatefulSet resource definition, change the image reference to be the newly uploaded Docker Hub image, as shown below (highlighted in bold):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>image: </b></span><span style="font-family: "courier new" , "courier" , monospace;"><b>pkdone/mongo-ent:3.4</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "mongod"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
Now re-perform all the normal steps to deploy the Kubernetes cluster and resources as outlined in the <a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">first blog post in the series</a>. Once the MongoDB Replica Set is up and running, you can check the output logs of the first "mongod" container, to see if the enterprise version of MongoDB is being used:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl logs mongod-0 | grep enterprise</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">2017-07-01T13:01:42.794+0000 I CONTROL [initandlisten] modules: enterprise</span><br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Summary</span></h2>
In this blog post I’ve shown how to use the enterprise version of MongoDB, when running a MongoDB Replica Set, using Kubernetes StatefulSets, on the Google Kubernetes Engine. This builds on the work done in previous blog posts in the series, around ensuring MongoDB Replica Sets are resilient and better tuned for production workloads, when running on Kubernetes.<br />
<br />
<i>[Next post in series: <a href="http://pauldone.blogspot.co.uk/2017/07/sharded-mongodb-kubernetes.html">Deploying a MongoDB Sharded Cluster using Kubernetes StatefulSets on GKE</a>]</i><br />
<br />
<br />
<i>Song for today: Standing In The Way Of Control by <a href="https://en.wikipedia.org/wiki/Gossip_(band)">Gossip</a></i><br />
<br />Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-78045134227572681612017-06-27T22:37:00.000+01:002020-02-13T20:03:51.503+00:00Configuring Some Key Production Settings for MongoDB on GKE Kubernetes<i>[Part 2 in a series of posts about running MongoDB on Kubernetes, with the Google </i><i>Kubernetes</i><i> Engine (GKE). See the GitHub project <a href="https://github.com/pkdone/gke-mongodb-demo">gke-mongodb-demo</a> for an example scripted deployment of MongoDB to GKE, that you can easily try yourself. The gke-mongodb-demo project combines the conclusions from all the posts in this series so far. </i><i>Also see: <a href="http://k8smongodb.net/">http://k8smongodb.net/</a>]</i><br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Introduction</span></h2>
In the <a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">first part of my blog series</a> I showed how to deploy a MongoDB Replica Set to GKE's Kubernetes environment, whilst ensuring that the replica set is secure by default and resilient to various types of system failures. As mentioned in that post, there are number of other "production" considerations that need to be made when running MongoDB in Kubernetes and Docker environments. These considerations are primarily driven by the best practices documented in MongoDB’s <a href="https://docs.mongodb.com/manual/administration/production-checklist-operations/">Production Operations Checklist</a> and <a href="https://docs.mongodb.com/manual/administration/production-notes/">Production Notes</a>. In this blog post, I will address how to apply some (but not all) of these best practices, on GKE's Kubernetes platform.<br />
<h2>
<span style="font-size: x-large;"><br />Host VM Modifications for Using XFS & Disabling Hugepages</span></h2>
For optimum performance, the <a href="https://docs.mongodb.com/manual/administration/production-notes/">MongoDB Production Notes</a> strongly recommend applying the following configuration settings to the host operating system (OS):<br />
<ol>
<li>Use an XFS based Linux filesystem for WiredTiger data file persistence.</li>
<li>Disable <a href="https://docs.mongodb.com/manual/tutorial/transparent-huge-pages/">Transparent Huge Pages</a>.</li>
</ol>
The challenge here is that neither of these elements can be configured directly within normally deployed pods/containers. Instead, they need to be set in the OS of each machine/VM that is eligible to host one or more pods and their containers. Fortunately, after a little googling I found a solution to incorporating XFS, in the article <a href="https://medium.com/@allanlei/mounting-xfs-on-gke-adcf9bd0f212">Mounting XFS on GKE</a>, which also provided the basis for deriving a solution for disabling Huge Pages too. It turns out that in Kubernetes, it is possible to run a pod (and its container) once per node (host machine), using a facility called a <a href="https://kubernetes.io/docs/concepts/workloads/controllers/daemonset/">DaemonSet</a>. A DaemonSet is used to schedule a "special" container to run on every newly provisioned node, as a one off, before any "normal" containers are scheduled and run on the node. In addition, for Docker based containers (the default on GKE Kubernetes), the container can be allowed to run in a <a href="https://developers.redhat.com/blog/2014/11/06/introducing-a-super-privileged-container-concept/">privileged mode</a>, which gives the "privileged" container access to other <a href="https://en.wikipedia.org/wiki/Linux_namespaces">Linux Namespaces</a> running in the same host environment. With heightened security rights the "privileged" container can then run a utility called <a href="http://man7.org/linux/man-pages/man1/nsenter.1.html">nsenter</a> ("NameSpace ENTER") to spawn a shell using the namespace belonging to the host OS ("/proc/1"). The script that the shell runs can then essentially perform any arbitrary root level actions on the underlying host OS.<br />
<br />
So with this in mind, the challenge is to build a Docker container image that, when run in privileged mode, uses "nsenter" to spawn a shell to run some shell script commands. As luck would have it, such a container has already been created, in a generic way, as part of the Kubernetes "contributions" project, called <a href="https://github.com/kubernetes/contrib/tree/master/startup-script">startup-script</a>. The generated "startup-script" Docker image has been registered and and made available in the <a href="https://cloud.google.com/container-registry/">Google Container Registry</a>, ready to be pulled in and used by anyone's Kubernetes projects.<br />
<br />
Therefore on GKE, to create a DaemonSet leveraging the "startup-script" image in privileged mode, we first need to define the DaemonSet's configuration:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat hostvm-node-configurer-daemonset.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">kind: DaemonSet</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: extensions/v1beta1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: hostvm-configurer</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> labels:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> app: startup-script</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> template:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> labels:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> app: startup-script</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> hostPID: true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: hostvm-configurer-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>image: gcr.io/google-containers/startup-script:v1</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> securityContext:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>privileged: true</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> env:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: STARTUP_SCRIPT</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> value: |</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> #! /bin/bash</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> set -o errexit</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> set -o pipefail</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> set -o nounset</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> </span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> # Disable hugepages</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> echo 'never' > /sys/kernel/mm/transparent_hugepage/enabled</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> echo 'never' > /sys/kernel/mm/transparent_hugepage/defrag</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> </b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> # Install tool to enable XFS mounting</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> apt-get update || true</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> apt-get install -y xfsprogs</b></span><br />
<br />
Shown in bold at the base of the file, you will notice the commands used to disable Huge Pages and to install the XFS tools for mounting and formatting storage using the XFS filesystem. Further up the file, in bold, is the reference to the 3rd party "startup-script" image from the Google Container Registry and the security context setting to state that the container should be run in privileged mode.<br />
<br />
Next we need to deploy the DaemonSet with its "start-script" container to all the hosts (nodes), before we attempt to create any GCE disks, that need to be formatted as XFS:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl apply -f hostvm-node-configurer-daemonset.yaml</span><br />
<br />
In the GCE disk definitions, described in the <a href="http://pauldone.blogspot.co.uk/2017/06/deploying-mongodb-on-kubernetes-gke25.html">first blog post in this series</a> (i.e. "<i>gce-ssd-persistentvolume?.yaml</i>"), an addition of a new parameter needs to be made (shown in bold below) to indicate that the disk's filesystem type needs to be XFS:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: "v1"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">kind: "PersistentVolume"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: data-volume-1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> capacity:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> storage: 30Gi</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> accessModes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - ReadWriteOnce</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> persistentVolumeReclaimPolicy: Retain</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> storageClassName: fast</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> gcePersistentDisk:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> fsType: xfs</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pdName: pd-ssd-disk-1</span><br />
<br />
Now in theory, this should be all that is required to get XFS working. <b>Except on GKE, it isn't!</b><br />
<br />
After deploying the DaemonSet and creating the GCE storage disks, the deployment of the "mongod" Service/StatefulSet will fail. The StatefulSet's pods do not to start properly because the disks can't be formatted and mounted as XFS. It turns out that this is because, by default, GKE uses a variant of <a href="https://cloud.google.com/container-optimized-os/docs/">Chromium OS</a> as the underlying host VM that runs the containers, and this OS flavour doesn't support XFS. However, GKE can also be configured to use a Debian based Host VM OS instead, which does support XFS.<br />
<br />
To see the list of host VM OSes that GKE supports, the following command can be run:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud container get-server-config</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Fetching server config for europe-west1-b</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">defaultClusterVersion: 1.6.4</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">defaultImageType: <b>COS</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">validImageTypes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- <b>CONTAINER_VM</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">- <b>COS</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<div>
<br /></div>
<div>
Here, "COS" is the label for the Chromium OS and "CONTAINER_VM" is the label for the Debian OS. The easiest way to start leveraging the Debian OS image is to clear out all the GCE/GKE resources and Kubernetes cluster from the current project and start deployment all over again. This time, when the initial command is run to create the new Kubernetes cluster, an additional argument (shown in bold) must be provided to define that the Debian OS should be used for each Host VM that is created as a Kubernetes node.</div>
<div>
<br /></div>
<div>
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud container clusters create "gke-mongodb-demo-cluster" <b>--image-type=CONTAINER_VM</b></span></div>
<div>
<br /></div>
<div>
This time, when all the Kubernetes resources are created and deployed, the "mongod" containers correctly utilise XFS formatted persistent volumes. </div>
<div>
<br /></div>
<div>
If this all seems a bit complicated, it is probably helpful to view the full end-to-end deployment flow, provided in my example GitHub project <a href="https://github.com/pkdone/gke-mongodb-demo">gke-mongodb-demo</a>.</div>
<div>
<br /></div>
<div>
There is one final observation to make before finishing the discussion on XFS. In Google's online documentation, it is stated that the <a href="https://cloud.google.com/container-engine/docs/node-image-migration">Debian Host VM OS is deprecated</a> in favour of Chromium OS. I hope that in the future Google will add XFS support directly to its Chromium OS distribution, to make the use of XFS a lot less painful and to ensure XFS can still be used with MongoDB, if the Debian Host VM option is ever completely removed.</div>
<br />
<b>UPDATE 13-Oct-2017: </b><i>Google has recently updated GKE and in addition to </i><i>upgrading</i><i> Kubernetes to version 1.7, has removed the old Debain container VM option and added an Ubuntu container VM option, instead. In the new Ubuntu container, the XFS tools are already installed, and therefore do not need to be configured by the DaemonSet. The <a href="https://github.com/pkdone/gke-mongodb-demo">gke-mongodb-demo</a> project has been updated accordingly, to use the new Ubuntu container VM and to omit the command to install the XFS tools. </i><br />
<br />
<h2>
<span style="font-size: x-large;">Disabling NUMA</span></h2>
For optimum performance, the <a href="https://docs.mongodb.com/manual/administration/production-notes/">MongoDB Production Notes</a> recommend that "<i>on NUMA hardware, you should configure a memory interleave policy so that the host behaves in a non-NUMA fashion</i>". The <a href="https://github.com/docker-library/mongo">DockerHub "mongo" container image</a> which has been used so far with Kubernetes in this blog series, already contains some <a href="https://github.com/docker-library/mongo/blob/master/3.4/docker-entrypoint.sh">bootstrap code</a> to start the "mongod" process with the "numactl --interleave=all" setting. This setting makes the process environment behave in a non-NUMA way.<br />
<br />
However, I believe it is worth specifying the "numactl" settings explicitly in the "mongod" Service/StatefulSet resource definition, anyway, just in case other users choose to use an alternative or self-built Docker image for the "mongod" container. The excerpt below shows the added "numactl" elements (in bold), required to run the containerised "mongod" process in a "non-NUMA" manner.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> image: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "numactl"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--interleave=all"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "mongod"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<br />
<div>
<br /></div>
<div>
<h2>
<span style="font-size: x-large;">Controlling CPU & RAM Resource Allocation Plus WiredTiger Cache Size</span></h2>
Of course, when you are running a MongoDB database it is important to size both CPU and RAM resources correctly for the particular database workload, regardless of the type of host environment. In a Kubernetes containerised host environment, the amount of CPU & RAM resource dedicated to a container can be defined in the "resource" section of the container's declaration, as shown in the excerpt of the "mongod" Service/StatefulSet definition below:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> image: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "mongod"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--wiredTigerCacheSizeGB"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "0.25"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--bind_ip"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "0.0.0.0"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--replSet"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "MainRepSet"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--auth"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--clusterAuthMode"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--keyFile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "/etc/secrets-volume/internal-auth-mongodb-keyfile"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "--setParameter"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - "authenticationMechanisms=SCRAM-SHA-1"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> resources:</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> requests:</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> cpu: 1</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> memory: 2Gi</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<br />
In the example (shown in bold), 1x virtual CPU (vCPU) and 2GB of RAM have been requested to run the container. You will also notice that an additional parameter has been defined for "mongod", specifying the <a href="https://docs.mongodb.com/manual/reference/program/mongod/#cmdoption-wiredtigercachesizegb">WiredTiger internal cache size</a> ("<i>--wiredTigerCacheSizeGB</i>"). In a containerised environment it is absolutely vital to explicitly state this value. If this is not done, and multiple containers end up running on the same host machine (node), MongoDB's WiredTiger storage engine may attempt to take more memory than it should. This is because of the way a container "reports" it's memory size to running processes. As per the <a href="https://docs.mongodb.com/manual/administration/production-notes/#allocate-sufficient-ram-and-cpu">MongoDB Production Recommendations</a>, the default cache size guidance is: "<i>50% of RAM minus 1 GB, or 256 MB</i>". Given that the amount of memory requested is 2GB, the WiredTiger cache size here, has been set to 256MB.<br />
<br />
If and when you define a different amount of memory for the container process, be sure to also adjust the WiredTiger cache size setting accordingly, otherwise the "mongod" process may not leverage all the memory reserved for it, by the container.</div>
<div>
<br /></div>
<div>
<br /></div>
<div>
<h2>
<span style="font-size: x-large;">Controlling Anti-Affinity for Mongod Replicas</span></h2>
When running a MongoDB Replica Set, it is important to ensure that none of the "mongod" replicas in the replica set are running on the same host machine as each other, to avoid inadvertently introducing a single point of failure. In a Kubernetes containerised environment, if containers are left to their own devices, different "mongod" containers could end up running on the same nodes. Kubernetes provides a way of specifying pod anti-affinity to prevent this from occurring. Below is an excerpt of a "mongod" Services/StatefulSet resource file which declares an anti-affinity configuration.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> serviceName: mongodb-service</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> replicas: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> template:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> labels:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>replicaset</b>: <b>MainRepSet</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> affinity:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>podAntiAffinity</b>:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> preferredDuringSchedulingIgnoredDuringExecution:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - weight: 100</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> podAffinityTerm:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> labelSelector:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> matchExpressions:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - key: <b>replicaset</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> operator: <b>In</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> values:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - <b>MainRepSet</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b>topologyKey: kubernetes.io/hostname</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">....</span><br />
<div>
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span></div>
Here, a rule has been defined that asks Kubernetes to apply anti-affinity when deploying pods with the label "replicaset" equal to "MainRepSet", by looking for potential matches on the host VM instance's hostname, and then avoiding them. <br />
<br />
<br />
<h2>
<span style="font-size: x-large;">Setting File Descriptor & User Process Limits</span></h2>
When deploying the MongoDB Replica Set on GKE Kubernetes, as demonstrated in the current GitHub project <a href="https://github.com/pkdone/gke-mongodb-demo">gke-mongodb-demo</a>, you may notice some warning about "rlimits" in the output of each containerised mongod's logs. These log entries can be viewed by running the following command:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl logs mongod-0 | grep rlimits</span><br />
<span style="font-family: "courier new" , "courier" , monospace; font-size: xx-small;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">2017-06-27T12:35:22.018+0000 I CONTROL [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 29980 processes, 1000000 files. Number of processes should be at least 500000 : 0.5 times number of files.</span><br />
<div>
<br /></div>
The MongoDB manual provides some recommendations concerning the system settings for the <a href="https://docs.mongodb.com/manual/reference/ulimit/">maximum number of processes and open files</a> when running a "mongod" process.<br />
<br />
Unfortunately, thus far, I've not established an appropriate way to the enforce these thresholds using GKE Kubernetes. This topic will possibly be the focus of a blog post for another day. However, I thought that it would be informative to highlight the issue here, with the supporting context, to allow others the chance to resolve it first.<br />
<br />
<b>UPDATE 13-Oct-2017: </b><i>Google has recently updated GKE and in addition to upgrading Kubernetes to version 1.7, has removed the old Debain container VM option and added an Ubuntu container VM option, instead. The default "rlimits" settings in the Ubuntu container VM, are already appropriate for running MongoDB. Therefore a fix is no longer required to address this issue. </i><br />
<i><br /></i></div>
<h2>
<span style="font-size: x-large;">Summary</span></h2>
In this blog post I’ve provided some methods for addressing certain best practices when deploying a MongoDB Replica Set to the Google Kubernetes Engine. Although this post does not provide an exhaustive list of best practice solutions, I hope it proves useful for others (and myself) to build upon, in the future.<br />
<br />
<i>[Next post in series: <a href="http://pauldone.blogspot.co.uk/2017/06/enterprise-mongodb-on-kubernetes.html">Using the Enterprise Version of MongoDB on GKE Kubernetes</a>]</i><br />
<br />
<br />
<i>Song for today: The Mountain by <a href="https://en.wikipedia.org/wiki/Jambinai">Jambinai</a></i><br />
<div>
<br /></div>
Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com8tag:blogger.com,1999:blog-1304066656993695443.post-55486607417919626622017-06-25T20:42:00.003+01:002022-08-15T23:52:41.995+01:00Deploying a MongoDB Replica Set as a GKE Kubernetes StatefulSet<i>[Part 1 in a series of posts about running MongoDB on Kubernetes, with the Google </i><i>Kubernetes</i><i> Engine (GKE). See the GitHub project <a href="https://github.com/pkdone/gke-mongodb-demo">gke-mongodb-demo</a> for an example scripted deployment of MongoDB to GKE, that you can easily try yourself. The gke-mongodb-demo project combines the conclusions from all the posts in this series so far. </i><i>Also see: <a href="http://k8smongodb.net/">http://k8smongodb.net/</a>]</i><br />
<i><br /></i>
<br />
<h2>
<span style="font-size: x-large;">Introduction</span></h2>
A few months ago, Sandeep Dinesh of Google wrote an informative blog post about <a href="http://blog.kubernetes.io/2017/01/running-mongodb-on-kubernetes-with-statefulsets.html">Running MongoDB on Kubernetes with StatefulSets</a> on Google’s Cloud Platform. I found this to be a great resource to bootstrap my knowledge of Kubernetes’ new <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSets</a> feature, and food for thought for approaches for deploying MongoDB on Kubernetes generally. StatefulSets is Kubernetes’ framework for providing better support for “stafeful applications”, such as databases and message queues. StatefulSets provides the capabilities of stable unique network hostnames and stable dedicated network storage volume mappings, essential for a database cluster to function properly and for data to exist and outlive the lifetime of inherently ephemeral containers.<br />
<br />
My view of the approach in the Google blog post, is it is a great way for a developer to rapidly spin up a MongoDB Replica Set, to quickly test that their code still works correctly (it should) in a clustered environment. However, the approach cannot be regarded as a best practice for deploying MongoDB in Production, for mission critical use cases. This assertion is not a criticism, as the blog post is obviously intended to show the art of the possible (which it does very eloquently), and the author makes no claim to be a seasoned MongoDB administration expert.<br />
<br />
So what are the challenges for Production deployments, in the approach outlined in the Google blog post? Well there are two problems, which I will address in this post:<br />
<ol>
<li>Use of a <a href="https://github.com/cvallance/mongo-k8s-sidecar">MongoDB/Kubernetes sidecar</a> per <a href="http://blog.kubernetes.io/2015/06/the-distributed-system-toolkit-patterns.html">Pod</a>, to control Replica Set configuration. Essentially, the sidecar wakes up every 5 seconds, checks which MongoDB pods are running and then reconfigures the replica-set, on the fly. It adds any MongoDB servers it can see, to the replica set configuration, and removes any servers it can no longer see. This is dangerous for many reasons. I’ve highlighted two of the most important reasons why here*:</li>
<ul>
<li>This introduces the real risk of split-brain, in the event of a network partition. For example, normally, if there is a 3 node replica set configured and the primary is somehow separated from the secondaries, the primary will step down as it can’t maintain a majority. Normally, the two secondaries that can now only see each other, will form a quorum and one of these two will then become the primary. In the sidecar implementation, during a network split, the sidecar on the primary believes the two secondaries aren’t running and it re-configures the replica set on the fly, to now just have one member. This remaining member believes it can still act as primary (because it has achieved a majority of 1 out of 1 votes). The sidecars still running on the other two members, now also reconfigure the replica set to be just those two members. One of these two members automatically becomes a primary (because it has achieved a majority of 2 out of 2 votes). As a result there are now two primaries in existence for the same replica-set, which a normal and properly configured MongoDB cluster would never allow to occur. MongoDB’s strong consistency guarantees are subverted and non-deterministic things will start happening to the data. In a properly deployed MongoDB cluster, if there is a 3 node replica set and 2 nodes appear to be down, it doesn’t mean you now have a 1 node replica set, you don’t. You still have a 3 node replica-set, albeit only one replica appears to be currently running (and hence no primary is permitted, to guarantee safety and strong consistency).</li>
<li>Many applications updating data in MongoDB will use “WriteConcerns” set to a value such as “majority”, to provide levels of guarantee for safe data updates across a cluster. The whole notion of a “WriteConcern” would become meaningless in the sidecar controlled environment, because the constantly re-configured replica set would always reflect a total replica-set size of just those replicas currently active and reachable. For example, performing a database update operation with “WriteConcern” of “majority” would always be permitted, regardless of whether all 3 replicas are currently available, or just 2 replicas are or just 1 replica is.</li>
</ul>
<li>Insecure by default, due to authentication not being enabled. In a Production environment, running MongoDB with authentication disabled should never be allowed. Even if the intention is to configure authentication as a later provisioning step, the database is potentially exposed and insecure for seconds, minutes or longer. As a result, the “mongod” process should always be started with authentication enabled (e.g. using “<i>--auth</i>” command line flag), even during any “bootstrap provisioning process”. MongoDB’s <a href="https://docs.mongodb.com/manual/core/security-users/#localhost-exception">localhost exception</a> should be relied upon to securely configure one or more database users.</li>
</ol>
<span style="font-size: x-small;">* If this was such an easy and safe thing to do, MongoDB replicas would be built to automatically perform these re-configurations, themselves, in a separate background thread running inside each “mongod” replica process. The brain controlling what the replica-set configuration should look like, lives outside the cluster for good reason (e.g. inside the head of an administrator, or preferably, inside a configuration file that is used to drive a higher level orchestration tool which operates above the “containers layer”).</span><br />
<br />
Additionally, there are number of other considerations that aren’t just specific to the approach in the referenced Google blog post, but are applicable to the use of Docker/Kubernetes with MongoDB, generally. These consideration can be categorised as ways to ensure that MongoDB’s best practices are followed, as documented in MongoDB’s <a href="https://docs.mongodb.com/manual/administration/production-checklist-operations/">Production Operations Checklist</a> and <a href="https://docs.mongodb.com/manual/administration/production-notes/">Production Notes</a>. I address some of these best practice omissions in the next post in this series: <a href="http://pauldone.blogspot.co.uk/2017/06/mongodb-kubernetes-production-settings.html">Configuring Some Key Production Settings for MongoDB on GKE Kubernetes</a>. It is probably worth me being clear here, that I am not claiming my blog series will get users 100% to where they need to be, to deploy a fully operational, secure and well-performing MongoDB Clusters on GKE. Instead, what I hope the series will do, is enable users to build on my findings and recommendations, so there are less gaps for them to address, when planning their own production environment.<br />
<br />
For the rest of this blog post, I will focus on the steps required to deploy a MongoDB Replica Set, on GKE, addressing the replica-set resiliency and security concerns that I've highlighted above.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">
Steps to Deploy MongoDB to GKE, using StatefulSets</span></h2>
The first thing to do, if you haven’t already, is sign up to use the Google Cloud Platform (GCP). To keeps things simple, you can sign up to a <a href="https://cloud.google.com/free/">free trial for GCP</a>. Note: The free trial places some restrictions on account resource quotas, in particular restricting storage to a maximum of 100GB. Therefore, in my series of the blog posts and <a href="https://github.com/pkdone/gke-mongodb-demo">my sample GitHub project</a>, I employ modest disk sizes, to remain under this threshold.<br />
<br />
Once your GCP account is activated, you should <a href="https://cloud.google.com/sdk/docs/quickstarts">download and install GCP’s client command line tool</a>, called “gcloud”, to your local Linux/Windows/Mac workstation.<br />
<br />
With “gcloud” installed, run the following commands to configure the local environment to use your GCP account, to install the main Kubernetes command tool (“kubectl”), to configure authentication credentials, and to define the default GCP zone to be deployed to:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud init</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud components install kubectl</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud auth application-default login</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud config set compute/zone europe-west1-b</span><br />
<br />
Note: If you want to specify an alternative zone to deploy to in the above command, you can first view the list of available zones by running the command: <span style="font-family: "courier new" , "courier" , monospace;">$ gcloud compute zones list</span><br />
<br />
You should now be ready to create a brand new Kubernetes cluster to the Google Kubernetes Engine. Run the following command to provision a new Kubernetes cluster called “gke-mongodb-demo-cluster”:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud container clusters create "gke-mongodb-demo-cluster"</span><br />
<br />
As part of this process, a set of 3 GCE VM instances are automatically provisioned, to run Kubernetes cluster nodes ready to host pods of containers.<br />
<br />
You can view the state of the deployed Kubernetes cluster using the <a href="https://console.cloud.google.com/">Google Cloud Platform Console</a> (look at both the “Kubernetes Engine” and the “Compute Engine” sections of the Console).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1VkyD0iQBTbPj5WZWsknq0V7QEJuwTBmOhNvrBxQHKVr5EyHje1rYJ6NTHPe7cauCjnx5t1OpKuroqSK_b3XN25l3t-EfTRbyT-52cM2qxAjlQSN8WViOYQFVRkjkvHxpX7qXAm6VBqo/s1600/console.png" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="699" data-original-width="1600" height="278" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj1VkyD0iQBTbPj5WZWsknq0V7QEJuwTBmOhNvrBxQHKVr5EyHje1rYJ6NTHPe7cauCjnx5t1OpKuroqSK_b3XN25l3t-EfTRbyT-52cM2qxAjlQSN8WViOYQFVRkjkvHxpX7qXAm6VBqo/s640/console.png" width="640" /></a></div>
<br />
Next, lets register GCE’s fast SSD persistent disks to be used in the cluster:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat gce-ssd-storageclass.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">kind: StorageClass</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: storage.k8s.io/v1beta1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: fast</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">provisioner: kubernetes.io/gce-pd</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">parameters:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> type: pd-ssd</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl apply -f gce-ssd-storageclass.yaml</span><br />
<br />
Then run the commands to allocate 3 lots of Google Cloud storage, of size 30GB, using the fast SSD persistent disks, followed by a query to show the status of those newly created disks:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud compute disks create --size 30GB --type pd-ssd pd-ssd-disk-1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud compute disks create --size 30GB --type pd-ssd pd-ssd-disk-2</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud compute disks create --size 30GB --type pd-ssd pd-ssd-disk-3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ gcloud compute disks list</span><br />
<br />
Now, declare 3 Kubernetes “Persistent Volume” definitions, that each reference one of the storage disks just created:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat gce-ssd-persistentvolume1.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: "v1"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">kind: "PersistentVolume"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: data-volume-1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> capacity:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> storage: 30Gi</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> accessModes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - ReadWriteOnce</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> persistentVolumeReclaimPolicy: Retain</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> storageClassName: fast</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> gcePersistentDisk:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pdName: pd-ssd-disk-1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl apply -f gce-ssd-persistentvolume1.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><i><br /></i></span>
<i>(repeat for Disks 2 and 3, using similar files, “gce-ssd-persistentvolume2.yml” and “gce-ssd-persistentvolume3.yml” respectively, with the fields “name: data-volume-?” and “pdName: pd-ssd-disk-?” set in each file)</i><br />
<br />
Once the three Persistent Volumes are configured, their status can be viewed with the following command:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl get persistentvolumes</span><br />
<br />
This will show that the state of each volume is marked as “available” (i.e. no container has staked a claim on each yet).<br />
<br />
A key deviation from the original Google blog post, is enabling MongoDB authentication immediately, before any "mongod" processes are started. Enabling authentication for a MongoDB replica set doesn’t just enforce authentication of applications using MongoDB, but also enforces <a href="https://docs.mongodb.com/v3.0/tutorial/enable-internal-authentication/">internal authentication</a> for inter-replica communication. Therefore, lets generate a keyfile to be used for internal cluster authentication and register it as a Kubernetes <a href="https://kubernetes.io/docs/concepts/configuration/secret/">Secret</a>:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ TMPFILE=$(mktemp)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ /usr/bin/openssl rand -base64 741 > $TMPFILE</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl create secret generic shared-bootstrap-data –from file=internal-auth-mongodb-keyfile=$TMPFILE</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ rm $TMPFILE</span><br />
<br />
This generates a random key into a temporary file and then uses the Kubernetes API to register it as a Secret, before deleting the file. Subsequently, the Secret will be made accessible to each “mongod”, via a volume mounted by each host container.<br />
<br />
For the final Kubernetes provisioning step, we need to prepare the definition of the Kubernetes <a href="https://kubernetes.io/docs/concepts/services-networking/service/">Service</a> and <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSet</a> for MongoDB, which, amongst other things, encapsulates the configuration of the “mongod” Docker container to be run.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: v1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">kind: Service</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: mongodb-service</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> labels:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ports:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - port: 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> targetPort: 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> clusterIP: None</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> selector:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> role: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">---</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">apiVersion: apps/v1beta1</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">kind: StatefulSet</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: mongod</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> serviceName: mongodb-service</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> replicas: 3</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> template:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> labels:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> role: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> environment: test</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> replicaset: MainRepSet</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> terminationGracePeriodSeconds: 10</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> volumes:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: secrets-volume</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> secret:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> secretName: shared-bootstrap-data</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> defaultMode: 256</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> containers:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongod-container</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> image: mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> command:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> <b> - "mongod"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--bind_ip"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "0.0.0.0"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--replSet"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "MainRepSet"</b></span><br />
<b style="font-family: "courier new", courier, monospace;"> - "--auth"</b><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--clusterAuthMode"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "keyFile"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--keyFile"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "/etc/secrets-volume/internal-auth-mongodb-keyfile"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "--setParameter"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><b> - "authenticationMechanisms=SCRAM-SHA-1"</b></span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ports:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - containerPort: 27017</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> volumeMounts:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: secrets-volume</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> readOnly: true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> mountPath: /etc/secrets-volume</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - name: mongodb-persistent-storage-claim</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> mountPath: /data/db</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> volumeClaimTemplates:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> - metadata:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> name: mongodb-persistent-storage-claim</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> annotations:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> volume.beta.kubernetes.io/storage-class: "fast"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> spec:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> accessModes: [ "ReadWriteOnce" ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> resources:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> requests:</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> storage: 30Gi</span><br />
<br />
You may notice that this Service definition varies in some key areas, from the one provided in the original Google blog post. Specifically:<br />
<ol>
<li>A “Volume” called “secrets-volume” is defined, ready to expose the shared keyfile to each of the “mongod” replicas that will run.</li>
<li>Additional command line parameters are specified for “mongod”, to enable authentication (“--auth”) and to provide related security settings, including the path where “mongod” should locate the keyfile on its local filesystem.</li>
<li>In the “VolumeMounts” section, the mount point path is specified for the Volume that holds the key file.</li>
<li>The storage request for the Persistent Volume Claim that the container will make, has been reduced from 100GB to 30GB, to avoid issues if using the free trial of the Google Cloud Platform (avoids exhausting storage quotas).</li>
<li>No “sidecar” Container is defined for the same Pod as the “mongod” Container.</li>
</ol>
Now it’s time to deploy the MongoDB Service and StatefulSet. Run:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl apply -f mongodb-service.yaml</span><br />
<br />
Once this has run, you can view the health of the service and pods:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl get all</span><br />
<br />
Keep re-running the command above, until you can see that all 3 “mongod” pods and their containers have been successfully started (<i>“Status=Running”</i>).<br />
<br />
You can also check the status of the Persistent Volumes, to ensure they have been properly claimed by the running “mongod” containers:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl get persistentvolumes</span><br />
<br />
Finally, we need to connect to one of the “mongod” container processes to configure the replica set and specify an administrator user for the database. Run the following command to connect to the first container:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl exec -it mongod-0 -c mongod-container bash</span><br />
<br />
This will place you into a command line shell directly in the container. If you fancy it, you can explore the container environment. For example you may want to run the following commands to see what processes are running in the container and also to see the hostname of the container (this hostname should always be the same, because a StatefulSet has been used):<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ ps -aux</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ hostname -f</span><br />
<br />
Connect to the local “mongod” process using the Mongo Shell (it is only possible to connect unauthenticated from the same host that the database process is running on, by virtue of the <a href="https://docs.mongodb.com/manual/core/security-users/#localhost-exception">localhost exception</a>).<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo</span><br />
<br />
In the shell run the following command to initiate the replica set (we can rely on the hostnames always being the same, due to having employed a StatefulSet):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">> rs.initiate({_id: "MainRepSet", version: 1, members: [</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> { _id: 0, host : "mongod-0.mongodb-service.default.svc.cluster.local:27017" },</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> { _id: 1, host : "mongod-1.mongodb-service.default.svc.cluster.local:27017" },</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> { _id: 2, host : "mongod-2.mongodb-service.default.svc.cluster.local:27017" }</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ]});</span><br />
<br />
Keep checking the status of the replica set, with the following command, until you see that the replica set is fully initialised and a primary and two secondaries are present:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">> rs.status();</span><br />
<br />
Then run the following command to configure an “admin” user (performing this action results in the “localhost exception” being automatically and permanently disabled):<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">> db.getSiblingDB("admin").createUser({</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> user : "main_admin",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> pwd : "abc123",</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> roles: [ { role: "root", db: "admin" } ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> });</span><br />
<br />
Of course, in a real deployment, the steps used above, to configure a replica set and to create an admin user, would be scripted, parameterised and driven by an external process, rather than typed in manually.<br />
<br />
That’s it. You should now have a MongoDB Replica Set running on Kubernetes on GKE.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">
Run Some Quick Tests</span></h2>
Let just prove a couple of things before we finish:<br />
<br />
1. Show that data is indeed being replicated between members of the containerised replica set.<br />
2. Show that even if we remove the replica set containers and then re-create them, the same stable hostnames are still used and no data loss occurs, when the replica set comes back online. The StatefulSet’s Persistent Volume Claims should successfully result in the same storage, containing the MongoDB data files, being attached to by the same “mongod” container instance identities.<br />
<br />
Whilst still in the Mongo Shell from the previous step, authenticate and quickly add some test data:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.getSiblingDB('admin').auth("main_admin", "abc123");</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> use test;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.testcoll.insert({a:1});</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.testcoll.insert({b:2});</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.testcoll.find();</span><br />
<br />
Exit out of the shell and exit out of the first container (“mongod-0”). Then using the following commands, connect to the second container (“mongod-1”), run the Mongo Shell again and see if the data we’d entered via the first replica, is visible to the second replica:<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl exec -it mongod-1 -c mongod-container bash</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.getSiblingDB('admin').auth("main_admin", "abc123");</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.setSlaveOk(1);</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> use test;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.testcoll.find();</span><br />
<br />
You should see that the two records inserted via the first replica, are visible to the second replica.<br />
<br />
To see if Persistent Volume Claims really are working, use the following commands to drop the Service & StatefulSet (thus stopping the pods and their “mongod” containers) and re-create them again (I’ve included some checks in-between, so you can track the status):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl delete statefulsets mongodb-statefulset</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl delete services mongodb-service</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl get all</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl get persistentvolumes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl apply -f mongodb-service.yaml</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl get all</span><br />
<br />
As before, keep re-running the last command above, until you can see that all 3 “mongod” pods and their containers have been successfully started again. Then connect to the first container, run the Mongo Shell and execute a query to see if the data we’d inserted into the old containerised replica-set is still present in the re-instantiated replica set:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ kubectl exec -it mongod-0 -c mongod-container bash</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.getSiblingDB('admin').auth("main_admin", "abc123");</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> use test;</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.testcoll.find();</span><br />
<br />
You should see that the two records inserted earlier, are still present.<br />
<br />
<br />
<h2>
<span style="font-size: x-large;">
Summary</span></h2>
In this blog post I’ve shown how a MongoDB Replica Set can be deployed, using Kubernetes StatefulSets, to the Google Kubernetes Engine (GKE). Most of the outlined steps (but not all) are actually generic to any type of Kubernetes platform. Critically, I have shown how to ensure the Kubernetes based MongoDB Replica Set is secure by default, and how to ensure the Replica Set can operate normally, to be resilient to various types of system failures.<br />
<br />
<i>[Next post in series: <a href="http://pauldone.blogspot.co.uk/2017/06/mongodb-kubernetes-production-settings.html">Configuring Some Key Production Settings for MongoDB on GKE Kubernetes</a>]</i><br />
<br />
<br />
<i>Song for today: Sun by <a href="https://en.wikipedia.org/wiki/The_Hotelier">The Hotelier</a></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com21tag:blogger.com,1999:blog-1304066656993695443.post-40579863653967112272016-03-18T15:54:00.003+00:002020-03-01T20:55:42.050+00:00SHOCKER: XA Distributed Transactions are only Eventually Consistent!Apologies for the tabloid-trash style headline. It could have been worse, I could have gone with my working title of "WARNING: XA will eat your first-born"!<br />
<br />
This topic has come up in a few conversations I've had recently. It turns out that most don't realise what I'd assumed to be widely understood. <a href="https://en.wikipedia.org/wiki/Two-phase_commit_protocol">2-phase-commit</a> (2PC), and <a href="https://en.wikipedia.org/wiki/X/Open_XA">XA</a> (it's widely used implementation) are <b>NOT</b> <a href="https://en.wikipedia.org/wiki/ACID">ACID</a> compliant. Specifically XA/2PC does not provide strong consistency guarantees and is in-fact just <a href="https://en.wikipedia.org/wiki/Eventual_consistency">Eventually Consistent</a>. It get's worse in practice. In places where I've seen XA/2PC used, it transpires that Atomicity and Durability are on shaky ground too (more on this later).<br />
<br />
Why have I seen this topic rearing it's head recently? Well some organisations have cases where they want to update data in an Oracle Database and in a <b>MongoDB</b> database, as a single transaction. Nothing wrong with that of course, it really just depends on how you choose to implement the transaction and what your definition of a "transaction" really is. All too often, those stating this requirement will then go on to say that MongoDB has a problem because it does not support the XA protocol. They want their database products to magically take care of this complexity, under the covers. They want them to provide the guarantee of "strong" consistency across the multiple systems, without having to deal with this in their application code. If you're one of those people, I'm here to tell that <i>these are not the <strike>droids</strike> protocols you are looking for</i>.<br />
<br />
Let me have a go at explaining why XA/2PC distributed transactions are not strongly consistent, This is based on what I've seen over the past 10 years or so, especially in the mid-2000s, when working with some UK government agencies and seeing this issue at first hand.<br />
<br />
<br />
<b>First of all, what are some examples of distributed transactions?</b><br />
<ul>
<li>You've written a piece of application code that needs to put the same data into two different databases (eg. Oracle's database and IBM's DB2 database), all or nothing, as a single transaction. You don't want to have a situation where the data appears in one database but not in the other.</li>
<li>You've written a piece of application code that receives a message off a <a href="https://en.wikipedia.org/wiki/Message_queue">message queue</a> (eg. <a href="https://en.wikipedia.org/wiki/IBM_WebSphere_MQ">IBM MQ Series</a>) and inserts the data, contained in the message, into a database. You want these two operations to be part of the same transaction. You want to avoid the situation where the dequeue operation succeeds but the DB insert operation fails, resulting in a lost message and no updated database. You also want to avoid the situation where the database insert succeeds, but the acknowledgement of dequeue operation subsequently fails, resulting in the the message being re-delivered (a duplicate database insert of the same data would then occur).</li>
<li>Another "real-world" example that people [incorrectly] quote, is moving funds between two bank systems, where one system is debited, say £50, and the other is credited by the same amount, as a single "transaction". Of course you wouldn't want the situation to occur where £50 is taken from one system, but due to a transient failure, is not placed in the other system, so money is lost <b>*</b>. The reason I say "incorrectly" is that in reality, banks don't manage and record money transfers this way. <a href="http://highscalability.com/blog/2013/5/1/myth-eric-brewer-on-why-banks-are-base-not-acid-availability.html">Eric Brewer explains</a> why this is the case in a much more eloquent way than I ever could. On a related but more existential note, Gregor Hohpe's classic post is still well worth a read: <a href="http://www.enterpriseintegrationpatterns.com/docs/IEEE_Software_Design_2PC.pdf">Your Coffee Shop Doesn’t Use Two-Phase Commit</a>.</li>
</ul>
<b style="font-size: small;">*</b><span style="font-size: x-small;"> although if you were the receiver of the funds, you might like the other possible outcome, where you receive two lots of £50, due to the occurrence of a failure during the 1st transaction attempt</span><br />
<b><br /></b>
<br />
<b><br /></b>
<b>So what's the problem then?</b><br />
<br />
Back in the mid-2000s, I was involved in building distributed systems with application code running on a <a href="https://en.wikipedia.org/wiki/Java_Platform,_Enterprise_Edition">Java EE</a> <a href="https://en.wikipedia.org/wiki/Application_server">Application Server</a> (<a href="https://en.wikipedia.org/wiki/Oracle_WebLogic_Server">WebLogic</a>). The code would update a record in a database (Oracle) and then place a message on a queue (IBM MQ Series), as part of the same distributed transaction, using XA. At a simplistic level, the transactions performed looked something like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaD3sQwCKXKPKdHKajFVxGuk8I2MQhIodDsF9Br5LhluXIIM37txfZ3jkJwEOK8vjBr3Qci0EOlSTO6rzL1yLJV-ISOajGMwk5_hyphenhyphenB4hWngiXErufnoRywZrDjTQlyMQ36ZYPXx9qTiQg/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="198" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaD3sQwCKXKPKdHKajFVxGuk8I2MQhIodDsF9Br5LhluXIIM37txfZ3jkJwEOK8vjBr3Qci0EOlSTO6rzL1yLJV-ISOajGMwk5_hyphenhyphenB4hWngiXErufnoRywZrDjTQlyMQ36ZYPXx9qTiQg/s400/1.png" width="400" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
If the update to the DB failed, the enqueue operation to put the message onto the message queue would be rolled back, and vice versa. However, as with most real world scenarios, the business process logic was more involved than that. The business process actually had two stages, which looked like this:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgp-LfV6oPI75oV7WQ9WrPj_gCKBq1NlYV1c27HFHIf1WsQ4HDDjPXv0UmWg7ljUw17IfCA6kqm_8vORvcIQ1SS0exu3SW7zyFu7UuL1gCKtUvqDX2jLP5UQnn1fnfQH5gkOQY_z-oDdzU/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgp-LfV6oPI75oV7WQ9WrPj_gCKBq1NlYV1c27HFHIf1WsQ4HDDjPXv0UmWg7ljUw17IfCA6kqm_8vORvcIQ1SS0exu3SW7zyFu7UuL1gCKtUvqDX2jLP5UQnn1fnfQH5gkOQY_z-oDdzU/s400/2.png" width="400" /></a></div>
<br />
Basically in the first stage of the process, the application code would put some data in the database and then put a message on a queue, as part of a single transaction, ready to allow the next stage of the process to be kicked off. The queuing system would already have a piece of application code registered with it, to listen for arrived messages (called a "message listener"). Once the message was committed to the queue, a new transaction would be initiated. In this new transaction the message would be given to the message listener. The listener's application code would receive the message and then read some of the data, that was previously inserted into the database.<br />
<br />
However, when we load tested this, before putting the solution into production, the system didn't always work that way. Sporadically, we saw this instead:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh94D9K72zo7hGbtbptBwiuSkBkphDR63OV2_l3Wx-9R_sO8PFdVhxF70SkmxjAMlx74d5gFUnQEEHMnOEyevrtjxb7gFQ-zDF9cETx30Nx94E-8_TUExpZ2RxE-qvTY2rRCE2nTgc3Zvc/s1600/3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="355" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh94D9K72zo7hGbtbptBwiuSkBkphDR63OV2_l3Wx-9R_sO8PFdVhxF70SkmxjAMlx74d5gFUnQEEHMnOEyevrtjxb7gFQ-zDF9cETx30Nx94E-8_TUExpZ2RxE-qvTY2rRCE2nTgc3Zvc/s400/3.png" width="400" /></a></div>
<br />
How could this be? A previous transaction had put the data in the database as part of the same transaction that put the message on the queue. Only when the message was successfully committed to the queue, could the second transaction be kicked off. Yet, the subsequent listener code couldn't find the row of data in the database, that was inserted there by the previous transaction!<br />
<br />
At first we assumed there was a bug somewhere and hunted for it in Oracle, MQ Series, WebLogic and especially our own code. Getting nowhere, we eventually started digging around the XA/2PC specification a little more, and we realised that the system was behaving correctly. It was correct behaviour to see such race conditions happen intermittently (even though it definitely wasn't desirable behaviour, on our part). This is because, even though XA/2PC guarantees that both resources in a transaction will have their changes either committed or rolled-back atomically, it can't enforce exactly when this will happen in each. The final commit action (the 2nd phase of 2PC) performed by each of those resource systems is initiated in parallel and hence cannot be synchronised.<br />
<br />
The asynchronous nature of XA/2PC, for the final commit process, is by necessity. This allows for circumstances where one of the systems may have temporarily gone down between voting "yes" to commit and then subsequently being told to actually commit. If it is never possible for any of the systems to go down, there would be little need for transactions in the first place (quid pro quo). The application server controlling the transaction keeps trying to tell the failed system to commit, until it comes back online and executes the commit action. The database or message queue system can never 100% guarantee to always commit immediately, and thus only guarantees to commit eventually. Even when there isn't a failure, the two systems are being committed in parallel and will each take different and non-deterministic durations to fulfil the commit action (including the variable time it takes to persist to disk, for durability). There's no way of guaranteeing that they both achieve this in exactly the same instance of time - they never will. In our situation, the missing data would eventually appear in the database, but there was no guarantee that it would always be there when the code in a subsequent transaction tried to read it. Indeed, upon researching a couple of things while preparing this blog post, I discovered that even Oracle now <a href="http://www.oracle.com/technetwork/products/clustering/overview/distributed-transactions-and-xa-163941.pdf">documents this type of race condition</a> (see section "Avoiding the XA Race Condition").<br />
<br />
Back then, we'd inadvertently created the perfect reproducible test case for <b><span style="color: red;">PROOF THAT XA/2PC IS ONLY EVENTUALLY CONSISTENT</span></b>. To fix the situation, we had to put some convoluted workarounds into our application code. The workarounds weren't pretty and they're not something I have any desire to re-visit here.<br />
<br />
<br />
<b>There's more! When the solution went live, things got even worse...</b><br />
<br />
It wasn't just the "C" in "ACID" that was being butchered. It turned out that there was a contract out to do a hatchet job on the "A" and "D" of "ACID" too.<br />
<br />
In live production environments, temporary failures of sub-systems will inevitably occur. In our high throughput system, some distributed transactions will always be in-flight at the time of the failure. These in-flight transactions would then stick around for a while and some would be visible in the Oracle database (tracked in Oracle's "<a href="https://docs.oracle.com/cd/B28359_01/server.111/b28310/ds_txnman007.htm">DBA_2PC_PENDING</a>" system table). There's nothing wrong with this, except the application code that created these transactions will have been holding a lock on one or more table rows. These locks are a result of the application code having performed an update operation as part of the transaction, that has not yet been committed. In our live environment, due to these transactions being in-doubt for a while (minutes or even hours depending on the type of failure) a cascading set of follow-on issues would occur. Subsequent client requests coming into the application would start backing up, as they tried to query the same locked rows of data, and would get blocked or fail. This was due to the code having used the very common "<a href="https://docs.oracle.com/cd/E11882_01/server.112/e40540/consist.htm#CNCPT1331">SELECT ... FOR UPDATE</a>" operation, which attempts to grab a lock on a row, ready for the row to be updated in a later step.<br />
<br />
Pretty soon there would be a deluge of blocking or failing threads and the whole system would appear to lock up. No client requests could be serviced. Of course, the DBA would then receive a rush of calls from irate staff yelling that the mission critical database had ground to a halt. Under such time pressure, all the poor DBA could possibly do was to go to the source of the locks and try to release them. This meant going to Oracle's "pending transactions" system tables and unilaterally <a href="https://docs.oracle.com/cd/B28359_01/server.111/b28310/ds_txnman007.htm">rolling back or committing</a> each of them, to allow the system to recover and service requests again. At this point all bets were off. The DBA's decision to rollback or commit would have been completely arbitrary. Some of the in-flight transactions would have been partly rolled-back in Oracle, but would have been partly committed in MQ Series, and vice versa.<br />
<br />
So in practice, these in-doubt transactions were neither applied Atomically or Durably. The "theory" of XA guaranteeing Atomicity and Durability was not under attack. However, the practical real-world application of it was. At some point, fallible human intervention was required to quickly rescue a failing mission critical system. Most people I know live in the real world.<br />
<br />
<br />
<b>My conclusions...</b><br />
<br />
You can probably now guess my view on XA/2PC. It is not a panacea. Nowhere near. It gives developers false hope, lulling them into a false sense of security, where, at best, their heads can be gently warmed whilst buried in the sand.<br />
<br />
It is impossible to perform a distributed transaction on two or more different systems, in a fully ACID manner. Accept it and deal with it by allowing for this in application code and/or in compensating business processes. This is why I hope MongoDB is never engineered to support XA, as I'd hate to see such a move encourage good developers to do bad things.<br />
<br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;"><b>Footnote</b>: Even if your DBA refuses to unilaterally commit or rollback transactions, when the shit is hitting the fan, your database eventually will, thus violating XA/2PC. For example, in Oracle, the database will unilaterally decide to rollback all pending transactions, older than the default value of 24 hours (see "<a href="http://docs.oracle.com/cd/E24329_01/web.1211/e24377/trxman.htm#WLJTA177">Abandoning Transactions</a>" section of Oracle's documentation).</span><br />
<br />
<br />
<i>Song for today: The Greatest by <a href="https://en.wikipedia.org/wiki/Cat_Power">Cat Power</a></i><br />
<i><br /></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com8tag:blogger.com,1999:blog-1304066656993695443.post-1197349769881133322015-12-16T07:53:00.002+00:002020-03-01T20:58:24.262+00:00MongoDB's BI Connector and pushdown of SQL WHERE clauses<i>[<b>EDIT 05-Apr-2018</b>: MongoDB BI Connector version 2+ uses a much more rich and powerful approach for pushing down SQL clauses to the database - for more info see <a href="https://docs.mongodb.com/bi-connector/current/">here</a>]</i><br />
<br />
In previous posts I showed how to use SQL & ODBC to query data from MongoDB, via <a href="http://pauldone.blogspot.co.uk/2015/12/mongodbbiconctrwindowsodbc.html">Windows clients</a> and <a href="http://pauldone.blogspot.co.uk/2015/12/mongodbbiconctrlinuxodbc.html">Linux clients</a>. In this post, I want to explore what happens to the SQL statement when it is sent to the BI Connector and onto the MongoDB database. For example, is the SQL WHERE clause pushed down to the database to resolve?<br />
<br />
Again, for these tests, I've used the same <a href="https://data.gov.uk/dataset/anonymised_mot_test">MOT UK car test results data set</a> as my last two posts.<br />
<br />
As I wanted to get a better insight into what MongoDB is doing under the covers to process SQL queries, I used the Mongo Shell to enable profiling for the MongoDB database holding the MOT data.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPGbUWaR-v1lR4S_WPvbSxk8Yt0wu1dl2jxxxL21-utFJDHRxkKUIE-V2Whfi7kO2SoW56GRVSKrvk9DgNpNRLZMjE0OOES3RpB9kSaDuRy9Z4dM98poKYQ5IOAqBLtKrGNARMLzZnZ-s/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="174" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiPGbUWaR-v1lR4S_WPvbSxk8Yt0wu1dl2jxxxL21-utFJDHRxkKUIE-V2Whfi7kO2SoW56GRVSKrvk9DgNpNRLZMjE0OOES3RpB9kSaDuRy9Z4dM98poKYQ5IOAqBLtKrGNARMLzZnZ-s/s640/1.png" width="640" /></a></div>
<br />
Then on my Linux desktop client, using the <a href="http://pauldone.blogspot.co.uk/2015/12/mongodbbiconctrlinuxodbc.html">ODBC settings</a> I'd configured in my last post, I fired up <i>isql</i> ready to start issuing queries against the MOT data set, via the BI Connector.<br />
<br />
I submitted a SQL statement to query all cars with a recorded mileage of over 500,000 miles, selecting specific columns only.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">> SELECT make, model, test_mileage FROM testresults WHERE test_mileage > 500000;</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSv66qmAPfwPCNpD4PPharFILYL_Ytv1twvqm-lN20RJ7aIhKi-riTROjCkbn9YNrqxJ_zQ3671MhOqDe5Ps4QiHZFf5FwjeoanJL6z1BW5wYW8QmWY49rUcnU2Chf-i0YR8jkBNaARQ0/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="614" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiSv66qmAPfwPCNpD4PPharFILYL_Ytv1twvqm-lN20RJ7aIhKi-riTROjCkbn9YNrqxJ_zQ3671MhOqDe5Ps4QiHZFf5FwjeoanJL6z1BW5wYW8QmWY49rUcnU2Chf-i0YR8jkBNaARQ0/s640/2.png" width="640" /></a></div>
<br />
The results were correctly returned in <i>isql</i>, but I was more interested to see what MongoDB was asked to do on the server-side, to fulfil this request. So using the Mongo Shell I queried the system profile collection to show the last recorded entry, displaying the exact request that MongoDB had received as a result of the translated SQL query.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.system.profile.find({ns: "mot:testresults"}).sort({$natural: -1}).limit(1).pretty()</span><br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVG60McEpuQAv4WnJLrzgRyBsJhZwJrTwes189lGfvdY5bZhWJb5yUCgzN_nNnnlOJNnS3IQ_uWtbFsNhRCpzTbOOnaMY_8eeJC1Z6IRsPdssy6L2LLy5eWfsA4IOKf1XzHafrtc3loZg/s1600/3b.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="354" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiVG60McEpuQAv4WnJLrzgRyBsJhZwJrTwes189lGfvdY5bZhWJb5yUCgzN_nNnnlOJNnS3IQ_uWtbFsNhRCpzTbOOnaMY_8eeJC1Z6IRsPdssy6L2LLy5eWfsA4IOKf1XzHafrtc3loZg/s640/3b.png" width="640" /></a></div>
<br />
As you can see in the output, the BI Connector has indeed pushed down to the database, the WHERE clause, plus the projection to return only specific fields. The profiler output shows that this has been achieved by the BI Connector, by assembling an <a href="https://docs.mongodb.org/v3.0/core/aggregation-pipeline/">Aggregation Pipeline</a>.<br />
<div>
<br /></div>
<div>
This is great to see. Most of the work to process the SQL query is being done at the database level, reducing the amount of data (less rows and less columns) that is returned to the ODBC client for final processing, and also enabling database indexes to be leveraged for maximum performance.<br />
<br /></div>
<div>
<i>Song for today: End Come Too Soon by <a href="https://en.wikipedia.org/wiki/Wild_Beasts">Wild Beasts</a></i></div>
<div>
<br /></div>
Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com1tag:blogger.com,1999:blog-1304066656993695443.post-44215601937185914512015-12-15T09:36:00.000+00:002020-03-01T21:01:51.816+00:00Accessing MongoDB data using SQL / ODBC on Linux<i>[<b>EDIT 05-Apr-2018</b>: MongoDB BI Connector version 2+ uses a different mechanism for connecting to (not using a PostgreSQL driver) - for more info see <a href="https://docs.mongodb.com/bi-connector/current/">here</a>]</i><br />
<br />
In my previous <a href="http://pauldone.blogspot.ie/2015/12/mongodbbiconctrwindowsodbc.html">post</a> I showed how generic Windows clients can use ODBC to query data from MongoDB using the new <a href="https://www.mongodb.com/products/bi-connector">BI Connector</a>. I'm not really a Windows type person though and feel more at home with a Linux desktop. Therefore, in this post, I will show how easy it is to use ODBC from Linux (Ubuntu 14.04 in my case) to query MongoDB data using SQL. I've used the same <a href="https://data.gov.uk/dataset/anonymised_mot_test">MOT UK car test results data set</a> as my <a href="http://pauldone.blogspot.ie/2015/12/mongodbbiconctrwindowsodbc.html">last post</a>.<br />
<br />
First of all I needed to install the Linux ODBC packages plus the PostgreSQL ODBC driver from the package repository.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo apt-get install unixodbc-bin unixodbc odbc-postgresql</span><br />
<br />
Then I googled how to use ODBC and the PostgreSQL driver to query an ODBC data source. Surprisingly, I didn't find much quality information out there. However, looking at the contents of the "odbc-postgresql" package....<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ apt-file list odbc-postgresql</span><br />
<br />
...it showed some bundled documents including...<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">/usr/share/doc/odbc-postgresql/README.Debian</span><br />
<br />
Upon opening this text file I found pretty much everything I needed to know, to get going. I really should learn to RTFM more often!<br />
<br />
The next step, as documented in that README, was to register the PostgreSQL ANSI & Unicode ODBC drivers in the <i>/etc/odbcinst.ini</i> file, using a pre-supplied template file that contained the common settings.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span><span style="font-family: "courier new" , "courier" , monospace;">$ sudo odbcinst -i -d -f /usr/share/psqlodbc/odbcinst.ini.template</span><br />
<br />
Then I needed to create the <i>/etc/odbc.ini</i> file where data sources can be registered. Here I used a skeleton template again.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo cat /usr/share/doc/odbc-postgresql/examples/odbc.ini.template >> ~/.odbc.ini</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo mv .odbc.ini /etc/odbc.ini</span><br />
<br />
However, this particular template only includes dummy configuration settings, so I needed to edit this file to register the remote MongoDB BI Connector's details properly.<br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ sudo vi /etc/odbc.ini</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"><br /></span>
<span style="font-family: "courier new" , "courier" , monospace;">$ cat /etc/odbc.ini </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">[mot]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Description = MOT Test Data</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Driver = PostgreSQL Unicode</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Trace = No</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">TraceFile = /tmp/psqlodbc.log</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Database = mot </span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Servername = 192.168.43.173</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">UserName = mot</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Password = mot</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">Port = 27032</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ReadOnly = Yes</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">RowVersioning = No</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ShowSystemTables = No</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ShowOidColumn = No</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">FakeOidIndex = No</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">ConnSettings =</span><br />
<br />
<span style="font-size: x-small;">( forgive my poorly secure username/password of "mot/mot". ;) )</span><br />
<br />
Now I was ready to launch the "<b>isql</b>" (Interactive SQL) tool, that is bundled with the Linux/UNIX ODBC package, to be able to issue SQL queries against my remote MongoDB BI Connector data source.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ isql mot mot mot</span><br />
<br />
<span style="font-size: x-small;">( first param is data source name, second param is username, third param is password )</span><br />
<br />
As you can see in the screenshot below, using "isql" I was then able to easily issue arbitrary SQL commands against MongoDB and see the results.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUWI-wAEDUInbRcsteoIXYzQ_ixEMJXw8gthzAKTITXpHi5pxdWSSD3uv7dgzQi0EcKSjS_BYUEDWcaFeOXskSAOmGQRF3liYiiDlfO0_hZYq0RNrVqqUv-6pgORl36Y2x06eGd-OV9_4/s1600/shot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="416" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUWI-wAEDUInbRcsteoIXYzQ_ixEMJXw8gthzAKTITXpHi5pxdWSSD3uv7dgzQi0EcKSjS_BYUEDWcaFeOXskSAOmGQRF3liYiiDlfO0_hZYq0RNrVqqUv-6pgORl36Y2x06eGd-OV9_4/s640/shot.png" width="640" /></a></div>
<br />
<br />
And that's it. Very simple, and I now have a powerful command line SQL tool at my disposal for further experiments in the future. :-)<br />
<br />
<br />
<i>Song for today: Swallowtail by <a href="https://en.wikipedia.org/wiki/Wolf_Alice">Wolf Alice</a></i><br />
<br />
<br />Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-13934807943554372632015-12-11T12:29:00.000+00:002020-03-01T21:05:34.018+00:00Accessing MongoDB data from SQL / ODBC on Windows, using the new BI Connector<i>[<b>EDIT 05-Apr-2018</b>: MongoDB BI Connector version 2+ uses a different mechanism for connecting to (not using a PostgreSQL driver) - for more info see <a href="https://docs.mongodb.com/bi-connector/current/">here</a>]</i><br />
<br />
The latest enterprise version of MongoDB (<a href="https://www.mongodb.com/download-center#enterprise">3.2</a>) includes a new <a href="https://www.mongodb.com/products/bi-connector">BI Connector</a> to enable business intelligence, analytics and reporting tools, that only "speak" <a href="https://en.wikipedia.org/wiki/SQL">SQL</a>, to access data in a MongoDB database, using <a href="https://en.wikipedia.org/wiki/Open_Database_Connectivity">ODBC</a>. Most of the <a href="https://www.youtube.com/watch?v=0kwopDp0bmg">examples</a> published so far show how to achieve this using rich graphical tools, like Tableau. Therefore, I thought it would be useful to show here that the data is accessible from any type of tool, that is capable of issuing SQL commands via an ODBC driver. Even from Microsoft's venerable Excel spreadsheet application. Believe it or not, I still come across organisations out there that are using Excel to report on the state of their business!<br />
<br />
For my example, I loaded a MongoDB database with the anonymised <a href="https://data.gov.uk/dataset/anonymised_mot_test">MOT tests results data</a>, that the <a href="https://www.gov.uk/government/statistical-data-sets/mot-testing-data-for-great-britain">UK government makes freely available to use</a>. As an explanation for non-residents of the UK, <a href="https://en.wikipedia.org/wiki/MOT_test">MOT tests</a> are the annual inspections that all UK road-going cars and other vehicles have to go through, to be legal and safe. There are millions of these car test records recorded every year, and they give a fascinating insight into the types and ages of cars people choose to drive in the UK.<br />
<br />
First I loaded the <a href="https://en.wikipedia.org/wiki/Comma-separated_values">CSV</a> file based MOT data sets into a MongoDB 3.2 database, using a small Python script I wrote, with each document representing a test result for a specific owner's car for a specific year. Below is an example of what one test result document looks like in MongoDB:<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0URGr-NsnjQsWJN8iT4FB_h3QYCJ8PeIHL1z6H5sSMqjnr6jSr1ssuN-P88I-Ej7ZvC3_-h27LHppfvn8NHo82JdeuT9N8cfmEIDxEevtEsfyDkGN1f5bI4xVVsh-HUoWxNGFoIUlnW0/s1600/0.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="433" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg0URGr-NsnjQsWJN8iT4FB_h3QYCJ8PeIHL1z6H5sSMqjnr6jSr1ssuN-P88I-Ej7ZvC3_-h27LHppfvn8NHo82JdeuT9N8cfmEIDxEevtEsfyDkGN1f5bI4xVVsh-HUoWxNGFoIUlnW0/s640/0.png" width="640" /></a></div>
<br />
I then followed the online <a href="https://docs.mongodb.org/manual/products/bi-connector/">MongoDB BI Connector documentation</a> to configure a BI Connector server to listen for ODBC requests for the "mot" database and translate these to calls to the underlying MongoDB "testresults" collection. I just used the default DRDL ("Document Relational Definition Language") schema file that was automatically generated by the "mongodrdl" command line utility (a utility bundled with the BI Connector).<br />
<br />
Then, on a separate desktop virtual machine running Windows 10, I <a href="http://www.postgresql.org/ftp/odbc/versions/msi/">downloaded</a> the latest PostgreSQL ODBC driver installer for Windows and installed it.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqiF9u4dICuVYH4K8-bPV8hea-qhuHF656N9PK9RPXbTxAutLyrQzdQth_O621nFBiuHUESt3ZjXhPHRlCWjG1hBr5E4abzsr_AmJg3M-xsEN7cCwqBiMIZVauxqZH7qPI0yj13PGTKes/s1600/1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="411" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiqiF9u4dICuVYH4K8-bPV8hea-qhuHF656N9PK9RPXbTxAutLyrQzdQth_O621nFBiuHUESt3ZjXhPHRlCWjG1hBr5E4abzsr_AmJg3M-xsEN7cCwqBiMIZVauxqZH7qPI0yj13PGTKes/s640/1.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ0dqUTTgwX6Ot1yGXes-tIDgNlSiAoTKUds2S8e0T02F8BTKDFlzTy8gkA9zfeuBbb36jzhW1IT2CqxrchhxOLXAE4MbEPMwQkI1XvAdFAhbqNJYjiWiFKJNq_f486679X0RJ8j_qY7A/s1600/2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="481" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZ0dqUTTgwX6Ot1yGXes-tIDgNlSiAoTKUds2S8e0T02F8BTKDFlzTy8gkA9zfeuBbb36jzhW1IT2CqxrchhxOLXAE4MbEPMwQkI1XvAdFAhbqNJYjiWiFKJNq_f486679X0RJ8j_qY7A/s640/2.png" width="640" /></a></div>
<br />
With the ODBC driver installed, I then proceeded to define a Windows ODBC Data Source to reference the MOT database that I was exposing via by the BI Connector (running on a remote machine).<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGkPbs-rRMd49GwwfE28Mk-8Vfunr9lRFglB_6OIn2BBDhcXCUP47z_AjOt67-eDLD8azOFyEfwhWC_vQwMabI_oN-iWAZ5kyauohQoYpJXusDuMA8rxvwx1atTHHT7W9pjO1WqPaXmKU/s1600/3.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGkPbs-rRMd49GwwfE28Mk-8Vfunr9lRFglB_6OIn2BBDhcXCUP47z_AjOt67-eDLD8azOFyEfwhWC_vQwMabI_oN-iWAZ5kyauohQoYpJXusDuMA8rxvwx1atTHHT7W9pjO1WqPaXmKU/s640/3.png" width="360" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaQo_-3jyhxRQ4kKZmcANF9pvxixS9bRUHA8ZIQr_PEP1ntLK6ONQrBlqCA-yyxNcKmTerBYrubI6kFHdgFyKp_QmsecftJJ800eO2fNN9mKOSgHa7D9lVWqhdfxwmlB9XXO8s4aaLYpY/s1600/4.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgaQo_-3jyhxRQ4kKZmcANF9pvxixS9bRUHA8ZIQr_PEP1ntLK6ONQrBlqCA-yyxNcKmTerBYrubI6kFHdgFyKp_QmsecftJJ800eO2fNN9mKOSgHa7D9lVWqhdfxwmlB9XXO8s4aaLYpY/s640/4.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdUqQISJWD_53bcbNuRjJgXJwlHYJkpm5JtXqUaUN13IdaHC3fEwOMkuNodSwhq3ffeeLqkpKcOzvOjhG549wa3BVYO9jjOxM2-ttBP-p-FH9E6JsqsP7fcj4Ukd0jkWBIawKYd7N7nh0/s1600/5.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="456" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjdUqQISJWD_53bcbNuRjJgXJwlHYJkpm5JtXqUaUN13IdaHC3fEwOMkuNodSwhq3ffeeLqkpKcOzvOjhG549wa3BVYO9jjOxM2-ttBP-p-FH9E6JsqsP7fcj4Ukd0jkWBIawKYd7N7nh0/s640/5.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkuOA16PWxi62CmhQO7CGsvoDT-adjj2YdPpPMoxC4B-UQOYStdUoET0vzSKwOuXLbZ0KE3llRDzEIK8oU7YPAa_B1dr2ew2aLkNGWfoS7z99R2KbO1occX3q5h24uRkpQ4BHkGkyHQLY/s1600/6.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="460" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkuOA16PWxi62CmhQO7CGsvoDT-adjj2YdPpPMoxC4B-UQOYStdUoET0vzSKwOuXLbZ0KE3llRDzEIK8oU7YPAa_B1dr2ew2aLkNGWfoS7z99R2KbO1occX3q5h24uRkpQ4BHkGkyHQLY/s640/6.png" width="640" /></a></div>
<br />
By default the BI Connector (running on a machine with IP address 192.168.1.174, in my case), listens on port 27032. Before hitting the Save button, I hit the Test button to ensure that the Windows client could make a successful ODBC connection to the BI Connector.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4NnNfBFvOZuD-LbhFETOTGw4NUQwN_zAkMu8LFOp5qs9T4diWB517itQcmWdmvAFAKLed6Sj9LGapw8-CB4H6flsR_JWGo4R_z51ls_mz86COcNN_8cx636mD2D1YhdT9GPtLo3XX9LA/s1600/7.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="460" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi4NnNfBFvOZuD-LbhFETOTGw4NUQwN_zAkMu8LFOp5qs9T4diWB517itQcmWdmvAFAKLed6Sj9LGapw8-CB4H6flsR_JWGo4R_z51ls_mz86COcNN_8cx636mD2D1YhdT9GPtLo3XX9LA/s640/7.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuqYVyS-rWQulyJSQptS_scpco-XPN25Kabb-SbWCOjWZ7uQzXkXZXUJ8MsC5Ert-lEnto4Hxxwfn1P_8dJkGsyCGWDTnZxFSOUUlqWS2PsUFWE7JkjC-bi5Qx7J3JQNprMthLL0U5xDM/s1600/8.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="459" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuqYVyS-rWQulyJSQptS_scpco-XPN25Kabb-SbWCOjWZ7uQzXkXZXUJ8MsC5Ert-lEnto4Hxxwfn1P_8dJkGsyCGWDTnZxFSOUUlqWS2PsUFWE7JkjC-bi5Qx7J3JQNprMthLL0U5xDM/s640/8.png" width="640" /></a></div>
<br />
With the new ODBC Data Source now configured (shown in screenshot above), I then launched Microsoft Excel so that I could use this new data source to explore the MOT test data.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZS6zyZ6cGcpbj7xQQlo8kbpTiXjo_5LXHSJB-JBibwzlNqXk1_71Kte31Km0OG3uZ_iEstO6o9WWC6S2YTJhGMGbE0YAcrA6KP6b5OehrjIXD1N3oMYTTYSZDk6mboz9KOcFxuXIMNqw/s1600/10.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="465" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiZS6zyZ6cGcpbj7xQQlo8kbpTiXjo_5LXHSJB-JBibwzlNqXk1_71Kte31Km0OG3uZ_iEstO6o9WWC6S2YTJhGMGbE0YAcrA6KP6b5OehrjIXD1N3oMYTTYSZDk6mboz9KOcFxuXIMNqw/s640/10.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJG0Q2_IEnb_YpFx08di8Gk-zmXlN-Te6mcH6y1HLVlahPbPrfLS22fVg2Cz2SOrZCquKQGqVcsfPEP4pivwWal9FcXcZuziwDZ16nlJko5X36RJgcuJxYuftN0pQdwb4Di-iLsi19cLE/s1600/11.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="348" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhJG0Q2_IEnb_YpFx08di8Gk-zmXlN-Te6mcH6y1HLVlahPbPrfLS22fVg2Cz2SOrZCquKQGqVcsfPEP4pivwWal9FcXcZuziwDZ16nlJko5X36RJgcuJxYuftN0pQdwb4Di-iLsi19cLE/s640/11.png" width="640" /></a></div>
<br />
Excel's standard query wizard was able to use the ODBC data source to discover the MongoDB collection's "schema". I chose to include all the "fields" in the query.<br />
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
</div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoQQCOonszpeCPKMTekYhdkO4G4rcXSJnhVDVMsjdkOu0uhuq9rhy_oXowFgOhdecjf9HAOCLhMRkKRmBytw7fDdVqZt66KBZfWJ3Z45BQtslv9S-lDe35DBGwIOkL28G8XWuk3ax0oKQ/s1600/12.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="412" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhoQQCOonszpeCPKMTekYhdkO4G4rcXSJnhVDVMsjdkOu0uhuq9rhy_oXowFgOhdecjf9HAOCLhMRkKRmBytw7fDdVqZt66KBZfWJ3Z45BQtslv9S-lDe35DBGwIOkL28G8XWuk3ax0oKQ/s640/12.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1qgFHL17JhP-bwYFypOClc1FK5skhexbr9EGIC150dmM0JSdcfqsngCiiHOpVMfs8N1dRiVpY_QRxuBC1KoBJ_Wk8evypDfIMj70q0sK10-rv1Ey4f6Z0Vgt3dQQBAhq2rkGNbrCbDbQ/s1600/13.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="408" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi1qgFHL17JhP-bwYFypOClc1FK5skhexbr9EGIC150dmM0JSdcfqsngCiiHOpVMfs8N1dRiVpY_QRxuBC1KoBJ_Wk8evypDfIMj70q0sK10-rv1Ey4f6Z0Vgt3dQQBAhq2rkGNbrCbDbQ/s640/13.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicfdU2ZVAWwVlgkvV22-ZesrT9kVgw9g8Nonc4TIkdEWlTnxUgj0CxzbjLgAyrtE6YXAFOXivadcM0CpMI9En20CQ9Ixb0TRZuUju4LJdB5DJULGJmrT5aoQOw5eQiXFOw-VpXji_AGJs/s1600/14.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="403" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEicfdU2ZVAWwVlgkvV22-ZesrT9kVgw9g8Nonc4TIkdEWlTnxUgj0CxzbjLgAyrtE6YXAFOXivadcM0CpMI9En20CQ9Ixb0TRZuUju4LJdB5DJULGJmrT5aoQOw5eQiXFOw-VpXji_AGJs/s640/14.png" width="640" /></a></div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: left;">
I thought it would be useful to ask for the MOT test results to be ordered by Test Year, followed by Car Make, followed by Car Model.</div>
<div class="separator" style="clear: both; text-align: center;">
<br /></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUh7HnWmDo4sSmYm0t25uzKxjh7EjPNuGNdBrOMxtg_SeRkObsckLjZcR2PM4orRoA7irol26jmLqYcUs03RCKh4wfU5VYyly7wjvg-QkVPx-coyvbLwhfDC0TnyVB2rqfACFycpPpePM/s1600/15.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="392" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgUh7HnWmDo4sSmYm0t25uzKxjh7EjPNuGNdBrOMxtg_SeRkObsckLjZcR2PM4orRoA7irol26jmLqYcUs03RCKh4wfU5VYyly7wjvg-QkVPx-coyvbLwhfDC0TnyVB2rqfACFycpPpePM/s640/15.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgipzjCWYUn5P6tksPjGJrJwt-Tv1XrfuhEUmb8aPnejqIROzL_v08gKARu_g9cme6fTbOW0zrz25xLyUwaCLjSH_QjUgAbZ5fq-4ZBDS_mlNLQQ7jZ8kUnKyYwzEsycT8lB1byppcTnYA/s1600/16.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="409" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgipzjCWYUn5P6tksPjGJrJwt-Tv1XrfuhEUmb8aPnejqIROzL_v08gKARu_g9cme6fTbOW0zrz25xLyUwaCLjSH_QjUgAbZ5fq-4ZBDS_mlNLQQ7jZ8kUnKyYwzEsycT8lB1byppcTnYA/s640/16.png" width="640" /></a></div>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNADRinyRWPRZlgv-h3UfU3lOlgGIwVK3PfpO3m3wBV-TjpmDDbBnDoCOIXLyvlsWit5p1ai_n4FAOoaeKbnA8kWHbwftU2GUyzxKC95WwAYFNCUi7s2G7SxuJhprrsGCXuJr8l4-iIUo/s1600/17.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="563" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgNADRinyRWPRZlgv-h3UfU3lOlgGIwVK3PfpO3m3wBV-TjpmDDbBnDoCOIXLyvlsWit5p1ai_n4FAOoaeKbnA8kWHbwftU2GUyzxKC95WwAYFNCUi7s2G7SxuJhprrsGCXuJr8l4-iIUo/s640/17.png" width="640" /></a></div>
<br />
Finally, upon pressing OK, Excel presented me with the results of the SQL/ODBC query, run directly against the MOT test data, sourced from the MongoDB collection.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSnC3uQgsAa9wrbSuR1dVt1sZSLUcYduQO9X0ctOoKEXb1cfJM05sXgno5ed4SSbqA5CjX0kRrxNzZVbDvw6jLzvRh8Z2v74nqnbC9x9RHGylejzR0j5GuNX-4nKMDImiI4q0Rzp4WHoE/s1600/18.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="457" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjSnC3uQgsAa9wrbSuR1dVt1sZSLUcYduQO9X0ctOoKEXb1cfJM05sXgno5ed4SSbqA5CjX0kRrxNzZVbDvw6jLzvRh8Z2v74nqnbC9x9RHGylejzR0j5GuNX-4nKMDImiI4q0Rzp4WHoE/s640/18.png" width="640" /></a></div>
<br />
Excel, then gave me many options, including settings to say whether to periodically refresh the data from source, and how often. I was also able to open Microsoft's built-in Query Builder tool to modify the query and execute again<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR0USe1dq7F82X7_Z2O1VCmtxFKK8kIjPTNucdb8LnxU-G8Q5zmRhRtpvHTQkCrB_UVtpPKY-qkQ1A5XlYYN1Ep37rFg9MewvXRVyxIRLLzi8Wlem3IHzcDu7vEmNElrS8WMcsgTvIh30/s1600/19.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="433" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiR0USe1dq7F82X7_Z2O1VCmtxFKK8kIjPTNucdb8LnxU-G8Q5zmRhRtpvHTQkCrB_UVtpPKY-qkQ1A5XlYYN1Ep37rFg9MewvXRVyxIRLLzi8Wlem3IHzcDu7vEmNElrS8WMcsgTvIh30/s640/19.png" width="640" /></a></div>
<br />
That's pretty much it. It's straight forward to configure a Windows client to access data from MongoDB, via MongoDB's new BI Connector, using ODBC.<br />
<br />
<br />
<i>Song for today: Dust and Disquiet by <a href="https://en.wikipedia.org/wiki/Caspian_(band)">Caspian</a></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com2tag:blogger.com,1999:blog-1304066656993695443.post-54075288977672540062014-09-26T10:10:00.000+01:002020-03-01T21:09:25.874+00:00Tracking Versions in MongoDBI'm honoured to have been asked by <a href="http://askasya.com/">Asya</a> to contribute a <a href="http://askasya.com/post/revisitversions">guest post</a> to her series on Tracking Versions in MongoDB.<br />
<br />
Here's Asya's full series on the topic, which I recommend reading in order:<br />
<br />
<ol>
<li><a href="http://askasya.com/post/trackversions">How to Track Versions with MongoDB</a></li>
<li><a href="http://askasya.com/post/mergeshapes">How to Merge Shapes with Aggregation Framework</a></li>
<li><a href="http://askasya.com/post/bestversion">Best Versions with MongoDB</a></li>
<li><a href="http://askasya.com/post/revisitversions">Further Thoughts on How to Track Versions with MongoDB</a> (my guest post)</li>
</ol>
<br />
<br />
<br />
<i>Song for today: Someday I Will Treat You Good by <a href="http://en.wikipedia.org/wiki/Sparklehorse">Sparklehorse</a></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com0tag:blogger.com,1999:blog-1304066656993695443.post-42084286479357419632014-09-11T11:54:00.002+01:002020-03-01T21:14:14.413+00:00Java Using SSL To Connect to MongoDB With Access Control<br />
In this post I've documented the typical steps you need to enable a Java application to connect to a MongoDB database over SSL. Specifically, I show the so-called "one-way SSL" pattern, where the server is required to present a certificate to the client but the client is not required to present a certificate to the server. Regardless of authentication, communication between client and server is encrypted in both directions. In this example, the client actually authenticates with the database, albeit using a username/password rather than presenting a certificate. In addition, I also show how the client application can run operations against a database that has access control rules defined for it.<br />
<br />
<b>Note</b>: Ensure you are running the <a href="http://www.mongodb.com/subscription/downloads">Enterprise version of MongoDB</a> to be able to configure SSL.<br />
<br />
<br />
<h3>
1. Generate Key+Certificate and Configure With MongoDB For SSL</h3>
<br />
Create a key and certificate:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ su -</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cd /etc/ssl/</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ openssl req -new -x509 -days 365 -nodes -out mongodb-cert.crt -keyout mongodb-cert.key</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ cat mongodb-cert.key mongodb-cert.crt > mongodb.pem</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ exit</span><br />
<br />
Add the following entries to <a href="http://docs.mongodb.org/manual/reference/configuration-options/">mongodb.conf</a> (or equivalent if <a href="http://docs.mongodb.org/manual/reference/program/mongod/">setting command line options</a>):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">sslOnNormalPorts=true</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">sslPEMKeyFile=/etc/ssl/mongodb.pem</span><br />
<br />
Start the <i>mongod</i> database server.<br />
<br />
Test the SSL connection from the Mongo shell.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo --ssl</span><br />
<div>
<br />
<br /></div>
<h3>
2. Configure Java Client For SSL</h3>
<br />
Create a new Java trust store in the local application's root directory, importing the certificate generated from the previous section. This is necessary because in this example a self-signed certificate is used.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ keytool -import -alias "MongoDB-cert" -file "/etc/ssl/mongodb-cert.crt" -keystore truststore.ts -noprompt -storepass "mypasswd"</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">$ keytool -list -keystore truststore.ts -storepass mypasswd</span><br />
<br />
In the client application's Java code, add the "ssl=true" parameter to the MongoDB URI to tell it to use an SSL connection, eg:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">MongoClientURI uri = new MongoClientURI("mongodb://localhost:27017/test<b>?ssl=true</b>");</span><br />
<br />
Modify the command line or script that runs the 'Java' executable for the client application, and add the following JVM command line property. This will allow the application to validate the server's certificate against the new trust store, when using SSL.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">-Djavax.net.ssl.trustStore=truststore.ts</span><br />
<br />
Run the Java client application to test that the SSL connection works.<br />
<br />
<br />
<h3>
3. Configure MongoDB database with Access Control Rules</h3>
<br />
Using the Mongo shell, connect to the database, and define an 'administrator user', plus a 'regular' user with an access control rule defined to enable it to have read/write access to a database (called 'test' in this example).<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo --ssl</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> use admin</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.addUser({user: "admin", pwd: "admin", roles: [ "userAdminAnyDatabase"]})</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> use test</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">> db.addUser({user: "paul", pwd: "password", roles: ["readWrite", "dbAdmin"]})</span><br />
<br />
Add the following entry to <a href="http://docs.mongodb.org/manual/reference/configuration-options/">mongodb.conf</a> (or equivalent if <a href="http://docs.mongodb.org/manual/reference/program/mongod/">setting command line options</a>):<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">auth=true</span><br />
<br />
Restart the <i>mongod</i> database server to pick up this change.<br />
<br />
Test the SSL connection to the 'test' database with username/password authentication, from the Mongo shell.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">$ mongo test --ssl -u paul -p password</span><br />
<br />
If the user specified doesn't have permissions to read collections in the database, an error similar to below will occur when trying to run <i>db.mycollection,find()</i>, for example.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">error: { "$err" : "not authorized for query on test.mycollection", "code" : 13 }</span><br />
<br />
<br />
<h3>
4. Configure Java Client To Authenticate</h3>
<br />
Modify the client application code to include the username and password in the URL, eg:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">MongoClientURI uri = new MongoClientURI("mongodb://<b>paul:password@</b>localhost:27017/test?ssl=true");</span><br />
<br />
Run the Java client application to test that using SSL with username/password authentication works and has the rights to access the sample database.<br />
<br />
If the user specified doesn't have correct permissions, an exception similar to below will occur.<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">Exception in thread "main" com.mongodb.MongoException: not authorized for query on test.system.namespaces</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> at com.mongodb.MongoException.parse(MongoException.java:82)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:292)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:273)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> at com.mongodb.DB.getCollectionNames(DB.java:400)</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> at MongoTest.main(MongoTest.java:25)</span><br />
<br />
<br />
<b>Note: </b>In a real Java application, you would invariably build the <a href="http://api.mongodb.org/java/current/com/mongodb/MongoClientURI.html">URL</a> <a href="http://api.mongodb.org/java/current/com/mongodb/MongoCredential.html">dynamically</a>, to avoid hard-coding a username and password in clear text in code.<br />
<div>
<br /></div>
<br />
<br />
<br />
<i>Song for today: My Sister in 94 by <a href="http://en.wikipedia.org/wiki/The_Paradise_Motel">The Paradise Motel</a></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com2tag:blogger.com,1999:blog-1304066656993695443.post-47227772523827972652014-05-02T14:45:00.001+01:002020-03-01T21:18:20.287+00:00MongoDB Connector for Hadoop with Authentication - Quick Tip<br />
If you are using the <a href="http://docs.mongodb.org/ecosystem/tools/hadoop">MongoDB Connector for Hadoop</a> and you have enabled authentication on your MongoDB database (eg. <i><b>auth=true</b></i>) you may find that you are prevented from getting data in to or out of the database.<br />
<br />
You may have provided the username and password to the connector (eg. <i>mongo.input.uri = "mongodb://<b>myuser:mypassword@</b>host:27017/mytestdb.mycollctn"</i>), for an Hadoop Job that pulls data from the database. The connector will authenticate to the database successfully, but early in in the job run, the job will fail with an error message similar to the following:<br />
<br />
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">14/05/02 13:17:01 ERROR util.MongoTool: Exception while executing job...<u></u><u></u></span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">java.io.IOException: com.mongodb.hadoop.splitter.<wbr></wbr><b>SplitFailedException</b>: <b>Unable to calculate input splits</b>: need to login<u></u><u></u></span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> at com.mongodb.hadoop.<wbr></wbr>MongoInputFormat.getSplits(<wbr></wbr>MongoInputFormat.java:53)<u></u><u></u></span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> at org.apache.hadoop.mapreduce.<wbr></wbr>JobSubmitter.writeNewSplits(<wbr></wbr>JobSubmitter.java:493)<u></u><u></u></span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> at org.apache.hadoop.mapreduce.<wbr></wbr>JobSubmitter.writeSplits(<wbr></wbr>JobSubmitter.java:510)<u></u><u></u></span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> at org.apache.hadoop.mapreduce.<wbr></wbr>JobSubmitter.<wbr></wbr>submitJobInternal(<wbr></wbr>JobSubmitter.java:394)<u></u><u></u></span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;"> at org.apache.hadoop.mapreduce.<wbr></wbr>Job$10.run(Job.java:1295)</span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; margin: 0cm 0cm 0.0001pt;">
<span style="font-family: "courier new" , "courier" , monospace; font-size: x-small;">......</span></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0cm 0cm 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; margin: 0cm 0cm 0.0001pt;">
This is because the connector needs to run the MongoDB-internal <a href="http://docs.mongodb.org/manual/reference/command/splitVector">splitVector</a> DB command, under the covers, to work out how to split the MongoDB data up into sections ready to distribute across the Hadoop Cluster. However, by default, you are unlikely to have given sufficient privileges to the user, used by the connector, to allow this DB command to be run. This issue can be simulated easily by opening a mongo shell against the database, authenticating with your username and password and then running the splitVector command manually. For example:</div>
<div class="MsoNormal" style="background-color: white; color: #222222; font-family: Calibri, sans-serif; font-size: 11pt; margin: 0cm 0cm 0.0001pt;">
<br /></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;">> var result = db.runCommand({<b>splitVector</b>: 'mytestdb.mycollctn', </span><span style="color: #222222; font-family: "courier new" , "courier" , monospace;">keyPattern: {_id: 1}, </span><br />
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;"> maxChunkSizeBytes: 32000000})</span></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;">> result</span></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;">{</span></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>"ok" : 0,</span></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>"errmsg" : "<b>not authorized on mytestdb to execute command</b> { </span><br />
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;"> splitVector: "mytestdb.mycollctn", </span><span style="color: #222222; font-family: "courier new" , "courier" , monospace;">keyPattern: { _id: 1.0 },</span><br />
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;"> maxChunkSizeBytes: 32000000.0 }",</span></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;"><span class="Apple-tab-span" style="white-space: pre;"> </span>"code" : 13</span></div>
<div class="MsoNormal" style="background-color: white; margin: 0cm 0cm 0.0001pt;">
<span style="color: #222222; font-family: "courier new" , "courier" , monospace;">}</span></div>
<br />
To address this issue, you first need to use the mongo shell, authenticated as your administration user, and run the <b>updateUser</b> command to give the connector user the <b>clusterManager</b> role, to enable the connector to run the DB commands it requires. For example:<br />
<br />
<span style="font-family: "courier new" , "courier" , monospace;">use mytestdb</span><br />
<span style="font-family: "courier new" , "courier" , monospace;">db.<b>updateUser</b>("myuser", {</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> roles : [</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> { role: "readWrite", db: "mytestdb" },</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> { role : "<b>clusterManager</b>", db : "<b>admin</b>" }</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> ]</span><br />
<span style="font-family: "courier new" , "courier" , monospace;"> })</span><br />
<br />
After this, your Hadoop jobs with the connector should run fine.<br />
<br />
<span style="font-size: x-small;"><br /></span>
<span style="font-size: x-small;">Note: In my test, I ran Cloudera CDH version 5, MongoDB version 2.6 and Connector version 1.2 (built with target set to 'cdh4').</span><br />
<br />
<br />
<i>Song for today: Spanish Sahara by <a href="http://en.wikipedia.org/wiki/Foals">Foals</a></i>Paul Donehttp://www.blogger.com/profile/09556312012162376804noreply@blogger.com2United Kingdom55.378051 -3.4359729999999912.188224499999997 -86.05316049999999 90 79.18121450000001