1. You set clear expectations up front, including priority in queue, terms, definitions, rules, appropriate granularity, and timing, and the person or team really missed them without two-way collaborative communication regarding why. That’s on them.

2. You didn’t set clear expectations. That’s on you.

I’ve seen #2 in out in the wild, much more than #1. You?

Next: Learnings

Share Ten Moments

touch fam && echo "yo!" > fam && mkdir -p hawaii && mv fam hawaii/

I used Docker to containerize my app. This means that I package the application and its dependencies into a Docker image, which can then be run consistently on any infrastructure. This eliminates the "it works on my machine" problem, because the application will run the same way in a container regardless of the underlying host system. Docker also integrates well with many popular continuous integration/continuous deployment (CI/CD) systems, making it a key part of any deployment pipeline.

Next: Domain Registrar

Share Ten Moments

touch fam && echo "yo!" > fam && mkdir -p hawaii && mv fam hawaii/

IAM (Identity and Access Management) is a web service that helps you securely control access to AWS resources. It controls who can sign in to your AWS infrastructure and who is authorized to use the resources (EC2, S3 buckets, and so on). It can be a real pain in the butt, but it provides essential functionality to keep resources and applications secure.

S3 (Simple Storage Service) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases.

CloudFront is a fast content delivery network (CDN) service that securely delivers data, videos, applications, and APIs to customers globally with low latency, high transfer speeds, all within a developer-friendly environment.

EC2 (Elastic Compute Cloud) is a web service that provides resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier by providing a simple interface that allows you to obtain and configure capacity with minimal friction. EC2 instances are created from "blueprints" called AMIs. I started creating my own custom AMIs in 2009, so that all required code, packages and OS-level customizations were ready to go as soon as the instance was up and running. These days that's typically done with containers (e.g. Docker).

ECR (Elastic Container Registry) is a fully-managed Docker container registry that makes it easy for developers to store, manage, and deploy Docker container images.

ECS (Elastic Container Service) is a highly scalable, high-performance container orchestration service that supports Docker containers and allows you to easily run and scale containerized applications on AWS. Getting it working requires jumping through some challenging IAM hoops, but once it's working, it's a fast way to run applications and services.

Fargate is a serverless compute engine for containers that works with both Amazon Elastic Container Service (ECS) and Amazon Elastic Kubernetes Service (EKS). Fargate makes it easy for you to focus on building your applications; I've done very little with it directly, in the lake_washington_and_sammamish_temps project.

Route 53 is a scalable and highly available Domain Name System (DNS) web service. It is designed to give developers and businesses an extremely reliable and cost-effective way to route end users to Internet applications. I've used it since 2009 for all of my DNS configuration needs.

CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers. CloudWatch provides you with data and actionable insights to monitor your applications, respond to system-wide performance changes, optimize resource utilization, and get a unified view of operational health. I find it to be kind of clunky.

EventBridge is a serverless event bus that makes it easy to connect applications together using data from your own applications, integrated Software-as-a-Service (SaaS) applications, and AWS services.

Certificate Manager is a service that lets you easily provision, manage, and deploy public and private Secure Sockets Layer/Transport Layer Security (SSL/TLS) certificates for use with AWS services and your internal connected resources. It's a pretty incredible, far cry from how certificate management had to be done years ago. Once your certificate is created by AWS, all of the renewals are handled for you automatically, making those browser warnings about expired certs a thing of the not-so-distant past.

Secrets Manager protects access to your applications, services, and IT resources. This service enables you to easily rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle. Authentication credential rotation can be a real pain, and AWS has made it very easy to make sure it's done on a regular schedule.

All for the wonderful low price of: reasonably inexpensive, for the lake_washington_and_sammamish_temps project. LMGBTY WRT monthly costs.

Next: Containerization

Share Ten Moments

touch fam && echo "yo!" > fam && mkdir -p hawaii && mv fam hawaii/

JSON (JavaScript Object Notation) is a popular data interchange format that's easy to read and write for humans and easy to parse and generate for machines. It's based on JavaScript syntax but is used in many programming languages beyond JavaScript. In my project, I might be using JSON to store configuration data, to communicate between a client and a server, or to store complex data structures for my application.

YAML (YAML Ain't Markup Language) is a human-friendly data serialization standard that's commonly used for configuration files, but can be used in many applications where data is being stored or transmitted. Like JSON, it's easy for humans to read and write. It also supports complex data structures. In my project, I might be using YAML for configuration files, especially if those files need to be edited by humans. YAML's ability to express complex data structures and its more human-friendly syntax can make it a better choice than JSON for these types of files.

All for the wonderful low price of: free.

Next: Cloud Services via Amazon Web Services (AWS)

Share Ten Moments

touch fam && echo "yo!" > fam && mkdir -p hawaii && mv fam hawaii/

PyTest is a popular Python testing framework that simplifies the process of writing and running tests. With its easy-to-write syntax, you can write tests quickly and without the boilerplate code that some other testing frameworks require. PyTest also supports a wide range of advanced testing scenarios, including parameterized tests and fixtures. It also can output the percentage of code covered by unit tests, in HTML format for quick and easy reference within your browser. In my workflow, I'v integrated PyTest with a Git pre-commit hook. This means that my tests are automatically run every time I commit changes to my Git repository. If any test fails, the commit is aborted, ensuring that I don't accidentally commit broken code. This setup helps maintain the quality and reliability of my codebase.

All for the wonderful low price of: free.

Next: Markup Languages

Share Ten Moments

touch me && echo "what!?" > me && mkdir -p hawaii && mv me hawaii/

Pandas is a software library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. It is free software released under the three-clause BSD license. The name is derived from the term "panel data", an econometrics term for data sets that include observations over multiple time periods for the same individuals.

Boto is the Amazon Web Services (AWS) SDK for Python. It allows Python developers to write software that makes use of services like Amazon S3, Amazon EC2, and others. With Boto, you can create, configure, and manage AWS services using Python code. This can be very useful for automating tasks, such as setting up infrastructure or deploying applications. Boto supports a wide range of AWS services, making it a key tool for anyone working with AWS.

All for the wonderful low price of: free.

Next: Unit tests and Test Coverage

Share Ten Moments

touch me && echo "what!?" > me && mkdir -p hawaii && mv me hawaii/

I absolutely love Python.

Python has long been my language of choice for a variety of reasons. First and foremost, Python's syntax is clean and easy to understand, which makes it a great language for beginners. But don't let its simplicity fool you - Python is a powerful and versatile language that's used in everything from web development to data science to machine learning. Python's philosophy is "There should be one-- and preferably only one --obvious way to do it", which leads to more readable and maintainable code.

Python also has a rich ecosystem of libraries and frameworks. Whether you're working with data (Pandas, NumPy), building a web app (Django, Flask), or delving into machine learning (scikit-learn, TensorFlow), there's likely a Python library that can make your job easier. Python's extensive standard library, often described as "batteries included," means that often you don't even need to install anything to start building.

The Zen of Python, a collection of 19 "guiding principles", really encapsulates why I love Python. If you run this command in a Python interpreter:

import this

You'll get:

The Zen of Python, by Tim Peters

Beautiful is better than ugly.
Explicit is better than implicit.
Simple is better than complex.
Complex is better than complicated.
Flat is better than nested.
Sparse is better than dense.
Readability counts.
Special cases aren't special enough to break the rules.
Although practicality beats purity.
Errors should never pass silently.
Unless explicitly silenced.
In the face of ambiguity, refuse the temptation to guess.
There should be one-- and preferably only one --obvious way to do it.
Although that way may not be obvious at first unless you're Dutch.
Now is better than never.
Although never is often better than *right* now.
If the implementation is hard to explain, it's a bad idea.
If the implementation is easy to explain, it may be a good idea.
Namespaces are one honking great idea -- let's do more of those!

It's a great reminder of what makes Python special, and a guide to writing good Python code.

As for BASH, I've already talked about why I love it so much: solid, simple commands that can be chained together reliably to produce powerful outcomes.

All for the wonderful low price of: free.

Next: Noteable Python Libraries

Share Ten Moments

touch me && echo "beer" > me && mkdir -p hawaii && mv me hawaii/

Git is a distributed version control system that has become the de facto standard for source code management in the software industry. It allows multiple developers to work on the same codebase without stepping on each other's toes. Git tracks changes to files over time, so you can always see who made what changes and when. This makes it easier to track down bugs and understand the evolution of a project. Git also supports branching and merging, which allows developers to experiment with new features or fixes without affecting the main codebase.

GitHub is a web-based hosting service for Git repositories. It provides a user-friendly interface for managing and viewing Git repositories, along with a host of other features that facilitate collaborative development. With GitHub, you can easily share your code with others, contribute to open-source projects, and review code changes. GitHub also integrates with many other tools, making it a central hub for many development workflows.

GitHub Actions is a CI/CD (Continuous Integration/Continuous Deployment) tool that allows you to automate your software workflows directly within your GitHub repository. With GitHub Actions, you can build, test, and deploy your code right from GitHub. You can also assign tasks and workflows based on certain events, such as pushing code to a branch or creating a pull request. This level of automation can greatly increase productivity and ensure consistent quality of code.

GitHub Pages is a static site hosting service that takes HTML, CSS, and JavaScript files straight from a repository on GitHub, optionally runs the files through a build process, and publishes a website. It's a great way to host project pages, documentation, or a personal blog. One of the key benefits of GitHub Pages is its tight integration with GitHub, which means you can manage your website content with the same Git workflows you use for your code.

The lake_washington_and_sammamish_temps project uses Github Actions to push code unit test coverage results to Github Pages whenever code is pushed to the main branch.

GitHub Issues is a robust tracking system that allows you to track bugs, enhancements, or other requests related to your project. It's more than just a bug tracker - it's also a great way to manage your development tasks and workflows. You can assign issues to specific team members, label them for easy searching, and reference them directly in your code commits. This makes GitHub Issues a central part of your project management toolkit. It's great, but frankly it doesn't yet quite compare to other project management tools like Asana or the toolset provided by Atlassian (Jira, etc), albiet at much higher costs.

All for the wonderful low price of: under $10 per month for a Github account.

Next: Coding Languages

Share Ten Moments

Anaconda is a powerful tool for Python and R data science and machine learning. It's an open-source distribution that simplifies package management and deployment. Anaconda comes with a suite of over a thousand data science packages, and a package manager called Conda. Conda makes it easy to install both Python and non-Python packages, manage environments, and ensure all packages work together without conflicts. Anaconda also includes Jupyter Notebook, an environment for interactive computing, which is widely used in data analysis and scientific research.

Apt, or Advanced Package Tool, is the default package manager for Ubuntu and other Debian-based Linux distributions. It handles the installation and removal of software on your system. Apt simplifies the process of managing software on Unix-like computer systems by automating the retrieval, configuration, and installation of software packages. It also resolves and handles dependencies automatically. This means that if you want to install a software package that relies on other packages to function, Apt will identify these dependencies and install all necessary packages.

Pip, which stands for "Pip Installs Packages", is the standard package manager for Python. It allows you to install and manage additional libraries and dependencies that are not distributed as part of the standard Python library. Pip provides a simple way to install packages from the Python Package Index (PyPI) and other package directories. It also supports virtual environments, which is a key feature for Python developers as it allows them to create isolated Python environments for different projects, ensuring dependencies are kept separate and organized.

All for the wonderful low price of: free.

Next: Revision Control and Related Tools

Share Ten Moments

VS Code's IntelliSense feature is a game-changer. It offers smart completions based on variable types, function definitions, and imported modules. This feature alone has saved me countless hours of time. The built-in Git support is another major plus. Being able to stage and commit changes, create new branches, and view diffs right from the editor with a modern-day UI is incredibly convenient.

The extensibility of VS Code is another reason why I love it. There's a vast marketplace of extensions that can add new languages, themes, debuggers, and more to the editor. From BASH intellisense to Docker and everything else, quick and easy...it's a game-changer. It's easy to customize VS Code to fit my exact needs. And despite all these features, VS Code remains fast and responsive, which is a testament to its well-designed architecture.

As for Jupyter notebooks, they have transformed the way I work with Python, especially for data analysis and visualization tasks. The ability to interleave code, outputs, and explanatory text in a single document is incredibly powerful. I can experiment with code, see the results, and make adjustments in a very efficient and interactive way. This is a stark contrast to the traditional edit-run-debug cycle that can be time-consuming and disruptive.

Jupyter notebooks also make it easy to share my work with others. The notebooks are self-contained and can be rendered in a web browser, making them accessible to anyone, regardless of their development environment. This has made Jupyter notebooks an essential tool for collaborative projects and for presenting my work in a clear and understandable way. I've started adding one by default to every repo I work in.

All for the wonderful low price of: free.

Next: Environment and Package Management

Share Ten Moments

One of the most powerful aspects of Linux that I absolutely love is its command-line interface and the suite of tools that come with it. These tools, often referred to as the "Unix philosophy," are designed to do one thing and do it well. They can be combined in countless ways to perform complex tasks. For example, the grep command can be used to search for a specific string in a file or stream of input. The ls command lists the contents of a directory. You can pipe the output of ls into grep to search for a specific file in a directory. The command would look like this:

ls | grep myfile.txt 

This will list all files in the current directory and then search that list for "myfile.txt". To get a bit cheeky, this will create a file named "me", fill "me" with beer, and move me to hawaii:

touch me && echo "beer" > me && mkdir -p hawaii && mv me hawaii/

This ability to chain reliable, well-documented, modular commands together allows for a level of power and flexibility that's hard to match in other operating systems.

After using Windows for many years and then switching many more years ago, I also have a deep appreciation for the macOS operating system. Being Unix-based, it combines the power and flexibility of Unix with the sleek and intuitive user interface that Apple is known for. This means I can leverage powerful command-line tools and scripts, similar to a Linux environment, while also enjoying a smooth, user-friendly desktop experience. The seamless integration between software and hardware in a Mac system also contributes to a stable and efficient operating environment. In terms of security, macOS has a strong track record. It's built on a Unix-based foundation which is known for its robust security features. Furthermore, Apple has implemented several security measures such as Gatekeeper, which blocks untrusted applications, and XProtect, an automatic malware detection tool. Additionally, the fact that macOS is less targeted by malware compared to other operating systems adds an extra layer of security.

I'm a big proponent of doing development work in the environment that matches where the code will end up running. So, if I'm working on server code, I'll typically be working on my Mac, but remoting into a cloud server or container that's running Ubuntu. No point in wrestling with package install challenges on my Mac when the target environment is Ubuntu! The lake_washington_and_sammamish_temps project was a bit of an exception; since it uses Selenium and Chrome, I found it easier to code and debug the web browser interactions on my local machine.

*nix OS provided for the wonderful low price of: free!

Next: IDE and Jupyter Notebooks

Share Ten Moments

• Operating Systems: OSX, Linux Ubuntu

• Integrated Development Environment (IDE): VS Code, Jupyter Notebooks

• Package Management: Anaconda, Apt, Pip

• Revision Control and Related Tools: Git, GitHub, Github Actions, Github Pages, Github Issues

• Coding Languages: Python, BASH

• Noteable Python Libraries: Selenium, Pandas, Boto

• Unit tests and Test Coverage: PyTest

• Markup Languages: Markdown, JSON, YAML

• Cloud Services via Amazon Web Services (AWS): IAM, EC2, S3, CloudFront, ECS, ECR, Fargate, Route 53, CloudWatch, EventBridge, Certificate Manager, Secrets Manager

• Containerization: Docker

• Domain Registrar: Namecheap

Next: Operating Systems

Share Ten Moments

My first step was to do some research on what has already been done on this front.

My findings were very interesting; I half-expected there to be an API(s) that surfaced lake water temperature data worldwide, or at least nationwide. After looking around quite a bit, and getting some help via email (thank you!) from Ronald Henderson and others at the USGS, it turns out that there apparently isn't a good source of this kind of data, outside of the county level. Interesting, given current climate concerns with regard to surface water temperatures!

This left me focused on the information provided on the King County website here. I found it a bit cumbersome to find what I was looking for (which was a good thing, in terms of substantiating my project), so I reached out and got some help from Curtis DeGasperi, Water Quality Engineer at King County Department of Natural Resources and Parks. They first pointed me to data sources here, but I was looking for more of a real-time data feed, so I followed up with more questions.

Per Curtis' response, I learned that the data sources for the pages here are updated only twice monthly (Mar-Nov) and monthly (Dec-Feb), and it takes ~45 days after each sampling event for the data generated via field and lab analysis to post to the web.

However, I was pointed to the Lake buoy data pages here, learning that real-time lake temperatures could be found by scrolling-over the buoy locations. Cool!

After reading about the buoy specifications here, researching their costs online, and reviewing the very well done maintenance and quality assurance plan here I came to the happy conclusion that researching, finding, buying and dropping my own sensor into the lake would not be the best next step; I decided to proceed with leveraging the data King County rigorously collects, and received permission to do so, noting their disclaimer here.

With all of this research, I landed on a plan to publish lake water temperature data that could be consumed in JSON format by any web-enabled application I might decide to build in the future. I'd get what I primarily wanted, a real-time temperature data JSON file. As a cool added outcome, I'd process all historical temperature data (since 1983) to determine what the all-time recorded high and low temperatures are, and the by-month highs and lows, regardless of year.

You may have noticed that my service posts data to swimming.withelvis.com. In one of my previous startups, I had a saying "with Elvis", e.g. "Next we're going to build a new client app! With Elvis." The idea being that Elvis is a cool legend, and doing anything "with Elvis" would make it that much more fun and better. I got so wrapped up in the idea that I purchased the domain name "withelvis.com", which had been sitting for years, unused. I got a kick out of the "swimming.withelvis.com" idea, and so it is.

Over many years in the startup world, I'd never published my own personal open-source project, and felt this was the perfect candidate and time to do it. So it is!

Ultimately I hope this fun little project serves to help others in some way. Cheers!

Next: technologies and tools overview

Share Ten Moments

I’ll write about why I did the work, my research, and the tools I chose.

Next: the why

Share Ten Moments

This is Ten Moments.

Subscribe now

I responded to my friend along the following lines:

1. If you’re going with an RDBMS, MySQL is the right, best choice in my opinion.  It’s worked very well for us over the years.

2. Since you’re going the standard SQLroute:

   1. If your database is expected to grow in step with traffic, and you’re thinking about sharding early – kudos.  You’re likely going to have to do it, sooner or later.

      1. Sooner vs. later if you’re in the cloud and running under its performance constraints.

      2. Do it out of the gate, if you have time, after you’ve figured out how you’re going to do it (i.e. whether you’re going to leverage a tool, DYI, etc).

         1. In other words, if you have time, don’t “see how long you can survive, scaling vertically”.

            1. Sharding while running the race : not a fun transition to make.

      3. I realize what I’m saying is counter to popular thinking, which is “don’t shard unless you absolutely have to”.

         1.  Without the assumption that your data set is going to grow in step with your traffic, I’d be saying the same thing.

   2. Designing your schema and app layer for sharding, sharded on as few keys as possible, ideally just one, is not future-proofing, it’s a critical P0.

3. Since you’re going to be sharding MySQL, your options are relatively limited last I checked.

   1. Ask for input from folks who have done it before.

   2. The other sharding options I started looking at over two years ago all had disallowing limitations, given our business model.

   3. At quick search-glance just now, it also does appear that dbShards is ruling this space at this point.

4. So barring any other options I’m missing, your best options that I’m aware of:

   1. dbShards

      1. Definitions we/they use, to help clarify discussion  :

         1. global tables : tables that contain the same data on every shard, consistency managed by dbShards.

         2. shard : two (primary and secondary) or more hosts that house all global table data, plus any shard-specific data.

         3. shard tree : conceptually, the distribution of sharded data amongst nodes, based on one or more shard keys.

         4. reliable replication : dbShards proprietary replication, more details on this below.

      2. pros

         1. The obvious : you’ll be able to do shard-count more reads and writes that you’d otherwise be able to do with a monolithic, non-sharded backend (approximately).

            1. Alternatively, with a single-primary read-write or write-only node, and multi-secondary read-only nodes you could scale reads to some degree.

               1. But be prepared to manage the complexities that come along with eventual read-consistency, including replication-lag instrumentation and discovery, beyond any user notifications around data not being up-to-date (if needed).

         2. It was built by folks who have only been thinking about sharding and its complexities, for many years

            1. who have plans on their roadmap to fill any gaps with their current product

               1. gaps that will start to appear quickly, to anyone trying to build their own sharding solution.

                  1. In other words, do-it-yourself-ers will at some point be losing a race with CodeFutures to close the same gaps, while already trying to win the race against their market competitors.

         3. It’s in Java, vs. some other non-performant or obscure (syntactically or otherwise) language.

         4. It allows for multiple shard trees; if you want (or have to) trade in other benefits for sharding on more than one key, you can.

            1. Benefits of just sharding on one key include, amongst other things, knowing that if you have 16 shards, and one is unavailable, and the rest of the cluster is available, 1/16th of your data is unavailable.

               1. With more than one shard tree, good luck doing that kind of math.

         5. It provides a solution for the auto-increment or “I need unique key IDs” problem.

         6. It provides a solution for the “I need connection pooling that’s balanced to shard and node count” problem.

         7. It provides a solution for the “I want an algorithm for balancing shard reads and writes”.

            1. Additionally, “I want the shard key to be based on a column I’m populating with the concatenated result of two other string keys”.

         8. It has a distributed-agent architecture, vs. being deeply embedded (e.g. there are free-standing data streaming agents, replication agents, etc instead of MySQL plugins, code modules, etc ).

            1. Provides future-proofing, scalability and plug-ability.

            2. Easier to manage than other design approaches.

         9. Streaming agents allow you to plug into the update/insert stream, and do what you like with changes to data.

            1. We use this to stream data into Redis, amongst other things.  Redis has worked out very well for us thus far, by the way.

            2. Other dbShards customers use this to replicate to other DBMS engines, managed by dbShards or not, such as a column store like MonetDb, InfoBright, even a single standalone MySQL server if it can handle the load.

         10. It supports consistent writes to global tables; when a write is done to a global table, its guaranteed to have been done on all global tables.

         11. It doesn’t rely on MySQL’s replication and its shortcomings, but rather on its own robust, low-maintenance and flexible replication model.

         12. Its command-line console provides a lot of functionality you’d rather not have to build.

             1. Allows you to run queries against the shard cluster, like you were at the MySQL command line.

             2. Soon they’re releasing a new plug-compatible version of the open source MyOSP driver, so we’ll be able to use the same mysql command line to access both dbShards and non-dbShards managed MySQL databases.

         13. Its web console provides a lot of functionality you’d rather not have to build.

             1. Agent management and reporting, including replication statistics.

             2. Displays warning, error, diagnostic information, and graphs query counts with types.

             3. Done via the “dbsmanage” host, which provides centralized shard node management as well.

         14. It’s designed with HA in mind.

             1. Each shard is two (or optionally more, I think) nodes.  We put all primary nodes in one AWS availability zone, secondaries in a different one, for protection against zone outages.

             2. Write consistency to two nodes; in other words DB transactions only complete after disk writes have completed on both nodes.  Secondary writes only require file-system (vs. MySQL) disk writes.

             3. Managed backups with configurable intervals; MySQL EC2/EBS backups aren’t trivial.

             4. Web-console based fail-over from primary to secondary; this is very helpful, particularly for maintenance purposes.

         15. Proven to work well in production, by us and others.

             1. We’ve performed 100K queries per second in load-testing, on AWS/EC2, using m1.xlarge instances.

         16. Designed with the cloud and AWS in mind, which was a great fit for us since we’re 100% in AWS.

         17. “dbsmanage” host

         18. Drivers included, of course.

             1. In addition to MyOSP, they have JDBC, PQOSP (native Postgres), ADO OSP (for .NET), and soon ODBC.

         19. Go-fish queries allow you to write standard SQL against sharded data

             1. e.g. sharded on user.id : SELECT * FROM user where FirstName=’Foo’;

                1. will return all results from all shards performing automatic aggregation

                   1. sorting using a streaming map-reduce method

         20. Relatively easy to implement and go live with; took us about six weeks of hard work, deadline-looming.

         21. It’s the market-leading product, from what I can tell.

             1. 5 of the Top 50 Facebook apps in the world run dbShards.

         22. It supports sharding RDBMSs besides MySQL, including Postgres, DB2, SQL Server, MonetDb, others coming.

         23. Team : top-notch, jump-through-their-butts-for-you, good guys.

         24. Ability to stream data to a highly performant BI backend.

      3. cons

         1. As you can see, some of these are on the pro list too, double-edged swords.

         2. Cost – it’s not free obviously, nor is it open source.

            1. Weigh the cost against market opportunity, and/or the additional headcount required to take a different approach.

         3. It’s in Java, vs. Python (snark).  Good thing we’ve got a multi-talented, kick-ass engineer who is now writing Java plugins when needed.

         4. Doesn’t rely on MySQL replication, which has its annoyances but has been under development for a long time.

            1. Nor is there enough instrumentation around lag.  What’s needed is a programmatic way to find this out.

         5. Allows for multiple shard trees.

            1. I’m told many businesses need this as a P0, and that might be true, even for us.

            2. But I’d personally prefer to jump through fire in order to have a single shard tree, if at all possible.

               1. The complexities of multiple shard trees, particularly when it comes to HA, are too expensive to justify unless absolutely necessary, in my humble opinion.

         6. Better monitoring instrumentation is needed, ideally we’d have a programmatic way to determine various states and metrics.

         7. Command line console needs improvement, not all standard SQL is supported.

            1. That said, we’ve managed to get by with it, only occasionally using it for diagnostics.

         8. Can’t do SQL JOINs from between shard trees.  I’ve heard this is coming in a future release.

            1. This can be a real PITA, but it’s a relatively complex feature.

            2. Another reason not to have multiple shard trees, if you can avoid them.

         9. Go-fish queries are very expensive, and can slow performance to a halt, across the board.

            1. We’re currently testing a hot-fix that makes this much less severe.

            2. But slow queries can take down MySQL (e.g. thread starvation), sharding or no.

         10. HA limitations, gaps that are on their near-term roadmap, I think to be released this year:

             1. No support for eventually-consistent writes to global tables means all primaries must be available for global writes.

                1. Async, eventually consistent writes should be available as a feature in their next build, by early October.

             2. Fail-over to secondaries or back to primaries can only happen if both nodes are responding.

                1. in other words, you can’t say via the console:

                   1. ‘ignore the unresponsive primary, go ahead and use the secondary’

                2. or:

                   1. ‘stand me up a new EC2 instance for a secondary, in this zone/region, sync it with the existing primary, and go back into production with it’

             3. Reliable replication currently requires two nodes to be available.

                1. In other words, if a single host goes down, writes for its shard are disallowed.

                   1. In the latest versions, there’s a configuration “switch” that allows for failing-down to primary

                      1. But not fail down to secondary.  This is expected in an early Q4 2012 version release.

             4. dbsmanage host must be available.

                1. dbShards can run without it or a bit, but stats/alerts will be unavailable for that period.

             5. Shard 1 must be available for new auto-increment batch requests.

             6. go-fish queries depend on all primaries (or maybe all secondaries via configuration, but not some mix of the two as far as I’m aware) to be available

   2. DYI

      1. I can rattle off the names of a number of companies who have done this, and it took many months longer than our deployment of dbShards (about six weeks, largely due to the schema being largely ready for it).

      2. Given a lot of time to do it, appeals to me even now, but I still wouldn’t go this route, given the pros/cons above.

   3. The latest release of MySQL Cluster may be an option for you, it wasn’t for us back with MySQL 5.0, and not likely now, due to its limitations (e.g. no InnoDB).

   4. AWS RDS was an option for us from the onset, and I chose to manage our own instances running MySQL, before deciding how we’d shard.

      1. For the following reasons:

         1. I wanted ownership/control around the replication stream, which RDS doesn’t allow for (last I looked) for things like:

            1. BI/reporting tools that don’t require queries to be run against secondary hosts.

               1. This hasn’t panned out as planned, but could still be implemented, and I’m happy we have this option, hope to get to it sometime soon.

            2. Asynchronous post-transaction data processing.

               1. This has worked out very well, particularly with dbShards, which allows you to build streaming plugins and do whatever you want when data changes, with that data.

                  1. Event-driven model.

                  2. Better for us than doing it at the app layer, which would increase latencies to our API.

         2. Concern that the critical foundational knobs and levers would be out of our reach.

            1. Can’t say for sure, but this has likely been a good choice for our particular use-case; without question we’ve been able to see and pull levers that we otherwise wouldn’t have been able to, in some cases saving our bacon.

         3. Their uptime SLAs, which hinted at unacceptable downtime for our use-case.

            1. Perhaps the biggest win on the decision not to use RDS; they’ve had a lot of down-time with this service.

         4. Ability to run tools, like mk-archiver (which we use extensively for data store size management), on a regular basis without a hitch.  Not 100% sure, but I don’t think you can do this with RDS.

         5. CloudWatch metrics/graphing is a very bad experience, and want/need better operational insights to what it provides.  Very glad we don’t depend on CW for this.

      2. All of these reasons have come at considerable cost to us as well, of course.

         1. Besides the obvious host management cycles, we have to manage :

            1. MySQL configurations, that have to map to instance sizes.

            2. Optimization and tuning of the configurations, poor-performance root-cause analysis,

            3. MySQL patches/upgrades.

            4. maybe more of the backup process than we’d like to.

            5. maybe more HA requirements than we’d like to; although I’m glad we have more control over this, per my earlier comment regarding downtime.

            6. maybe more of the storage capacity management than we’d like to.

         2. DBA headcount costs.

            1. We’ve gone through two very expensive and hard-to-find folks on this front, plus costly and often not-helpful, cycle-costing out-sourced DBA expertise.

            2. Currently getting by with a couple of experienced engineers in-house and support from CodeFutures as-needed.

      3. As I’ve seen numerous times in the past, AWS ends up building in features that fill gaps that we’ve either developed solutions for, or worked around.

         1. So if some of the RDS limitations can be worked-around, there’s a good chance that the gaps will be filled by AWS in the future.

         2. But it’s doubtful they’ll support sharding any time soon, there’s too much design and application-layer inter-dependencies involved.  Maybe I’m wrong, that’s just my humble opinion.

This was originally posted last week here, but I wanted to re-post here and will be updating with our latest status and learnings, if there’s any interest.  Let me know.