The CASA model with a JavaScript SPA

November 12, 2020 Leave a comment


This will be a short one, but came from a question asked while I was working on a green field project for auditing work. The user interface was developed using REACT, which is a JavaScript SPA (Single Page Application) framework. In the exercise, I set up both an API (service layer) solution and a web solution, both using the Core AS Application (CASA) model. The model is shown below:

The CASA (Core AS Application) Model

The two solutions fit together like so, with the web application connecting to the API application via an adapter known as a service proxy. The actual naming was CompanyName.ProductName.Persist.Audit.ServiceProxy.

Website connected to an API

The idea was there was a core for the web project, which was in developed in React, and the API, which was developed using ASP.NET Core WebAPI. A few questions arose during the set up.

Why not have the react website connect directly to the API instead of using its own back end pieces?

What the suggestion was is a picture that looks more like this:

Web application connected directly to API

The answer to this question is multi-fold.

  1. Consistency: You will be using this pattern for all other types of APIs. While using different patterns is not an issue, the more consistency you use in your code the easier it is to …
    • Maintain – If there is a consistent organization, you can find the code in the same place for every connection. There is always an adapter to abstract the connection.
    • Train – If you consistently follow the pattern, you can easily train new developers.
  2. Abstraction: You lose the abstraction in front of the API. In this case, there is no issue, as both are internal, but think about some other capability, like single sign on. If you started with an internal API to log someone on, but the company switch to another solution, like Okta, you now have to completely set up Okta and then switch over (from experience, this is not always easy).
  3. Future Proofing: You switch from abstraction using contracts (interfaces in .NET) and adapters to configuration. If you ever switch APIs, this is not an issue as long as the contracts are identical. If not, abstraction protects you from change. The reality is something will change. Being able to control changes in small steps is a much easier way to future proof your software from change.

All three of these are important. Consistency in patterns definitely helps with maintainability and training. This allows you to be more efficient in your development efforts. But, the abstraction is the bigger one for me, as it future proofs the solution. I have an abstraction to create adapters on so I don’t have to deploy all of the software at one time in a “big bang” integration.

A couple of other notes on this:

  • There are plenty of examples of React in an ASP.NET core application. Talking back to this location and then connecting back to the API through libraries is well documented.
  • When you use adapters, they adhere to contracts through interfaces, fitting the I in SOLID, or Interface Segregation Principle. If you merely connect to the API directly in JavaScript, you have to change implementations through configuration, which means you lose the interface. Can you create an interface in JavaScript? Yes, at least in TypeScript and EF6. But this is a bit newer to many developers. Note that the use of contracts helps with Dependency Inversion (the D in SOLID).
  • Separating everything out helps you adhere to the Single Responsibility Principle, or the S in SOLID.

Can we remove the core on the React application solution and connect the ASP.NET Core application directly to the Service Proxy Persistence classes?

This one comes with the assumption you will not use business logic in the web application. In this case, the Core becomes a pass through with methods that only contain two lines of code (the instantiation of the persistence object and invoking a method). This is not a bad thing, as it will not add a lot of weight to the call and it makes it easier to add any business logic that might be required. But, as you are having .NET code calling the persistence library, you are making changes in the same solution (different project) and it is more of a refactor than a huge change in architecture. Personally, I would leave the pass through, as my experience shows business logic tends to get added as you mature the product.

Summary

The power of the CASA model is it sets up software in a way that is easy to reconfigure. I won’t say it is the only way of doing this. If you have another method to set things up in a consistent manner and leave abstractions between portions of the application, then use your own method.

No matter how you set up software, be sure you have abstractions in place that allow you to change out pieces without having to go through major surgery on your applications.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Uncategorized

Why Code Organization is so Important in Software

November 12, 2020 Leave a comment


I am a pattern person. This makes me great at reading and quickly memorizing maps. I can look at a Sudoku board and start mentally filling in without having to think about the numbers too much. I can also look at a software architecture problem, from a single application to integrated systems, and start figuring how to map from where we are to where we need to be. I can also see patterns in software development, not only in what we are doing, but the how and the why.

Today, software development is difficult. Some of this comes from the overly optimistic approach most developers take when sizing a task. Some of it comes from the practices we use. And others still come from business or how we interact with business. In this post, I want to take a look at why code organization is a foundational skill for developing quality maintainable and acceptable solutions.

First a bit of history …

Finding the Root

There are a few things I notice in various places I have consulted or worked over the years.

  1. There is very little testing over the code base.
  2. Refactoring is a luxury when there is
  3. Agility is “practiced” in the development team, but not throughout the Enterprise
  4. Very little is automated (in DevOps words, I see far more CI and far too little CD – a topic for another post?)
  5. There is too much technical debt in the solution

It is not that we don’t have knowledge of many practices that would alleviate these tendencies, it’s just that we have precious little time. This is partially due to our optimism, but also due to pressure of business compounded by the debt built up from the lack of time to refactor. And it goes on.

My first exploration into this topic came from working with my first DevOps assignment. How do you take an organization from fully manual deployment to the point where a code check in could potentially push all the way to production, in discrete, controlled steps?

I started from the end, production deploy, and walked back to the beginning, code check in, with the bookends – or start and end states – being fully manual deployment via documents to a fully automated deployment triggered by a code check in (another lovely topic for a blog post?).

Here are the steps to get from manual to fully automated (and, as you will see, the steps for getting development under control).

  1. Organize the Code Base
    • Find seams in the code
    • Add interfaces
    • Refactor
    • Add Tests
    • Repeat this cycle
  2. Test – Ensure code is covered for acceptability
  3. Refactor
    Repeat the first three steps, as needed
  4. Productize the code base
  5. Create a CI build
  6. Create a CD build for Development
  7. Add in Feedback loops
    Continue steps 5 through 7 until the pipeline is complete

As I spent more time with additional clients, and different issues, I found organizing the code to be the foundation for almost all of them. Here are my statements

  • It is difficult to automate the delivery of software without properly organized code
  • It is difficult to test for acceptability without properly organized code
  • It is difficult to maintain an application without properly organized code
  • It is difficult to modernize a legacy application without properly organized code
  • It is difficult to add new features without properly organized code

In some cases, I would replace “difficult” with “impossible” or at least “nearly impossible”. So, how do we get to organization.

Mi CASA es su CASA

Around 2008, I was working with a client who wanted to build a web application with the same features as a desktop application they currently had. Having spent time in Enterprise software development, my first thought was this: If you are writing the web application in the same language as the desktop application, why do you need to write a new application?

It is still a valid question today. As a result of asking that question, I started thinking about developing a framework for software development. In 2010, I came up with a name: Core As Application. Succinct and expresses the fact our core business logic really is our application. Also hard to digest. Wanting a cool name like “scrum”, I shortened it to CASA (which still means Core AS Application). It took awhile, but I realized it was neither a framework not a methodology, but a model. Below is a copy of the model from 2018.

The CASA (Core AS Application) model

The main point here is the center circle is the beginning point when thinking about an application. It is built on top of domain centric objects (generally state only, no behavior) and any frameworks required. It is fronted by a means of presenting the core to a “user”, which can be a human or another system that will work in sessions of time. Is is back ended by some means of persisting, or preserving, the state between “users” sessions. And, it is topped by enough tests to ensure acceptability of the application from the owners and users of the application.

Key to the model is the idea that the core is linked to the presentation and persistence portions of the system via contracts. In most languages, you call these contracts interfaces. But using these contracts, or interfaces, is where you are able to do things like switching from SQL Server to Oracle, or making a new mobile application with identical features as your current website.

Yes, it gets a bit more complex, as you end up with multiple “cores” in the average system, as you have multiple “applications” or products (as shown in the diagram below where you have the website connecting to one or more APIs). It can also be taken to the point of extreme in microservices architecture.

CASA across solutions/products

The main rule is “each class goes into a library that is identified by its purpose and/or function”, you can start to see the value in adding the area of concern in the project, library and/or assembly names (with apologies using the word assembly if you are a non-.NET developer).

Something New?

Hardly. It is the idea of creating a taxonomy for your code. In the COM days, it was common to think of applications in terms of layers or tiers. Microsoft coined it n-tier architecture, although it was generally in three tiers (Client/UI, Business and Data – see below).

N-tier architecture

And the concepts align well with clean architecture, as you are starting from the business domain out.

Clean Architecture

Think of the domain models at the bottom of the CASA model as Entities, the Core as Use Cases, the contracts and adapters as Controllers, Gateways and Presenters, and the outer points (presentation and persistence) as the blue outer circle in Clean Architecture (External Interfaces, DB, Devices, Web, and UI).

We can also see it in Hexagonal architecture (developed by Alastair Cockburn, one of the original Agile manifesto guys).

Hexagonal Architecture

In Hexagonal architecture, the outside points are presentation and persistence, connected to the Core via ports and adapters. As an even better view of Hexagonal architecture, here is one with a bit of an overlay of clean architecture.

Hexagonal Architecture with Clean Architecture concepts embedded

The Setup

When I am consulting, one of the first things I do with a solution is add in the folders corresponding to the type of code. The folders set up are:

  • Core: The core business logic, aka the “application”. These are behavior objects that contain code specific to the business.
  • Data: This will contain any projects specific to data persistence mechanisms, like a database. In the .NET world, a SQL Server Data Tools project would be an example of what lies here. These are generally segregated out early to productize the database separately from the application so it can be an independent product (avoid “big bang” deployment).
  • Domain: The domain entities and exceptions. These are state objects and general just getters and setters.
  • Framework: These are behavioral in nature by can apply to multiple applications and possibly even multiple domains. I will eventually “retire” these from the application solution and create a product out of the library and put it in a package manager (NuGet for .NET). I will occasionally subclass this further. The primary example would be separating out utility from the core framework libraries.
    • Utility – A general product focused on performing a specific cross-domain function. For example, a print library.
  • Persistence: Behavioral objects related to persisting state between sessions. This will generally be repository objects over a database or service proxies to contact APIs.
  • Presentation: This is where the application connects to “users”, which can be human or other systems. I subdivide this folder into specific types of presentation.
    • API – a web based API endpoint, generally REST in current times. In the past, I have also called this service, but that can be confused with a server based service.
    • Desktop – A user interface to run on the desktop.
    • Mobile – A user interface for a mobile device.
    • Service – A server based service (think windows services).
    • Web – A user interface using HTML or other web constructs, except APIs.
  • Test – Anything related to automated testing of the application. The following subfolders are common, but I will have to write a more extensive post soon on testing, as there are different patterns one can take.
    • Accept – Tests focused on acceptance. While one can cover a great deal of acceptability of the solution in unit and integration tests, these would be focused more on acceptance. An example might be Fitnesse. In general, automate everything you can here so user acceptance is more focused on preference of layout, colors, etc.
    • Integ – Tests for integration. In these tests, actual dependencies are utilized rather than mocks. It should be noted the unit tests will be identical in signature to integration tests, but will utilize mocked dependencies. NOTE: Tests in products like selenium will fall into the Integration folder if they are included in the solution.
    • Load – Tests for load testing. This is where load specific tests would be if you include them in the solution.
    • Unit – Tests focused on testing a unit of code. All dependencies will be mocked with known return values. It should be noted the integration tests will be identical in signature to unit tests, but will utilize the actual dependencies.

As you go through the above list, you should note the two in bold italics. There are the types of projects you productize and remove from solutions. The data projects will become automated database deployment and changes should be completely decoupled from deployment of applications (in other words, you should take small steps in data changes and do them ahead of application deployment rather than create a need to deploy at the same time). The Framework should move out so they can be independently updated and packaged for all applications that use them.

How does this look in development? Below is an empty solution in Visual Studio with the folders.

Visual Studio Solution Using CASA folders

In Practice

The practice varies slightly between green field (new development) and brown field (currently deployed development) solutions. Let’s look at each independently.

Green Field

With green field development, I will start by creating the various projects in the solution in the proper folders, as shown below. 

Sample CASA solution with projects in named folders

The general naming convention is:

Organization_Name.Product_Name.Area.SubArea

In this naming convention, the area is the folders: Core, Domain, Persistence, Presentation and the like.

It takes a bit more time, but I like to match up my code folders in the solution with the file system. This is done as follows.

Create the New Project by selecting the folder it will go in, right clicking and choosing add project.

Adding a new project to the solution

Add the correct type of project. In this case, I am setting up the Core business logic in .NET Core (unintentional play on words?).

Adding .NET Core for the Core

Name the project. In this example, it is the Core project for the CASA_Example product. Note that the folder is changed (highlighted).

Add the folder name to the file location

Once I am done, all of the projects will be in their correct folders, both on the file system (left) and in the solution (right).

Side by side to show the file system matches the view in Visual Studio

Brown Field

If the brown field product was already organized and the team followed the practice religiously, there is nothing to do. But, if this is a solution that requires reorganization, you have some work. This is a huge topic and worthy of a blog post of its own. I will do this soon on a post on legacy modernization.

One common example would be a solution in which all, or nearly all, of the code was present in the user interface/presentation project. In this case, you have to start organizing the code. These would be the steps I would take to do this.

  1. Rename the presentation project. This is not completely necessary, but doing this and moving the project to the correct folder on the drive is a good first step, as it starts setting up the process in your team early. A few steps here.
    • Go ahead and fix the namespace of the files in this project now. This is not mandatory, but it helps you avoid having to fix using statements later. This may take a bit of time, but it pays to keep your library maintained.
    • Double check naming on custom folders and adjust this now, as well.
  2. Search for boundaries/seams in the code. The natural seams are:
    • Any code that access a persistence mechanism
    • Any code that has core business logic (use cases)
    • Any code that serves a utility (or helper) function
  3. Migrate this code into a folder in the project with the proper area name.
  4. Create an interface for the new class. This is your contract.
  5. Create tests.
  6. Refactor. You may have to run back over different steps above at this time.
  7. Repeat steps 1-3 until you have all code that is not presentation related segregated. I start from persistence, then to framework, then to core, etc. The idea is migrate out the dependencies that are farther out in the clean architecture diagram first (or focus on the far right of the CASA model first.).
  8. Migrate code to new projects in the proper areas. Scroll back up to green field to see how to do this.
  9. You may, again, have to iterate.

I will focus on a post on legacy modernization of .NET applications some time soon to go through all of these.

Discussion

I have done this exercise with various companies with a lot of success. It is a bit difficult to get used to at first, but once the team is used to the concept, it flows nicely. It also gives a nice model to set up where code should go and makes it very easy to find the source of a problem with the code base.

When I introduce this concept, I get a few questions on a regular basis.

How strict should I be to adhering to names in my projects, namespaces, etc.? This is a naming convention. Like any naming convention, you should be very strict to using it consistently. It should be part of your code reviews like naming classes.

But do I have to use the names Core, Domain, Persist, Present, etc.? Honestly, I don’t care if you name it Fred, Wilma, Barney, and Betty, as long as the names make sense to you. I will state, however, I think Flintstones names are stupid, but mostly because it becomes another thing to train new developers on. I can see eyes rolling when you say “In this company, the core business logic is called Fred because Fred was the head of the Flintstone household.” Whatever floats your boat. 😀

I had a client want to change Present, Core, and Persist to UI, Business, and Data. I don’t particularly like these names, as not all presentations are user interfaces, not all persistence is through data storage mechanisms, and using the three sounds very n-tierish. But, it was what they wanted.

I had another client who asked if Core could be App, for application. I was fine with this. I prefer not to, as I feel Core is a better representation of “core business logic”, but if you regard this an Application, or App, role with it.

The important part is be consistent and be rigorous in upholding your standards. When you get loosey goosey on this one, you start making it more difficult to maintain your code base and you return to square one. You also risk introducing technical debt when you move away from consistency.

Can I use Clean Architecture? Sure. This would mean using the following folders.

  • Enterprise
    • Entities (instead of domain)
  • Application
    • UseCases (instead of core)
  • InterfaceAdapters
    • Controllers
    • Gateways
    • Presenters
  • Framework
    • Web
    • UI
    • ExternalInterfaces
    • DB
    • Devices

The thing I am not completely sold on here is the Presentation and Persistence are both under the Framework area and it muddies the word Framework, as the .NET Framework would be completely different from your website.

But, if this is how you want to do it, that is your choice. You can do the same with Onion Architecture or Hexagonal Architecture.

Summary

Code organization is one of the issues I see most often with clients. This goes in the direction of single monolithic solutions or very granular solutions to ensure the code can be found for maintenance. Very seldom do I find code organized in a way it is very easy to maintain.

Once your code is divided into proper spheres of influence, you will still have to take some additional steps to complete the maintainability of the solution. Tests are a key component, as is determining the names for different types of classes. As an example, do you call a core class that acts as a pipeline for a particular type of information a manager, a service, or something else.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Uncategorized Tagged with , , ,

Architecture: The CASA Model- Getting to the Core

December 4, 2018 Leave a comment


About 9 years ago, I was looking at code from a client and dealing with how hard it was to maintain compared to new code I was writing. In the process, I started seeing patterns in the code that I saw in numerous code bases I had worked in over the years. In particular:

  1. Most code was not organized well for maintainability
  2. Best practices were known by all developers, but few were practiced
  3. Very little automated testing
  4. The systems were not domain focused
  5. Switching from one user interface (say windows forms application) to another (web) required a lot of work, as the developers created a new application.

One issue that led to this mass of spaghetti code was the n-tier architecture idea, which gave people the principle that an application is made up of a user interface, business logic and data. Thus, a windows application and a web application were two different things. Naturally, since they were different things, you had to build them separately. In reality web application and a windows applications, as we know them, are not applications. Web and windows are two means of presenting applications. 

That simple paradigm shift lead me to start considering methodologies used for development, so I created what I thought was a new, or at least varied, methodology. I called it Core as Application. The original premise looked like this.

 

Core As Application, circa 2010

In this model, the Core IS the Application. This means an application is made up of domain objects (State) and actions on the domain objects (Behavior). In short, software is a bunch of BS? 

To facilitate the application, I need a means of presenting information to an interacting with users (May be human users or machine users). And, I need a means of persisting state between sessions. I should be able to switch presentations and persistence mechanisms without writing a NEW application, as the Core IS the application. 

NOTE: Core AS Application was a progression of Core Is Application so I could have a nice acronym.

There were a couple of things I did not realize at the time.

  1. Core as Application is NOT a methodology. It is a model. 
  2. Other people were working the same model at the same time, although focused very much on the architecture. You know them as Onion Architecture, Hexagonal Architecture and Clean Architecture

On point #1, I refined the concept to the Core AS Application model (CASA for short, since it needed a cute name). On point #2, all of the “architectures” mentioned deal with the same concept: domain models are the center, followed by business logic. I can can cover these in another post in the future.

Before moving forward, let’s cover a few terms as I envision them right now in CASA. The definitions are oversimplifications to an extent, but help keep you, the reader, on track.

Capability: A business capability is a function or service a business provides to its clients, internal and/or external.  Applications should be centered around capabilities and systems around 

Application: The software solution providing centered around a business capability.  In CASA, the application will be a single core with contracts separating it from presentation and persistence mechanisms.

System: A program or set of programs to solve a business problem that can span multiple capabilities. System can be the same as an application, in simple business problems, but can also required integration with other applications. As an example, an application that connects to multiple services to accomplish its work is a system. In CASA, if you see more than one core, it is an integrated system.

Solution: Solution is used here to indicate a packaging of separate projects to create an application. In this regard, it is synonymous to a Visual Studio Solution. A project will be a single Visual Studio project. 

In this post, I want to cover CASA as it exists today and why it works well in development.

The CASA Model 2018

The general principle of CASA is the application is the business logic at the center of the model. This means I can switch presentation and persistence without changing the application. I sit this core (behavior) on top of domain models (state), as the domain is the foundation of the application. I place contracts on either side of the Core, as I want to facilitate testing, which sits on top (the Contracts serve more than just aiding tests – to this in a moment). And testing, primarily automated, sits at the top. The model looks like this today.

There are a couple of differences between this model and the original.

  1. The domain is no longer part of the core, but the foundation. 
  2. Contracts have been added between the core and the persistence mechanism (in general, this will be a database)
  3. Contracts have been added between the core and the presentation mechanism (normally a user interface, but could be a service of some other machine to machine mechanism).
  4. Tests have been added on top.
  5. Adapters have been placed next to the contracts. 

In general, each of the various boxes will be contained in separate projects in a solution (assuming .NET, but the concept can be used in other programming paradigms). Here is a sample solution showing how this is used. 

For a better understanding of the value of the model, let me tie it back to some other industry concepts you might be familiar with, some of which were included in the design of the CASA model: Domain Driven Design, Contract-Fist Development, SOLID Design Principles, DRY, YAGNI and KISS, although some of the concepts are implicit rather than explicit.

  • Domain Driven Design – I don’t use all of Eric Evan’s domain driven principles, as I am rather stringent on separating my BS (behavior and state). I have 2 common domain projects in every solution: one for models and another for exceptions and I do not include behavior in a domain object, unless it is related to the object. As an example, an address may have a ToString() method that delivers a concatenated address – this is behavior, but specific to the object.
  • Contract First Development – The concept of writing concepts prior to developing is not new. In C++, you create interfaces first, in the form of .h (header) files. Delphi, which may, in some ways, be considered the inspiration for C#, also had a contract first approach. With REST APIs, the contract is a bit more implicit and found in the URL. For CASA, the idea is write the contracts first to ensure you understand the seams in your application.
  • Test Driven Development – I am definitely one that believes writing tests first helps you ensure your code is focused on what your users desire. I use code coverage metrics with my teams, but it is just a measure not a goal. 100% code coverage with bad tests is as unacceptable as very low coverage. I should also note acceptability is the real goal, not writing tests. 
  • SOLID Design Principles – Organizing code into specific areas of concern cover the (S Single Responsibility), while contracts handle the I (Interface Segregation) and aid the D (Dependency Inversion). The O (Open/Closed Principle) and L (Liskov Substitution) are not directly addressed by CASA, although using interfaces will help you think twice about breaking contracts, especially when you end up with multiple classes utilizing the same interface.
  • DRY – Do Not Repeat Yourself. This is not inherent in the model, so you have to create a process to handle this, as well as governance. For the process, when you find repeat code in multiple classes (or worse, projects), you should start refactoring ASAP. A couple of normal methods to employ would be to move the common code into a base class (two classes on the same interface, for example). Or you move the code into a separate class and/or project and reference it. As for the governance, if you are not using code reviews as part of your development process, you should. 
  • YAGNI and KISS – Also not directly covered in CASA, but you find the separation of the code makes it much easier to solve problems through testing and focusing on one problem at a time. Once you grasp the concept, you will find it easier to keep things simple and avoid bloating your code. If you do try to think ahead, you will find the model fights against you. In the future, I plan on talking a bit about testing with CASA and will add the link here.

Expanding CASA

As I started examining the model, I started thinking about solutions I had seen in various Enterprises. The area CASA seemed to fail is when you added a services layer to the application. First, some developers had an issue with the paradigm shift involved with thinking of services as a presentation concept. A bigger issue is what happens when you add a tier of services, as shown below:

I adjusted this by moving services to its own folder in the solution. This helped with the paradigm shift. Recently, I have found this also works with mobile applications, as you can use the same persistence library for the mobile application and the server side application. 

I should note, the service holds one application and the user interface, which used the service, holds another (at least potentially). So I had two cores, which means another solution. From an organization standpoint, you have two applications. When I add a second service, it becomes more evident. If you focus on microservices, which are mini applications with a service front end (yes, a simplification), it is absolutely clear an application with a service, layer or not, is actually multiple applications. I am not one for having too many projects in a solution, but as long as you can develop without massive compile and test cycles, it is a place to start. Solve then refactor.

From the standpoint of the model, it became clear I simply needed to chain two cores together with a service, which looks like this:

Moving to Mobile

In the past 6 months, I have gotten heavily focused on mobile presentation in React and Xamarin. As React cannot be directly compiled using Visual Studio, I won’t cover it here (I actually think I might be able to solve that problem, but that is another story). Xamarin is a bit easier, but changes a few things in organization.

In particular, with Xamarin, you end up having to compile at least parts of your core for each platform, especially in an offline application. You will be persisting to both the service (server side) and the mobile database from the mobile device. And to a server side database when the mobile application contacts the server to persist.

  1. Create a subfolder called Mobile under the Present folder. This is where the Xamarin.Forms project and the iOS and Android specific libraries go (I will likely rework this in the future, as these are not technically presentation libraries, but utilities).
  2. Make sure any libraries that are used on the server and the mobile device are .NET standard. NOTE: I think Core can work, but standard provides a few other options. 
  3. If you require any functionality that uses the full .NET Framework, ensure it is behind a service, as Xamarin cannot compile the full framework to native code.
  4. To facilitate the mobile application saving locally and on the server, configure it to use 2 implementations of the same contract: one to save locally and one to save via the service. The service will use one implementation of the contrract, of course.

I currently have the service in the same solution as the mobile application. When I separate it out as a separate product, I will have two solutions pointing to the same contracts for persistence. This may necessitate placing the contracts as their own product to avoid development in one solution breaking the other. I think the team is small enough to not jump either of these hoops right now (NOTE: I envision an entry on product mindset this week and will link it here). 

So, here is the solution at present. (NOTE: This is a work in progress, as I am still developing this. Thus far it is working)

This seems awfully hard?

Any time you switch paradigms, it is hard. For years after .NET was released I saw developers coding standard VB in VB.NET projects, as they had not made the paradigm shift. Microsoft made it easy to avoid making the shift by adding training wheels to VB in the form of the library Microsoft.VisualBasic.Compatibility. During the early .NET years, I used to recommend switching to C# as the paradigm shift would become part of learning the new language syntax. 

Once you get the paradigm shift, however, you will notice it becomes very easy to find code when you discover a bug. Part of this is in the organization the model provides, but a good deal of it is in learning the concept of a domain (which is outside the scope of this post).

Does it Work?

I have yet to see a case it does not. But I am also not arrogant enough to state you must use CASA to get the benefits. Below are the design principles I am using. If you already adhere to all of these principles, you might think twice before adopting what I have written here. Some of these principles go beyond the concepts in CASA and will be covered on this blog later.

  • Domain Driven Design – Focus on the business problem you are solving. Domain Driven Design will aid you in this principle. Your state objects should be named in terms familiar to your business.
  • Contract First Development – Understand the seams in your application before coding anything around the seams. To simplify, as you deal with different technical concerns, consider it a seam. In CASA, there is a contract for persistence, one for the core and one for any dependency that can be swapped out, or capability. If you ever feel another team could write a separate implementation, you write a contract. Why first? Because, like Test Driven Development, you learn about your code before you code it.
  • Requirements – Write requirements for acceptance. This is a pre-step for testing, as understanding what is acceptable will make it much easier to write a full enough set of tests.
  • Test Early – I am not a 100% purist in test driven, but I am a stickler for writing a test first on any problem you are not 100% sure of (which is almost everything) and adding tests on top of those that you are. (NOTE: Over time you will find some things you were 100% sure of at first, you should have written the test first, as you were cocky in your confidence). One more point: Acceptability should be a goal for all testing, not just acceptance testing. In standard TDD style, this means writing more test methods. In behavior driven, it means acceptability in your specifications.
  • Coverage – Code for acceptability, not percent. Percent is a good metric to find areas that might be lacking, but never incent on code coverage, as humans are very good at being creative to get incentives. Good coverage includes the following: 1) covering business use cases completely to ensure user acceptance 2) covering unacceptable cases 3) covering the fringe. I will focus on this in another post (and link here).
  • Testing errors – If a bug is found, or a lack of acceptability is discovered, you immediately write a failing test. If you cannot write a test that covers the condition that FAILS, you do not understand the bug well enough to fix it, so you SHOULD NOT try to fix it. (NOTE: lack of acceptability is not the same thing as a bug. If a user discovers something does not work how he or she intended, it is not a bug, but a change in understanding of what is acceptable – same process, but don’t have your team take the blame when someone realizes their vision does not work as they would like).
  • Build the tests up from the unit – If you are focused on covering for acceptability, you will find the exact same tests you run as unit tests are often run as integration testing. With contracts, you will mock the contract for your unit test and use a concrete implemenation of the contract for integration tests. If you use behavior focused tests, you will find you are running the same tests, just changing dependencies (did that blow your mind? If not, you are already in the Zen garden). Because I see the persistence store (database generally) as the castle that protects the king (data is king for business), I write an integration test on this early on. I might write it first, especially on non-user interfacing persistence applications. But I sometimes unit test the presentation first to ensure I understand what the user intended and then go to the database. Knowing how the user will interact with the system and how I store (optimized data store, not just a CRUD mock of the business objects), I can mock the persistence layer and unit test all core business functionality. If what I just wrote sounds a bit confusing, I will write a blog entry about this soon.
  • Product Mindset – This is the most complex topic (and deserving of its own blog entry – link here in the future). A few things I recommend. If you find you are no longer actively developing on part of your code, consider making it into a product. If you started with CASA at the start, these will likely end up being utility classes that you want to reuse, but this technique can also serve well if you have not use CASA, or similar, in the past. The main point is once you find a capability, add a contract and move the code to its own project, then to its own solution, and finally put it in your own NuGet repository. (I guess I also need to talk capability thinking in the future? – link here in the future).
  • Agility – This really has nothing to do with CASA, at least not directly. But Agile gets you thinking in small bits of functionality at a time, which works very well with CASA.

In Summary

In this entry, I focused on the Core AS Application, or CASA, model. CASA is a model that focuses on a visual representation of code organization with a focus on best practices, like Domain Driven Design, Contract First Development and SOLID Design Principles.

In the future, I will break this down further, focusing on how this works with various concerns, such as Requirements, Testing, Product Focus and Agile Development.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Uncategorized

Automated Testing is NOT optional

November 24, 2018 Leave a comment


Once upon a time there was a young apprentice who did not understand why automated testing was so critical. As he was under pressure to churn out magic for an evil wizard, he felt he did not have time to test his spells. In addition, he did not know how to tell if the spells worked correctly. Thusly, he randomly Harry Pottered the incantations and hoped for the best. As he had learned not to fry off his own appendages, he felt this fire and observe method was adequate and did nothing to change.

A medieval tale? Nay, I say. This is today. And I am finding despite years of talking testing, and it’s variants, at conferences, on blogs and even in video courses, so many people still don’t get it. I will distill why automated testing is important and then see if I can drill into some details and get you back on the right course. And, if you are already doing everything I state in this post, then find a place with two for one spells and that second beer is on me. 😀

A cautionary tale

In this story, our hero has inherited an app. Well, not exactly, as the app has to be stabilized and released prior to takeover. The only thing standing in the way is testing. And, since there is precious little automated, the testing is manual. And slow. And error ridden. And … okay, I will stop there, as I think you get the picture.

In each testing cycle, as things are fixed, the testing gets deeper and deeper, and finds more bugs. These bugs were not caught in the original runs, as the manual tests were running into surface bugs. As the surface bugs disappear, the testers has time to start hitting fringe cases. And, as these are wiped out, the testers start testing at the same time and run into cross account bugs, meaning one person sees someone else’s data. Along the way, you see circular bugs, where fixing one brings up the other and vice versa.

Sound familiar? Every single one of these problems can be solved by automated testing. Every single one.

But, Greg, automated tests do not solve this problem. You still have to go after surface bugs (those on the happy path) and drill deeper.

True, but once I write an automated test, it is part of my library of tests which I can run over and over again. Yes, I can do this by creating test plans and checklists and manually testing, but – and here is the key – I have to rely on discipline to ensure all tests are completed each time. Plus, the automated test session is a few minutes in length (seconds, or less, in the case of unit testing) whereas my manual testing session may take hours. I have been playing this game for the past week and I have spent more than 8 hours, plus 8 hours of another employee’s time, to manually test a solution lacking automated tests.

Yes, there is the other side of the coin, which is the time necessary to write the tests. Initially, you take a productivity hit writing tests, as it takes time. They pay back dividends when you get to maintaining your solution, however.

Here is what I have found in the past week (all of which could have been found through automated tests):

  • Buttons not wired up – UI testing
  • Items that do not delete – Unit testing
  • Items that appear to delete, but stay – UI testing
  • Validation failures – Unit tests (backend) and UI testing (front end)
  • Items that cross accounts – Unit testing and/or UI testing
  • Slow reports – performance testing
  • Autosave issues – JavaScript unit testing

Every single test above could have been automated. Let’s start there and I will talk about maintainability and acceptability first.

Lions and Tigers and Test Frameworks … Oh My!

One of the first questions I get asked is “which test framework is best?” My answer: YES.

Honestly, I don’t care what test framework you use. I most often use MSTest, which is included with Visual Studio. I sometimes augment with a BDD framework like SpecFlow. But, I have worked with NUnit, XUnit, JUnit and LMNOP (which stands for “insert your favorite here”). I can tell you strengths and weaknesses of some of them, but I have yet to find a case where one of the ones I have been using for some time just sucked. Start somewhere.

In your toolbelt, you should consider, at minimum:

  • A unit test framework – in .NET, this will likely be MSTest, NUnit or XUnit – as mentioned, I generally use MSTest. Wikipedia has a nice listing here which covers a plethora of languages.
  • A test runner – Visual Studio has one built in. Resharper adds its own, which has some nice features (you use Resharper right?). NUnit comes with one, etc. In other words, don’t sweat the small stuff.
  • A mocking framework – I bounce between RhinoMocks and MOQ, but this has been a customer driven decision.
  • A JavaScript unit test framework – Unless you never do web or hybrid mobile, I would definitely add this. I happen to gravitate to Jasmine, but have played a bit with JSUnit, Karma and Mocha.
  • A user interface focused test tool – Selenium is the main open source tool for this type of testing. I have picked up Katalon recently to test, as we are on Mac and PC now. Katalon sits on top of Selenium and works in Chrome, which was a requirement for us.

A few other things you might be interested in

  • A BDD Framework – I use SpecFlow in .NET, but there is a bit of a learning curve. Plus, you can write behavior focused tests without making the BDD paradigm shift (see later)
  • An acceptance test tool – I happen to be fond of Fitnesse, as you can set it up with either Excel or a Wiki and let business add test conditions. It gets a bit strange when you tests magically fail because they have edited the wiki, but it sure shortens the feedback loop. Please note you can test for acceptance without a separate tool by making sure you identify all of the conditions
  • Some type of static code analysis – Visual Studio contains some tooling, which you can accentuate with extensions like Microsoft Code Analysis. I also recommend having static code analysis as part of your delivery pipeline if you use Continuous Delivery disciplines.

It is also nice to have a means of getting to fringe conditions in your tests – Intellitest in Visual Studio Enterprise (the evolution of Pex) is a great tool for this (Pex and Moles were tools from Microsoft research that were included in earlier versions of Visual Studio).

Well, I don’t accept that

What are you trying to get from automated testing? When I ask this question, the general answer focuses on quality code. But what is quality code? Certainly bug free, or at least critical bug free, is a must. But is that enough? I say no.

When we look at development and maintenance, maintenance is the long pole. The better your tests are, the easier your solution is to maintain. This should be obvious, as the tests provide guides to ensure any maintenance fixes avoid creating additional bugs you now have to conquer (think about those circular, fixing A causes B and fixing B causes A, types of bugs).

But I think we have to go beyond quality alone and think in terms of acceptability. This means you have to ask questions to determine what behaviors are acceptable. More important, you have to ask the RIGHT questions. Below is a sample Q&A with a business users.

User: A person can enter a number between one and one hundred in this box.

Dev: What if they put in zero?

User: It should warn them.

Dev: Same with negative numbers?

User: Yes, and everything over one hundred as well.

Dev: Can we make it a drop down so they can only pick one to one hundred?

User: We find drop downs get too unwieldy for our users when there are 100 numbers.

Dev: And no letters, right?

User: Exactly.

Dev: Can we put in something that ignores when someone types in letters in that field and text like the user name and password fields saying 1 to 100.

User: Oh, both would be great.

From this conversation, we understand that acceptance includes a user interface that stops a user from entering anything other than the integers 1 to 100. For many developers, adding constraints to the user interface would be enough to ensure this. But one should understand the business code must also contain checks, because code should never trust user input.

As you go through this exercise with your end users, you will find acceptability is beyond quality alone. It deals with perception, as well. And users will help you create a more comprehensive test solution. In summary, think both about maintainability and acceptability as you.

Software is a Bunch of BS

Some days I think of male bovine output when I see code, as some of it really smells. But when I talk about software being BS in conference talks, I am thinking about the dual aspects of code, which are behavior and state.

  • Your users are most concerned with state. When they pull out a record and change it, they want to be confident the new data is placed in the database, so they can find it in that state during their next session.
  • Your dev team is most concerned with behavior, or what they do to state (domain objects) as a user interacts with it. The behavior should result in the correct state changes.

This is why I recommend testing that focuses on behavior. I have, in the past, used a tool called Specflow when coding in .NET languages (SpecFlow can also be loaded directly into Visual Studio through extensions – I have the search here, as there are different extensions for different versions of Visual Studio. 2017 is here). SpecFlow requires a unit testing framework to work and works with many (MSTest, nUnit and xUnit are all included, although xUnit requires a separate NuGet package).

The upside of SpecFlow, for me, was when I pushed code over to teams in India. I could write the cucumber and generate the scaffolding and have the use BDD to create the code. For the first week or two, I would correct their tests, but once they got it down, I knew I could create the test scaffolds and get the desired results. The downside was the learning curve, as writing tests in cucumber takes some practice. There is also a paradigm shift in thinking from standard unit tests.

If SpecFlow is a bit too much of a curve right now, you can write behavior tests in a unit test framework by moving away from a single “UserTests” class with multiple tests to a single test per behavior. I will write a complete entry on this later, but the following pattern works well for behavior (I have the template outline in bullets and then a very simple example).

  • Class is named for the behavior being tested. In this case the class is named “WhenMultiplyingFiveTimesFive”
  • The code that runs the behavior is in the Class Initialization method
  • The variables accessed by the tests are kept in private static variables
  • The tests contain expected values and assertions

And here is the class.

[TestClass]
public class WhenMultiplyingFiveTimesFive {
    private static _inputA = 5;
    private static _inputB = 5;
    private static int _actual;

    [ClassInitialize]
    public static ClassInitialize(TestContext context) {
        _actual = MathLib.Multiply(_inputA, _inputB);
    }

    [TestMethod]
    public void ShouldReturn25() //This can also be ThenIGet25 or similar (cucumber)
    {
        int expected = 25;
        Assert.AreEqual(expected, _actual, string.Format("Returned {0} instead of {1}.", _actual, expected);
    }
}

The downside here is you end up with a lot of individual test classes, many with a single test. As you start dealing with more complex objects, your _actual will be the object and you can test multiple values. On the positive side, I can  generate a good portion of my tests by creating a simple program to read my domain models and auto generate stub test classes. It can’t easily figure out the When condition, but I can stub in all of the Should results (i.e. the tests). If you opt for the SpecFlow direction, you will generate your scaffold test classes from cucumber statements, like:

    Given I am Logged In
    When I Call the Multiply method with 5 as A and 5 as B
    Then it returns 25

Or similar. This will write out annotated methods for each of the conditions above, which is much like a coding paint by numbers set.

Process This

I will have to draw this subject out in another blog entry, as testing discipline is very important. For the time being, I will cover the basics you should include the DOs and DO NOTs of a good testing strategy.

  • DO write tests for all behavior in your system
  • DO NOT write tests for domain objects. If the language vendor has screwed up getters and setters, you have a bigger problem than testing. One exception would be domain objects with behavior (although in my shop you would have to have a really good argument for including behavior in state objects)
  • DO ensure you have proper test coverage to match acceptability
  • DO NOT use code coverage metrics as a metric of success. 100% coverage with crap tests still yields crap software.
  • DO ensure every bug has a test. DO write the test BEFORE you fix the bug. If you cannot write a test that goes red for the condition, you do not understand the bug. If you do not understand the bug, you should not be attempting to fix it.
  • DO ask questions of your users to better understand the user interactions. This will help you create tests focused on acceptability.
  • DO look at fringe conditions and conditions outside of the box.
  • DO add tests prior to refactoring (assuming there are not adequate tests on the code at the time).

None of this is automagic. You will have to instill this discipline in yourself and your team. There are no free lunches.

With legacy code, prioritize what needs to be tested. If you are replacing certain sections of the application completely, writing tests is probably a waste of time. Also focus on the things that need to be updated first, as a priority, so you have the tests completed. I should note you may have to reorganize a bit prior to adding tests. This is dangerous. But, using refactoring tools (Visual Studio or Rehsharper), you can lessen the risk. You need only pull things out far enough you can add unit tests, so extract method is your best friend here.

Summary

If you have never read the book The Checklist Manifesto, I would advice considering it for your bookshelf. The book focuses on how checklists have been used to save lives. Literally! Once you see the importance of a checklist, you can start thinking about automating the checks. Then, you will understand why automated testing is so important.

In this post, I focused on the very high level, mostly aiming at what the developer can do up front to greatly increase quality AND acceptability of the solutions coded. I strongly adhere to the idea of testing as much as is possible. Once I see a pattern in the tests, I generate them (often with Excel, which is a whole story in and of itself). But I focus on ensuring every condition I can think of is put in a test. I will use code coverage as a means of discovering areas I don’t have tests in. This does not mean I will create a test for every piece of code (domain objects being a prime example), but that I use the tools to help me help myself … and my team.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Best Practices, Computers and Internet Tagged with , , ,

Why EF Code First Sucks … and how to fix it if you must use it

November 23, 2018 Leave a comment


NOTE: EF is a tool. As with all tools, it has pros and cons. The headline is designed to get you to examine your useage.

I am a contrarian in many cases. I look at what is in front of me and ask one of the following questions:

  • When would I use this thing I don’t like
  • When would I not use this thing I do like

By constantly questioning, I find myself being a good devil’s advocate. It also keeps me focused on solving problems the “right way”, which means adding in the variables as I find them and validating my knee jerk reaction over and over again. Sometimes this means not adopting the latest FUD, no matter how awesome it is, as it will cause too much disruption (read: too much risk).

As a developer, I think you should explore everything you find and constantly be curious. As an architect, and now a manager, I think you have to take in account how many moving parts you are introducing before making a decision to add something new. I also am very strong at examining pros and cons before striking out. This does not mean I don’t make very quick decisions at times, but that I strive to always do at least some research before firing up the engine and traveling away from home.

For me, Entity Framework makes things easy and much faster to develop. But, it also puts you in a box, like all software designed to develop (even Visual Studio puts you in a box, just larger). There are some projects that fit in the box, but some that do not. As I work on mission critical applications, in an Enterprise setting, I find the maintainability and acceptability of the solution to outweigh time to market in most cases. This makes EF less attractive to me, especially since most EF developers I have met understand the basics and not much more. When you build software any idiot can use, I can guarantee you some of the people using it are idiots. The easier it is, the more idiots.

In particular, this article is focused on issues I am having with a current project I am testing. The problems I have found are not unique, however, as I have seen them in other instances. In this case, there are a few additional issues that exacerbate the problem. And that takes us to “why this post?”

To put it plainly, if you are going to use EF Code First, don’t use the defaults. Let’s dive in.

The problem with EF Code First

Let me be fair. This is not really a problem with EF Code First as much as understanding how to use it correctly. The issue I see is so many people DON’T know how to use it correctly. In fact, it seems to be a one-size fits all tool for so many people. And a great number of them are ignorant about the pros and cons of the product and the cause and effect nature of the issues it creates when used in default. And, as mentioned in the last section, some are idiots. 😉

Auto-generation is attractive. I utilize my skill in coding domain objects, push build and deploy and voila (wah-lah for those who hate French?). Now I have a site. If I just went with defaults, however, I have all of the following problems.

  • My database looks just like my business objects.
  • My interfaces to my data are CRUDdy (simply create, read, update and delete on tables that simply reconstitute my business objects)
  • My database is hard to optimize if looking just like my business objects is not the optimal way to solve it.
  • If I use GUIDs, which are great for certain scenarios (multiple live servers, creation of IDs on disconnected devices, etc.) the default clustered index can create issues down the line (mostly fragmentation)
  • If I use strings in my business objects, they are NVARCHAR(MAX) in my database
  • By default, I have no constraints (other than the auto-generated primary keys). This means no foreign keys in my database.
  • Filtering is done on the user interface.

Once again, I fully realize much of this is ignorance on how you SHOULD design your application with EF Code First, but with no warnings or guidance in Visual Studio to stop the developer from doing this, I am finding this type of bad EF Code First design is more the rule than the exception.

NOTE: I use the word business objects rather than domain models in this post, as I think the lack of domain focus exacerbates this problem. I will write about domain models in a future post and why domaincentric design (I use the made up domaincentric word to distinguish from a purist Domain Driven Design approach – see: Eric Evans).

Annotating with Attributes

I can cure most of the ills with attributes on my business objects. This includes putting keys of the correct type, adding max lengths and avoiding naked strings on objects. I will go through each of the issues I have discovered (not a complete list).

  • Clustered keys on GUIDs (uniqueidentifier data type in SQL Server)
  • NVARCHAR(MAX) fields
  • Foreign keys

Clustered Keys on Guids

When you use a clustered key, the items in a table are stored in the order of the key. With numeric keys, this means 1 comes before 2, etc. With a common key type being an IDENTITY field, or autoincrementing integer, this type of field works nicely. As each key is created, the new data is placed after previous data. You do not end up with table fragmentation very often, which makes maintenance of your database easier.

As long as you continue to add to the end of the table, the clustered key is very efficient. But, when you stick something in the middle of existing records, it causes some extra work, as it fragments the data table. Without getting too much into the internals of RDBMS, or SQL Server in particular, the underlying pages get fragmented. if the data completely fills a page, the fragmentation is cured by reorganizing pages. This is not trivial, but it is far less intrusive than when multiple rows exist in a single page and the internal data of the page has to be reorganized. Yes, there are things you can do to lessen the blow, like altering the fill percentage of the page, but there is a risk of fragmentation.

Why is fragmentation bad? It reduces the efficiency of the database, which can impact performance.

When you use a Globally Unique IDentified (GUID, which is represented as a uniqueidentifier in SQL Server), the GUIDs may be created sequentially, but this is not guaranteed, especially over time. This has to do with the algorithm to create GUIDs to “guarantee” they are unique (there is still a minuscule risk of creating 2 GUIDs that are alike, but it would be extremely rare – 2^128 or 1 in 340,282,366,920,938,000,000,000,000,000,000,000,000 – a very minuscule risk). When a new GUID is created, it can be sequential for quite some time. Then it is not. And over time, you end up with a lot out of sequence. And you start fragmenting. And performance suffers. And little children start crying … really loud.

Fortunately, this is easy to fix.

Version 1: Fragmentation city

[Key()]
public GUID Id { get;set;}

This is the same as the following in SQL Server.

tbd

Version 2: Non-fragmented bliss

Please read this entire section before attempting (Why? Because I am a smart a**). Let’s solve this problem.

[Key(NonClustered)]
public GUID Id { get;set;}

Bliss!

Yes, I a screwing with you. Entity Framework leaves NO WAY to add a non-clustered primary key in Code First with data annotations. So, you have to sling some code. This will require at least EF 6.2 (released a year ago, so you are on this if you are new to EF). In the OnModelCreating event, add the following:

modelbuilder.Entity().HasKey(k => k.Id, config => config.IsClustered(false));

I have also seen it done like this (very slight variation).

p.HasKey(k => k.ColumnId).ForSqlServerIsClustered(false);

The important take away is you dink with Is Clustered and setting it to false. Both create a migration that sets up a non-clustered index.

Cat’s and dogs sleeping together … and world peace.

If you are prior to EF 6.2, I feel really sorry for you. You can create non-clustered indexes, but not non-clustered primary keys. In that case, i would stop using Code First and cry. 🙂

NVARCHAR(MAX)

Before getting into the problem, I want to rant a bit. NVARCHAR(MAX) was created to solve a problem that, for the most part, did not exist. The problem is this:

What if I have a field that normally contains a manageable amount of data, but occasionally contains the entire context of War and Peace?

I see this as a possibility, but a very rare one. In most cases, when you have a s**t ton of data in a field, that is the norm, and when you don’t, that is the norm. Usually you can either constrain or not. And, in cases where you have small amounts of data, it is more often from bad test plans on BLOB (Binary Large OBject) fields where the tester does not want to copy and paste and wait.

NOTE: I am willing to listen to cases where NVARCHAR(MAX) is a proper design choice, as I realize I do not have all of the answers. Just ping me and let’s argue over it. 😉

Okay, I am being a bit too snarky. Let me regain my composure.

NVARCHAR(MAX) is designed to add some performance gains for the rare “I can either type in my name or enter War & Peace” scenarios. Anything small enough to fit into the SQL Server page is put in the page. Anything not, is placed in a type of BLOB storage. This means it can be much faster on records that fit in the page and only slow down on those that overflow.

Sounds Great, right? Yes, but it has some implications. You cannot index an NVARCHAR(MAX) field in SQL Server. I think this should be obvious for anyone who things about it, but let me describe the scenario and see if you can spot the problem.

  • You have a table with NVARCHAR(MAX)
  • There are 1,000,000 records in the table
  • You add the contents of War & Peace into 1/2 the records
  • You create an index on that field

Do you see the problem. Hint: How big is the index.

To solve the fact a person could create a multiple terrabyte index on a field, you cannot index using any BLOB field, even one, like NVARCHAR(MAX), that is pseudo-blob.

But, guess what EF does on strings? If you guessed “creates an NVARCHAR(MAX) field” give yourself a golden star.

Is this a problem? Probably not in most cases, but if you know you are creating a field like city, will you ever have a city that contains 1,000,000 characters? Not until we explore Zeekbart in the Omega quadrant (they are very verbose aliens) or start coding in Entish (LOTR’s creatures for the non-Tolkein fans).

Fortunately, you can solve this one very, very easily. Just add a data annotation.

[MaxLength(100)]
public string City { get; set; }

 

Wala Wala Washington. Now it will no longer create a MAX field and you can index city. And, yes, this is a common use case, although I would consider extracting the city information into it’s own table if you wish to use it as part of your queries (one primary reason for indexing this field is ad hoc searching on demographic fields, like “how many users in New York City had a problem”).

Foreign Keys

This is not an issue, as attributing for foreign keys is extremely easy to do. The problem is I find many people using EF Code first FAIL to do it. Your code is auto-generating your database folks. And constraints are a necessary element of ensuring the database catching bad data by throwing an exception when you try to put bad data in. Never rely on the brilliance of your developers to keep you company in business. Instead constrain the database so they can’t mess things up.

I ran into this problem (not EF, but lack of constraints) with a client in the early 2000s. I was asked to look at a problem with their data. When I logged on an examined the schema, I found no foreign keys. Asking the dev manager, he stated “the foreign keys were causing errors, so we removed them”. I said, “of course they were causing errors, because the code was set up to create data problems.” I got paid for identifying the problem, but they were unhappy I could not fix the missing data. Needless to say, they are out of business now. All because of a lack of foreign keys.

Fortunately, you can avoid this type of embarassment by adding … dun, dun, duh … foreign keys. Guess how? Yes, data annotations.

[ForeignKey(“KeyName”)] public int OtherTableId { get; set; }

I am not going to cover it in this post, but you can do composite (multiple field) keys, as well. I try not to use composite keys as a rule. Not because I am against the idea, per se, but because I find it harder on coders in many instances. If you start with the idea each table has a derived key as its primary key, it is very clean. You can then refactor the idea where a composite key would be better.

The Data Looks the Same

This is more of a design issue, but EF makes it very easy to simply pass objects back and forth between layers. If you adhere to principles of Domain Driven Design, you understand each domain is unique in how it treats models. Let’s take an eCommerce company as an example. In particular, let’s look at the customer object.

  • When an order is taken, a customer ID is added. For the sales department, when the order is retrieved, the customer information, including addresses, is needed to track down any issues.
  • For the warehouse, there is no need to have a customer object. You only need an order number with the items to pull.
  • For the shipping department, you need an address and phone number and nothing else.

I could go on, but if you use EF in default mode, you will be tempted to have a single domain and filter on the user interface.

This is what was done in an application I recently tested. Grab everything for all roles and then filter out what is not needed. But, since the application was not designed out enough, the following problems surfaced.

  • Some smaller tables were retrieved in toto and not filtered for the location or the user. Thus, if I added something, someone in another location saw it, even though they should not have.
  • As binding was not constrained, the tables had all of the information. If I grabbed a user object, I got all of the data if it was not filtered correctly.

Sure, the above have some other design issues that contribute to the problem, but EF in default forces you to filter after retrieval. When you do not, you are likely adding extra work, which may not be justified over other methods of data access (a topic for another day).

The bigger issue is the database storage looks just like the domain model. And, if you are using the EF defaults and auto-generation, you end up stuck in this paradigm. Data can be optimized with a few tools, like indexes, but once you figure it out, you need to make sure the changes are reflected in the EF model. In some cases, this is hard, and that which is hard is rarely tried (until someone threatens to fire you – then again, they threaten and you find a new job, so it is still not done).

In Summary

I am not a huge fan of Entity Framework in the Enterprise (important caveat?). Not because it does not work in many cases, but because it has a lot of moving parts and some that are missing. In most cases, you end up having to use custom migrations to get around the fact it is so easy to step outside of the box created for you. Once you start slinging code with every migration, you start asking whether going to scripted database changes is a better approach than data migrations in EF (I will argue that it IS in a future post, but your mileage may vary and I accept that each tool has its pros and cons).

A hill I will die on is “defaults are bad”, at least when it comes to Entity Framework in an Enterprise setting. I will also state you have to annotate your data if you are going to use your model to generate it. I would prefer don’t design your RELATIONAL database with OOP objects, but your mileage may vary.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Best Practices, Computers and Internet, Entity Framework Tagged with , ,

Windows 10: Getting around “Buy Wi-Fi” on membership networks

October 6, 2015 Leave a comment


So I found a nice “feature” in Windows 10 that really annoys the crap out of me. It is the buy Wi-Fi feature. It seems like it might be a nice feature at times, like when you are somewhere, cannot find free Wi-Fi (rare) and need to connect to get some business done.

The problem with the feature is networks you have membership on, that Microsoft has contracted with, default to “buy wi-fi”, as in the follow capture

image

That’s right, with a Marriott Platinum status, sitting in a room in the hotel, the default is to buy Internet. This is really irritating.

Getting On Without Paying (This case “paying twice”)

So this really bugged me until I got to playing with it. If the Network is one you should be able to get on, you can click on the network and the following pops up. Click on the link circled below:

image

Once you click “other options from the provider”, the logon screen for the hotel appears. In this case, it shows I am still connected from last night. The network will boot me off in a bit.

image

If you select view services, you will then see your purchase options for the network.

image

And there is one more caveat. To get on the network, if you don’t know this already, you have to download the Windows 10 Wi-Fi app from the Microsoft Store just to see the options. Until you download the app, you don’t even have the option to explore other options with the provider.

Personally, I think this is a fail and Microsoft needs to rethink. It would be better, in my opinion to have “other options” be the buy option, as this irritates the crap out of users. Oh, wait, Windows is the only option when you have certain careers?

Peace and Grace,
Greg

Filed under Uncategorized

Microservices in .NET part 2: Silver Bullets and Free Lunches?

June 11, 2015 1 Comment


I have spent the better part of this week digging into micro services and I love the idea. Here are some benefits I see that can be realized by using a microservices approach:

  1. The granularity level allows developers to stay on a single context while solving a problem. This singularity of focus makes it much easier to dig into the details of a specific object or set of objects. In many cases, it will quickly expose poor planning in an organization and provide rationale for fixing the process. As an example, a service that has a lot of churn is probably one that was not planned out well (not talking finding additional uses, but rather having to rebuild contracts and re-architect the service on a regular basis)
  2. The services are simple,making it easy to maintain the individual components.
  3. The methodology forces a good separation of concerns.
  4. You can use the best tool for the job rather than stick to a single platform, programming language or paradigm, etc. This is a dual edged sword, as I will uncover a bit later.
  5. Isolated solution problems can easily be fixed without much impact. If you find your employee microservice has an issue, you can fix it without deploying the entire solution.
  6. Working with multiple services enables the use of concepts like Continuous Integration (CI) and Continuous Delivery (CD). This is also dual edged, as you almost have to go to a full blown CD implementation to use microservices architectures. I will hit this later, as well.
  7. You can get multiple teams working independently of each other. This was always possible, of course, as I have pointed out in my Core As Application blog entries (one here), if you will take the time to plan out you contracts and domain models first. (NOTE: In 2010, I was told “you cannot design contracts first, as you don’t know all of the requirements up front”. By 2011, I proved this wrong by delivering using a contract first approach, both ahead of time and under budget – a bit of planning goes a long way).
  8. Systems are loosely coupled and highly cohesive.

This is just a short list of the benefits. The problem I see is everyone is focusing on the benefits as if we have finally found the silver bullet (do you have werewolves in your organization?) and gained a free lunch. This article focuses on some of the downsides to microservices.

DevOps

As you move to smaller and smaller services, there are many more parts that have to be deployed. In order to keep the solutions using the microservices up and running, you have to be able to push the services out to the correct location (URI?) so they can be contacted properly by the solutions using them. If you go to the nth degree, you could conceivably have tens, if not hundreds, of small services running in an Enterprise.

As each service is meant to be autonomous, this means you have to come up with a strategy for deployment for each. You also have to plan for high availability and failover. And there has to be a solid monitoring and instrumentation strategy in place. In short, you need all of the pieces of a good API Management strategy in place, and you need to do this BEFORE you start implementing microservices. And I have not even started focusing on how everything is wired together and load balancing of your solutions. On the plus side, once you solve this problem, you can tune each service independently.

There is a burden on the Dev side that needs to be tackled up front, as well. You need to start thinking about the requirements for monitoring, tracing and instrumenting the code base and ensure it is part of the template for every service. And you have to plan for failure, which is another topic.

As a final point on this topic, your dev and ops team(s) must be proficient in the combined concept of DevOps to have this be a success. Developers can no longer pitch items over to Ops with a deployment document. They have to be involved in joint discussions and help come up with plans for the individual services, as well as the bigger solutions.

Planning for Failure and Avoiding Failure

Services will fail. Looking at the plethora of articles on microservices, it is suggested you use patterns like circuit breakers (avoid hitting a failed service after a few attempts) and bulkheads (when enough “compartments” are “under water”, seal the rest of the solution from the failure point). This is a fine avoidance strategy, but what if the service failing is a critical component to the solution working?

Not mentioned in the articles I have read is a means of managing services after failure. Re-deploying is an option, and you can make redeployment easier using quickly set up virtual environments and/or containers, but what if reaching that portion of the network is the point of failure. I would love to hear comments on this next idea: Why not look at some form of registry for the services (part of API Management, similar to UDDI, etc.) or a master finder service that exists in various locations and that all applications are aware of. Another idea would be to include as part of the hyper-media specification backup service locations. But either of these solution further exacerbates the reliance on DevOps, creating even more need for planning solutions and monitoring released solutions.

I don’t see microservices working well without a good CI/CD strategy in place and some form of API management. The more I look into microservices the more I see the need for a system that can discover its various points on the fly (which leads me back to the ideas of using a finder service or utilizing hypermedia to inform the solutions utilizing microservices where other nodes exist).

Contracts and Versioning

When you develop applications as a single Visual Studio solution (thinking in terms of projects and not products?), you have the ability to change contracts as needed. After all, you have all of the code sitting in front of you, right? When you switch to an internal focus on services as released products, you can’t switch out contracts as easily. You have to come up with a versioning strategy.

I was in a conversation a few weeks ago where we discussed versioning. It was easy to see how URI changes for REST services required versioning, but one person disagreed when I stated changes to the objects you expose should be a reason for versioning in many instances. The answer was “we are using JSON, so it will not break the clients if you change the objects”. I think this topic deserves a side bar.

While it is true JSON allows a lot of leeway in reorganizing objects without physically breaking the client(s) using the service, there is also a concept of logical breakage. Adding a new property is generally less of a problem, unless that new element is critical for the microservice. Changing a property may also not cause breakage up front. As an example, you change from an int to a long as planning for the future. As long as the values do not exceed the greatest value for an int, there is no breakage on a client using an int on their version of the object. The issue here is it may be months or even years before a client breaks. And finding and fixing this particular breakage could be extremely difficult and lead to long down times.

There are going to be times when contract changes are necessary. In these cases, you will have to plan out the final end game, which will include both the client(s) and service(s) utilizing the new contract, as well as transitional architectures to get to the end game without introducing a “big bang” approach (which microservices are said to help us avoid). In short, you have to treat microservice changes the same way you approach changes on an external API (as a product). Here is a simple path for a minor change.

  1. Add a second version of the contract to the microservice and deploy (do not remove the earlier version at this time)
  2. Inform all service users the old contract is set to deprecate and create a reasonable schedule in conjunction with the consumers of the microservice
  3. Update the clients to use the new contract
  4. When all clients have updated, retire the old version of  the contract

This is standard operating procedure in external APIs, but not something most people think about a huge deal when the APIs are internal.

I am going to go back to a point I have made before. Planning is critical when working with small products like microservices. To avoid regular contract breakages, your architects and Subject Matter Experts (SMEs) need to make sure the big picture is outlined before heading down this path. And the plan has to be conveyed to the development teams, especially the leads. Development should be focused on their service when building, but there has to be someone minding the shop to ensure the contracts developed are not too restrictive based on the business needs for the solutions created from the services.

Duplication of Efforts

In theory, this will not happen in microservices, as we have the individual services focusing on single concerns. And, if we can imaging a world where every single class had a service (microservices to the extreme?) we can envision this, at least in theory. But should we break down to that granlular a level? I want to answer that question first.

In Martin Fowlers article, he talks about the Domain Driven Design (DDD) concept of a bounded context. A bounded context is a grouping of required state and behavior for a particular domain. Martin Fowler uses the following diagram to show two bounded contexts.

In the diagram above, you see some duplication in the bounded contexts in the form of duplication of customer and product. In a microservices architecture, you could conceivably move customer and product to its own service and avoid the duplication, but moving a concept out simply to avoid duplication is not the best motivation in all cases. If you can also make a customer or product business capability, I would wholeheartedly support this approach, but it is not always the case (another side bar).

When would you not want to separate out Customer and Product? In short, when the domain concept of these objects is different. In the sales context, a customer contains sales specific information, including terms of sale (net 60 days?) and other items that may not exist in a support context. If we are talking a company that ships products (as opposed to a service only company), we can add other contexts, like shipping and warehousing, that have radically different customer views. In the warehouse, a customer is completely unimportant, as they are focused on pulling orders. From a shipping standpoint, a customer is a name, a shipping address and a phone number. No need for any additional information. A customer microservice either spits out a complete object, allowing the services to filter (not a great idea from a security standpoint) or it provides multiple interfaces for each of the clients (duplication of efforts, but in a single service rather than multiple consumers and/or services, so it does not avoid duplication). A product can also be radically different in each of these contexts.

My advice to starting out is starting with bigger contexts and then decomposing as needed. The original “microservice” can act as an aggregator as you move to more granular approaches. Example of transitional states from the contexts above.

  1. Discover of duplication in the sales and support microservices leads to a decision the customer and product should be separate services
  2. New customer and product services created
  3. Sales and support services altered to use the new product and customer services
  4. new version of sales and support services created to avoid serving product and customer information
  5. Clients altered to use the new services as well as the sales and support services

This is one idea of migration, as we will discover in the next section.

Where do we Aggregate?

If we go back to the bounded context discussion in the last section, we see the need to aggregate. The question is where to we aggregate? You need to come up with a strategy for handling aggregation of information. I am still groking this, so I am not offering a solution at this time. Here are some options I can see.

Option 1 – Client: In a full microservices architecture, the client may be responsible for all aggregation. But what if the user’s client is a mobile application. The chattiness of a microservice architecture is hard enough to control across your internal multi-GB network infrastructure. Moving this out onto the Internet and cell networks expounds the latency. I am not saying this is a bad option in all cases, but if you opt for this approach, more focus on the latency issue is required from your mobile development team. On a positive note, if the client application can handle single service failures gracefully, you reduce the likelihood of a single point of failure.

Option 2 – Final service boundary: In this approach, the outermost service contacts the many microservices it requires to get work done and aggregates for the client. I find this more appealing, in general, for mobile clients. And it reduces the number of “proxies” required for web, simplifying the user interface client. As a negative, it creates a single point of failure that has to be handled.

Option 3 – Aggregation of dependencies: In this approach, the higher level service (closer to the client) aggregates what it requires to work for the client. At first, I liked this option the best, as it fits a SOA approach, but the more and more I read about the microservices idea, the more I see this as a potential combination of the bad points of the first two options, as you introduce numerous points of failure on the aggregate level while still potentially creating multiple points for latency in your client applications. I still think this might be something we can think through, so I am providing it.

If you can think of other options, feel free to add them in the comments.

Testing One, Two, Three

I won’t spend a lot of time on testability, but the more moving parts you have to test, the harder it is. To understand why this is so, create an application fully covered in unit tests at every level, but developed by different teams, and then integrate. The need for integration testing becomes very clear at this moment. And what if you are not only integrating multiple libraries, but multiple discrete, and very small, services. There is a lot of discipline.

I find the only reasonable answer is to have a full suite of unit tests and integration tests, as well as other forms of testing. To keep with the idea of Continuous Integration, only the smaller tests (unit tests) will be fired off with each CI build, but there will be a step in the CD cycle that exercises the full suite.

There is also a discipline change that has to occur (perhaps you do this already, but I find most people DON’T): You must now treat every defect as something that requires a test. You write the test before the fix to verify the bug. If you can’t verify the bug, you need to keep writing before you solve it. Solving something that is not verified is really “not solving” the problem. You may luck out … but then again, you may not.

Summary

There are no werewolves, so there are no silver bullets. There is no such concept as a free lunch. Don’t run around with hammers looking for nails.The point here is microservices is one approach, but don’t assume it comes without any costs.

As a person who has focused on external APIs for various companies (start ups all the way to Fortune 50 companies), I love the idea of taking the same concepts inside the Enterprise. I am also intrigued by the idea of introducing more granularity into solutions, as it “forces” the separation of concerns (something I find so many development shops are bad at). But I also see some potential gotchas when you go to Microservices.

Here are a few suggestions I would have at this point in time:

  1. Plan out your microservices strategy and architecture as if you were exposing every service to the public. Thinking this way pushes you to figure out deployment and versioning as a product rather than a component in a system.
  2. Think about solving issues up front. Figure out how you are going to monitor your plethora of services to find problems before they become huge issues (downtime outside of SLAs, etc). Put together a disaster recovery plan, as well as a plan to failover when you can’t bring a service back up on a particular node.
  3. In like mind, plan out your deployment strategy and API management up front. If you are not into CI and CD, plan to get there, as manually pushing out microservices is a recipe for disaster.
  4. Create a template for your microservices that includes any pieces needed for logging, monitoring, tracing, etc. Get every developer in the organization to use the template when creating new microservices. These plumbing issues should not requiring solving again and again.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Computers and Internet, Development Practices, Microservices, Service Oriented Architecture

A foray into micro services in .NET

June 6, 2015 2 Comments


Last month, I was given an assignment to attend a workshop with one of our clients. The workshop, as it was called, turned into something more like a hackathon, with various groups attempting to POC some concepts. The big talk of the workshop was micro services.

If you are not familiar with micro services, you should take a look at Martin Fowler’s article. The idea behind micro services is simple. You take an application (which might be a service) and then break it down into smaller, very focused services.

As an example, consider eCommerce. You have online store, which uses a service to handle the various actions required for a customer to order items and have them shipped and tracked. If you examine this from a micro-services perspective, you will separate out the customer shopping cart, order processing, fulfillment and tracking into different services. As you dig deeper you might even go further.

In this entry, I want to dig into micro services a bit deeper, from a Microsoft .NET perspective, but I first want to give some foundation to the micro services concept by understanding how we got here in the first place. That is the main focus of this entry, although I will talk about a high level implementation and some pros and cons.

NOTE: For this article, as with most of my posts, I will define the word “application” as a program that solves a business problem. From this definition, it should not matter how the application interacts with users, and you should not have to rewrite the entire application to create a different presentation (for example, windows to web). I use define the word “solution” as a particular implementation of an application or applications.

Foundations

When we first learn to create applications, the main focus is on solving the problem. This is most easily accomplished by creating a single executable that contains all of the code, which is commonly called a monolith (or monolithic application).

TBD

Monoliths are most often considered bad these days, but there are pluses and minuses to every approach. With the monolith, you have the highest performance of any application type, as everything runs in a single process. This means there is no overhead marshalling across application processes spaces. There is also no dealing with network latency. As developers often focus on performance first, the monolith seems to get an unfair amount of flack. But there are tradeoffs for this level of performance.

  1. The application does not scale well, as the only option for scale is vertical. You can only add so many processors and so much memory. And hardware only runs so fast. Once you have the most and fastest, you are at maximum scale.
  2. There is a reusability issue. To reuse the code, you end up copying and pasting to another monolith. If you find a bug, you now have to fix it in multiple applications, which leads to a maintainability issue.

To solve these issues, there is a push to componentizing software and separating applications into multiple services. Let’s look at these two topics for a bit.

Componentizing for Scale (and Reusability)

While the monolith may work fine in some small businesses who will never reach a scale that will max out their monolith, it is disastrous for the Enterprise, especially the large Enterprise. To solve this “monolith problem”, there have been various attempts at componentizing software. Here are a list of a few in the Microsoft world (and beyond):

COM: COM, or the Component Object Model is Microsoft’s method for solving the monolith problem. COM was introduced in 1993 for Windows 3.1, which was a graphical shell on top of DOS. COM was a means of taking some early concepts, like Dynamic Data Exchange (DDE), which allowed applications to converse with each other, and Object Linking and Embedding (OLE), which was built on top of DDE to allow more complex documents by embedding types from one document type into another. Another common word in the COM arena is ActiveX, which is largely known as an embedding technology for web applications that competed with Java applets.

DCOM and COM+: DCOM is a bit of a useless word, as COM was able to be distributed rather early on. It was just hard to do. DCOM, at its base level, was a set of components and interfaces that made communication over Remote Procedure Call (RCP) easier to accomplish. DCOM was largely in response to the popularity of CORBA as a means of distributing components. COM+ was a further extension of DCOM which allowed for distributed transactions and queueing. COM+ is a merger of COM, DCOM libraries and Microsoft Transaction Server (MTS) and the Microsoft Message Queue service (MSMQ).

.NET: If you examine Mary Kirtland’s early articles on COM+ (one example here), you will read about something that sounds a lot like .NET is today. It is designed to be easily distributed and componentized. One problem with COM+ was DLL Hell (having to clean out component GUIDs (globally unique identifiers) to avoid non-working applications). .NET solved this by not registering components and returning to the idea of using configuration over convention (a long fight that never ends?).

Competing technologies in this space are EJB and RMI in the Java world and CORBA and the CCM as a more neutral implementation. They are outside of the scope of this document.

The main benefit of technologies are they make it easier to reuse code, reducing maintainability, and allow you to more easily distribute applications, providing greater availability and scalability. You can still choose to build monolithic applications, when they make sense, but you are not tied to the limitations of the monolith.

Services

One issue with many of the component technologies was they tightly coupled the consumer to the application, or the “client” to the “server”. To get away from this, a group of developers/architects at Microsoft came up with SOAP (Yes, I know Don Box was actually working for DevelopMentor as a contractor at Microsoft and Dave Winer was heading UserLand software and only partnering with Microsoft on XML and RPC communication, but the majority of the work was done there). With the creation of SOAP, we now had a means of creating applications as services, or discrete applications, which could focus on one thing only. That is sounding a lot like micro services … hmmmm.

In the Microsoft world, SOAP was used to create “web services”. The initial release of .NET in 2002 allowed one to create services using HTTP and SOAP as ASMX services (a type of document in ASP.NET) as well as create faster RPC type services with Remoting (generally these were internal only, as it was hard to play with them outside of the Enterprise, due primarily to more tight coupling to technologies, much less play with them outside of the Microsoft world).

By 2006, with the release of .NET 3.0, Microsoft had merged the concepts of Remoting and ASMX web services in the Windows Communication Foundation (WCF). You could now develop the service and add different endpoints with ease, allowing for an RCP implementation and a web implementation off the same service. WCF really came to fruition about a year later with the release of .NET 3.5.

The latest service technology to enter the fray is Representational State Transfer (REST). In the Microsoft world, REST was first introduced in REST toolkit, an open source project. From the standpoint of an official Microsoft release, it was released as the WCF Web API. It was a bit kludgy, as WCF works in a completely different paradigm than REST, so the project was moved over the web group and is now implemented on top of ASP.NET MVC as the ASP.NET Web API.

Methodologies, Technologies and Tools

One more area we should look at before moving to micro services are methodologies, technologies and tools used to solve the monolith “problem”.

Methodologies

The first methodology that gained a lot of acceptance in the Microsoft world was the n-tier development methodology. The application was divided into UI, Business and Data tiers (note: today Microsoft calls this Presentation, Middle and Data tiers), with the push towards separating out the functionality into discrete, purposed pieces. Below is a typical n-tier diagram for an application.

Around 2010, I realized there was a problem with n-tier development. Not so much with the methodology, as it was sound, but with the way people were viewing the models. Below is an n-tier model of an application:

The issue here is people would see an application as the merger of presentation, business logic and data. Is this true? Let’s ask a couple of questions.

1. If you create a web application with the exact same functionality as a windows application is it 2 applications or one? In implementation, it was often treated as two, but if it was logically and physically two applications, you were duplicating code.

2. If you want to add a web service to expose the functionality of the application, do you rebuild all n tiers? If so, should you?

My view is the application is the part that solves business problems, or the core of the functionality. You should be able to change out where you persist state without changing this functionality. I think most people understand this as far as switching out one database server for another, like SQL Server to Oracle. When we get to the logical level, like changing out schemas, a few get lost here, but physical switches with the same schema is well known and most find it easy to implement. Switching out presentation is what most people find more difficult, and this is generally due to introducing logic other than presentation logic in the presentation portion of the solution.

NOTE: The naming of the tiers in 3-tier and n-tier architecture has changed over time. Originally, it was common to see UI, Business and Data. The illustration above calls the tiers Client, Middle and Data. I have also seen Presentation, Application and Persistence, which is closer to the naming I would use in my models.

To better illustrate this concept, in 2010 I came up with a methodology called Core as Application (seen in this article on the importance of domain models). In this model the core libraries ARE the application. The libraries for presentation can easily be switched out, and have responsibility for shaping the data for presentation.

image

The “Core as Application” model requires you start with your domain models (how data is represented in the application) and your contracts, both for presentation and persistence. Some of the benefits of this model are:

  1. Focusing on domain models and contracts first pushes the team to plan before developing (no, there is no surefire way to force people other than tasers ;->). This is a good representation no matter what methodology or model you use, but it is critical if you are going to have multiple teams working on different parts of a solution.
  2. You can have multiple teams working in parallel rather than relying on completing one layer prior to working on another. You will have to resynchronize if any team determines the contract needs to be changed, but the amount of rework should be minimal.

When you look at “Core as Application” from a SOA standpoint, each service has its own core, with a service presentation layer. The persistence for higher level applications is the individual services. This will be shown a bit later.

Technologies

We have already covered some technologies used to solve the monolith problem. COM and .NET are good examples. But as we move even deeper, we find technologies like the ASP.NET Web API is useful. The technologies do not force us to not create monoliths, as even Microsoft pushes out some examples with data access, as LINQ to SQL, in a controller in an MVC application. But they do get us thinking about creating cohesive libraries that serve one purpose and classes that give us even more fine grained functionality.

Tools

We also have tools at our service. Visual Studio helps us organize our code into solutions and projects. The projects focus on a more fine grained set of functionality, helping us break the monolith up. If we follow best practices, our solutions end up with more projects, which can easily be broken down into individual micro services. Speaking of which, this a good time to segue.

Onward to Micro Services

Micro services are being presented as something new, but in reality, they are nothing more than Service Oriented Architecture (SOA) taking to the nth degree. The idea behind micro services is you are going extremely fine grained in your services. It is also stated you should use REST, but I don’t see REST as an absolute requirement. Personally, I would not aim for SOAP as there is a lot of extra overhead, but it is possible to use SOAP in micro services, if needed. But I digress … the first question to answer is “what is a micro service?”

What is a Micro Service?

I am going to start with a more “purist” answer. A micro service is a small service focused on solving one thing. If we want to be purist about it, the micro service will also have its own dedicated database. If we were to illustrate the order system example we talked about earlier, using the “Core as Application” model, the micro-services implementation would be something like the picture below.

image

If you want to take it to an extreme, you can view micro services the way Juval Lowy viewed in 2007 (video gone, but you can read the description). His idea was every single class should have a WCF service on top of it. Doing so would create a highly decoupled system possible, while maintaining a high level of cohesiveness. Micro services does not dictate this type of extreme, but it does recommend you find the smallest bounded context possible. I will suggest a practical method of doing this a bit later.

image

One difference between Juval Lowy’s suggested “one service per class” and micro services is the suggested service methodology has changed a bit. In 2007, WCF was focused on SOAP based services. Today, REST is the suggested method. Technically, you can develop a micro service with SOAP, but you will be adding a lot of unnecessary overhead.

Below are Martin Fowler’s characteristics of a micro service:

· Componentization via services – This has already been discussed a bit in the previous two paragraphs. Componentization is something we have done for quite some time, as we build DLLs (COM) or Assemblies (.NET – an assembly will still end in .DLL, but there is no Dynamic Link Library capability for COM inherently built in without adding the Interop interfaces). The main difference between an assembly (class library as a .dll file) and a micro service is assembly/dll is kept in process of the application that utilizes it while the micro-service is out of process. For maximum reuse, you will build the class library and then add a RESTful service on top using the ASP.NET Web API. In a full micro services architecture, this is done more for maintainability than need.

· Organized around business capabilities – This means you are looking for bounded contexts within a capability, rather than simply trying to figure out the smallest service you can make. There are two reasons you may wish to go even smaller in micro services. The first, and most reasonable, is finding a new business capability based on a subset of functionality. For example, if your business can fulfill orders for your clients, even if they are not using your eCommerce application, it is a separate business capability. A second reason is you have discovered a piece of functionality can be utilized by more than one business capability. In these cases, the “business capability” is internal, but it is still a capability that has more than one client. Think a bit more about internal capabilities, as it may make sense to duplicate functionality if the models surrounding the functionality are different enough a “one size fits both (or all)” service would be difficult to maintain.

  • Product Not Projects – This means every service is seen as a product, even if the only consumer is internal. When you think of services as products, you start to see the need for a good versioning strategy to ensure you are not breaking your clients.
  • Smart Endpoints and dumb pipes – I am not sure I agree completely with the way this one is worded, but the concept is the endpoint has the smarts to deliver the right answer and the underlying delivery mechanism is a dumb asynchronous message pump.
  • Decentralized Governance – Each product in a micro services architecture has its own governance. While this sounds like it goes against Enterprise concepts like Master Data Management (MDM), they are really not opposites at all, as you will see in the next point
  • Decentralized Data Management – This goes hand in hand with the decentralized governance, and further illustrates the need for Master Data Management (MDM) in the organization. In MDM, you focus on where the golden record is and make sure it is updated properly if a change is needed. From this point on, the golden record is consulted whenever there is a conflict. In micro services, each micro service is responsible for the upkeep of its own data. In most simple implementations, this would mean the micro service contains the golden record. If there are integrated data views, as in reporting and analytics, you will have to have a solution in place to keep the data up to date in the integrated environment.
  • Infrastructure Automation – I don’t see this a mandatory step in implementing micro services, but it will be much harder if you do not have automation. This topic will often start with Continuous Integration and Continuous Delivery, but it gets a bit deeper, as you have to have a means of deploying the infrastructure to support the micro service. One option bandied about on many sites is a cloud infrastructure. I agree this is a great way to push out microservices, especially when using cloud IaaS or PaaS implementations. Both VMware and Microsoft’s Hyper-V solutions provide capabilities to easily push out the infrastructure as part of the build. In the Microsoft world, the combination of the build server and release management are a very good start for this type of infrastructure automation. In the Linux world, there is a tool called Docker that allows you to push out containers for deployment. This capability also finds its way to the Microsoft world in Windows Server 2016.
  • Design For Failure – Services can and will fail. You need to have a method of detecting failures and restoring services as quickly as possible. A good application should have monitoring built in, so the concept is nothing new. When your applications are more monolithic, you more easily determine where you problem is. In micro services, monitoring becomes even more critical.
  • Evolutionary Design – I find this to be one of the most important concepts and one that might be overlooked. You can always decompose your micro services further at a later date, so you don’t have to determine the final granularity up front. As a micro service today can easily become an aggregator of multiple micro services tomorrow, There are a couple of concepts that will help you create your microservices we will discuss now: Domain Driven Design and Contract First Development.
Domain Driven Design

Domain Driven Design (DDD) is a concept formulated by Eric Evans in 2003. One of the concepts of DDD that is featured in Fowler’s article on micro services is the Bounded Context. A bounded context is the minimum size a service can be broken down into and still make sense. Below is a picture from Fowler’s article on Bounded Contexts.

When you start using DDD, you will sit down with your domain experts (subject matter experts (SMEs) on the domain) and find the language they use. You will then create objects with the same names in your application. If you have not read Eric Evans Domain Driven Design book, you should learn a bit about modeling a domain, as it is a process to get it right.

NOTE: you are not trying to make your data storage match the domain (ie, table names matching domain object names); let your database experts figure out how to persist state and focus on how the application uses state to create your domain objects. This is where Contract First comes into play.

Contract First Development

Once you understand how your objects are represented in your domain, and preferably after you have a good idea of how the objects look in your presentation projects and your data store, you can start figuring out the contracts between the application, the presentation “mechanism” and the persistent stores.

In general, the application serves its objects rather than maps them to presentation objects, so the contract focuses on exposing the domain. The presentation project is then responsible for mapping the data for its own use. The reason for this is presenting for one type of presentation interface will force unnecessary mapping for other types. As an example, I have seen n-tier applications where the business layer projects formatted the data as HTML, which forced writing an HTML stripping library to reuse the functionality in a service. Ouch!

How about the persistence “layer”? The answer to this question really depends on how many different applications use the data. If you are truly embracing micro services, the data store is only used by your service. In these cases, if storage is different from the domain objects, you would still desire spitting out the data shapes that fit the domain objects.

How to Implement Microsoft Services, High Level

Let’s look at a problem that is similar to something we focused on for a client and use it to determine how to implement a micro services architecture. We are going to picture an application for an Human Resources (HR) services company that helps onboard employees We will call the company OnboardingPeople.com (and hope it is not a real company?).

Any time a new employee comes on board there is a variety of paperwork that needs to be completed. In the HR world, some of the forms have a section that is filled out by the employee and another that is filled out by a company representative. In the micros services application architecture, we might look at creating a micro service for the employee portion and another for the employer portion. We now have a user facing application that has two roles and uses the two services to complete its work. Each service surrounds a bounded context that focuses on a single business capability.

image

Over time, we start working with other forms. We find the concept of an employee is present in both of the forms and realize employee can now represent a bounded context. The capability may only be internal at this time, so there is a question whether it should be separated out. But we are going to assume the reuse of this functionality is a strong enough reason to have a separate capability. There is also a possibility the employer agent can be separated out (yes, this is more contrived, but we are envisioning where the solution(s) might go).

image

If we take this even further, there is a possibility we have to deal with employees from other country, which necessitates an immigration micro service. There are also multiple electronic signatures needed and addresses for different people, so these could be services.

image

In all likelihood, we would NEVER break the solution(s) into this many micro services. As an example, addresses likely have a slightly different context in each solution and are better tied to services like the employee service and employer services rather than a separate service that is able to keep context

Pros and Cons

While micro services are being touted in many articles as THE solution for all applications, silver bullets don’t exist, as there are no werewolves to kill in your organization. Micro services can solve a lot of pain points in the average Enterprise, but there is some preparation necessary to get there and you need to map it out and complete planning before implementing (I will focus on another article to go into more detail on implementation).

Pros

One of the main pros I see mentioned is the software is the right size with micro services. The fact you are implementing in terms of the smallest unit of business capability you can find means you have to separate the functionality out so it is very focused. This focus makes the application easier to maintain. In addition, a micro services architecture naturally enforces tight cohesion and loose coupling. Another benefit is you naturally have to develop the contract up front as you are releasing each service as a discreet product.

You also have the flexibility to choose the correct platform and language on a product by product (service by service) basis. The contract has to be implemented via standards for interoperability, but you are not tied into single technology. (NOTE: I would still consider limiting the number of technologies in use, even if there is some compromise, as it gets expensive in manpower having to maintain multiple technologies).

Micro services will each be in control of their own data and maintain their own domains. The small nature of the services will mean each domain is easy to maintain. It is also easier to get a business person focused on one capability at a time and “perfect” that capability. It goes without saying micro services work well in an Agile environment.

Micro services architecture also allows you the ability to scale each service independently. If your organization has adopted some form of cloud or virtual infrastructure, you will find it much easier to scale your services, as you simply add additional resources to the service.

Cons

Micro services is a paradigm shift. While the concepts are not radically different, you will have to force your development staff to finally implement some of the items they “knew” as theory but had not implemented. SOLID principles become extremely important when implementing a micro services architecture. If your development methodologies and staff are not mature enough, it will be a rather major paradigm shift. Even if they are, a shift of thinking is in order, as most shops I have encountered have a hard time viewing each class library as a product (Yes, even those who have adopted a package manager technology like NuGet).

There is a lot of research that is required to successfully implement a micro services architecture.

· Versioning – You can no longer simply change interfaces. You, instead, have to add new functionality and deprecate the old. This is something you should have been doing all along, but pretty much every developer I have met fudges this a good amount of the time. It is internal, so I can fix all of the compiler errors, no problem. This is why so many shops have multiple solutions with the same framework libraries referenced as code. You should determine your versioning strategy up front.

· URI Formatting – I am assuming a REST approach for your micro services when I include URI formatting, but

· API Management – When there are a few services, this need will not be as evident. As you start to decompose your services in to smaller services, it will become more critical. I would consider some type of API management solution, like Layer 7 (CA), Apigee, or others, as opposed to building the API management or relying on an Excel spreadsheet or internal app to remind you to set up the application correctly.

· Instrumentation and Monitoring – Face it, most of us are bad at setting the plumbing, but it becomes critical to determine where an error occurs in a micro services architecture. In theory, you will know where the problem is, because it is the last service deployed, but relying on this idea is dangerous.

· Deployment – As with the rest of the topics in this section, there is a lot of planning required up front when it comes to deployment. Deployment should be automated in the micro services world. But deployment is more than just pushing applications out, you need to have a rollback plan if something goes wrong. In micro services, each service has its own deployment pipeline, which makes things really interesting. Fortunately, there are tools to help you with build and release management, including parts of the Microsoft ALM server family, namely build server and release management.

In short, micro services are simple to create, but much harder to manage. If you do not plan out the shift, you will miss something.

Summary

This is the first article on micro services and focuses on some of the information I have learned thus far. As we go forward, I will explore the subject deeper from a Microsoft standpoint and include both development and deployment of services.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Uncategorized

Solving the Microsoft Mahjong Classic Puzzle–January 9th

January 23, 2015 Leave a comment


This particular puzzle was a bit of a b*tch. Despite being flagged easy, it takes quite a bit of time, largely due to the placement of a few tiles that requires a strategy that is a bit out of the norm. Here is my guide to solving the puzzle using the tranquility tile set. (If you are using another tile set, you can still use this post, but will have to go by position alone.

First, here is how it looks as you start.

Starting-Point

To explain the solution, I need to do 2 things:

  1. Identify the tiles in shorthand
  2. Grid off the board

Identifying the Tiles

In Mahjong, there are a group of tiles that can only match an identical tile; these are 3 suits (wheels (or dots), bamboo (or sticks) and numbers (or cracks)). the four winds (north, south, each and west) and a set of three dragons (red, green and white).

There are also two suits in which you can match any of the tiles: Seasons (spring, summer, winter, fall) and flowers (plum, orchid, bamboo and mum).

Wheels

The wheels are easy to identify. They have a dot or wheel. You count the number of wheels to determine the number of the tile. I label these with a lower case w after the number: 1w – 9w.

Bamboo

Bamboo is also easy, as you count the number of bamboo fronds. The exception is the 1 of bamboo, which looks like a bird. I label these with a lower b after the number: 1b – 9b.

Numbers

Numbers are a bit different unless you read Chinese. Here is how to identify them. 1, 2 and 3 of numbers have 1, 2 and 3 vertical lines across the middle of the top of the tile. The 4 looks like an arabic W. 5 is the most complex symbol. It has an up and down stroke followed by what appears to be a lower case h with a cross at the top and underline. 6 is a stick figure without the body segment. 7 appears a bit like a t, 8 like a broken upside down v and 9 is like a cursive r. I label numbers with a lower case n: 1n – 9n. Some of the numbers are shown below.

numbers

Winds

Winds are easy to identify, as they have the initial of the wind direction in black in the upper left corner of the tile: N, S, E, W.

Dragons

There are three dragons: red, green and white. The red dragon is a stick with a circle colored red. The green dragon appears like a green bush or a green flame. And the white dragon is a blue square. I have a red and blue dragon shown below, with the numbers all ready on the graphic:

dragons

Seasons

Seasons are found on green tiles. You can match any season with any other season.

seasons

Flowers

 

Gridding off the board

On this board, there are 5 rows from top to bottom, starting from the 2 of wheels (or dots) in the middle top. and ending with row5 which has a south wind on the far left side. I am going to label these rows 1 through 5.

Rows

There are 13 rows, which I label A through M. Leaving the rows in place, here are the columns.

Rows-&-Columns-&-Half

Notice that some of the tiles, like the 2 of bamboo (2b) on row 4 (partially hidden by the West wind (Wwind). These are half columns, which I label by adding an X. In particular, this is column Ax. Here is a shot showing the rows and the half columns

Rows-&-Halfs

And here is rows, columns and half columns.

Rows-&-Columns-&-Half

Designating Position

To designate a position, I also need to indicate the level of the tile. On this board, the 4 tiles on row 4 (starting with a flower and ennding with the West wind are at level 4. The North wind located at 5-M is at level 2 (there is one tile underneath it.

My location designator consists of row, column and level, with hyphens in between. The North wind at the lower left, for example, is 5-M-2. I will add the tile name (or type in flowers or seasons) in front of the location. Here is a map of a few positions, so you can understand the system.

Locations

Solution

As pointed out on the Mahjong FaceBook page, you have to focus on the bottom row. This is normal when you are solving a Mahjong puzzle, as you should focus on three areas.

  1. long lines
  2. Tall stacks
  3. tiles that cover more than one tile

Not focusing on these areas is bad. This particular puzzle is a bit insidious as there are some tiles that only have one pair and are located under other tiles that have only one pair. This makes it hard to clear off the board. Here are the areas to look out for:

North on top of north
North-on-North

2 6 of bamboo (only pair of them in game) on top of row 5
2-6-of-bamboo

Two white dragons on row 5 at level 2
2-white-dragons

Two red dragons on the bottom level of row 5:
2-red-dragons

The general strategy here (as mentioned on the Facebook page) is to start row 5, level 3 from the left side, then level 2 from the right side and level 1 from the left side. This is not 100% true, as you will see.

The first step is to start clearing row 5, level 3, from the left. There is a 5n here at 5-B-3. You can see another 5n under one of the flowers. The designation for clearing this tile, fitting the scheme is 5n 2-J-2 5-B-3. To get to this tile, you have to clear flowers, which are designated flowers 2-Ix-3 3-G-4.

First-2-moves

You then continue on the left side of row 3 for 2 more moves. Here are the first four moves. Counter intuitive moves are marked with a starStar.

Move Tile Location 1 Location 2
1 flowers 2-Ix-3 4-G-3
2 5n 2-J-2 5-B-3
3 Star 1n 2-C-2 5-C-3
4 Wd 1-J-1 5-D-3

The 3rd move, above, is counter intuitive. You would think you should clear off 5-B-3 with 3-K-2, but I found this move makes the puzzle unsolvable. At the end of the three moves, you have a board that looks like this:
board-after-4-moves

It should be clear enough how to use the moves table now. Here are the moves to clear off the rest of the top of row 5. Note that we will start from the left side now (purple), as we will with row 2 (blue). Also note this table has some rows that say ALL, followed by a number of pairs. This means clear everything of that type off the board.

Move Tile Location 1 Location 2
5 Nwind 4-H-4 5-M-2
6 Wwind ALL (3 pairs)  
9 Rd 4-F-3 4-I-4
10 4n 2-Hx-4 4-E-2
11 Swind ALL (2 pairs)  
13 EWind 3-I-4 5-k-3
14 9n 1-I-2 4-Bx-2
15 8n 3-B-3 4-Bx-1
16 4b 1-I-1 4-Ax-2
17 2w 2-Gx-3 4-G-3
18 3w ALL (2 pairs)  
20 7w 3-Ax-1 4-L-1
21 5b 1-H-1 3-I-3
22 Mug Nwind 2-G-2 5-I-3
23 4w 3-Ex-1 5-H-3
24 6w 2-F-1 2-K-1
25 5w 2-J-1 4-J-3
26 3b 4-I-3 5-G-3
27 1n 3-K-2 5-F-3
28 Wd 3-J-2 5-E-3

Move 22 (beer icon) is also critical, as there is a north on a north in position 2G (at level 2 and 1). Exchanging either of these north winds for the north wind at 4-G-2 At this point in time, the top of row 5 should be cleared, as shown below:
After-28-moves

The next step is to clear off both row 4 and 5. Because of the 6 of bamboo (6b), you have to start row 5, level 2 from the right. Here are the steps to clear row 5, level 2 (purple) and much of row 4, level 2 (blue). 

Move Tile Location 1 Location 2
29 1b ALL (2 pairs)  
31 9w 3-G-3 4-G-3
32 Nwind 1-G-1 4-G-2
33 7b 5-K-1 5-L-2
34 Ewind 3-Kx-1 4-Ax-1
35 6n 2-A-2 3-G-2
36 2b 2-Hx-2 3-Jx-1
37 8b 3-Bx-1 3-I-2
38 Swind 3-H-2 4-H-2
39 1w 2-I-2 5-J-2
40 Seasons ALL (2 pairs)  
42 3n 4-J-2 5-H-2
43 9b 4-K-2 5-G-2
44 4n 3-G-2 5-F-2
45 Gd 2-H-2 5-E-2
46 6b 5-B-2 5-D-2
47 1w 3-Ix-1 5-C-2

Here is the board after these moves:
Near-Solving

You should now be able to solve this without my help, but here are the moves.

Move Tile Location 1 Location 2
48 flower 2-H-1 5-B-1
49 2w 1-G-1 4-K-1
50 Gd 4-J-1 5-C-1
51 2n ALL (2 pairs)  
53 8w 4-H-1 5-E-1
54 3n 5-F-1 5-K-1
55 Rd 5-G-1 5-J-1
56 7n 5-H-1 5-I-1
57 3b 3-Hx-1 4-G-1
58 2b 2-A-1 3-Gx-1

Hope this helps.

Peace and Grace,
Greg

Twitter: @gbworld

Filed under Uncategorized

Analyzing the Michael Brown Shooting

December 4, 2014 1 Comment


I got a chance to go through all of the witness testimony the grand jury released (minus any embedded in the Grand Jury transcripts). I did this to see whether this was a clear cut case of a police murder or something more justified.  Everything in here is taken from my notes, and I have linked numerous documents so you can read the originals. You can find the bulk of the documents linked from this page. You can also find the same on USA Today (which also includes public videos entered into evidence) and the New York Times site.

Maps

Below are a couple of maps of the area. The first was taken from the web and shows some locations. This is the map that gave me the idea to determine where people were.

And here is my graphic, taken from Google maps, with the locations of various witnesses marked (these are keyed by the same numbers shown in the post). NOTE: X in a box is where Michael Brown’s body was at the end of the altercation. The blue dot is where Officer Wilson’s car was stopped.

BrownMap

NOTES

  • Numbers are color coded per the description in the next section. Red are those that appear to be false (either outright lying or recollecting from false testimony) and orange are suspect in at least part of the testimony.
  • I am not completely sure the order of witnesses 30, 34 and 40. Assume 30 to be closer as he states the cop was between him and the boy and 34 burned out, which would have been easier near the parking lot entrance. It is known both were in their approximate locations, just the ordering is suspect.
  • The exact positions of 38 and 45 are also unknown, so I have placed them near the dumpster.
  • Witness 44 started back a bit further and likely ended a bit closer than his marker, based on his testimony. The marker is an approximation of mid-point and good enough for this exercise (to see why examine discredited witnesses).
  • I have no clue which apartment witness 57 heard the shooting from or which stop sign witness 64 was at. There are no stop signs at any street where a witness would have visibility to the shooting, at least not on the current views on Google maps.
  • The question marks indicate where I think the witnesses were based more on testimony than their statement of location. In other words, they are educated guesses.
  • The phone symbol is where one of the videos was shot that appears to fit Wilson’s side of the story. There is also one from Piaget Crenshaw (witness 43 – correction on this; witness is male, Piaget’s location very close, however) and apparently one from witness 41, although I cannot find it.

Wilson Interview

NOTE: Prior to the incident, Wilson had gone to a call in the Sharondale apartments, northeast of the point of the altercation. This is on track 324, page 1 of the radio log. Wilson got a call of a “stealing” going on. The information on the suspect is on page 7 (track 358) of the radio log. On track 364, page 8, he puts himself back into service (10-8). He then asks 25 or 22 if they need assistance.

NOTE: The stealing info is interesting, as people have stated there is no way Wilson could have known about the shoplifting, as the owner had not called anything in.

Saw two black males walking down Canfield. He told them to get on the sidewalk and they said they were almost to their destination. He asked “but what’s wrong with the sidewalk” and they answered “fuck what you have to say”. Wilson then called in for backup (track 369, page 9) and backed up his vehicle. Officer 25 answers the call (log page 10, track 372).

After backing up about 10 feet, Wilson tries to open the door and Brown says “What the fuck you gonna do?” and shuts the door on him, leg out. He then tries to reopen the door, Brown says something, closes the door and starts punching through the window. The first blow is a glancing blow. Brown then hands the cigarillos to Johnson and hits him again. Wilson thinks of using his mace, but it is on the wrong side, so he grabs his gun and aims it at Brown, who puts his hand over the top of the gun and pushes it towards Wilson.

Wilson states a shot was fired in the car and Brown ran. He exits the vehicle, states “shots fired. Send me more cars”. Brown stopped, turned, made a grunting noise and the most aggressive face he had ever seen and then charged with his hands in his waistband. Wilson ordered him to stop and then fired multiple times. Wilson yelled at him to get on the ground again. Brown, still hand in waistband, charged Wilson, and Wilson fired until Brown went down. He was about 8-10 feet away at this time. He then called “send me every car we got and a supervisor”. About 15-20 seconds later, two marked cars show up and started blocking off everything.

 

Dorian Johnson Story

Dorian Johnson was the closest to Brown at the time of the initial contact.

August 9

 

Witness Interviews

These are taken in order of when they were interviewed. Closest to the event are usually more accurate, although this is not always the case. I have included markings to indicate which witnesses are not credible and explain the reasoning later.

* (asterisk)  is used to indicate a witness I find not credible. red is used to indicate a witness that is likely lying or recollecting things rather than going by actual observations (I believe 2 are outright lying, while 2 are filling in way too many gaps), while orange is suspect, used for a witness that has some useful details, but some that are not likely to be true (either filled in some how (recollected?) or added due to personal feelings). Some of the witnesses in these categories state the shooting was execution style, while other take the other extreme, and some are in the middle.

Same day as shooting

* Witness 16 (8/9/2014 2:19 PM and 9/11/2014): Located approximately 150 feet from the police car in a third floor apartment (assume this was also 2973 Canfield Court instead of building next door, as she could not see where Brown was in the street from the other building). She heard a “skirr”, or car “screech”  and looked out her bedroom window and saw Brown tussling with Wilson in the car. She then heard two shots, one which she assumes is the one that hit the building across the street, and saw officer Wilson get out of the car with a  red face, which she assumed was from anger. She then moves from the bedroom to the living room and sees Michael Brown coming towards Wilson with his hands up, at his shoulders, and then shooting him at 2-3 yards. Under a second interview with the FBI, she describes getting her purse, then walking to the living room, then goes and gets her cell phone so she can record the scene (she has a video after the shooting that can be viewed) and then has to open the blinds in the living room. She then admits it took too long to get back and she only saw the tussle and then Brown laying in the street.

* Witness 43 (8/9/2014 4:12 PM): In father’s apartment, upstairs.Not sure where this apartment is, as the description does not get into location. I believe this is Piaget Crenshaw. NOTE: She later mentions seeing Brown with his hands up, although that is missing from her initial testimony (she looked away) and she has a lot of details. Can confirm not Piaget from end of video.

She heard screaming from Brown as he was reaching into Wilson’s car. Wilson tried to taze Brown, but failed. He then pulled out his gun and tried to shoot Brown, but failed. Brown started running and he shot him in the back. She looked away, as she thought it was over, but then heard 4-5 more shots and went back to see Brown lying in the street.

Witness 12  (8/9/2014 (twice, first at 4:40 PM, second 4:45 PM), 8/13/2014 and at least 1 media interview): This is Michael Brady, based on testimony. Location, first floor at 2973 Canfield Court, approximately 150 feet from the police cruiser. He was located Heard an altercation and came to the window to see Brown and Wilson fighting through the window of the police cruiser (“arms goin’ through the window”). Heard 2 shots from in car and then Brown runs. Wilson then shoots two more shots (8/13/2014) or 6 shots (8/9/2014) or 3-4 shots (8/13/2014). Brown then turns around curled up and walks towards Wilson, with his hands “probably up” and Wilson shoots 3-4 more times (most interviews) or 4-5 times (MSNBC interview). Brady’s testimony is all over the place, but the main thread stays consistent. Later testimony seems more recollection than remembrance.

Witness 25 (8/9/2014 4:59 PM): In kitchen on phone when fiancée called him to the window. Saw Wilson waking after a running Brown shooting steadily at him. Brown then turned to Wilson and was shot 2-3 more times.

Witness 22 (8/9/2014 5:06 PM): Located in a third story apartment, not sure of exact location. Sleeping, but awakened by the first two shots (from inside the car?). Boyfriend told her to come to the window. Saw Brown grabbing stomach or side and Wilson just kept firing. Then saw Brown kneeling in the street with his hands up and Wilson shooting him 3 times.

Witness 32 (August 9, 2014 6:40 PM): In car right behind Wilson, about 10 feet behind, heading west on Canfield. Saw Brown and Johnson in the middle of the street walking towards. Wilson said something to him (assumed, not heard) and Brown tussled with him in the car and a shot rang out. Brown backed up and ran east on Canfield, past the witness, while Johnson tries to get in the car. Both Brown and Wilson pass by car and through rearview mirror sees Brown fall after 3 more shots. Then takes off quickly, driving partially over the grass.

Witness 40 (August 9th journal entry): Heading east on Canfield very close to scene (since states Johnson told him to leave after shooting, most likely either in front of or behind witness 34, putting him 5-15 feet from the police car). White person, evidenced by first line in journal entry stating “need to understand the black race better so I stop calling blacks n*******s and start calling them people.

Saw police car back up and almost hit Brown and Johnson. Brown hit the door and looked pissed. Saw Brown hit car door with belly (this fits Johnsons description of Wilson trying to open the door). Johnson hits mirror on police car. Saw Brown in the window and then saw Brown running. Wilson got out of car with his left hand on his face and right hand grabbing his gun. Wilson yelled something at Brown, although the witness could not hear it as other people were already yelling outside. Brown turned around at Wilson (“with attitude”) and started running at Wilson, head down, like a football player. Wilson shot 3 times, but Brown continued advancing. Wilson fired 2 more shots, and Brown backed up. Mentions Johnson told him to leave.

August (not same day)

Witness 10 (8/11/2014): Working in the building at 2973 Canfield Court – same building as witness 12 and 16); not sure of exact distance, as he was taking things in and out of the building (120-150 feet). Saw Brown wrestling with Wilson through the window of the car and a shot went off. Brown took off running (witness thought Wilson had been murdered). Brown turns and makes a movement and then starts charging Wilson. Wilson shoots 4-6 shots. Brown stops and Wilson stops firing. Brown then charges Wilson again and Wilson fires 4-5 more shots and Brown falls to the ground.

Witness 14 (8/12/2014 and 9/24/2014): Located in apartment that looks straight down Canfield towards West Florissant (same building as Witness 10, 12 & 16?). Saw two young men walking down street approached by police cruiser. Then saw Brown tussling with Wilson while he was still in the truck. Heard a shot and then saw Brown run 25-30 feet from the car. Wilson gets out about 3 seconds later and comes around car to passenger’s side. Yells “stop” at Brown. Brown takes 2-3 steps and was shot 2-3 times. Brown lifts hands to shoulder level (pleading?) with his palms out at his rib cage level, and continues to step forward. Wilson yells stop again and then “lets loose” on Brown.

Witness 30 (August 13, 2014): Was driving east on Canfield, heard 3-4 shots and saw Brown hit in the leg. Brown turned around and lifted at least one hand. Witness thought he had a gun. Wilson then shot Brown.

* Witness 35 (August 13, 2014): Located in 3rd floor window, nearest apartment building to Northwinds apartments (2909 Canfield Drive), which puts him more than 200 feet from where Brown ended up in the street and close to 300 feet from Wilson’s car. Witness self-described as Brown’s best friend and Johnson’s cousin. Says he was on the phone when the first shot rang out (missed fight in car). Ran to window and saw Brown on knees with blood rushing from shoulder or rib cage. He heard Brown tell Dorian to run for his life and then saw Wilson get out of his truck and shoot him in the head. At this time, Brown was no more than 5 feet from the police cruiser. Also mentions saw Brown shot 4 times, then heard him pleading for his life and saying “don’t shoot me”, heard 4 more shots as he ran downstairs to the scene. Stated Wilson shot his 10 times at close range.

* Witness 48 (8/14/2014, also phone call follow up): In the middle row of a van on Canfield turning left at a stop sign (had to be Coppercreek Road to be near a stop sign and able to see any of the altercation). This would put her at least 200 feet from the police car and farther from where Brown was shot. Heard 2 shots and then sees Brown run. Brown turns to Wilson and runs at him, hands balled up. Wilson yells stop at least three times. Wilson shoots three times, stops and then shoots Brown again.

Witness 46 (8/15/2014): In car heading east on Canfield. Pulled over for police car to pass, as he/she had an outstanding warrant for tickets. If the story is true (doubtful), he was right in front of the shooting location. He saw “two mens” talking to a cop (who just passed him not too long ago?). The boys were heading east, the cop was heading east. The cop then stuck his gun out the window and shot Brown, and ended up shooting about 2 shots. After the shooting stop, the witness got out of the car and asked Wilson if he could help and Wilson told him “get the f*ck on”.

* Witness 45 (8/16/2014): Was near the dumpster at 2973 Canfield Court (In front of Brady’s (witness 12) apartment), about 75-100 feet from police car. A police car with 2 officers approaches Brown and Johnson, drives off and then backs up and almost hits them. Brown then goes to the window “either to defend hisself or to give up”. Saw Wilson grab Brown by the collar and then the first shot rang out. Brown backed up with a blood spot on his shirt. Wilson fired the second shot as he started to run. Brown turns around and puts his arms up and Wilson guns him down. The second officer than gets out of the vehicle.

* Witness 44 (8/16/2014 and 9/25/2014): Walking on the south side of Canfield, near the leasing office. Approximately 150 feet from the police car. Saw Wilson back up the car, hitting Brown and Johnson and then Brown running around to the driver’s side of the car for about 15 seconds. Brown then fights Wilson through the window, while Johnson runs away. A shot rings out and Brown backs up and checks himself, and then runs away. The gun falls to the ground. Brown runs about 10-12 steps and then turns to face Wilson, who is now out of the car (number of steps changes a lot in the interviews). Brown puts his hands up  like “I’m done” or “arrest me” and Wilson shoots him 7 or 8 times, 6 of them while Brown is backing up.

* Witness 42 (8/16/20148/16/2014):: In apartment on Canfield (very close to shooting, so most likely 2943 or 2947 Canfield. This would put him somewhere between 30-50 feet of where Michael Brown was shot, although some of the scene would have been obstructed depending on which apartment he was in.

He was on the phone when he heard the first shot (in the car?). Saw Wilson running after Brown and then shoot him in the back. Brown turned around and threw his hands in the air. Wilson unloaded his gun in Brown’s direction, unknown number of shots (“it was gruesome”). Shooting started about 20 feet apart, but ended with an execution style shooting with Brown on his knees (less than an arm’s length) while Michael Brown was saying “don’t shoot”. While Brown on the ground, he continued to shoot the body.

* Witness 38 (8/16/2014): Was near the dumpster at 2973 Canfield Court (In front of Brady’s (witness 12) apartment), about 75-100 feet from police car. Saw Wilson talking to Brown and Johnson. They move on and Wilson puts the car in front of them. The witness continues to the dumpster, but hears two shots. Thought it was just a warning shot, but then he heard another 6-7 shots. Ducked down and when he came back up the police were on the scene putting out crime scene tape.

* Witness 37 (8/18/2014): Driving east on Canfield, near leasing office, 125-150 feet from police car. When he came on the scene he saw Wilson dragging Brown into the car by his shirt. Hears a shot and Brown breaks lose. Brown gets 10-15 feet, with Wilson shooting 3-4 shots from his vehicle. Brown then turns around and puts his hands up. Wilson then casually walks up to Brown and shoots him 203 times at point blank range. He then stands over the body and fires 203 shots into it while it is on the ground. In total, 10 shots were fired.

* Witness 41 (August 26, 2014, also here): On second floor “between buildings” in a building with the west side facing Copperfield, so it could only be 2973/2975 Canfield Court (Brady’s building) or 2977/2999 Canfield Court. After hearing the first two shots (from inside car?), the witness went to the lot. Saw Wilson get out of the car and shoot Brown execution style, while on knees at close range. Wilson fired 9 shots so fast the gun had to be a full automatic,which the witness thought was illegal for police use. In later testimony the witness mentioned he/she saw Brown from the back only and saw a bruise on Office Wilson’s face.

September

Witness 34 (9/3/2014): Located in car heading west on Canfield, very near the police car (10-20 feet). Saw Brown and Wilson tussling for about 2 minutes. Wilson was holding Brown’s shirt and Brown threw “a couple blows”. Wilson was leaning towards center console. Shot rang out and Brown ran. He ran about 203 car lengths and then placed his hand on a brown car (as if winded?). Wilson came out and seemed “shaken”. Saw Brown turn around and come at Wilson and he was shot a couple of time. The witness then took off quickly: “I turn around right there in front of the officer truck and I burn out”.

November

Witness 57 (11/6/2014): Located in apartment in area (not sure where, but it does not matter). Heard shot and then saw tussling in car. Thought it was over until other shots were fired. Got cell phone and went outside to record the scene.

Witness 62 (11/6/2014): Treated officer Wilson at NW Healthcare. Wilson came in complaining of jaw pain. Stated he was punched twice and requested x-rays of jaw to see if it was broken. Had redness in the area of his jaw and some bruising. Also had scratch marks on the back of his neck.

Witness 64 (11/11/2014): At stop sign when he/she heard 2 shots. Saw brown running and appeared to be hit by shot in legs or hip. Turns to Wilson with hands up at chest level. Wilson about 8 feet away when shot.

Other Documents

I have examined a few other documents:

  • Medical record has diagnosis reason 95909 (other/unspecified injury to the face and neck) and 78492 (jaw pain). The primary diagnosis code 920 (contusion of face/scalp/neck except eye). TRIAGE: Chief complaint “he needs x-rays he was hit in the face a couple of times”. MEDICAL SCREENING: Skin pink, dry and warm (evidence of injury). Pain index of 6 out of 10. Prescriptions: Naprosyn 500mg tablets, 20, for pain
  • Wilson’s drug screen – negative on all counts (also see this)
  • Brown’s drug screen – Positive for cannabinoids (DELTA-9-THC and 11-NOR-DELTA-9-THC-COOH in blood) and 11-HYDROXY-THC and 11-NOR-DELTA-THC-COOH in urine. I personally find this uninteresting, as it was marijuana, but you can draw your own conclusions.

Discussion

Here are my thoughts on this, organized into different headings:

Witnesses

First there are some witnesses I find problems with. Some are major (I would think they are outright lying) and some are suspect, colored to belief system or similar (of some value, but should be taken with a grain of salt. I have stated why I think the witnesses should be discredited, for discussion.

Discredited witnesses – Either outright liar’s, as in the case of witness 46 and most likely witness 35, or someone who has pieced together so much it is difficult to accept any of the testimony at face value.

  • Witness 35 was too far to have heard anything Brown through a closed window (other witnesses closer to the scene did not hear anything said from either party. He also describes the shooting taking place next to the cruiser (within 5 feet) and the shooting at point blank range. Since the resting place of Brown was much farther and a point blank shooting would have left powder on more than just Brown’s hands, this does not fit the facts. He also would have missed the past 4 shots as he was heading downstairs and had no angle, so could not have seen the “don’t shoot” hands up when it happened. More than likely his grief over losing a friend caused him to make up most of his testimony. He is warned numerous times not to lie under oath.
    • Witness 44 describes being so blind he could not find his friends in his high school café without his glasses. Admits not wearing contact lenses or glasses on the day of the shooting. He is located about 150 feet from the police cruiser and farther from where Brown hit the ground. More than likely he saw a blur and filled in the story with what he heard talking to other people.
    • Witness 45 describes seeing two officers, so it is more likely he/she came up after the investigation started, which was shortly after the shooting took place (within minutes). The witness also describes blood on the shirt, after Wilson shot Brown from in the car.
    • There is no credibility whatsoever to witness 46’s statement. He has Brown and Johnson heading the wrong direction. He has the number of shots completely wrong. Has Wilson sticking his hand out the window to shoot Brown and then sets himself up as an attempted hero, where he is foiled by a cop telling him to “get the f*ck on”.

Suspect witnesses – may be something of value, but the story has evidence of coloring to fit a certain viewpoint.

  • Witness 16 admits she did not see anything beyond the altercation in the police cruiser, as she was retrieving her phone and purse and did not see the shooting. Her testimony on the screech is useful, as it corroborates the car backing up before the fight, as is the information about the volleys of shots (matches other witnesses). The description of anything beyond the altercation is recollection, filled in by other people’s stories.
  • Witness 37 describes an execution that could not have taken place based on physical evidence. About the only thing that can be taken at face value is there was a tussle in the car. He is also warned numerous times not to lie under oath.
  • Witness 40 is not suspect in his shading of whether Brown had “attitude”. For this reason, I cannot accept the “charging like a football player”, especially since there is only one other witness that has Brown running at Wilson (there are many saying he was advancing). The events, overall, match other witnesses, but the racial beginning of the journal entry suggests a bit of bias in how the scene was reported. It is also a personal journal and not witness testimony.
  • Witness 41 could not have seen much of the altercation from the second floor landing (examining angles on Google maps) and would not have seen Brown on the street on his knees once on the lot level, as there were cars in the way. The physical evidence refutes a close shooting and had the witness seen Brown from back only, would have never seen him on his knees, as he would have been. About the only thing I find interesting is the description of the bruises on Wilson’s face, but these would have had to have been witnessed later than at the time of the shooting, as it takes times for bruising to come up.
  • Witness 42 describes an execution style shooting at very close range, which is refuted by physical evidence. He also mentions Wilson finishing Brown off while he was lying face down in the street, which would not match any of the wounds. It is also clear, from the interview, that he knows about the media stories being told.
  • Witness 43 has embellished the story to include a taser, which Officer Wilson did not carry. This makes the rest of the testimony suspect. Since she did not witness the actual shooting, her testimony, beyond stating Wilson and Brown were struggling in the car, is rather useless.
  • Witness 48 was over 200 feet from the scene and in the middle row of a passenger van, making it hard to see everything that was going on. Also describes Brown charging Wilson, as did witness 40

    Story

Here is what happened, as best as I can piece together from witness testimony, with some weight towards those that I cannot find issues with on my first reading.

Michael Brown and Darian Johnson are walking down the middle of Canfield Court, heading east to Darian Johnson’s apartment on the third floor of 2909 Canfield Drive. We know Brown and Johnson were recently at the Ferguson Market and Liquor, located at 9101 W Florissant Avenue, where Brown shoplifted some cigars and pushed the owner when confronted. It is approximately 11:40 AM. on August 9, 2014.

Officer Darren Wilson is driving a Ferguson Police SUV west on Canfield, having left a domestic disturbance in the Northwinds apartments. He comes on the teens and tells them to get out of the middle of the road. Some witnesses (not included in the testimony) have stated he said “get the f*ck out of the road” or “get the f*ck on the sidewalk”. Wilson drives by the two and then one of them says something back to Wilson (this is assumed by some witnesses). Wilson backs up the truck quickly (creating a “screech” or “skirr” described by witness 16), pulling up next to Brown and Johnson and attempts to open the door. The door bumps Brown (Johnson’s testimony). Brown then starts fighting the officer. Wilson is leaning away from Brown, but holding onto his shirt. Brown hits Wilson at least 2 times (witness testimony as well as medical reports and testimony). Two shots then ring out (one that hits Brown on the thumb at minimum (autopsy), and one that hits a window or window frame across the street – 2964 Canfield Court).

Brown and Johnson then run. Johnson ducks behind a car driving west on Canfield after trying to get in to the car (witness 34) , while Brown continues down the middle of the street. Wilson gets out of the car shaken from being hit and heads after Brown, albeit a bit slower than Brown. He shoots at Brown at this time, more than likely causing the forearm wound.

Brown then turns and faces Wilson. Wilson tells him to stop but Brown advances. The speed of the advance is unclear. It is also unclear whether Brown advances and is shot and then advances and is shot again (two volleys) or Brown advances once and is shot. He has his hands up at chest or shoulder level, but it is unclear whether the hands are in a surrender position or balled up.

After the altercation, there is an onsite investigation, after which Wilson is taken to Northwest Healthcare with evidence of bruising on his face.

Additional bits

Here are some additional bits I have found.

Wilson’s Testimony

Here are some images that corroborate Wilson’s testimony.

NOTE: There is evidence of the fight in pictures (taken at NW Healthcare):

There is also evidence of a shot fired inside the car

here is another from the scene that shows the same piece sticking out

There is also forensic evidence of blood in the vehicle, as well as blood splatters

From one side of the body, he appears to have his hand in his waistband, as stated by Officer Wilson

From the other, his hand is out, so perhaps it was up, or up at chest level

Video of Scene

I find this one interesting, as it supports Wilson’s claim Brown rushed him (fast forward to 6:30 and start listening as one witness talks about Brown coming back at Wilson (after someone asks “why his body come this way though”)

Next thing I know he coming back towards unintelligible(?). The police had his gun drawn.

More repeated about 7:12 bout Brown coming at Wilson.

Conclusion?

It is hard to drive a firm conclusion, as the testimony is all over the place. Here are things that are clear.

  1. Brown and Johnson were walking down the center of Canfield
  2. Wilson stopped and confronted them
  3. They walked off and Wilson backed up
  4. There was an altercation in the vehicle and Brown was in the vehicle at some point, not just with a hand around his throat outside the police car
  5. Brown struck Wilson at least once on the left cheek
  6. A shot was fired in the car that struck Brown
  7. Brown got clear and ran off
  8. Wilson pursued
  9. Brown turned to face Wilson before being fatally shot

Some witnesses state Brown rushed Wilson, per Wilson’s testimony, including one overheard on a video shot just after the event. Others state he had his hands up. It is obvious some witnesses have changed their stories over time, while others outright lied about events they saw.

Brown’s hand wound appears to be at close range as the official autopsy states there is particulate matter in the hand, indicating a shot at close range to the hand. Brown’s private autopsy does not have this finding, but states there is no stipple.

It is also  highly likely at least one shot hit Brown as he ran away (forearm), although those stating he had his hands up offer another potential story.

Filed under Brown Shooting, Ferguson, General Rants

← Older posts