Enterprise configuration management

Almost every application requires some form of configuration information. This information can be as simple as a database connection string or as complex as multipart and hierarchical user preference information. How and where to store an application’s configuration data are questions you often face as a developer.

Any large enterprise application has many moving blocks. They all need to be configured for a proper working of the application. As the application size increases or for scalability the same configuration has to be repeated in different applications. For most applications once the configuration has been changed the application needs to be restarted.

Sample Code (create a blank console project, add json.net nuget)

using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using System.Collections.Generic;
using System.Linq;

namespace CM
{
    // configuration management API, 
    //1. allow clients specify typesafe models for configuations
    //2. Store flat data on server (table) which is easy to edit
    //3. can be extended to have inheritance of values (overrides)
    //4. can be extended to lock / unlock certain property by admins etc.
    
    class Program
    {
        //Sample configuration model
        internal class SampleConfigModal
        {
            public SampleConfigModal()
            {
                Address = new Address();
            }
            public string Name { get; set; }
            public Address Address { get; set; }
            public int Age { get; set; }


        }
        public class Address
        {
            public string Street { get; set; }

        }

        // this is how client api will look like
        static void Main(string[] args)
        {
            // sample client code
            var data = new SampleConfigModal() { Name = "Rajnish", Age = 18, Address = new Address() { Street = "Oxley" } };

            // save Configuration
            SaveConfiguration("app", "section", data);

            //get Configuration
            var data2 = GetConfiguration("app", "section");
        }

        // Client side framework api -> call to rest end point
        private static T GetConfiguration(string appName, string SectionName) where T : new()
        {
            var defaultValue = new T();
            var samplePayload = JsonConvert.SerializeObject(defaultValue);
            var payload = GetConfiguration(appName, SectionName, samplePayload);
            return JsonConvert.DeserializeObject(payload);
        }

        // Client side framework api -> call to rest end point
        private static void SaveConfiguration(string appName, string SectionName, T data)
        {
            var payload = Newtonsoft.Json.JsonConvert.SerializeObject(data);
            SaveConfigurationa(appName, SectionName, payload);
        }


        //---------------------------------------- Server Code -------------------------- 
        //-------------- server has no knowledge of configuration structure or model

        private static Dictionary<string, string> storage;

        private static void SaveConfigurationa(string appName, string SectionName, string payload, string enumForHierarchyLevel = null)
        {
            // transformer
            var section = string.Format("{0}.{1}", appName, SectionName);
            var data = (JObject)JsonConvert.DeserializeObject(payload);
            var keyValueData = Flatten(data, section);

            // check if user has permission for level overrides
            // store with proper overides

            // store the flat list in sql or data 
            //| KEY |           |Value|            |OverrideType| - default,sysadmin,appadmin,groups,user etc
            //app.section.Name, Rajnish
            //app.section.Address.Street, Oxley
            //app.section.Age, 18

            storage = keyValueData;
        }

        private static string GetConfiguration(string appName, string SectionName, string samplePayload)
        {
            var section = string.Format("{0}.{1}", appName, SectionName);
            var data = (JObject)JsonConvert.DeserializeObject(samplePayload);
            var keyValueSample = Flatten(data, section);
            // update data from sql or data store
            // apply property override rules and get value from overrides if exists
            var keyValueData = keyValueSample.Select(x => new KeyValuePair<string, string>(x.Key, storage[x.Key]));

            //read these
            //app.section.Name, Rajnish
            //app.section.Address.Street, Oxley
            //app.section.Age, 18

            UnFlatten(data, section, keyValueData);

            var formatedData = JsonConvert.SerializeObject(data);
            /*
             * {
                  "Name": "Rajnish",
                  "Address": {
                    "Street": "Oxley"
                  },
                  "Age": "18"
                }
             * */
            return formatedData;

        }
        
        // Server side json helper

        private static void UnFlatten(JObject jsonObject, string prefix, IEnumerable<KeyValuePair<string, string>> data)
        {
            foreach (var item in data)
            {
                var keyName = item.Key.Substring(prefix.Length + 1);
                var storageValue = item.Value;
                if (keyName.Contains("."))
                {
                    var keys = keyName.Split('.');
                    var jtoken = (JToken)jsonObject;
                    foreach (var k in keys)
                    {
                        jtoken = jtoken.SelectToken(k);
                    }
                    ((JValue)jtoken).Value = storageValue;
                }
                else
                {
                    jsonObject[keyName] = storageValue;
                }
            }
        }

        private static Dictionary<string, string> Flatten(JObject jsonObject, string prefix)
        {

            IEnumerable jTokens = jsonObject.Descendants().Where(p => p.Count() == 0);
            Dictionary<string, string> results = jTokens.Aggregate(new Dictionary<string, string>(), (properties, jToken) =>
            {
                properties.Add(string.Format("{0}.{1}", prefix, jToken.Path), jToken.ToString());
                return properties;
            });
            return results;
        }
    }
}

BoundedContext – DDD

Earlier in the article Software Architecture Patterns we briefly discussed domain driven design. In this article we will take a real example and dive into best practices and design of solution based on hypothetical problem or scenario.

Bounded Context is a central pattern in Domain-Driven Design and It is the focus of DDD’s strategic design section which is all about dealing with large models and teams. The DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships.

Strategic design  deals with situations that arise in complex systems, larger organizations, interactions with external system.Strategic design decisions are made by teams, or even between teams. Strategic design enables the goals of DDD to be realized on a larger scale, for a big system or in an application that fits in an enterprise-wide network.

DDD is about designing software based on models of the underlying domain. A model acts as a Ubiquitous language to help communication between software developers and domain experts. It also acts as the conceptual foundation for the design of the software itself.

It is hard to model a larger domain and build a single unified model. In real world, small domain models are build and together they represent the larger domain. Now lets image a real world example from electricity utility – smart meters ! –  here the word “meter” meant subtly different things to different domain experts coming from different parts of the organization. Lets try to understand the domain and try to break into sub domain models.

Smart meters are the next generation of gas and electricity meters and offer a range of intelligent functions.The smart metering system is made up of: one electricity smart meter, one gas smart meter, a communications hub and an in-home display unit-the smart energy monitor on which you can view your energy.Smart meters measure actual, total gas and electricity usage and put consumers in control of their energy use, allowing them to adopt energy efficiency measures that can help save money on their energy bills.

 

smart meter

 

Now lets split the into domains and sub domains

smart domain

 

In the above diagram the subdomain build on foundation however one sub domain is interrelated to one or more other sub domain. They don’t exists in isolation in real world. The total unification of the domain model for a large system will not be feasible. So instead DDD divides up a large system into Bounded Contexts, each of which can have a unified model.

A bounded context typically represents a slice of the overall system with clearly defined boundaries separating it from other bounded contexts within the system. If a bounded context is implemented by following the DDD approach, the bounded context will have its own domain model and its own ubiquitous language.

A bounded context is the context for one particular domain model. Similarly, each bounded context (if implemented following the DDD approach) has its own ubiquitous language, or at least its own dialect of the domain’s ubiquitous language, entities, services etc. as shown below.

Bounded Context

Bounded Contexts have both unrelated concepts – such as a support ticket only existing in a customer support context, but also share concepts such as products and customers both exists in sales and support contexts.

A large complex system can have multiple bounded contexts that interact with one another in various ways. A context map is the documentation that describes the relationships between these bounded contexts. It might be in the form of diagrams, tables, or text.

In the next series we will dive into how we can apply CRQS (Command Query Responsibility Segregation Pattern) and vertically slice the layered architecture to deliver the highly scalable yet composite solution.

Software Architecture Patterns

Extreme Programming (XP) is one of the more well known Agile methodologies. It is a programmer-centric methodology that emphasizes technical practices to promote skillful development through frequent delivery of working software.This methodology takes “best practices” to extreme levels and that’s why its named as Extreme Programming. Code reviews are a good example of Extreme programming. If code reviews are good, then doing constant code reviews would be extreme; but would it be better? This led to practices such as pair-programming and refactoring, which encourage the development of simple, effective designs, oriented in a way that optimizes business value.

Extreme Programming defines 4 basic activities (coding, testing, listening & designing) and several practices like Pair Programming, Planning game, Test driven development, continuous integration, design improvement, coding standards, collective code, simple design etc.

Projects suited to Extreme Programming are those that:

  • Involve new or prototype technology, where the requirements change rapidly, or some development is required to discover unforeseen implementation problems
  • Are research projects, where the resulting work is not the software product itself, but domain knowledge
  • Are small and more easily managed through informal methods

Below are some software development process based on a concept of Extreme Programming and are about how to approach your design.

Test driven design (TDD)

tdd

TDD is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved to pass the new tests, only. It offers them a technique to explore the concepts behind the customers requirements, questioning that requirement and uncovering likely pitfalls. The developer can deliver these benefits without spending valuable time building and perfecting a graphical user interface. it stops developers from over engineer the product and encourage them to think from different prospective.

TDD relies on the repetition of a very short development cycle :

  • Write an automated test case that defines a new feature – no code yet, so test will fail
  • Produce the minimum amount of code to pass that test
  • Refactor the new code to acceptable standards.

Domain driven design (DDD)

DDD is the process of being informed about the Domain before each cycle of touching code. Domain is a set of functionality that you are attempting to mimic that lies outside of your application.Domain Driven Design (DDD) is about mapping business domain concepts into software artifacts.Driven Design (DDD) focuses on the core model (the domain) and tries to keep other stuff like UI’s and databases separate.Domain Driven Design is all about understanding the customer real business need and emphases focuses more into the business need not focusing on the technology.

It promotes important agile principles:-

  • Maintain the projects primary focus on the core domain of the delivery
  • Use models to refine a complex design and
  • Get the key team members together to collaborate deeply to derive their designs.

Domain modeling and DDD play a important role in Enterprise Architecture (EA). Since one of the goals of EA is to align IT with the business units, the domain model which is the representation of business entities, becomes a core part of EA. This is why most of the EA components (business or infrastructural) should be designed and implemented around the domain model. Domain driven design is a key element of Service Oriented Architecture (SOA) because it helps in encapsulating the business logic and rules in domain objects. The domain model also provides the language and context with which the service contract can be defined.

Domain driven design effort begins where domain modeling ends.

There should be more focus on domain objects than services in the domain model.

  • Start with domain entities and domain logic.
  • Start without a service layer initially and only add services where the logic doesn’t belong in any domain entity or value object.
  • Use Ubiquitous Language, Design by Contract (DbC), Automated Tests, CI and Refactoring to make the implementation as closely aligned as possible with the domain model.

From the design and implementation stand-point, a typical DDD framework should support the following features.

  • It should be a POCO based framework.
  • It should support the design and implementation of a business domain model using the DDD concepts.

Bounded Context is a central pattern in Domain-Driven Design and It is the focus of DDD’s strategic design section which is all about dealing with large models and teams. – for more details on bounded context continue reading ddd here – series 2 of DDD.

Behaviour driven design (BDD)

BDD is a software development process based on Test-driven Development (TDD), that combines the general techniques and principles of TDD with ideas from Domain-driven Design (DDD) and Object-oriented Analysis and Design to provide software developers and business analysts with shared tools and a shared process to collaborate on software development, with the aim of delivering “software that matters”.While it is a refinement to TDD, it concentrates in understanding the user’s behaviour, and yields nicely to a good acceptance of the end system. In the though process means thinking from outside the system in. The benefit is that it offers a more precise and organized conversation between developers and domain experts

BDD is also often heralded because BDD testing tools can be arguably more human readable to non-developers such as Domain Experts

Event driven architecture (EDA)

-TODO

Command Query Responsibility Segregation Pattern (CQRS)

-TODO

Cloud Computing

 

Cloud

You’re probably using cloud computing right now, even if you don’t realize it. If you use an online service to send emails, edit documents, watch films or TV, listen to music, play games, or store pictures and other files, it’s likely that cloud computing is making it all possible behind the scenes.

Cloud computing stack

Most cloud computing services fall into four broad categories: On Premises, infrastructure as a service (IaaS), platform as a service (PaaS) and software as a service (SaaS).

Cloud Solutions Model

IaaS  – Infrastructure as service

This is where pre-configured hardware is provided via a virtualised interface or hypervisor. There is no high level infrastructure software provided such as an operating system, this must be provided by the buyer embedded with their own virtual applications.

PaaS – Platform as service

PaaS goes a stage further and includes the operating environment included the operating system and application services. PaaS suits organisations that are committed to a given development environment for a given application but like the idea of someone else maintaining the deployment platform for them.

SaaS – Software as service

Saas offers fully functional applications on-demand to provide specific services such as email management, CRM, web conferencing and an increasingly wide range of other applications & services.

Type of Cloud deployment

Based on the security and management required, the clouds can be built in following three ways to suit the needs of the businesses:

Public cloud
Public clouds are owned and operated by a third-party cloud service provider, which delivers computing resources such as servers and storage over the Internet. Microsoft Azure is an example of a public cloud. With a public cloud, all hardware, software and other supporting infrastructure are owned and managed by the cloud provider. You access these services and manage your account using a web browser.

Private cloud
A private cloud refers to cloud computing resources used exclusively by a single business or organisation. A private cloud can be physically located on the company’s on-site data centre. Some companies also pay third-party service providers to host their private cloud. A private cloud is one in which the services and infrastructure are maintained on a private network.

Hybrid cloud
Hybrid clouds combine public and private clouds, bound together by technology that allows data and applications to be shared between them. By allowing data and applications to move between private and public clouds, hybrid cloud gives businesses greater flexibility and more deployment options.

Community Cloud

Type of cloud hosting in which the setup is mutually shared between many organisations that belong to a particular community, i.e. banks and trading firms. It is a multi-tenant setup that is shared among several organisations that belong to a specific group which has similar computing apprehensions. The community members generally share similar privacy, performance and security concerns.

Actor Model

An actor is an isolated, independent unit of compute and state with single-threaded execution. The actor pattern is a computational model for concurrent or distributed systems in which a large number of these actors can execute simultaneously and independently of each other. Actors can communicate with each other and they can create more actors.

The concurrency models we have considered so far have the notion of shared state in common. Shared state can be accessed by multiple threads at the same time and must thus be protected, either by locking or by using transactions.

We now have a look at an entirely different approach but without the notion of shared state. State is still mutable, however it is exclusively coupled to single entities that are allowed to alter it, so-called actors.

An actor is a computational entity that, in response to a message it receives, can concurrently:

  • send a finite number of messages to other actors
  • create a finite number of new actors
  • designate the behavior to be used for the next message it receives.

There is no assumed sequence to the above actions and they could be carried out in parallel.

Actor-Model

 

For communication, the actor model uses asynchronous message passing.
When implementing the actor model, it is important to adhere to the set of rules defined by the original idea. First and foremost, actors must not share any state. This disallows actors to pass references, pointers or any other kind of shared data as part of a message. Only immutable data and addresses (i.e. “names”) of actors should be sent. Message passing between actors is often enriched with a few more guarantees compared to the entirely best-effort style. Most implementations ensure that two messages sent from one actor to another maintain their order at arrival. Messaging is always asynchronous and the interleaving of incoming messages sent by multiple actors is indeterminate.

Frameworks for the Actor Model

There are many frameworks for the development of a distributed system based on the actor model. The most popular are Akka, Orleans, Service Fabric, TPL Dataflow..

TPL Dataflow model promotes actor-based programming by providing in-process message passing for coarse-grained dataflow and pipelining tasks.These dataflow components are useful when you have multiple operations that must communicate with one another asynchronously or when you want to process data as it becomes available.The TPL Dataflow Library provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applications that have high throughput and low latency. It also gives you explicit control over how data is buffered and moves around the system.

Service Fabric Reliable Actors is an implementation of the actor design pattern.Although the actor design pattern can be a good fit to a number of distributed systems problems and scenarios, careful consideration of the constraints of the pattern and the framework implementing it must be made.

As general guidance, consider the actor pattern to model your problem or scenario if:

  • Your problem space involves a large number (thousands or more) of small, independent, and isolated units of state and logic.
  • You want to work with single-threaded objects that do not require significant interaction from external components, including querying state across a set of actors.
  • Your actor instances won’t block callers with unpredictable delays by issuing I/O operations.

Validate SSIS package on server

SSIS server validation

Untitled

 

  • Any change in database schema will break the associated SSIS package.
  • Generally each change is responsible to identify dependent components to be included in the change scope.
  • But what about any missed dependencies. E.g. Recent production issue.
  • Such broken SSIS can be identified in Pre-Prod only if pre-prod is well controlled and identical copy of prod environment.
  • Process is required to identify packages impacted by dependencies changes like database schema.
  • Proactive validation of SSIS package could help to identify the issue and reduce the system downtime.

High level architecture

Untitled2

 

The solution is capable of validating all ssis packages for sql server version 2008 on-wards. Instead of publishing the full source code below is the pseudo code and sql scripts to validate the packages.

  1. Create C# console base project which takes the server name you want to validate. The app calls the web service to validate the SQL server (validate SSIS packages on that server).
  2. Create web api project and expose following methods (Job Controller)
    1. Validate – start the validation process by creating SQL agent job dynamically on target server
    2. ValidationFinished – this will be invoked by job when validation is finished.
    3. GetValidationStatus – this will provide the SQL agent job status currently validating server

The Validate method works as follows..

  • Get the SQL Server version
private Version GetServerVersion(string serverName)
 {
 var connectionString = string.Format("Server={0};Database=master;Trusted_Connection=True;", serverName);
 using (var connection = new SqlConnection(connectionString))
 {
 connection.Open();
 return new Version(connection.ServerVersion);
 }
 }
  • Create interface IProcessor and implement it for sql server 2008 and 2012 as they both have different way to validate package
  • Based on server version get the eligible concrete implementation.
  • The validation engine will create the validation job on target sql server. for each proxy account, the script will create job step to trigger a network share package running under proxy account. (for details see createValidationJob.sql)
  • Each validation step gets the package scheduled under the current account and validate the package, populate the results back on central server. and in the end extract the job history and delete the job.)
  • The network deployed ssis package uses same engine to validate 2008 and 2012 via IProcessor since validation is done differently for both servers.

The common scripts used for both servers are here (download and rename doc to zip)

Common-Scripts

The SQL server 2008 validation is done via DTEXEC with /validate command (download and rename doc to zip)

SQl 2008-Scripts

The SQL 2012 validation is done via EXECUTE SSISDB.catalog.validate_project stored proc. (download and rename doc to zip)

SQL 2012 Scripts

The centralized server which keeps the status of SSIS validation looks like

Untitled3

The font end to display the data was build on angular 2 API..

ViewModel First – ViewCaching

With View model first approach, view are usually created via DataTemplate, and each time a view model is injected in the content control, the corresponding view is recreated.

If you have complex view where it take bit time to create view, you may see the performance hits. To avoid the performance hit you may want to cache the view and use the cached view when available. This can be done via creating CacheContentControl and you choice of view factory.

The basic idea is to wrap the view inside CacheContentControl and delegate the view creation logic to your view factory. The attached sample uses weakreference view factory,say if GC has not collected the view, you may use the cached view instead of creating view each time. This will work with controls like tab control docking controls etc.

The CacheContentControl  code

public class CacheContentControl : ContentControl
    {
        public CacheContentControl()
        {
            Unloaded += ViewCache_Unloaded;
            ViewFactory = WeakReferenceViewFactory.Instance;
        }

        void ViewCache_Unloaded(object sender, RoutedEventArgs e)
        {
            Content = null;
        }

        private Type _contentType;
        public Type ContentType
        {
            get { return _contentType; }
            set
            {
                _contentType = value;
               //  use you favorite factory
                Content = ViewFactory.GetView(value);
            }
        }

        public IViewFactory ViewFactory { get; private set; }
    }

In the DataTemplate, use the ViewCache, pass the type of the real view you want to use:

<DataTemplate DataType=”{x:Type moduleB:Panel3Vm}”>

<Border Background=”Green”>

<local:CacheContentControl ContentType=”{x:Type moduleB:Pane3 }” Margin=”5″/>

</Border>

</DataTemplate>

The example uses standard data template (red option) to demonstrate each time you navigate to panes the corresponding views are recreated with new hash code.

If you select CacheContentControl (green option) and navigate the view is only created once. If you click on button to collected the GC then view are created on navigation if they are claimed by GC.

Why ViewModel first ?

I prefer to use view model first approach. For many reasons:

  • Vms are your application containing most of logic apart from glue code in form of behaviors or triggers.
  • If you creates views then you are responsible for its life and cleanup code. You have to deal with threading and other issues which are difficult to test. On the other hand of you create vms and leave the view creation logic with WPF via data template.. you don’t have to worry about threading issue. And there will be better separation of concerns.
  • With vm first approach zero code behind.
  • With a project level isolation for view and vms you can restrict developers using view specific things like dispatcher in the view model leaving more cleaner and testable code base. I.e view project sprojec to vm. And vm project should not refer to any presentation lib.
  • If there is clear boundary between view and vm. Both can evolve and will be less fragile.
  • Allows more complete testing of logic to open new Views and ViewModels
  • Tends to be DRYer (don’t repeat yourself ) as applications get larger
  • View and ViewModel are more independent and can be worked on separately more easily.

For more details see attached source code (download and rename docx to zip)

ViewCache Source Code