Pixytech

Lead Architect  •  Full Stack Engineer

Tag: .NET

  • Enterprise configuration management

    Almost every application requires some form of configuration information. This information can be as simple as a database connection string or as complex as multipart and hierarchical user preference information. How and where to store an application’s configuration data are questions you often face as a developer.

    Any large enterprise application has many moving blocks. They all need to be configured for a proper working of the application. As the application size increases or for scalability the same configuration has to be repeated in different applications. For most applications once the configuration has been changed the application needs to be restarted.

    Sample Code (create a blank console project, add json.net nuget)

    using Newtonsoft.Json;
    using Newtonsoft.Json.Linq;
    using System.Collections.Generic;
    using System.Linq;
    
    namespace CM
    {
        // configuration management API, 
        //1. allow clients specify typesafe models for configuations
        //2. Store flat data on server (table) which is easy to edit
        //3. can be extended to have inheritance of values (overrides)
        //4. can be extended to lock / unlock certain property by admins etc.
        
        class Program
        {
            //Sample configuration model
            internal class SampleConfigModal
            {
                public SampleConfigModal()
                {
                    Address = new Address();
                }
                public string Name { get; set; }
                public Address Address { get; set; }
                public int Age { get; set; }
    
    
            }
            public class Address
            {
                public string Street { get; set; }
    
            }
    
            // this is how client api will look like
            static void Main(string[] args)
            {
                // sample client code
                var data = new SampleConfigModal() { Name = "Rajnish", Age = 18, Address = new Address() { Street = "Oxley" } };
    
                // save Configuration
                SaveConfiguration("app", "section", data);
    
                //get Configuration
                var data2 = GetConfiguration("app", "section");
            }
    
            // Client side framework api -> call to rest end point
            private static T GetConfiguration(string appName, string SectionName) where T : new()
            {
                var defaultValue = new T();
                var samplePayload = JsonConvert.SerializeObject(defaultValue);
                var payload = GetConfiguration(appName, SectionName, samplePayload);
                return JsonConvert.DeserializeObject(payload);
            }
    
            // Client side framework api -> call to rest end point
            private static void SaveConfiguration(string appName, string SectionName, T data)
            {
                var payload = Newtonsoft.Json.JsonConvert.SerializeObject(data);
                SaveConfigurationa(appName, SectionName, payload);
            }
    
    
            //---------------------------------------- Server Code -------------------------- 
            //-------------- server has no knowledge of configuration structure or model
    
            private static Dictionary<string, string> storage;
    
            private static void SaveConfigurationa(string appName, string SectionName, string payload, string enumForHierarchyLevel = null)
            {
                // transformer
                var section = string.Format("{0}.{1}", appName, SectionName);
                var data = (JObject)JsonConvert.DeserializeObject(payload);
                var keyValueData = Flatten(data, section);
    
                // check if user has permission for level overrides
                // store with proper overides
    
                // store the flat list in sql or data 
                //| KEY |           |Value|            |OverrideType| - default,sysadmin,appadmin,groups,user etc
                //app.section.Name, Rajnish
                //app.section.Address.Street, Oxley
                //app.section.Age, 18
    
                storage = keyValueData;
            }
    
            private static string GetConfiguration(string appName, string SectionName, string samplePayload)
            {
                var section = string.Format("{0}.{1}", appName, SectionName);
                var data = (JObject)JsonConvert.DeserializeObject(samplePayload);
                var keyValueSample = Flatten(data, section);
                // update data from sql or data store
                // apply property override rules and get value from overrides if exists
                var keyValueData = keyValueSample.Select(x => new KeyValuePair<string, string>(x.Key, storage[x.Key]));
    
                //read these
                //app.section.Name, Rajnish
                //app.section.Address.Street, Oxley
                //app.section.Age, 18
    
                UnFlatten(data, section, keyValueData);
    
                var formatedData = JsonConvert.SerializeObject(data);
                /*
                 * {
                      "Name": "Rajnish",
                      "Address": {
                        "Street": "Oxley"
                      },
                      "Age": "18"
                    }
                 * */
                return formatedData;
    
            }
            
            // Server side json helper
    
            private static void UnFlatten(JObject jsonObject, string prefix, IEnumerable<KeyValuePair<string, string>> data)
            {
                foreach (var item in data)
                {
                    var keyName = item.Key.Substring(prefix.Length + 1);
                    var storageValue = item.Value;
                    if (keyName.Contains("."))
                    {
                        var keys = keyName.Split('.');
                        var jtoken = (JToken)jsonObject;
                        foreach (var k in keys)
                        {
                            jtoken = jtoken.SelectToken(k);
                        }
                        ((JValue)jtoken).Value = storageValue;
                    }
                    else
                    {
                        jsonObject[keyName] = storageValue;
                    }
                }
            }
    
            private static Dictionary<string, string> Flatten(JObject jsonObject, string prefix)
            {
    
                IEnumerable jTokens = jsonObject.Descendants().Where(p => p.Count() == 0);
                Dictionary<string, string> results = jTokens.Aggregate(new Dictionary<string, string>(), (properties, jToken) =>
                {
                    properties.Add(string.Format("{0}.{1}", prefix, jToken.Path), jToken.ToString());
                    return properties;
                });
                return results;
            }
        }
    }
    
    
  • BoundedContext – DDD

    Earlier in the article Software Architecture Patterns we briefly discussed domain driven design. In this article we will take a real example and dive into best practices and design of solution based on hypothetical problem or scenario.

    Bounded Context is a central pattern in Domain-Driven Design and It is the focus of DDD’s strategic design section which is all about dealing with large models and teams. The DDD deals with large models by dividing them into different Bounded Contexts and being explicit about their interrelationships.

    Strategic design  deals with situations that arise in complex systems, larger organizations, interactions with external system.Strategic design decisions are made by teams, or even between teams. Strategic design enables the goals of DDD to be realized on a larger scale, for a big system or in an application that fits in an enterprise-wide network.

    DDD is about designing software based on models of the underlying domain. A model acts as a Ubiquitous language to help communication between software developers and domain experts. It also acts as the conceptual foundation for the design of the software itself.

    It is hard to model a larger domain and build a single unified model. In real world, small domain models are build and together they represent the larger domain. Now lets image a real world example from electricity utility – smart meters ! –  here the word “meter” meant subtly different things to different domain experts coming from different parts of the organization. Lets try to understand the domain and try to break into sub domain models.

    Smart meters are the next generation of gas and electricity meters and offer a range of intelligent functions.The smart metering system is made up of: one electricity smart meter, one gas smart meter, a communications hub and an in-home display unit-the smart energy monitor on which you can view your energy.Smart meters measure actual, total gas and electricity usage and put consumers in control of their energy use, allowing them to adopt energy efficiency measures that can help save money on their energy bills.

     

    smart meter

     

    Now lets split the into domains and sub domains

    smart domain

     

    In the above diagram the subdomain build on foundation however one sub domain is interrelated to one or more other sub domain. They don’t exists in isolation in real world. The total unification of the domain model for a large system will not be feasible. So instead DDD divides up a large system into Bounded Contexts, each of which can have a unified model.

    A bounded context typically represents a slice of the overall system with clearly defined boundaries separating it from other bounded contexts within the system. If a bounded context is implemented by following the DDD approach, the bounded context will have its own domain model and its own ubiquitous language.

    A bounded context is the context for one particular domain model. Similarly, each bounded context (if implemented following the DDD approach) has its own ubiquitous language, or at least its own dialect of the domain’s ubiquitous language, entities, services etc. as shown below.

    Bounded Context

    Bounded Contexts have both unrelated concepts – such as a support ticket only existing in a customer support context, but also share concepts such as products and customers both exists in sales and support contexts.

    A large complex system can have multiple bounded contexts that interact with one another in various ways. A context map is the documentation that describes the relationships between these bounded contexts. It might be in the form of diagrams, tables, or text.

    In the next series we will dive into how we can apply CRQS (Command Query Responsibility Segregation Pattern) and vertically slice the layered architecture to deliver the highly scalable yet composite solution.

  • Software Architecture Patterns

    Extreme Programming (XP) is one of the more well known Agile methodologies. It is a programmer-centric methodology that emphasizes technical practices to promote skillful development through frequent delivery of working software.This methodology takes “best practices” to extreme levels and that’s why its named as Extreme Programming. Code reviews are a good example of Extreme programming. If code reviews are good, then doing constant code reviews would be extreme; but would it be better? This led to practices such as pair-programming and refactoring, which encourage the development of simple, effective designs, oriented in a way that optimizes business value.

    Extreme Programming defines 4 basic activities (coding, testing, listening & designing) and several practices like Pair Programming, Planning game, Test driven development, continuous integration, design improvement, coding standards, collective code, simple design etc.

    Projects suited to Extreme Programming are those that:

    • Involve new or prototype technology, where the requirements change rapidly, or some development is required to discover unforeseen implementation problems
    • Are research projects, where the resulting work is not the software product itself, but domain knowledge
    • Are small and more easily managed through informal methods

    Below are some software development process based on a concept of Extreme Programming and are about how to approach your design.

    Test driven design (TDD)

    tdd

    TDD is a software development process that relies on the repetition of a very short development cycle: requirements are turned into very specific test cases, then the software is improved to pass the new tests, only. It offers them a technique to explore the concepts behind the customers requirements, questioning that requirement and uncovering likely pitfalls. The developer can deliver these benefits without spending valuable time building and perfecting a graphical user interface. it stops developers from over engineer the product and encourage them to think from different prospective.

    TDD relies on the repetition of a very short development cycle :

    • Write an automated test case that defines a new feature – no code yet, so test will fail
    • Produce the minimum amount of code to pass that test
    • Refactor the new code to acceptable standards.

    Domain driven design (DDD)

    DDD is the process of being informed about the Domain before each cycle of touching code. Domain is a set of functionality that you are attempting to mimic that lies outside of your application.Domain Driven Design (DDD) is about mapping business domain concepts into software artifacts.Driven Design (DDD) focuses on the core model (the domain) and tries to keep other stuff like UI’s and databases separate.Domain Driven Design is all about understanding the customer real business need and emphases focuses more into the business need not focusing on the technology.

    It promotes important agile principles:-

    • Maintain the projects primary focus on the core domain of the delivery
    • Use models to refine a complex design and
    • Get the key team members together to collaborate deeply to derive their designs.

    Domain modeling and DDD play a important role in Enterprise Architecture (EA). Since one of the goals of EA is to align IT with the business units, the domain model which is the representation of business entities, becomes a core part of EA. This is why most of the EA components (business or infrastructural) should be designed and implemented around the domain model. Domain driven design is a key element of Service Oriented Architecture (SOA) because it helps in encapsulating the business logic and rules in domain objects. The domain model also provides the language and context with which the service contract can be defined.

    Domain driven design effort begins where domain modeling ends.

    There should be more focus on domain objects than services in the domain model.

    • Start with domain entities and domain logic.
    • Start without a service layer initially and only add services where the logic doesn’t belong in any domain entity or value object.
    • Use Ubiquitous Language, Design by Contract (DbC), Automated Tests, CI and Refactoring to make the implementation as closely aligned as possible with the domain model.

    From the design and implementation stand-point, a typical DDD framework should support the following features.

    • It should be a POCO based framework.
    • It should support the design and implementation of a business domain model using the DDD concepts.

    Bounded Context is a central pattern in Domain-Driven Design and It is the focus of DDD’s strategic design section which is all about dealing with large models and teams. – for more details on bounded context continue reading ddd here – series 2 of DDD.

    Behaviour driven design (BDD)

    BDD is a software development process based on Test-driven Development (TDD), that combines the general techniques and principles of TDD with ideas from Domain-driven Design (DDD) and Object-oriented Analysis and Design to provide software developers and business analysts with shared tools and a shared process to collaborate on software development, with the aim of delivering “software that matters”.While it is a refinement to TDD, it concentrates in understanding the user’s behaviour, and yields nicely to a good acceptance of the end system. In the though process means thinking from outside the system in. The benefit is that it offers a more precise and organized conversation between developers and domain experts

    BDD is also often heralded because BDD testing tools can be arguably more human readable to non-developers such as Domain Experts

    Event driven architecture (EDA)

    -TODO

    Command Query Responsibility Segregation Pattern (CQRS)

    -TODO

  • Actor Model

    An actor is an isolated, independent unit of compute and state with single-threaded execution. The actor pattern is a computational model for concurrent or distributed systems in which a large number of these actors can execute simultaneously and independently of each other. Actors can communicate with each other and they can create more actors.

    The concurrency models we have considered so far have the notion of shared state in common. Shared state can be accessed by multiple threads at the same time and must thus be protected, either by locking or by using transactions.

    We now have a look at an entirely different approach but without the notion of shared state. State is still mutable, however it is exclusively coupled to single entities that are allowed to alter it, so-called actors.

    An actor is a computational entity that, in response to a message it receives, can concurrently:

    • send a finite number of messages to other actors
    • create a finite number of new actors
    • designate the behaviour to be used for the next message it receives.

    There is no assumed sequence to the above actions and they could be carried out in parallel.

    Actor-Model

     

    For communication, the actor model uses asynchronous message passing.
    When implementing the actor model, it is important to adhere to the set of rules defined by the original idea. First and foremost, actors must not share any state. This disallows actors to pass references, pointers or any other kind of shared data as part of a message. Only immutable data and addresses (i.e. “names”) of actors should be sent. Message passing between actors is often enriched with a few more guarantees compared to the entirely best-effort style. Most implementations ensure that two messages sent from one actor to another maintain their order at arrival. Messaging is always asynchronous and the interleaving of incoming messages sent by multiple actors is indeterminate.

    Frameworks for the Actor Model

    There are many frameworks for the development of a distributed system based on the actor model. The most popular are Akka, Orleans, Service Fabric, TPL Dataflow..

    TPL Dataflow model promotes actor-based programming by providing in-process message passing for coarse-grained dataflow and pipelining tasks.These dataflow components are useful when you have multiple operations that must communicate with one another asynchronously or when you want to process data as it becomes available.The TPL Dataflow Library provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applications that have high throughput and low latency. It also gives you explicit control over how data is buffered and moves around the system.

    Service Fabric Reliable Actors is an implementation of the actor design pattern.Although the actor design pattern can be a good fit to a number of distributed systems problems and scenarios, careful consideration of the constraints of the pattern and the framework implementing it must be made.

    As general guidance, consider the actor pattern to model your problem or scenario if:

    • Your problem space involves a large number (thousands or more) of small, independent, and isolated units of state and logic.
    • You want to work with single-threaded objects that do not require significant interaction from external components, including querying state across a set of actors.
    • Your actor instances won’t block callers with unpredictable delays by issuing I/O operations.
  • Validate SSIS package on server

    SSIS server validation

    Untitled

     

    • Any change in database schema will break the associated SSIS package.
    • Generally each change is responsible to identify dependent components to be included in the change scope.
    • But what about any missed dependencies. E.g. Recent production issue.
    • Such broken SSIS can be identified in Pre-Prod only if pre-prod is well controlled and identical copy of prod environment.
    • Process is required to identify packages impacted by dependencies changes like database schema.
    • Proactive validation of SSIS package could help to identify the issue and reduce the system downtime.

    High level architecture

    Untitled2

     

    The solution is capable of validating all ssis packages for sql server version 2008 on-wards. Instead of publishing the full source code below is the pseudo code and sql scripts to validate the packages.

    1. Create C# console base project which takes the server name you want to validate. The app calls the web service to validate the SQL server (validate SSIS packages on that server).
    2. Create web api project and expose following methods (Job Controller)
      1. Validate – start the validation process by creating SQL agent job dynamically on target server
      2. ValidationFinished – this will be invoked by job when validation is finished.
      3. GetValidationStatus – this will provide the SQL agent job status currently validating server

    The Validate method works as follows..

    • Get the SQL Server version
    private Version GetServerVersion(string serverName)
     {
     var connectionString = string.Format("Server={0};Database=master;Trusted_Connection=True;", serverName);
     using (var connection = new SqlConnection(connectionString))
     {
     connection.Open();
     return new Version(connection.ServerVersion);
     }
     }
    
    • Create interface IProcessor and implement it for sql server 2008 and 2012 as they both have different way to validate package
    • Based on server version get the eligible concrete implementation.
    • The validation engine will create the validation job on target sql server. for each proxy account, the script will create job step to trigger a network share package running under proxy account. (for details see createValidationJob.sql)
    • Each validation step gets the package scheduled under the current account and validate the package, populate the results back on central server. and in the end extract the job history and delete the job.)
    • The network deployed ssis package uses same engine to validate 2008 and 2012 via IProcessor since validation is done differently for both servers.

    The common scripts used for both servers are here

    Common-Scripts

    The SQL server 2008 validation is done via DTEXEC with /validate command

    SQl 2008-Scripts

    The SQL 2012 validation is done via EXECUTE SSISDB.catalog.validate_project stored proc.

    SQL 2012 Scripts

    The centralized server which keeps the status of SSIS validation looks like

    Untitled3

    The font end to display the data was build on angular 2 API..