Rajnish Noonia

Tag: Distributed Systems

  • Building a TypeScript-First Frontend SDK: An Architectural Journey

    Over the past few years I have spent much of my time on one recurring problem: how do you let many teams build many data-intensive web applications without each of them reinventing the same foundations? This post walks through the architectural journey of designing a TypeScript-first frontend SDK to answer that question. It focuses on the patterns and trade-offs rather than any particular implementation, because the ideas travel well beyond the project that prompted them.

    Application layer Individual applications built on the platform Data grid apps Dashboards Workflow tools Admin tools Others… Framework layer Native React and Angular — thin bindings over the core Angular bindings Components · Directives · DI bridge React bindings Components · Hooks · DI bridge Core Framework-agnostic TypeScript — no UI framework dependency MVVMViewModel · State IoC / DIContainer · Discovery Data gridConfig · Streaming FormsRules · Validation ShellConfig-driven LayoutPanels · Persist ThemingTokens · Dark mode MessagingEvents · Notifications VisualisationCharts · Maps StreamingObservables ObservabilityLogging · Tracing · APM SecurityAuth · Permissions AI-ready contextContext files · Agent rules Scaffolding CLI + delivery pipeline CI/CD · Containers · AWS · dev / test / prod environments
    A layered SDK: framework-agnostic core, thin React/Angular bindings, and applications on top

    The Starting Point: Why a Component Library Is Not Enough

    The usual answer to “we need consistency across teams” is a component library: styled buttons, inputs and grids published as a package. In my experience that solves perhaps a fifth of the real problem. It says nothing about where state lives, how view logic is tested without rendering, how the same feature behaves identically in two UI frameworks, how dependent form fields validate each other, or how any of it stays coherent across years of changing teams.

    So the goal shifted from “shared components” to “shared architecture”: a TypeScript-first, framework-agnostic SDK in which React and Angular are equal, native rendering targets. The choice of view framework becomes a team preference, not an architectural fork.

    Principle 1: MVVM Keeps the View Thin

    The most important decision was to adopt Model–View–ViewModel as the backbone. The view is a template that binds to properties and commands. The ViewModel is a plain TypeScript class that owns state and behaviour. The model is data and services. Nothing else is allowed to blur those lines.

    Three benefits follow. Testability: a ViewModel has no JSX and no decorators, so business logic is tested as ordinary unit tests instead of by mounting components and poking the DOM. Reuse across frameworks: the same ViewModel drives a React view and an Angular view, so behaviour is written once. Clarity: engineers always know where logic belongs, which removes a whole category of code-review debate.

    Principle 2: Dependency Injection Owned by the Core

    MVVM only works well when ViewModels can ask for what they need without knowing where it comes from. Neither framework’s native mechanism fits: Angular’s injector only exists in Angular, and React context is not really DI. The answer was an IoC container that lives in the core, independent of any UI framework, with thin bridges into each one – a hook-based resolver for React and an injector bridge for Angular.

    Services are registered against interfaces, and can be discovered by capability rather than concrete type. This is where SOLID stops being a slide and becomes the path of least resistance: new behaviour arrives as a new registration, not an edit to existing classes, and every dependency can be swapped for a test double.

    Principle 3: Layers With Enforced Boundaries

    The SDK is organised as a monorepo with three layers:

    • Core: pure TypeScript – MVVM primitives, the container, state, messaging, streaming, configuration, theming tokens and platform services. No UI framework dependency.
    • Framework bindings: thin React and Angular packages that render the core’s concepts natively and bridge the container into each framework.
    • Applications: products built on a shared, configuration-driven shell, deployable to the browser or to desktop containers.

    Build-time boundary rules stop applications from reaching into the core directly and reject circular dependencies. Architecture that is only documented erodes; architecture that fails the build stays intact.

    Principle 4: Solve Cross-Cutting Concerns Once

    Most of the value of a platform is in the unglamorous capabilities that every application needs and no team wants to build. Rather than leave them to each project, the SDK provides them as shared services:

    • Configuration-driven shell and layout: navigation, panels and workspaces declared in configuration, with dockable panels whose arrangement is persisted per user.
    • State persistence: any screen can opt in to having column widths, filters, sort order and layout restored on reload.
    • Data grid abstraction: grids configured declaratively, with streaming updates delivered through observables, so teams write configuration and a ViewModel rather than grid plumbing.
    • Forms with a rule engine: field dependencies, visibility and validation expressed as rules; client and server validation surface through one pipeline.
    • Design tokens: semantic colours, spacing and typography exposed as typed constants and CSS variables, with dark mode as a first-class theme.
    • Observability and security: structured logging, tracing and application performance monitoring wired in by default, alongside authentication and permission handling.

    Principle 5: Start Every Project From Working, Not Blank

    A scaffolding CLI turns the SDK into a starting point rather than a set of instructions. A few commands produce an application that already has the shell, theming, authentication, observability, CI/CD pipelines, container configuration and cloud deployment templates for AWS across dev, test and production environments. The decisions that usually consume a project’s first sprint are already made, and every project starts in the same shape – which makes moving between them much easier for engineers.

    Principle 6: Design for AI-Assisted Development

    A newer lesson: an SDK now has two audiences, human developers and AI coding agents. Each package carries a concise context file describing its patterns, naming conventions and rules, and a routing structure points an agent at only the context relevant to its task. Generated projects inherit these files automatically.

    Treating that context as a maintained deliverable – updated alongside code and checked in CI – keeps agents inside the architecture instead of inventing their own. The result is AI assistance that accelerates teams without quietly eroding standards.

    Principle 7: Governance That Scales

    • Enforced module boundaries at build time
    • Semantic versioning with deprecation windows, so nothing disappears without warning
    • Living documentation: component catalogues and visual regression tests in the pipeline
    • Context files as a contract for both humans and agents

    What I Took Away

    The measurable outcome was that applications which once took months could be delivered in weeks, with a consistent look, feel and behaviour. But the more lasting lessons were architectural:

    • Separate behaviour from rendering early; it is very hard to retrofit.
    • Own your dependency injection in the core if you want framework independence.
    • Encode rules in the build, not in wiki pages.
    • Invest in the boring cross-cutting services – that is where teams lose the most time.
    • Treat AI context as part of the product, not an afterthought.

    A good platform is measured by how rarely the teams building on it need to think about it.

  • Cloud Computing

     

    Cloud

    You’re probably using cloud computing right now, even if you don’t realize it. If you use an online service to send emails, edit documents, watch films or TV, listen to music, play games, or store pictures and other files, it’s likely that cloud computing is making it all possible behind the scenes.

    Cloud computing stack

    Most cloud computing services fall into four broad categories: On Premises, infrastructure as a service (IaaS), platform as a service (PaaS) and software as a service (SaaS).

    Cloud Solutions Model

    IaaS  – Infrastructure as service

    This is where pre-configured hardware is provided via a virtualised interface or hypervisor. There is no high level infrastructure software provided such as an operating system, this must be provided by the buyer embedded with their own virtual applications.

    PaaS – Platform as service

    PaaS goes a stage further and includes the operating environment included the operating system and application services. PaaS suits organisations that are committed to a given development environment for a given application but like the idea of someone else maintaining the deployment platform for them.

    SaaS – Software as service

    Saas offers fully functional applications on-demand to provide specific services such as email management, CRM, web conferencing and an increasingly wide range of other applications & services.

    Type of Cloud deployment

    Based on the security and management required, the clouds can be built in following three ways to suit the needs of the businesses:

    Public cloud
    Public clouds are owned and operated by a third-party cloud service provider, which delivers computing resources such as servers and storage over the Internet. Microsoft Azure is an example of a public cloud. With a public cloud, all hardware, software and other supporting infrastructure are owned and managed by the cloud provider. You access these services and manage your account using a web browser.

    Private cloud
    A private cloud refers to cloud computing resources used exclusively by a single business or organisation. A private cloud can be physically located on the company’s on-site data centre. Some companies also pay third-party service providers to host their private cloud. A private cloud is one in which the services and infrastructure are maintained on a private network.

    Hybrid cloud
    Hybrid clouds combine public and private clouds, bound together by technology that allows data and applications to be shared between them. By allowing data and applications to move between private and public clouds, hybrid cloud gives businesses greater flexibility and more deployment options.

    Community Cloud

    Type of cloud hosting in which the setup is mutually shared between many organisations that belong to a particular community, i.e. banks and trading firms. It is a multi-tenant setup that is shared among several organisations that belong to a specific group which has similar computing apprehensions. The community members generally share similar privacy, performance and security concerns.

  • TPL Dataflow – Concurrent Programming

    TPL DataFlow

    TPL Dataflow is an in-process actor library on top of the Task Parallel Library enabling more robust concurrent programming.

    Parallel computing is a form of computation in which multiple operations are carried out simultaneously.Parallel computing is closely related to asynchronous programming, using many of the same core concepts and support. Asynchronous programming is an approach to writing code that involves invoking operations such that they don’t block the current thread of execution.Many personal computers and workstations have two or four or 8 cores (that is, CPUs) that enable multiple threads to be executed simultaneously. Computers in the near future are expected to have significantly more cores. To take advantage of the hardware of today and tomorrow, you can parallelize your code to distribute work across multiple processors. In the past, parallelization required low-level manipulation of threads and locks.

    The purpose of the TPL is to make developers more productive by simplifying the process of adding parallelism and concurrency to applications. The TPL scales the degree of concurrency dynamically to most efficiently use all the processors that are available. In addition, the TPL handles the partitioning of the work, the scheduling of threads on the ThreadPool, cancellation support, state management, and other low-level details. By using TPL, you can maximize the performance of your code while focusing on the work that your program is designed to accomplish.

    Data parallelism refers to scenarios in which the same operation is performed concurrently (that is, in parallel) on elements in a source collection or array. In data parallel operations, the source collection is partitioned so that multiple threads can operate on different segments concurrently.

    The Task Parallel Library (TPL) is based on the concept of a task, which represents an asynchronous operation. In some ways, a task resembles a thread or ThreadPool work item, but at a higher level of abstraction. The term task parallelism refers to one or more independent tasks running concurrently. Tasks provide two primary benefits:More efficient and more scalable use of system resources & More programmatic control than is possible with a thread or work item.

    The concurrency models we has discussed so far have the notion of shared state (data) in common.Shared state can be accessed by multiple threads at the same time and must be thus protected, either by locking or by using transactions. Both, mutability and sharing of state are not just inherent for these models, they are also inherent for the complexities.Unfortunately, programmers have found it very difficult to reliably build robust multi-threaded applications using the shared data and locks model, especially as applications grow in size and complexity.Making things worse, testing is not reliable with multi-threaded code. Since threads are non-deterministic, you might successfully test a program one thousand times, yet still the program could go wrong the first time it runs on a customer’s machine.

    We now have a look at an entirely different approach that bans the notion of shared state altogether. State is still mutable, however it is exclusively coupled to single entities that are allowed to alter it, so-called actors.The actor model in computer science is a mathematical model of concurrent computation that treats “actors” as the universal primitives of concurrent digital computation: in response to a message that it receives, an actor can make local decisions, create more actors, send more messages, and determine how to respond to the next message received.For communication, the actor model uses asynchronous message passing. In particular, it does not use any intermediate entities such as channels. Instead, each actor possesses a mailbox and can be addressed. These addresses are not to be confused with identities, and each actor can have no, one or multiple addresses. When an actor sends a message, it must know the address of the recipient. In addition, actors are allowed to send messages to themselves, which they will receive and handle later in a future step.

    The Task Parallel Library (TPL) provides dataflow components to help increase the robustness of concurrency-enabled applications. These dataflow components are collectively referred to as the TPL Dataflow Library. This dataflow model promotes actor-based programming by providing in-process message passing for coarse-grained dataflow and pipelining tasks.The TPL Dataflow Library provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applications that have high throughput and low latency. It also gives you explicit control over how data is buffered and moves around the system.

    If you want to scale your application beyond single machine or process then ServiceBus (NServiceBus, Microsoft Azure, etc) are the best candidates. These are designed around message oriented architecture and you can achieve highly reliable, available and scalable application. As an architect i always focus on reliability, after all, a highly available and scalable service that produces unreliable results isn’t very valuable. – We will need another post to cover the in-depth of service bus..

    TPL Dataflow (TDF) is a library for building concurrent applications. It promotes actor/agent-oriented designs through primitives for in-process message passing, dataflow, and pipelining. I have been playing with dataflow since its CTP was released and i found its very use in cases where you have to process data in form of a pipeline.With just few in-build blocks you can easily and quickly build concurrent app..

    The primitive blocks provided by dataflow are

    • Buffering Blocks – Holds data for use by data consumers.
      • BufferBock(T) – FIFO queue of message that can be written to multiple sources or read from by multiple targets.
      • BroadcastBlock(T) – Used when you must pass multiple messages to another component.
      • WriteOnceBlock(T) – similar to broadcastblock except object can be written to one time only
    • Execution Block – call a user provided delegate for each piece of received data
      • ActionBlock(t) – calls a delegate when it receives a data – excepts synchronous or asynchronous delegates
      • TransformBlock(Tinout,TOutput) – call function delegates to transform the incoming message to another type- excepts synchronous or asynchronous delegates
      • TransformManyBlock(TInput , TOutput) – similar to TransformBlock except it can produce zero or more output values for each input value, instead of only one output value for each input value. – excepts synchronous or asynchronous delegates

    Degree of Parallelism

    Every ActionBlock<TInput>, TransformBlock<TInput, TOutput>, and TransformManyBlock<TInput, TOutput> object buffers input messages until the block is ready to process them. By default, these classes process messages in the order in which they are received, one message at a time. You can also specify the degree of parallelism to enable ActionBlock<TInput>, TransformBlock<TInput, TOutput> and TransformManyBlock<TInput, TOutput> objects to process multiple messages concurrently.

    Now lets look at the implementation details of a web crawler

    Request Buffer
    |
    PageDownload
    / Save     ParseLink
    |
    RaiseLinkFound

    The messages to download a Url is received in the request buffer which is downloaded by a TranformBlock and converted into type safe page type message. The page message is then broadcasted using Broadcast block, which is further received by save ActionBlock and ParseLink Block. The parse link block parses the urls in the page and if they belong to same page it will raise an event for each url. The consumer of engine will receive the url and if its a new URL it will be posted back to engine.. The save action block will save the page to disk.

    This is very basic example but you can see the message based approach is much more simpler than a shared resource + threading approach.

    The TPL dataflow is good in case you don’t want to scale the solution beyond single machine as it offer a in process message base approach.With a proper service bus like NServiceBus you can scale out the solution beyond single machine and multiple servers could process the request to achieve the high throughput and off-course with easy to build,maintain clean code base.

    In production there are more things you have to take care like logging, error handling, transactions, unexpected failure recovery, dependency injection,loosely coupled components, extensibility, scalability and so on.. Frameworks like NServiceBus provides all these features alone with API to handle more complex business problems.

    Download Code : Here (Partially finished but working POC)

  • Actor Model

    An actor is an isolated, independent unit of compute and state with single-threaded execution. The actor pattern is a computational model for concurrent or distributed systems in which a large number of these actors can execute simultaneously and independently of each other. Actors can communicate with each other and they can create more actors.

    The concurrency models we have considered so far have the notion of shared state in common. Shared state can be accessed by multiple threads at the same time and must thus be protected, either by locking or by using transactions.

    We now have a look at an entirely different approach but without the notion of shared state. State is still mutable, however it is exclusively coupled to single entities that are allowed to alter it, so-called actors.

    An actor is a computational entity that, in response to a message it receives, can concurrently:

    • send a finite number of messages to other actors
    • create a finite number of new actors
    • designate the behaviour to be used for the next message it receives.

    There is no assumed sequence to the above actions and they could be carried out in parallel.

    Actor-Model

     

    For communication, the actor model uses asynchronous message passing.
    When implementing the actor model, it is important to adhere to the set of rules defined by the original idea. First and foremost, actors must not share any state. This disallows actors to pass references, pointers or any other kind of shared data as part of a message. Only immutable data and addresses (i.e. “names”) of actors should be sent. Message passing between actors is often enriched with a few more guarantees compared to the entirely best-effort style. Most implementations ensure that two messages sent from one actor to another maintain their order at arrival. Messaging is always asynchronous and the interleaving of incoming messages sent by multiple actors is indeterminate.

    Frameworks for the Actor Model

    There are many frameworks for the development of a distributed system based on the actor model. The most popular are Akka, Orleans, Service Fabric, TPL Dataflow..

    TPL Dataflow model promotes actor-based programming by providing in-process message passing for coarse-grained dataflow and pipelining tasks.These dataflow components are useful when you have multiple operations that must communicate with one another asynchronously or when you want to process data as it becomes available.The TPL Dataflow Library provides a foundation for message passing and parallelizing CPU-intensive and I/O-intensive applications that have high throughput and low latency. It also gives you explicit control over how data is buffered and moves around the system.

    Service Fabric Reliable Actors is an implementation of the actor design pattern.Although the actor design pattern can be a good fit to a number of distributed systems problems and scenarios, careful consideration of the constraints of the pattern and the framework implementing it must be made.

    As general guidance, consider the actor pattern to model your problem or scenario if:

    • Your problem space involves a large number (thousands or more) of small, independent, and isolated units of state and logic.
    • You want to work with single-threaded objects that do not require significant interaction from external components, including querying state across a set of actors.
    • Your actor instances won’t block callers with unpredictable delays by issuing I/O operations.