A long series of learning notes on infrastructure

Materials taken from the presentation from here. All tributes go to InfoQ and the Presenter.

November 6, 2018

Previously

Application

Initial login:

• Download full workspace model with all channels, users, emoji, etc.
• Establish real time websocket

While connected:

• Push updates via websocket
• API calls for channel history, message edits, create channels, etc.

Database

Workspace Sharding

• Assign a workspace to a DB and MS shard at creation
• Metadata table lookup for each API request to route

“Herd of Pets”

• DBs run in active/active pairs with application failover
• Service hosts are addressed in config and manually replaced

Challenges with Scalling up

• …

A long series of learning notes on infrastructure

Materials taken from the presentation from here. All tributes go to InfoQ and the Presenter.

March 7th, 2017

• Scale out
• Scale up
• Scale the dev team

Scale out

Two types of resources:

• Storage: needs to be consistent across data centers
• Computing: driven by user traffic, as needed basis

Database:

• PostgreSQL: relational database: user, media, friendship etc. One master for write and slave replicas for read.
• Cassandra: user feeds, activities etc. Masterless replicas that have different read/write eventual consistency configuration.

Memcache:

Due to attributes below, memcache is not suitable for cross region update.

• High performance…

Creational Pattern — Adapter

Simply put, the Adapter pattern allows two incompatible piece to work together. This is done by creating an adapter to convert one of the pieces’ interface to be acceptable by another.

• Client : calls the Adapter to talk to Adaptee
• Adapter : twist the Adaptee’s interface to make it acceptable by the Client
• Adaptee : does nothing, simply has its interface to be twisted by Adapter.

This allows for loose coupling as the two otherwise incompatible pieces can separate from each other, and it’s down to the adapter to create the communication channel. …

Creational Pattern — Singleton

Simply put, a Singleton is a pattern where there is only one instance of the class instantiated. This is normally implemented with saving the instance to a static property and having a guard clause to check if the property exist.

Singleton class is normally made accessible globally (though it’s not always the case). This makes the lifecycle of a Singleton class instance persists till the end of the application. This means the instance won’t be garbage collected until the end of the application.

Note why we prefer a Singleton vs. a global variable:

• Global variables are prone to modification/changes. Local…

Creational Pattern — Builder

Simply put, Builder is a creational design pattern that decouples the construction of an object from its class representation, so you can use the same construction process to create objects of similar kinds with different steps.

At the first glance, it may seem that this is very similar to the Factory pattern from last blog. It does indeed. The difference is that the Builder pattern applies to objects that are more complex, and the nuances are really in the different construction steps for different sub products. An example will be a HttpRequestPreparation Class that will go through: getUrl-->getHttpMethod-->getParam-->setHeader-->setAuthentication before return…

Creational Pattern — Factory

Simply put, the factory pattern wraps creation (normally through class constructor) of different types of objects with a simple API instead of multiple if..else statement.

This comes in handy when you want to group several Classes under one API interface for abstraction. It demonstrates in Dynamic programming where the type of objects is decided at runtime.

But be careful with overuse it, as a lot of use cases can be solved simply using Dependency Injection.

See below with an example is in Python. Note the use of ABCMeta Classe to define an agreed interface among child classes.

from abc import…

Not long ago, I was tasked to work on a project about configuring the SSO with AWS Cognito. The project itself went fine, but before I dived into the project, I spent quite some time to figure out the concept of authentication & authorisation, the typical workflows, as well as the industry standards.

In short:

• Authentication: Who the user is
• Authorisation: What the user can access (authentication is normally the first step of authorisation)

Single sign-on

Single sign-on (SSO) is an authentication scheme that allows a user to log in with a single ID and password to any of the group of…

This is a series of learning notes on Designing Data-Intensive Applications.

Transmitting event streams

Stream is usually a sequence of events: a small, self-contained, immutable objects with a timestamp. Related events are usually grouped together into a topic.

An event is generated once by a producer (publisher or sender), and then processed by consumers (subcribers or recipients).

Messaging systems

Within the publish/subscribe model, we can differentiate the systems by asking two questions:

1. What happens if the producers send messages faster than the consumers can process them? …