Global Software Architecture Summit 2019 – episode V

Sketch by Simon Brown on Software Architecture for Developers

This is the fifth, and last, episode in a series of posts reviewing the first Global Software Architecture Summit that took place in Barcelona the 10th of October. To see the whole list of episodes, please visit this.

Architectural Thinking

Mark Richards
Mark Richards

This talk, given by Mark Richards (@markrichardssa), was the best first class in Software Architecture that I have ever seen, and the best definition of what Software Architecture is that anyone might find out there. He also emphasized the natural existence of tradeoffs in every architecture alternative there is, and the key role of software architects in making explicit the prons and cons of them, in order to pick the one that fits in the scenario in hand.

So, should have you the opportunity to attend one of Mark’s courses, or attend to any of his talks, just go!

He also gave us the advice to spend the first 20 minutes of our days learning something, and in particular something we were not even aware that existed: the unknown unknowns. How could you learn about anything that you didn’t know that you did not know? Staying tune for what is said out there, in related media: podcats, blogs, aggregators, meetups, etc.

Effective Architecture Practices Debate

This third debate was moderated by Álvaro García (@alvarobiz), and the invitees on stage were Sandro Mancuso (@sandromancuso), George Fairbanks (@GHFairbanks), Juan Manuel Serrano (@juanshac), Carlos Blé (@carlosble), and Eoin Woods (@eoinwoodz).

Their interventions were eventually centered on how implement changes in architecture style in practice, with particularly wise advices from Sando Mancuso. For instance, it is always key to have a roadmap as an expression of a direction from now to where the organization envision to go, even though being concious that the straight line drawn in the roadmap is going to become a curvy road in real life.

Incidentally, someone mentioned the risk of overengineering that might arise when teams try out what influencers in technology are claiming. IMHO, this is a debate born out of a prejudice. In my more than 20 years in the profession, I only found one example of overengineering, whereas I had to struggle with plenty of examples of bad code, bad practices, or bad architecture. I am not saying overengineering is a fake problem, just that I feel like we should put our efforts on the 95% of the issues before facing the (being generous) 5% of them.

Choosing the Right Architecture Style Debate

The last debate, and the last session, of the event was moderated by Christian Cotes (@ccotesg) and had Alex Soto (@alexsotob), Viktor Farcic (@vfarcic), and Ana-Maria Mihalceanu (@ammbra1508) as guests.

Again, the debate circled around the adoption of microservices as architecture style, and more in general about how hard it seems to distinguish between what is hyped and what is reasonable. It was certainly a very lively conversation, mostly because of the funny style of Viktor Farcic, the author of the book “DevOps Paradox“.

Unfortunately, there was nothing new to add to what we had already listened during previous sessions: the microservices debate, and the fear of change some professionals exhibit, though they likely consider themselves wise and not fearful. Maybe the organizers should consider how to avoid this reiteration of topics in further editions of the event.

Data-Driven Scalability and Cost Analysis for Evolutionary Architecture In the Cloud

Between the two debates above, Ian Gorton presented the results of a study that the research team he leads in the University of Northampton in Seattle conducted to analyse how the cloud costs vary depending on the programming language, and also how those costs depend on which are the values we set in the scale configuration panels of cloud services.

In my opinion, was not a surprise that Go was the most performant programming language in Google Cloud, not because both are initiatives born at Google headquarters, but because Go community shows to have one, and only one, focus: performance.

No surprising neither that the default values suggested by Google Cloud were not eventually the cheapest. That is why companies which whole business model is help organizations reduce their cloud bills exist.

Copyright notice: Featured image taken from PlanetGeek.ch blog. Visit the blog for a full set of other enjoyable pictures.

Global Software Architecture Summit 2019 – episode IV

This is the fourth episode in a series of posts reviewing the first Global Software Architecture Summit that took place in Barcelona the 10th of October. To see the whole list of episodes, please visit this.

Reactive Architecture Patterns Debate

This second debate was moderated by Jakub Marchwicki (@kubem), and speakers were David Farley (@davefarley77), Mark Richards (@markrichardssa), Len Bass (@LenBass4), and Christian Ciceri. It had the Reactive architecture style, and incidentally the Reactive Manifesto and its advocacy of asynchronous procedures, as its key topic, deriving into complexity often too.

Actually, I signed the Manifesto 5 years ago (under the name of my company at a time, though), and have been working with asynchronous messaging since, whenever no dialogue with humans is required, because it improves performance at the UI level, and this is something that, as it is immediately obvious, makes sense.

Reactive architecture style has become extremely popular: Node.js was asynchronous from scratch, all programming languages got their implementations of asynchronicity in recent years, and several flavours of Event-centric architecture styles arose as well. Even a Reactive Foundation showed up, now as part of the Linux Foundation. Clearly, async came to stay.

However, if decoupled transactions, cut in pieces to be handled by autonomous services, were already hard to govern, making them asynchronous turns out to be even harder, as Len Bass repetedly said during the debate. Are our teams ready to handle this complexity? And more importantly, is this complexity necessary in all the contexts when it is being applied now? Unfortunately, this debate on complexity kept the speakers to explore other implications, or technologies which implement reactiveness which might not be known by the audience.

Cover picture of the end of work book
The End of Work book

Another fascinating idea dropped by one of the speakers, but not followed up, was this seek of humanless that technology seems to be in seek of recently. I am afraid people with no technical background is increasingly feeling upset, even scared, by this pursuit, and I do not see that people with technical background (like the speakers, for instance) spend enough time stating clearly that the work is not ending. What we seek is automation of tasks that, by their own nature, do not require understanding to be completed.

For Artificial Intelligence, no matter its name says, is not turning machines into smart entities: machines can be trained to recognise cats in pictures, but not to know what a cat is. Some people believe that machines might achieve that ability (let’s say true intelligence) in the future, but I do not see this happening soon because just a few animal species reached that point, and evolution needed thousands of milions of years for them to appear. To behold tiny algorithms transcending their machine nature and become beings is not a spectacle I am expecting to see in my life.

Global Software Architecture Summit 2019 – episode III

ci-cd pipeline picture

This is the third episode in a series of posts reviewing the first Global Software Architecture Summit that took place in Barcelona the 10th of October. To see the whole list of episodes, please visit this.

Software architecture evolution, what to expect? Panel Discussion

This panel was moderated by Alvaro García (@alvarobiz) and the participants were Eoin Woods (@eoinwoodz), Michael Feathers (@mfeathers), and Ian Gorton. They had an animated conversation, during which the trouble with Microservices came up again.

No one doubts that Microservices is a very demanding architectural style. Everything is easier if done in a monolith than with microservices. Unfortunately, as I pointed out here, monoliths aversion to change prevent organizations to evolve by making this evolution tougher day by day. And there are plenty of reasons for organizations to embrace change: for it makes them more competitive, quicker to response to unexpected shifts in economy, or laws and regulations; in a word: keep them safe from obsolescence.

So, on one hand, monoliths are easier and cheaper, but age us. On the other hand, microservices (as an example of decoupled architectures, or, in a wider sense, of those more complex architecture styles we heard about in the Episode II of this series), keep us afresh, though they are tough, and expensive. What to do, then?

As I see them, microservices and monoliths are two opposite poles in a whole spectrum of factually possible adoptions of architecture styles. I mean, it looks reasonable (and I saw it frequently in the past three years) to have 5, 10, or 15 microservices running in the very same ecosystem that a heavier set of coupled functionalities that we may still call a monolith. This is not a mess, this is an implementation in progress towards a more and more efficient ecosystem step by step.

Monoliths and microservices are two poles in a full spectrum of possible architectures

Maybe microservices detractors feel afraid of gigantic projects which purpose is to decouple completely a fully functional monolith into an ecosystem of microservices, all in all within a timeframe of, let’s say, two or three years. Well, I must agree with them that this kind of transformation is not only too risky, but, in most cases, unnecessary. Mixed architectures, laying somewhere in that spectrum I mentioned above, may be the best possible answer.

The panel also emphasized the key importance of experimentation and observation. Again, they advocated the adoption of the Scientific Method (come up with a theory, run experiments, gather information, and obtain useful knowledge), in a neverending process a colleegue of mine named “trial and error“. But this name does not feel right to me, because I am afraid it is missing the relevant part of the point, and makes it something trivial, when it is not.

I’d rather call this Continuous Change, as a practice and as a third companion for the Continouous Integration/Continuous Delivery, duo. To me, the Continuous Change would be a more general approach to the creation of software, whereas CI/CD would be a particular application of it. For CI/CD, as in the famous diagram modeled out of the Möbius strip, tighly integrates the creation of the code with its deployment, in a way, for instance, that makes testing in production an obvious consequence.

But they are not. The Möbius strip is one possible topology to picture the production of software, among others. In this topology velocity prevails, in a way that the bigger the number of iterations in a given period, the better. Actually, the CI/CD cycle prevents experimentation unless it is reducible enough to be feature flagged, in a mechanism which purpose is to make changes as atomic as possible. Following this approach further is how even chunk commits look self-evident.

Unfortunately, I was not able to find a better idea to integrate experiments in the common CI/CD cycle in a more satisfactory way. A topic for another day, I am afraid.

Copyright notice: Featured image taken from the Linux Juggernaut web site without permission.

Global Software Architecture Summit 2019 – review series

GSAS 2019 logo

Thanks to my employer, I had the opportunity to attend to the first Global Software Architecture Summit that took place in Barcelona the 10th of October. The event, organized by Apiumhub, consisted in a one day track of speaches, debates, and one panel, as well as the usual breaks to mingle with the community, all around the topic of Software Architecture. 8 different sessions, overall provided by a bunch of illuminating, and funny, speakers who brought a huge level of expertise to the stage, and that ended up being an extremely inspiring, though exhausting, day for me.

So, I decided to share the ideas I came up with during the event here, hoping that some of you might eventually feel that inspiration too, at least as much as I can recreate it. Problem is, there was so much to say, and so many paths to follow starting at any of the topics dropped by the speakers during the event, that it came eventually obvious that I would rather cut the whole review in smaller, digestable pieces.

This is how this series came into light. The episodes are the following:

Summary

Some of the episodes are longer than others, whereas some topics show up repeatedly in some, or all, the sessions. Should I list the main topics, I’d clearly separate the incidental from the fundamental: micro-services, kubernetes, and complexity, as incidental; and the high value driven by experimentation, as fundamental.

Certainly, overall the event was highly inspiring, and I left the place with plenty of ideas that, I am sure, worth spending some time exploring.

As a final note, my advice to everyone involved in producing software: attend to events, mingle with the community, learn, and enjoy!

Copyright notice: Featured image taken from the GSAS Twitter account without permission.

Global Software Architecture Summit 2019 – episode II

A depiction of observability by Haproxy

This is the second episode in a series of posts reviewing the first Global Software Architecture Summit that took place in Barcelona the 10th of October. To see the whole list of episodes, please visit this.

Applying architectural principles, processes, and tools

Picture of Len Bass
Len Bass

The second speaker was Len Bass (@LenBass4), who started by giving us what seemed to me a first-day lesson in college on the 5 key characteristics a software architect must be proficient in, but when the time for questions arrived, dropped plenty of valuable thoughts.

One of the main vectors of change in how software is created has been the recent incursion of Operations in it. Traditionally, developers cared very few of how the code they produced was actually run. Since the DevOps, the role of developers has evolved quite quickly, and right now a developer who does willingly ignore the operational part of the software gets (or should get) easily unemployed.

Which, incidentally, drives in another of the main leit-motives of the whole event: which level of architectural complexity should we accept because of that invasion of Operations in software. Illustrated with the neverending monolith versus microservices debate. Some of the speakers in the debates were contrary to the adoption of microservices and others were in favour of them, if and only if microservices fit the problem and context. This latter seems a pretty cautious position to me, but definitely it was not conservative enough for those who complain every time things evolve faster than they can follow.

We will have time to come to this again. Before that, I would like to note three other thoughts from Len Bass: one was the idea that it is very rare that software crafters participate in the creation of any application which is not similar to other applications already created. An example of a true novelty is the Amazon’s invention of a bunch of new technologies, tools, and procedures in its seek of Infrastructure Scaling. In general, though, we should keep ourselves aware of what other people is doing all the time, as well as being open and communicative so that others may also benefit of our work.

A second thought is that log files are quite more important than we think: log files mainly exist to help first responders in case of an emergency. But they also help us gather deeply valuable information of our software’s behaviour. Which reminded me of Observability, a field that fascinates me for its promise of providing us with the ability to query the system any time, and which depends on log files to deliver its magic.

For those also interested in Observability, I highly recommend this introductory page in the Honeycomb website (I have no liaison with this particular company, except by the fact that I got completely mesmerized by a speech of their former CEO, and current CTO, Charity Majors, back in 2018 in London). There is a podcast that I follow too: o11ycast. And a conference, O11ycon, which happened in San Francisco last year. Slides and videos are free to access online.

A third idea that Len Bass said that shocked me was that Microservices potentially drive technology inconsistency into our application ecosystem. This is a common criticism, though I doubt that it is fair. It seems reasonable to assume that there is a limit in how much diversity organizations can handle. Where this limit lies on, and why is the blame put on microservices when they just expose this limit, are the two factors that look unfair to me.

Bearing in mind that Microservices allow organizations to isolate disparate technologies as long as they communicate via standard protocols, like HTTP or asynchronous messaging (by the way, another schoking moment was when someone in the audience asked a question in a way thay made me think this person believes microservices are asynchronous per se, which is wrong), the potential incosistency in adopting those disparate technologies cannot come from them as an architecture style. For it is completely possible, even reasonable in some contexts, to adopt microservices using just one technology.

I believe that microservices allow us to pick the technology that best fits in every case, but this is my believe, not something that microservices force organizations to do. And even I accept that this is an ideal: in practice, the resources are scarce, and organizations must focus on less than 5 technologies. I worked in a company where even more than 5 thecnologies were in use, and everyone was aware of the reasons why, but I do neither recommend that, nor see it happening a lot. On the contrary, my advice would be to focus on the operational, and the infrastructure, issues, get skilled on that, before bringing in additional diversity.

In my opinion, inconsistency comes out of us making bad decisions, not by the adoption of any particular architecture style. Microservices, as any other architecture style, are neutral, optional. Pick them if that fit your needs, simply ignore them if not.

Copyright notice: Featured picture taken from the Haproxy website without permission

Global Software Architecture Summit 2019 – episode I

A picture of Peter Naur by Duo Duo Zang

This is the first episode in a series of posts reviewing the first Global Software Architecture Summit that took place in Barcelona the 10th of October. To see the whole list of episodes, please visit this.

Code is your partner in thought

George Fairbanks
George Fairbanks

George Fairbanks (@Fairbanks) was the first speaker, and introduced to us some ideas which originally appeared in a book by Peter Naur, in 1985, being the most relevant of them the realization that the act of Programming, understood as the overall set of activities involved in the production of software, is by itself a task of invention of theories to understand a problem and finding suitable solutions to address it. Actually, the title of the Naur’s book was “Programming as Theory building“.

To achieve a solution to a given problem, we first invent theories, and the code we eventually write is a representation of that theory. Which is another powerful reason why the code must be as understandable as possible: because the code is the main instrument of communication of that theory to other people, now and in the coming future; a purpose, by the way, that the old-fashioned procedure of writing tidy documentation upfront of the code was unable to fulfill ever.

Computer programs only make sense to the team who have developed it

Peter Naur (1985)

In other words, the code repository is not only where the documentation of the solution, and of the problem it addresses, live. Since that code is also a representation of the theory we invented in the process of understanding the problem and generating ideas to handle it, git is not only in charge of handling a repository of code, but also a repository of knowledge.

Actually, this is an idea which stands out as the pivotal summary of the whole event: the creation of software is an exploration of thoughts inspired by the necessity to address some problem, and it looks, and it should be practiced, more as Science is, via the scientific method. Come up with an idea, produce some experiments to test it, and, if the experiment ends well, make it happen in Production, just before keeping the exploration going on.

Even more, this procedure of coming up with a theory, test it in practice, and eventually put it into production in the form of runnable software, is a long term learning process. As metaphor, we might imagine a wider cycle around the quicker Agile cycles that we are used to call sprints: a process extremely important, in my opinion, to keep Scrum teams from falling to the sickness of delivering new features time and again, but never reaching any goal.

Fairbanks dropped other interesting thoughts, as the concept of the Model-code gap (in essence, the troubles which arise whenever the representation of our thoughts that we put in our code is crooked, or biased, instead of the finest match to them possible), or that documentation is a good place to explain those ideas or decisions which did not get through: precisely because the code does not represent them, it may be a good practice to include them in the code repo as README files, for instance.

In essence, it was a very nice introduction of the main topics that we were going to see showing up throughout the whole event.

Copyright notice: Painted portrait of Peter Naur by Duo Duo Zhuang.

Going further with Events

Recently Mathias Verraes published several posts in his blog around patterns for Events. This was very fortunate indeed, not only because these are valuable insights for those who architect decoupled applications using Events as a key instrument, but also, quite incidentally, because it gave me the opportunity to streamline my own ideas on Events and asynchronous decoupled architectures in general.

Inspired by some of the ideas he introduces there, I came up with the concept of Story as an implementation of the Event Summary pattern. Let’s see how and why.

Event Summary pattern

One of the patterns that intrigued me more is the Event Summary pattern. In short, this would be an individual Event whose body gathers data already present in the bodies of other, previously dispatched Events, according to some relationship they share that makes them convenient for being dispatched together, and consumed all at once as part of an aggregate.

Let me give you an example. Just imagine a music concert and a service that dispatches a SongPlayed Event every time the band ends playing a song. Now, also imagine another service that only acts on music concerts as they end, for instance, to communicate the list of songs played to a music rights management entity.

Clearly, all the SongPlayed Events dispatched during a particular music concert share a common relationship, which is that they were played that day, in that music concert. This is the fact by which we may found useful to dispatch a summarising message at the end of the concert.

In this scenario, would not be easier (and cheaper) for that latter service to consume just  an Event Summary dispatched at the end of the concert, with information gathered whenever a song was played, or even taken from the bodies of those SongPlayed events, instead of consuming all those annoying SongPlayed events one by one?

Likely everyone, even I myself, would answer YES.

Regardless of my affirmative answer, there are some details around this Event Summary pattern that, in my opinion, deserve further attention before we move forward with my proposal for an implementation.

A tale of species

The circumstance that we have got an infrastructure of asynchronous messaging should not drive us to say that every structured message travelling up and down throughout that infrastructure is an Event.

In my opinion, we should define a kind of a taxonomy of messages so that applications may behave one way or another depending of what kind of message they got to have to deal with. In practice, automated services are ready to handle a short list of messages, and silently let the others go away. However, the list of message patterns is growing, and IMO we architects would appreciate to have them organised somehow around their features: headers, body, name, etc.

Following the thread of posts presented by Mathias Verraes in his blog, there are at least two main features which may help us build that taxonomy:

  • Fullness, which allows us to classify messages depending on whether they bring all the information available in the context of the message dispatcher, or they just left some data behind for any reason, in particular because stayed unchanged (i.e., their values were the same before and after the fact that the message informs about happened).
  • Time Atomicity, which allows us to identify messages as atomic, were they representing a fact occurred in a given instant of time, or not atomic, were they bringing information related to a happening broader in time.

A definition of Event

With all that arsenal on hand, let’s start by streamlining the definition of Event:

Events are full representations of facts that happened in a moment in time.

When we say the Events represent Facts we mean that Events bring what happened in the real world to the application using the terms the application understands. So Events not only transport information, but translate it as well from the common language words we all understand to the specific terms that we would find in the code were we reading it, and that were invented in the process of designing the application.

If we look at the definition of Event under the light of the two message features above:

  • All the raw information available in the context of the fact should be included in the  body of the Event that represents it. There are operative details to account here, but let’s just say that the context of a fact is all the data that we may collect about it without doing any additional operation, like querying a data source or make some calculation.
  • Facts refer to an instant, roughly a date and time, up to the second or as minutely as needed. This usually means that facts got an Event representing them as soon as they end. In English, this is indicated by past participles of verbs.

So, according to these remarks above, it seems clear that messages from the past which are not exhaustive, but selective, when taking data from the fact being represented, should not be considered Events.

This condition of Fullness avoids dispatching Events with only the data any given Consumer would claim for. Consumers must take all or nothing, and simply ignore whatever data they do not need from the body of the Events they consume. 

No matter reasonable this Fullness may look, it is just convenient. Actually, it is easy to imagine scenarios in which breaking this Fullness rule is far a better option. Mathias Verraes also talks about this in the post Segregated Event Layers in his series.

Fullness is not Completeness

I think it is worth mentioning that the Fullness condition introduced above refers to every particular message dispatched in the application. Actually we may be dispatching messages which are selective, in the sense that some properties available in the context of the dispatcher at dispatch time are not included in the message body, instead of exhaustive, so that they could not be called Events.

Even though, we may be dispatching an Event whenever a fact happens in the business context mapped in our application via one, or more, services. In this case, we would be talking about Completeness. I took this term also out of Mathias Verraes blog, where he calls it a Completeness Guarantee.

Obviously, it takes much more effort to accomplish that Completeness Guarantee that just to ensure that every message dispatched is full. In my opinion, it is a feature of our applications that is achieved incrementally, in a journey which might likely never reach its end (unless we started up with a requirement for Completeness Guarantee fulfilment in mind from minute zero).

Stories

Summary Events is a message pattern which breaks the rule of Atomicity because they represent facts which life spans longer than just an instant. These messages may, or may not, fulfil the Fullness rule. Whenever they do, I propose to call them Stories.

Stories are full representations of facts that spanned during a period of time.

(I must recognise that this is not so an original name: we are living in the times of Snapchat’s stories and Instagram’s instastories.)

The key point in the definition of Stories is the past tense. Nothing can be considered a fact if is still going on, not only because otherwise we cannot know when it ends, but also because we cannot say for sure that it will end.

Therefore, anything that should be triggered in an application during the lifespan of a still ongoing Story, should come out of true facts (for instance, an already played song in a still ongoing music concert), which means an Event. This is how the Event Summary pattern comes implemented with Stories.

In general, uninterrupted periods of time can easily be represented as Stories. However, there are situations in which consistently unique facts exist, even though they might be interrupted. For instance, a Wimbledon tennis match which was interrupted by the rain. That interruption should be included in the Story’s body, in order to make sense of the partial facts, or Events, which are part of it.

In practice, there are (at least) two ways of implementing Stories:

  • As a deferred fact.
    We know for sure that the facts are going to end some moment in the future, and we keep gathering information out of its context up to the very moment it ends, and then we dispatch the whole Story downstream.
  • As a projection produced after the last Event in the Story is dispatched.

An example of a Story: a motorbikes race

Let me finish this long post with an example taken from the company I am currently working at, and consider a motorbikes race of the World MotoGP Championship.

A race has a clear starting at point in time, but it is not possible to know the exact time at which it will end. However, it constitutes a unique fact in time, which makes it eligible to be represented as a Story.

There are plenty of facts that happen in precise an instant during a race, some of them expected (i.e., the first rider crosses the line after completing the first round, etc.), whereas some of them are unexpected (i.e., accidents, rider falls, red flags, etc.). Every time any relevant fact happens, an Event should be dispatched. This is what makes this a Story, instead of simply a collection of data we may produce in anticipation.

Actually, this kind of Story is extremely simple to implement: the data is gathered from the context as it is, with no further operation. And, just at the end of the race, whenever the last rider crosses the line, the Story can be dispatched.

Event Summary pattern revisited

However broad the bunch of real situations in which they may be the rightest message representation of choice, Stories are not a simple aggregation of data. They collect the information available of a longer-than-one-instant fact and bring it into our application.

Going back to the post in the blog of Mathias Verraes where the Event Summary pattern is presented, a daily sales report summarising all the sales in a given day is clearly not a Story because all those sales operations are not related whatsoever, except for the circumstance that they happened the same day.

Did I came up with a name for this third kind of messages? Not with one that convinces me: I’d call them Reports. But let’s leave them for another day.

Photograph credit: https://pixabay.com/users/geralt-9301/