★ Inclusive developer

So where have we got to? Access is important, but inclusion is bigger than access. Inclusive design means making something valuable, not just accessible, to as many people as we can.

What the Heck Is Inclusive Design? (cache)

Job titles are hard. I always wondered which is mine. I think Heydon Pickering nailed it with his last article on 24ways. I’m working on inclusiveness due to recent changes within my team and as part of my road to become a senior developer. Being inclusive means a lot of things to me:

  • help newcomers jumping into projects smoothly with explicit documentation of governance and participation
  • enjoy nonviolent communication during developments (pair-programming, code-reviews and so on) and quick feedback loop on iterative releases
  • avoid burn-out/isolation for co-workers with healthy lifestyle and solidarity
  • turn open-source into free software (more in a future article)

Inclusive developers lead to inclusive teams. Inclusive teams lead to inclusive products. Not the other way around. The values you share within your team will be the ones transmitted by your product. It’s hard to have empathy for your users if you don’t have empathy for your colleagues first. I’m paying the price these days for not being careful enough about people I’m working with. It will be one of my take-aways from 2016. No matter the importance of your achievements, you are interacting with humans and taking the time to listen to them is important.

I hope I’ll be able to qualify myself as an inclusive developer, someday.

★ Cultural Intimacy

I just came across that campaign from Spotify titled “Thanks, 2016. It’s been weird.” where we can read on billboards things like:

Dear 3,749 people who streamed “It’s the End of the World As We Know It” the day of the Brexit vote, Hang in there.

or

Dear person who played “Sorry” 42 times on Valentine’s Day, What did you do?

When a campaign like this is cheered as being “Brilliant”, I’m afraid that we lost our collective mind (and my battles about privacy). It literally means that people at Spotify are aggregating data to make fun of you. And you enjoyed it. Smiled at it. Shared it. Even paid for it!

In that case, they are dealing with big data — maybe anonymised — so it doesn’t elect as a privacy issue, right? Well, it’s maybe worse than that. Something affecting communities in a deeper sense. It’s a matter of cultural intimacy. Revealing such moments is segregating more than being inclusive. And you should be ashamed of that. More insidious is the fact a company can deliberately report playing with customers’ data on gigantic billboards and nobody reacts. It makes me so sad.

My FLAC library doesn’t spy me each and every time I play a music file. I can listen to Wicked Game interpreted by James Vincent McMorrow 20 times in a row if I want to and nobody will judge me, except maybe my neighbours. Think about it next time you play something on someone else’s computer (a.k.a. cloud), he knows it and will make fun of it. A torrent file does not.

Thanks, Spotify. YOU are creepy.

★ Blogrolls et découverte

Il faut également ouvrir de nouvelles routes. Sous l’influence de Google, nos blogs se sont repliés sur eux-mêmes. Nous devons en revenir au principe des blogrolls. Créer des liens vers les blogs amis. Non dans un but de référencement, mais avec l’espoir que nos visiteurs effectuent un pas de côté, cela depuis chez nous, sans remonter jusqu’à Google ou un réseau social.

Comment sauver notre Web ? (cache)

Les derniers billets de Thierry Crouzet étaient assez pessimistes, et puis celui-ci donne des pistes pour améliorer la situation. En remettant notamment à l’ordre du jour l’usage de la blogroll, ce réseau social numérique avant l’heure. Je l’avais supprimée de cet espace car ce n’est pas évident à maintenir (techniquement et émotionnellement) et je trouve que mon OPML est trop personnel pour le publier fréquemment.

Je me suis mis en quête de blogs en français, encore alimentés, un minimum techniques et relativement intimes — dans l’idée que je m’en fais, ils sont peut-être très populaires — pour proposer de nouvelles pistes de lectures. L’algorithme pour arriver à cette liste est humain, il n’a pas la neutralité d’une plateforme et il est explicitement anti-Medium.

Le constat est assez terrible de ne pas réussir à en lister davantage. Du coup j’étends un peu en acceptant la veille :

Quelle difficulté pour trouver 15 liens… je ne sais pas trop ce que je dois en conclure si ce n’est que ma bulle est toute petite. J’espère que #nowwwel (cache) sera l’occasion de découvrir de nouvelles sources. J’espère surtout que vous allez m’en envoyer d’autres par courriel :-).

PS : vous remarquerez que j’utilise le nom des personnes dans mon agrégateur, c’est pour moi très important dans la lecture de pouvoir mettre un visage ou au moins un humain. Si vous voulez une liste d’entreprises, je vous conseille celle compilée par Éric D (cache).

Réactions (ajout manuel)

La blogroll est une bonne idée. Les blogs avec la possibilité de commenter aussi, tout autant que ceux qui permettent de les rétrolier.

Blogrolls et blogs (cache)

Ça ressemblerait presque à des commentaires distribués avec cache local !

★ Accompagner un enfant

Alexandre,

Voici le compte-rendu d’une expérience, celle de ces trois dernières années. Je ne sais pas si tu auras la volonté ou la capacité de lire ce texte à un moment de ta vie, je le produis en guise de mémoire et peut-être de pistes à explorer pour toi ou d’autres. Ou pas, sa durée de vie est liée à l’attention que je porte à mes textes. Il est empli d’incertitudes et de doutes que je te souhaite d’avoir un jour à ton tour.

Ces années ont été parsemées d’émotions, de peurs, d’apprentissages, de culpabilités, de bonheurs, de stress en tout genres, d’amour, de contradictions et de bienveillance. À tel point qu’il est difficile de classifier ou d’ordonner cela. Je vais tenter de m’en référer à la souvenance que j’en ai.

Conception

La crise consiste justement dans le fait que l’ancien meurt et que le nouveau ne peut pas naître : pendant cet interrègne on observe les phénomènes morbides les plus variés […] Le vieux monde se meurt, le nouveau monde tarde à apparaître et dans ce clair-obscur surgissent les monstres.

Antonio Gramsci

Ta conception biologique a été simple. C’est tout le processus en amont qui a été très long pour accepter de mettre au monde un nouvel être. Pour gagner la confiance suffisante en moi, en nous et dans les autres. J’envie parfois presque les personnes qui ne se posent pas toutes ces questions mais j’ai l’illusion de croire qu’elles passent peut-être à côté de l’essentiel : cette remise en question permanente.

J’espère me tromper en voyant surgir petit à petit des monstres qui ont de plus en plus de pouvoir. Il va sûrement falloir apprendre à vivre avec car nous évoluons au sein du même environnement. Je souhaite pour autant ne faire peser aucun espoir sur tes épaules. Sois.

Être

Pensez-y en cette nouvelle année scolaire, vos élèves apprendront beaucoup plus ce que vous êtes que seulement ce que vous savez. Vous êtes ce que vous savez. Si les apprenants apprennent ce que vous êtes, ils apprendront nécessairement ce que vous savez. Par contre, ils ne deviendront jamais ce que vous êtes en apprenants seulement ce que vous savez. Vous pouvez relire ce paragraphe plus d’une fois.

Il vous restera à régler la question qui tue: « Les élèves sont-ils intéressés à devenir ce que vous êtes ? »

Être ce que l’on enseigne! (cache)

Lorsque tu as découvert l’air libre, j’ai envoyé un message à la famille proche : « Alexandre est né » et l’on m’a reproché d’avoir été trop bref. Je pense au contraire avoir été trop long. « Alexandre est » aurait amplement suffit. Et pourtant je ne saurais dater précisément ta naissance qui a eue lieue bien avant cette libération. Si la vie est un état, la mise au monde est davantage un processus qu’une date, et celui-ci est toujours en cours.

Vivant

Un adulte créatif est un enfant qui a survécu.

Ursula K. Le Guin

Trois ans et tu es toujours vivant. J’aime ta vitalité et j’essaye d’interférer le moins possible avec ta créativité et tes expériences car je considère que ce sont des instants très importants. J’aime voir l’étincelle dans cet œil qui préfigure de nouvelles connexions neuronales. J’aime que tu t’essayes à l’humour. J’aime ta capacité de concentration quand tu souhaites comprendre quelque chose. Ou quand tu t’enfermes dans ta chambre pour dessiner un cachalot.

Tu as vu très peu d’écrans pour l’instant. Ton imaginaire s’alimente beaucoup dans les livres dont la sélection s’avère être de plus en plus compliquée pour éviter la violence sans pour autant tomber dans le moralisateur. Tu n’as pas été sur beaucoup d’écrans non plus, ton existence numérique semble tellement futile en regard de la richesse d’autres interactions.

(Non)Violence

Enfance sans confiance n’est que ruine de l’Humanité.

C’est la principale raison pour laquelle nous sommes allés au Québec. L’envie de trouver un cadre différent, avec moins de violence à la fois physique et psychologique. L’humain est ainsi fait qu’il a besoin de boucs-émissaires, les régions inhospitalières ont cet avantage de moins s’en prendre aux minorités qu’au climat. Ce choix fut particulièrement difficile car il impliquait une mise à distance vis-à-vis de la famille.

En te faisant confiance, je n’ai pas besoin de violence. En limitant les stress, je réduis les sources de tensions. C’est un luxe qui demande de l’espace et du temps. Et de la volonté. Le résultat est pour l’instant bien au-delà de mes espérances et je me demande ce que cela va produire par la suite.

Mimétisme

Grandir c’est se spécialiser. L’adulte n’est pas moins intelligent, il est spécialisé : spécialisé dans sa langue, dans sa culture, dans sa pensée, dans ses comportements sociaux, etc. Et vivre avec l’enfant, c’est participer à sa spécialisation. Nos façons de parler, de réagir, ce que nous faisons avec lui ou devant lui, va littéralement participer au câblage de son cerveau.

Les lois naturelles de l’enfant (cache)

Je constate chaque jour à quel point l’inné est mince face à l’acquis. La co-évolution culture-génome théorique me semble être clairement déséquilibrée en pratique et ce déséquilibre tend à s’accentuer. Cela donne une incroyable responsabilité à l’accompagnant qui en vient à guider quasi malgré lui. Partager ses expériences sans orienter est un vrai challenge. Faire s’épanouir un libre arbitre et un esprit critique qui ne soient pas les nôtres.

Je ne cherche pas pour autant à me mettre en spectateur comme pourrait le faire un photographe. Je préfère égoïstement partager avec toi sans penser aux likes des proches. J’aime participer à cet apprentissage et j’essaye autant que possible d’avoir des moments explicites d’échanges qui ne soient pas encombrés de flatulences numériques. La seule chose que je puisse et veuille t’offrir c’est de l’attention.

Alimentation

Tu manges ce que tu veux et presque quand tu veux. Tu manges souvent froid. On mange souvent devant un spectacle bien plus enthousiasmant que la télévision. Je finis beaucoup de plats (froids aussi). Aucun repas n’est équilibré. Aucune semaine ne semble déséquilibrée pour autant. Tes goûts évoluent d’un jour sur l’autre et me font reconsidérer cette notion.

Il y a des moments où le parasitisme devient symbiose et cela me met en joie. Je réalise que l’accompagnement est réciproque, ce qui garantit sa pérennité. En plus du savoir-faire et du savoir-être, il y aurait un savoir-échanger à développer ensemble, la famille étant ce pot commun dont nous sortons tous grandis.

Instruction

Je me suis renseigné sur de nombreuses formes d’instructions. On a visité des écoles plus ou moins traditionnelles. Difficile de trouver un lieu qui soit en accord avec nos valeurs. Peut-être que l’instruction en famille sera finalement choisie au prix d’une certaine marginalisation. Si c’est l’option retenue, il faudra absolument que l’on trouve d’autres sources de connaissances et d’émotions que tes parents, aussi incroyables (et modestes !) soient-ils :p.

De multiples inspirations pour une expérience forcément unique. Et en même temps avoir la volonté de partir d’une page blanche pour apprendre à l’écrire ensemble, d’expériences en réajustements. Chaque déséquilibre motivant le prochain pas, vaincre la peur récurrente du vide en se faisant confiance mutuellement.

Tabous

Appartenance, fierté, amour, chantage, beauté. Il y a des jugements que je me refuse à ton égard car je ne les trouve pas sains. Je ne veux pas que tu aies besoin de mon regard pour te trouver beau. Je ne souhaite pas introduire de chantage entre nous, encore moins affectif, c’est trop facile. Il ne peut y avoir d’amour dans une relation imposée comme celle d’un lien de filiation. C’est autre chose. Dans la fierté réside un espoir initial égoïste, une projection que l’on atteint enfin.

Ces termes ne sont pas tabous pour autant et nous aurons l’occasion d’échanger beaucoup plus longuement sur ces sujets. Et plein d’autres. Tu m’auras vraisemblablement fait changer d’ici là.

Conclusion

Il n’y a pas de conclusion. La futilité du quotidien reprend son droit. La beauté de l’éphémère et du moment présent. La conscience de vivre ensemble pleinement. L’inconscience de savoir que ça ne pourra durer éternellement. L’intime croyance qu’un lien se tisse durablement entre deux hommes qui se transforment.

Aujourd’hui c’était une chouette journée. Hier pas mal non plus. Et demain ? On verra bien. Ensemble ? Avec plaisir !

En recherche d'emploi - Looking for a job

English version

Je suis à la recherche d'un nouveau poste dans le développement web. Comme pour mes précédents jobs, je recherche une société éditrice de ses produits et non financée par la publicité. J'aimerais intégrer une équipe qui s'interroge régulièrement sur ses méthodes de travail. Idéalement, le télétravail est accepté dans cette entreprise.

Si vous êtes intéressés par mon profil, mon CV est disponible. Si vous avez une idée de société qui pourrait m'intéresser, n'hésitez pas à me contacter.


I'm looking for a new job in web development. Like previous gigs, I'm looking for a company that works on its own products and with a business model that does not rely on ads. I'd like to be part of a team that regularly questions its methods of working. Ideally, remote work is accepted in that company (I'm staying in France).

If my profile seems interesting, take a look at my resume. If you have an idea of a company that might be suitable, please leave me a note

★ Tools and teams

Use the right tool for the job.

I had been guilty for so many years failing to apply that simple old saying. Until I realised that if your job is not well designed you cannot find the right tool. Until I realised that you cannot know every existing tools. Until I realised that knowing your tools is sometimes more effective than picking the perfect one. Until I learnt the hard way that in an evolving context you have to adapt your tool across time. Today I’m more inclined to say: “Use the right toolset for the team at a given moment.”

It might be counter-intuitive at first but after all our job is to solve pretty basic technical problems: display that data, allow interactions with this one, make sure everything is smooth for everybody, secure as much as you can. Nothing more.

We’re not paid to write code, we’re paid to add value (or reduce cost) to the business. Yet I often see people measuring their worth in code, in systems, in tools—all of the output that’s easy to measure. I see it come at the expense of attending meetings. I see it at the expense of supporting other teams. I see it at the expense of cross-training and personal/professional development. It’s like full-bore coding has become the norm and we’ve given up everything else.

You Are Not Paid to Write Code (cache)

The problem arises when we introduce layers to solve these basic needs, even worse when they are not resilient enough to guarantee their accessibility toward as much people as we can. Frameworks are popular today — bashing them too. We are often forgetting that framework starts with frame. This is the frame within which the author of the framework have (or worse, has?) his own job. When you choose a framework you are betting that your job will stay within its frame. Otherwise it’s a nightmare because breaking the frame will dismantle the consistency you were looking for at first with that solution. When all you have is a framework, everything you achieve looks like a patchwork.

A micro-framework tends to extend that frame and to turn it into something more loose that you will strengthen yourself for a given domain in order to accomplish the initial job. The result is still a frame but your current job hopefully is at the center with more room to evolve and find his audience. The trade-off is that you will have to write more code and your team must be concept-competent, not framework-competent.

When your team acquires experience about a given tool it is a short-term advantage over your competitors, you will be able to iterate quickly. When your team acquires knowledge on a given concept it is a long-term goal for your product, you will be able to pivot faster. The balance on the business side is to still be alive when it happens. The goal on the developers side is to still be competent when the framework and/or the product fades.

Code clarity

Let me tell you a story about code clarity. If you’ve met me, I’ve probably told you this story before.

Back when I was in engineering school, I had some algorithmic courses. The teacher would often ask us how many lines a particular algorithm would take to solve. Some students would take guesses: - 10? - No. - 16? - No. - 13? - No. 3.

The first time we heard that answer, we thought we were in for some very clever solving that we haven’t heard about yet. And then the teacher would write down three function calls. Wait, that didn’t solve anything! They just tricked us. Those three lines don’t accomplish anything, we still need to write the bulk of the work in those functions. And in the end, we’re going to see fifteen to twenty lines so we weren’t far off.

The teacher asked us this question many more times. We were not expecting amazingly short algorithms anymore but we’d still think about solving the whole puzzle when guessing line numbers. And the teacher would never go above five lines for any algorithm.

At the time, we were mocking him. We couldn’t see the value of this way of thinking.

Fast forward now and it is one of the thing I do most when writing code. I’ll take a piece of paper and write the function names I’m going to implement. Or I’ll do it directly in my code editor, with real functions or comments. And I’ll also give that advice to any developer I meet.

It allows you to focus on one problem at a time. When you’re writing those function names, you are thinking about what the code should be doing. When you’re implementing the functions, you are thinking about how the code should do it. This clarity gives you the best chance to write code that is easy to understand for the next person reading it.

Team effort, cheap iteration

Your mind is focused on the problem to solve, on the different ways you could solve it. Once you’ve broken down the problem, you can discuss with your team which solution they’d like to implement. That iteration was pretty cheap because it didn’t take very long to imagine those different solutions. If you come with one solution fully implemented but the team would prefer another one, you might stick to your (less appreciated) implementation because it’s already done. Or the time you spent implementing is lost because now you have to start again.

Once the problem is broken down, you can even share the implementation work for a large task. Thanks to the plan you outlined, everyone is on the same page and will be alright working on their part.

Not sidetracked until needed

When you are thinking about the what, you don’t need to focus on the features of the language you are using. You’re barely using the language or maybe even using a pseudo-language. You’re less likely to get sidetracked looking up documentation or Stack Overflow, finding an arcane behaviour of the language, framework or library you’re using. You’re focused on the big picture.

When comes the time for the implementation, you can get stuck and confused. But that’s alright, that won’t distract you from the big picture, it has already been solved.

Good naming

As you create your plan to solve a problem, you’ll have to give names to the concepts you’re working with. Because you don’t have the full code to show your team, those names need to explain clearly what is happening if you want them to understand.

Those function or variable names can often replace the need for some comments1. The team will think those names clearly express the intent because they’ve agreed to the plan. And when the code evolves, it is more likely for the names in the code to be updated. We’ve all seen comments being completely out-of-date with the code surrounding them.

Take the time to do it, it’s worth it

Something I thought was a joke as a student is now one of my favourite tools as a more experienced engineer. As a mentor to new developers on a team2, I’ll often ask them to do that work and present it. That often leads to very interesting conversations on the architecture of the project, why we do things in a certain way. You can also easily discuss the benefits and drawbacks of each solution.


  1. I said some comments, not all. 

  2. Juniors or seniors 

★ Si c’est gratuit…

Si c’est gratuit, c’est toi le produit.

Martelé. Jusqu’à l’écœurement. Cette idée d’être consommé pour ce qui nous est propre. Nos données personnelles. Nos déplacements personnels. Nos envies personnelles. Nos possessions personnelles. Pour un graphe impersonnel.

Si c’est gratuit, c’est toi qui produis.

On passe de la consommation à la production. Micro-tâches. Agrégation de travail non rémunéré. Annotations collectives. Liens multiples. Masse critique. Prolétarisation.

Si c’est gratuit, c’est toi qui enseigne.

Sujets d’apprentissage pour intelligences artificielles. Addiction volontaire. Ludification généralisée. Infantilisation éhontée. Transmettre sans comprendre. Déshumanisation de l’autre. De la confiance à la preuve (cache).

Si c’est gratuit, c’est nous qui échangeons.

Réciprocité. Gain mutuel. Le bien commun n’étant plus une production de la société civile mais de l’Homme et de la Machine. Jusqu’à ce que la symbiose soit évaluée comme un parasitisme. Auquel cas, cela pourrait finir par nous coûter cher.

An Advanced Elasticsearch Architecture for High-volume Reindexing

I’ve found a new and funny way to play with Elasticsearch to reindex a production cluster without disturbing our clients. If you haven’t already, you might enjoy what we did last summer reindexing 36 billion documents in 5 days within the same cluster.

Reindexing that cluster was easy because it was not on production yet. Reindexing a whole cluster where regular clients expect to get their data in real time offers new challenges and more problems to solve.

As you can see on the screenshot below, our main bottleneck the first time we reindexed Blackhole, the well named, was the CPU. Having the whole cluster at 100% and a load of 20 is not an option, so we need to find a workaround.

Marvel showing how we're CPU bound

This time, we won’t reindex Blackhole but Blink. Blink stores the data we display in our clients dashboards. We need to reindex them every time we change the mapping to enrich that data and add new feature our clients and colleagues love.

A glimpse at our infrastructure

Blink is a group of 3 clusters built around 27 physical hosts each, having 64GB RAM and 4 core / 8 threads Xeon D-1520. They are small, affordable and disposable hosts. The topology is the same for each cluster:

  • 3 master nodes (2 in our main data center and 1 in our backup data center plus a virtual machine ready to launch in case of major outage)
  • 4 http query nodes (2 in each data center)
  • 20 data nodes (10 in each data center)

The data nodes have 4*800GB SSD drives in RAID0, about 58TB per cluster. The data and nodes are configured with Elasticsearch zones awareness. With 1 replica for each index, that makes sure we have 100% of the data in each data center so we’re crash proof.

Elasticsearch with zone awareness

We didn’t allocate the http query nodes to a specific zone for a reason: we want to use the whole cluster when possible, at the cost of 1.2ms of network latency. From Elasticsearch documentation:

When executing search or GET requests, with shard awareness enabled, Elasticsearch will prefer using local shards — shards in the same awareness group — to execute the request. This is usually faster than crossing racks or awareness zones.

In front of the clusters, we have a layer 7 load balancer made of 2 servers each running Haproxy and holding various virtual IP addresses (VIP). A keepalived ensures the active load balancer holdes the VIP. Each load balancer runs in a different data center for fault tolerance. Haproxy uses the allbackups configuration directive so we access the query nodes in the second data center only when the two first ones are down.

frontend blink_01
 bind 10.10.10.1:9200 
 default_backend be_blink01
backend be_blink01
 balance leastconn
 option allbackups
 option httpchk GET /_cluster/health 
 server esnode01 10.10.10.2:9200 check port 9200 inter 3s fall 3
 server esnode02 10.10.10.3:9200 check port 9200 inter 3s fall 3
 server esnode03 10.10.10.4:9200 check port 9200 inter 3s fall 3 backup
 server esnode04 10.10.10.5:9200 check port 9200 inter 3s fall 3 backup

So our infrastructure diagram becomes:

Elasticsearch with zone awareness and load balancing

In front of the Haproxy, we have an applicative layer called Baldur. Baldur was developed by my colleague Nicolas Bazire to handle multiple versions of a same Elasticsearch index and route queries amongst multiple clusters.

There’s a reason why we had to split the infrastructure in multiple clusters even though they all run the same version of Elasticsearch, the same plugins, and they do exactly the same things. Each cluster supports about 10,000 indices, and 30,000 shards. That’s a lot, and Elasticsearch master nodes have a hard time dealing with so much indexes and shards.

Baldur is both an API and an applicative load balancer built on Nginx with the LUA plugin. It connects to a MySQL database and uses Nginx memory for caching. Baldur was built for 2 reasons:

  • to tell our API the active index for a dashboard
  • to tell our indexers which indexes they should write in, since we manage multiple versions of the same index.

In Elasticsearch, each index has a defined naming:

<mapping version>_<dashboard id>

In Baldur, we use have 2 tables:

The first one is the indexes table with the triplet

id / cluster id / mapping id

That’s how we manage to index into multiple versions of a same index with the ongoing data during the migration process from one mapping to another.

The second table is the reports table with the triplet

client id / report id / active index id

So the API knows which index it should use as active.

Just like the load balancers, Baldur holds a VIP managed by another Keepalived, for fail over.

The infrastructure with the Baldur layer

Using Elasticsearch for fun and profit

Since you know everything you need about our infrastructure, let’s talk about playing with our Elasticsearch cluster the smart way for fun and, indeed, profit.

Elasticseach and our indexes naming allows us to be lazy so we can watch more cute kitten videos on Youtube. To create an index with the right mapping and settings, we use Elasticsearch templates and auto create index patterns.

Every node in the cluster has the following configuration:

action:
 auto_create_index: +<mapping id 1>_*,+<mapping id 2>_*,-*

And we create a template in Elasticsearch for every mapping we need.

PUT /_template/template_<mapping id>
{
  "template": "<mapping id>_*",
  "settings": {
    "number_of_shards": 1
  },
  "mappings": {
    "add some json": "here"
  }
}
  },
  "mappings": {
    "add some json": "here"
  }
}

Every time the indexer tries to write into a not yet existing index, Elasticsearch creates it with the right mapping. That’s the magic.

Except this time, we don’t want to create empty indexes with a single shard as we’re going to copy existing data.

After playing with Elasticsearch for years, we’ve noticed that the best size / shard was about 10GB. This allows faster reallocation and recovery at a cost of more Lucene segments during heavy writing and more frequent optimization.

On Blink, 1,000,000 documents weight about 2GB so we’re creating indexes with 1 shard for each 5 million documents + 1 when the dashboard already has more than 5 million documents.

Before reindexing a client, we run a small script to create the new indexes with the right amount of shards. Here’s a simplified version without error management for your eyes only.

curl -XPUT http://esnode01:9200/<new mapping id>_<dashboard id> -d ‘{ “settings.index.number_of_shards” : ‘$(( $(curl -XGET http://esnode01:9200/<old mapping id>_<dashboard_id>/_count | cut -f 2 -d : | cut -f 1 -d “,”) / 5000000 + 1))’}’

Now we’re able to reindex, except we didn’t solve the CPU issue. That’s where fun things start.

What we’re going to do is to leverage Elasticsearch zone awareness to dedicate a few data nodes to the writing process. You can also add some new nodes if you can’t afford removing a few from your existing cluster, it works exactly the same way.

First, let’s kick out all the indexes from those nodes.

PUT /_cluster/settings
{
 “transient” : {
 “cluster.routing.allocation.exclude._ip” : “<data node 1>,<data node 2>,<data node x>”
 }
}

Elasticsearch then moves all the data from these nodes to the remaining ones. You can also shutdown those nodes and wait for the indexes to recover but you might lose data.

Then, for each node, we edit Elasticsearch configuration to assign these nodes to a new zone called envrack (f$#!ed up in French). We put all these machines in the secondary data center to use the spare http query nodes for the indexing process.

node:
 zone: 'envrack'

Then restart Elasticsearch so it runs with the new configuration.

We don’t want Elasticsearch to allocate the existing indexes to the new zone when we bring back these nodes online, so we update these index settings accordingly.

curl -XPUT http://esmaster01:9200/<old mapping id>_*/_settings -d ‘{
 “routing.allocation.exclude.zone” : “envrack”
}’;

The same way, we don’t want the new indexes to be allocated to the production zones, so we update the creation script.

#!/bin/bash
shards=1
counter=$(curl -XGET http://esnode01:9200/<old mapping id>_<dashboard_id>/_count | cut -f 2 -d : | cut -f 1 -d “,”)
if [ $counter -gt 5000000 ]; then
 shards=$(( $counter / 5000000 + 1 ))
fi
curl -XPUT http://esnode01:9200/<new mapping id>_<dashboard id> -d ‘{
 “settings” : {
 “index.number_of_shards” : ‘$counter’,
 “index.numer_of_replicas” : 0,
 “routing.allocation.exclude.zone” : “barack,chirack”
 }
}

More readable than a oneliner isn’t it?

We don’t add a replica for 2 reasons:

  • The cluster is zone aware and we only have one zone for the reindexing
  • Indexing with a replica means indexing twice, so using twice as much CPU. Adding a replica after indexing is just transferring the data from one host to another.

Indeed, losing a data node means losing data. If you can’t afford reindexing an index multiple times in case of crash, don’t do this and add another zone or allow your new indexes to use the data from the existing zone in the backup data center.

There’s one more thing we want to do before we start indexing.

Since we’ve set the new zone in the secondary data center, we update the http query nodes configuration to make them zone aware so they read the local shards in priority. We do the same with the active nodes so they read their zone first. That way, we can query the passive http query nodes when reading during the reindexing process with little hassle on what the clients access.

In the main data center:

node:
 zone: 'barack'

And in the secondary:

node:
 zone: 'chirack'

Adding a new zone in our infrastructure

It’s now time to reindex.

We first tried to reindex taking the data from our database clusters, but it put them on their knees. We have large databases and our dashboard are made of documents crawled over time, which means large queries on a huge dataset, with random accesses only. In one word: sluggish.

What we’re doing then is copy the existing data, from the old indexes to the new ones, then add the stuff that makes our data richer.

To copy the content of an existing index into a new one, Logstash from Elastic is a convenient tool. It takes the data from a source, transforms it if needed and pushes it into a destination.

Our Logstash configuration are pretty straightforward:

input {
 elasticsearch {
 hosts => [ “esnode0{3,4}” ]
 index => “<old mapping id>_INDEX_ID”
 size => 1000
 scroll => “5m”
 docinfo => true
 }
}
output {
 elasticsearch {
 host => “esdataXX”
 index => “<new mapping id>_INDEX_ID”
 protocol => “http”
 index_type => “%{[@metadata][_type]}”
 document_id => “%{[@metadata][_id]}”
 workers => 10
 }
 stdout {
 codec => dots
 }
}

We can now run Logstash from a host inside the secondary data center.

Here, we:

  • read from the passive http query nodes. Since they’re zone aware, they query the data in the same zone in priority
  • write on the data nodes inside the indexing zone so we won’t load the nodes accessed by our clients

The full infrastructure

Once we’ve done with reindexing a client, we update Baldur to change the active indexes for that client. Then, we add a replica and move the freshly baked indexes inside the production zones.

curl -XPUT http://esnode01:9200/<new mapping id>_<dashboard id>/_settings -d ‘{
 “index.numer_of_replicas” : 1,
 “routing.allocation.exclude.zone” : “envrack”,
 “routing.allocation.include.zone” : “barack,chirack”
}

Now, we’re ready to delete the old indexes for that client.

curl -XDELETE http://esnode01:9200/<old mapping_id>_<dashboard id>

Conclusion

This post doesn’t deal with cluster optimization for massive indexing on purpose. The Web is full of articles on that topic so I decided it didn’t need another one.

What I wanted to show is how we managed to isolate the data within the same cluster so we didn’t disturb our clients. Considering our current infrastructure, building 3 more clusters might have been easier, but it has a double cost we didn’t want to afford.

First, it means doubling the infrastructure, so buying even more servers you won’t use anymore after the reindexing process. And it means buying these servers 1 or 2 months upfront to make sure they’re delivered in time.

I hope you enjoyed reading that post as much as I enjoyed sharing my experience on the topic. If you did, please share it around you, it might be helpful to someone!

★ Senior developer

Defining “senior” is an ongoing and surprisingly difficult process, but we do it because it’s business-critical for us. Without a clear definition of “senior developer", we have no clear path for our own employees to get there. We have no concrete way to evaluate people joining the team, no way to hold ourselves accountable, and no way of improving the process.

The Conjoined Triangles of Senior-Level Development (cache)

There is a moment in your developer career when you wonder if you’re senior enough to depict yourself as a senior developer. This is not at all a matter of how old you are (cache), neither how much you’re being paid. This is more related to how many and diverse experiments you made, how many different peers you helped onboarding a project, how easy it becomes to transmit your knowledge, how much confidence you accumulated and how fast you can admit you’re totally wrong. Actually, this is all about the fluidity you can have with a team within an evolving complex context. That is the moment you realize you are more valuable than the code you produce.

You’re here to speed up the learning process but not too much, otherwise your fellow companions are totally missing the potential failures and are pursuing without accumulating knowledge. Going fast is useful only if everybody within the boat is aware of what has been tried before and what was wrong (and right!) for that particular journey. It can only be achieved with a ton of communication.

When you’re lucky enough to be part of a team of highly skilled developers, you know that everybody will still progress technically because it’s part of the team’s DNA. Besides some long-running trolls, you know that the hard part will not be about technical capabilities anymore, the team is confident enough on that side to learn quickly if necessary. The hard part will be to consider the team — present and future — as a whole. It requires a tremendous amount of empathy to make the right social decisions.

Senior team members should be expected to spend half their time mentoring and helping others on the team get better. Their job isn’t just to be the code hero bottleneck.

Want to be an Engineering Manager? (cache)

Here the important word is bottleneck and I think that better than trying to reach the senior label individually, it has to be gained as a team. It’s way more challenging to be part of something bigger than yourself. You can mesure how “senior” a team is by how good it is at reducing bottlenecks and sharing responsibilities.

Finally, it also creates social problems as well. Bugs that span multiple services and require many changes can languish as multiple teams need to coordinate and synchronize their efforts on fixing things. It can also breed a situation where people don’t feel responsible, and will push as many of the issues onto other teams as possible. When engineers work together in the same codebase, their knowledge of each other and the system itself grows in kind. They’re more willing and capable when working together to tackle problems, as opposed to being the kings and queens of isolated little fiefdoms.

Microservices - Please, don’t (cache)

Choosing carefully which trends you’re following is key. Some are particularly destructive for the social interactions. I already talked about GraphQL, I think that microservices are even worse. This is a particular case when there is so much tensions within the team that you need to separate people and their products to still be able to deliver some value. A senior developer has to be inclusive in his productions and reactions, sometimes at the expense of speed or relevance.

The last step is to write about it. This could be a blog post, a book, or a conference talk. When I write about a topic, I explore the edges of what I know, the edges outside of what I needed to initially implement the idea.

How do I learn? (cache)

One part of becoming a senior developer is to be able to go just a bit deeper than the average developer and be able to share it. That’s a tiny advantage that makes all the difference. Sharing can take many forms, from blogging to giving a presentation or pushing some code on a repository. The end-result is not the most important (except for ego maybe). The moment you dig into the concrete issue and spend some time on it, the process of acquiring that knowledge and being capable of transmitting it. That’s the key point.

We are knowledgeable and productive, yes, but we also understand that we may actually know fewer (useful) things than we did at a prior point in our career. A non-trivial amount of our knowledge has decayed, and we may not have had the time to accumulate enough new knowledge to compensate.

[…]

We realize that it’ll require real effort to just maintain our level proficiency - and without that effort, we could be worse at our jobs in 5 years than we are today.

Reflections of an "Old" Programmer (cache)

The combination of our knowledge decay being extremely fast and our knowledge accumulation rate being quite slow leads to burnouts and endless questioning. Both being quite destructive on the long term. Senior developers are survivors. The ones finding a steady pace in their learning and a clear balance between theory and practice on a day-to-day basis. The ones taking the time to transmit their experience and to be kind enough (cache) to reduce the pain for newcomers. The ones avoiding depression and dead-ends like management and entrepreneurship. The ones escaping the craftsmanship and perfection rabbit holes. The ones considering themselves not senior enough to push the limits of its definitions. What is your one?