Migrate your ES cluster from one continent to another without downtime

ES migrating

I just migrated a whole Elasticsearch cluster from Canada to France without downtime.

With only 1.8TB of data, the cluster is quite small. However, crossing the ocean on an unreliable network made the process long and hazardous.

My main concern was about downtime: it was not an option. Otherwise I would have shutdown the whole cluster, rsync the data and restarted the Elasticsearch processes.

To avoid downtime, I decided to connect both clusters and rely on Elasticsearch elasticity. It was made possible because this (rather small) cluster relies on unicast for discovery. With unicast discovery, you add a list of node in your Elasticsearch configuration, and you let it discover his pairs. This is something I did once, but not cross continent!

First step is to connect both clusters using unicast. To do this, I’ve added the IP address of the Canadian master nodes to one of the French cluster nodes configuration. I updated both machines firewall rules to they were able to communicate on port 9300, then restarted the Elasticsearch process.

At first, I only launched one French node, the one I planned to communicate with the Canadian one as a gateway. After a few hours of shard relocation, everything was green again, and I just I was able to shutdown the first Canadian data node.

That’s what I launched the 2 other French nodes. They only knew about each other and the gateway node. They did not know anything about the Canadian ones, but it worked like a charm.

If for some reasons you can’t expose your new Elasticsearch cluster, what you can do is adding a http only node you will use as a bridge. Just ensure it can communicate with both clusters by adding 1 IPs of each of their nodes, it works quite well, even with 1 public and 1 private subnets. This gateway provides another advantage: you don’t need to update your clusters configuration to make them discover each other.

ES cluster replication with a gateway

Once again, it took a few hours to relocate the shards within the cluster, but it was still working like a charm, getting his load of reads and writes from the application.

Once the cluster was all green, I could shutdown the second Canadian node, then the third after some relocation madness.

You may have noticed that at that time, routing nodes were still in Canada, and data in France. That’s right. The latest part of it was playing with DNS.

The main ES hostname the application accesses is managed using Amazon Route53. Route53 provides some nice round robin thing so the same A record can point on many IPs or CNAME with a weight system. It’s pretty cool even though it does not provide failover. If one of your nodes crash, it needs to unregister itself from route53.

Configuring route53 for weighted routing

As soon as the data transfer was OK, I was able to update route53, adding 3 new records to route53. Then, I deleted the old records and removed the routing nodes from the cluster. Mission successful.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

Your startup idea is stupid, it won’t work but I don’t have the balls to tell you

Idea!

You just had the best idea ever, you know it. It’s the next big thing. Goodbye Google, adios Facebook, pasta la vista, Twitter! You’re convinced the next world changing idea lies in your hands.

Overexcited, you turn your garage into an office. Waiting to switch full time, every night becomes a new work day, each weekend another week. You lose sleep. Your wife threatens to leave you and go back to her mom and your kids don’t know you anymore. Your idea turns into an obsession and the one thing you think about.

After a few days, you’ve filled more notebooks than Leonardo in his whole life. You’re more careful in building your first prototype than Roy Neary was in sculpting the mountain in Close Encounters of the Third Kind.

You start talking about it around you. You talk it to your mom, your barber, your cat, your colleagues, people at the bus stop. Actually you talk about it to every living beings you pass by, sometimes to your car or your office furniture. You talk about it because you need it and your close family can’t bear it anymore. You also need feedback from people of trust, lots of feedback. It will work, you know it. Everyone around you seems as enthusiastic as you are, they all told you:

Awesome! I love it! I don’t think I’m your target but my friends will adore it!

You start feeling a bit uneasy. You’re not sure they really paid attention to what you were saying. But it’s OK after all. Their friends, their families, they’ll be your clients, and all these people will be your best ambassador to their relatives!

For God’s sake, come down to earth! Here’s what they told you.

Your idea is totally stupid, it won’t work and I don’t give a fuck about it. I want you to leave me alone but I don’t have the balls to tell you.

Don’t lose faith in your idea. You didn’t talk to the right people.

“No one is a prophet in his own country.” (Luc 4,24). Stop trying to get feedback or clients from your relatives, they’re not the droids you’re looking for. Try to find people who have the problem you’re trying to solve instead.

Asking your relative’s opinion is a common trap. It provides instant gratification because they tell you what you want to hear. But your business won’t live on instant gratification.

Take it the other way. Look for people having the same problem as you have. Tell them about your solution. Tell them about your idea. Even better, show them your prototype and get immediate feedback. Do you solve their problem? If not, are they your target? Do you understand your market they way you think you do? And if you do, congratulation, you’ve just found your first users. Iterate, release and come back to them.

The hardest part is coming. It’s time to make money.

This post is a translation of Ton Idée Est Nulle, Ça Ne Marchera Pas, Mais Je N’Ai Pas Le Courage De Te Le dire I first published on 08 Jan 2013 in a series called Startup Life


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

Once you’ve done it, it’s not a challenge anymore

Writing as a challenge

If you like challenging yourself, I bet you’ll be interested with upcoming NaNoWriMo.

For 15 years now, November has been the time of the National Novel Writing Month, a literary challenge where you have 31 days to perform a 50,000 words novel. 50,000 words is about the length of Aldous Huxley’s A Brave New World, and it’s considerer the minimum for a novel. Writing one in a single month is committing yourself to write at least 1667 words of fiction each day, 7 days a week.

If you have ever wanted to write a novel but never started because you were afraid of the task, be happy: NaNoWriMo was created by and for people like you. Last year, the challenge gathered more than 250,000 people, making it de facto the biggest writing challenge in the world.

I’ve tried and won NaNoWriMo 4 times from 2010 to 2013, and I’ve decided no to do it this year. I really recommend you to try it though for you’ll learn a lot of things about yourself.

My first NaNoWriMo was by far the hardest. I had put an incredible an useless pressure on myself as I could not imagine losing. I wrote the 50,000 words in 19 days but it left me physically and mentally drained for months. After November the 20th, I was unable to open my editor without being about to vomit, and it wasn’t because of some really trash scenes.

This first victory was very important, if not elegant. When you give yourself a challenge it’s important to get the appropriate result. If your challenge is to write a 50,000 words novel and you do it, your challenge is successful. Don’t blame yourself for not finishing your story or having 200 pages of pure unreadable crap. Finishing your story or writing the next Nobel prize were not the challenge you initially setup.

The 2 following years were much easier despite starting badly. Impostor syndrome striking back, I had spent 2011 convinced winning NaNoWriMo was pure luck. I did not put myself as much pressure as the year before and it helped me a lot. Truth was I had nothing to prove to myself or anyone anymore.

That second victory was not as important as the first one since I had already won the first time. When giving yourself a challenge it’s important and helpful to mark multiple stages so you get that feeling of achievement from time to time. A 31 days challenge without those stages can look like a long tunnel with no end, and it can be discouraging. Most people losing around me lost because they could not keep with writing daily, even a few hundreds words so they eventually stopped on day 20 or 25. The reason why they broke their writing pace was because the challenge was too big. It would have been easier for them sliced in small sub challenges.

It seemed like winning NaNoWriMo was becoming a nice yearly habit. Until last year where it was painful again. I had a hard time finishing, hated what I was writing about and spent every single day hoping for December. I didn’t care about NaNoWriMo anymore because it was no more the challenge it used to be.

Every year came with its lessons and 2013 had the most important one. Whatever you add to make it harder, whatever the way you do it for the second, third or fourth time, a challenge is a once in a lifetime achievement. Once you’ve done it, it’s not a challenge anymore.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

Données et secret

Je fais le serment de remplir mes fonctions avec conscience, indépendance, et humanité.
Je m’engage à suivre les standards du web, de la qualité et de l’accessibilité pour que le web reste universel, neutre, libre et ouvert.
Je m’engage à respecter et protéger le secret dû aux données personnelles et à la vie privée dont j’aurai connaissance dans l’exécution de mon travail.
Je suis un travailleur du web, j’en suis fier, et j’assumerai mon rôle avec dignité.

Le serment du Beffroi de Montrouge

Cela fait quelques jours que ce serment tourne sur Twitter et j’ai du mal. J’ai donc fini par regarder la vidéo et derrière l’enrobage un peu pompeux et les assertions à faire sourire Miss France, il est surtout question de données personnelles (des autres). J’ai pas mal réfléchi à la question et j’en suis malheureusement arrivé à la conclusion qu’il était illusoire de prétendre avoir le moindre contrôle là-dessus. Les données stockées sur mon ordinateur sont incontrôlables. Je ne sais pas qui y a accès et où elles sont envoyées. Et vous ne savez pas non plus. À moins que :

  • votre ordinateur soit déconnecté ET protégé de tout réseau ET inaccessible physiquement ;
  • vous ayez construit votre ordinateur ainsi que le firmware de chaque puce ;
  • vous ayez pu passer en revue chaque logiciel installé ET chaque mise à jour ET chaque moyen pour mettre à jour ce logiciel.

La liste est bien plus longue mais ces 3 items montrent déjà l’ampleur de la tâche. Aussi « Je m’engage à respecter et protéger le secret dû aux données personnelles et à la vie privée dont j’aurai connaissance dans l’exécution de mon travail. » est un vœux pieux qui ne pourra être respecté. Ou alors il faut être explicite sur les mesures prises pour essayer d’honorer cette déclaration. Par exemple :

  • je m’engage à ne pas effectuer de sauvegardes distantes de ces données en vous éduquant sur ce que cela implique en cas de crash ;
  • je m’engage à chiffrer ces données lorsqu’elles ne sont pas utilisées quotidiennement ET à les supprimer après 30 jours sans utilisation ;
  • je m’engage à ne pas transmettre intentionnellement ces données à un tiers ET à restreindre l’accès physique à ma machine ;
  • je m’engage à vous avertir immédiatement en cas d’accès non autorisé à vos données OU de leur transmission (indépendante de ma volonté) sur un quelconque réseau.

Ce sont les engagements que je m’efforce de tenir au niveau pro avec plus ou moins de succès. C’est faible mais c’est bien au-delà de ce qui est pratiqué habituellement.

Si le problème est un problème d’image et de confiance, je ne vois pas trop ce qu’un serment pourrais nous apporter en matière de crédibilité. Et ce même s’il était signé par — soyons fous — 80% de la profession. La confiance se construit avec de la communication et de l’éducation. Si notre image est actuellement mise à mal à cause des données personnelles qui nous sont confiées : communiquons mieux, éduquons plus.

Would you invest 1 million in your next employer?

I will follow him

Every time I meet a potential employer, I ask myself this dead simple question:

If I had 1 million to invest for the 5 next years, would I bet on that guy regardless of the product and team?

If you’re working for a government or a large company, this question does not apply to you. Despite a few corporate missions, I’ve always worked for 40 people max young companies. When you join such a venture, you soon realize your boss is at least as much important as your closest colleagues.

To answer that question, I make my future boss pass the mountain guide test.

I love hiking in the mountain. I can walk alone for days, not talking to anyone, and I can hire a guide to discover new places. Sometimes the walk is easy and sometimes it’s incredibly hard. Your legs hurt like hell. The backpack braces cut your shoulders. And every single step becomes a challenge harder than climbing the whole mountain.

In difficult times, I expect the guide to be the one who awakens the group and takes it to the next summit.

That’s what I expect from my future boss. I expect a startup CEO to show me the highest summit and tell me:

Look, this is where we’re going. I don’t know the path yet, I don’t know how hard it will be but that’s where I want you to come with me.

Indeed, it’s not the only thing to weight before you join a company. You need to check the team, the market, the product, the technology, the company core values. But it’s a crucial one.

If all lights green but I find myself unable to share his vision and passion. If I feel he’s not a true leader who’ll open the way to a world changing adventure, then I won’t invest my million and follow him. It’s a no go.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

How to bring your company to open source in spite of it in 5 easy steps

Open Source ninjas

Most if not all IT related companies use or rely on open source software, but only a tiny part of them contributes back.

Most of the time, reasons for not contributing are lack of time, packaging and maintenance code, necessity to create a great code to show the company technical skills, or fear to empower the competition.

These are fallacious reasons but they show your company won’t contribute to open source projects unless you help it in spite of it.

There’s something you can do for open source without releasing the company’s code, taking the time to package it or build awesome, 10 people reviewed libs. You can improve the free or open source software you’re using when you need it.

  1. Create a Github organization for your company. It’s free and you don’t need to be an official to do it. Name it yourcompany-labs so you leave your company officially go on Github someday. Name it something else more or less related if you fear to be kicked in the butt for this, you can rename it later.
  2. When you find a bug or a small missing feature in a piece of code you use, don’t fix it and keep it to yourself. Fork the project under your company’s organization.
  3. Create an issue upstream. Document it well. You want your open source contribution to mirror your company’s technical skills, and writing good documentation is part of it.
  4. Don’t wait for someone to fix your bug or code that missing feature. Do it, and create a pull request from it. You don’t even need to be a badass in the language the project uses. My first contribution to Ansible was also my very first experience with Python.
  5. Embed your colleagues who’re like minded so they can do the same.

Indeed you can contribute and push that code under your own name. It’s easier and you (probably) won’t be yelled at for wasting the company’s resources. Indeed, you’ll have to trick your company and it may end badly, so the best way to do it is bringing the management with you and show how that no cost operation can have a very positive effect on the company’s reputation in the tech field, and furthermore in recruitments.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

This is the support team you're looking for

A great support team

Doing a postmortem analysis of the crisis you’ve faced is not only a great way to understand what happened so it never happens again. It’s also the best way I know to improve crisis response management for the next time.

Because there’s always a next time.

Looking back at all the crises I’ve faced in the past 10 years, what I’ve missed the most to solve them efficiently was an awesome support team.

An awesome support is not (only) the one that answers you Saturday 9:00 PM and replaces your son’s Kindle within 18 hours so he can read during his vacation. An awesome support is also the front line that manages the client crisis side while you’re working.

The awesome support team is product and tech savvy, composed half of people with great empathy and engineers. It removes pressure from the crisis solving team both from the inside and the outside by filtering new tickets, understanding if they’re related to the problem you’re facing or something completely unrelated.

The awesome support team comes with the awesome product, not after it’s released. And as a front line team, it deserves its part of honor after the battle when the medals are given.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

The GPG Suite for Mac is not free anymore (and the sick sad future of personal cryptography)

Padlock

In An Open Letter About Our Future, the GPG Tools team announces the future versions of their software won’t be free anymore. The GPG Suit is the best personal PGP graphical interface on Mac OS X, and the only one that integrates well with Mail.

I understand the GPG Tools team position and I’ll even pay the price for a good personal cryptography experience. Since I’ve started using Max OS X 10.10 beta a few months ago, I’ve been using nerdy workarounds to keep using PGP with Mail. This is not the experience I want.

15 years ago, I had that no software but free software state of mind. I spent nights and days compiling and configuring poorly integrated software to make them work together and ensure a decent user experience. A young know it all, I considered code had to be free both as in freedom and free beer before everything else.

At 36, I’ve switched to a no software but good software (better if free) state of mind. I want to do things with my computer instead of doing things for my computer. I’ve been working in the software world for too long too understand the value of code and the value of a good user experience, and to pay for it or donate to my favorite project.

I understand the move of the GPG Suit team, but I strongly disagree with it.

I’m concerned it makes a precedent and becomes a major step back for the global personal cryptography use outside of the computer science sphere.

For a few years personal data protection has been a major concern. Wikileaks and Edward Snowden revelations have raised awareness of the data privacy situation outside of the nerdy and global conspiracy spheres. It raised an interest in what corporations and governments actually do with our communications and data far beyond the usual security scene. If it did not really change anything, at least people can’t say we didn’t know anymore.

However, I don’t feel like the use of personal cryptography has improved in any way in the general public neither for personal nor professional use.

Despite some user experience improvement, using personal cryptography is still a pain in the ass. You need to understand the basics of applied cryptography: public / secret key, key exchange, signature, expiry date… Signing or encrypting an email, connecting to a VPN still needs you to add some operations to your usual, simple workflow.

And no one wants you to use personal cryptography anyway. Your government doesn’t. Your ISP doesn’t. Your employer doesn’t.

A strong example is about man in the middle attack (MITM). MITM is a technique where an attacker intercepts a data stream, spoofing the emitter he’s the legitimate receiver before letting it go. In a corporate environment, MITM works even on encrypted trafic.

15 years ago, man in the middle was a strong little known attack. Today, it has turned into a corporate security measure. That’s why no one wants you to use encryption, and that’s why Google giving a better rank to SSL using sites makes corporate ITs crazy.

Personal cryptography is a pain, so no one wants to use it. Having to pay for it won’t improve anything. If we want a strong adoption, personal cryptography must be free, and being free is part of a good general user experience: it removes adoption friction.

12 years ago, I was doing lots of CSS. Internet Explorer 6 had 90% of the global market, people used table based layouts and Web standards was a geek thing.

Hopefully, Firefox came, and spread. Web standards started to spread too and most people quickly stopped using table based layouts. It was a hard time for evangelization, and even in 2006 I still had to develop for IE6, tables and inline style.

Geeks started to install Firefox on their friend’s, parents and school computers. They were able to do it because it was free, both as in free beer and free speech. What would have happened if they had to pay a small fee to use Firefox? It would have known the same fate as advertisement powered Opera.

This was made possible because despite many people working full time on Firefox, they found another way (I.E. Google) for funding. As Truecrypt is dead, we need more general public level cryptographic projects, and we need them to be free. It’s a question of freedom.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.

Enseignement et acquisition

Easier to ask for forgiveness than permission. This common Python coding style assumes the existence of valid keys or attributes and catches exceptions if the assumption proves false. This clean and fast style is characterized by the presence of many try and except statements. The technique contrasts with the LBYL style common to many other languages such as C.

EAFP

J’ai appris qu’il y avait eu une suite à l’intervention de SudWeb au sujet de l’enseignement de l’intégration lors de ParisWeb (merci Boris !). On peut dire que ça tombe au bon moment. Tout cela m’amène à faire un parallèle entre des paradigmes de langages de programmation (EAFP vs. LBYL) et l’opposition pratique vs. théorique que l’on rencontre forcément lorsque l’on souhaite transmettre ses connaissances. À quel point faut-il prévenir plutôt que guérir ? Qu’est-ce qui est le plus formateur ?

J’ai pour l’instant pris l’option très expérimentale : produisez, je vous corrige. Et j’espère bien arriver jusqu’à un point où cela deviendra : produisez, corrigez-vous ! Ce vous correspondant au groupe et aux connaissances accessibles en ligne. Mais je suis tiraillé. Ces étudiants ont la chance d’avoir une formation et je leur propose de devenir autodidactes. Est-ce que je ne les prive pas ainsi d’une théorie qui m’a manquée pour pouvoir progresser plus rapidement il y a 10 ans ? Est-ce que les trentenaires du Web (huhuhu) ne se cachent pas derrière cette mise en pratique car ils n’ont connu que ça ?

Et puis je me raccroche à la permissivité du Web, à cette inconsistence inscrite dans son ADN, à ces paquets qui errent entre 2 continents avec l’espoir d’arriver quelque part. J’imagine ces étudiants qui souhaitent à tout prix être autonomes alors qu’il va leur falloir apprendre à faire ensemble. Qui veulent appliquer de la bonne pratique sans forcément en comprendre le sens et le besoin. Je les observe, perdus mais volontaires, et je garde espoir. Ils ont encore le temps pour faire des erreurs et l’énergie pour se relever.

It's a book about a corporation family kid who starts his own company

Jonathan Livingston Seagull

It took me 20 years to understand why I was offered Jonathan Livingston Seagull for Xmas 1991. My parents had been giving me one book every year ever since before I could read, but it was the first time my mother wrote something on the first page.

Xmas 1991. This book and Jonathan look like you.

If I wanted to completely ruin the plot, I’d say Jonathan is a book about a seagull who want to fly.

It’s much more than this.

It was my first book about entrepreneurship and startups. It’s a book about someone who decide to follow his dreams despite everybody around him saying he’s a fool. He focuses on his dream and eventually makes it a reality, inspiring people around him to do the same.

It’s the book every teenager should read if they don’t feel like being like everyone else around them. If you’re a non conformist, it will help you passing those difficult years even though you won’t realize it.

Last week, something happened in my boy’s class. He just had a 17/20 and was given a chance to change his mark for the better or for the worst. His friends told him 17/20 was good enough and he should stop. He didn’t and scored 20/20. After that, his best friend told him:

You were right not to listen to what everybody said. If you do what everybody tells you to do, you’ll become like everybody. Don’t.

Telling me that story with the fear I was about to yell at him for endangering a good mark, he didn’t realize how proud of him I was. He knew it was something important and I can’t wait for him to grow up and read that book.

This book is not only about corporate family kid who starts his own company.

It’s a book about freedom.


Cet article a été publié par Frédéric de Villamil sur Le Rayon UX | Si vous l'avez lu ailleurs sans qu'un lien ait été fait vers l'article original, c'est qu'il a été reproduit illégalement.