At Synthesio, we have recently started to release part of our Ansible deployment stack on Github. It’s the achievement of a 2 years long project. We wanted to do it, but didn’t for many good and bad reasons.
The code was not good enough to be released. That’s the excuse I hear the most from companies that are reluctant to open source their code.
We can’t release that code, it’s crappy and people will think our engineering team sucks.
That’s the wrong way of thinking. Don’t wait for your code to be perfect or you won’t release anything. Push something that might be useful for someone. If people use it, they will contribute and improve your code.
We didn’t have the time to do it. To tell the truth, open sourcing our code was not a priority. We had to deliver fast, fix many things, so doing simple stuff like writing documentation or pushing on Github came second.
The code had Synthesio specific stuff we couldn’t push. That might be the only good reason to keep our code closed for so long. We had to make our code less Synthesio specific by moving things from the core to the configuration. It didn’t take long, and made our code more readable and reusable as our infrastructure is growing. The process is still ongoing and we’ll keep pushing stuff as we clean them.
If you want to open source part of your code, here’s a way to do it without causing a mess or getting crazy.
Split your code source into modules
The first part of the job is splitting your existing code into modules you can release. In this example, we’re releasing our Mesos deployment Ansible role, which used to be in our Ansible stack core.
To do this, we’ll rely on Git submodules. Many people hate submodules, but in our case, that’s the best way to go. We’ll be able to have a separate Git repository on our internal Gitlab we can mirror on Github in the blink of an eye when we update it.
First, create 2 new git repositories:
- One on your internal git infrastructure, we’ll call ansible-mesos-internal
- One on Github, we’ll call infra-ansible-mesos because that’s how we called it.
We want to keep the code on our internal Git infrastructure in case of Github would close or be down someday.
Now, we can actually split the code into modules. Since we don’t want to lose the git history, we’ll use some git tricks to keep that code and its revisions only.
Clone your local repository into a new one:
git clone ./ansible ./ansible-release
Make sure the place is clean before you start working
git checkout master
git reset --hard
for remote in $(git remote); do git remote rm $remote; done
for branch in $(git branch -a | grep -v master); do git branch -D $branch; done
It’s now time to do the real thing:
git filter-branch --tag-name-filter cat --prune-empty --subdirectory-filter roles/users -- --all
You’re now left alone with your mesos role. Good. Some cleaning might be needded before we push to our new repositories.
git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d
git reflog expire --expire=now --all
git gc --aggressive --prune=now
Remove references to your specific / secret code
Our Mesos role used to have some Synthesio internal things we don't want to release, like machine hostnames, or usernames / passwords. To do it, we had to clean Git history so people won't find it while browsing Github.
If you've just deleted that file, it's easy:
git filter-branch -f --index-filter 'git update-index --remove defaults/main.yml' <sha1 of introduction>..HEAD
However, we didn't delete the file, we replaced the data so we need some more git tricks to get rid of the history until the replacement.
First, you need to find the commit SHA1 you replaced your sensitive data with.
git log defaults/main.yml
Now, create a branch ending to that commit
git checkout -b secrets <sha1 of the commit>
git checkout master
git filter-branch -f --index-filter 'git update-index --remove defaults/main.yml' <sha1 of introduction>..secrets
You're done! Your file is still alive but all the sensitive history has been deleted.
Pushing and using
Add a LICENSE and a README file, you're now ready to push your code
git remote add origin ansible-mesos internal
git push -u origin master
git remote add github infra-ansible-mesos
git push -u github master
Now, make use of the newly separated module in your main project. Since we don't want the whole mesos history thing, we'll delete it as well here.
git checkout -b feature/split-mesos
git filter-branch -f — index-filter ‘git update-index — remove roles/mesos’ <sha1 of introduction>..HEAD
git submodule add roles/mesos ansible-mesos-internal
git add .gitmodules
git commit -m "Splitting mesos from the main project"
git push origin feature/split-mesos
Get your commit reviewed, tell your pals to update. TADA! Your company is now a proud open source contributor!
A Technical Guide on Open Sourcing your Code without Pain was originally published in Fred Thoughts on Medium, where people are continuing the conversation by highlighting and responding to this story.