I'm Thomas Schoffelen, edtech entrepreneur, co-founder of Includable and lead developer for Hoy and Street Art Cities. Before that, I co-founded NearSt.

I do some user interface design, and write occasionally.

Recent Dribbble shots →

Recent writings →

2018/07 – Microservice Forensics 101

Recently we were brought in by a customer to investigate a problem with their microservices architecture where specific transactions weren’t yielding the expected results. Since some of their microservices heavily rely on one another, it wasn’t immediately clear which one of them was the one misbehaving.

After a few days of combing through their code, we finally found the culprit. These are some of the lessons we learned while debugging.

1. Run it locally

With a microservices infrastructure where at least a dozen docker containers are required to debug a specific part of a transaction, you might be inclined to delay setting up a local development environment as long as possible. Sure, it’s a hassle, but being able to make changes directly in the code and SSH’ing directly into boxes to debug and see logs makes it so much easier to actually reproduce a transaction in a way that lets you find the root cause of the problem. In the end, it’s much faster than only poking around in production logs and going through code.

2. One service at a time

Don’t try to understand what the full lifecycle of a transaction is through the myriad of API calls that connect microservices to each other. Instead, go for a one-by-one approach. Understand what a certain service does with your input data, look at the logs for that one service when you replay the transaction, verify the output contains what you would expect it to contain, and move on to the next service.

In cases where microservices don’t talk directly to each other but use message busses or queues, make sure to shut down the consumer service so that you have time to inspect the messages being put into the queue before they are being processed. That goes more or less back to the first point, since you probably shouldn’t be shutting down services in the production environment 🙃.

3. Have a central place for logs

Having multiple places where production services send their log files is not only really annoying, but might also make it very hard to get a full picture of what happened to a certain transaction in the different stages of it being processed.

Also, save logs of all severities. Nothing is more painful than realising you’re missing an important part of the puzzle because you didn’t save verbose/debug logs. Or, better yet: build a mechanism to temporarily enable debug-level logs for all production services when necessary.

4. Follow the code

It is really easy to be led astray by confusing or vague log messages. Make sure you understand what code gets executed when you send a certain request to a service, and only then run it and explore the logs. That way you know exactly what to look out for, as well as where potential pain points in the code might be located.

5. Write tests

When you finally have found the cause of your issue, don’t just patch it and go on with your life, but write a few unit or integration tests to make sure that if someone else breaks the same thing again, they won’t have to waste as many hours as you just did on finding out what they did wrong.

2018/06 – A new economy of work, on the blockchain

How The Bounties Network and Gitcoin are working towards a new way of working.

I’ve spend my whole — albeit short — life working for myself, either as a freelancer or contractor, or building startups, or more recently running a development agency. I got my first paying gig as a development contractor when I was just 12, by someone who was crazy enough to trust me to build a website for the event they were organising.

I’ve never really needed to search for work, but there have been times where work was slightly more scarce than I wanted it to be. In such periods it sucks to be a contractor, because your choices are more or less limited to cold calling or emailing potential customers, or waiting. There’s of course these contracting websites like Upwork or Fiverr, but those are crowded with people that are way more experienced in using those platforms, and margins are very small.


Recently, however, I found an alternative: Gitcoin. Nope, it’s not another cryptocurrency, even though it sounds like that. Instead, it is a way for blockchain projects to outsource work to the community in the form of bounties, which are basically Github issues with some form of reward attached to them. Not only is that a great, low-barrier, way to make a quick buck, but it’s also a great way to get involved in the blockchain space as a developer, something I was having a hard time getting into as a developer that used to be mainly focussed on mobile and web.

StandardBounties and The Bounties Network

I spent some evenings the last few weeks completing some small tasks through Gitcoin, and in the process met a few of the people behind some very cool blockchain projects. Then, this week, I learned that Gitcoin is just part of something much bigger: StandardBounties, a standard for issuing bounties for work on the Ethereum platform.

From the “Rationale” section of the Github:

Ethereum smart contracts can trivially facilitate transactions of resources (or tokens) between individuals or groups, but service transactions are more complex. Requesting individuals (issuers) must first approve the work they’re receiving before paying out funds, meaning bounty hunters must have trust that the issuer will pay them in the spirit of the original contract.
The StandardBounties.sol contract facilitates transactions on qualitative data (“deliverables”), allowing bounty issuers to systematically approve work received.

StandardBounties is an initiative of The Bounties Network, an open platform for Ethereum-based bounties. Using their beta web interface you can find bounties across all of the Ethereum network, not just from Gitcoin, but from every bounty system that implements the StandardBounties standard!


This new network and open standard makes it possible for any type of developer (not just those that can code Solidity contracts!) to have a consistent source of work whenever necessary. I’m super excited about the possibilities of this, from students being able to make rent without having to go deep into debt, to startup founders that need to make some money on the side while their core business grows.

Are you willing to give it a try?

2018/04 – The Debrief: RChain Developer Conference

Earlier this month, some members of the Includable and Moonshot.Express teams were invited to Boulder, a small town near Denver, CO, to join the RChain Developer Conference. There, the team behind RChain showed off their progress in the development of a new blockchain platform that promises near-infinite scalability by using asynchronous smart contracts and a new programming language called Rholang.

We came there with not much more knowledge about the RChain platform than the very short executive summary I had prepared for the team, but that didn’t stop us from quickly getting the development environment for writing smart contracts set up and even building our own tool to test smart contracts online after the first day of the conference. More on that later.

Without further ado, here are some of our main takeaways from the conference.

The flight to Mercury

Musk might have set his sights on Mars, but the RChain community is currently focussing on Mercury. That’s the code name for the first major release of the RChain platform (later releases are named Venus and Earth). As much of excitement there is for this release, which is currently planned for the end of this year, there’s has also been quite a bit of unclarity about what is and what isn’t included in this release.

The deployment of the first testnet, which will allow you to actually test smart contracts working on multiple nodes is planned for the start of this summer, with a first version of the communications layer and namespace logic being finished right now.

Then in December, the launch of the RChain main net is planned, which will allow developers to actually deploy their first real dApps on the RChain platform.

A global roadmap for the Mercury release can be found on the RChain Confluence documentation site, as well as very detailed briefings for the minor releases (0.3, 0.4, etc.) that are scheduled for the next months.


Applications for the RChain network are built in a new language called Rholang, which implements Rho Calculus, an extension of Pi Calculus. Pi Calculus is quite a unique way of building processes that run in parallel (think Javascript async/await, but for statement you write).

A great way to learn about it is using JsonPi, a sample implementation of Pi Calculus by Glen Braun, with some great documentation on how the basics of it work.

Currently trying out Rholang itself requires you to boot up a RChain node (a server on the blockchain) using Docker, which can be slightly tedious, especially on Windows machines. That’s why our team (lead by Ahsan Fazal and myself) immediately sprung into action to build a simple web-based tool to execute Smart Contracts (programs) written in Rholang. You can find it on rchain.cloud. There is a link to the Github repository for the project there as well, so feel free to contribute to it if you feel like it!


The community behind RChain is quickly growing, and it is quite clear why: it is a blockchain platform that is built on solid theory, and the product of more than ten years of research by the founders.

Excitement about the possibilities is very great, and although it will take a while before the platform is built out far enough to be able to actually get started on building those apps that people in the RChain community are fantasising about, but there is a pretty tight-woven group of people both inside the organisation and big group of volunteers that try to get to this reality as fast as possible.

One of the ways in which this is being achieved is through the RChain bounty program, which can be joined by everyone, and is a great way to collaborate on this awesome project while also making some bucks.


It is also great to see that blockchain platforms can comfortably live next to each other (for now). We met some great Ethereum stakeholders at the conference as well!

Extending in Europe

Includable is supporting Inblockio in getting RChain excitement going in Europe. To that effect we’re working on developing learning tools, as well as organising community meetups in The Netherlands and Germany. The first one will be held soon in Amsterdam, so stay tuned!

Back to Boulder

For me, it was the second time being in Boulder. It’s a beautiful small town, in a valley just 10 minutes from the Rocky Mountains by car. We went there two years ago during Boulder Startup Week to show some officials from my hometown, Heerlen, what a city of that size could be in terms of supporting startups and creating a Silicon Valley-like environment (the good parts!).

Boulder Startup Week starts on the 14th of May this year, so if you have the opportunity to drop by, I would certainly recommend it!

2018/04 – Tutorial: setting up Continuous integration with Includable

It is now super easy to set up continuous integration for projects running on the Includable platform. In this tutorial we’ll show you how to use Travis CI to build static assets and deploy the result to Includable.

The basics

First make sure you have the latest version of the Includable CLI installed:
npm i -g includable

Then, you can start creating your first Includable module: inc create. Let’s go for the svelte starter pack. Svelte is a super cool front-end framework that is compiled beforehand, resulting in a super tiny javascript file for deployment. By default, this starter kit will require the developer to compile their assets themselves (happens automatically when running inc run) and checks those into their Git repo.

A cleaner solution would be to automatically create those when a new change is committed, and automatically deploying those assets to the Includable Platform. In order to do that, we’ll use Travis CI, one of the many available continuous integration solutions (some others to check out are Circle CI and Codeship).

Updating Gitignore

First off, we want to make sure that compiled static assets are never checked into version control, since they will be created before deployment by our Travis CI script.

In order to do so, we add the following line to .gitignore:


Creating Travis Config

Every CI solution has their own way of configuring what scripts and tests to run. Travis CI requires you to add a file called .travis.yml to your repo. Let’s add that with the following contents:


You can now create you initial commit, and push your code to a public Github repository in the usual way.

Getting started with Travis CI

The only thing left is to tell Travis CI to build your project. For that, simply sign in to travis-ci.org with your Github account.

You can then flip the switch next to the repository you just created to trigger your first build. What you need to do, though, to make sure the build succeeds, is add your Includable token, otherwise Includable won’t know if you are authorized to update your module or not.

You can get your token by going back to your command line, and running inc login --token . The value you will then get back you can add to your environment variables in the Travis CI settings:


That should do it, you have now set up a continuous integration flow to automatically build and publish any changes you push to your Github repository automatically to Includable.

If you need any help, feel free to ping me @tschoffelen on Twitter.

2018/03 – FISIM

(“Fuck it, ship it”-mentality)

Investors (and startups looking for investment) know this as no other: the team is what makes or breaks a tech company, or any company for that matter. Part of that is team culture, team productivity, but also the mentality you have as a team when it comes to products.

I am somewhat of a perfectionist, most of the time. And I work a lot with people to which that label would also stick quite easily. I’ve learned however, a long time ago, that when it comes to tech, there usually isn’t really a “perfect” version of a product. There will always be features to add, bugs to squash, user experiences to improve, not to speak of quality of code, unit testing coverage, integration tests, continuous integration scripts and configuration. A tech product simply is never done. If you feel like it is, you either succeeded in building a really great MVP (which is also quite rare), or you haven’t delved deeply enough into your code base for a while.

That “never being done“ makes it super hard for some teams to actually release. The mantra “release early, release often” sounds so great on paper, and almost nobody will disagree with that basic premise, but when it comes to releasing the first version of the product, it often sounds much easier than it is, because:

  1. The experience isn’t great enough. Are people going to get how this app/website/tool even works?
  2. We’re still missing features X and Y. Will people want to use the product without those?
  3. There’s this bug, that’s really hard to reproduce. How many people will experience that when using the product?

Note that all of those usually lead back to a certain question or uncertainty. Many of them will be about product-market fit. Experience shows that the only way to really learn if that exists is to let people play with your product.

Also, users on the internet know how to make their voices heard. If they have an opinion, they will definitely find a way to tell you about it. So stop guessing, stop spending hours on perfecting your app or fixing your style sheet, and get the first version of your product out there. If you’re proud of it, you’ve waited too long. If you’re proud of that by launching, you’ve just created a way to validate your assumptions, you’re on the right track!

Make sure you, and all of your team are on the same track, knowing that striving for perfection is important, but validating your idea is more important, so just fuck it, ship it 🚀.

2018/03 – A hypocrite that turned 21

I feel so alone sometimes. Not really lonely, just alone. And when I feel that way, I’m being so unfair to myself. Why? Because I’m anything but alone.

It’s almost 5:00 in the morning as I’m writing this, on the day of my birth, the eleventh of March. I turned twenty-one about five hours ago. Just before that, at about 11:20 at night, on the tenth of March, I suddenly, randomly, got a text from a dear old friend, telling me that she is in a bar in Amsterdam, actually no more than 5 minutes from my house. By 11:40 I was there, I met her friends, at at midnight they all congratulated me. I was supposed to finish some work for a client of mine (if you’re reading this, mea culpa!), but instead I spent the last five hours having fun (and consuming alcohol) with that group of people.

Why am I awake after all of that? A normal person would have thrown their body into bed as soon as they entered their apartment, and took a few winks before even attempting to write anything Medium-worthy after such a night. Well, the reason: in a few hours I’ll be flying to London, not only to have fun there with my Dutch friends who already flew over yesterday, but also to have birthday drinks with some of my London friends later in the day.

Therefore, the only two hours of today that I’ll be spending alone will be the two or three hours between now and when I leave for the airport to catch my flight, together with the friend and business partner that I’ve been working together with for over six years now. So, the next time I feel alone (or, for that matter, lonely), let this post be a reminder of the hypocrite asshole I am for feeling that way. I’m so fortunate to have all of those people in my life! Just like real hunger, I’ve probably actually never experience the feeling of really being alone in this world.

Also, to future (sober) Thomas: happy birthday! Perhaps you might want to delete this post. Or, at the very least, properly check it for grammar and spelling mistakes!

2017/12 – Friday night

Quit Gmail. Quit my project management application, the one with the bars that present the hours you’ve worked this week. Quit those folders with the source code of those client project. Quit all those browser tab. Why do you even still have a Facebook account? Fuck off. Open up the top buttons of you jacket. Maybe even get some junkfood on the way home, you’re drunk anyways. And you can’t count calories while you’re drunk, right?

It’s quite interesting, that winding down, just after midnight on a Friday, Saturday technically, after all the stress of the week is gone, after all the meetings are done, the emails are sent. Open up that list of movies you still want to see, look up which of them can be found on Netflix. End up at The Pirate Bay because none of them can be found elsewhere.

Open up that brightly coloured box with ‘Chicken McNuggets’ printed in big bold letters on the top. You can’t count calories while you’re drunk.

Press play.

Sink away.

Tomorrow, tomorrow, tomorrow is a new day.

And thankfully not a weekday.