Creativity is an important part of life. For technical work it’s important as well. Such as the field of data I work in. I believe that putting some restrictions down will improve creativity, and therefore, productivity and enjoyment at work.

Examples where putting barriers and restricting movement creates better performance is, for example, South Park. The show is funny because of legal restrictions. No, the lawyers would say, you can’t say this or that about a person or company, they’ll sue us! And the team continues to twist and turn until a new joke appears that won’t cause legal action and is often funnier than the original.

This creativity by limitation we find in tech. I like the fast-inverse square root of id Software they used in Quake.1 Working with limits of computers at the time; they had to do something clever. The fact people try to run DOOM (also id Software!) on anything with a primitive input and output interface as a challenge of their own creativity and knowledge.2

Considering data software and cloud platforms, such as AWS, Azure and GCP, I wonder if the same is true. Confinement to the place and systems available, your creativity gets stimulated to help develop unique solutions. It’s a puzzle with finite pieces instead of infinite “maybe” ones.

Places like Netflix, Uber, Twitter, Google and anything else massive have these restrictions as well, their limits being the actual limits of tools and systems available. They’re natural limits, not artificially imposed. Google faced unseen amounts of data. They couldn’t just turn to a vendor to fix their issues, they had their creative minds come up with a solution! (Such as distributed file systems and distributed compute.)

If any piece of software on the market is on the table, we forget our own creativity and just try to find the next shiny thing that fulfills our current need. As soon as the next challenge appears, instead of learning our tools, having built creative solutions and creative minds, we hunt for the next fast-food-like fix for our architecture.

For quite some time, I built only on the Azure platform with ADF and Azure SQL DB. Azure because my employer was Microsoft-only. ADF and Azure SQL DB because they were familiar with our target audience. There were many reasons to switch to “something with Python” to increase efficiency in some programming challenges, but down the road I’m glad we didn’t. We ended up with a simple cohesive platform resulted.

It was very rewarding to build something so valuable with a few basic elements!

Don’t underestimate the benefit of simplicity. This setup was quite simple, but it was easy to onboard new developers and to expand ideas. Not for a minute we had to wonder which vendor might solve our issues, it was just us and our tools. A new tool brings unknown-unknowns and new challenges. First you learn the tool, then you integrate it into your architecture, and then maintain it for a while.

There are no temporary solutions, especially within enterprises. If it works, it works, and will continue to be leveraged until the cost to maintain excessively outweighs the benefits. The cumbersome legacy component that kind of does the job — we all know these. The simpler things are, the easier they’re replaced.

Complexity always exists. It’s just the question of where we will put it. The “high value” vendors who manage a lot of complexity are rarely lean in operation and support. Lean solutions seldom cover all your complexities. Lean solutions, however, allow you to be more efficient and master and simplify the complexity you have to deal with. Think of the cost-quality-time triangle.

In my case the complexity was supporting different source systems into a single canonical data model. About five financial software vendors and five HR software vendors. That meant dealing with intricacies of each system, different configurations and usage for each client. Then we had to unify them. It took several sessions in front of a whiteboard, but instead of buying middleware that might fit, we created a solution that perfectly suits our use case.

Now that I’m in a new environment with a different work ethic, I notice this employer does not like constraints. Everything that is even remotely is difficult, we petition the software vendors for a new platform to add to the architecture. It works, but it feels convoluted and the short term easy way out. I’m not against buying the proper tools, but buying shouldn’t be first on the list.

It’s not about how many tools you have, it’s how you use them. And maybe which ones you won’t use?

  1. Understanding Quake’s Fast Inverse Square Root ↩

  2. It Runs Doom showcases all the systems that strange people have managed to get to run the popular 1993 action game Doom ↩