Building Psithur Connect
When I set-out to build Psithur Connect, I wanted to build a product that delighted its users and made them feel as if there was a little bit of magic at their fingertips.
Co-founding Psithur
I never studied computer science; my formal education is in Economics where I learned the concepts of game-theory, behavioural economics, cost benefit analysis and more differential maths and matrix algebra than I’d care to admit. While I no longer work in the field, my Economics degree continues to serve me: Economics is a useful framework for thinking about the world and complex systems: and software engineering certainly requires a healthy dose of cost-benefit analysis and systems thinking!
Everything I’ve learned in software engineering, I’ve learned through trial and error, a fair amount of luck and from listening to the wisdom of others. I hope to never stop learning, but like anything, learning needs to be practised.
I started writing software 20 years ago in high school - where I taught myself C++. Since then, I’ve continued to learn new programming languages while working as a research economist, data scientist, founder, data engineer, software architect and software engineer.
The longer I spend working on technical and data-focussed systems the more I appreciate the perspective gained from having a sense of the costs and benefits of various technologies and technology choices.
Psithur is the continuation of many years of learning how to navigate these trade-offs in order to achieve systems that deliver real value and help organisations solve real problems.
Gardening
My favourite analogy for software development is gardening: they share many similarities.
A landscape gardener is like a software architect; she takes her client’s desire for a beautiful and functional garden with various features. Using her knowledge of the climate and the soil, she examines the landscape and selects and sites suitable plants. Some gardens might need a lot of preparatory work, some only a little, but chances are once the landscape gardener is paid and gone, the garden is anything but complete.
Aside from regular weeding, watering and pruning, over time the garden will change. Things that might have seemed neat and tidy will grow and become messy. Plants may die, some will flourish out of control. If the garden is not maintained it becomes wild and loses its function. Cared for - however - a garden improves and may have new function over time. Gardening without the right tools is ineffective and inefficient. A gardener caring for a garden needs patience, stamina, perseverance and perspective. Reflection is as important as action.
I started working on Psithur Connect (at the time called Symplast) in August 2020, and I wasn't confident that I could build something to match the vision Jason and I shared. I hadn’t built something of this scale or complexity with Clojure before and I didn’t know if it would work. Eighteen months later though, I can say that the foundations are solid and much trial and error has paid off.
Psithur now has a product that we think is exceptional, and although there's still much we want to build, I wanted to share some reflections from the last 18 months and try to articulate why Psithur Connect’s foundations help make it an exceptional product.
Clojure
I was introduced to Clojure a decade ago by a colleague. Specifically, a video by Richard Hickey (Clojure's creator) titled "Simple Made Easy". In this video, Hickey introduces some of the concepts that drove him to create a new programming language. While I didn't start using Clojure properly for another 5 years, I recognised the wisdom in Hickey's arguments about software and complexity and I was drawn to learn more.
Exploring more of Hickey's work led me to think differently about programming. I started changing the way I wrote Python and adopted some of the functional paradigms at the core of Clojure (such as immutable data). I was lucky at the time to have a boss who was happy for me to learn Python on the job. Over 5 years of writing lots of Python and JavaScript with a slowly growing unease at what I couldn't do in these languages, I started using Clojure properly.
In hindsight, my unease was a manifestation of the lack of expressiveness of the languages I was using; that is, the measure of how much work it is to express a particular idea (or rather, how much code you need to write to solve a particular problem).
About five years ago I started using Clojure properly, albeit for hobby projects. It was a steep learning curve, but the more I used it (and the more I learned how to use it), the more I recognized its value.
I won't write more about Clojure itself here, but rather I'll point you to a blog post by Gene Kim, whose book The Phoenix Project should be on every project manager's shelf, "Love Letter to Clojure".
Graph
Over my career, almost every data problem I’ve worked on has had an aspect of Graph: more often than not, I didn't have access to the tools to apply a Graph-based solution.
One example that comes to mind; over a decade ago, working as a Data Miner at a large Australian Government Department, I was tasked with improving fraud detection algorithms within a category of transactions the Department was responsible for processing. The Department had invested in an Enterprise Data Warehouse - which was very good at storing very large tables of data.
This Warehouse was also reasonably good at answering simple questions such as "find all the transactions that occurred from accounts with the same email address and order them by quantity". The problem was that the patterns of fraud weren't limited to one attribute. That is, the question needed to be generalised to "discover every group of transactions where the account linked to each transaction shares at least one attribute with another account and order the results by the total number of accounts in that group".
You might recognise this as a classical Graph Analytics problem (social network analysis). My crude SQL solution using stored procedures and set theory was actually in the end very successful at finding "interesting" networks, but the large Enterprise Data Warehouse was not the right tool for the job.
Today, if asked, I would define a Graph enabled system as a system that has the ability to navigate arbitrary relationships in the system's data: a capability that boils down to two things: the programmatic interface to the database and the choices made by the database in indexing its data. While SQL is good for some things, it's not suitable as an interface to data that is structured as a Graph.
Psithur Connect is built with Graph capabilities at its core, to be able to navigate arbitrary relationships (nearly) instantly, both in the algorithms and business logic running on the backend, as well as the user interface on the frontend.
ClojureScript
As a data scientist and data engineer, I spent a lot of time writing server-side Python to manipulate data. While this type of effort is a significant and necessary part of any data-related project, I always felt the options for non-technical users to derive value and insight from their data was limited. This was because the languages that I was using were good for manipulating data but of no use to almost anyone else.
What was missing was expressive, well-designed, performant user interfaces where data could be brought to life, be explored, triaged and shared, and most importantly where insights were accessible, not just to those people who could code.
In my last startup, I built bespoke full-stack analytics and business productivity platforms using Python and JavaScript. While I was able to craft some really neat user experiences, again, the tools I was using didn’t match the ambition I had.
One of Clojure's super-powers is that it works both on the server (or backend) for data processing as well as the frontend, for user interfaces. ClojureScript is the Clojure variant that compiles to JavaScript and hence runs in a web-browser. This means that a project (like Psithur Connect) that uses Clojure can share code and business logic between the frontend and backend and, one developer, working in the same programming language, can build both.
This reduces complexity (no more translating data or models between two disjoint systems) and enables a single system that not only feels coherent to develop but, far more vitally, feels coherent to its users.
XTDB
We've built our backend tightly integrated with XTDB, a database that offers a unique combination of features that no-other database has :-
A schemaless, document oriented model (such as MongoDB)
Full change history (MarkLogic, Datomic, countless retrofitted EDWs)
Graph traversal (such as TigerGraph and Neo4j).
Full text search (actually uses Lucene, such as ElasticSearch)
A datalog interface and pull syntax support (Datomic)
In addition to those features, XTDB is built with Clojure and is OpenSource and extensible through a modular (aka “unbundled”) architecture - in-fact we're using three custom developed XTDB modules currently.
We didn't start building with XTDB though. Psithur Connect - as it is today - has undergone many rounds of innovation, of trial and error, of improvement, resulting in a system built from many components that are integrated together to create a coherent whole.
Datomic
A little while into my journey to Clojure, I came across another of Richard Hickey's projects - Datomic. If you'll accept my assertion that Clojure was a revelation to me as a Python programmer, you should also believe me when I say that learning about Datomic was a revelation as a Data Engineer.
For many years, I'd been thinking in terms of data locality, indexes, schemas and ontologies, audit and history logging. Datomic is a database built on the same concepts as Clojure and it's designed for a certain domain of problems - core business information systems - which are - in their essence - models of real world relationships. It is opinionated in its design (and unapologetic). But the revelation was the possibility of a database that can solve a unique set of intersecting requirements, specifically, operating as both a graph database and relational database while maintaining a complete history of facts over time.
While we have now migrated our platform from Datomic to XTDB, we owe a large debt to the work that the Datomic team put into their product. The ideas Datomic embodies have been critical to the design of Psithur Connect.
Clojure and Psithur Connect
Clojure is the reason we've been able to build an application like Psithur Connect.
Clojure is expressive so complex ideas can be expressed tersely (which aids simplicity) and complex problems can be solved iteratively without rewriting large parts of the whole.
Clojure is extensible, so we can leverage the best open-source libraries; not only from the Clojure and ClojureScript ecosystem, but also the Java and JavaScript ecosystem.
As if that was not enough, in the last few months we have also integrated machine learning libraries into our product using the JVM's Project Panama which extends further our integration options to include native C implementations of popular machine learning and vector search libraries. These are now integrated into the core of Psithur Connect.
As a developer, working with Clojure has spoiled me - for now I know what it's possible to achieve when you have the right tools for the job.
As a company, we're excited to bring Psithur Connect to more customers and while we very rarely talk about the technology that it is built on (because it's actually irrelevant to almost everyone), I wanted to at least pay some homage to the foundations that Clojure has provided for us to build a software product and company where we can deliver a rock solid user experience that delights our customers and maybe even feels a little bit magic.