Over the years, we’ve written a lot of data processing applications. Sometimes we used homegrown frameworks; sometimes we used frameworks created by third parties. Invariably, one of the most difficult problems we faced was managing state.
We’d often default to sticking state into a database, but then we had a couple of challenges. First, our performance and scalability were bounded by the database. Second, and even harder was handling failures. We’d end up rolling a homegrown solution to make sure that we never lost any message, or duplicated any updates.
And you know what? We always had bugs. At best, it was frustrating. At worst, it cost the business money.
State management and Wallaroo
When we set out designing Wallaroo, we wanted to put state at the core of what we did. To paraphrase our original design document:
Most current systems leave the developer holding the bag for accuracy. Programmers have to roll their own idempotence. We envision a system that doesn’t force the programmer to be a distributed systems expert. Idempotent processing is hard. State management is hard. But it doesn’t have to be.
Wallaroo provides the application developer APIs that help them manage state. We give programmers APIs for creating, updating, and querying their application state, so that the platform handles failures for them. Not duplicating updates becomes the platform’s problem, not yours.
Check out this snippet of code from our Python Word Count example:
class CountWord(): def name(self): return "Count Word" def compute(self, word, word_totals): # increment the number of times we've seen this `word` word_totals.update(word) # return current count for `word` return (word_totals.get_count(word), True)
That’s what we call a ‘StateComputation’. It’s one of the building blocks of a Wallaroo application. It’s pretty straightforward Python code that does some pretty awesome things because it’s running in Wallaroo.
‘word_totals’ is the state that Wallaroo is managing. In this case, a dictionary that contains a mapping from a word to the number of times we’ve seen it. For example:
Wallaroo makes sure that as you update the map, that the updates are written out to an event log. That event-log can be used to replay in the case of failure. Our optional exactly-once message processing ensures the programmer never has to worry about duplicating updates.
Watch our video demo below to see it in action.
Need more details? Below is a short video created by our engineering team that has helped people understand what Wallaroo does.
This video will give you:
- An overview of the problem we are solving with our scale-independent API
- A short intro to the Python API
- A demonstration of our autoscale functionality
- An idea of the power of scale-independent APIs in action