Welcome

Status

This documentation is a work in progress. Articles which have content in them appear normally in the chapter navigation. Articles which are incomplete appear with a (E) next to their name, indicating it is an empty unwritten article.

Hi there! You've discovered the comprehensive Holochain guidebook.

Holochain is an open source software library that provides a way for businesses, communities, and other groups to build and run applications which are hosted and validated by the "users" themselves. Doing so provides a superior level of agency and autonomy over heavy reliance on the so-called "cloud" and other third parties.

These applications are known more widely as peer-to-peer, decentralized applications, or dApps. To distinguish between dApps built on Blockchains and those built on Holochain, we preemptively call the latter "hApps". A more detailed comparison between Blockchain dApps and Holochain hApps is available here.

Holochain provides a cross-platform framework for the development and execution of these applications. By running these kinds of applications, "users" cease to merely "use". They become "user-participants" who are also responsible for hosting and validating the network's data. Applications can be developed utilizing any of the major operating systems, and run on virtually any device.

The many benefits and opportunities associated with peer-to-peer dApps (e.g. offloaded server costs, elimination of single points of failure, and flexible governance structures) are made available, and often amplified through the Holochain hApp architecture, on desktops, laptops, and Android (arm64) devices.

This book provides an in-depth explanation of Holochain's functions, from data validation to data propagation, so that you can get to work straightaway on developing applications that will serve your business, community, or otherwise.

Overview

Depending on whether your interest is specific, or general, you may wish to read it front to back, or skip to specific sections of the book. If the summary of a section describes you, then it's a section for you!

Planning a dApp

Readers: You can't put the horse before the cart. First things first: understanding the landscape of decentralized apps. What are the mechanics of a functioning decentralized app? You have an idea of what you want to build, and you need to get a grasp on what it's going to take. You need to know what your blind spots are, and what are the common pitfalls. You need to know which of your assumptions to hold on to, and which to let go.

Writing: General Audience

Building Holochain Apps

Readers: Whether you're a seasoned developer, or just starting out, you're in it to write code. You've either got an app project (or five) on the go, or you're in it to experiment and test the limits. You need to know what you need to know. You want to talk Holochain's language. You're an intrepid explorer of technical documentation.

Writing: Technical, Explanatory & How-to

Going Live with Holochain Apps

Readers: You're involved in the conception, development, or design of a Holochain app, and you've got to know how to get your app out into the world, into the hands of the people who need it. You've got questions like "How do updates to the app work?", "How do I track performance of the app?", "What are best practices for security?"

Writing: General Audience

Extending the Holochain Platform

Readers: You want to look at Holochain itself, not what you can build with it, but to see what you can tweak, or contribute. You've got ideas for Holochain and got skills to pull them off. You're reading the Holochain source code, or source documentation. Maybe you want to enable app development in a whole other language not available yet, or maybe to run Holochain on a device or platform not supported yet. You can make sense of terse technical language, and direct yourself well.

Writing: Technical, Explanatory & How-to

How to contribute

This book uses a tool that builds HTML files from markdown files called 'mdbook'. The markdown files are stored on GitHub, in the main holochain-rust repository. Because they are on GitHub, they have built-in version control, meaning that it's easy for anyone to contribute, and to propose, track and merge changes.

There are two main pathways to contribute. One is by editing files directly in GitHub, and the other is cloning the repository to your computer, running mdbook locally, and then submitting change requests. The following covers both scenarios.

Writing Guidelines

Please do not edit the SUMMARY.md file, which stores the raw chapter structure, without advanced signoff from the documentation team in https://chat.holochain.org/appsup/channels/hc-core.

More forthcoming!

How the Book Works

Writing is in classic markdown, so a new page is a new markdown file. These files get rendered in the book panel. One markdown file, SUMMARY.md stores the structure of the book as an outline.

The HTML files used in the Book get automatically built from the markdown files. Even though they're auto-generated, static HTML files, one can search within the text.

What to Contribute

For a current list of open documentation issues, check out the 'documentation' label for github issues.

Contributing via GitHub

Getting there

  1. Log on to your GitHub account

  2. In the Holochain Rust repo, everything is under the doc/holochain_101/src folder. All markdown files are there, and some are nested in subfolders. Navigate to the following link: https://github.com/holochain/holochain-rust/tree/develop/doc/holochain_101/src

  3. Determine whether you are making, editing, or reviewing an article.

Access Rights

If you don't have write access to the repository you need to create a fork to contribute. Forking is easy. Click the "Fork" button in the top right hand corner of the Github UI.

Making a new article

  1. Click "Create New File"

  2. Name this file what you intend to name the article, plus the .md extension, i.e. how_to_contribute.md

  3. Use classic markdown to set up the page title, i.e. "# How to contribute"

  4. Write the rest of your text, checking the "Preview" tab to see how it would look.

  5. Scroll to the bottom of the page and select the option "create a new branch for this commit and start a pull request". You can name a branch, though GitHub will set one automatically. If you know it, mention the issue that the request addresses.

  6. Click "Propose New File". Proceed to the Making Multiple Edits or Opening a Pull Request section.

Editing an article

  1. Navigate to the article you want to edit.

  2. Click the 'pencil' icon to edit the article. There's a built-in text editor in GitHub, where you can write a change and also why you changed it (so that a reviewer can understand the rationale for the change).

  3. Select the branching method for making your change. (See Making Multiple Edits for clarification)

  4. Click "Propose File Change". Proceed to the Making Multiple Edits or Opening a Pull Request section.

Making Multiple Edits On One Branch & Pull Request

A "branch" is a series of divergent changes from the main version. If you want to make multiple edits at once, you will need to make each of those changes on the same branch as you named your original edit. Check which branch you are on by looking for the "Branch: ?" dropdown. Use the dropdown to switch to your branch if you're on the wrong one.

Opening a Pull Request

  1. Once redirected to the "comparing changes" page, prepend your pull request title with "MDBOOK: " and then a very short statement of what changed.

  2. Add a more detailed description of what changes you made and why in the text box.

  3. If there is an open issue related to the article you're submiting or editing, tag it by using the "#" plus the issue number.

  4. Add the "documentation" label.

  5. If appropriate, click "Reviewers" and select one or more people to request reviews from.

  6. Click "Create Pull Request".

Reviewing a Pull Request

  1. Under the Pull Request tab, look for ones starting with "MDBOOK". Go to the Pull Request of your choice, and then click on the "Files Changed" tab.

  2. Start a review by hovering over a line and pressing the blue "add" symbol to add comments to a line

  3. Click the green "Review Changes" button. If you approve of the changes, select "Approve". If you would like further changes to be made before it gets merged, select "Request Changes". If you are just weighing in, select "Comment". Then, click "Submit Review".

Merging a Pull Request

  1. Under "Conversation" you can merge the pull request, which integrates it into the develop branch. Changes automatically deploy to https://holochain.github.io/holochain-rust within ~30 minutes. Merge the pull request once it has received two approved reviews.

Contributing by Cloning and Running Mdbook (advanced)

You will need to have cloned holochain-rust to your computer. You will also need Docker installed.

There is a Docker build that allows local build, serve, watch and live reload for the book.

From the root of the repo, run:

. docker/build-mdbook-image && . docker/run-mdbook

Once the book has built and is serving, visit http://localhost:3000 in the browser.

You can edit the markdown files in doc/holochain_101/src and the book will live reload.

To do a one-time build of the files to HTML, run:

. docker/build-mdbook

Edit the files to your satisfaction, commit them to a branch (or a fork) and then open a pull request on GitHub. Once its on GitHub, the same things as mentioned above apply.

Planning a dApp

What is a dApp?

A dApp is a distributed application. This means that the data associated with the application is stored by each user rather than in a central database.

Basic expectations for dApps

Generally speaking, you need to know the following in order to build a Holochain dApp:

how to install Holochain

how to use the command line tools

how to configure your application with a "DNA" file

how to write your application code in a language that compiles to WebAssembly

how to think through building a distributed application

how to build a user interface for your app

how to test your application code

This article will help you plan a dApp by providing practical considerations about the specifics of distributed applications in general, and Holochain dApps in particular. It has been remarked that holochain dApps require us to make a mental shift, first from applications whose data is centrally organized, and also from blockchain-based, data-centric dApps.

Here we will provide a basic overview of concepts from cryptography that are central to holochains. Then, we will consider the consequences of Holochain's cryptographic architecture for data permissioning, access, and security. Because app data storage is distributed amongst user-participants, one must expect that data encryption and permissions are important for protecting privacy in accordance with the jurisdictions in which the app is operating.

Remember that, as user-participants leave the application, they take their data with them. They also retain copies of other data that they held to support the DHT.

One must also re-think the dApp's business model such that it does not rely on a central authority's ability to whitelist access to a given resource.

Cryptography in Holochain dApps

Distributed systems rely more heavily on cryptographic patterns and techniques than centralized systems. The basic concepts below explain how data integrity, ownership, and security are achieved natively with holochain's architecture. They are time-worn, relatively intuitive ideas that are critical for planning a holochain dApp.

Hashes

Hashes ensure the reliability of information by representing any given piece of data with a unique, consistent string of random looking characters. This makes changes to data visible because one can see that a hash has changed without needing to inspect the data itself.

However, it is impossible to get the original data from a hash -- its purpose is to prove that the data to which it corresponds has not been altered. The same data consistently gives the same hash, and different data always give a completely different hash.

These features imply that one can use small, portable hashes to verify data. One could also use a database containing data and their hashes as a table of contents, indexing (though not reading) data associated with a given hash.

In the context of Holochain hashes are frequently used to look up content, both in our internal implementations as well as on the DHT. Therefore we frequently refer to the hash of some item (i.e. an entry on the chain) as its Address.

Signatures

Signatures provide an additional type of data verification, answering the question "who created this data?" Signatures look like hashes. They are unique, reliable, and like hashes, cannot be used to retrieve the data to which they correspond. Signatures also come with a pair of keys. One is public, and the other private.

The private key designates a unique author (or device), and the public key lets anyone verify a signature made by one specific private key. This key infrastructure addresses the problem of single points of failure associated with centralized systems by making authors responsible for securing their unique private key.

Encryption

What if one needs to restrict access in addition to verifying data? Two types of encryption are possible. Symmetric encryption has one key for reading and writing data. Asymmetric encryption has two keys, where one creates messages and the other reads them.

Encryption is a two way process, so the right key enables one to decrypt an encrypted message. With this added benefit come the drawbacks of the size of encrypted messages (at least as large as the original data) and broken encryption stripping the author of control of the original data.

Key Management

Once you are using cryptographic functions, either signing or encrypting, managing and securing the keys that unlock these functions, becomes a huge challenge to maintaining the overall system's integrity because there have to be safe ways to generate and store keys and also verify the provenance of the keys being used. In Holochain we have designed for a Distributed Public Key Infrastructure (DPKI) in which we leverage the distributed nature of Holochain itself to help manage all the aspects of this challenge.

Data access paradigms

The following are five data access paradigms. Note that in real-world scenarios it is common to mix these options by combining separate dApps. In instances when many separate dApps are needed to share data, Holochain supports bridging between dApps. Bridges between two networks with different data privacy models specify who can use the bridge, what data crosses the bridge, and tasks that might run in response to the bridge (e.g. notifications)

The default model for Holochain data is public data shared on a public network, and every Holochain dApp has its own network and data, and creates networks for user-participants as soon as they join a dApp. The dApp code sets sharing and verification rules.

Public, shared data on a public network

Public data works like Bittorrent:

  • anybody can join a network
  • anybody can request any data they want from the network
  • any data is available as long as at least one person is sharing it
  • if some data is not shared by enough people, a new random person on the network must share it
  • there is no "local only" data

As stated above, an additional requirement for Holochain dApps is that new data must have a digital signature.

Public, shared data on a private network

The functionality is the same as a public network, but private networks use cryptography for access control to the network itself.

Each device on the network must open a P2P connection to another device before it can send any data. The devices that are already on the private network send a challenge to any new device before sending any more data. The new device must sign the challenge with the network private key. The network public key is set in the dApp configuration, available to Holochain. Holochain can then refuse any connection with a bad challenge signature. Data within the network is public and shared. Every device on the network has “logged in” with a signed challenge, so has full access.

Encrypted data on a public or private network

Encryption relies on dApp developers encrypting and decrypting data within the dApp software.

Holochain exposes a set of industry standard encryption algorithms (e.g. AES) to each dApp that cover both symmetric and asymmetric encryption options, in addition to hashing and signing tools.

This option is very flexible for dApp developers but security becomes more subtle. Much like the private network, any one user-participant losing a key can become a serious security problem.

Note that encryption can pose problems for Holochain's native validation method.

Private, local only data

Any data in a dApp can be validated and stored by its author without being shared to the network. Private, local data can provide a useful database for portable user preferences and notes and avoids the complexity of encryption and key-based security.

Private data is hashed in the same way as public data, and the hash is public. Accordingly, one could tell that private data exists without being able to access it or take advantage of this with dApps that feature the eventual reveal of previously authored, private data -- think a distributed guessing game, like "rock, paper, scissors" or a digital classroom that operates with signatures disconnected from real-world identity and uses this method to prevent cheating.

Hybrid model with servers

Holochain supports many different environments and interfaces so that Holochain is easy to integrate with existing infrastructure. Any connected system with an API can push data through a dApp, as when one's phone sends a summary of private calendar data to a Holochain dApp. Any data in a dApp immediately becomes transparent, auditable and portable.

The version of Holochain in active development covers the following integrations:

  • Command line tools
  • Web server
  • Android
  • Qt/QML
  • Unity 3D games engine

Security - best practices

A great way to begin offsetting the governance crises now typical of distributed systems (i.e. DAO hack) is to think in terms of protecting and enabling the community of user-participants in addition to cryptography.

In essence, one must consider how to prevent undesired access to the DHT. If membranes are not properly built in the dApps' DNA, having access to the source code also means having access to the entire network's entries via the DHT. Developers must treat the code, or at least the DNA taken as a whole, as if it's a key to the data. Note, too, that one can easily fork a Holochain dApp without disrupting its activity, making it possible to retain the benefits of open-source code without some of the risks.

Membranes

Security efforts begin with the specification of membranes, lest the code itself become a target. Though holochains rely on the cryptography above to create trust in data's provenance and immutability, trust is a distinctly human affair at the level of creating membranes. Different applications will require different levels of security, and Holochain is uniquely suited to accommodate a high degree of flexibility. DNA could define a closed list of participants, Proof of Service, or social triangulation requirements, for example.

Sybil attacks are attacks launched through the creation of many nodes explicitly for the purpose of the attack. These nodes are identifiable by having no history. Blockchains prevent Sybil Attacks with Proof of Work. PoW is an implied membrane since it constrains who can verify blocks. In that case, the clearing node must have done "work" to maintain the network. Holochain requires membranes to identify and filter out Sybil nodes so that an attacker cannot use them to overrun the DHT.

Immune system

Holochain relies on random users across the network validating every piece of data. This keeps the verification rules reliable and unbiased. This is called an "immune system" for validating content.

Note that when using encrypted data, it is not possible to verify contents without the appropriate key. Encryption is a choice that should be made carefully, as it undermines Holochain's native immune system.

Scenarios to consider

  1. p2p platforms
  2. supply chains and open value networks
  3. social networks
  4. collaboration apps

Building Apps

If you're looking to build a Holochain app, it is important to first know what a Holochain app is.

First, recall that Holochain is an engine that can run your distributed apps. That engine expects and requires your application to be in a certain format that is unique to Holochain. That format is referred to as the "DNA" of your application. The DNA of an application exists as a single file, which is mounted and executed by Holochain.

Writing your application in a single file would not be feasible or desirable, however. Instead, you are supplied the tools to store your application code across a set of files within a folder, and tools to build all that code down into one file, in the DNA format.

While there are lots of details to learn about Holochain and DNA, it can be useful to first look from a general perspective.

Holochain and DNA

Recall that a goal of Holochain is to enable cryptographically secured, tamper-proof peer-to-peer applications. DNA files play a fundamental role in enabling this. Imagine that we think of an application and its users as a game. When people play any game, it's important that they play by the same rules -- otherwise, they are actually playing different games. With Holochain, a DNA file contains the complete set of rules and logic for an application. Thus, when users independently run an app with identical DNA, they are playing the same game -- running the same application with cryptographic security.

What this allows in technical terms is that these independent users can begin sharing data with one another and validating one anothers data. Thus, users can interact with the data in this distributed peer-to-peer system with full confidence in the integrity of that data.

The key takeaway from this is that if you change the DNA (the configuration, validation rules, and application logic) and a user runs it, they are basically running a different app. If this brings up questions for you about updating your application to different versions, good catch. This concern will be addressed later in this section.

Before exploring the details of Holochain DNA, take a minute to explore the different platforms that you can target with Holochain.

Introduction to DNA: Configuration

As a developer, you won't have to interact directly with the contents of a DNA file that often. However, it is quite important to grasp its role and structure.

Holochain DNA files are written in a data format known as JSON. It stores sets of key-value pairs, and allows a nested tree structure. It looks like this:

{
  "property_name": "property_value",
  "nest_name": {
    "nested_property_name": "nested_property_value"
  }
}

JSON is usually used for configuration and static data, but in the case of Holochain, these DNA files also contain compiled code, which is executable by Holochain.

As previously mentioned, you do not need to edit this "master" DNA file directly. Holochain command line tools can be used to build it from your raw files.

Learn more about the package command which fulfills this function

Configuration

For the configuration-related parts of your DNA, they will come from actual JSON files stored in your application folder. There will be multiple JSON files nested in the folder structure. An application folder should have a file in its root called app.json.

This file should define various properties of your application. Some of these properties Holochain fully expects and will not work without, others can be customised to your application.

app.json Properties

A default app.json file looks roughly like this:

{
  "name": "Holochain App Name",
  "description": "A Holochain app",
  "authors": [
    {
      "indentifier": "Author Name <[email protected]>",
      "public_key_source": "",
      "signature": ""
    }
  ],
  "version": "0.0.1",
  "dht": {},
  "properties": null
}

Intro to DNA: Code

The functionality of Holochain applications is written as a collection of logical modules called "Zomes".

Zomes are created inside a folder called zomes, and each Zome should have its own sub-folder within that, in which the configuration and code for that particular Zome should be placed.

These Zomes can call and access the functionality of the others, but they are written independently.

When the DNA file is being packaged, the code for these Zomes is encoded using Base64 encoding and combined with the configuration file associated with the Zome.

The configuration file should be a JSON file, stored in the Zome folder. The file can be named anything, but the default is zome.json.

This Zome file is extremely simplistic at this point, and contains only a description property, which is a human readable property that describes what the Zome is for.

The only coding language that Holochain knows how to execute is WebAssembly. However, it is unlikely that you'll want to write WebAssembly code by hand. Instead, most people will write their Zomes' code in a language that can compile to WebAssembly, such as Rust or Assemblyscript, and then define a build step in which it is compiled to WebAssembly. There is already a large, and growing, number of languages that compile to WebAssembly.

If this is sounding complex, don't worry. There are tools supplied to make this easy, and you'll be writing in a language that's familiar, or easy to learn.

With this overview in mind, the details of app development can be explored.

Intro to Command Line Tools

There are a set of custom-designed tools for working with Holochain that can be installed as utilities to your command line to simplify and accelerate the process of Holochain app development. These command line tools are required if you wish to be able to attempt what you are reading about as you continue through the articles on building an app.

To install the command line tools, use the quick start installation guide.

In addition to the installation instructions, the README has an overview of the different functions of the command line tools. You will also learn these organically just by proceeding through the other articles in this chapter and the ones that follow.

Create A New Project

The command line tools discussed in the last article can be used to easily create a new folder on your computer, that contains all the initial folders and files needed for a Holochain application.

You will typically want to create a new project folder for a Holochain application this way. This one approach will suit the creation of a new Holochain app or implementing an existing app with Holochain instead.

In your terminal, change directories to one where you wish to initialize a new Holochain app. The command will create a new folder within the current directory for your app.

Come up with a name for your application, or at least for your project folder.

Copy or type the command below into your terminal, except replace your_app_name with the name you came up with. Press Enter to execute the command.

hc init your_app_name

hc specifies that you wish to use the Holochain command line tools. init specifies to use the command for initializing a new project folder. your_app_name is an argument you supply as the app, and folder name.

This has created a new folder in which you have the beginnings of a Holochain app.

Check out the next article to see what the contents of a DNA source code folder looks like.

Project Source Folders

The source code folder for a Holochain DNA project looks something like this, where the ellipses (...) indicate a folder

  • test
    • ...
  • zomes
    • ...
  • .gitignore
  • .hcignore
  • app.json

test contains some starter code for writing tests.

zomes will contain sub-folders, each of which represents a "Zome", which can be thought of as a submodule of the source code of your DNA.

.gitignore contains useful defaults for ignoring files when using GIT version control.

.hcignore is utilized by the packaging commands of the hc command line tools

app.json is the top level configuration of your DNA.

Carry on to the next article to see about making changes to the configuration of a new project.

Configuring an App

As mentioned in Intro to DNA: Configuration at the top level of a Holochain app source code folder there should be a file named app.json. This file is useful for two primary things:

  1. When executing your application, Holochain can adopt specific behaviours, that can be configured in the app.json file. These mostly relate to how the Distributed Hash Table and P2P gossip functions.
  2. You can give app users, and other developers background info about your application, such as the name of the app, and the author.

Here are the properties currently in use:

Property Description
name Give this application or service a name.
description Describe this application or service for other people to read.
authors Optionally provide contact details for the app developer(s). It is an array, so multiple people can be referenced.
authors.identifier A string including a name, and a public email for the contact person.
authors.public_key_source Can reference a publicly hosted cryptographic "public key" from a private-public key-pair.
authors.signature The app developer can optionally add a string that is signed by their private key, so that app users could verify the authenticity of the application.
version Provides a version number for this application. Version numbers are incredibly important for distributed apps, so use this property wisely.
dht This is a placeholder for the configuration options that Holochain will implement, regarding the Distributed Hash Table. It will provide a number of ways that the DHT behaviour can be customized.
properties Properties, if used, can be an object which implements numerous app specific configuration values. These can be up to the app developer to define, and, when implemented, will be able to be called using the property function of the Zome API.

The minimum recommended values to set when you initialize a new project folder are:

  1. name
  2. description
  3. authors
  4. version

To edit them, just open app.json in a text editor (preferably one with syntax highlighting for JSON), change the values, and save the file.

Writing in Rust

It has always been in the designs for Holochain to support programming in multiple languages. In the prototype of Holochain, Zomes could be written in variants of Javascript (ES5) and Lisp (Zygomys). In the new version of Holochain the primary "Ribosome", where there was a JS one and Lisp one before, interprets WebAssembly code.

While we will provide a small introduction to WebAssembly shortly, we should also briefly introduce Rust, since it is the first language that has a first class Holochain app development experience, with its' WebAssembly compilation. This is accomplished via a Holochain Development Kit (HDK) library which has been written for Rust.

For the time being, writing Holochain apps requires writing code in Rust. This will not always be the case. Assemblyscript, a language based off Typescript, is the next likely language in which there will be an HDK library. If it happens to interest you, there is an article here about writing an HDK, since that is something we also invite and encourage the community to do.

From Wikipedia: "Rust is a systems programming language with a focus on safety, especially safe concurrency, supporting both functional and imperative paradigms."

Rust is a strongly typed language, which is desirable for the development of secure P2P applications, and compilation from Rust to WebAssembly is extremely easy.

If Rust is new to you, don't worry. With lots of Holochain app development happening in an open source way, and through learning resources like this guidebook, and the "Rust book" you will have lots to reference to get started.

While there are lots of other materials available for learning Rust, the base materials for the language are always a good resource to go back to: Rust Docs.

In many articles in the following chapters, you will find that there is a dedicated section at the bottom of the article called "Building in Rust". This typically follows after a general discussion of a concept, and is done this way because implementation details may differ based on the language developers are writing Zomes in.

Writing in Assemblyscript

As mentioned in writing in Rust Assemblyscript is a language based off of Typescript, which is designed to compile to WebAssembly. It is hoped that Assemblyscript will soon be mature enough, and have the necessary features, to be able to have an HDK for it. Work on an HDK for Assemblyscript commenced mid 2018, but hit roadblocks.

Updates on this should appear in a number of places:

Intro to WebAssembly

What is WebAssembly exactly?

"WebAssembly is a standard being developed by the W3C group for an efficient, lightweight instruction set. This means we can compile different types of programming languages ranging from C/C++, Go, Rust, and more into a single standard... WebAssembly, or WASM, for short, is memory-safe,platform independent, and maps well to all types of CPU architectures efficiently." - source

Though initially designed for use by major browsers IE, Chrome, Firefox and Safari, WASM has quickly been taken up as a portable target for execution on native platforms as well.

WebAssembly.org describes it as a binary instruction format for a stack-based virtual machine.

Despite being a binary format, "WebAssembly is designed to be pretty-printed in a textual format for debugging, testing, experimenting, optimizing, learning, teaching, and writing programs by hand."

This textual format is called WAT.

Not because it needs to be understood, but so that you can get a glimpse of what WAT looks like, here's a little sample:

(module
    (memory (;0;) 1)
    (func (export "public_test_fn") (param $p0 i64) (result i64)
        i64.const 6
    )
    (data (i32.const 0)
        "1337.0"
    )
    (export "memory" (memory 0))
)

Once the above code is converted from WAT to binary WASM it is in the format that could be executed by the Holochain WASM interpreter.

Often times, for a language that compiles to WASM, you will have a configuration option to generate the (more) human readable WAT version of the code as well, while compiling it to WASM.

While the compilation to WASM mostly happens in the background for you as an app developer, having a basic understanding of the role of WebAssembly in this technology stack will no doubt help you along the way.

Updating from holochain-proto to holochain-rust

If you wrote an application for holochain-proto, you are likely wondering what it may take to port your app to the new holochain-rust version of Holochain.

The following should provide multiple levels of insight into what this could involve.

At a very general level:

  • In terms of code, you have at least 2 options
    • rewriting the code in Rust
    • waiting for Assemblyscript support, and migrating at that point to Assemblyscript (the caveat to this approach is that it is not yet known at which point this support will arrive)
  • The API between a user interface and Holochain has switched from HTTP to Websockets (for now), and so any user interface must be updated to use this approach.
  • The DNA file has been simplified. Less is defined as JSON in the dna.json file and more is defined in the code.
  • Testing of DNA utilizes Nodejs to run tests, using the testing library of your choice. This replaces the custom (and limited) JSON test configuration employed by holochain-proto.
  • Schemas for entry types are no longer defined using json-schema, but using native Rust structs.

At the level of the code, in more detail, the changes are as follows (note that this is in reference to Javascript Zomes being ported to Rust Zomes):

  • all camel case function names are now snake case
  • makeHash is now named entry_address
  • commit is now named commit_entry
  • get is now named get_entry
  • update is now named update_entry
  • remove is now named remove_entry
  • Links are no longer created using commit, but instead have their own method, named link_entries
  • Instead of being implicitly imported, the Zome API functions are explicitly imported into Zomes, e.g. extern crate hdk;
  • The code of each Zome must now utilize a Rust "macro" called "define_zome!", and its various subproperties, which did not previously exist.
  • Many aspects of validation have changed, see the section below on validation

Updating Validation

There is a conceptual change to the approach to validation of entries, and even whereabouts that logic lives in the code.

In holochain-proto, there were a number of hooks which Holochain would call back into, to perform validation, such as

  • validateCommit
  • validatePut
  • validateMod
  • validateDel
  • validateLink

Regardless of how many entry types there were, there would still be only 5 callbacks defined maximum. These validation callbacks were performed at a certain stage in the lifecycle of an entry.

Now, an entry type is defined all in one place, including its validation rules, which are unique to it as an entry type. This could look as follows:


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct Person {
    name: String,
}
#}

# #![allow(unused_variables)]
#fn main() {
entry!(
    name: "person",
    description: "",
    sharing: Sharing::Public,
    native_type: Person,
    validation_package: || {
        hdk::ValidationPackageDefinition::Entry
    },
    validation: |person: Person, validation_data: hdk::ValidationData| {
        (person.name.len() >= 2)
            .ok_or_else(|| String::from("Name must be at least 2 characters"))
    }
)
#}

The callback validation, replaces validateCommit and all the rest from holochain-proto. However, validation still happens at various times in the lifecycle of an entry, so if the validation is to operate differently between initial commit to the chain, update, or remove, then that logic must be written into this single validation function. To determine which context validation is being called within, you can check in a property of the second parameter of the callback, which in the example above is called validation_data.

For this, you can use the Rust match operator, and check against the validation_data.action. It will be one of an enum that can be seen in detail in the API reference.

Yet to cover:

  • Capabilities
  • Traits
  • UI

Built With Holochain

Please click "suggest an edit", and add something you built with Holochain to the list!

  • Holochain Basic Chat
  • Todo list UI (Gatsby/Redux)
  • TODOlist example
  • HoloChat Simple UI
  • HoloVault
  • Simple App
  • Todo List tutorial
  • Hylo: a social network for community management, messaging and collaboration. They have a working network, although apparently with no activity. See also https://github.com/Hylozoic/hylo-holo-dnas.
  • Coolcats2: happ intended to replace Twitter.
  • Learn Using Test Driven Development
  • Holo. (People often confuse Holo and Holochain. To make it clear they are not the same thing, Holo is actually a web hosting happ, built on Holochain, to make it easier to bridge users to Holochain via web apps without having to know anything about Holochain, or do anything with it. It is not the same thing as Holochain, the agent-centric distributed app framework with unlimited scalability, or with scalability at least proportional to the number of nodes, with the limit on that number only being constrained by physics, number of computers,).
  • Producer’s Market. Provides a market or exchange for producers and consumers, with at least the initial focus on agriculture. See also this blog post.
  • Morpheus Network: automated, global, platform-agnostic supply-chain network.
  • Holosupply: digital supply chain integration platform.
  • Hypergroove: a data-driven, SaaS platform for more sustainable, inclusive, community-driven agri-food systems, with a focus on urban agriculture and p2p trade.
  • Redgrid: a SaaS platform for the Internet of Energy, connecting energy generation and storage devices and owners with power-consuming appliances and users. The first app built will be "Market Adaptive Demand Management, which will allow users’ high consumption devices—such as HVACs—to automatically respond to external market prices and signals in order to reduce consumption and costs. This will be applied to households, commercial buildings and multi-building sites. The application will be integrated into utility demand management programs to pay users for reducing their demand peaks, easing strain on the grid." It also aims to provide access to the 1.2b people in energy poverty. See also https://holo.host/project/redgrid/.
  • Metavents: a planning platform for events and humanitarian project with event fees donated to the same humanitarian projects, using the transparency of Holochain. It uses visual planning with 2D, 3D, VR/AR and AI.
  • Bridgit / Crowdfact. Bridgit is a Distributed Search and Discovery protocol and a protocol for the OverWeb, a new layer of trust, ideas, interactions, and experiences that display on top of the page source. Available as a Chrome extension. Crowdfact is part of Bridgit and involves crowd-sourced fact-checking.
  • Scribe: The SCRIBE Project aims to build a distributed Conceptionary Manager which is a distributed application that allows to manage one or more Conceptionaries. There's a Github repo at https://github.com/iPlumb3r/Th3Sr1b3Pr0j3ct.
  • Prtcl (The Underscore Protocol): a way for people to share ideas, content creation, and conversation with each other. It is inspired by Git, which is a version control system for distributed tracking changes in software development. It allows collaborators to work together on idea contexts and to develop branches of multiple perspectives within those contexts. It also supports arranging and nesting different but related contexts into more complex idea structures. Video timestamp. See also this video here.
  • Set Match Games: a platform for collaborative for video game design. See also this video here.
  • Hololution: personal productivity apps that will then transform into collaboration apps. Video timestamp: https://youtu.be/Y8gXQankAu8?t=503.
  • Omni: The Omni project establishes a global scholarly commons. Omni is a distributed scholarly network for direct scholarly communication, treated as main access point for all other Omni Project applications.
  • humm: a beautiful p2p publishing platform. Control your content and its distribution, own your creativity and humanity. They're aiming to foster a new kind of economy around content creation (which Holochain would be suitable for with microtransfers). Video timestamp: https://youtu.be/Y8gXQankAu8?t=597.
  • Comet. Holochain powered distributed Reddit-like application, with differences. "Because it is built on Holochain, Comet has no censorship or moderation, and can run on the devices of the users without the help of servers. Comet also replaces the idea of "subreddits" with tags, and allows posts to be created and crossposted with more than 1 tag. Perhaps the most interesting feature of Comet, though, is its voting system. Firstly, votes can be fractional, they range from -1 to 1. Votes are also not counted as "objective". The scores of posts and comments are always calculated from the perspective of a particular agent (the user). This agent will have upvoted and downvoted other posts from other people. The score is counted depending on how the agent has voted on the other voters in the past. This helps fight spam, vote manipulation, and helps maintain community in an otherwise lawless space." See also this blog post and https://youtu.be/Y8gXQankAu8?t=636.
  • Junto. A social network designed for authenticity. See also https://holo.host/project/junto/.
  • Core.Network: a social network designed to unify all social networks into a single visual dashboard.
  • Haven: see https://www.youtube.com/watch?v=Y8gXQankAu8&t=809s.
  • Holohouse: Coliving, Coworking and Coop, sustainable and with a purpose
  • Allianceblock: "The All-In-One Investment Ecosystem"
  • Orion: a liquidity aggregator protocol. "A standard for connecting to centralized and decentralized exchanges, enabling an ecosystem of dApps to solve liquidity issues and price parity. The Orion Protocol enables cross chain trading, omni-exchange accessibility, and liquidity."
  • Pacio’s Tender. A zero-fee transaction platform for distributed apps, built on Holochain.
  • Jala: "A digital platform that makes participation in any venture verifiable, intuitive and automated without the need for intermediaries."
  • Bizgees: Transforming refugees into entrepreneurs using FinTech.
  • bHive: "The bHive Cooperative is a community owned person-to-person sharing economy platform being developed for Bendigo [bank] by a team of five local entrepreneurs. bHive is the future of work."
  • Joatu: skills and goods marketplace for local communities.
  • 2RowFlow: "A Digital Commons for Treaty People in Canada"
  • Arcade City Realgame. A decentralized ride-sharing platform. See also https://github.com/ArcadeCity/unter and https://forum.arcade.city/t/pivoting-upward/37. It's not clear whether the latest closed-source implementation is built on Holochain—so it's best to assume that it's not.
  • Infinite World Game: "an operating system for humanity. A whole systems breakthrough in service to life. The Infinite World Game (IWG) is an unprecedented innovation. A vision and whole-systems platform for a society and world that does things differently. Cocreative players in the IWG organise themselves as living cells in a planetary super-organism (or ‘Universal Fractal Organism’, UFO), where ‘thriveability’ is the new name of the game for humanity and the planet. Drawing on a new world view and the principles of living systems, the IWG moves beyond our current concepts of economics and social organisation and offers a fresh way to share and regulate resources, interrelate, develop innovations, realise ourselves and co-ordinate our intelligence. As we elevate our capacity as a species, the IWG reorganises our social processes to go beyond product and profitability alone towards a world where every action is creating more life, love and thriveability for all."
  • Sacred Capital: infrastructure for building and exchanging reputational credit. See also this blog post.
  • HoloREA: happ infrastructure that uses the Resources, events, agents (REA) accounting model, with an implementation based on the ValueFlows protocol, and is built on Holochain. See also this video.
  • Cogov: a framework for experimentation with social system innovation. See also http://cogov.tech/.
  • Creafree: "Creafree registers, publishes and promotes creations to better meet the needs of the 21st century."
  • Ulex: an open source legal system. "In the case of Ulex, the “software” is not necessarily code (although you can access Tom W Bell’s Ulex Gitrepository here), rather they are rules that a legal system can operate under. Because Ulex is open source, its foundation is not tied to any one country and can be used in different jurisdictions. The goal of Ulex is to foster an open source community that creates and tries different variants of Ulex. Ulex protects human rights with an efficient and fair dispute resolution process, promoting the rule of law. It is not imposed by any one government, but instead is adopted by the mutual consent of those it governs. " See also https://physes.github.io/Ulex/ and https://ulex.app/.

See also

Building for Different Platforms

Holochain is designed to run on many different platforms. Essentially it can run on any platform that Rust and the Rust WASM interpreter targets. Thus Holochain DNA's will be able to run on platforms ranging from Raspberry Pis to Android smartphones once the tools have been fully developed.

We have experimented with C bindings that allowed us to run Holochain DNAs, in a Qt and Qml based cross-platform deployment, which besides running on desktop machines also worked on Android.

We expect other approaches to running Holochain apps on different platforms to proliferate, including compiling Holochain directly in your native application, whether it be an Electron app, a command-line Rust based app, or an Android app.

Building Holochain Apps: Zome Code

Recall that for the DNA of a hApp, there can be many Zomes, and each one will have their own source code. Think of Zomes as the fundamental unit of composability for DNA. As a DNA developer you can think of Zomes as modules. We expect developers to reuse Zomes written by others, and thus Zomes can call one another's functionality, using the call API function. Though currently Rust is the only available language for writing Zomes, note that these Zomes could be written in different languages (any language that compiles to WebAssembly) from one another in the future, and still access one another's functionality.

While writing the source code for DNA, it is extremely important to verify, before putting it into people's hands, that the code works as expected. For this reason, there are tools for testing included by default in newly generated projects. While there are technically a variety of ways that testing could be accomplished, and you could build your own, the most accessible of those is included by default, which is a JavaScript/nodejs Holochain Conductor. What this means is that the full scope of writing DNA, as of this writing, is likely for most people to include source code in two languages:

  • Rust
  • JavaScript

In the near future, this is likely to expand in diversity on both sides, Zome code and testing code.

Throughout this chapter, there will be plenty of examples given as to writing Zome code in Rust.

This chapter starts with a step by step tutorial, and goes on with a detailed explanation of the affordances of Holochain, and how to use its core functions.

First steps writing Holochain hApps with Rust


This tutorial builds for the 0.0.9-alpha release but as the API and HDK are changing it will likely fail under newer releases.


Holochain hApps are made of compiled WebAssembly that encodes the rules of the hApp, the data it can store and how users will interact with it. This means that any language that can compile to WebAssembly can one day be used for Holochain.

Writing WebAssembly that complies with the Holochain runtime can be tricky. To make development as streamlined as possible the core team has been developing a Holochain-dev-kit (HDK) for the first supported language, Rust! In the near future the community is encouraged to develop an HDK for their language of choice.

In this article we will walk through the steps of creating a simple hApp using Rust.

Requirements

First step is to download the appropriate dev preview release for your OS. If you decide to build the latest version from source, be warned that the API is undergoing rapid change, so some of the steps in this article may not work. The release contains the binary for the holochain developer command line tool, hc, which is used to generate a skeleton app, run tests and build the app package. Follow the installations on this page to install the required dependencies.

Ensure that hc is available on your path. If you instead decide to build from source cargo will ensure the binaries are on your path automatically.

If you want to jump ahead to see what the completed project will look like, the full source code is available on GitHub.

First steps

We will be making a classic to-do list hApp. A user can create new lists and add items to a list. They should also be able to retrieve a list by its address and all of the items on each list.

Let's begin by generating an empty hApp skeleton by running:

hc init holochain-rust-todo

This will generate the following directory structure:

holochain-rust-todo/
├── app.json
├── test
│ └── …
└── zomes

Notice the zomes directory. All Holochain hApps are comprised of one or more zomes. They can be thought of as similar to modules in JavaScript, each one should provide some self-contained functionality. Every zome has its own build system so it is possible to combine zomes written in different languages to produce a single hApp.

We will create a single zome called lists that uses a Rust build system:

cd holochain-rust-todo
hc generate zomes/lists rust

The project structure should now be as follows:

├── app.json
├── test
│ └── …
└── zomes
 └── lists
 ├── code
 │ ├── .hcbuild
 │ ├── Cargo.toml
 │ └── src
 │  └── lib.rs
 └── zome.json

Writing the lists zome

The Rust HDK makes use of Rust macros to reduce the need for boilerplate code. The most important of which is the define_zome! macro. Every zome must use this to define the structure of the zome, what entries it contains, which functions it exposes and what to do on first start-up (genesis).

Open up lib.rs and replace its contents with the following:


# #![allow(unused_variables)]
#fn main() {
#[macro_use]
extern crate hdk;
 
define_zome! {
    entries: [
    ]
 
    genesis: || {
        Ok(())
    }
 
    functions: [
    ]
 
    traits: {
    }
}
#}

This is the simplest possible zome with no entries and no exposed functions.

Adding some Entries

Unlike in holochain-proto, where you needed to define a JSON schema to validate entries, holochain entries in Rust map to a native struct type. We can define our list and listItem structs as follows:


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize, Debug, Clone, DefaultJson)]
struct List {
    name: String
}

#[derive(Serialize, Deserialize, Debug, Clone, DefaultJson)]
struct ListItem {
    text: String,
    completed: bool
}

#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct GetListResponse {
    name: String,
    items: Vec<ListItem>
}
#}

You might notice that the List struct does not contain a field that holds a collection of ListItems. This will be implemented using links, which we will discuss later.

Also be sure to add the following to the list of imports:


# #![allow(unused_variables)]
#![feature(try_from)]
#fn main() {
#[macro_use]
extern crate hdk;
#[macro_use]
extern crate serde_derive;
#[macro_use]
extern crate holochain_core_types_derive;
use hdk::{
    error::ZomeApiResult,
    holochain_core_types::{
        hash::HashString,
        error::HolochainError,
        dna::entry_types::Sharing,
        json::JsonString,
        cas::content::Address,
        entry::Entry,
    }
};
#}

The Serialize and Deserialize derived traits allow the structs to be converted to and from JSON, which is how entries are managed internally in Holochain. The DefaultJson derived trait comes from the holochain HDK itself and allows for seamless converting between data stored in the DHT and rust structs.

These structs on their own are not yet valid Holochain entries. To create these we must include them in the define_zome! macro by using the entry! macro:


# #![allow(unused_variables)]
#fn main() {
// -- SNIP-- //

define_zome! {
    entries: [
        entry!(
            name: "list",
            description: "",
            sharing: Sharing::Public,
            validation_package: || hdk::ValidationPackageDefinition::Entry,
            validation: |validation_data: hdk::EntryValidationData<List>| {
                Ok(())
            },
            links: [
                to!(
                    "listItem",
                    link_type: "items",
                    validation_package: || hdk::ValidationPackageDefinition::Entry,
                    validation: |_validation_data: hdk::LinkValidationData| {
                        Ok(())
                    }
                )
            ]
        ),
        entry!(
            name: "listItem",
            description: "",
            sharing: Sharing::Public,
            validation_package: || hdk::ValidationPackageDefinition::Entry,
            validation: |validation_data: hdk::EntryValidationData<ListItem>| {
                Ok(())
            }
        )
    ]

// -- SNIP-- //
#}

Take note of the native_type field of the macro which gives which Rust struct represents the entry type. The validation_package field is a function that defines what data should be passed to the validation function through the ctx argument. In this case we use a predefined function to only include the entry itself, but it is also possible to pass chain headers, chain entries or the full local chain. The validation field is a function that performs custom validation for the entry. In both our cases we are just returning Ok(()).

Take note also of the links field. As we will see later links are the main way to encode relational data in holochain. The links section of the entry macro defines what other types of entries are allowed to link to and from this type. This also includes a validation function for fine grain control over linking.

Adding Functions

Finally we need a way to interact with the hApp. We will define the following functions: create_list, add_item and get_list. get_list will retrieve a list and all the items linked to each list.

For each of these functions we must define a handler, which is a Rust function that will be executed when the conductor calls the function. (For more on conductors, read Nico's recent post.) It is best practice for functions to always return a ZomeApiResult<T>, where T is the type the function should return if it runs without error. This is an extension of the Rust Result type and allows zome functions to abort early on errors using the ? operator. At the moment the handler function names cannot be the same as the function itself so we will prefix them with handle_. This will be fixed in an upcoming release. The handler for create_list could be written as:


# #![allow(unused_variables)]
#fn main() {
fn handle_create_list(list: List) -> ZomeApiResult<Address> {
    // define the entry
    let list_entry = Entry::App(
        "list".into(),
        list.into()
    );

    // commit the entry and return the address
    hdk::commit_entry(&list_entry)
}
#}

The hdk::commit_entry function is how a zome can interact with holochain core to add entries to the DHT or local chain. This will trigger the validation function for the entry and if successful will store the entry and return its hash/address.

The add_item function requires the use of holochain links to associate two entries. In holochain-proto this required the use of a commit with a special Links entry but it can now be done using the HDK function link_entries(address1, address2, link_type, tag). The link_type must exactly match one of the types of links defined in an entry! macro for this base (e.g. link_type: "items" in this case). The tag can be any string we wish to associate with this individual link. We will just use an empty string for this example. The add item handler accepts a ListItem and an address of a list, commits the ListItem, then links it to the list address:


# #![allow(unused_variables)]
#fn main() {
fn handle_add_item(list_item: ListItem, list_addr: HashString) -> ZomeApiResult<Address> {
    // define the entry
    let list_item_entry = Entry::App(
        "listItem".into(),
        list_item.into()
    );

    let item_addr = hdk::commit_entry(&list_item_entry)?; // commit the list item
    hdk::link_entries(&list_addr, &item_addr, "items", "")?; // if successful, link to list address
    Ok(item_addr)
}
#}

At the moment there is no validation done on the link entries. This will be added soon with an additional validation callback.

Finally, get_list requires us to use the HDK function get_links(base_address, link_type, tag). As you may have guessed, this will return the addresses of all the entries that are linked to the base_address with a given link_type and a given tag. Both link_type and tag are Option types. Passing Some("string") means retrieve links that match the type/tag string exactly and passing None to either of them means to retrieve all links regardless of the type/tag. As this only returns the addresses, we must then map over each of then and load the required entry.


# #![allow(unused_variables)]
#fn main() {
fn handle_get_list(list_addr: HashString) -> ZomeApiResult<GetListResponse> {

    // load the list entry. Early return error if it cannot load or is wrong type
    let list = hdk::utils::get_as_type::<List>(list_addr.clone())?;

    // try and load the list items, filter out errors and collect in a vector
    let list_items = hdk::get_links(&list_addr, Some("items"), None)?.addresses()
        .iter()
        .map(|item_address| {
            hdk::utils::get_as_type::<ListItem>(item_address.to_owned())
        })
        .filter_map(Result::ok)
        .collect::<Vec<ListItem>>();

    // if this was successful then return the list items
    Ok(GetListResponse{
        name: list.name,
        items: list_items
    })
}
#}

Phew! That is all the handlers set up. Finally the function definitions must be added to the define_zome! macro. Before doing that, it is worth briefly discussing a new concept in Holochain, traits. Traits allow functions to be grouped to control access and in the future will allow hApps to connect to other hApps that implement a particular trait. At this time the only trait we need to consider is the hc_public trait. This is a special named trait that exposes all of the contained functions to the outside world.

The function field of our zome definition should be updated to:

define_zome! {

    // -- SNIP-- //
    functions: [
        create_list: {
            inputs: |list: List|,
            outputs: |result: ZomeApiResult<Address>|,
            handler: handle_create_list
        }
        add_item: {
            inputs: |list_item: ListItem, list_addr: HashString|,
            outputs: |result: ZomeApiResult<Address>|,
            handler: handle_add_item
        }
        get_list: {
            inputs: |list_addr: HashString|,
            outputs: |result: ZomeApiResult<GetListResponse>|,
            handler: handle_get_list
        }
        ]
        traits: {
        hc_public [create_list, add_item, get_list]
        }
}

and there we have it! If you are coding along the full lib.rs should now look like this:


# #![allow(unused_variables)]
#![feature(try_from)]
#fn main() {
#[macro_use]
extern crate hdk;
#[macro_use]
extern crate serde_derive;
#[macro_use]
extern crate holochain_core_types_derive;

use hdk::{
    error::ZomeApiResult,
    holochain_core_types::{
        hash::HashString,
        error::HolochainError,
        dna::entry_types::Sharing,
        json::JsonString,
        cas::content::Address,
        entry::Entry,
    }
};

 
define_zome! {
    entries: [
        entry!(
            name: "list",
            description: "",
            sharing: Sharing::Public,
            validation_package: || hdk::ValidationPackageDefinition::Entry,
            validation: |validation_data: hdk::EntryValidationData<List>| {
                Ok(())
            },
            links: [
                to!(
                    "listItem",
                    link_type: "items",
                    validation_package: || hdk::ValidationPackageDefinition::Entry,
                    validation: |_validation_data: hdk::LinkValidationData| {
                        Ok(())
                    }
                )
            ]
        ),
        entry!(
            name: "listItem",
            description: "",
            sharing: Sharing::Public,
            validation_package: || hdk::ValidationPackageDefinition::Entry,
            validation: |validation_data: hdk::EntryValidationData<ListItem>| {
                Ok(())
            }
        )
    ]
 
    genesis: || {
        Ok(())
    }
 
    functions: [
        create_list: {
            inputs: |list: List|,
            outputs: |result: ZomeApiResult<Address>|,
            handler: handle_create_list
        }
        add_item: {
            inputs: |list_item: ListItem, list_addr: HashString|,
            outputs: |result: ZomeApiResult<Address>|,
            handler: handle_add_item
        }
        get_list: {
            inputs: |list_addr: HashString|,
            outputs: |result: ZomeApiResult<GetListResponse>|,
            handler: handle_get_list
        }
    ]
    traits: {
        hc_public [create_list, add_item, get_list]
    }
}


#[derive(Serialize, Deserialize, Debug, Clone, DefaultJson)]
struct List {
    name: String
}

#[derive(Serialize, Deserialize, Debug, Clone, DefaultJson)]
struct ListItem {
    text: String,
    completed: bool
}

#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct GetListResponse {
    name: String,
    items: Vec<ListItem>
}

fn handle_create_list(list: List) -> ZomeApiResult<Address> {
    // define the entry
    let list_entry = Entry::App(
        "list".into(),
        list.into()
    );

    // commit the entry and return the address
    hdk::commit_entry(&list_entry)
}


fn handle_add_item(list_item: ListItem, list_addr: HashString) -> ZomeApiResult<Address> {
    // define the entry
    let list_item_entry = Entry::App(
        "listItem".into(),
        list_item.into()
    );

    let item_addr = hdk::commit_entry(&list_item_entry)?; // commit the list item
    hdk::link_entries(&list_addr, &item_addr, "items")?; // if successful, link to list address
    Ok(item_addr)
}


fn handle_get_list(list_addr: HashString) -> ZomeApiResult<GetListResponse> {

    // load the list entry. Early return error if it cannot load or is wrong type
    let list = hdk::utils::get_as_type::<List>(list_addr.clone())?;

    // try and load the list items, filter out errors and collect in a vector
    let list_items = hdk::get_links(&list_addr, "items")?.addresses()
        .iter()
        .map(|item_address| {
            hdk::utils::get_as_type::<ListItem>(item_address.to_owned())
        })
        .filter_map(Result::ok)
        .collect::<Vec<ListItem>>();

    // if this was successful then return the list items
    Ok(GetListResponse{
        name: list.name,
        items: list_items
    })
}
#}

The Zome we created should now build if we run:

hc package

from the root directory. This will compile the Rust to WebAssembly and produce a holochain-rust-todo.dna.json file in the dist folder which contains the compiled WASM code and the required metadata. This is the file that we can load and run using hc.

Writing tests

The testing framework is built on JavaScript around Tape.js and allows for writing single agent and muti-agent tests using javascript async/await syntax. Opening up the test/index.js file you will see a skeleton test file already created:

// This test file uses the tape testing framework.
// To learn more, go here: https://github.com/substack/tape
const { Config, Scenario } = require("@holochain/holochain-nodejs")
Scenario.setTape(require("tape"))

const dnaPath = "./dist/holochain-rust-todo.dna.json"
const agentAlice = Config.agent("alice")
const dna = Config.dna(dnaPath)
const instanceAlice = Config.instance(agentAlice, dna)
const scenario = new Scenario([instanceAlice])

scenario.runTape("description of example test", async (t, { alice }) => {
  // Make a call to a Zome function
  // indicating the function, and passing it an input
  const addr = alice.call("my_zome", "create_my_entry", {"entry" : {"content":"sample content"}})
  const result = alice.call("my_zome", "get_my_entry", {"address": addr.Ok})

  // check for equality of the actual and expected results
  t.deepEqual(result, { Ok: { App: [ 'my_entry', '{"content":"sample content"}' ] } })
})

This illustrates the app.call function that is exposed by the conductor for each app and that can be used to call our functions. Take note that the input-data should be a JSON object that matches the function signature. call will also return a JSON object.

Lets add some tests for our todo list:

const { Config, Scenario } = require('@holochain/holochain-nodejs')
Scenario.setTape(require('tape'))
const dnaPath = "./dist/holochain-rust-todo.dna.json"
const dna = Config.dna(dnaPath, 'happs')
const agentAlice = Config.agent('alice')
const instanceAlice = Config.instance(agentAlice, dna)
const scenario = new Scenario([instanceAlice])

scenario.runTape('Can create a list', async (t, { alice }) => {
  const createResult = await alice.callSync('lists', 'create_list', { list: { name: 'test list' } })
  console.log(createResult)
  t.notEqual(createResult.Ok, undefined)
})

scenario.runTape('Can add some items', async (t, { alice }) => {
  const createResult = await alice.callSync('lists', 'create_list', { list: { name: 'test list' } })
  const listAddr = createResult.Ok

  const result1 = await alice.callSync('lists', 'add_item', { list_item: { text: 'Learn Rust', completed: true }, list_addr: listAddr })
  const result2 = await alice.callSync('lists', 'add_item', { list_item: { text: 'Master Holochain', completed: false }, list_addr: listAddr })

  console.log(result1)
  console.log(result2)

  t.notEqual(result1.Ok, undefined)
  t.notEqual(result2.Ok, undefined)
})

scenario.runTape('Can get a list with items', async (t, { alice }) => {
  const createResult = await alice.callSync('lists', 'create_list', { list: { name: 'test list' } })
  const listAddr = createResult.Ok

  await alice.callSync('lists', 'add_item', { list_item: { text: 'Learn Rust', completed: true }, list_addr: listAddr })
  await alice.callSync('lists', 'add_item', { list_item: { text: 'Master Holochain', completed: false }, list_addr: listAddr })

  const getResult = await alice.callSync('lists', 'get_list', { list_addr: listAddr })
  console.log(getResult)

  t.equal(getResult.Ok.items.length, 2, 'there should be 2 items in the list')
})

Running hc test will build the test file and run it using node which is able to load and execute holochain hApps via the holochain node conductor. If everything has worked correctly you should see some test output with everything passing.

Pro tip: Pipe the output to tap-spec (which must be installed via npm first) to get beautifully formatted test output.

Conclusion

And there we have it! A simple Zome created with Holochain using the Rust HDK.

The complete working version of this project is available on github. This builds under the 0.0.9-alpha release but as the API and HDK are changing it will likely fail under newer releases.

Adding a Zome

After creating a new project, in the resulting folder there is an empty folder called 'zomes'. The name of this folder is important and should not be changed. It will serve as the root folder for the one or more Zomes in a project.

Every Zome should have its own folder within the 'zomes' root folder, and the name of those folders is also important, it should be the name of the Zome. It shouldn't have any spaces, and it should be a valid folder name.

While you could go about creating a new Zome manually through your file system, it will be far faster to use the Holochain command line tools to generate one, with all the basic files you need included.

To do this, navigate in a command line to the root directory of your hApp project. In the command line, run

hc generate zomes/your_zome_name

hc specifies that you wish to use the Holochain command line tools. generate specifies to use the command for initializing a new Zome. zomes/your_zome_name is an argument you supply as the path to, and the name of the Zome to generate.

The output should be as follows

cargo init --lib --vcs none
Created library package
Generated new rust Zome at "zomes/your_zome_name"

Note that in the case of a Rust Zome, which is the only language for a Zome we can generate at the moment, it will rely internally on Rust related commands (cargo init), meaning that Rust (and its package manager, cargo) must already ALSO be installed for this command to work successfully.

This has created a new folder (zomes/your_zome_name) in which you have the beginnings of a Zome.

What's in a Zome?

A Rust based Zome folder looks something like this:

  • code
    • src
      • lib.rs
    • .hcbuild
    • Cargo.toml
  • zome.json

code is a folder that should always exist in a Zome, and should contain either pre-compiled WASM, or the source code and instructions to generate WASM. Everything within code is contextual to the language the Zome is written in, in the case above, a Rust "crate". Files within code will be explained in detail below.

zome.json is the top level configuration of your Zome.

Rust crate Zomes

As mentioned above, the files within code are contextual to the language the Zome is written in, and in this case, that's Rust.

As developers tend to do, Rust developers gave their own unique name to Rust projects: "crates". There are two types of Rust crates: library and binary. Since Zome code is getting compiled to WebAssembly, not standard binary executables, Zome crates use the library style, which is why we see under code/src a lib.rs file.

The most minimalistic library crate would look like this:

  • src
    • lib.rs
  • Cargo.toml

Notice that the Zome we generated has one extra file, .hcbuild. This is the only Holochain specific file in the code folder. The rest is standard Rust. The .hcbuild file is discussed in another chapter.

In general, the generated files have been modified from their defaults to offer basic boilerplate needed to get started writing Zome code.

src/lib.rs is the default entry point to the code of a library crate. It can be the one and only Rust file of a Zome, or it can use standard Rust imports from other Rust files and their exports, taking full advantage of the Rust module system natively.

Cargo.toml is Rust's equivalent to nodejs' package.json or Ruby's Gemfile: a configuration and dependency specification at the same time.

Note that with the Cargo dependency system, Zome developers can take advantage of pre-existing Rust crates in their code, with one condition: that those dependencies are compatible when compiling to WebAssembly. This will be gone into in more detail elsewhere.

Intro to Language HDKs

Within any Zome, there are a number of conventions that must be followed by the WASM code, in order to work with Holochain core. For example parameters are passed using a specific memory allocation scheme that needs to be followed to access the parameters that a Zome function receives. Although it would be possible for Zome authors to code directly to the Holochain core API, it makes more sense to provide a software library for each language to make it as easy as possible to use those standard functions and behaviours. We call such a library a "Holochain Development Kit", or HDK.

So in order to get familiar with coding for Holochain, it will involve familiarity with an HDK library.

An HDK performs many important functions for developers.

  • It aids with the memory management of the WASM code that your Zomes compile into. WASM has some strict memory limitations, like 64 KiB paged memory.
  • It creates helper functions that hide tedious semantics that seem to plague compilation of languages to WASM.
  • It implements a complete type system that's compatible with Holochain.
  • It addresses hidden functions that Holochain needs from Zome code, but that would be redundant and confusing to make Zome developers write again and again.

The HDK for a given language, if it has deep integration into the build tools, such as the Zome generator, should include this helper library by default, which is the case of the Rust HDK.

There is an in-depth article on writing an HDK if this sounds interesting to you.

The Rust HDK

The HDK with priority development and support is implemented in Rust, and is included right in the core repository along with the Holochain core framework. Other HDKs may be implemented in different languages, and exist in separate repositories. This HDK implements all of the above features for developers, so just know that as you develop your Zome, a lot is going on behind the scenes in the HDK to make it all work.

The Rust HDK has documentation for each released version, available at developer.holochain.org/api. This documentation will be invaluable during your use of the HDK, because it can show you how the definitions of the various custom Holochain types, and give examples and details on the use of the API functions exposed to the Zomes.

Notice that in the Cargo.toml file of a new Zome, the HDK is included. For example,

...
[dependencies]
...
hdk = { git = "https://github.com/holochain/holochain-rust", branch = "master" }
...

Setting the version of the HDK

Once Holochain stabilizes beyond the 0.0.x version numbers, it will be published to the Rust package manager, crates.io and versioning will be simplified. For now, Cargo installs the HDK specified as a GIT dependency, fetching it from the specified commit, branch, or tag of the repository.

If you wanted to lock the HDK at a specific version, you could adjust the HDK dependency like this:

hdk = { git = "https://github.com/holochain/holochain-rust", tag = "v0.0.7-alpha" }

Use of the HDK in Rust code

Notice now that within the src/lib.rs file of a new Zome, the HDK is already imported here too:


# #![allow(unused_variables)]
#fn main() {
#[macro_use]
extern crate hdk;
...
#}

The #[macro_use] statement on the first line is very important, since it allows the usage of Rust macros (discussed in the Define Zome article) defined in the HDK to be used in your code, and the macros will be needed. Now, within the Rust code files, exposed constants, functions, and even special macros from the HDK can be used.

The very first thing to familiarize with is the define_zome! macro. Read on to learn about it.

Intro to Zome Definition

The adding a zome section explored the file structure of a Zome. Intro to HDK covered Holochain Development Kit libraries for languages that Zomes can be written in. Once a Zome has been generated, and the HDK imported, it is time to start adding definitions to it.

There are multiple aspects to defining a Zome. They will each be covered in detail in the following articles.

What are the characteristics of a Zome, that need defining?

A Zome has

  • name: the name of the containing folder
  • description: defined in JSON in the zome.json file within a Zome folder
  • config: Not Implemented Yet
  • validating entry types: definition may vary based on the language
  • a genesis function: a callback that Holochain expects and requires, defined in the code itself
  • fn_declarations: a collection of custom functions declarations,
  • traits: sets of named function groups used for composability
  • code: the core application logic of a Zome, written in a language that compiles to WASM, which Holochain interprets through that compiled WASM

To develop a Zome, you will have to become familiar with these different aspects, the most complex of which are the validating entry types, and the traits and function definition. Implementation details will differ depending on the language that you are developing a Zome in.

Building in Rust: define_zome!

As discussed in the intro to HDK article, by setting the HDK as a dependency in the Cargo.toml file, and then referencing it in src/lib.rs, Zome code in Rust gains access to a host of features.

The first line in the following code snippet (from a src/lib.rs) is important: #[macro_use]. This imports access to custom Rust macros defined by the HDK.

What are Rust macros? Generally speaking, they are code that will actually generate other code, when compiled. They are shortcuts. Anywhere in Rust that you see an expression followed immediately (no space) by an exclamation mark (!) that is the use of a macro.

In the case of Zome development, it was discovered that much code could be saved from being written, by encapsulating it in a macro.

That is how define_zome! came about. It is a Rust macro imported from the HDK which must be used for every Zome (unless you read the source code for it yourself and write something that behaves the same way!)

The following is technically the most minimalistic Zome that could be implemented. It does nothing, but still conforms to the expectations Holochain has for a Zome.


# #![allow(unused_variables)]
#fn main() {
#[macro_use]
extern crate hdk;

define_zome! {
    entries: []

    genesis: || {
        Ok(())
    }

    functions: []
}
#}

entries represents the validating entry type definitions. Note that it is an array, because there can be many. What validating entry types are will be explained next.

genesis represents the previously mentioned genesis callback that Holochain expects from every Zome. Skip here for details.

functions is where the functions are defined. Skip here for details.

These are the three required properties of define_zome!.

App Entry Type Definitions

An "entry" is a data element that an agent authors to their source-chain (stored on their local device), which is then propagated to peers. The entry is backed by a "chain header", which is the data element used for verification of the integrity of itself, as well as the entry.

Entries are a fundamental, primitive type within Holochain. Entries are an abstraction, they can technically be persisted to a device in a variety of ways, using a variety of databases, which can be as simple as files in the file system.

There are types of entries which cannot be written to the chain by users of an application. These are generally called system entries. They include DNA, and initial Agent entries, which are always the first two entries written to a chain.

There are a special type of entries called App Entries. These are entries which are created through the active use of an application by a user. They must have an entry type which, rather than being system defined, is defined by the Zome developer.

Defining App Entry Types

Creating a Zome for a hApp will almost always involve defining app entry types for that Zome. This means looking closely at the data model.

What types of data will the Zome be designed to handle? Is it dealing in "users", "transactions", "friendships", "tasks", or what? These will be the entry types needing definition in a Zome.

Broadly speaking, when defining the entry type, the developer of a Zome is designing the behaviour and generic properties of data of that type. This includes these important aspects:

  • how that data is shared, or not, with peers
  • the schema for entries of the type
  • custom validation behaviour for entries of the type
  • types of relationships (links) that can exist between entry types

An entry type is given a name that is used when an agent is attempting to write an entry to the chain. That's how Holochain knows what to do with the data for the entry that it has been given.

An entry type should also be given a basic description so that other people reading it understand the use of the entry type.

A third important property is sharing. The primary options for this at this time are 'Private' and 'Public'. Private means entries of this type will stay only the device of the author. Public means entries of this type will be gossiped to other peers sharing copies of the DNA. Public does NOT mean that it will be shared publicly on the internet.

Examining a .dna.json file closely, nested within the JSON configuration for a Zome, for an entry type you might see something like the following:

"entry_types": [
    {
        "entry_type_name": "post",
        "description": "A blog post entry which has an author",
        "sharing": "public",
        "links_to": []
    }
]

This is a Zome that implements only a single entry type, post.

These values are likely not to be modified within a JSON file, but within some code itself, where the entry type is defined. The validation rules for the entry type, will of course be defined within the code as well. Since this can be a complex topic, defining the validation logic has its' own article.

Setting up the entry types for a Zome is an often logical starting point when creating a Zome.

Building in Rust: Defining an Entry Type

Recall that in define_zome!, there was an array called entries. The most minimalistic Zome could look like this:


# #![allow(unused_variables)]
#fn main() {
#[macro_use]
extern crate hdk;

define_zome! {
    entries: []

    genesis: || {
        Ok(())
    }

    functions: []

    traits: {}
}
#}

entries is where we will populate the Zome with entry type definitions. It expects an array of ValidatingEntryType. So how can one be created?

Easy: the entry! macro. It can be used to encapsulate everything needed to define an entry type. All of the following must be defined:


name


# #![allow(unused_variables)]
#fn main() {
entry!(
    name: "post",
    ...
)
#}

This should be a machine-readable name for the entry type. Spaces should not be used. What will the entry type be that will be given when new entries are being created?


description


# #![allow(unused_variables)]
#fn main() {
entry!(
    ...
    description: "A blog post entry which has an author",
    ...
)
#}

This should be a human-readable explanation of the meaning or role of this entry type.


sharing


# #![allow(unused_variables)]
#fn main() {
use hdk::holochain_core_types::dna::entry_types::Sharing;

entry!(
    ...
    sharing: Sharing::Public,
    ...
)
#}

As mentioned above, sharing refers to whether entries of this type are private to their author, or whether they will be gossiped to other peers to hold copies of. The value must be referenced from an enum in the HDK. Holochain currently supports the first two values in the enum: Public, and Private.


native_type


# #![allow(unused_variables)]
#![feature(try_from)]
#fn main() {
extern crate serde;
extern crate serde_json;
#[macro_use]
extern crate serde_derive;
#[macro_use]
extern crate holochain_core_types_derive;

#[derive(Serialize, Deserialize, Debug, DefaultJson,Clone)]
struct Post {
    content: String,
    date_created: String,
}

entry!(
    ...
    native_type: Post,
    ...
)
#}

Clearly, native_type is where things start to get interesting. It requires the introduction of quite a number of dependencies, first of all. Why is that?

It is important to remember that the Rust code of a Zome is compiled into WASM before it can be executed by Holochain. This introduces a certain constraint. How is data passed between Holochain, and the WASM Zome code? Answer: it is stored in the WASM memory as stringified JSON data, and accessed by the WASM code and by Holochain, running the WASM interpreter.

JSON was chosen as the interchange format because it is so universal, and almost all languages have serializers and parsers. Rust's is called serde. The three serde related dependencies all relate to the need to serialize to and from JSON within Zomes.

Note that the top line in the snippet above is important. It switches on a Rust feature that would otherwise be off, allowing attempted conversions between types, which is exactly what the JSON parsing is doing.


# #![allow(unused_variables)]
#![feature(try_from)]
#fn main() {
#}

Additionally, the HDK offers built-in conversion functions from JSON strings to Entry structs. This comes from the DefaultJson derive.

Every struct used as a native_type reference should include all 4 derives, as in the example:


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize, Debug, DefaultJson)]
#}

Serialize and Deserialize come from serde_derive, and DefaultJson comes from holochain_core_types_derive.

Then there is the struct itself. This is the real type definition, because it defines the schema. It is simply a list of property names, the 'keys', and the types of values expected, which should be set to one of the primitive types of the language. This will tell serde how to parse JSON string inputs into the type. Note that conversion from JSON strings into the struct type can easily fail, in particular if the proper keys are not present on the input.


validation_package


# #![allow(unused_variables)]
#fn main() {
use hdk::ValidationPackageDefinition;

entry!(
    ...
    validation_package: || {
        ValidationPackageDefinition::Entry
    },
    ...
)
#}

At the moment, what validation_package is will not be covered in great detail. In short, for a peer to perform validation of an entry from another peer, varying degrees of metadata from the original author of the entry might be needed. validation_package refers to the carrier for that extra metadata.

Looking at the above code, there is a required import from the HDK needed for use in validation_package, and that's the enum ValidationPackageDefinition. The value of validation_package is a function that takes no arguments. It will be called as a callback by Holochain. The result should be a value from the ValidationPackageDefinition enum, whose values can be seen here. In the example, and as the most basic option, simply use Entry, which means no extra metadata beyond the entry itself is needed.

Further reading is here.


validation


# #![allow(unused_variables)]
#fn main() {
use hdk::EntryValidationData;

entry!(
    ...
    validation: |_validation_data: ValidationData<Post>| {
        Ok(())
    }
)
#}

validation is the last required property of entry!. Because it is such an important aspect, it has its' own in depth article.

It is a callback that Holochain will call during different moments in the lifecycle of an entry, in order to confirm which action to take with the entry, depending on its' validity. It will be called with two arguments, the first representing the struct of the entry itself, and the second a struct holding extra metadata that can be used for validation, including, if it was requested, the validation_package.

The callback should return a Rust Result type. This is seen in Ok(()). The example above is the simplest possible validation function, since it doesn't perform any real logic. While this is ok in theory, great caution should be taken with the validation rules, and further reading is recommended.

validation for a ValidatingEntryType should either return Ok(()) or an Err containing the string explaining why validation failed.

The validity of an entry is therefore defined by the author of a Zome. First of all, data which doesn't conform to the schema defined by the native_type will fail, but validation allows for further rules to be defined.

Note that not only the entry author will call this function to validate the entry during its' creation, but other peers will call this function to validate the entry when it is requested via the network that they hold a copy of it. This is at the heart of how Holochain functions as peer-to-peer data integrity layer.

Further reading can be found here.


Putting It All Together

Taken all together, use of the entry! macro may look something like the following:


# #![allow(unused_variables)]
#fn main() {
...
entry!(
    name: "post",
    description: "A blog post entry which has an author",
    sharing: Sharing::Public,
    validation_package: || {
        ValidationPackageDefinition::Entry
    },
    validation: |validation_data: EntryValidationData<Post>| {
        Ok(())
    }
)
#}

This can be embedded directly inside of the entries array of the define_zome!, like so:


# #![allow(unused_variables)]
#fn main() {
...
define_zome! {
    entries: [
        entry!(
            name: "post",
            description: "A blog post entry which has an author",
            sharing: Sharing::Public,
   
            validation_package: || {
                ValidationPackageDefinition::Entry
            },
            validation: | _validation_data: EntryValidationData<Post>| {
                Ok(())
            }
        )
    ]

    genesis: || {
        Ok(())
    }

    functions: []

    capabilitites: {}
}
#}

If there is only entry type, this can be fine, but if there are multiple, this can hurt readability of the source code. You can wrap the entry type definition in a function, and call it in define_zome!, like so:


# #![allow(unused_variables)]
#fn main() {
...
fn post_definition() -> ValidatingEntryType {
    entry!(
        name: "post",
        description: "A blog post entry which has an author",
        sharing: Sharing::Public,

        validation_package: || {
            hdk::ValidationPackageDefinition::Entry
        },

        validation: |_validation_data: Validation<Post>| {
            Ok(())
        }
    )
}

define_zome! {
    entries: [
        post_definition()
    ]

    genesis: || {
        Ok(())
    }

    functions: []

    capabilitites: {}
}
#}

Use of this technique can help you write clean, modular code.

If you want to look closely at a complete example of the use of entry! in a Zome, check out the API reference, or the "app-spec" example app.

Summary

This is still a pretty minimal Zome, since it doesn't have any functions yet, and the most basic genesis behaviour, so read on to learn about how to work with those aspects of define_zome!.

Genesis

The Initialization Process

Recall that every peer will be running instances of DNA on their device. This is how peers join the network for a given DNA. There are some actions that a developer may wish to initiate, upon a new peer joining a network. For this reason, a hook into this moment of the lifecycle is implemented by Holochain. It also provides an opportunity to reject the user from joining, if for some reason there is an issue with the way they are attempting to.

This lifecycle stage is called genesis. It is a callback that Holochain expects every single Zome to implement, because it will call it during initialization. If it does not exist within the WASM code for a Zome it will cause an error and peers will not be able to launch an instance of the DNA.

This function also has the opportunity to reject the success of the launch of the instance. If it succeeds, the expected return value is just an integer (in WASM) representing that, but if it fails, a string is expected to be passed, explaining why.

When Holochain is attempting to launch an instance of a Zome, it will iterate through all the Zomes one by one, calling genesis within each. If each succeeds, success. If any one fails, the launch will fail, and the error string will be returned to the peer.

Holochain will wait up to 30 seconds for a genesis response from the Zome, before it will throw a timeout error.

Of course, this also indicates that genesis is a reserved function name and should not be used as the name of any other function that is publicly callable in the Zome.

Building in Rust: genesis

How is genesis used within the Rust HDK?

Previously, the general structure of define_zome! has been covered. It includes a Rust function closure called genesis, which is passed zero arguments. This is the hook that Holochain is expecting. It expects a Rust Result as a return value, which is either Ok(()) or an Err, with the string explaining the error.

In the following two examples, nothing interesting will happen in the genesis functions, they are simply to illustrate how to return success, and how to return failure.

More complex capabilities will be possible during genesis in the future, yet for now, using the first simple example that succeeds is recommended.

If genesis should succeed:


# #![allow(unused_variables)]
#fn main() {
define_zome! {
    entries: []

    genesis: || {
        Ok(())
    }

    functions: []

    traits: {}
}
#}

If genesis should fail:


# #![allow(unused_variables)]
#fn main() {
define_zome! {
    entries: []

    genesis: || {
        Err("the error string".to_string())
    }

    functions: []

    traits: {}
}
#}

Zome Functions

It is time to address the core application logic of Zomes.

What Zome functions you write depends, of course, on what you are building your application to do. By exposing a number of native capacities of Holochain to Zomes, developers have been given access to a rich suite of tools that offer limitless ways they can be combined. Holochain achieves this by exposing core functions to the WASM code of Zomes.

A core feature of an HDK is that it will handle that native interface to Holochain, wrapping the underlying functions in easy to call, well defined and well documented functions that are native to the language that you're writing in!

So within Zome code, by calling functions of the HDK, you can do powerful things with Holochain, like:

  • Read and write data to and from the configured storage mechanism, and from the network
  • Transport messages directly between nodes of a network
  • Call functions in other Zomes, and even "bridged" DNA instances
  • Use cryptographic functions like signing and verification to handle data security and integrity
  • Emit "signals" containing data from a Zome, to a UI for example

How these different functions work and how to use them will be covered throughout the rest of this chapter in detail. This article will provide a general overview of what Zome functions themselves are and how they work.

Recall that Zomes will be written in diverse programming languages, any one that compiles to WebAssembly. Towards the bottom of this article, "Building in Rust" gives examples of what writing functions in Rust will be like. It is difficult to show what a function in WASM looks like, since even the "human-readable" version of WASM, WAT, is not highly readable.

DNA, Zomes, Functions, Traits, and Capabilities

When Holochain loads a DNA file, to start an instance from it, it expects the presence of one or more Zomes in the definition. Here is a skeletal (incomplete) DNA JSON file to illustrate this:

{
    "name": "test",
    "zomes": {
        "test_zome": {
            "name": "test_zome",
            "traits": {
                "hc_public": {
                    "functions": [],
                }
            },
            "fn_declarations": [],
            "code": {
                "code": "AAECAw=="
            }
        }
    }
}

This theoretical DNA has one Zome, "test_zome". However, it has no functions. Note that the nested fn_declarations property is an empty array.

There are a few things to learn from this DNA JSON. The first, is that the code, Base64 encoded WASM, is actually embedded in the Zome's definition, nested under code.code. All the functions Holochain expects to be implemented need to be encapsulated within that WASM code.

The second is that even outside of the WASM code, Holochain expects a certain level of visibility into the functions contained within, at least the ones meant to be called via Holochain (as oppose to private/internal functions).

There are at least two reasons for this:

  • to be able to reason about data inputs and outputs for those functions
  • to group those functions semantically for composition

These will both be discussed below.

Function Declarations

All of the Zome's functions are declared in the fn_declarations array. Here is an example of one:

"fn_declarations": [
    {
        "name": "get_task_list",
        "inputs": [{"name": "username", "type": "string"}],
        "outputs": [{"name": "task_list", "type": "json"}]
    }
]

Each function declaration is an object that includes the name, and the inputs and outputs expected for the function. Since WebAssembly only compiles from code languages with a type system, the generation of these inputs and outputs can expected to be automated.

The name is the most important thing here, because when a function call to an instance is being performed, it will have to match a name which Holochain can find in the functions. If the function isn't declared, Holochain will treat it as if it doesn't exist, even if it is an exposed function in the WASM code.

Traits

Traits provide a way to group functions by name. The primary use of this feature is for creating a composibility space where DNA creators can implement different DNAs to emergent function interfaces and then compose with them in the conductor by matching on the function group names and signatures. Additionally Holochain may reserve a few special trait names that have specific side-effects. The first of such reserved names is hc_public. Functions grouped in this name will have automatically added to a public capability grant that happens at genesis time, thus making them accessible to any caller. For more details on the Holochain security model please see the Capabilities section.

Here is an example of what a trait definition using the public reserved trait name might look like:

"traits": {
    "hc_public": {
        "functions": ["get_task_list"]
    }
}

Data Interchange - Inputs and Outputs

In order to support building zomes in a variety of languages, we decided to use a simple language agnostic function specification format, using JSON. Other formats may be supported in the future.

This has two big implications: Holochain Conductor implementations must handle JSON serialization and deserialization on the "outside", and HDKs and Zomes must handle JSON serialization and deserialization on the "inside". Holochain agrees only to mediate between the two by passing a string (which should represent valid JSON data).

How Zome Functions Are Called

Function calls are received by Holochain from client requests (which there are a variety of implementations of, discussed later). When function calls are being made, they will need to include a complete enough set of arguments to know the following:

  • which Zome?
  • which function?
  • what values should the function be called with?

Before making the function call, Holochain will check the validity of the request, and fail if necessary. If the request is deemed valid, Holochain will mount the WASM code for a Zome using its' WASM interpreter, and then make a function call into it, giving it the arguments given to it in the request. When it receives the response from the WASM, it will then pass that return value as the response to the request. This may sound complex, but that's just what's going on internally, actually using it with an HDK and a Conductor (which is discussed later) is easy.

Building in Rust: Zome Functions

So far, in entry type definitions and genesis, the most complex example of define_zome! was still very simple, and didn't include any functions:


# #![allow(unused_variables)]
#fn main() {
...

#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct Post {
    content: String,
    date_created: String,
}

define_zome! {
    entries: [
        entry!(
            name: "post",
            description: "A blog post entry which has an author",
            sharing: Sharing::Public,
            native_type: Post,
            validation_package: || {
                ValidationPackageDefinition::Entry
            },
            validation: |_post: Post, _validation_data: ValidationData| {
                Ok(())
            }
        )
    ]

    genesis: || {
        Ok(())
    }

    functions: []
}
#}

functions is where the function declarations will be made.

Adding Traits:

Here are some sample traits


# #![allow(unused_variables)]
#fn main() {
...

define_zome! {
    ...
    traits: {
        hc_public [read_post]
        authoring [create_post, update_post]
    }
}
#}

In this example, hc_public is the reserved trait name which creates a Public Capbility-type grant at genesis time for access to the read_post function. Additionally it names an authoring trait for the create_post and update_post functions.

Adding a Zome Function

In order to add a Zome function, there are two primary steps that are involved.

  1. declare your function in define_zome!
  2. write the Rust code for the handler of that function, calling any HDK functions you need

Step 1

The functions section looks a bit like an array of key-value pairs:


# #![allow(unused_variables)]
#fn main() {
...

define_zome! {
    ...
    functions: [
        send_message: {
            inputs: |to_agent: Address, message: String|,
            outputs: |response: ZomeApiResult<String>|,
            handler: handle_send_message
        }
    ]
}
#}

In this example, send_message is the given name of this function, by which it will be referenced and called elsewhere. There are three properties necessary to provide send_message, and any function declaration: inputs, outputs, and handler.

inputs expects a list or argument names, and types, for the send_message function to be called with.

outputs expects a single declaration of a return type. The name (which in the example is response) is arbitrary, call it anything.

handler expects the name of a function which will handle this function call, and which matches the function signature of inputs and outputs. In this case, handle_send_message, which has yet to be defined.

Step 2

Here is an example of a simplistic function, for illustration purposes. It centers on the use of a function call to an HDK function.


# #![allow(unused_variables)]
#fn main() {
fn handle_send_message(to_agent: Address, message: String) -> ZomeApiResult<String>  {
    hdk::send(to_agent, message, 60000.into())
}
#}

Notice right away how the arguments match perfectly with the inputs: |...| section of the function declaration. Any differences will cause issues. This is also true of the return type of the output. Note the pairing of ZomeApiResult<String> as the return type.

The name of the function, handle_send_message is the same as the name given as the handler in the define_zome! function declaration.

Within the function, handle_send_message makes use of a Holochain/HDK function that sends messages directly node-to-node.

The available functions, their purpose, and how to use them is fully documented elsewhere, in the API reference and the List of API Functions.

In the example, handle_send_message simply forwards the result of calling hdk::send as its' own result.

Here are the above two steps combined:


# #![allow(unused_variables)]
#fn main() {
...

fn handle_send_message(to_agent: Address, message: String) -> ZomeApiResult<String>  {
    hdk::send(to_agent, message, 60000.into())
}

define_zome! {
    ...
    functions: [
        send_message: {
            inputs: |to_agent: Address, message: String|,
            outputs: |response: ZomeApiResult<String>|,
            handler: handle_send_message
        }
    ]
}
#}

To see plenty of examples of adding functions, check out a file used for testing the many capacities of the HDK.

Adding Traits:

Here are some sample traits


# #![allow(unused_variables)]
#fn main() {
...

define_zome! {
    ...
    traits: {
        hc_public [read_post]
        authoring [create_post, update_post]
    }
}
#}

In this example, hc_public is the reserved trait name which create a Public Capability-type grant at genesis time for access to the read_post function. Additionally it names an authoring trait the create_post and update_post functions.

Continue reading to learn all about the API Functions and examples of how to use them.

Capabilities

Overview

Holochain uses a modified version of the capabilities security model. Holochain DNA instances will grant revokable, cryptographic capability tokens which are shared as access credentials. Appropriate access credentials must be used to access functions and private data.

This enables us to use a single security pattern for:

  • connecting end-user UIs,
  • calls across zomes within a DNA,
  • bridging calls between different DNAs,
  • and providing selective users of a DNA the ability to query private entries on the local chain via send/receive.

Each capability grant gets recorded as a private entry on the grantor’s chain. The hash (i.e. address) of that entry is then serves as the capability token usable by the grantee when making zome function call, because the grantor simply verifies the existence of that grant in it's chain. Thus, all zome functions calls include a capability request object which contains: public key of the grantee and signature of the parameters being used to call the function, along with the capability token being used as the access credential.

Using Capabilities

Public Capabilities

You can declare some functions as "public" using the special hc_public marker trait in your define_zome! call. Functions in that trait will be added to the public capability grant which gets auto-committed during genesis. Like this:

define_zome! {

...

   traits: {
       hc_public [read_post, write_post]
   }

...

}

Grant Capabilities

You can use the commit_capability_grant HDK function to create a custom capability grant. For example, imaging a blogging use-case where you want to grant friends the ability to call the create_post function in a blog zome. Assuming the function is_my_friend(addr) correctly examines the provenance in CAPABILITY_REQ global which always holds the capability request of the current zome call, then the following code is an example of how you might call hdk::commit_capability_grant:


# #![allow(unused_variables)]
#fn main() {
pub fn handle_request_post_grant() -> ZomeApiResult<Option<Address>> {
    let addr = CAPABILITY_REQ.provenance.source();
    if is_my_friend(addr.clone()) {
        let mut functions = BTreeMap::new();
        functions.insert("blog".to_string(), vec!["create_post".to_string()]);
        Ok(Some(hdk::commit_capability_grant(
            "can_post",
            CapabilityType::Assigned,
            Some(vec![addr]),
            functions,
        )?))
    } else {
        Ok(None)
    }
}
#}

Capabilities in Bridging

TBD.

Read & Write Data Operations

Entry Validation

Linking

Node to Node Messaging

Calling Other Zomes

Crypto Functions

Holochain DNA instances are designed to function in the context of a Distributed Public Key Insfrastructure (DPKI) which:

  1. manages key creation and revocation
  2. manages an agency context that allows grouping and verifying that sets of DNA instances are controlled by the same agent.
  3. creates a context for identity verification

Holochain assumes that there may be different DPKI implementations but provides a reference implementation we call DeepKey. We assume that the DPKI implementation is itself a Holochain application, and we provide access to a set of generic cryptographic functions. These functions also allow DNA authors to build ad-hoc cryptogrpahic protocols.

For each Holochain DNA instance, the conductor maintains a Keystore, which holds "secrets" (seeds and keys) needed for cryptographic signing and encrypting. Each of the secrets in the Keystore is associated with a string which is a handle needed when using that secret for some cryptographic operation. Our cryptographic implementation is based on libsodium, and the seeds use their notions of context and index for key derivation paths. This implementation allows DNA developers to securely call cryptographic functions from wasm which will be executed in the conductor's secure memory space when actually doing the cryptographic processing.

Here are the available functions:

  • keystore_list() -> returns a list of all the secret identifiers in the keystore
  • keystore_new_random(dst_id) -> creates a new random root seed identified by dst_id
  • keystore_derive_seed(src_id,dst_id,context,index) -> derives a higherarchical deterministic key seed to be identifided by dst_id from the src_id. Uses context and index as part of the derivation path.
  • keystore_derive_key(src_id,dst_id,key_type)-> derives a key (signing or encrypting) to be identified by dst_id from a previously created seed identified by src_id. This function returns the public key of the keypair.
  • keystore_sign(src_id,payload) -> returns a signature of the payload as signed by the key identified by src_id
  • keystore_get_public_key(src_id) -> returns a the public key of a key identified by src_id. Returns an error if you pass an identifier of a seed secret.
  • sign(payload) -> signs the payload using the DNA's instance agent ID public key. This is a convenience function which is equivalent to calling keystore_sign("primary_keybundle:sign_key",payload)
  • sign_one_time(payload_list) -> signs the payloads with a randomly generated key-pair, returning the signatures and the public key of the key-pair after shredding the private-key.
  • verify_signature(provenance, payload) -> verifies that the payload matches the provenance which is a public key/signature pair.

Not Yet Implemented:

  • encrypt(payload) -> encrypts
  • keystore_encrypt(src_id,payload)

Bundling

Emitting Signals

API DNA Variables

Note: Full reference is available in language-specific API Reference documentation.

For the Rust hdk, see here

Name Purpose
DNA_NAME Name of the Holochain DNA taken from the DNA.
DNA_ADDRESS The address of the DNA
AGENT_ID_STR The identity string used to initialize this Holochain
AGENT_ADDRESS The address (constructed from the public key) of this agent.
AGENT_INITIAL_HASH The hash of the first identity entry on the local chain.
AGENT_LATEST_HASH The hash of the most recent identity entry that has been committed to the local chain.
CAPABILITY_REQ The capability request that was used to run the zome call

Zome API Functions

Overview

A Zome API Function is any Holochain core functionality that is exposed as a callable function within Zome code.

Compare this to a Zome Callback Function, which is implemented by the Zome code and called by Holochain.

So, Zome functions (functions in the Zome code) are called by Holochain, which can optionally call Zome API Functions, and then finally return a value back to Holochain.

Holochain "blocks" (meaning it pauses further execution in processor threads)
  -> calls Zome function
  -> executes WASM logic compiled from Zome language
  -> Zome logic calls zome API function
    -> Holochain natively executes Zome API function
    -> Holochain returns value to Zome function
  -> Zome function returns some value
  -> Holochain receives final value of Zome function

Each Zome API Function has a canonical name used internally by Holochain.

Zome code can be written in any language that compiles to WASM. This means the canonical function name and the function name in the Zome language might be different. The Zome language will closely mirror the canonical names, but naming conventions such as capitalisation of the zome language are also respected.

For example, the canonical verify_signature might become verifySignature in AssemblyScript.

When a Zome API function is called from within Zome code a corresponding Rust function is called. The Rust function is passed the current Zome runtime and the arguments that the zome API function was called with. The Rust function connects Zome logic to Holochain core functionality and often has side effects. The return value of the Rust function is passed back to the Zome code as the return of the Zome API function.

Property

Canonical name: property

Not Yet Available.

Returns an application property, which are defined by the developer in the DNA. It returns values from the DNA file that you set as properties of your application (e.g. Name, Language, Description, Author, etc.).

Entry Address

Canonical name: entry_address

Returns the address that a given entry will hash into. Often used for reconstructing an address for a "base" when calling get_links.

View it in the Rust HDK

Debug

Canonical name: debug

Debug sends the passed arguments to the log that was given to the Holochain instance and returns None.

View it in the Rust HDK

Call

Canonical name: call

Enables making function calls to an exposed function from another app instance via bridging, or simply another Zome within the same instance.

View it in the Rust HDK

Sign

Canonical name: sign

Not yet available.

Enables the signing of some piece of data, with the private keys associated with the acting agent.

Verify Signature

Canonical name: verify_signature

Not yet available.

A "signature" is a piece of data which claims to be signed by the holder of a private key associated with a public key. This function allows that claim to be verified, when given a "signature" and a public key.

Commit Entry

Canonical name: commit_entry

Attempts to commit an entry to your local source chain. The entry will have to pass the defined validation rules for that entry type. If the entry type is defined as public, it will also publish the entry to the DHT. Returns either an address of the committed entry as a string, or an error.

View it in the Rust HDK

Update Entry

Canonical name: update_entry

Commit an entry to your local source chain that "updates" a previous entry, meaning when getting the previous entry, the updated entry will be returned. update_entry sets the previous entry's status metadata to Modified and adds the updated entry's address in the previous entry's metadata. The updated entry will hold the previous entry's address in its header, which will be used by validation routes.

View it in the Rust HDK

Update Agent

Canonical name: update_agent

Not yet available.

Remove Entry

Canonical name: remove_entry

Enables an entry, referred to by its address, to be marked in the chain as 'deleted'. This is done by adding a new entry which indicates the deleted status of the old one. This will changes which types of results that entry would then show up in, according to its new 'deleted' status. It can still be retrieved, but only if specifically asked for. This function also returns the Hash of the deletion entry on completion

View it in the Rust HDK

Get Entry

Canonical name: get_entry

Given an entry hash, returns the entry from a chain or DHT if that entry exists.

Entry lookup is done in the following order:

  • The local source chain
  • The local hash table
  • The distributed hash table

Caller can request additional metadata on the entry such as type or sources (hashes of the agents that committed the entry).

Get Links

Canonical name: get_links

Consumes three values, the first of which is the address of an entry, base, the remaining two are Optional types for the link_type and tag. Passing Some("string") will return only links that match the type/tag exactly. Passing None for either of those params will return all links regardless of the type/tag. Returns a list of addresses of other entries which matched as being linked by the given link type. Links are created in the first place using the Zome API function link_entries. Once you have the addresses, there is a good likelihood that you will wish to call get_entry for each of them.

Link Entries

Canonical name: link_entries

Consumes four values, two of which are the addresses of entries, and two of which are strings that determine which link_type to use and a tag string that should be added to the link. The link_type must exactly match a type defined in an entry! macro. The tag can be any arbitrary string. Later, lists of entries can be looked up by using get_links and optionally filtered based on their type or tag. Entries can only be looked up in the direction from the base, which is the first argument, to the target, which is the second. This function returns a hash for the LinkAdd entry on completion.

View it in the Rust HDK

Query

Canonical name: query

Returns a list of addresses of entries from your local source chain, that match a given entry type name, or a vector of names. You can optionally limit the number of results, and you can use "glob" patterns such as "prefix/*" to specify the entry type names desired.

View it in the Rust HDK

Send

Canonical name: send

Sends a node-to-node message to the given agent. This works in conjunction with the receive callback, which is where the response behaviour to receiving a message should be defined. This function returns the result from the receive callback on the other side.

View it in the Rust HDK

Grant Capability

Canonical name: commit_capability_grant

Creates a capability grant on the local chain for allowing access to zome functions.

View it in the Rust HDK

Start Bundle

Canonical name: start_bundle

Not yet available.

Close Bundle

Canonical name: close_bundle

Not yet available.

Building Holochain Apps: Packaging

The hc package command will automate the process of compiling your Zome code, encoding it, and inserting into the .dna.json file. In order to get these benefits, you just need to make sure that you have the right compilation tools installed on the machine you are using the command line tools from, and that you have the proper configuration files in your Zome folders.

hc package works with two special files called .hcignore files and .hcbuild files.

.hcbuild Files

In the process of building a .dna.json file during packaging, here is what Holochain does:

  • It iterates Zome by Zome adding them to the JSON
  • For each Zome, it looks for any folders containing a .hcbuild file
  • For any folder with a .hcbuild file, it executes one or more commands from the .hcbuild file to create a WASM file
  • It takes that built WASM file and Base64 encodes it, then stores a key/value pair for the Zome with the key as the folder name and the encoded WASM as the value

When using hc generate to scaffold a Zome, you will have a .hcbuild file automatically. If you create your Zome manually however, you will need to create the file yourself. Here's the structure of a .hcbuild file, using a Rust Zome which builds using Cargo as an example:

{
  "steps": {
    "cargo": [
      "build",
      "--release",
      "--target=wasm32-unknown-unknown"
    ]
  },
  "artifact": "target/wasm32-unknown-unknown/release/code.wasm"
}

The two top level properties are steps and artifact.

steps is a list of commands which will be sequentially executed to build a WASM file.

artifact is the expected path to the built WASM file.

Under steps, each key refers to the bin(ary) of the command that will be executed, such as cargo. The value of cargo, the command, is an array of arguments: build, and the two -- flags. In order to determine what should go here, just try running the commands yourself from a terminal, while in the directory of the Zome code.

That would look, for example, like running:

cargo build --release --target=wasm32-unknown-unknown

Building in Rust: Rust -> WASM compilation tools

If we take Zome code in Rust as an example, you will need Rust and Cargo set up appropriately to build WASM from Rust code. To enable it, run the following:

$ rustup target add wasm32-unknown-unknown

This adds WASM as a compilation target for Rust, so that you can run the previously mentioned command with --target=wasm32-unknown-unknown.

Ignoring Files Using A .hcignore File

Sometimes, you'll want to exclude files and folders in your project directory to get a straight .dna.json file that can be understood by Holochain. In order to do that, just create a .hcignore file. It has a similar structure to .gitignore files:

README.md
dist
.DS_Store

The hc package command includes patterns inside .gitignore files automatically, so you don't have to write everything twice. Also hidden files are ignored by default as well.

Because hc package will attempt to package everything in the directory that is not explicitly ignored, Holochain will return an error if the DNA package is malformed. It is a common mistake to forget to exclude files or folders in the .hcignore file, so that your DNA will be valid.

Building Holochain Apps: Testing

In order to provide a familiar testing framework, a nodejs version of the Holochain framework has been compiled using Rust to nodejs bindings. It is called "holochain-nodejs" and is a publicly installable package on the NPM package manager for nodejs. It enables the execution of Holochain and DNA instances from nodejs.

At a basic level, here is how testing the Holochain DNA you are developing works:

  • Use the hc test command to run a series of steps optimal for testing
  • call a JS file containing tests
  • In the JS file, import the nodejs Holochain Conductor
  • load your packaged DNA into the Conductor, and otherwise configure it
  • use exposed methods on the Conductor to make function calls to the DNA
  • check that the results are what you expect them to be

For checking the results, a basic JavaScript test framework called Tape has received priority support thus far, but other test frameworks can be used.

You have the flexibility to write tests in quite a variety of ways, open to you to explore. This chapter will overview how to approach testing Holochain DNA.

Running Tests

By default, when you use hc init to create a new project folder, it creates a sub-directory called test. The files in that folder are equipped for testing your project. The contents of the folder represent a simple nodejs package (in that they have a index.js file and a package.json file).

Tools to help with testing are also built right into the development command line tools.

hc test

Once you have a project folder initiated, you can run hc test to execute your tests. This combines the following steps:

  1. Packaging your files into a DNA file, located at dist/your.dna.json. This step will fail if your packaging step fails.
  2. Installing build and testing dependencies, if they're not installed (npm install)
  3. Executing the test file found at test/index.js (node test/index.js)

The tests can of course be called manually using nodejs, but you will find that using the convenience of the hc test command makes the process much smoother, since it includes the packaging step for when you change the DNA source files.

hc test also has some configurable options.

If you want to run it without repackaging the DNA, run it with

hc test --skip-package

If your tests are in a different folder than test, run it with

hc test --dir tests

where tests is the name of the folder.

If the file you wish to actually execute is somewhere besides test/index.js then run it with

hc test --testfile test/test.js

where test/test.js is the path of the file.

Intro to holochain-nodejs

The purpose of the holochain-nodejs module is to make integration tests and scenario tests able to be written simply and with as little boilerplate as possible. However, the module also provides even more basic functionality, making it possible to build tests with whatever tradeoff between convenience and customization is right for your project.

There are two primary capabilities of the module, which are introduced below.

Simple, Single Node Integration Tests

The point of this mode of testing is simply to call Zome functions, and ensure that they produce the result you expect. It is discussed further in calling zome functions and checking results.

Scenario Tests

The point of this mode of testing is to launch multiple instances, call functions in one, then another, and to ensure that processes involving multiple agents play out as intended. The module conveniently provides a way to sandbox the execution of these scenarios as well, so that you can test multiple without worrying about side effects. This is discussed further in scenario testing.

Configuration

Config is an object with helper functions for configuration that is exported from holochain-nodejs and can be imported into your code. The functions can be combined to produce a valid configuration object to instantiate a Conductor instance with.

Import Example

const { Config } = require('@holochain/holochain-nodejs')

Agent

Config.agent(agentName) => object

Takes an agent name and creates a simple configuration object for that agent


Name agentName

Type string

Description An identifying string for this agent


Example

const agentConfig = Config.agent('alice')
console.log(agentConfig)
/*
{
    name: 'alice'
}
*/

DNA

Config.dna(dnaPath, [dnaName]) => object

Takes a path to a valid DNA package, and optionally a name and creates a simple configuration object for that DNA


Name dnaPath

Type string

Description The path to a .dna.json file containing a valid DNA configuration


Name dnaName Optional

Type string

Description The path to a .dna.json file containing a valid DNA configuration

Default The same as the given dnaPath


Example

const dnaConfig = Config.dna('path/to/your.dna.json')
console.log(dnaConfig)
/*
{
    path: 'path/to/your.dna.json',
    name: 'path/to/your.dna.json'
}
*/

Instances

Config.instance(agentConfig, dnaConfig, [name]) => object

Takes an agent config object and a dna confid object, and optionally a unique name, and returns a full configuration object for a DNA instance.


Name agentConfig

Type object

Description A config object with a name property, as produced by Config.agent


Name dnaConfig

Type object

Description A config object with a name and path property, as produced by Config.dna


Name name Optional

Type string

Description The name acts like the instance ID, and in fact will be used as such when calling Zome functions

Default The same as the name property of the given agentConfig (agentConfig.name)


Example

const agentConfig = Config.agent('alice')
const dnaConfig = Config.dna('path/to/your.dna.json')
const instanceConfig = Config.instance(agentConfig, dnaConfig)
console.log(dnaConfig)
/*
{
    agent: {
        name: 'alice'
    },
    dna: {
        path: 'path/to/your.dna.json',
        name: 'path/to/your.dna.json'
    },
    name: 'alice'
}
*/

Bridges

Config.bridge(handle, callerInstanceConfig, calleeInstanceConfig) => object

Takes three arguments: the bridge handle, the caller, and the callee (both instances)


Name handle

Type string

Description The desired bridge handle, which is used by the "caller" DNA to refer to the "callee" DNA. See the bridging section of the docs for more detail.


Name callerInstanceConfig

Type object

Description A config object as produced by Config.instance, which specifies the instance which will be making calls over the bridge


Name calleeInstanceConfig

Type object

Description A config object as produced by Config.instance, which specifies the instance which will be receiving calls over the bridge


Example

const agentConfig1 = Config.agent('alice')
const agentConfig2 = Config.agent('bob')
const dnaConfig = Config.dna('path/to/your.dna.json')
const instanceConfig1 = Config.instance(agentConfig1, dnaConfig)
const instanceConfig2 = Config.instance(agentConfig2, dnaConfig)
const bridgeConfig = Config.bridge('bridge-handle', instanceConfig1, instanceConfig2)
console.log(bridgeConfig)
/*
{ handle: 'bridge-handle',
  caller_id: 'alice',
  callee_id: 'bob' }
*/

DPKI

Config.dpki(instanceConfig, initParams) => object

Takes two arguments: an instance object, as specified by Config.instance, and an object which gets passed into the init_params conductor config object.


Name instanceConfig

Type object

Description A config object with a name property, as produced by Config.instance


Name initParams

Type object

Description A config object which will be passed directly through to the conductor config (as dpki.init_params)

Full Conductor Configuration

Config.conductor(conductorOptions) => object

Config.conductor(instancesArray, [conductorOptions]) => object

There are two ways to construct a valid Conductor configuration from these Config helpers. Using the first way, you put all the config data into a single required object. Using the second "shorthand" style, you specify an array of Config.instance data, along with an optional object of extra options. The second way can be more convenient when you are just trying to set up a collection of instances with nothing extra options.

Consumes an array of configured instances and produces an object which is a fully valid Conductor configuration. It can be passed into the Conductor constructor, which is covered in the next articles.

This function is mostly useful in conjunction with manually instantiating a Conductor.


Name conductorOptions Optional

Type object

Description conductorOptions.instances array Pass in an array of instance configuration objects generated by Config.instance to have them within the final configuration to be instantiated by the Conductor. Note: If using the two-argument "shorthand" style of Config.conductor, the first instancesArray argument will override this property.

Description conductorOptions.bridges array Pass in an array of instance configuration objects generated by Config.bridges to have them within the final configuration to be instantiated by the Conductor

Description conductorOptions.debugLog boolean Enables debug logging. The logger produces nice, colorful output of the internal workings of Holochain.

Default { debugLog: false }


Name instancesArray

Type array

Description When using the two-argument "shorthand" style of Config.conductor, you can specify the list of instances as the first argument, rather than folding it into the conductorOptions object.


Example

const agentConfig = Config.agent('alice')
const dnaConfig = Config.dna('path/to/your.dna.json')
const instanceConfig = Config.instance(agentConfig, dnaConfig)
const conductorConfig = Config.conductor({
    instances: [instanceConfig]
})

Or, equivalently, using the shorthand style:

const conductorConfig = Config.conductor([instanceConfig])

Example With conductorOptions

const agentConfig = Config.agent('alice')
const dnaConfig = Config.dna('path/to/your.dna.json')
const instanceConfig = Config.instance(agentConfig, dnaConfig)
const conductorConfig = Config.conductor({
    instances: [instanceConfig],
    debugLog: true
})

Or, equivalently, using the shorthand style:

const conductorConfig = Config.conductor([instanceConfig], {debugLog: true})

Multiple Instances Example, with Bridges

const { Config } = require('@holochain/holochain-nodejs')

// specify two agents...
const aliceName = "alice"
const bobName = "bob"
const agentAlice = Config.agent(aliceName)
const agentBob = Config.agent(bobName)
// ...and one DNA...
const dnaPath = "path/to/happ.dna.json"
const dna = Config.dna(dnaPath)
// ...then make instances out of them...
const instanceAlice = Config.instance(agentAlice, dna)
const instanceBob = Config.instance(agentBob, dna)

const bridgeForward = Config.bridge('bridge-forward', instanceAlice, instanceBob)
const bridgeBackward = Config.bridge('bridge-backward', instanceAlice, instanceBob)

// ...and finally throw them all together
const config = Config.conductor({
    instances: [instanceAlice, instanceBob],
    bridges: [bridgeForward, bridgeBackward]
})

Configuration Alternatives

It is possible to use the same configuration as you would for the holochain Conductor, and pass it to the constructor for Conductor. The configuration may be a string of valid TOML, or a JavaScript object with the equivalent structure. To review the configuration, go here.

To see some examples of what these configuration files can look like, you can check out this folder on GitHub.

Using a Plain Old Javascript Object

const { Conductor } = require('@holochain/holochain-nodejs')
const conductor = new Conductor({
    agents: [],
    dnas: [],
    instances: [],
    bridges: [],
    // etc...
})

Using TOML

const { Conductor } = require('@holochain/holochain-nodejs')
const toml = `
[[agents]]
<agent config>

[[dnas]]
<dna config>

[[instances]]
...etc...
`
const conductor = new Conductor(toml)

Scenario Testing

Scenario is a class that is exported from holochain-nodejs and can be imported into your code. It can be used to run tests individually for a single node, or to orchestrate multi-node tests, which is why it is called Scenario. It does all the work of starting and stopping conductors and integrating with various test harnesses.

Import Example

const { Scenario } = require('@holochain/holochain-nodejs')

Scenario Testing Setup

constructor(instancesArray, conductorOptions) => Scenario

Instantiate a Scenario with an array of instance configurations and some optional configuration overrides. Note that this function has the same signature as Config.conductor.


Name instancesArray

Type array

Description Pass in an array of instance configuration objects generated by Config.instance to have them within the final configuration to be instantiated by the Conductor


Name conductorOptions Optional

Type object

Description conductorOptions.debugLog boolean Enables debug logging. The logger produces nice, colorful output of the internal workings of Holochain.

Default { debugLog: false }


Example

const dna = Config.dna("path/to/happ.dna.json")
const instanceAlice = Config.instance(Config.agent("alice"), dna)
const instanceBob = Config.instance(Config.agent("bob"), dna)
const scenario = new Scenario([instanceAlice])

With conductorOptions Example

const dna = Config.dna("path/to/happ.dna.json")
const instanceAlice = Config.instance(Config.agent("alice"), dna)
const instanceBob = Config.instance(Config.agent("bob"), dna)
const scenario = new Scenario([instanceAlice], { debugLog: true })

Inject Tape Version

Scenario.setTape(tape) => null

setTape should be called prior to usage of the Scenario class, if you intend to use tape as your testing framework. It sets a reference internally to the version of tape, since there are many variations, that you wish to use. It is used in conjunction with Scenario.runTape.


Name tape

Type object

Description A reference to the imported tape package you wish to use as your test framework.


Example

const { Config, Scenario } = require('@holochain/holochain-nodejs')

Scenario.setTape(require('tape'))

Full Multiple Instances Example

The following example shows the simplest, most convenient way to start writing scenario tests with this module. We'll set up an environment for running tests against two instances of one DNA, using the tape test harness:

const { Config, Scenario } = require('@holochain/holochain-nodejs')

Scenario.setTape(require('tape'))

// specify two agents...
const agentAlice = Config.agent("alice")
const agentBob = Config.agent("bob")
// ...and one DNA...
const dna = Config.dna("path/to/happ.dna.json")
// ...then make instances out of them...
const instanceAlice = Config.instance(agentAlice, dna)
const instanceBob = Config.instance(agentBob, dna)

// Now we can construct a `scenario` object which lets us run as many scenario tests as we want involving the two instances we set up:
const scenario = new Scenario([instanceAlice, instanceBob])

Run Tests With Tape

scenario.runTape(description, runner) => null

Each invocation of scenario.runTape does the following:

  1. Starts a fresh Conductor based on the configuration used to construct scenario
  2. Starts a new tape test
  3. Injects the values needed for the test into a closure you provide
  4. Automatically ends the test and stops the conductor when the closure is done running

It will error if you have not called Scenario.setTape first.


Name description

Type string

Description Will be used to initialize the tape test, and should describe that which is being tested in this scenario


Name runner

Type function

Description runner is a closure: (t, runner) => { (code to run) }. When this function ends, the test is automatically ended, and the inner Conductor is stopped.

  • t is the object that tape tests use
  • runner is an object containing an interface into each Instance specified in the config. The Instances are keyed by "name", as taken from the optional third parameter of Config.instance, which itself defaults to what was given in Config.agent.

Example

scenario.runTape("test something", (t, runner) => {
    const alice = runner.alice
    const bob = runner.bob
    // fire zome function calls from both agents
    const result1 = alice.call('zome', 'function', {params: 'go here'})
    const result2 = bob.call('zome', 'function', {params: 'go here'})
    // make some tape assertions
    t.ok(result1)
    t.equal(result2, 'expected value')
})

// Run another test in a freshly created Conductor
// This example uses destructuring to show a clean and simple way to get the Instances
scenario.runTape("test something else", (t, {alice, bob}) => {
    // write more tests in the same fashion
})

Other Test Harnesses

Only tape is currently supported as a fully integrated test harness, but you can also run tests with more manual control using scenario.run. Using run allows you to manage the test yourself, only providing you with the basic help of starting and stopping a fresh Conductor instance.

The example does still use tape to show how it compares to using runTape, but it could use any test harness, like Jest or Mocha. In fact, runTape simply calls run internally.

scenario.run(runner) => null

Each invocation of scenario.run does the following:

  1. Starts a fresh Conductor based on the configuration used to construct scenario
  2. Injects the values needed for the test into a closure you provide

Name runner

Type function

Description runner is a closure: (stop, runner) => { (code to run) }.

  • stop is a function that shuts down the Conductor and must be called in the closure body
  • runner is an object containing an interface into each Instance specified in the config. The Instances are keyed by "name", as taken from the optional third parameter of Config.instance, which itself defaults to what was given in Config.agent.

Example

This example does also use tape as an illustration, but each test harness would have its own particular way

const tape = require('tape')

// Create a scenario object in the same fashion as in other examples
const scenario = new Scenario([instanceAlice, instanceBob])

// scenario.run only manages the Conductor for us now, but we have to manage the test itself
scenario.run((stop, runner) => {
    const alice = runner.alice
    const bob = runner.bob
    tape("test something", t => {
        const result = alice.call('zome', 'function', {params: 'go here'})
        t.equal(result, 'expected value')
        // the following two steps were not necessary when using runTape:
        t.end() // end the test
        stop() // use this injected function to stop the conductor
    })
})

// This example uses destructuring to show a clean and simple way to get the Instances
scenario.run((stop, {alice, bob}) => {
    tape("test something else", t => {
        // write more tests in the same fashion
        t.equal(2 + 2, 4)
        t.end()
        stop() // but don't forget to stop the conductor when it's done!
    })
})

DNA Instances

DnaInstance is a class that is exported from holochain-nodejs and can be imported into your code. This class is used externally and instances of it are built automatically for you to use, so you typically should not have to construct a DnaInstance yourself.

A DnaInstance represents a running version of a DNA package by a particular agent. This means that the agent has a source chain for this DNA. In addition to these basic properties on a DnaInstance that are covered below, the following articles cover how to make function calls into the Zomes.

Import Example

const { DnaInstance } = require('@holochain/holochain-nodejs')

Instantiate A DnaInstance

constructor(instanceId, conductor) => DnaInstance

Instantiate a DnaInstance based on an instanceId, and the conductor where an instance with that id is running. Calling this manually is not typically necessary, since the Scenario testing returns these natively. A DnaInstance can make calls via that Conductor into Zome functions.


Name instanceId

Type string

Description The instance id of the DnaInstance as specified in the configuration of conductor. Note that when using the Config.instance helper, the instance ID defaults to the agent name (as specified in Config.agent) if not explicitly passed as a third argument.


Name conductor

Type Conductor

Description A valid, and running Conductor instance


Example


const aliceInstance = new DnaInstance('alice', conductor)

DnaInstance Attributes

dnaInstance.agentId

The agentId for an instance.

Example

console.log(alice.agentId)
// alice-----------------------------------------------------------------------------AAAIuDJb4M

dnaInstance.dnaAddress

The address of the DNA for an instance.

Example

console.log(alice.dnaAddress)
// QmYiUmMEq1WQmSSjbM7pcLCy1GkdkfbwH5cxugGmeNZPE3

Calling Zome Functions

dnaInstance.call(zomeName, functionName, callParams) => object

A DnaInstance can use the Conductor in which it's running to make calls to the custom functions defined in its Zomes. This is necessary in order to be able to test them. It calls synchronously and returns the result that the Zome function provides. An error could also be thrown, or returned.

Note that Holochain has to serialize the actual arguments for the function call into JSON strings, which the Conductor will handle for you automatically. It also parses the result from a JSON string into an object.

This function will only succeed if conductor.start() has been called for the Conductor in which the DnaInstance is running.


Name zomeName

Type string

Description The name of the Zome within that instance being called into


Name functionName

Type string

Description The name of the custom function in the Zome to call


Name callParams

Type object

Description An object which will get stringified to a JSON string, before being passed into the Zome function. The keys of this object must match one-to-one with the names of the arguments expected by the Zome function, or an error will occur.


Example

// ...
scenario.runTape("test something", (t, runner) => {
    const alice = runner.alice
    // scenario.run and scenario.runTape both inject instances
    const callResult = alice.call('people', 'create_person', {name: 'Franklin'})
})

Note that there are some cases where, for the purposes of testing, you may wish to wait for the results of calling a function in one instance, in terms of chain actions like commits and linking, to propogate to the other instances. For this, extra ways of performing calls have been added as utilities. Check them out in handling asynchronous network effects.

Handling Asynchronous Network Effects

In the previous example, we used alice.call() to call a zome function. This returns immediately with a value, even though the test network created by the conductor is still running, sending messages back and forth between agents for purposes of validation and replication, etc. In many test cases, you will want to wait until all of this network activity has died down to advance to the next step.

For instance, take the very common scenario as an example:

  1. Alice runs a zome function which commits an entry, then adds a link to that entry
  2. Bob runs a zome function which attempt to get links, which should include the link added by alice

If the test just uses call() to call that zome function, there is no guarantee that the entries committed by alice.call will be available on the DHT by the time bob.call is started. Therefore, two other functions are available.

alice.callSync returns a Promise instead of a simple value. The promise does not resolve until network activity has completed. alice.callWithPromise is a slightly lower-level version of the same thing. It splits the value apart from the promise into a tuple [value, promise], so that the value can be acted on immediately and the promise waited upon separately.

// If we make the closure `async`, we can use `await` syntax to keep things cleaner
scenario.run(async (stop, {alice, bob}) => {
    tape("test something", t => {
        // we can await on `callSync` immediately, causing it
        // to block until network activity has died down
        const result1 = await alice.callSync('zome', 'do_something_that_adds_links', {})
        // now bob can be sure he has access to the latest data
        const result2 = bob.call('zome', 'get_those_links', {})
        t.equal(result, 'expected value')
        // the following two steps were not necessary when using runTape:
        t.end() // end the test
        stop() // use this injected function to stop the conductor
    })
})

Even though we can't solve the eventual consistency problem in real life networks, we can solve them in tests when we have total knowledge about what each agent is doing.

(E) Checking Results

Managing the Conductor

Conductor is a class that is exported from holochain-nodejs, and can be imported into your code. It is mostly used internally in the library, but can be useful in some of your own use cases.

Import Example

const { Conductor } = require('@holochain/holochain-nodejs')

Simple Use

Conductor.run(conductorConfig, runner) => Promise

Spin up a Conductor with a Conductor configuration. When you're done with it, call stop, a function injected into the closure.


Name conductorConfig

Type string or object

Description should be a TOML configuration string, as described here or an equivalent JavaScript object constructed manually, or setup using the Config helper functions described here.


Name runner

Type function

Description runner is a closure: (stop, conductor) => { (code to run) }

  • stop is a function that shuts down the Conductor and must be called in the closure body
  • conductor is a Conductor instance, from which one can make Instances and thus Zome calls.

Example

// ...
Conductor.run(Config.conductor([
    instanceAlice,
    instanceBob,
    instanceCarol,
]), (stop, conductor) => {
    doStuffWith(conductor)
    stop()
})

Manually Instantiating a Conductor

constructor(conductorConfig) => Conductor

Instantiate a Conductor with a full Conductor configuration.


Name conductorConfig

Type string or object

Description should be a TOML configuration string, as described here or an equivalent JavaScript object constructed manually, or setup using the Config helper functions described here.


Example

// config var can be defined using the Config helper functions
const conductor = new Conductor(config)

Manually Starting and Stopping a Conductor

conductor.start() => null

Start running all instances. No Zome functions can be called within an instance if the instance is not started, so this must be called beforehand.

Example

conductor.start()

conductor.stop() => Promise

Stop all running instances configured for the conductor. This function should be called after all desired Zome calls have been made, otherwise the conductor instances will continue running as processes in the background.

Returns a Promise that you can optionally wait on to ensure that internal cleanup is complete.

Example

conductor.stop()

Access Instance Info

Other info about running Instances in the Conductor can be retrieved via functions on a Conductor.

conductor.agent_id(instanceId) => string

Get the agent_id for an instance, by passing an instance id.


Name instanceId

Type string

Description Specifies an instance by its instanceId. This instanceId should be the equivalent to an instanceConfig.name which was passed to Config.instance. This in turn would be equivalent to the original name given to Config.agent, unless you overrode it when calling Config.instance. See more here.


Example

const aliceAgentId = conductor.agent_id('alice')
console.log(aliceAgentId)
// alice-----------------------------------------------------------------------------AAAIuDJb4M

conductor.dna_address(instanceId) => string

Get the address of the DNA for an instance, by passing an instance id.


Name instanceId

Type string

Description Specifies an instance by its instanceId. This instanceId should be the equivalent to an instanceConfig.name which was passed to Config.instance. This in turn would be equivalent to the original name given to Config.agent, unless you overrode it when calling Config.instance. See more here.


Example

const dnaAddress = conductor.dna_address('alice')
console.log(dnaAddress)
// QmYiUmMEq1WQmSSjbM7pcLCy1GkdkfbwH5cxugGmeNZPE3

Running Holochain Apps: Conductors

To introduce Conductors, it is useful to zoom out for a moment to the level of how Holochain runs on devices.

Holochain was designed to be highly platform and system compatible. The core logic that runs a DNA instance was written in such a way that it could be included into many different codebases as a library, thus making it easier to build different implementations on the same platform as well as across platforms (MacOSX, Linux, Windows, Android, iOS, and more). Architecturally, Holochain DNAs are intended to be small composable units that provide bits of distributed data integrity functionality. Thus most Holochain based applications will actually be assemblages of many "bridged" DNA instances. For this to work we needed a distinct layer that orchestrates the data flow (i.e. zome function call requests and responses), between the transport layer (i.e. HTTP, Websockets, Unix domain sockets, etc) and the DNA instances. We call the layer that performs these two crucial functions, the Conductor, and we have written a conductor_api library to make it easy to build actual Conductor implementations.

Conductors play quite a number of important roles:

  • installing, uninstalling, configuring, starting and stopping instances of DNA
  • exposing APIs to securely make function calls into the Zome functions of DNA instances
  • accepting information concerning the cryptographic keys and agent info to be used for identity and signing, and passing it into Holochain
  • establishing "bridging" between DNA instances
  • serving files for web based user interfaces that connect to these DNA instances over the interfaces

Those are the basic functions of a Conductor, but in addition to that, a Conductor also allows for the configuration of the networking module for Holochain, enables logging, and if you choose to, exposes APIs at a special 'admin' level that allows for the dynamic configuration of the Conductor while it runs. By default, configuration of the Conductor is done via a static configuration file, written in TOML.

In regards to the Zome functions APIs, Conductors can implement a diversity of interfaces to perform these function calls, creating an abundance of opportunity. Another way to build Holochain into an application is to use language bindings from the Rust built version of the Conductor, to another language, that then allows for the direct use of Holochain in that language.

There are currently three Conductor implementations:

  • Nodejs
    • this is built using the language bindings approach, using neon
  • hc run
    • this is a zero config quick Conductor for development
  • holochain executable
    • this is a highly configurable sophisticated Conductor for running DNA instances long term

The articles that follow discuss these different Conductors in greater detail.

What is now known as a "Conductor" used to be called a "Container", so if you see the language of Container from other versions know that these refer to the same thing. Fun fact: because this component has such a variety of functions, there was some difficulty in naming it. The word "Conductor" was finally chosen because it actually implies multiple metaphors, each of which resonates with an aspect of what the Conductor does. Like an orchestra conductor, it helps several parts work together as a whole. Like a train conductor, it oversees and instructs how the engine runs. Like an electricity conductor, it allows information to pass through it.

Development Conductor

The easiest Conductor to run is built right into the development command line tools. It has no required configuration and is launched via the hc run command. Meant primarily for accelerating the development process it is useful for testing APIs or prototyping user interfaces. The hc run command expects to be executed from inside a directory with valid DNA source files: The command is simply:

hc run

This will start the DNA instance in a Conductor and open, by default, a WebSocket JSON-RPC server on port 8888. You can find more details on how to use the API in your UI in the JSON-RPC interfaces article.

The following are the options for configuring hc run, should you need something besides the defaults.

Packaging

-b/--package

Package your DNA before running it. Recall that to package is to build the yourapp.dna.json file from the source files. hc run always looks for a DNA package file in the root of your DNA folder that should have the same name as the directory itself with suffix: .dna.json, so make sure that one exists there when trying to use it. hc run --package will do this, or run hc package beforehand.

example

hc run --package

Storage

--persist

Persist source chain and DHT data onto the file system. By default, none of the data being written to the source chain gets persisted beyond the running of the server. This will store data in the same directory as your DNA source code, in a hidden folder called .hc.

example

hc run --persist

Interfaces

--interface

Select a particular JSON-RPC interface to serve your DNA instance over.

The JSON-RPC interface will expose, via a port on your device, a WebSocket or an HTTP server. It can be used to make function calls to the Zomes of a DNA instance. These are covered in depth in the JSON-RPC interfaces article.

The default interface is websocket.

examples To run it as HTTP, run:

hc run --interface http

To explicitly run it as WebSockets, run:

hc run --interface websocket

Port

-p/--port

Customize the port number that the server runs on.

example

hc run --port 3400

Networking

--networked

Select whether the Conductor should network with other nodes that are running instances of the same DNA. By default this does not occur, instead the instance runs in isolation from the network, allowing only the developer to locally access it.

This option requires more configuration, which can be read about in the configuring networking article.

Stopping the Server

Once you are done with the server, to quit type exit then press Enter, or press Ctrl-C.

Configuring Networking for hc run

hc run uses mock networking by default and therefore doesn't talk to any other nodes.

In order to have hc run spawn a real network instance, start it with the --networked option:

hc run --networked

You should see something like this:

Network spawned with bindings:
     - ipc: wss://127.0.0.1:64518/
     - p2p: ["wss://192.168.0.11:64519/?a=hkYW7TrZUS1hy-i374iRu5VbZP1sSw2mLxP4TSe_YI1H2BJM3v_LgAQnpmWA_iR1W5k-8_UoA1BNjzBSUTVNDSIcz9UG0uaM"]
...

Starting A Second Node

Starting up a second node is a little bit more work:

Step 1

Set the HC_N3H_BOOTSTRAP_NODE environment variable to the external p2p bound address listed by the first node. Copy-paste it from the string from the terminal log of the first node, the one that starts with "/ip4/192.168".

Step 2

Specify a different agent id than the first node, by setting the HC_AGENT environment variable. Since the first agent by default will be testAgent, testAgent2 is suitable.

Step 3

Specify a different port than the first node to run on. Since the port for the first node by default will be 8888, 8889 is suitable.

Running the command could look like this:

HC_AGENT=testAgent2 HC_N3H_BOOTSTRAP_NODE=wss://192.168.0.11:64519/?a=hkYW7TrZUS1hy-i374iRu5VbZP1sSw2mLxP4TSe_YI1H2BJM3v_LgAQnpmWA_iR1W5k-8_UoA1BNjzBSUTVNDSIcz9UG0uaM hc run --port 8889

In the terminal logs that follow, you should see:

(libp2p) [i] QmUmUF..V71C new peer QmeDpQLchA9xeLDJ2jyXBwpe1JaQhFRrnWC2JfyyET2AAM
(libp2p) [i] QmUmUF..V71C found QmeDpQLchA9xeLDJ2jyXBwpe1JaQhFRrnWC2JfyyET2AAM in 14 ms
(libp2p) [i] QmUmUF..V71C ping round trip 37 ms
(libp2p) [i] QmUmUF..V71C got ping, sending pong

This means that the nodes are able to communicate! Watch the logs for gossip, as you take actions (that alter the source chain) in either node.

Production Conductor

In addition to the zero config development Conductor using hc run, there is a highly configurable sophisticated Conductor for running DNA instances long term.

This Conductor will play an important role in making the use of Holochain truly easy for end users, because it supports all the functionality that those users are likely to want, in terms of managing their Holochain apps, or hApps, just on a low level. On that note, a graphical user interface that exposes all the functionality of this Conductor to users is under development.

For now, use of this Conductor must happen mostly manually, and by tech-savvy users or developers.

This Conductor is simply a command line tool called holochain. Its only function is to boot a Conductor based on a configuration file, and optionally, the ability to write changes back to that file. Within that Conductor many DNA instances can be run for one or more agents, multiple types of interfaces to the APIs can be exposed, UI file bundles can be served, and logs from all of that can be accessed.

The first step to using holochain is of course installing it. Instructions for installation can be found in its README. If you wish to attempt any of the things you read in this chapter while going through it, you will need to have installed the executable.

Like Holochain core, this particular Conductor is written in Rust. View it on GitHub here.

To understand how to configure the holochain Conductor, check out the next article.

Intro to TOML Config Files

To configure the holochain Conductor, a configuration file format called TOML is used. It stands for "Tom's Obvious Minimal Language" and was created by Tom Preston-Werner, one of the original founders of GitHub. The documentation on GitHub for it is very good.

holochain configuration files make heavy use of tables and arrays of tables.

A table is actually a collection of key/value pairs, and it looks like this:

[table-1]
key1 = "some string"
key2 = 123

An array of tables looks like this:

[[products]]
name = "Hammer"
sku = 738594937

[[products]]
name = "Nail"
sku = 284758393
color = "gray"

This represents two "product" items in an array.

In the following articles, how to configure the various properties of the holochain Conductor using these will be expanded on. First, knowing how to reference the configuration file for use by holochain will be covered below.

holochain Config Files

holochain requires a configuration file to run, which must exist in the default location, or be provided as an explicit argument. holochain will return an error if neither is given. The default location for the configuration file is in a subdirectory of the HOME directory on a device, at the path:

# Unix (Mac & Linux)
$HOME/.holochain/conductor/conductor-config.toml

# Windows
%HOME%\.holochain\conductor\conductor-config.toml

When executing holochain in a terminal, a path to a configuration file can be given. This can be done with the following option:

--config

or for short

-c

This could look like:

holochain -c ./conductor-config.toml

The holochain-nodejs Conductor also accepts the same TOML based configuration.

Examples

To jump ahead into what these configuration files can look like, you can check out this folder on GitHub which has a number of examples. Otherwise, read on to understand each part.

Agents

agents is an array of configurations for "agents". This means that you can define, and later reference, multiple distinct agents in this single config file. An "agent" has a name, ID, public address and is defined by a private key that resides in a file on their device.

Required: agents is a required property in the config file. It is the ONLY required property.

Properties

id: string

Give an ID of your choice to the agent

name: string

Give a name of your choice to the agent

public_address: string

A public address for the agent. Run hc keygen and copy the public address to this value

keystore_file: string

Path to the keystore file for this agent. Copy the path from when you ran hc keygen into this value.

Example

[[agents]]
id = "test_agent2"
name = "HoloTester2"
public_address = "HcSCJts3fQ6Y4c4xr795Zj6inhTjecrfrsSFOrU9Jmnhnj5bdoXkoPSJivrm3wi"
keystore_file = "/org.holochain.holochain/keys/HcSCJts3fQ6Y4c4xr795Zj6inhTjecrfrsSFOrU9Jmnhnj5bdoXkoPSJivrm3wi"

DNAs

dnas is an array of configurations for "DNAs" that are available to be instantiated in the Conductor. A DNA is a packaged JSON file containing a valid DNA configuration including the WASM code for the Zomes. How to package DNA from source files can be read about here.

Optional

Properties

id: string

Give an ID of your choice to this DNA

file: string

Path to the packaged DNA file

hash: string Optional

A hash can optionally be provided, which could be used to validate that the DNA being installed is the DNA that was intended to be installed.

Example

[[dnas]]
id = "app spec rust"
file = "example-config/app_spec.dna.json"

Instances

instances is an array of configurations of DNA instances, each of which is a running copy of a DNA, by a particular agent. Based on these configurations, the Conductor will attempt to start up these instances, initializing (or resuming) a local source chain and DHT. It is possible to use the same DNA with multiple different agents, but it is not recommended to run two instances with the same DNA and same agent. An instance has a configurable storage property, which can be set to save to disk, or just store temporarily in memory, which is useful for testing purposes.

Optional

Properties

id: string

Give an ID of your choice to this instance

agent: string

A reference to the given ID of a defined agent

dna: string

A reference to the given ID of a defined DNA

storage: StorageConfiguration

A table for configuring the approach to storage of the local source chain and DHT for this instance

StorageConfiguration.type: enum

Select between different storage implementations. There are three so far:

  • memory: Persist actions taken in this instance only to memory. Everything will disappear when the Conductor process stops.
  • file: Persist actions taken in this instance to the disk of the device the Conductor is running on. If the Conductor process stops and then restarts, the actions taken will resume at the place in the local source chain they last were at.
  • pickle : Persists to a fast memory call which is eventually persisted to a file storage every 5 seconds. The actions taken will also resume at the place in the local source chain they were last. If an application error does occur, it will make sure to persist the latest data prior to any shutdown occurring.

StorageConfiguration.path: string

Path to the folder in which to store the data for this instance.

Example

[[instances]]
id = "app spec instance 1"
agent = "test agent 1"
dna = "app spec rust"

    [instances.storage]
    type = "file"
    path = "example-config/tmp-storage"

Interfaces

interfaces is an array of configurations of the channels (e.g. http or websockets) that the Conductor will use to send information to and from instances and users. Interfaces are user facing and make Zome functions, info, and optionally admin functions available to GUIs, browser based web UIs, local native UIs, and other local applications and scripts. The following implementations are already developed:

  • WebSockets
  • HTTP

The instances (referenced by ID) that are to be made available via that interface should be listed. An admin flag can enable special Conductor functions for programatically changing the configuration (e.g. installing apps), which even persists back to the configuration file.

Optional

Properties

id: string

Give an ID of your choice to this interface

driver: InterfaceDriver

A table which should provide info regarding the protocol and port over which this interface should run

InterfaceDriver.type: enum

Select between different protocols for serving the API. There are two so far:

  • websocket: serve the API as JSON-RPC via WebSockets
  • http: serve the API as JSON-RPC via HTTP

These are discussed in great detail in Intro to JSON-RPC Interfaces, and the following articles.

InterfaceDriver.port: u16

An integer value representing the port on the device to run this interface over

admin: bool Optional

Whether to expose admin level functions for dynamically administering the Conductor via this JSON-RPC interface. Defaults to false.

instances: array of InstanceReferenceConfiguration

An array of tables which should provide the IDs of instances to serve over this interface. Only the ones which are listed here will be served.

InstanceReferenceConfiguration.id: string

A reference to the given ID of a defined instance

Example Without Admin

[[interfaces]]
id = "websocket interface"

    [[interfaces.instances]]
    id = "app spec instance 1"

    [interfaces.driver]
    type = "websocket"
    port = 4000

Example With Admin

[[interfaces]]
id = "http interface"
admin = true

    [[interfaces.instances]]
    id = "app spec instance 1"

    [interfaces.driver]
    type = "http"
    port = 4000

Bridges

bridges is an array of configuration instances that are configured to be able to make calls to Zome functions of another instance. You can think of this of as configuring an internal direct interface between DNA instances. The section on bridging provides more information on how this ability is used to to compose complex applications out of many DNA instances.

Optional

Properties

caller_id: string

A reference to the given ID of a defined instance that calls the other one. This instance depends on the callee.

callee_id: string

A reference to the given ID of a defined instance that exposes capabilities through this bridge. This instance is used by the caller.

handle: string

The caller's local handle for this bridge and the callee. A caller can have many bridges to other DNAs and those DNAs could by bound dynamically. Callers reference callees by this arbitrary but unique local name.

Example

[[bridges]]
caller_id = "app1"
callee_id = "app2"
handle = "happ-store"

UI Bundles

ui_bundles is an array of configurations of folders containing static assets, like HTML, CSS, and Javascript files, that will be accessed through a browser and used as a user interface for one or more DNA instances. These are served via UI Interfaces, which is covered next.

Optional

Properties

id: string

Give an ID of your choice to this UI Bundle

root_dir: string

Path to the folder containing the static files to serve

hash: string Optional

A hash can optionally be provided, which could be used to validate that the UI being installed is the UI bundle that was intended to be installed.

Example

[[ui_bundles]]
id = "bundle1"
root_dir = "ui"

UI Interfaces

ui_interfaces is an array of configurations for "UI Interfaces", meaning there can be multiple within one Conductor. UI Interfaces serve UI Bundles over HTTP.

Optional

Properties

id: string

Give an ID of your choice to this UI Interface

bundle: string

A reference to the given ID of a defined ui_bundle to serve over this interface

port: u16

An integer value representing the port on the device to run this interface over. Must not conflict with any of the interface ports, nor another UI Interface port.

dna_interface: string Optional

A reference to the given ID of a defined interface this UI is allowed to make calls to. This is used to set the CORS headers and also to provide an extra virtual file endpoint at /_dna_config/ that allows hc-web-client or another solution to redirect Holochain calls to the correct ip/port/protocol

Example

[[ui_interfaces]]
id = "ui-interface-1"
bundle = "bundle1"
port = 3000
dna_interface = "websocket_interface"

Logging

logger is a table for the configuration of how logging should behave in the Conductor. Select between types of loggers and setup rules for nicer display of the logs. There is only one logger per Conductor.

Optional

Properties

type: enum Optional

Select which type of logger to use for the Conductor. If you leave this off, "simple" logging is the default

  • debug: enables more sophisticated logging with color coding and filters
  • simple: a most minimal logger, no color coding or filtering

rules: LogRules Optional

A table for optionally adding a set of rules to the logger

LogRules.rules: LogRule

An array of tables containing the rules for the logger

LogRule.pattern: Regex string

A Regex pattern as a string to match a log message against, to see whether this rule should apply to it.

LogRule.exclude: bool Optional

Whether to use this pattern to exclude things that match from the logs. Defaults to false. This option is useful for when the logs seem noisy.

LogRule.color: enum Optional

What color to use in the terminal output for logs that match this pattern. Options:

black, red, green, yellow, blue, magenta, cyan, white

Example

[logger]
type = "debug"
    [[logger.rules.rules]]
    color = "red"
    exclude = false
    pattern = "^err/"

    [[logger.rules.rules]]
    color = "white"
    exclude = false
    pattern = "^debug/dna"

    [[logger.rules.rules]]
    exclude = true
    pattern = "^debug/reduce"

    [[logger.rules.rules]]
    exclude = false
    pattern = ".*"

Networking

network is a table for the configuration of how networking should behave in the Conductor. The Conductor currently uses mock networking by default. To network with other nodes Holochain will automatically setup the n3h networking component. How n3h behaves can be configured with the following properties in a Conductor configuration file.

Optional

Properties

n3h_persistence_path: string

Absolute path to the directory that n3h uses to store persisted data. The default is that a temporary self-removing directory for this transient data will be used.

bootstrap_nodes: array of string Optional

List of URIs that point to other nodes to bootstrap p2p connections.

n3h_log_level: char

Set the logging level used globally by N3H. Must be one of the following: 't', 'd', 'i', 'w', 'e' Each value corresponding to the industry standard log level: Trace, Debug, Info, Warning, Error.

n3h_ipc_uri: string Optional

URI pointing to an n3h process that is already running and not managed by this Conductor. If this is set the Conductor does not spawn n3h itself and ignores the path configs above. Default is this value is empty.

Example

[network]
n3h_persistence_path = "/tmp"
bootstrap_nodes = []

Persistence Directory

This is a simple key/value pair specifying a directory on the device to persist the config file, DNAs, and UI bundles, if changes are made dynamically over the JSON-RPC admin API. This is only relevant if you are running one of the interfaces with admin = true. The default value is in a subdirectory of the $HOME directory, $HOME/.holochain/conductor.

Optional

If you start a Conductor that has this value set, but then make no changes via the JSON-RPC admin interface, the persistence directory will not be utilized and the Conductor config file you started with will not be moved into that directory. On the other hand, if you do make any changes to the configuration by calling one of the dynamic admin functions then whatever the value of the persistence_dir is for that Conductor config, it will create that directory, and then persist the modified Conductor configuration file there. It would then be wise to utilize that Conductor config in the future, instead of the original.

Within this persistence_dir that is now on the device, there are a number of possible files and folders.

conductor-config.toml is the new configuration file, which will be repeatedly written to with any further dynamic updates. This is useful so that when the Conductor is stopped, or if it dies for some reason, when you restart it will behave the same as before.

storage is a directory used for persisting the data for instances, in particular when new instances are added via the admin/instance/add admin function.

dna is a directory used for copying DNA package files into if the admin/dna/install_from_file admin function is called.

static is a directory used for copying UI Bundle files into if the admin/ui/install admin function is called.

Example

persistence_dir = "/home/user/my_holochain"

Intro to JSON-RPC Interfaces

The JSON-RPC interface will expose, via a port on your device, a WebSocket or an HTTP server, via which you can make function calls to the Zomes of your DNA.

JSON-RPC

JSON-RPC is a specification for using the JSON data format in a particular way, that follows the "Remote Procedure Call" pattern. Holochain uses the Version 2 specification of JSON-RPC. You can see general examples of JSON-RPC here.

The format for the JSON-RPC request/response pattern is really simple. A request is a JSON object with just a few mandatory values which must be passed.

jsonrpc: specifies the JSON-RPC spec this request follows. The JSON-RPC spec used by Holochain Conductors is 2.0.

id: specifies the ID for this particular request. This is so that the request and response can be matched, even if they get transmitted out of order.

method: specifies the method on the "remote" (Holochain) to call.

params: (optional) contains a JSON object which holds the data to be given as arguments to the method being called, if the method expects them.

Conductor JSON-RPC API

Querying Running DNA Instances

Holochain Conductors expose a method info/instances. This method returns a list of the running DNA instances in the Conductor. For each running instance, it provides the instance "ID", the name of the DNA, and the agent "id". The instance IDs will be particularly useful in other circumstances.

The method info/instances doesn't require any input parameters, so params can be left off the request.

Example

example request

{
    "jsonrpc": "2.0",
    "id": "0",
    "method": "info/instances"
}

example response

{
    "jsonrpc": "2.0",
    "result": [{"id":"test-instance","dna":"hc-run-dna","agent":"hc-run-agent"}],
    "id": "0"
}

Calling Zome Functions

The following explains the general JSON-RPC pattern for how to call a Zome function.

Unlike info/instances, a Zome function call also expects arguments. We will need to include a JSON-RPC args field in our RPC call.

To call a Zome function, use "call" as the JSON-RPC method, and a params object with four items:

  1. instance_id: The instance ID, corresponding to the instance IDs returned by info/instances
  2. zome: The name of the Zome
  3. function: The name of the function
  4. args: The actual parameters of the zome function call

In the last example, the instance ID "test-instance" was returned, which can be used here as the instance ID. Say there was a Zome in a DNA called "blogs", this is the Zome name. That Zome has a function called "create_blog", that is the function name.

Any top level keys of the args field should correspond exactly with the name of an argument expected by the Zome method being called.

Example Zome Function Arguments

{ "blog": { "content": "sample content" }}

Example Request

example request

{
    "jsonrpc": "2.0",
    "id": "0",
    "method": "call",
    "params": {
        "instance_id": "test-instance",
        "zome": "blog",
        "function": "create_blog",
        "args": {
            "blog": {
                "content": "sample content"
            } 
        }
    }
}

example response

{
    "jsonrpc": "2.0",
    "result": "{\"Ok\":\"QmUwoQAtmg7frBjcn1GZX5fwcPf3ENiiMhPPro6DBM4V19\"}",
    "id": "0"
}

This response suggests that the function call was successful ("Ok") and provides the DHT address of the freshly committed blog entry ("QmU...").

HTTP

Any coding language, or tool, which can make HTTP requests can make requests to a running DNA instance. Based on the API exposed by Holochain, these must be POST requests, use the "application/json" Content-Type, and follow the JSON-RPC standard.

The HTTP example below will demonstrate how easy it is to make calls to a running DNA instance, just using the cURL tool for HTTP requests from a terminal.

Any of these methods could be similarly called from whatever client you are using, whether that is JS in the browser, nodejs, Ruby or any other language. For maximum ease of use, we recommend searching for a JSON-RPC helper library for your language of choice, there are lots of good ones out there.

Starting an HTTP Server with hc run

hc run --interface http

Starting an HTTP Server with holochain

To review how to start an HTTP Server with holochain, review the interfaces article.

HTTP Example

This whole example assumes that one of the methods listed above has been used to start an HTTP server on port 8888 with a valid DNA instance running in it.

info/instances

The following is a starter example, where a special utility function of Holochain is called, which accepts no parameters, and returns an array of the instances which are available on the HTTP server.

In another terminal besides the server, we could run the following cURL command: curl -X POST -H "Content-Type: application/json" -d '{"jsonrpc": "2.0","id": "0","method": "info/instances"}' http://localhost:8888

A response something like the following might be returned:

{
    "jsonrpc": "2.0",
    "result": [{"id":"test-instance","dna":"hc-run-dna","agent":"hc-run-agent"}],
    "id": "0"
}

Calling Zome Functions

The following discusses how to use cURL (and thus HTTP generally) to make calls to Zome functions.

The JSON-RPC "method" to use is simply "call".

The instance ID (as seen in the info/instances example), the Zome name, and the function name all need to be given as values in the "params" value of the JSON-RPC, in addition to the arguments to pass that function. This part of the "params" object might look like this: {"instance_id": "test-instance", "zome": "blog", "function": "create_post"}

Unlike info/instances, Zome functions usually expect arguments. To give arguments, a JSON object should be constructed, and given as "args" key of the "params" value. It may look like the following: "args": {"content": "sample content"}

Combining these, a request like the following could be made via cURL from a terminal: curl -X POST -H "Content-Type: application/json" -d '{"id": "0", "jsonrpc": "2.0", "method": "call", "params": {"instance_id": "test-instance", "zome": "blog", "function": "create_post", "args": { "content": "sample content"} }}' http://127.0.0.1:8888

A response like the following might be returned:

{
    "jsonrpc": "2.0",
    "result": "{\"Ok\":\"QmUwoQAtmg7frBjcn1GZX5fwcPf3ENiiMhPPro6DBM4V19\"}",
    "id": "0"
}

This response suggests that the function call was successful ("Ok") and provides the DHT address of the freshly committed blog entry ("QmU...").

This demonstrates how easy it is to call into Zome function from clients and user interfaces!

WebSockets

Any coding language which has WebSockets support can communicate with the WebSocket server interface for Holochain. Based on the API exposed by Holochain, the messages must follow the JSON-RPC standard.

We recommend searching for a JSON-RPC Websockets library for the language of your choice. In this example, we will use a Javascript based JSON-RPC library.

Starting a WebSocket Server with hc run

hc run

Starting a WebSocket Server with holochain

To review how to start a WebSocket Server with holochain, check out the interfaces article.

WebSocket Example

This whole example assumes that one of the methods listed above has been used to start a WebSocket server on port 8888 with a valid DNA instance running in it.

The JavaScript JSON-RPC library this example will use is rpc-websockets.

The overall pattern this example illustrates should be very similar for other languages.

For nodejs, and using NPM, install the rpc-websockets package by running: npm install rpc-websockets

The following code snippet just does the setup for interacting with your running DNA instance:

// import the rpc-websockets library
let WebSocket = require('rpc-websockets').Client

// instantiate Client and connect to an RPC server
let holochainUri = 'ws://localhost:8888'
let ws = new WebSocket(holochainUri)
 
// create an event listener, and a callback, for when the socket connection opens
ws.on('open', function() {
  // do stuff in here
})

info/instances

The following is a starter example, where a special utility function of Holochain is called, which accepts no parameters, and returns an array of the instances which are available on the WebSocket server.

The name of this special method is info/instances. The following code shows how to use rpc-websockets to call it. (Note the previous code is collapsed in the ellipsis for brevity)

...
ws.on('open', function() {
  
  let method = 'info/instances'
  let params = {}
  // call an RPC method with parameters
  ws.call(method, params).then(result => {
      console.log(result)
  })
})

If this code was run in nodejs, the output should be:

[ { id: 'test-instance', dna: 'hc-run-dna', agent: 'hc-run-agent' } ]

Calling Zome Functions

The following discusses how to use rpc-websockets to make calls to Zome functions.

The JSON-RPC "method" to use is simply "call".

The instance ID (as seen in the info/instances example), the Zome name, and the function name all need to be given as values in the "params" value of the JSON-RPC, in addition to the arguments to pass that function. This part of the "params" object might look like this: {"instance_id": "test-instance", "zome": "blog", "function": "create_post"}

Unlike info/instances, Zome functions usually expect arguments. To give arguments, a JSON object should be constructed, and given as "args" key of the "params" value. It may look like the following: { blog: { content: "sample content" }}

The following code shows how to use rpc-websockets to call Zome functions.

...
ws.on('open', function() {
    let method = 'call'
    let params = {
        instance_id: "test-instance",
        zome: "blog",
        function: "create_post",
        args: {
            content: "sample content"
        }
    }

    // call an RPC method with parameters
    ws.call(method, params).then(result => {
        console.log(result)
    })
})

If this code was run in nodejs, the output should be:

{ "Ok": "QmRjDTc8ZfnH9jucQJx3bzK5Jjcg21wm5ZNYAro9N4P7Bg" }

This response suggests that the function call was successful ("Ok") and provides the DHT address of the freshly committed blog entry ("QmR...").

Closing the WebSocket Connection

When you are done permanently with the connection, it can be closed.

...
// close a websocket connection
ws.close()

All in all, calling into Zome functions from clients and user interfaces is easy!

hc-web-client

To make it even easier in particular for web developers, there is a simple JavaScript library called hc-web-client developed which wraps the rpc-websockets library. Find it here, with instructions on how to use it: hc-web-client

Administering Conductors

It is possible to dynamically configure a Conductor via a JSON-RPC interface connection. There is a powerful API that is exposed for doing so.

To do this, first, recall that the admin = true property needs to be set for the interface that should allow admin access. Second, it is helpful to review and understand the behaviours around the persistence_dir property for the Conductor.

You can find details of the API for this functionality in the full API reference material. Scroll to view the with_admin_dna_functions comment block and the with_admin_ui_functions comment block. Calling these functions works exactly the same way as the other JSON-RPC API calls.

As mentioned in production Conductor, there is a GUI in development that will cover all this functionality, so that it does not have to be done programmatically, but can be done by any user simply point and click.

Building Holochain Apps: User Interfaces

Holochain is designed to be flexible and accomodating when it comes to building user interfaces. There is not a single approach to developing user interfaces that is enforced by Holochain, though there are some approaches that have extra tooling, support and focus. First and foremost, Holochain supports APIs that make building UIs with the technologies of the web (HTML, CSS, JS, etc) easy.

Put simply, it is entirely possible to build a user interface for Holochain completely in HTML, CSS, and JavaScript.

Building Holochain Apps: Bridging

As you saw in Building Apps each DNA has a unique hash that spawns a brand new DHT network and creates isolated source chains for each agent. Even when you change the DNA, releasing a new version of the app, it will spawn a brand new DHT network and source chains.

So if every app lives in an entirely separated world how can they talk to each other? This is where bridging comes into play.

A bridge is a connector between two apps (or two versions of the same app, for that matter) that allows a synchronous bidirectional transfer of information between them.

To use a bridge, right now you need to configure a production Holochain conductor, at least two instances configured, along the lines of the following example setup (in a conductor-config.toml file):

[[instances]]
id = "caller-instance"
dna = "caller-dna"
agent = "caller-agent"
[instances.logger]
type = "simple"
[instances.storage]
type = "memory"

[[instances]]
id = "target-instance"
dna = target-dna"
agent = "target-agent"
[instances.logger]
type = "simple"
[instances.storage]
type = "memory"

[[bridges]]
caller_id = "caller-instance"
callee_id = "target-instance"
handle = "sample-bridge"

Then on the caller DNA you have to initiate the bridge call using hdk::call like this:


# #![allow(unused_variables)]
#fn main() {
    let response = match hdk::call(
        "sample-bridge",
        "sample_zome",
        Address::from(PUBLIC_TOKEN.to_string()), // never mind this for now
        "sample_function",
        json!({
            "some_param": "some_val",
        }).into()
    ) {
        Ok(json) => serde_json::from_str(&json.to_string()).unwrap(), // converts the return to JSON
        Err(e) => return Err(e)
    };
#}

And the corresponding target / callee DNA on the other end should have a zome called "sample_zome", with a function as follows:


# #![allow(unused_variables)]
#fn main() {
pub fn handle_sample_function(some_param: String) -> ZomeApiResult<Address> {
    // do something here
}

define_zome! {
    entries: []

    genesis: || { Ok(()) }

    functions: [
        sample_function: {
            inputs: |some_param: String|,
            outputs: |result: ZomeApiResult<Address>|,
            handler: handle_sample_function
        }
    ]

    traits: {
        hc_public [
            sample_function
        ]
    }
#}

Remember that the call will block the execution of the caller DNA until the callee (target) finishes executing the call, so it's best to mind performance issues when working with bridges. Try to make contextual or incremental calls rather than all-encompassing ones.

Going Live with Holochain Apps

Creating Versioned Releases

Building Holochain Apps: Advanced Topics

Serialization and JsonString

Why serialize anything? Why JSON?

Holochain zomes are written in WASM.

WASM only supports working directly with integers and manually allocating memory. This means that sharing any data between holochain core and zome functions must be serialized. There is no way that WASM functions can understand the Rust type system natively. Serialized data can be allocated for WASM to read out and deserialize into Rust types/structs/enums.

Any developers using the Rust HDK get the serialization/deserialization and type handling almost "for free". The macros for defining entities and zomes automatically wrap the memory work and serialization round trips for anything that implements Into<JsonString> and TryFrom<JsonString> (see below).

We use serde for our serialization round trips as it is by far the most popular and mature option for Rust. Many serialization formats other than JSON are supported by serde but JSON is a solid option. JSON allows us to easily bring the Rust type system across to WASM with decent performance.

From the serde_json github repository README:

It is fast. You should expect in the ballpark of 500 to 1000 megabytes per second deserialization and 600 to 900 megabytes per second serialization, depending on the characteristics of your data. This is competitive with the fastest C and C++ JSON libraries or even 30% faster for many use cases. Benchmarks live in the serde-rs/json-benchmark repo.

Holochain aims to support all WASM languages not just Rust/JS

The official Holochain HDK is Rust. The Rust HDK will always be the most tightly integrated HDK with core simply because Holochain itself is Rust based.

Generally though, we are hoping and expecting many different WASM zome languages build an ecosystem over time. Personally I'm hoping for a decent LISP to appear ;)

To encourage as many languages as possible we want to keep the minimum requirements for interacting with holochain core as minimal as possible.

Currently the two requirements for writing zomes in <your favourite language>:

  • Must compile to WASM
  • Must be able to serialize UTF-8 data and allocate to memory read by core

We can't do much about the first requirement but here are some lists to watch:

  • https://github.com/appcypher/awesome-wasm-langs
  • https://github.com/mbasso/awesome-wasm

The second requirement means that we must be very mindful of choosing a serialization format that can round trip through as many languages as possible.

In the end, this is the main reason we chose JSON for communication with core.

Note that at the time of writing, the AssemblyScript (ostentisbly JavaScript) WASM implementation does not even provide a native JSON.parse() method! To do something as apparently simple as serialize JSON in JavaScript we have had to implement a custom JSON parser. At least JSON (naturally) maps very well to JavaScript native data, other serialization/language combinations are even further from maturity.

WASM is very promising but very immature so esoteric serialization options are not really viable options right now, even if serde supports them in Rust.

JSON serialization only pertains to communication with core

Holochain often makes a distinction between "app data" and "core data". Following the biomimicry theme we sometimes call this "conscious" vs. "subconscious" when this data is used in zomes or core logic respectively.

The most obvious example of this is the Entry enum that has an Entry::App variant explicitly for app data, and other variants for system logic.

The Entry enum itself is serialized via JSON so that is has maximal compatibility across all zome languages (see above) across the core/wasm boundary. However, the contents of Entry::App(..) are treated as an opaque UTF-8 string by Holochain core. Naturally the HDK macros we offer provide sugar to work with the value of app entries but this is not enforced anywhere within core. Because the Rust serialization round tripping must work across both core and the HDK it must work equally well while treating the app entry values as opaque in the subconscious and meaningful structs in the conscious. This is achieved through a healthy dose of compiler and macro magic.

This means that zome developers can implement their own serialization logic for their own data if they wish. Simply by wrapping a zome-serialized app entry value in "\"...\"" it becomes a string primitive from core's perspective. The zome can do anything needed with this, including custom validation logic, etc. The RawString type handles this automatically with JsonString (see below).

Serialization through Rust types

How Rust serializes: serde from 1000m

The serde crate leans heavily on the Rust compiler for serialization round tripping.

Using the "vanilla" serde_json crate affords this logic on the way in:


# #![allow(unused_variables)]
#fn main() {
let foo_json = serde_json::to_string(foo).unwrap();
#}

Notes:

  • There is an unwrap but this can't fail for simple structs/enums in practise
    • The unwrap can fail e.g. serializing streams but we don't do that
    • The compiler enforces that everything we pass to serde can Serialize
  • foo can be anything that implements Serialize
  • we have no direct control over the structure of the JSON output
    • the Serialize implementation of foo decides this for us
    • in the case of nested data e.g. hash maps, Serialize works recursively

OR using the manual json! macro:


# #![allow(unused_variables)]
#fn main() {
let foo_json = json!({"foo": foo.inner()});
#}

Notes:

  • We no longer have an unwrap so there is slightly less boilerplate to type
  • We have a lot of direct control over the structure of our output JSON
  • For better or worse we avoid what the compiler says about Serialize on Foo
  • We must now manually ensure that "{\"foo\":...}" is handled everywhere
    • Including in crates we don't control
    • Including when we change our JSON structure across future releases
    • Including across WASM boundaries in HDK consumers

AND on the way out:


# #![allow(unused_variables)]
#fn main() {
let foo: Foo = Foo::try_from(&hopefully_foo_json)?;
#}

Notes:

  • Serde relies on compiler info, the type Foo on the left, to deserialize
  • Serde requires that hopefully_foo_json makes sense as Foo
    • This definitely can fail as the json is just a String to the compiler
    • In real code do not unwrap this, handle the Err carefully!

JSON structure, the Rust compiler and you

All this means that our JSON data MUST closely align with the types we define for the compiler. There is a lot of flexibility offered by serde for tweaking the output (e.g. lowercasing names of things, modifying strings, etc.) but the tweaks involve a lot of boilerplate and have limits.

For example this can be awkard when handling Result values. The Result enum has two variants in Rust, Ok and Err. Both of these, like all enum variants in Rust, follow the title case convention.

This means that in a JS conductor/HDK consuming JSON values returned from zome functions that return a Result (a good idea!) we see this JavaScript:

const result = app.call(...)
const myVar = result.Ok...

We get a result.Ok rather than the result.ok that we'd expect from idiomatic JavaScript.

As the JSON structure comes from the Rust compiler, we have two options:

  • Force serde to output JSON that follows the conventions of another language
  • Force conductors/HDKs to provide sugar to map between Rust/XXX idioms
  • Force developers to work with a very leaky abstraction over the Rust compiler

As the first option requires a lot of boilerplate and isn't interoperable across all languages anyway (e.g. kebab case, snake case, etc.) we currently are pushing this sugar down to conductor/HDK implementations. Additionally, the serialized form of entries is used to calculate Address values for storage and retrieval from the local chain and DHT so we need to be very careful here as it will be hard to change in the future.

That said, we are open to constructive feedback on what this sugar looks like and how it works! Ideally zome development is as idiomatic as possible across as many languages as possible 🕶

Binary data as base64

We recommend base64 encoding binary data straight into an app entry string that you can use in your zome logic directly (see above).

Yes this uses more space than binary data, 33% more to be specific :(

But there are benefits:

  • It is UTF-8 and web (e.g. data URI) friendly
  • Simply wrapped in "\"..\"" it becomes valid JSON (see RawString below)
  • It has wide language support (see above for why this is important)
  • It will be supported by all persistence backends for the forseeable future
    • At least these storage systems require base64 encoded data at some point:
      • Browser based localStorage
      • MongoDB
      • Elasticsearch
      • Amazon SimpleDB
      • Amazon DynamoDB.

The performance penalty can be minimal:

https://lemire.me/blog/2018/01/17/ridiculously-fast-base64-encoding-and-decoding/

JSON is lame! Can Holochain support <my favourite serialization format>?

Yes... and no...

It depends what you mean by "support".

Right now, most serialization formats are supported in app/zome data simply by wrapping the output in double quotes so core sees it as a JSON string literal. Holochain core won't try to interpret/mangle any of that data so the zome can theoretically do whatever it wants at that point without a performance hit.

In practise, there are some limitations as mentioned in this doc:

  • WASM languages tend to have no or limited serialization options
    • you may need to roll your own parse/stringify logic
    • seriously... e.g. we pushed our own JSON.parse implementation upstream for the AssemblyScript team, that's JSON parsing in JavaScript!
    • don't underestimate how bleeding edge and limited the WASM tooling still is
      • to work directly with WASM you must be prepared to bleed
  • If you don't use JSON you can't use hdk macros for that part of your zome
  • Only valid UTF-8 strings are supported (may change in the future)

If you're looking for a way to provide core data in non-JSON format then NO that is not supported and won't be in the short-mid term future.

Yes, serde supports many serialization options but:

  • Not all data in core uses default serde serialization logic
    • e.g. this document explaining non-default serde serialization logic
  • Swapping to a different serializer in serde is not just a matter of passing config to serde
    • we'd have to centralise/match everywhere and swap out serde_json for analogous crates in each other format we'd want to use
    • even using a SerialString instead of JsonString (see below) would not clear out every implementation without a lot of work
  • Serde is already quite heavy in compilation/WASM files so we don't want to bloat that more with edge-case serialization needs
    • every new format is a new crate
  • We don't (yet) have any use-cases showing that JSON is a problem/bottleneck
  • Adding more serialization options would exacerbate non-idiomatic conductor and HDK data structure mapping issues (see above)

JsonString

The problem and our solution

Sometimes we want to nest serialization (e.g. hdk::call) and sometimes we want to wrap serialization (e.g. Entry::App), sometimes converting to a string uses entirely different logic (e.g. error values). Ideally we want the compiler to guide us through this process as mistakes are common and difficult to debug. We also want serialization logic to be as invisible as possible to zome developers using our HDKs.

Serde will serialize anything that implements Serialize, including String so we added a type JsonString that does not automatically round trip to act as a logical "checkpoint" in our code.

JsonString doesn't "do" anything beyond giving ourselves and the compiler a shared target while stepping through the serialization round trip.

Essentially we trade this:


# #![allow(unused_variables)]
#fn main() {
// foo_a is a Foo
// foo_json is a String
// Foo implements Serialize and Deserialize
let foo_json = serde_json::to_string(&foo_a)?;
let foo_b: Foo = serde_json::from_str(&foo_json)?;
#}

for this:


# #![allow(unused_variables)]
#fn main() {
// foo_a is a Foo
// JsonString implements From<Foo>
// Foo implements TryFrom<JsonString>
let foo_json = JsonString::from(foo_a);
let foo_b = Foo::try_from(hopefully_foo_json)?;
#}

Which looks very similar but protects us from this bug:


# #![allow(unused_variables)]
#fn main() {
let foo_json = serde_json::to_string(&foo_a)?;
let foo_json = serde_json::to_string(&foo_json)?; // <-- double serialized :/
let foo_b: Foo = serde_json::from_str(&foo_json)?; // <-- will fail :(
#}

Because nesting JsonString::from() calls is a compiler error:


# #![allow(unused_variables)]
#fn main() {
let foo_json = JsonString::from(JsonString::from(foo_a)); // <-- compiler saves us :)
#}

and this bug:


# #![allow(unused_variables)]
#fn main() {
let foo_a: Foo = serde_json::from_str(&string_but_not_json)?; // <-- runtime error :(
#}

Because calling Foo::try_from(String) is (probably) a compiler error:


# #![allow(unused_variables)]
#fn main() {
let foo_a = Foo::try_from(string_but_not_json)?; // <-- compiler saves us again :)
#}

and this bug:


# #![allow(unused_variables)]
#fn main() {
type Foo = Result<String, String>;
let foo_json_a = json!({"Err": some_error.to_string()}); // <-- good key `Err`
// somewhere else... maybe a different crate or old crate version...
let foo_json_b = json!({"error": some_error.to_string()}); // <-- bad key `error` :/

let foo: Foo = serde_json::from(&foo_json_a)?; // <-- works, key matches variant name
let foo: Foo = serde_json::from(&foo_json_b)?; // <-- runtime error! :(
#}

Because the structure of the JSON data is defined centrally at compile time:


# #![allow(unused_variables)]
#fn main() {
// Result<Into<JsonString>, Into<JsonString>> is implemented for you by HC core
let foo_json_a = JsonString::from(Err(some_error.to_string()));
// only one way to do things, automatically consistent across all crates
// doing anything different is a compiler issue
let foo_json_b = JsonString::from(Err(some_error.to_string()));
#}

Which is great for the majority of data that needs serializing. There are some important edge cases that we need to cover with additional techniques/tooling.

String handling

JsonString::from_json(&str) requires the &str passed to it is already a serialized JSON value. We may add the option to validate this for debug builds at runtime in the future.

Previously JsonString implemented the From<String> trait but this was removed. Strings are a special case as they may either contain serialized json or be used as a JSON string primitive. JsonString::from_json makes it explicit that you mean the former.

We can use serde_json::to_string and json! to create JSON data that we can then wrap in JsonString.


# #![allow(unused_variables)]
#fn main() {
// same end result for both of these...
let foo_json = JsonString::from_json(&serde_json::to_string(&foo));
let foo_json = JsonString::from(foo);
#}

More commonly useful, we can move back and forward between String and JsonString without incurring serialization overhead or human error:


# #![allow(unused_variables)]
#fn main() {
// this does a round trip through types without triggering any serde
JsonString::from_json(&String::from(JsonString::from(foo)));
#}

This is helpful when a function signature requires a String or JsonString argument and we have the inverse type. It also helps when manually building JSON data by wrapping already serialized data e.g. with format!.

An example taken from core:


# #![allow(unused_variables)]
#fn main() {
fn result_to_json_string<T: Into<JsonString>, E: Into<JsonString>>(
    result: Result<T, E>,
) -> JsonString {
    let is_ok = result.is_ok();
    let inner_json: JsonString = match result {
        Ok(inner) => inner.into(),
        Err(inner) => inner.into(),
    };
    let inner_string = String::from(inner_json);
    JsonString::from_json(&format!(
        "{{\"{}\":{}}}",
        if is_ok { "Ok" } else { "Err" },
        inner_string
    ))
}

impl<T: Into<JsonString>, E: Into<JsonString> + JsonError> From<Result<T, E>> for JsonString {
    fn from(result: Result<T, E>) -> JsonString {
        result_to_json_string(result)
    }
}
#}

Which looks like this:


# #![allow(unused_variables)]
#fn main() {
let result: Result<String, HolochainError> =
    Err(HolochainError::ErrorGeneric("foo".into()));

assert_eq!(
    JsonString::from(result),
    JsonString::from("{\"Err\":{\"ErrorGeneric\":\"foo\"}}"),
);
#}

When given a Result containing any value that can be turned into a JsonString (see below), we can convert it first, then wrap it with String::from + format!.

String serialization

Sometimes we want a String to be serialized as a JSON string primitive rather than simply wrapped in a JsonString struct. JsonString::from won't do what we need because it always wraps strings, we need to nest the String serialization.


# #![allow(unused_variables)]
#fn main() {
let foo = String::from(JsonString::from_json("foo")); // "foo" = not what we want
let foo = ???; // "\"foo\"" = what we want
#}

To keep the type safety from JsonString and nest String serialization use RawString wrapped in JsonString. RawString wraps String and serializes it to a JSON string primitive when JsonStringified.


# #![allow(unused_variables)]
#fn main() {
// does what we need :)
let foo = String::from(JsonString::from(RawString::from("foo"))); // "\"foo\""
#}

An example of this can be seen in the core version of the Result serialization from above that deals with String error values:


# #![allow(unused_variables)]
#fn main() {
impl<T: Into<JsonString>> From<Result<T, String>> for JsonString {
    fn from(result: Result<T, String>) -> JsonString {
        let is_ok = result.is_ok();
        let inner_json: JsonString = match result {
            Ok(inner) => inner.into(),
            // strings need this special handling c.f. Error
            Err(inner) => RawString::from(inner).into(), // <-- RawString here!
        };
        let inner_string = String::from(inner_json);
        format!(
            "{{\"{}\":{}}}",
            if is_ok { "Ok" } else { "Err" },
            inner_string
        )
        .into()
    }
}
#}

Which looks like this:


# #![allow(unused_variables)]
#fn main() {
let result: Result<String, String> = Err(String::from("foo"));

assert_eq!(
    JsonString::from(result),
    JsonString::from("{\"Err\":\"foo\"}"),
)
#}

If we didn't do this then the format! would return invalid JSON data with the String error value missing the wrapping double quotes.

RawString is useful when working with types that have a .to_string() method or similar where the returned string is not valid JSON.

Examples of when RawString could be useful:

  • Error descriptions that return plain text in a string
  • Base64 encoded binary data
  • Enum variants with custom string representations
  • "Black boxing" JSON data that Rust should not attempt to parse

Implementing JsonString for custom types

As mentioned above, there are two trait implementations that every struct or enum should implement to be compatible with core serialization logic:

  • impl From<MyType> for JsonString to serialize MyType
  • impl TryFrom<JsonString> for MyType to attempt to deserialize into MyType

Note that TryFrom is currently an unstable Rust feature. To enable it add !#[feature(try_from)] to your crate/zome.

Based on discussions in the Rust community issue queues/forums, we expect this feature to eventually stabilise and no longer require feature flags to use.

The TryFrom trait will need to be added as use std::convert::TryFrom to each module/zome implementing it for a struct/enum.

Boilerplate

To defer all the logic to standard serde defaults with some sensible debug logic in the case of an error, there are two utility functions in core, default_to_json and default_try_from_json.

The standard minimal boilerplate looks like this:


# #![allow(unused_variables)]
#fn main() {
struct MyType {}

impl From<MyType> for JsonString {
  fn from(my_type: MyType) -> Self {
    default_to_json(my_type)
  }
}

impl TryFrom<JsonString> for MyType {
  type Error = HolochainError;
  fn try_from(json_string: JsonString) -> Result<Self, Self::Error> {
    default_try_from_json(json_string)
  }
}
#}

Automatic derive

The standard boilerplate has been implemented as a derive macro in the holochain_core_types_derive crate.

Simply #[derive(DefaultJson)] to add the above boilerplate plus some extra conveniences (e.g. for references) to your type.

DefaultJson requires:

  • JsonString is included
  • HolochainError is included
  • MyType implements Serialize, Deserialize and Debug from serde/std

# #![allow(unused_variables)]
#fn main() {
use holochain_core_types::json::JsonString;
use holochain_core_types::error::HolochainError;

#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct MyType {}
#}

Using JsonString as the property of a struct/enum

Because JsonString cannot automatically be round tripped with Serialize and Deserialize, the following can cause difficulty:


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize)]
struct Foo {
  bar: JsonString,
}
#}

The compiler will complain about this because anything deriving Serialize recursively must consist only of values that also implement Serialize.

There are a few approaches here, each with benefits and tradeoffs.

  1. Swap out JsonString with String
  2. Use a serde attribute to manually serialize Bar
  3. Use a serde attribute to skip Bar
  4. Create a "new type" or wrapper/conversion struct

Swap JsonString with String

This approach is quick and dirty. Simply change the type of Bar to String. When prototyping or on deadline, this might be the most attractive option ;)

This will likely cause problems upstream and downstream of what you are doing, or may be symptomatic of poorly handled JSON somewhere. This is roughly how Entry used to work, with a String valued SerializedEntryand JsonString valued Entry that could be swapped between using a From implementation.

Done correctly we can "onboard" values to Foo by simply carefully wrapping and unwrapping the String. Done badly, we reintroduce the possibility for invalid wrap/nest/etc. logic to creep in.

This works best when the fields on Foo are private and immutable, exposed only through getter/setter/new style methods that internally convert between JsonString and String.

This option is less suitable if we want to double serialize the nested JSON data when serializing Foo. For an example of where we preserve JSON rather than trying to automatically deserialize or wrap it with structs, see the return values from hdk::call (not using structs, but similar ideas).

Also consider that somebody reading your code might entirely miss the fact that Foo::bar is JSON data if all they read is the struct definition.

It may be worthwhile adding methods to Foo to enforce this:


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize)]
pub struct Foo {
  bar: String,
}

impl Foo {
  pub fn new(bar: JsonString) -> Foo {
    Foo {bar: String::from(bar)}
  }

  pub fn bar(&self) -> JsonString {
    JsonString::from(self.bar.clone())
  }
}
#}

Treat bar as though it was going to be stored as a JsonString right until the last moment.

Avoid this:


# #![allow(unused_variables)]
#fn main() {
let bar_json = json!({"bar": bar.inner()}).to_string();
// somwhere later...
let foo = Foo{bar: bar_json};
#}

Because then everything that needs to use Foo must consistently implement the manual jsonification logic. This is especially important if Foo and/or bar is to be used across multiple crates.

Instead, prefer this:


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct Bar {
  bar: ..
}

let bar_json = JsonString::from(Bar{bar: ..});
let foo = Foo::new(bar); // assuming impl Foo::new from above
#}

The result is still a raw String in Foo but the validity and consistency of the JSON data is enforced across all crates by JsonString::from(bar).

It is even possible to internalise the JsonString completely within the Foo methods using Into<JsonString>. This is covered in more detail below.

Using serde attributes

Serde allows us to set serialization logic at the field level for structs.

The best example of this is handling of AppEntryValue in core. As all zome data is treated as JSON, assumed to line up with internal structs in the HDK but potentially opaque string primitives (see above) we simply alias AppEntryValue to JsonString.

The Entry enum needs to be serialized for many reasons in different contexts, including for system entries that zome logic never handles directly.

It looks something like this (at the time of writing):


# #![allow(unused_variables)]
#fn main() {
#[derive(Clone, Debug, Serialize, Deserialize, DefaultJson)]
pub enum Entry {
    #[serde(serialize_with = "serialize_app_entry")]
    #[serde(deserialize_with = "deserialize_app_entry")]
    App(AppEntryType, AppEntryValue),

    Dna(Dna),
    AgentId(AgentId),
    Delete(Delete),
    LinkAdd(LinkAdd),
    LinkRemove(LinkRemove),
    LinkList(LinkList),
    ChainHeader(ChainHeader),
    ChainMigrate(ChainMigrate),
}
#}

Note that Entry:

  • Derives Serialize and Deserialize and even DefaultJson!
  • Contains AppEntryValue in a tuple, which is a JsonString
  • Uses some serde serialization attributes

This works because the serialization attributes tell serde how to handle the JsonString in this context. This is a double edged sword. We have explicit control over the serialization so we can never accidentally wrap/nest/etc. JSON data in an invalid way. We also only define the serialization for this type in this one place. If AppEntryValue was used in some other struct/enum, we would have to manually remember to use the same or compatible serialize/deserialize callbacks.

This approach also gives a lot of control over the final JSON structure. We can avoid stutters and reams of redundant data in the final output. This can mitigate the verbosity and awkwardness of compiler-driven JSON structures when sending data to other languages (see above).

The serde documentation explains in great (technical) detail how to implement custom serialization and deserialization logic for many different data types:

https://serde.rs/field-attrs.html

For reference, the callbacks used in Entry above look like this:


# #![allow(unused_variables)]
#fn main() {
pub type AppEntryValue = JsonString;

fn serialize_app_entry<S>(
    app_entry_type: &AppEntryType,
    app_entry_value: &AppEntryValue,
    serializer: S,
) -> Result<S::Ok, S::Error>
where
    S: Serializer,
{
    let mut state = serializer.serialize_tuple(2)?;
    state.serialize_element(&app_entry_type.to_string())?;
    state.serialize_element(&app_entry_value.to_string())?;
    state.end()
}

fn deserialize_app_entry<'de, D>(deserializer: D) -> Result<(AppEntryType, AppEntryValue), D::Error>
where
    D: Deserializer<'de>,
{
    #[derive(Deserialize)]
    struct SerializedAppEntry(String, String);

    let serialized_app_entry = SerializedAppEntry::deserialize(deserializer)?;
    Ok((
        AppEntryType::from(serialized_app_entry.0),
        AppEntryValue::from(serialized_app_entry.1),
    ))
}
#}

Obviously this is a lot of boilerplate for one tuple, and is really only the tip of the iceberg for how complex custom serde implementations can get. Use this for surgical implementations along critical path type safety/ergonomics.

Skip the attribute

Serde also allows for attributes to be completely skipped during serialization.

In the context of a JsonString this is unlikely to be the desired behaviour. If we are serializing the outer struct we probably want the inner JSON data to also be serialized, but not necessarily, or perhaps we don't need it and so can live without it.

This option has very clear tradeoffs. We lose the JSON data when the outer struct is serialized but also don't have to worry about how it might be represented.

This option is very handy during development/prototyping/debugging when you want to sketch out a larger idea without immediately tackling serde logic.

Simply add the #[serde(skip)] attribute to your struct.


# #![allow(unused_variables)]
#fn main() {
#[derive(Serialize, Deserialize)]
struct Foo {
  #[serde(skip)]
  bar: JsonString,
}
#}

Wrap/convert to a new type or struct

If it is possible to create a struct that better represents the data, or a new type to hold it, then that struct can implement to/try_from JsonString.

This is very similar to the first option where we put a String into Foo but it provides semantics, information for the compiler and somewhere to hook into() for our code.


# #![allow(unused_variables)]
#fn main() {
// Bar as a new type
#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct Bar(String)

#[derive(Serialize, Deserialize)]
struct Foo {
  bar: Bar,
}

impl Foo {
  fn new(bar: Bar) -> Foo {
    Foo { bar }
  }

  fn bar(&self) -> Bar {
    self.bar.clone()
  }
}

// somewhere else...
let json = JsonString::from(..);
let bar = Bar::from(json);
let foo = Foo::new(bar);

// or...
let json = JsonString::from(..);
let foo = Foo::new(json.into());
#}

The biggest drawback to this approach is the potential for stutter. With lots of nested types we give the compiler more power but also can incidentally bloat the JSON output a lot.

Many ad-hoc/once-off types can also become confusing for humans and lead to duplicated/redundant code over time.

It is easy to end up with JSON like {"Foo":{"bar":{"Bar":[".."]}}} with a poorly chosen combination of enum variants and tuples.

As per all the considerations outlined for using String directly on Foo, avoid using json! or similar to build up the internal String of Bar.

Hiding JsonString with Into<JsonString>

It is possible in function signatures to simply leave an argument open to anything that can be converted to JsonString.

This is exactly like using Into<String> but for JSON data. An even looser option is to only require TryInto<JsonString> but this makes little or no difference to us in practise.

An example of this is the store_as_json used to pass native Rust typed data across the WASM boundary. This is used internally by the define_zome! macro for all zome funtions:


# #![allow(unused_variables)]
#fn main() {
pub fn store_as_json<J: TryInto<JsonString>>(
    stack: &mut WasmStack,
    jsonable: J,
) -> Result<SinglePageAllocation, RibosomeErrorCode> {
    let j: JsonString = jsonable
        .try_into()
        .map_err(|_| RibosomeErrorCode::ArgumentDeserializationFailed)?;

    let json_bytes = j.into_bytes();
    let json_bytes_len = json_bytes.len() as u32;
    if json_bytes_len > U16_MAX {
        return Err(RibosomeErrorCode::OutOfMemory);
    }
    write_in_wasm_memory(stack, &json_bytes, json_bytes_len as u16)
}
#}

The relevant into() or try_into() method is called internally by the function accepting Into<JsonString>, meaning the caller needs to know almost nothing about how the serialization is done. Additionally, the caller could do its own custom serialization, passing a String through, which would be wrapped as-is into a JsonString.

Unfortunately this doesn't work as well for structs because of the way trait bounds work (or don't work) without complex boxing etc. See above for simple strategies to cope with nested/wrapped serialization in nested native data structures.

This approach can be combined with the "quick and dirty" Foo with private String internals to create a Foo that can store anything that round trips through JsonString:


# #![allow(unused_variables)]
#fn main() {
struct Foo {
  bar: String,
}

impl Foo {
  fn new<J: Into<JsonString>> (bar: J) -> Foo {
    Foo{ bar: String::from(JsonString::from(bar)) }
  }

  fn bar<T: TryFrom<JsonString>>(&self) -> Result<T, HolochainError> {
    Ok(JsonString::from(self.bar.clone()).try_into()?)
  }
}

// somewhere later..
// we can build MyBar ad-hoc to send to Foo as long as it implements JsonString
// we could create MyOtherBar in the same way and send to Foo in the same way
#[derive(Serialize, Deserialize, Debug, DefaultJson)]
struct MyBar { .. }

let my_bar = MyBar::new(..);
// auto stores as String via. JsonString internally
let foo = Foo::new(my_bar);
// note we must provide the MyBar type at restore time because we destroyed
// that type info during the serialization process
let restored_bar: MyBar = foo.bar()?;
#}

This is how the ContentAddressableStorage trait used to work. It would "magically" restore the correct Content from storage based on an Address and type alone, provided the compiler had the type info available at compile time.

We had to sacrifice this neat trick due to incompatible constraints from the type system elsewhereon the CAS, but it should work well in most scenarios :)

Building For Android

Note: These instructions for building Holochain on Android are adapted from here.

In order to get to libraries that can be linked against when building HoloSqape for Android, you basically just need to setup up according targets for cargo.

Given that the Android SDK is installed, here are the steps to setting things up for building:

  1. Install the Android tools:

    a. Install Android Studio b. Open Android Studio and navigate to SDK Tools: - MacOS: Android Studio > Preferences > Appearance & Behaviour > Android SDK > SDK Tools - Linux: Configure (gear) > Appearance & Behavior > System Settings > Android SDK c. Check the following options for installation and click OK: * Android SDK Tools * NDK * CMake * LLDB d. Get a beverage of your choice (or a full meal for that matter) why you wait for the lengthy download

  2. Setup ANDROID_HOME env variable:

On MacOS

export ANDROID_HOME=/Users/$USER/Library/Android/sdk

Linux: (assuming you used defaults when installing Android Studio)

export ANDROID_HOME=$HOME/Android/Sdk
  1. Create standalone NDKs (the commands below put the NDK in your home dir but you can put them where you like):
export NDK_HOME=$ANDROID_HOME/ndk-bundle
cd ~
mkdir NDK
${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch arm64 --install-dir NDK/arm64
${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch arm --install-dir NDK/arm
${NDK_HOME}/build/tools/make_standalone_toolchain.py --api 26 --arch x86 --install-dir NDK/x86
  1. Add the following lines to your ~/.cargo/config:
[target.aarch64-linux-android]
ar = "<your $HOME value here>/NDK/arm64/bin/aarch64-linux-android-ar"
linker = "<your $HOME value here>/NDK/arm64/bin/aarch64-linux-android-clang"

[target.armv7-linux-androideabi]
ar = "<your $HOME value here>/NDK/arm/bin/arm-linux-androideabi-ar"
linker = "<your $HOME value here>/NDK/arm/bin/arm-linux-androideabi-clang"

[target.i686-linux-android]
ar = "<your $HOME value here>/NDK/x86/bin/i686-linux-android-ar"
linker = "<your $HOME value here>/NDK/x86/bin/i686-linux-android-clang"

(this toml file needs absolute paths, so you need to prefix the path with your home dir).

  1. Now you can add those targets to your rust installation with:
rustup target add aarch64-linux-android armv7-linux-androideabi i686-linux-android

Finally, should now be able to build Holochain for Android with your chosen target, e.g.:

cd <holochain repo>
cargo build --target armv7-linux-androideabi --release

NOTE: there is currently a problem in that wabt (which we use in testing as a dev dependency) won't compile on android, and the cargo builder compiles dev dependencies even though they aren't being used in release builds. Thus as a work around, for the cargo build command above to work, you need to manually comment out the dev dependency section in both core/Cargo.toml and core_api/Cargo.toml

Extending Holochain

Embedding Holochain

Core API is a library for embedding a Holochain instance (an hApp) in your own code. So this is for the use case of writing your own software that runs hApps. A common use case might be "glueing" several hApps together or adding centralized services, like file storage, on top of a hApp.

Core API

Distributed Public Key Infrastructure (DPKI)

Context

Like most other distributed systems, Holochain fundamentally relies on public key cryptography. Among other uses, Holochain nodes are identified by their public keys, and thus provenance of messages from nodes can be checked simply by checking a message signature against the node's identifier. This fundamental use requires nodes to keep the private key secret, lest the node's very agency be compromised. Keeping such secrets in the digital age is a non-trivial problem. Additionally, distributed systems imply an abundance of nodes with public keys, many of which may wish to be identified as being under the control of a single actor. Solutions to this need create yet another layer of keys which sign other keys, and the management of all this can become quite complex.

Addressing these needs is the function of Public Key Infrastructure, and in our case, because we use the power of Holochain itself to do this, a Distributed Public Key Infrastructure: DPKI.

Requirements

DPKI needs to fulfill at least the following design requirements:

  1. Provide a way to create new keys for nodes.
  2. Provide a way to revoke compromised keys and re-issue keys for a node.
  3. Provide a way to verify the provenance of keys by grouping them as originating from a single actor.
  4. Securely manage the private keys.
  5. Reliably distribute and make available information about public keys.

In designing a solution for DPKI we recognize that this is a complex and difficult enough problem that any solution will need to evolve, and in fact there will be multiple solutions necessary for different contexts. Thus, we have built into the Holochain conductor a simple interface for the fundamental needed functions, e.g. creating new keys when installing a DNA for the first time, that can then be implemented by specialized DPKI applications. Furthermore we've implemented a reference implementation of a Holochain based DPKI application, which we call DeepKey.

DeepKey

To deliver on the basics of Distributed Public Key Infrastructure, we need a way to generate keys of various types (revocation, identity, encryption, signing) from seeds, and we need to be able to generate such seeds from primary seeds, so that a human agent can create related "device agents" provably under their control.

After studying a number of uses cases, including initial sign-up, key revocation, etc, the central insight we came to was the need to create a Hierarchical Deterministic Key generation system, based on a Primary Seed, from which additional seeds can be generated which then are in turn used to actually generate many key-pairs. This allows us, by-convention, to use the first seed generated by the Primary seed as the seed for revocation keys, and subsequent seeds as seeds for keys of separate Holochain devices that can be proven to be under the control of the holder of Primary Seed.

DeepKey

TODO: merge the various docs we developed to explain DeepKey here.

  • https://medium.com/holochain/part-2-holochain-holo-accounts-cryptographic-key-management-and-deepkey-bf32ee91af65
  • https://hackmd.io/UbfvwQdJRKaAHI9Xa7F3VA?view
  • https://hackmd.io/8c8rZCyaTTqH_7TIBVtEUQ
  • https://hackmd.io/oobu0sKMSMadLXza4rHY_g

DPKI Implementation Technical Details

Keystore

For each Holochain DNA instance, the Conductor maintains a Keystore, which holds "secrets" (seeds and keys) needed for cryptographic signing and encrypting. Each of the secrets in the Keystore is associated with a string which is a handle needed when using that secret for some cryptographic operation. Our cryptographic implementation is based on libsodium, and the seeds use their notions of context and index for key derivation paths. This implementation allows DNA developers to securely call cryptographic functions from WASM which will be executed in the conductor's secure memory space when actually doing the cryptographic processing.

Decrypting a secret from the keystore invokes a passphrase manager service, which is used to collect the passphrase from some end-user. This service is implementation specific. Currently we have implementations for command-line passphrase collection for use in the hc keygen command, and also a command-line implementation within the conductor.

Standalone Mode

If the Conductor config does not include a DPKI section, then the conductor assumes it's in standalone mode and takes responsibility for adding generating new agent secrets when it receives admin/add_agent requests through the admin interface. This mode is also used by the hc command-line tool.

DPKI Mode

If the Conductor config specifies a DPKI instance, then the conductor will initially bootstrap the DPKI instance, and also delegate any admin/add_agent requests to the DPKI app for processing. Note that the Conductor does assume that basic agent key will be created by the DPKI app so that it can actually create the agent keystore file on behalf of the DPKI app.

The Holochain conductor expects the following exposed functions to exist in any DPKI application as a trait so that it can call them at various times to manage keys and identity.

  • init(params): Called during bootstrap with initialization parameters retrieved from the DPKI configuration. This function is only called if a prior call to is_initialied() returned false.
  • is_initialized() -> bool : Should return a boolean value if the DPKI DNA has been initialized or not
  • create_agent_key(agent_name) : Called any time the conductor creates a new DNA instance. Should create a keystore record for the instance.

TODO: add more functions for the trait.

Naming things

There are only two hard things in Computer Science: cache invalidation and naming things.

Rust naming conventions

If in doubt refer to the Rust conventions.

https://doc.rust-lang.org/1.0.0/style/style/naming/README.html

Holochain naming conventions

There are gaps where the Rust conventions are either silent or following them would make things too ambiguous.

Actions & reducers

  • Action is VerbNoun or Verb if there is no available noun and matches the underlying function e.g. GetEntry
  • ActionResponse is ActionName e.g. Action::GetEntry results in ActionResponse::GetEntry
  • reducer name is reduce_action_name e.g. reduce_get_entry

Actors & protocols

  • Actor Protocol is VerbNoun or Verb if there is no available noun and matches the underlying function e.g. PutEntry or Setup
  • Result of a Protocol is VerbNounResult or VerbResult e.g. PutEntryResult or SetupResult

Method names

  • method names that access something directly "for free" are the name of the thing being accessed, e.g. entry()
  • method names that have side effects or an expensive lookup are verb_noun() e.g. put_entry()

Short names

avoid micro names like t, e, h when table, entry, header is clearer.

avoid shorthand names like table when table_actor is clearer.

in the long run the legibility and unambiguity saves orders of magnitude more time than the typing costs.

Writing a Development Kit

The end goal of a Development Kit is to simplify the experience of writing Zomes that compile to WASM for Holochain apps.

At the time of writing, there is currently one active Developer Kit being written, for the Rust language. While it is possible to look at the Rust language HDK as a reference, this article is a more general guide that outlines what it takes to build a Development Kit.

If you are interested in supporting developers to write Zomes in an unsupported language, you will want to first of all check whether that language can be compiled to WebAssembly, as that is a requirement.

Why Development Kits

Development Kits are important because the WASM interface between Zomes and Holochain is constrained to singular 64 bit integers.

The WASM spec allows for multiple function arguments and defines integers as neither signed nor unsigned, but Holochain only supports a single u64 input and output for all zome functions.

WASM implements a single linear memory of bytes accessible by offset and length.

Holochain sends and receives allocated bytes of memory to zomes by treating the 64 bit integer as two 32 bit integers (high bits as offset and low bits as length).

If no bytes of memory are allocated (i.e. the 32 bit length is 0) the high bits map to an internal enum. This enum is contextual to the zome but typically represents errors:


# #![allow(unused_variables)]
#fn main() {
pub enum RibosomeErrorCode {
    Unspecified                     = 1 << 32,
    ArgumentDeserializationFailed   = 2 << 32,
    OutOfMemory                     = 3 << 32,
    ReceivedWrongActionResult       = 4 << 32,
    CallbackFailed                  = 5 << 32,
    RecursiveCallForbidden          = 6 << 32,
    ResponseSerializationFailed     = 7 << 32,
    NotAnAllocation                 = 8 << 32,
    ZeroSizedAllocation             = 9 << 32,
    UnknownEntryType                = 10 << 32,
}
#}

Each development kit should abstract memory handling in some contextually idiomatic way.

The Rust Development Kit WASM Solution

The standard development kit implements a simple memory stack.

The WasmAllocation struct represents a pair of offset/length u32 values.

The WasmStack struct is a single "top" u32 value that tracks the current end of linear memory that can be written to (either allocation or deallocation).

Use of these structs is optional inside zome WASM, Holochain core will always write/read according to the input/output position represented by the u64 arg/return values.

Reads and write methods are provided for both primitive Rust UTF-8 strings and JsonString structs.

Write new data to WasmStack as stack.write_string(String) and stack.write_json(Into<JsonString>).

If the allocation is successful a WasmAllocation will be returned else an AllocationError will result.

Allocation to the stack can be handled manually as stack.allocate(allocation) and the next allocation can be built with stack.next_allocation(length).

Allocation on the stack will fail if the offset of the new allocation does not match the current stack top value.

To read a previous write call let s = allocation.read_to_string() and standard let foo: Result<Foo, HolochainError> = JsonString::try_from(s) for JSON deserialization.

To write a deallocation call stack.deallocate(allocation).

Deallocation does not clear out WASM memory, it simply moves the top of the stack back to the start of the passed allocation ready to be overwritten by the next allocation.

Deallocation will fail if the allocation offset + length does not equal the current stack top.

Holochain compatible encodings of allocations for the return value of zome functions can be generated with allocation.as_ribosome_encoding().

The development kit:

  • Implements the simple stack/allocation structs and methods
  • Manages a static stack for consistent writing
  • Exposes convenience functions for the Holochain API to handle relevant allocation/deallocations
  • Maps u64 values to/from encoded error values and u32 offset/length values for memory allocations

For more details review the unit/integration test suites in hdk-rust and wasm_utils.

Crafting the API

Using its WASM interpreter, Holochain exposes its callable Zome API functions by making them available as "imports" in Zome WASM modules. Per the memory discussion above, each of the Zome API functions have the same explicit function signature, but different implicit function signatures. The native functions have each been given a prefix so that Development Kit wrappers can expose a regular function name. Here is a complete list:

  • hc_debug
  • hc_call
  • hc_sign
  • hc_verify_signature
  • hc_commit_entry
  • hc_update_entry
  • hc_update_agent
  • hc_remove_entry
  • hc_get_entry
  • hc_link_entries
  • hc_query
  • hc_send
  • hc_start_bundle
  • hc_close_bundle

There is a special additional one called hc_init_globals which we will discuss further.

The Development Kit should implement and export one function per each native function from the list. The function should be called the same as its native form, but without the prefix. E.g. hc_update_agent should be called update_agent or updateAgent. That function should internally call the native function and handle the additional complexity around that.

In order to call these "external" functions, you will need to import them and provide their signature, but in a WASM import compatible way. In Rust, for example, this is simply:


# #![allow(unused_variables)]
#fn main() {
extern {
  fn hc_commit_entry(encoded_allocation_of_input: RibosomeEncodingBits) -> RibosomeEncodingBits;
}
#}

TODO: define or link to meaningful function signatures

Working with WASM Memory

The goal of the Development Kit is to expose a meaningful and easy to use version of the API functions, with meaningful arguments and return values. There is a bit of flexibility around how this is done, as coding languages differ. However, the internal process will be similar in nature. Here it is, generalized:

  1. declare, or use a passed, single page 64 KiB memory stack
  2. join whatever inputs are given into a single serializable structure
  3. serialize the given data structure as an array of bytes
  4. determine byte array length
  5. ensure it is not oversized for the stack
  6. allocate the memory
  7. write the byte array to memory
  8. create an allocation pointer for the memory a. use a 16 bit integer for the pointers offset b. use a 16 bit integer for the pointers length
  9. join the pointers into a single 32 bit integer a. high bits are offset b. low bits are length
  10. call the native function with that 32 bit integer and assign the result to another 32 bit integer a. e.g. encoded_alloc_of_result = hc_commit_entry(encoded_alloc_of_input)
  11. deconstruct that 32 bit integer into two variables a. use a 16 bit integer for the pointers offset b. use a 16 bit integer for the pointers length
  12. read string data from memory at the offset address
  13. deallocate the memory
  14. deserialize the string to JSON if JSON is expected

That looks like a lot of steps, but most of this code can be shared for the various functions throughout the Development Kit, leaving implementations to be as little as 5 lines long. Basically, the process inverts at the point of the native function call.

WASM Single Page Stack

TODO

App Globals

When writing Zome code, it is common to need to reference aspects of the context it runs in, such as the active user/agent, or the DNA address of the app. Holochain exposes certain values through to the Zome, though it does so natively by way of the hc_init_globals function mentioned. Taking care to expose these values as constants will simplify the developer experience.

This is done by calling hc_init_globals with an input value of 0. The result of calling the function is a 32 bit integer which represents the memory location of a serialized JSON object containing all the app global values. Fetch the result from memory, and deserialize the result back into an object. If appropriate, set those values as exports for the Development Kit. For example, in Rust, values become accessible in Zomes using hdk::DNA_NAME. It's recommended to use all capital letters for the export of the constants, but as they are returned as keys on an object from hc_init_globals they are in lower case. The object has the following values:

  • dna_name
  • dna_address
  • agent_id_str
  • agent_address
  • agent_initial_hash
  • agent_latest_hash
  • public_token

See the API global variables page for details on what these are.

Publish It and Get In Touch

If you've made it through the process so far, good work. The community is an important part of the success of any project, and Holochain is no different. If you're really proud of your work, get in touch with the development team on the chat server, mention you're working on it, and request help if necessary. This book could be updated to include links to other HDKs. Whether you would like to, or you'd like the team to, the HDK could be published to the primary package manager in use for the language, to be used by developers around the world. For example, RubyGems for Ruby or npm for nodejs.

Zome implementation

Zome API functions

Each zome API function is implemented under nucleus::ribosome::api.

There is a fair bit of boilerplate at the moment, sorry!

To co-ordinate the execution of an API function across Rust and WASM we need to define a few related items.

Within nucleus::ribosome::api:

  • A variant in the ZomeApiFunction enum
  • The same canonical string in both as_str and from_str
  • A mapping to the API function under as_fn

As a new module under nucleus::ribosome::api:

  • A ribosome module implementing the invocation logic as invoke_*
  • A struct to hold/serialize any input args if needed

In ::action:

  • An action if the zome API function has side effects

Zome API function definition

Simply add the name of the new zome API function to the end of the enum.

Make sure to add the canonical names carefully. The Rust compiler will guide you through the rest if you miss something.

DO add a doc comment summarising what the zome function does and sketching the function signature.

DO extend the relevant unit tests.

Do NOT add to the start or middle of the enum as that will renumber the other zome functions.

Zome API function ribosome module

Each zome API function should have its own module under nucleus::ribosome::*.

Implement a public function as invoke_<canonical name>. The function must take two arguments, a &mut nucleus::ribosome::Runtime and a &wasmi::RuntimeArgs.

This function will be called by the invocation dispatch (see above).

Zome API function arguments

The wasmi::RuntimeArgs passed to the Zome API function contains only a single u64 value. This is an encoded representation of a single page of memory supported by the memory manager. The 16 high bits are the memory offset and the 16 low bits are the memory length. See the wasm_utils crate for more implementation details.

You don't have to work with the memory manager directly, simply pass the runtime and runtime args to nucleus::runtime_args_to_utf8 to get a utf-8 string from memory.

You DO have to handle serialization round trips if you want to pass anything other than a single utf-8 string to a zome API function.

The simplest way to do this is implement a struct that derives Serialize and Deserialize from serde, then use serde and .into_bytes() co-ordinate the round trip.

For an example implementation of a struct with several fields see:

  • nucleus::ribosome::commit::CommitArgs for the input args struct
  • nucleus::ribosome::commit::tests::test_args_bytes serializing the struct as bytes
  • nucleus::ribosome::commit::invoke_commit deserializing the struct from the runtime

Zome API function action dispatch

If the function has a side effect it must send an action to the state reduction layer.

Actions are covered in more detail in the state chapter.

In summary, if you want to send an action and wait for a return value:

  • create an outer channel in the scope of your invoke function that will receive the return value
  • call ::instance::dispatch_action_with_observer with:
    • the runtime's channels
    • the action the reducer will dispatch on
    • an observer sensor, which is a closure that polls for the action result and sends to your outer channel
  • block the outer channel until you receive the action result

Zome API function return values

The zome API function returns a value to wasm representing success or a wasm trap.

The success value can only be a single u64.

Traps are a low level wasm concern and are unlikely to be directly useful to a zome API function implementation.

See https://github.com/WebAssembly/design/blob/master/Semantics.md#traps

To get complex values out of wasm we use the memory manager, much like the input argument serialization (see above).

The util function nucleus::runtime_allocate_encode_str takes a string, allocates memory and returns the value that the zome API function must return.

To return an error relevant to holochain, return Ok with an HcApiReturnCode error enum variant.

For an example implementation returning a complex struct see:

  • agent::state::ActionResponse::GetEntry containing an Entry struct
  • nucleus::ribosome::get::invoke_get
    • match the action result against the correct enum variant
    • serialize the entry using serde
    • return the result of runtime_allocate_encode_str
    • if the action result variant does NOT match then return HcApiReturnCode::ErrorActionResult

Zome API function agent action

If the zome API function will cause side effects to the agent state then it must implement and dispatch an action.

Actions are covered in more detail in the state chapter.

In summary, if a new agent action (for example) is needed:

  • extend the action::Action enum
    • this sets the data type, the ActionWrapper provides a unique ID
    • use the canonical name if that makes sense
  • extend an ActionResult enum if the action has a return value
  • implement a reducer for the new action

State & Actions

Holochain uses a hybrid global/local state model.

In our bio mimicry terms the global state is for "short term memory" and local state wraps references to "long term memory".

The global state is implemented as Redux style reducers. Any module can dispatch an action to the global state. The action will be "reduced" to a new state tree value by the modules responsible for each branch of the state tree. The response values from a reduction must be polled directly from the state tree in a thread using a "sensor" closure in an observer.

Actions are stateless/immutable data structures that are dispatched by modules to communicate a request to do something potentially state changing. Everything in the system should be either stateless or change state only in response to an incoming action.

The global state is called "short term memory" because it is highly dynamic, readily inspectable, and volatile. It does not survive indefinitely and is best thought of as a cache of recent history.

Local state is implemented using actors to co-ordinate memory and threads in Rust for external, persistent state. The classic example is a database connection to the database that stores entries and headers. The db actor receives read/write messages, and a reference to the sender is stored in the global state.

Actions

The action module defines actions and action wrappers:

  • ActionWrapper: struct contains a unique ID for the action and the Action
  • Action: enum of specific data to a given action, e.g. Action::Commit

Processing an incoming action is a 3 step process:

  1. Implement reduce to resolve and dispatch to a handler
  2. Resolve the action to an appropriate handler
  3. Implement handler logic

Reduce

The reduce implementation is essentially copypasta. It handles resolving and dispatching to a handler with a new state clone. The handler resolution and dispatch logic should be split to facilitate clean unit testing.


# #![allow(unused_variables)]
#fn main() {
pub fn reduce(
    old_state: Arc<FooState>,
    action_wrapper: &ActionWrapper,
    action_channel: &Sender<ActionWrapper>,
    observer_channel: &Sender<Observer>,
) -> Arc<AgentState> {
  let handler = resolve_action_handler(action_wrapper);
  match handler {
      Some(f) => {
          let mut new_state: FooState = (*old_state).clone();
          f(&mut new_state, &action_wrapper, action_channel, observer_channel);
          Arc::new(new_state)
      }
      None => old_state,
  }
}
#}

Resolve an appropriate handler

The action handler should map signals to action handlers.


# #![allow(unused_variables)]
#fn main() {
fn resolve_action_handler(
    action_wrapper: &ActionWrapper,
) -> Option<fn(&mut AgentState, &ActionWrapper, &Sender<ActionWrapper>, &Sender<Observer>)> {
    match action_wrapper.action() {
        Action::Commit(_, _) => Some(handle_commit),
        Action::Get(_) => Some(handle_get),
        _ => None,
    }
}
#}

Implement the handlers

Each handler should respond to one action signal and mutate the relevant state.

The standard pattern is to maintain a HashMap of incoming action wrappers against the result of their action from the perspective of the current module. Each action wrapper has a unique id internally so there will be no key collisions.


# #![allow(unused_variables)]
#fn main() {
fn handle_foo(
    state: &mut FooState,
    action_wrapper: &ActionWrapper,
    _action_channel: &Sender<ActionWrapper>,
    _observer_channel: &Sender<Observer>,
) {
    let action = action_wrapper.action();
    let bar = unwrap_to!(action => Action::Bar);

    // do something with bar...
    let result = bar.do_something();

    state
        .actions
        .insert(action_wrapper.clone(), ActionResponse::Bar(result.clone()));
}
#}

WARNING: Actions are reduced in a simple loop. Holochain will hang if you dispatch and block on a new action while an outer action reduction is also blocking, waiting for a response.

Global state

instance::Instance has a state::State which is the one global state. Each stateful module has a state.rs module containing sub-state slices.

See src/agent/state.rs and src/nucleus/state.rs and how they are put together in src/state.rs.

State is read from the instance through relevant getter methods:


# #![allow(unused_variables)]
#fn main() {
instance.state().nucleus().dna()
#}

and mutated by dispatching an action:


# #![allow(unused_variables)]
#fn main() {
let entry = Entry::App( ... );
let action_wrapper = ActionWrapper::new(&Action::Commit(entry));
instance.dispatch(action_wrapper);
#}

Instance calls reduce on the state with the next action to consume:


# #![allow(unused_variables)]
#fn main() {
pub fn consume_next_action(&mut self) {
    if self.pending_actions.len() > 0 {
        let action = self.pending_actions.pop_front().unwrap();
        self.state = self.state.clone().reduce(&action);
    }
}
#}

The main reducer creates a new State object and calls the sub-reducers:


# #![allow(unused_variables)]
#fn main() {
pub fn reduce(&mut self, action_wrapper: &ActionWrapper) -> Self {
    let mut new_state = State {
        nucleus: ::nucleus::reduce( ... ),
        agent: ::agent::reduce( ... )
    }

    new_state.history.insert(action_wrapper);
    new_state
}
#}

Each incoming action wrapper is logged in the main state history to facilitate testing and "time travel" debugging.

Sub-module state slices are included in state::State as counted references.

The sub-module reducer must choose to either:

  • If mutations happen, return a cloned, mutated state slice with a new reference
  • If no mutations happen, return the reference to the original state slice

The reduce copypasta above demonstrates this as the possible return values.

Redux in Rust code was used as a reference from this repository.

Local state

Coming Soon.

@TODO @see https://github.com/holochain/holochain-rust/issues/176

Internal actors

Actors are discussed in two contexts:

  • Each Holochain agent as an actor in a networking context
  • Riker actors as an implemenation detail in the Holochain core lib

This article is about the latter.

Actor model

The actor model is a relatively safe approach to co-ordinating concurrency.

At a high level:

  • An actor is the "primitive", like objects are the primitive of the OO paradigm
  • Actors are stateful but this state is never exposed to the rest of the system
  • Actors manage their internal state
  • Actors maintain a message queue or "inbox"
  • Messages can be received concurrently but must be processed sequentially in FIFO order
  • The messages have a preset format
  • Actors update their internal state in response to messages
  • Actors can send messages to each other
  • Messages are always processed at most once
  • Actors can "supervise" each other to create a fault tolerent system
  • A supervisor can restart or stop a failed actor, or escalate the decision to another supervisor

The guarantees provided by the message queue allow actors to use stateful logic that would not be safe otherwise in a concurrent context.

For example, we can implement logic that reads/writes to the file system without locks or other co-ordination. Then put an actor in front of this logic and only interact with the file system through the relevant actor.

Riker

Riker is an actor library for Rust.

The actor implementation in Riker has a few key concepts:

  • protocol: a set of valid messages that can be sent (e.g. an enum)
  • actor system: manages and co-ordinates all actors
  • actor: anything implementing the Actor trait to create new actor instances and handle receiving messages
  • actor instance: an instance of the actor struct that has internal state and is tracked by the actor system
  • actor ref(erence): an ActorRef that can tell messages to the actor instance it references via. the actor system

The actor reference is a "killer feature" of Riker for us.

  • known size at compile, safe as properties of structs/enums
  • small size, almost free to clone
  • safe to share across threads and copy, no Arc reference counting, no locks, etc.
  • safe to drop (the actor system maintains a URI style lookup)
  • known type, no onerous generic trait handling
  • no onerous lifetimes

Frequently Asked Questions

  1. How is Holochain different from blockchain?
  2. Why do you call it "Holochain"?
  3. How is Holochain different from a DHT (Distributed Hash Table)?
  4. What kind of projects is Holochain good for? What is Holochain not good for?
  5. What is Holochain's consensus algorithm?
  6. Can you run a cryptocurrency on Holochain?
  7. How is Holochain different from __________?
  8. What language is Holochain written in? What languages can I use to make Holochain apps?
  9. Is Holochain open source? 10 How is Holochain more environmentally ethical than blockchain?
  10. How are data validated on Holochain?
  11. What happens to data when a node leaves the network?
  12. Should I build my coin/token on Holochain?
  13. What does “agent-centric” mean? How is this different from “data-centric”?
  14. What is the TPS (Transactions Per Second) on Holochain?

How is Holochain different from blockchain?

Holochain and blockchain are built for fundamentally different use cases. Blockchain is relatively good for systems where it’s absolutely necessary to maintain global consensus. Holochain is much better than blockchain at anything that requires less than universal consensus (most things): It’s faster, more efficient, more scalable, adaptable, and extendable.

Long before blockchains were hash chains and hash trees. These structures can be used to ensure tamper-proof data integrity as progressive versions or additions to data are made. These kinds of hashes are often used as reference points to ensure data hasn't been messed with—like making sure you're getting the program you meant to download, not some virus in its place.

Instead of trying to manage global consensus for every change to a huge blockchain ledger, every participant has their own signed hash chain (countersigned for transactions involving others). After data is signed to local chains, it is shared to a DHT where every node runs the same validation rules (like blockchain nodes all run the same validation rules. If someone breaks those rules, the DHT rejects their data—their chain has forked away from the holochain.

The initial Bitcoin white paper introduced a blockchain as an architecture for decentralized production of a chain of digital currency transactions. This solved two problems (time/sequence of transactions, and randomizing who writes to the chain) with one main innovation of bundling transactions into blocks which somebody wins the prize of being able to commit to the chain if they solve a busywork problem faster than others.

Now Bitcoin and blockchain have pervaded people's consciousness and many perceive it as a solution for all sorts of decentralized applications. However, when the problems are framed slightly differently, there are much more efficient and elegant solutions (like holochains) which don't have the processing bottlenecks of global consensus, storage requirements of everyone having a FULL copy of all the data, or wasting so much electricity on busywork.

Why do you call it "Holochain"?

A variety of reasons: it's a composed whole of other technologies, it's structurally holographic, and it empowers holistic patterns.

A unified cryptographic whole

Holochain is made from multiple cryptographic technologies composed into a new whole.

  • Hashchains: Hashchains provide immutable data integrity and definitive time sequence from the vantage point of each node. Technically, we're using hash trees—blockchains do too, but they're not called blocktrees, so we're not calling these holotrees.

  • Cryptographic signing of chains, messages, and validation confirmations maintain authorship, provenance, and accountability. Countersigning of transactions/interactions between multiple parties provide non-repudiation and "locking" of chains.

  • DHT (Distributed Hash Table) leverages cryptographic hashes for content addressable storage, while randomizing of interactions by hashing into neighborhoods to impede collusion, and processing validation #1 and #2 to store data on the DHT.

Holographic storage

Every node has a resilient sample of the whole. Like cutting a hologram, if you were to cut a Holochain network in half (make it so half the nodes were isolated from the other half), you would have two whole, functioning systems, not two partial, broken systems.

This seems to be the strategy used to create resilience in natural systems. For example, where is your DNA stored? Every cell carries its own copy, with different functions expressed based on the role of that cell.

Where is the English language stored? Every speaker carries it. People have different areas of expertise, or exposure to different slang or specialized vocabularies. Nobody has a complete copy, nor is anyone's version exactly the same as anyone else, If you disappeared half of the English speakers, it would not degrade the language much.

If you keep cutting a hologram smaller and smaller eventually the image degrades enough to stop being recognizable, and depending on the resiliency rules for DHT neighborhoods, holochains would likely share a similar fate. Although, if the process of killing off the nodes was not instantaneous, the network may be able to keep reshuffling data per redundancy requirements to keep it alive.

Holarchy

Holochains are composable with each other into new levels of unification. In other words, Holochains can build on decentralized capacities provided by other Holochains, making new holistic patterns possible. Like bodies build new unity on holographic storage patterns that cells use for DNA, and a society build new unity on the holographic storage patterns of language, and so on.

How is Holochain different from a DHT (Distributed Hash Table)?

DHTs enable key/value pair storage and retrieval across many machines. The only validation rules they have is the hash of the data itself to confirm what you're getting is probably what you intended to get. They have no other means to confirm authenticity, provenance, timelines, or integrity of data sources.

In fact, since many DHTs are used for illegal file sharing (Napster, Bittorrent, Sharezaa, etc.), they are designed to protect anonymity of uploaders so they won't get in trouble. File sharing DHTs frequently serve virus infected files, planted by uploaders trying to infect digital pirates. There's no accountability for actions or reliable way to ensure bad data doesn't spread.

By embedding validation rules as a condition for the propagation of data, our DHT keeps its data bound to signed source chains. This can provide similar consistency and rule enforcement as blockchain ledgers asynchronously so bottlenecks of immediate consensus become a thing of the past.

The DHT leverages the signed source chains to ensure tamper-proof immutability of data, as well as cryptographic signatures to verify its origins and provenance.

The Holochain DHT also emulates aspects of a graph database by enabling people to connect links to other hashes in the DHT tagged with semantic markers. This helps solve the problem of finding the hashes that you want to retrieve from the DHT. For example, if I have the hash of your user identity, I could query it for links to blogs you've published to a holochain so that I can find them without knowing either the hash or the content. This is part of how we eliminate the need for tracking nodes that many DHTs rely on.

What kind of projects is Holochain good for?

Sharing collaborative data without centralized control. Imagine a completely decentralized Wikipedia, DNS without root servers, or the ability to have fast reliable queries on a fully distributed PKI, etc.

  • Social Networks, Social Media & VRM: You want to run a social network without a company like Facebook in the middle. You want to share, post, publish, or tweet to shared space, while automatically keeping a copy of these things on your own device.

  • Supply Chains & Open Value Networks: You want to have information that crosses the boundaries of companies, organizations, countries, which is collaboratively shared and managed, but not under the central control of any one of those organizations.

  • Cooperatives and New Commons: You want to create something which is truly held collectively and not by any particular individual. This is especially good for digital assets.

  • P2P Platforms: Peer-to-Peer applications where every person has similar capabilities, access, responsibilities, and value is produced collectively.

  • Collective Intelligence: Governance, decision-making frameworks, feedback systems, ratings, currencies, annotations, or work flow systems.

  • Collaborative Applications: Chats, Discussion Boards, Scheduling Apps, Wikis, Documentation, etc.

  • Reputational or Mutual Credit Cryptocurrencies: Currencies where issuance can be accounted for by actions of peers (like ratings), or through double-entry accounting are well-suited for holochains. Fiat currencies where tokens are thought to exist independent of accountability by agents are more challenging to implement on holochains.

What is Holochain not good for?

You probably should not use Holochain for:

  • Just yourself: You generally don't need distributed tools to just run something for yourself. The exception would be if you want to run a holochain to synchronize certain data across a bunch of your devices (phone, laptop, desktop, cloud server, etc.)

  • Anonymous, secret, or private data: Not only do we need to do a security audit of our encryption and permissions, but you're publishing to a shared DHT space, so unless you really know what you're doing, you should not assume data is private. Some time in the future, I'm sure some applications will add an anonymization layer (like TOR), but that is not native.

  • Large files: Think of holochains more like a database than a file system. Nobody wants to be forced to load and host your big files on their devices just because they are in the neighborhood of its hash. Use something like IPFS if you want a decentralized file system.

  • Data positivist-oriented apps: If you have built all of your application logic around the idea that data exists as an absolute truth, not as an assertion by an agent at a time, then you would need to rethink your whole approach before putting it in a Holochain app. This is why most existing cryptocurrencies would need significant refactoring to move from blockchain to Holochain, since they are organized around managing the existence of cryptographic tokens.

What is Holochain's consensus algorithm?

Holochains don't manage consensus, at least not about some absolute perspective on data or sequence of events. They manage distributed data integrity. Holochains do rely on consensus about the validation rules (DNA) which define that integrity, but so does every blockchain or blockchain alternative (e.g. Bitcoin Core). If you have different validation rules, you're not on the same chain. These validation rules establish the "data physics," and then applications are built on that foundation.

In making Holochain, our goal is to keep it "as simple as possible, but no simpler" for providing data integrity for fully distributed applications. As we understand it, information integrity does not require consensus about an absolute order of events. You know how we know? Because the real world works this way—meaning, the physically distributed systems outside of computers. Atoms, molecules, cells, bodies each maintain the integrity of their individual and collective state just fine without consensus on a global ledger.

Not only is there no consensus about an absolute order of events, but if you understand the General Theory of Relativity, then you'll understand there is in fact no real sequence of events, only sequences relative to a particular vantage point.

That's how holochains are implemented. Each source chain for each person/agent/participant in a Holochain preserves the immutable data integrity and order of events of that agent's actions from their vantage point. As data is published from a source chain to the validating DHT, then other agents sign their validation, per the shared "physics" encoded into the DNA of that Holochain.

The minor exception to the singular vantage point of each chain, is the case when a multi-party transaction is signed to each party's chain. That is an act of consensus -- but consensus on a very small scale -- just between the parties involved in the transaction. Each party signs the exact same transaction with links to each of their previous chain entries. Luckily, it's pretty easy to reach consensus between 2 or 3 parties. In fact, that is already why they're doing a transaction together, because they all agree to it.

Holochains do sign every change of data and timestamp (without a universal time synchronization solution), This provides ample foundation for most applications which need solid data integrity for shared data in a fully distributed multi-agent system. Surely, there will be people who will build consensus algorithms on top of foundation (maybe like rounds, witnesses, supermajorities of Swirlds),

However, if your system is designed around data having one absolute true state, not one which is dynamic and varied based on vantage point, we would suggest you rethink your design. So far, for every problem space where people thought they needed an absolute sequence of events or global consensus, we have been able to map an alternate approach without those requirements. Also, we already know this is how the world outside of computers works, so to design your system to require (or construct) an artificial reality is probably setting yourself up for failure, or at the very least for massive amounts of unnecessary computation, communication, and fragility within your system.

How is Holochain more environmentally ethical than blockchain?

Holochain removes the need for global consensus, and with it the expenditure of massive amounts of electricity to synchronize millions of nodes about data that aren't relevant to them.

There are two reasons Holochain is vastly more efficient than blockchain and more ethical in a green sense:

  1. It eliminates the need for all nodes to be synchronized with each other in global consensus. Sharding is usually enabled on Holochain. This means that when two nodes make a transaction, each node saves a countersigned record of that transaction. Additionally, the transaction is published to the Distributed Hash Table (sent to and saved by some unpredictably random nodes that can be looked up later for retrieval).

    Sharding is configurable by app, and in some cases it's a good idea to turn it off. For example, imagine a distributed Slack-like team messaging app. With only 40-50 members, full synchronization would be worth the extra bandwidth requirement for the benefit of offline messages and reduced load times. But for most applications, global synchronization isn't really needed and sharding is kept on.

    Because of DHTs, and the sharding they enable, Holochain actually doesn't rely on the transfer of large amounts of redundant information, and uses vastly less bandwidth than blockchain.

  2. There's no mining on Holochain. Blockchain's proof-of-work system provides a hefty incentive for thousands of people to spend the processing power of their CPUs and GPUs using up huge amounts of electricity on solving a meaningless cryptographic puzzle. Holochain doesn't have mining.

How is Holochain different from __________?

TODO: Update with reference to Rust project.

Please see the Comparisons page.

What language is Holochain written in? What languages can I use to make Holochain apps?

Holochain is written in the Rust programming language. At a low level, Holochain runs WebAssembly code, but for all practical purposes developers will write applications in a language that compiles to WebAssembly such as Rust, C, C++, Go, etc. For now, only Rust has tier 1 support for writing apps, because it has a "Holochain Development Kit" library which makes writing WebAssembly apps easy.

Is Holochain open source?

Yes, it has an open source license.

Can you run a cryptocurrency on Holochain?

Theoretically, yes—but for the moment, we'd discourage it.

If you don't know how to issue currencies through mutual credit, or how to account for them through double entry accounting, then you probably shouldn't build one on Holochain. If you do understand those key principles, than it is not very difficult to build a cryptocurrency for which Holochain provides ample accounting and data integrity.

However, you probably shouldn't try to do it in the way everyone is used to building cryptocurrencies on a global ledger of cryptographic tokens. Determining the status of tokens/coins is what create the need for global consensus (about the existence/status/validity of the token or coin). However, there are other approaches to making currencies which, for example, involve issuance via mutual credit instead of issuance by fiat.

Unfortunately, this is a hotly contested topic by many who often don't have a deep understanding of currency design nor cryptography, so we're not going to go too deep in this FAQ. We intend to publish a white paper on this topic soon, as well as launch some currencies built this way.

How are data validated on Holochain?

On Holochain, each node that receives a record of a transaction validates it against the shared application rules and gossips it to their peers. If the rules are broken, that transaction is rejected by the validator.

There is no overall, global "correctness" (or consensus) built in to Holochain. Instead, each node that receives a record of a transaction validates it against the shared application rules and gossips it to their peers. If the rules are broken, that transaction is rejected by the validator. If foul play is detected on a node's part (the node is either propagating or validating bad data) that node is blocked and a warning is sent to others. Here's an infographic describing this process. In summary, instead of a global consensus system, Holochain uses an accountability-based system with data validation by peers.

Applying this to the example of 'Ourbnb', an imaginary distributed version ofAirbnb: The Ourbnb Holochain app would certainly be written with a rule, "don't rent your apartment to two parties at the same time." So the moment a user rents to two parties at the same time, nodes receiving that datum on the DHT attempt to validate it against the app rules, detect a collision, and reject it. Holochain's gossip protocol is designed to operate at a rate at which collisions will be detected nearly immediately by gossiping peers. And since Holochain doesn't have a coin built into it, it incentivizes users to cooperate and co-create.

As a user, you don't need to trust the provider of the application you're using, only agree with the shared protocols that make up the application itself. Aside from being responsible for the maintenance and security of apps they provide, application providers on Holochain are not like traditional application providers today (think Facebook, Twitter, etc.). They don't host your data because your data is stored by you and a random subset of the users of the application.

What happens to data when a node leaves the network?

The DHT of a Holochain app makes sure that there are always enough nodes on the network that hold a given datum.

When people running Holochain apps turn off their device, they leave the network. What happens to their data and the data of other people they were storing? There are always enough nodes that hold a given piece of data in the network so as to prevent data loss when nodes leave. The DHT and Holochain gossip protocol are designed this way. Also, the redundancy factor of data on a given DHT is configurable so it can be fine-tuned for any purpose. For example, a chat app for a small team might set a redundancy factor of 100% in order to prevent long loading times, while an app with thousands of users might have a very small redundancy factor.

Should I build my coin/token on Holochain?

Since it's agent-centric instead of data-centric like traditional blockchains, Holochain isn't the best platform on which to build a token or coin.

The idea of tokens or coins is a direct representation of a system being data-centric. While theoretically it would be possible to create a token on Holochain, it would be taking a step back instead of a step forward. The more exciting possibility is creating mutual credit currencies on Holochain. These are agent-centric currencies that are designed to facilitate active exchange of value and flourishing ecosystems instead of hoarding.

What does “agent-centric” mean? How is this different from “data-centric?”

Agent-centric systems view data not as an object, but as a shared experience.

Traditional blockchains are data-centric: they rely on and are built around the concept that data is a thing—an object. Holochain transitions to agent-centricism: the idea that data is shared experiences seen from many points of view. It's not a thing. It's a collection of shared, relative experiences. Einstein discovered this about the physical world a hundred years ago—Relativity. So why are modern blockchains that are supposedly "cutting edge" still falling back on this antiquated idea that data is an object, and for two agents to have different views of one piece of data is wrong?

Holochain is deeply agent-centric. Using tech that embodies this mindset enables vastly richer interactions and collaboration to happen through its technology while at the same time being thousands of times more efficient.

What is the TPS (Transactions Per Second) on Holochain?

Holochain doesn't have a set TPS (transactions per second) like other blockchain-based or blockchain-derived projects might because there's central point through which all transactions must pass. Instead, Holochain is a generalized protocol for distributed computing.

It's common to ask a blockchain project, "How much can your technology handle? What's its TPS?" This is because nearly all of these projects are built around the limiting idea of a global ledger.

But you are not asking, how many posts per second Facebook can do. Why? Because there is no technical problem, adding more servers to Facebook's data center (only maybe monetary problems).

You are not asking how many emails per second the internet can handle, because there is no single bottleneck for email-sending, like there would be with a centralized approach.

Why are we seeing a transaction limit with blockchain networks? Because blockchain in a strange way marries a decentralized p2p network of nodes with the logical notion of one absolute truth, i.e. the blockchain being one big decentralized database of transactions. It tries to maintain this way of thinking about apps that we are used to from centralized servers. It forces every node into the same "consensus". That is implemented by having everybody share and validate everything. That does work, and maybe there are few usecases (like a global naming system maybe?) where it might be advantageous.. but applying that for everything is nonsensical.

Holochain is not forcing such a model. Instead it allows for building applications that are like email. The application is rather like a protocol, or grammar, or (I prefer this language) like a dance. If you know the dance (If you have a copy of the validation rules of the app) you can tell who else is dancing that dance and who is not. The difference between Holochain and something like email is that (similarly to blockhain) Holochain is applying 1. cryptographic signatures and 2. tamper proof hash-chains (hence Holochain) so that you can build a distributed system you can trust in. You know it is impossible (I'd rather say: very very hard) to game somebody. This so far was only possible by having trusted authorities like banks or Facebook.

So, Holochain as an app framework does not pose any limit of transactions per second because there is no place where all transactions have to go through. It is like asking, "how many words can humanity speak per second?" Well, with every human being born, that number increases. Same for Holochain.

Glossary

Agent

Keys

DNA

Zome

Source Chain

Distributed Hash Table

Local Hash Table

implementation details

First, read about state actors.

The 1:1 API implementation between actors and their inner table is achieved by internally blocking on an ask from riker patterns.

https://github.com/riker-rs/riker-patterns

The actor ref methods implementing HashTable sends messages to itself.

Calling table_actor_ref.commit(entry) looks like this:

  1. the actor ref constructs a Protocol::PutPair message including the entry
  2. the actor ref calls its own ask method, which builds a future using riker's ask
  3. the actor ref blocks on its internal future
  4. the referenced actor receives the Commit message and matches/destructures this into the entry
  5. the entry is passed to the commit() method of the inner table
  6. the actor's inner table, implementing HashTable, does something with commit (e.g. MemTable inserts into a standard Rust, in-memory HashMap)
  7. the return value of the inner table commit is inserted into a CommitResult message
  8. the CommitResult message is sent by the actor back to the actor ref's internal future
  9. the actor ref stops blocking
  10. the CommitResult message is destructured by the actor ref so that the return of commit satisfies the HashTable trait implementation

Riker ask returns a future from the futures 0.2.2 crate.

table_actor.block_on_ask() calls block_on and unwrap against this ask.

Both the block and the unwrap should be handled better in the future.


suggest an edit