Announcing Experimental Rust Filesystem and Path Support for Xous PDDB!
The "filesystem" on xous is called PDDB – short for the Plausibly Deniable Data Base. This is a dict-key-value store with built-in overlay support. We've just released experimental support for File and Path support for Xous! This requires a recent
184.108.40.206 or later), as well as a version of
Xous Core from 6 July 2022 or later.
The Structure of PDDB
The default view of PDDB is a "Union" filesystem. It is a table of dicts, where each dict has one or more keys. Each key contains binary data.
The selection of dicts that are currently visible depends on the number of bases open. Bases can be added or removed to the running system, meaning dicts and keys can appear and vanish. Most of the time users will use the "Union" basis which is all of the currently-loaded bases overlaid on top of one another. If the same key appears in two dicts, the most recently added basis takes priority when opening keys, and the least recently added basis takes priority when writing to keys.
Since PDDB is a flat system, we've grafted path support onto dicts. We use the
: character as a path separator, which means a dict with a name like
web:documents would be represented as a directory
web with a directory
documents inside of it. This works well for the small number of dicts currently present in PDDB. It might not scale to thousands of directory entries, but the API won't need to change much so we can improve performance within the PDDB service itself and programs will automatically benefit.
An Example Rust Program
With this change, the following program will compile and run on Xous:
Additionally, any sort of directory-traversal routine will work, enabling you to write a program to visit every "file" on the "filesystem" using the same method you'd use on desktop Rust.
Happily, this works as long as you have libstd support installed on your system, and can have zero additional dependencies. You can create a brand-new directory and start building software for Xous:
Inventing a Path Spec
There is a peculiarity with PDDB in that paths must specify a dict and a key, and may optionally specify a basis. This is similar to some systems where there is a drive letter and no root. Curiously, the greatest challenge of this project was coming up with a reliable path format that is suited to PDDB. The format that was finally decided on is:
That is, you MUST specify a dict. This is required when listing directories. You cannot create a dict as a file, so if you want to open a file you must also specify a key. This is done using the form
If you want to specify a heirarchy, you may add additional dicts by appending them with
If you'd like to specify a basis, you may explicitly specify the default basis
::, or give a complete basis name
:.System:. As a special case not mentioned above, the empty path implies all root dicts, and a bare basis name implies all root dicts in a particular basis.
Finally, you can also specify
: as a path, which refers to the list of bases.
Test cases were created for all of these in pddb/src/libstd/utils.rs and may be run with
cargo test -p pddb.
At this time, PDDB has no restrictions on dict or key names, meaning it's possible to create a key with a
: in the name. The standard library functions won't be able to disambiguate these paths at this time, and so this character will likely be made illegal in key names. This is somewhat remeniscent of the Windows Registry where
\ is a path separator that is illegal in path names yet is allowed in key names.
Senres: Sending Data Between Processes
Most heavy services in Xous use
rkyv to send data between processes. By annotating a struct or enum with
#[derive(rkyv::Archive, rkyv::Serialize, rkyv::Deserialize)] that struct gains the ability to be efficiently serialized into a format that's very easy to unpack.
Unfortuantely, when adding code to the Rust standard library it's difficult to add new crates. This is because each crate increases compilation burden for all targets. When we get upstream support for Xous we will need to do it while adding as few new dependencies as possible – ideally zero. It will be difficult to justify adding
xous-rs as a dependency, and very difficult to add
rkyv and its dependencies, particularly since it works its magic with proc macros.
As part of this project I developed
Senres, which is like a lighter version of
rkyv. It supports serializing data to be sent across process boundaries where it can be efficiently unpacked, analyzed, and responded to.
To send a message that's 4096 bytes stored on the stack, you would first allocate the backing:
One nice feature thanks to
const_generics is that the size of the message must be a multiple of 4096. If you try to specify a different size, the program would refuse to compile.
Once you have your backing, you next create a writer with a 4-byte type signature and append data to the request:
Finally, you send it to the server using a given opcode:
On the server side, you'd create a Backing from the memory message slice, then start reading from it:
When you're done parsing the structure and processing its contents, it's time to construct a reply. This is done in exactly the same manner as the request was made:
The message will automatically be responded to when it goes out of scope. And thanks to Rust's borrow checker, you can be sure you're never writing to a message that still has live references pointing to it.
Status of API Support
This release provides basic support for
std::path. As an example, the following features work:
- Opening and closing files
- Reading from and writing to files
- Seeking within an open file
- Creating new files
- Deleting files
- Listing directories
- Creating directories
- Deleting directories
There are a lot of features that do not work, or do not currently make sense. We'll add these features if there is demand:
- Copying files
- Truncating currently-open files
- Duplicating file descriptors
- Getting creation/access/modification times on files
- Readonly files
- Renaming files
Additionally, the PDDB supports callbacks to notify senders of various file events such as deletions and updates. This has not yet been properly implemented, and remains experimental.
Give it a Shot!
The initial release is out, so give it a try! As this is a new API there are bound to be issues. Let us know if you find any, or if you encounter any rough edges. It's early days on this, and there's still lots to do.
The Betrusted project, including the Xous operating system, are made possible thanks to financial assistance from NLNet and the NG10 Privacy & Trust Enhancing Technologies Fund. Thank you to them for their support.
This work was funded by NLNet.