xd009642

My Experience as a Rice Farmer

2026-04-01T00:00:00+00:00

I’ve been inspired to write something for April Cools Club</a>, and what fits better from my normal content than my experience rice farming in rural Japan!

For those who aren’t aware, in 2025 I spent January-July in Japan staying with my wife’s family. During that time we helped out on the family rice farm near Shuzenji in the Shizuoka prefecture. I unfortunately had to leave before the full harvest process was done but I’ll take you as far as I got and also try and share other insights I gleaned.

Unfortunately, while I thought I took a lot of photos it seems I’m missing things I would have liked to have captured for this. Where applicable there’ll be other sources and at least one video linked for more information.

Quick Farm Intro</a></h1>
The farm is primarily a rice farm, there’s no animals (ignoring the koi fish that were in the garden pond). There is a portion of a bamboo forest and space to plant non-rice crops so we also grow or harvest for consumption:

Bamboo roots</li>
Mushrooms</li>
Potatoes</li>
Pumpkins/gourds</li>
Carrots</li>
Warabi - an edible bracken. Not so much planted it just grows everywhere</li>
Whatever other vegetables they decide to plant</li> </ol>
And looking out from the driveway this is the view at the end of winter before everything starts growing and spring properly kicks in:

Obviously, me and my wife aren’t around all the time. When we’re not my brother-in-law and mother-in-law work on the farm part time often 1-2 days per week each.
Preparation</a></h1>
At the start of spring we come back to the fields. They’ve been left fallow over the winter and the dead rice plants from last year cover them. They’re currently dry, we’ll flood them later on once they’ve been prepared. Because it’s hard and spiky we have cordless strimmers with metal blades to cut through this and not take forever.
I don’t have a picture of a rice field before clearing but here is one of the fields partially cleared:

As part of clearing and getting ready we also dig the drainage ditches along the side of the fields. When the fields were drained last year the soft mud flows back into the ditches and hardens again so they need to be redug.
The field will also be ploughed to break up the soil and loosen it, and we’ll remove large rocks we find. After ploughing we can level the field to flatten it. With the field level, the rice will be at equal depths and the planting process is more consistent.
This might be the first time someone’s prepared a rice field wearing a Rust London t-shirt. It’s definitely my first time driving a tractor!

But before we get that far we have to prepare the route for the water to get into the field. This work is only actually needed for one of our fields, the others have a fairly direct route from the river to the field. But for one field we have to clear a few hundred metres of a channel that goes along the edge of a bamboo forest clearing the dead bamboo and other natural detritus.
I don’t have any pictures of this, but imagine all the joys of clearing out hundreds of meters of ditches among dense vegetation in high humidity.
I do have a picture after ploughing with the drainage ditch for the field dug next to the river that will supply our water:

One last thing we might do before flooding is drive metal rods into the perimeter of the field as part of building a fence. This doesn’t have to be done for every field, just the ones that border the bamboo forest where the wild boar and deer might sneak through at night and eat the rice.
Water Water Everywhere</a></h1>
Rice fields are typically placed near rivers, before planting we have to flood and level the fields. We’ll go down into the river, and place a wooden board by a drainage pipe at the edge redirecting water down that pipe and into a channel next to the field. We can then open a hole and let water go from that channel into the field. Water can then drain out of the edge of the field when it gets full, continuing on into other fields and eventually back into the river. The water will rejoin the river in part so that farming doesn’t dry out rivers and ensure the longevity of the environment.
For the field with the more onerous ditch clearing that water flows under the field and eventually back into the river. For that there’s an ad-hoc construction of some old drain and bamboo to move the water across into the field:

And the water entering that field:

After flooding, a tractor with a flat rear blade will be moved over the field a few times to level it. When the planting vehicle goes over the field little arms pluck off some rice and stick it down. If the soil is too far from the arm you end up with loose rice floating around on top of the water. Obviously, we don’t want to waste rice like this so levelling is an important step.
One thing to note, with a rice field the deeper soil is compacted and firm, it shouldn’t be able to drain by the water going into the water table and disappearing. However, our field with the tricky water intake did suffer from a minor sinkhole as water was able to go down and rejoin the stream that flows under the field. This resulted in some work to dig down and fill the area letting water out with rocks and harder mud and compacting it with the bucket of a digger. After this work was done the field held it’s water and we were able to think about planting.
After poking around to figure out why water was draining I managed to get this picture of the hole that started to open up. I guess that’s a sinkhole of sorts.

An interesting fact is that rice doesn’t actually need standing water to grow. The water helps by stopping weeds growing around the rice taking resources and protects the rice from certain pests that would eat it.
For some further watching, this video</a> shows a more advanced but very similar for a different farm. The main difference is that they don’t need to manually go to the inlet gates to open/close them and instead have some more modern gates controlled via mobile phone.
Planting Day</a></h1>
It’s planting day, turning up I can see the neighbours have already planted and here you can see our ready but empty field next to their freshly planted field:

But here we go, everything we’ve been working toward. The previous process has taken from mid-February and now it’s early May. We go off and buy seed trays of rice to load into the Rice Transplanter. Below you can see a picture of the planting process:

An arm will move along the bottom of the rice and pull off a clump of rice and then plunge it into the ground. It will keep moving back and forth doing this at regular intervals. It the motion of it working is reminiscent of a typewriter at work.
After it’s done there’s some leftover rice, and there might be gaps where things weren’t perfectly level. We go out into the field wearing jika-tabi</a>. These are boots with a split between the big toe and the other toes. It’s meant to help our feet not get stuck in the wet mud. Grabbing rice in small bunches we pull them from the seed tray and plant them about an inch deep into the mud and compact some mud around it.
Fun language note, my wife asked me if I saw the tabi once and I thought she meant a tabby cat. I wasn’t aware of the name of the footwear.
Now the rice is in the field we’re at risk of attack. Wild boar and deer just love to snack on our hard work - this means it’s time to put up the electric fence. This is fairly simple drive the poles into the ground at regular intervals, then feed the wire along it wrapping it round the clips making sure it’s moderately taut. Also check for any breaks in the wire and if so get a bit of electrical tape and fix it.
After wiring we place a box which is just a solar panel and battery on a timer next to the fence and try to hammer it into the ground or prop it up securely enough with rocks where the dirt is too shallow.
We’ll have to come back every week or so to cut the grass that sprouts up on the edge of the field. If we don’t it will ground the fence and drain the battery and our rice will fall victim to the local wildlife.
After planting our fields look like this:

Draining Time</a></h1>
When the rice gets older - around waist height the field is drained. Some sort of narrow plough is moved between the rows pushing the mud up around the rice to hold it up and then the water intake is closed and the field left to drain and dry out. Then the rice will continue to grow until it’s harvest time.

Unfortunately, I left Japan a couple of weeks after draining and I haven’t experienced the final stage of harvest yet. I have this picture I was sent of the rice near harvest time but the final stages will have to remain shrouded in mystery for now. I’m not ready for spoilers when I may learn this in future firsthand:

Attacked!</a></h1>
A spectre has been looming over this post. The wild boar. I got an update one day it seems a baby boar managed to squeeze under an unelectrified part of the fence and help itself to an all you can eat buffet:

Luckily someone came to the farm the day before and the day after it happened and it was closed up before the boars started visiting nightly. But it seems important to remain vigilant of your defences. I’ve still not seen a boar in the wild even going through the nearby forests - they’re nocturnal and rather dangerous so I’m glad of that!
Local Fauna</a></h1>
In rice fields you can see a lot of interesting wildlife. Frogs and salamanders help protect the crop by eating bugs that might feed on the rice. You also might see snakes nearby that feed on them as well.


When clearing grass once I saw a snake dart out from under a pickup truck we’d had parked up for a few hours as I walked past. I then looked at the grass I was going to cut and saw it hunkered down in the grass but obscured enough to not get a picture and not wanting to disturb it I moved on. After all I don’t know how dangerous it might be.
I asked my brother-in-law about the snakes later on when he came to the truck to get a drink and asked if they’re dangerous. He asked if it’s brown or “blue” (aoi 青い) - it was brown. Also blue here isn’t blue but green, historically ao used to mean the entire blue-green spectrum so for some older terms (often things like animal colours), aoi is still used instead of the more modern word for green (midori 緑). Anyway, his response to my answer is how I first heard the Japanese word 有毒な (yūdokuna) - or venomous in English. Not speaking any English, he further translated it by grabbing his throat and miming frothing at the mouth.
There are also black kites flying around, they’ve been known to swoop down and snatch up kittens and there are warning signs in some more populated places about keeping close to small pets. I’ve seen them circling in the heat but it’s hard to get a good photograph of birds with a normal smartphone camera. But I have my best capture of one:

Economics of Rice Farming</a></h1>
When I was in Japan there was a rice price crisis (try saying that three times fast). With a 95% increase in price, it actually became cheaper to fly to South Korea, fill a suitcase with rice and fly back. Eventually, the government released part of its emergency rice supply kept in storage to tackle food shortages and mitigate against disasters. This situation is likely to occur again, and as an outsider looking at how Japan’s farming system is organised it seems unavoidable without significant reforms.
In Japan the average age of a full-time rice farmer is around 70. For younger generations they can only afford to do it part-time, 1 or 2 days a week. They also own 4-6 rice fields. There are no factory farms and large scale operations.
In this respect my wife’s family is very average. Rice farming doesn’t generate enough income to do it full time so my Mother-in-law and Brother-in-law only farm 1 or 2 days a week maximum. Without more time they’re able to just plant enough fields to account for the family rice consumption and not to sell rice.
Part of the reason of this is the Gentan system, designed to protect small-scale farmers income it prevents large scale factory farming of rice and encourages ownership of smaller farms. It has been officially abolished but it still shapes how the rice economy works. This was part of a system to discourage communism initially by encouraging ownership of business and preventing absentee landlords accumulating large tracts of land where people who work the fields would be forced into renting. It should be noted the UK’s system is like this with rich landowners accumulating more farmland for tax reasons and renting it to farmers who often struggle to make farming profitable.
Farmers also sell their crops via a centrally managed system which fixes the price. Historically, crops used for animal feed have fetched higher price than human quality rice leading to a number of farmers planting rice for themselves and then selling animal feed to make a living.
Another issue is automation of farming. Reading this account of rice farming you might think this seems very manual and it is. In America rice is aerially planted. With consistency in fields and the distribution of the rice leads to higher yields. And if you’re dealing with such small farm area that becomes more important - and things like aerial planting become less economically viable. An American farm can be roughly 100 times larger than a Japanese one.
Additionally, with rising cost of living a lot of the youth of Japan move to cities like Tokyo, Osaka and Nagoya where they can find better paid office work. Local rural economies struggle more as they lose people and income from work doesn’t scale with the costs of living. It seems unavoidable we’ll see more and more rice farms close and further impacts due to decreased output.
If you’re interested in this there’s a video about this on Asianometry</a>.
Finally</a></h1>
Reading this last section it might seem to end in doom and gloom. This isn’t really how I wanted to sign off on things. Rice farming was a positive experience for me, a connection with nature, building relationships with my wife’s family and growing my Japanese skills. Doing a day of manual labour, chatting shit, then going for the onsen and some BBQ and beers is far better than grinding away at some enterprise SaaS that will probably disappear in a few years.
Farming becoming economically unviable seems to be something afflicting many countries. At some point I expect a wakeup call or transition. Either things are changed to make it viable full-time or Japan’s system of small independent farms will gradually fade away. Only time will tell, but I hope that rural communities can continue to survive and also thrive.

Automatically Running Cargo on Workspace Changes

2026-02-26T00:00:00+00:00

Given a Rust project using workspaces with potentially a large amount of packages how can we make it quicker to run tests or other tooling when the project changes. Using build systems like Bazel and Buck2 you can use the build dependency graph and only rerun relevant things. But I’ve touched Bazel briefly when I was a C++ developer and I swore never again for my own mental health. Buck2 looks intriguing and I might look at it in the future if I have some stupid big project to work on - but for Rust projects it’s gonna be hard to beat the developer experience of just using Cargo.
Authors disclaimer: this project isn’t a serious project I expect people to use. I had an idea and I implemented it because it felt interesting and possible with a low time commitment.
Implementation Plan</a></h1>
The idea is simple at it’s core. If we look at the files changed in a commit and can correlate them to packages in the workspace we can run our commands just on those packages which have changed. If we can build a dependency graph between packages in the workspace we can also run our commands on dependents of changed packages to ensure that changing a dependency doesn’t change how it’s users work.
One of the things that first drew me to this was that I could use a trie to represent workspace paths and check which package a file exists in by looking for the strongest prefix match in the trie. I hadn’t used a trie in anger before, only in University style assignments, so using one for a real project seemed like something that might be entertaining.
The next part of the plan was using Minijinja</a> for templating to allow any command to be ran like this - on the changed packages only. I’ve been using it recently for some work projects so it was in my mind.
With this rough idea we have 4 components:

Git based identification of what’s changed</li>
Building a dependency graph of the workspace, likely via cargo metadata</li>
Finding each files owned package (if there is one)</li>
Generating a command and executing it</li> </ol>
Steps 1-3 are really just “identifying relevant packages” and 4 is creating/running command. I’ve fleshed out 1-3 more already in my head just because that’s the area my mind went to first. I know this is due to a bias on what’s more interesting to me but I felt it correlated well with the difficulty. Spoiler alert: it did correlate well.
We can also special case some commands we already know we’ll use to make it more ergonomic. Running something like dc --template "cargo test {% for package in packages %} -p {{package}} {% endfor %}</code> is definitely not as friendly as dc test</code>.
Identifying Impacted Packages</a></h1> Finding Changes via Git</a></h2> My first thought here is we don’t want to run on a package if any file in it has changed. That would be safer but we can probably ignore anything that’s not a source file, manifest or clearly generating code. A fairly simple function provides this check: use std::path::Path; pub fn is_considered(path: &Path) -> bool { let ext = match path.extension().and_then(|e| e.to_str()) { Some(e) => e.to_ascii_lowercase(), None => return path.is_dir(), }; matches!( ext.as_str(), "rs" | "c" | "cpp" | "h" | "hpp" | "cc" | "cxx" | "toml" | "pb" ) }</code></pre> Then after that, given the path to the root repository as we might want to run on a different folder than our working directory. We can use git2</a> to identify the files that changed between the current commit and the parent commit. use git2::{DiffOptions, Repository}; use std::path::PathBuf; pub fn get_changed_source_files(root: &Path) -> anyhow::Result<Vec<PathBuf>> { let repo = Repository::open(root)?; let head = repo.head()?; let commit = head.peel_to_commit()?; // Last commit should have at least one parent (could look further back as well) let parent = commit.parent(0)?; let commit_tree = commit.tree()?; let parent_tree = parent.tree()?; let mut diff_opt = DiffOptions::new(); let diff = repo.diff_tree_to_tree( Some(&parent_tree), Some(&commit_tree), Some(&mut diff_opt) )?; let mut considered_files = vec![]; diff.foreach( &mut |delta, _| { if let Some(path) = delta.new_file() .path() .or_else(|| delta.old_file().path()) { if is_considered(&root.join(path)) { considered_files.push(path.to_path_buf()); } } true }, None, None, None, )?; Ok(considered_files) }</code></pre> This wasn’t too hard, the library maps well to a knowledge of git and there’s also helpful examples</a> to get me looking in the right places. For a more fully featured one I’d include dirty changes (or have the option to include them), as well as allowing comparisons between more than the current head and parent. But this is intended to run in CI and I’m happy running on every commit - for now. Using a Trie to Represent Structure</a></h2> A trie is a tree structure also called a prefix tree, the wikipedia</a> is a reasonably good resource if you’ve not heard of it before. The reason I’m using a trie for prefix matching is to get ergonomic checks on if something is within a directory tree. To visualise this here’s a potential trie we might construct from a cargo workspace: Notice I say ergonomic not fast. While tries are used to speed up prefix based matching, we don’t really know if it’ll impact things at this scale. The trie is going to be constructed when the program runs and used. But depending on the number of changes and the size of the project it might always be faster to just loop through the packages and do file.starts_with(package_root)</code> for all the packages in the workspace. If this was for a server or something long running the cost of trie creation could be amortized and we could be more sure of savings. But that’s not the case here. When selecting a trie library I vaguely recalled a Cloudflare blog on a trie library they wrote trie-hard</a>. In the blog they mentioned rafix_trie</a> as the fastest existing implementation before theirs and in the readme for trie-hard it also mentions radix_trie as more robust and fully featured. Also, radix_trie was updated more recently whereas trie-hard seems to have not been touched since their blogpost. With that in mind I chose radix_trie. Working out Workspace Packages</a></h2> In my experience, whenever you write a dev-tool that works on Rust projects you will reach for cargo_metadata</a>. This is a minimal library that takes the json structured cargo metadata</code> output and gives you Rust types for it. From that we can we the metadata for the package, loop over the workspace members, find out what dependencies they have (and which ones are in our workspace as well). This code once again feels straightforward to the point I’ll just paste it all in here. We construct a metadata command that points to the project root. We then iterate over the workspace members and for each member iterate over it’s dependencies keeping ones which exist within our project root. We then construct a package type with the name of the package, manifest path and the list of dependencies. This is inserted into a trie that maps from the workspace path to the package and the trie is returned - with a result if the metadata command fails. use cargo_metadata::MetadataCommand; use radix_trie::Trie; use std::path::{Path, PathBuf}; #[derive(Debug, Default, Eq, Hash, Ord, PartialEq, PartialOrd)] pub struct Package { pub name: String, pub manifest: PathBuf, pub dependencies: Vec<PathBuf>, } fn check_path(root: &Path, path: Option<&Path>) -> bool { match path { Some(p) => p.starts_with(root), None => false, } } pub fn find_packages(root: &Path) -> anyhow::Result<Trie<PathBuf, Package>> { let metadata = MetadataCommand::new().current_dir(root).exec()?; let mut packages = Trie::new(); for package in &metadata.workspace_members { let package = &metadata[package]; let dependencies = package .dependencies .iter() .filter(|x| check_path(root, x.path.as_ref().map(|x| x.as_std_path()))) .map(|x| x.path.clone().unwrap().into_std_path_buf()) .collect(); let pack = Package { name: package.name.to_string(), manifest: package.manifest_path.clone().into_std_path_buf(), dependencies, }; packages.insert( package .manifest_path .parent() .unwrap() .as_std_path() .to_path_buf(), pack, ); } Ok(packages) }</code></pre>Generating Commands</a></h1> As mentioned before I’m going to use minijinja for this. The motivation is that we have a list of packages, we’ll have a list of changed packages and probably a list of other arguments to apply to the command. Depending on the command we might have different ways of providing those lists as arguments and a Minijinja template lets us express things like loops to generate our command string. Off the bat I created this template for a simple cargo test -p package_1 -p package_2</code> given a list of those two packages: cargo test {% for pkg in packages %} -p {{ pkg }} {% endfor %}</code></pre> This can easily be expanded to add a list of other arguments as well or changed from -p</code> to -e</code> to exclude packages. To aid in writing templates, Minijinja has a playground</a> as well! I’m also then using shell-words</a> to take the command string and split it up into the command name and args so I can construct a std::process::Command</code>. Rules on splitting arguments involves handling things like quotes and I didn’t want to roll that myself - and knowing the world there’s probably some cursed edge-cases. With Minijinja we start by creating an environment and adding our template. We then get the template out and can see what undeclared variables are present. These will be where we insert things like our packages or args. This does rely on us having some reserved names for certain things which the user will make use of. I’ve gone for packages</code> for the packages to include and args</code> for the other args. We could also not use undeclared variables and just insert all the variables available to us. But why do extra work when it’s trivial to avoid it? Minijinja uses json internally a lot, and the Minijinja value type is analogous to the serde_json::Value</code>. This also means we can create Mininja values from types that implement Serialize</code> and we don’t have to worry too much as long as things all look like they’d serialize into json without a care. After that we render the expression, split using shell-words and execute the command. use minijinja::{Environment, Value}; fn generate_command( template: &str, included_packages: &BTreeSet<&str>, args: &[String], ) -> anyhow::Result<Command> { let mut env = Environment::new(); env.add_template("cmd", template)?; let expr = env.get_template("cmd")?; let variable_names = expr.undeclared_variables(true); let mut variables = HashMap::new(); for var in variable_names.iter() { match var.as_str() { "packages" => { variables.insert("packages", Value::from_serialize(included_packages)); } "args" => { variables.insert("args", Value::from_serialize(args)); } s => anyhow::bail!("Unsupported variable `{}`", s), } } let result = expr.render(&variables)?; let parts = shell_words::split(result.as_str())?; let mut part_iter = parts.into_iter(); let exe = part_iter.next().context("No program name")?; let mut cmd = Command::new(exe); cmd.args(part_iter) .stdout(Stdio::inherit()) .stderr(Stdio::inherit()); Ok(cmd) }</code></pre> For commands where we don’t choose packages to include but also exclude I added another allowed variable of excludes</code>. Adding this to the code we change the function to add an argument of packages: &Trie<PathBuf, Package></code>. This allows us to get the difference between the set of packages and the included packages for excludes. This changes the match on variable names to be as follows (adding the arg is omitted): match var.as_str() { "packages" => { variables.insert("packages", Value::from_serialize(included_packages) ); } "excludes" => { variables.insert( "excludes", Value::from_serialize(generate_exclude_list( packages.values(), included_packages, )), ); } "args" => { variables.insert("args", Value::from_serialize(args)); } s => anyhow::bail!("Unsupported variable `{}`", s), }</code></pre> With generate_exclude_list</code> defined as follows: fn generate_exclude_list<'a>( packages: impl Iterator<Item = &'a Package>, included_packages: &BTreeSet<&str>, ) -> BTreeSet<&'a str> { packages .filter(|x| !included_packages.contains(x.name.as_str())) .map(|x| x.name.as_str()) .collect::<BTreeSet<_>>() }</code></pre>Integrating the Current Pieces</a></h1> Given the root of the project, we can now: Get the changes</li> Get the packages in the project</li> </ol> Using radix_trie</code> if we do trie.get_ancesntor_value(&changed_file)</code> it will give us a package if one exists. Using the initial set of changes we get the initial packages impacted. Then we can get these packages, check the dependencies for ones in the workspace. Add it to the list of packages and then repeat. Continue until no new packages are added. An astute reader might think “couldn’t we have constructed the dependency graph and just do a graph traversal”. And yes we could. But for the size of workspaces I work with this felt like overkill. Plus I’ve only got 7 direct dependencies so far and 3 of them are >1.0 let considered_files = repository::get_changed_source_files(&root)?; let packages = cargo::find_packages(&root)?; let mut changed_packages = BTreeSet::new(); let mut end_package_names = BTreeSet::new(); for file in &considered_files { if let Some(package) = packages.get_ancestor_value(&root.join(file)) { changed_packages.insert(root.join(file)); end_package_names.insert(package.name.as_str()); } } let mut changed_packages_previous = 0; while changed_packages_previous != changed_packages.len() { changed_packages_previous = changed_packages.len(); for (key, val) in packages.iter() { if val .dependencies .iter() .any(|x| changed_packages.contains(x)) { let path = root.join(key) if let Some(package) = packages.get_ancestor_value(&path) { changed_packages.insert(root.join(key)); end_package_names.insert(package.name.as_str()); } } } }</code></pre>Designing the CLI</a></h1> The final step is to create the CLI, for this I’ll be using Clap</a>. Here we’ll create an enum of commands with some predefined ones so we can do dc test</code>, dc nextest</code>, dc build</code> and dc run</code> where run is the only one without a presupplied template. We’ll pop some convenience methods on the arg types to get out the right templates, paths, and other things depending on defaults or what the user supplies. Aside from that every command will have a --no-run</code> option and other args will be supplied after --</code>. Clap doesn’t seem to allow me to grab all the unexpected args as a Vec<String></code> or some sort of map otherwise, which makes sense but is a bit of added friction to the CLI. dc test -- --all-features</code></pre> That CLI interface defined with Clap is as follows:: const CARGO_TEST_TEMPLATE: &'static str = "cargo test {% for pkg in packages %} -p {{ pkg }} {% endfor %} {% for arg in args %} {{ arg }} {% endfor %}"; const CARGO_NEXTEST_TEMPLATE: &'static str = "cargo nextest {% for pkg in packages %} -p {{ pkg }} {% endfor %} {% for arg in args %} {{ arg }} {% endfor %}"; const CARGO_BUILD_TEMPLATE: &'static str = "cargo build {% for pkg in packages %} -p {{ pkg }} {% endfor %} {% for arg in args %} {{ arg }} {% endfor %}"; const CARGO_BENCH_TEMPLATE: &'static str = "cargo build {% for pkg in packages %} -p {{ pkg }} {% endfor %} {% for arg in args %} {{ arg }} {% endfor %}"; #[derive(Debug, Parser)] pub enum RunCommand { Test(RequiredArgs), Nextest(RequiredArgs), Build(RequiredArgs), Bench(RequiredArgs), Run(Args), } impl RunCommand { pub fn required_args(&self) -> &RequiredArgs { match self { Self::Test(a) | Self::Nextest(a) | Self::Build(a) | Self::Bench(a) => a, Self::Run(a) => &a.required, } } pub fn command(&self) -> Option<Cow<'_, str>> { match self { Self::Test(_) => Some(CARGO_TEST_TEMPLATE.into()), Self::Nextest(_) => Some(CARGO_NEXTEST_TEMPLATE.into()), Self::Build(_) => Some(CARGO_BUILD_TEMPLATE.into()), Self::Bench(_) => Some(CARGO_BENCH_TEMPLATE.into()), Self::Run(a) => a.command.as_ref().map(|x| x.into()), } } } #[derive(Debug, Parser)] pub struct RequiredArgs { /// Get the project to run on, runs in current directory otherwise. #[arg(short, long)] input: Option<PathBuf>, /// Generate command but don't run it #[arg(long)] no_run: bool, /// These will be passed to the minijinja template as the args variable #[arg(last = true)] args: Vec<String>, } impl RequiredArgs { fn path(&self) -> PathBuf { match self.input.as_ref() { Some(s) => s.clone(), None => env::current_dir().unwrap(), } } } #[derive(Debug, Parser)] pub struct Args { /// Run the following command. This accepts a minijinja template where /// `packages` is a list of packages that can be included and `excludes` /// is a list of packages that can be excluded. For a cargo test you can /// write the template /// `cargo test {% for pkg in packages %} -p {{ pkg }}{% endfor %}` #[arg(short, long)] command: Option<String>, #[command(flatten)] required: RequiredArgs, }</code></pre>Missing Pieces</a></h1> With this the user can add their own -p</code> or -e</code> to the cargo test templates for things they always want to test i.e. when things might break because of stuff not in the package workspace. I haven’t made any effort to remove conflicts in that case. There’s also probably stuff with propagating environment to the process which is easy enough to do but I don’t have a need for it. Another thing would be applying this to multiple commits or between branches. I’ll likely look into that the first moment I feel a need for it myself - I don’t see it being too much extra just maybe a bit of faff. Overall, I’ve made something that works on my projects and I don’t have a strong motivation to make into a more general project for a wider community. I have a feeling this is solved by other systems better and that’s fine - I don’t do a lot of this stuff looking for users more to scratch an itch. Will I Use This?</a></h1> Probably not, but maybe. Recently Github has been having “issues” as I’m sure a lot of people are aware. Seems like they’re targeting the lofty heights of one 9 of uptime. And during this at $day_job we’ve seen CI pipelines in workspaces that normally take 4 minutes take upwards of 50 minutes. Maybe this would make them take more like 20 minutes when Github Actions are having a moment. But then with degraded performance like that no bets are off. The main thing is I got to do something a bit different and have fun so I’ll chalk that up as a win and move on with my life.
2025 in Review 2026-01-03T00:00:00+00:00 I did one last year so it’s probably time for another year in review. It’ll be mentioned more in the non-technology section but I did spend 6 months in Japan and that will be a theme in why I haven’t made as much progress in side projects as I desired. tarpaulin</a></h1> Tarpaulin had some significant speed improvements in 2025 as well as improvements on memory usage in the llvm coverage parsing. Additionally, the llvm profraw parsing code I’ve written in llvm-profparser</a> is being used by Ferrous Systems in a new project. They’ve helped by contributing more testing, and fixes to parts of the library which Tarpaulin doesn’t hit and it’s been a pleasure collaborating with them on something I wrote. The focus in 2026 will be taking more of the existing issues and closing more and more of them. Hopefully, UX issues with workspaces and excluding code from results can be solved in a way that’s more intuitive to people. Tarpaulin has seen 14 releases in 2025 (and including New Years Day). That’s 3 more releases than last year and from 12 contributors (including me and ignoring dependabot). streaming audio API blog series</a></h1> In 2024 I started a blog series on streaming audio APIs</a> with 4 posts released and a working system which could be used as a basis for things like streaming transcription services. In 2025 I only managed to release a single post but with 2 others in a draft state. Currently, there’s a post on implementing the production metrics for these sorts of APIs. It’s in revisions and I hope to get it out within a month or two. But first I want to add more details to the section on tokio-metrics… After that I aim on releasing the one on batching futures for performance. I might be ambitious and aim for the Opentelemetry one but given how terrible the Otel developer experience is that one might take a long time… wiremocket</a></h1> Last year I released wiremocket</a> to minimal interest. This was expected as not everyone likes mocking, and most people don’t like websockets. However, it’s as complete as I need for the few times I need it which means unless people raise issues or I need more it can be seen as feature complete. My current plan is just to keep bumping up dependency versions as new versions come out and otherwise keep it in a light maintenance mode. asyncapiv3</a></h1> I did start some work on improving asyncapiv3</a> submitting 7 PRs in the end. My hope was to make something similar to an openapi experience for websocket APIs. I’ll have to go back into my notes for this but I believe the library should be at a place where it can be used to power something similar to utoipa</a> in Rust. Currently at my day job, clients consuming websocket APIs is lower priority than it used to be. This does mean my work on this has somewhat stalled. trustfall</a></h1> Last year I got into trustfall</a> some more and wrote a post on it</a>. I also, revived Predrag’s WIP PR for a trustfall docs site and filled in some more details (See the PR here</a>). There’s a tracking issue</a> for ideas for the docs and missing holes. I hope to get back to this work and hopefully something will be up and hosted this year. Trustfall is a cool piece of technology and helping making it easier to adopt would make me personally happy! outside of technology</a></h1> I mentioned in the past year in review I’d be spending some months in Japan. All in all that was 6 months during which I did get married with a traditional Shinto ceremony. Obviously, a large landmark in life and definitely a unique experience, the kimono was like 5-7 layers and in 85% humidity it does become a feat of human endurance. In Japan my Japanese progress has obviously leapt forwards. It’s not where I want it to be but spending most of my time around people with little to no English has sharpened up my listening comprehension and speaking skills. I also helped out on the family rice farm splitting my time between my day job and rice farming for a few months. I did get to see a reasonable amount of Japan, a lot around the Shizuoka prefecture where I was living, Tokyo, Nagoya, Nagano, Hiroshima, Imabari, Yamanashi. I should be back again this year and hopefully I’ll have time to explore more of Japan north of Tokyo. Reading wise I just managed to finish Hyperion by Dan Simmons, and I’m partway through How to Blowup a Pipeline by Andreas Malm. I didn’t get to any of my other reading targets too many other things going on! I did start reading more manga though, mainly classics and seinen with Beserk and Oyasumi Punpun being the ones that have taken the most of my time. I did definitely play a lot more games though. I finished everything I planned to last year (1000xRESIST, Chasing Static, Hollowbody, Half Life 1). I also picked and completed other fun indies like Parking Garage Rally Circuit</a> and Selaco</a>. Cult of the Lamb</a> was also something I really enjoyed. But the most time I’ve put in a game last year has to be Dwarf Fortress</a> where I’ve racked up around 100 hours in 2025. In terms of TV/Film, I’ve mainly watched a lot of anime since getting back to the UK. All of Naruto + Naruto Shippuden, Orb, Dandadan, Spy X Family and Yuru Camp are the ones that spring to mind as standouts. For film I’ve watched a lot and some standouts (not animated) are: Shin Kamen Rider (Dir. Hideaki Anno)</li> Onibaba (Dir. Kaneto Shindō)</li> Decision to Leave (Dir. Park Chan-wook)</li> Memories of Murder (Dir. Bong Joon Ho)</li> Pulse (Dir. Kiyoshi Kurosawa)</li> Weapons (Dir. Zach Cregger)</li> In the Mood for Love (Dir. Wong Kar-Wai)</li> </ul> I’ve also started trying to watch all Akira Kurosawa films: Seven Samurai, Ran and Throne of Blood are all amazing. For animated films Jin Roh - Wolf Brigade is an amazing work of animation and 100 meters might get me on a sport anime kick. Plus the Chainsaw Man movie was an amazing spectacle. Ideas of Crater and Coverage 2025-11-15T00:00:00+00:00 Recently I’ve been doing tackling some issues in my open source projects related to Ferrous System’s work to get code coverage for libcore - the Rust core library. Because of this I’ve been talking to Jyn</a> a fair bit and this has lead to some ideation my side, about combining this work with crater. Jyn telling me this idea sounded cool is what gave me the push to write this up - so blame or thank them as necessary 😅. Of course libcore isn’t libstd, but having coverage for core should result in coverage for libstd potentially being possible with no extra work! What’s Crater?</a></h1> If you know what crater is feel free to skip ahead, this won’t be anything groundbreaking in terms of insights! Crater</a> helps detect regressions in the Rust compiler by grabbing all the projects on crates.io and github and then running cargo build/check</code> and cargo test</code> on them. This helps make sure that changes to the language or standard library don’t cause any breakages in users projects. It won’t run every project, ones with flaky tests or that can’t run in the environment (a linux host) often get blacklisted. But it will run through a lot. Looking, at the crater queue</a> I can see one of the current runs is going over 1471548 jobs which is more than the number of crates on crates.io, looking at a report I can see it’s testing multiple versions of crates so that’s likely why. What is Coverage?</a></h1> Code coverage is a means to measure how much of your code is hit by your tests. The simplest metrics are very coarse, just lines or functions, and they get more complex looking at branches, boolean subconditions or combinations of them (MC/DC coverage). It’s a reasonable heuristic for spotting what part of your code isn’t being exercised by your tests, but it doesn’t help you determine the quality of your tests. Because of this it’s often effective when used in combination with things like mutation testing, or other testing methods. Coverage might also be coupled with formal verification tooling like kani to ensure that the verification harness isn’t overly constrained. In the safety critical world you’ll typically have some form of requirements-based testing and your tests contain a written brief which references your requirements and design and what they’re exercising. Then you can assess both your code coverage and you requirements coverage. If you have 100% requirements coverage and only 60% code coverage something is probably wrong - either some requirements are poorly tested or your system is under-specified. Likewise if you had 60% requirements coverage and 100% code coverage you can’t have confidence that all your tests are meaningfully testing the system. This is just a general idea of the concept and how it’s useful in engineering. When I was working in a safety-critical domain we had requirements based tests, real world data simulation tests and something akin to property based testing. With all of these in combination it’s possible to deliver incredibly robust software like flight control computers etc. Okay Then What’s Your Idea?</a></h1> Well it should be obviously at this point. But use crater to get coverage statistics of how rust crates in the ecosystem are using the standard library. Are there any parts that tests in the wild just don’t hit? Mainly just curiosity at what the data says. Having a lot of eyes (crates) on an API makes it easier to spot regressions, so what aren’t we looking at? Also, given the limitations to what crates crater can run and the impacts of the blacklist of some crates. This could function of a benchmark of craters effectiveness as a smoke test. Maybe we could spot some areas where changes to what crater can run could have the largest gains in it’s usefulness. Plus when it comes to assessing nightly (library) features, which ones are more or less widely tested? With source code analysis we can come up with usage but we don’t necessarily know how much that code is being ran. There’s also potential to couple source analysis and coverage analysis, an initial thought is that the ratio of what people use but don’t test could show where things are hard to write tests for. Of course getting more data for analysis and doing something useful with that analysis is another matter entirely. But by putting this idea out maybe some people might have useful ideas, and that’s the aim of this post. Making CLI Tools with Trustfall 2025-08-18T00:00:00+00:00 For those who aren’t aware, Trustfall</a> is a pretty cool tool. From the repo about section it says: A query engine for any combination of data sources. Query your files and APIs as if they were databases! </blockquote> And this is pretty powerful as a concept, and I’ve read blogposts - mostly by Predrag (blog here</a>), I’ve seen a talk or two. And as a result, I’ve been looking for a place to play with Trustfall. Well, I’ve finally done it and here’s my writeup of the general experience and how to use Trustfall yourself. But before I get into that it would be remiss to not mention there’s already a nifty CLI tool using Trustfall and that’s cargo-semver-checks</a>. If you haven’t seen it before check it out it’s very useful. If you want to skip ahead here’s the link</a> for the project I’m talking about in this post. The Tool</a></h1> Docker. It bloats up my storage, I have tons of images as part of work with dev builds and for some projects they’re around 7GB an image. It’s not great. And I’ve been suffering writing bash functions to grep all the images and remove ones with a certain substring in the name, attempts to delete anything over a size. It ends up fiddly and a bit annoying. But each image has a bunch of data associated with it, it seems reasonable to query it and print information or delete images that match a query. Looks like I’ve got a reason to use Trustfall. Side note I do also use podman on some machines, this will be relevant later. Okay, the start of a plan. I’ll make a tool that can query the docker images on my system and retrieve the images that match some predicates and delete them! Getting Started with Trustfall</a></h1> Trustfall uses a GraphQL-esque language to your schema and query. First off we need to create a schema to describe the shape of our data - similar to the relational model. To help create this schema it can be helpful to think of the queries that we want to represent. Using the following CLI command I get can get a list of all the docker images on my machine in an easy to parse format. This is newline delimited json so for ease of reading I’ll show one of the lines in formatted json! $ docker image ls --format=json { "Containers":"N/A", "CreatedAt":"2024-06-07 13:00:09 +0100 BST", "CreatedSince":"13 months ago", "Digest":"\u003cnone\u003e", "ID":"35a88802559d", "Repository":"ubuntu", "SharedSize":"N/A", "Size":"78.1MB", "Tag":"latest", "UniqueSize":"N/A", "VirtualSize":"78.05MB" }</code></pre> Looking at this data, and thinking about what I want to do I can end up with some queries: Finding docker images within a size range (min or max)</li> Docker images older or younger than some date</li> Ones with an exact name match (not using tag)</li> Ones with a name that matches a regex</li> Ones with a name containing a substring</li> </ol> This is just driven off what data I can get via docker image ls --format=json</code>. There is a possibility to get more data, but that would need another call to docker inspect <IMAGE></code>. From this I’ve made this schema in the end: schema { query: Query } type Query { Image: [Image!]! } type Image { repo: String, tag: String, size: Int!, created: String! # Filtering via edges (with parameters) size_in_range(min: Int, max: Int): [Image!]! created_after(timestamp: String!): [Image!]! created_before(timestamp: String!): [Image!]! has_name(name: String!): [Image!]! name_matches(regex: String!): [Image!]! name_contains(substring: String!): [Image!]! }</code></pre> Why is there size_in_range</code> but created_after</code> and created_before</code>? Mainly just laziness and what looks natural in a written query. Internally they can map to the same code and could add a created_in_range</code>. From Schema to Code</a></h1> Firstly, we want to install trustfall_stubgen</code>. This will take our schema and generate some starting code with todo!</code> stubs dotted around and some boilerplate we’d rather not write ourselves. To install it: cargo install trustfall_stubgen</code></pre> Then I’ll make a temp directory and generate the code into it so I can have a look at it: mkdir tmp trustfall_stubgen --schema schema.graphql --target tmp/</code></pre> In this there’s an adapter</code> folder with our generated code, plus our schema is included in the folder as it neededs to be outputted as a string by one method. I tend to generate it into a separate folder instead of directly into my source folder just to avoid potentially overwriting something and not noticing. Which is definitely a bit paranoid given I’m using version control but nevertheless. In our brand new adapter module we have: adapter_impl.rs</li> edges.rs</li> entrypoints.rs</li> mod.rs</li> properties.rs</li> schema.graphql</li> tests.rs</li> vertex.rs</li> </ul> We don’t have to touch all of these, edges.rs, entrypoints.rs, properties.rs and vertex.rs are the only files I’ve had to edit. So one by one, let’s go through and fill in this code! I’m going to present these in an order which builds things up gradually and (in my opinion) is the clearest order to implement. Writing Your vertex.rs</a></h2> In the Trustfall model a vertex is like a table in SQL, this will contain our parsed docker image data, we don’t really have any other relations in application. It’s also an enum and this is provided in the file for us to fill in as so: #[non_exhaustive] #[derive(Debug, Clone, trustfall::provider::TrustfallEnumVertex)] pub enum Vertex { }</code></pre> Now I typically don’t like putting my data definitions in here and instead define them outside of the adapter and import them in. So in my src/image.rs</code> I add a type definition for the Image</code> and any methods, conversions or other utilities that I need for it. That way my vertex.rs is just vertex related stuff. My image definition is as follows, taking all the useful data from the inspected image in useful form: #[derive(Debug, Clone, Eq, Hash, PartialEq, Ord, PartialOrd)] pub struct Image { pub hash: String, pub repository: String, pub tag: String, pub size: usize, pub created_at: Timestamp, }</code></pre> And then my vertex.rs I update to be: use std::sync::Arc; #[non_exhaustive] #[derive(Debug, Clone, trustfall::provider::TrustfallEnumVertex)] pub enum Vertex { Image(Arc<crate::Image>), }</code></pre> The eagle-eyed amongst you may realise this Image</code> type doesn’t match the schema: type Image { repo: String, tag: String, size: Int!, created: String! }</code></pre> It mostly does, it’s just created is a different type. This is because Trustfall doesn’t (yet) have a timestamp type and I’m also using jiff::Timestamp</code>. Later on when going between the Trustfall queries and my types there’ll have to be some conversion between the two. There is a clear difference between Rust types that are nice to work with, and queries that are nice to work with. At times these can clash and then you get these small differences. Writing Your entrypoints.rs</a></h2> An entrypoint is where you populate your data to query, it outputs a VertexIterator</code>. Here we want to take our data source and output our vertices. This could be calling an API, loading a file and deserializing it, calling a CLI and parsing it’s output. Here is where podman becomes relevant, because the docker image ls --format=json</code> output is different between the two. Docker outputs a newline delimited json objects, whereas podman outputs a json list. Additionally, the fields in each object vary (fun). With this in mind I made an ImageOutput</code> type and a conversion to my Image</code> type used in my Vertex</code> as follows: #[derive(Deserialize)] #[serde(untagged)] pub enum ImageOutput { Podman(podman::Image), Docker(docker::Image), } impl From<ImageOutput> for Image { fn from(x: ImageOutput) -> Self { match x { ImageOutput::Podman(p) => p.into(), ImageOutput::Docker(d) => d.into(), } } } impl From<podman::Image> for Image { fn from(img: podman::Image) -> Self { // SNIP } } impl From<docker::Image> for Image { fn from(img: docker::Image) -> Self { // SNIP } }</code></pre> With that boilerplate out of the way here is my entrypoint: pub(super) fn image<'a>(_resolve_info: &ResolveInfo) -> VertexIterator<'a, Vertex> { let version = Command::new("docker") .args(["--version"]) .output() .expect("couldn't get docker version"); let version = String::from_utf8_lossy(&version.stdout); let is_podman = version.contains("podman"); let images = Command::new("docker") .args(["image", "ls", "--format", "json"]) .output() .expect("failed to run docker"); let images: Vec<ImageOutput> = if is_podman { serde_json::from_slice(&images.stdout).expect("couldn't deserialize the json output") } else { let s = String::from_utf8_lossy(&images.stdout); let mut v = vec![]; for line in s.lines() { v.push(serde_json::from_str(line).expect("couldn't deserialize the json output")); } v }; Box::new( images .into_iter() .map(|x| Vertex::Image(Arc::new(x.into()))), ) }</code></pre> This is all relatively simple code, some code to detect if it’s docker or a podman alias, deserialization and then return a type that meets the requirements of VertexIterator</code> which is a Box<dyn Iterator<Item = VertexT> + 'vertex></code>. As a side, you can also just remove this file. And instead in adapter_impl.rs</code> have the type which the trustfall::provider::Adapter</code> is implemented for generate your Vertex</code>. That approach might be desirable if the type that implements Adapter</code> has a number of member variables you want to use when generating the vertices. Also, don’t worry about ResolveInfo</code>, it’s for optimisation purposes and more complicated. For this project I have one source and that’s the docker images on my machine. The fact there’s no fancy configuration means I can keep things simple and stick to the generated structure. Writing Your properties.rs</a></h2> Now we have a Vertex</code>, and we have the means to get a sequence of vertices into the query engine. The only things missing to execute queries is extracting data from the vertex and running our defined queries. First, let’s get the data out. That way we can run simple queries with no filtering like print every docker image. The starting point that trustfall_stubgen</code> gives us is as follows: use trustfall::{FieldValue, provider::{AsVertex, ContextIterator, ContextOutcomeIterator, ResolveInfo}}; use super::vertex::Vertex; pub(super) fn resolve_image_property<'a, V: AsVertex<Vertex> + 'a>( contexts: ContextIterator<'a, V>, property_name: &str, _resolve_info: &ResolveInfo, ) -> ContextOutcomeIterator<'a, V, FieldValue> { match property_name { "created" => { todo!("implement property 'created' in fn `resolve_image_property()`") } "repo" => todo!("implement property 'repo' in fn `resolve_image_property()`"), "size" => todo!("implement property 'size' in fn `resolve_image_property()`"), "tag" => todo!("implement property 'tag' in fn `resolve_image_property()`"), _ => { unreachable!( "attempted to read unexpected property '{property_name}' on type 'Image'" ) } } }</code></pre> Here we have a ContextIterator</code> and we want to transform that into a ContextOutcomeIterator</code>. These type aliases with some simplification look as follows: pub type ContextIterator<VertexT> = Box<dyn Iterator<Item = DataContext<VertexT>>>; pub type ContextOutcomeIterator<VertexT, OutcomeT> = Box<dyn Iterator<Item = (DataContext<VertexT>, OutcomeT)>>;</code></pre> The contexts are what we want to resolve, so for each property name I’ll create a mapping from that context and property to the same context paired with the value of that property. I believe this all came together from looking at some example Trustfall adapter implementations from Predrag’s blog and talks but it was a while ago so I can’t trace back to exactly how I learned this. Just implementing this for the created</code> property looks as follows: pub(super) fn resolve_image_property<'a, V: AsVertex<Vertex> + 'a>( contexts: ContextIterator<'a, V>, property_name: &str, _resolve_info: &ResolveInfo, ) -> ContextOutcomeIterator<'a, V, FieldValue> { let func = match property_name { "created" => |v: DataContext<V>| match v.active_vertex() { Some(Vertex::Image(img)) => { let created_at = img.created_at.to_string().to_string(); (v, created_at) }, None => (v, FieldValue::Null), }, "repo" => todo!("implement property 'repo' in fn `resolve_image_property()`"), "size" => todo!("implement property 'size' in fn `resolve_image_property()`"), "tag" => todo!("implement property 'tag' in fn `resolve_image_property()`"), _ => { unreachable!( "attempted to read unexpected property '{property_name}' on type 'Image'" ) } }; Box::new(contexts.map(func)) }</code></pre> Here I have to convert my jiff::Timestamp</code> to a string and then use the Into</code> to turn it into a FieldValue</code>, for types with no conversion this is a bit simpler. But generally speaking this is reasonably straightforward. Get the active vertex (of which we only have one potential type), extract the property and return it. One important thing to note, cloning v</code> is easy and what I initially did. However, it’s not necessary and often you can avoid a potentially expensive clone by binding the second element of the tuple to a new variable so the compiler doesn’t try and take a reference to a temporary value. Writing Your Edges.rs</a></h2> Last is edges, here we have the most generated code and we implement the queries in our schemas. If you scroll right down to the bottom you’ll see a function for each query like: pub(super) fn created_after<'a, V: AsVertex<Vertex> + 'a>( contexts: ContextIterator<'a, V>, timestamp: &str, _resolve_info: &ResolveEdgeInfo, ) -> ContextOutcomeIterator<'a, V, VertexIterator<'a, Vertex>> { resolve_neighbors_with( contexts, move |vertex| { let vertex = vertex .as_image() .expect("conversion failed, vertex was not a Image"); todo!("get neighbors along edge 'created_after' for type 'Image'") }, ) }</code></pre> The steps here are once again not too hard, we have our vertex already extracted for us. We just need to see if it matches the query and return it in an iterator if it does. Here we get our timestamp once again in a string so we need to do the conversion. Also, because I want to return that vertex I’ve removed the name shadowing. But below is the code: pub(super) fn created_after<'a, V: AsVertex<Vertex> + 'a>( contexts: ContextIterator<'a, V>, timestamp: &str, _resolve_info: &ResolveEdgeInfo, ) -> ContextOutcomeIterator<'a, V, VertexIterator<'a, Vertex>> { let ts = timestamp.parse::<Timestamp>().unwrap(); resolve_neighbors_with(contexts, move |vertex| { let image = vertex .as_image() .expect("conversion failed, vertex was not a Image"); if image.created_at > ts { Box::new(std::iter::once(vertex.clone())) } else { Box::new(std::iter::empty()) } }) }</code></pre> Repeat something like this for all the queries and then we have a working Trustfall adapter! Using the Adapter</a></h1> Using this adapter if I wanted to get all the docker images created in June this year I could use the following query: { Image { created_after(timestamp: "2025-06-01 00:00:00+00") created_before(timestamp: "2025-07-01 00:00:00+00") repo @output tag @output size @output created @output } }</code></pre> If I want to then execute that query it looks like so (the query is in the variable query</code> for brevity): use adapter::*; use trustfall::{FieldValue, execute_query}; use std::{collections::BTreeMap, sync::Arc}; let adapter = Arc::new(Adapter::new()); let args: BTreeMap<Arc<str>, FieldValue> = BTreeMap::new(); let vertices = execute_query(Adapter::schema(), adapter, query, args).unwrap();</code></pre> The vertices is an iterator over BTreeMap<Arc<str>, FieldValue></code>. Each BTreeMap</code> will contain the repo, tag, size and created date of a docker image. Now for a CLI each CLI argument will map to at least one query, and you can iteratively build up a Trustfall query going over the CLI args. Currently, Trustfall is “GraphQL-like” in that it doesn’t support everything in GraphQL. This means features like query arguments aren’t yet supported and you should place the values straight in the constructed query string. My query string construction is a fair amount of code, but not very nested and more repetitive. Here’s a snippet of it to see how I use it: let mut query_str = "{Image{".to_string(); if let Some(created_before) = filter.created_before { query_str.push_str(&format!( "created_before(timestamp: \"{}\")\n", created_before )); } // SNIP if let Some(contains) = &filter.name { query_str.push_str(&format!("has_name(name:\"{}\")\n", contains)); } query_str.push_str("repo @output\ntag @output\nsize @output\ncreated @output\n"); // Mainly for ease of readability query_str.push_str("}}");</code></pre> Then once I have the list of vertices I can map it to a printout, run docker commands on them. Anything I desire! The main part here is just using the queried data and creating a nice CLI interface to interact with it. Small Bonus UX</a></h2> Before I go, on the theme of nice CLI interface, I’m using the human-size</a> crate for working with sizes like 2GB</code> etc. Because of this all my size based arguments accept either the number of bytes or a human size. For the snippet I’ll put below I can enter --smaller-than 2000000000</code> or --smaller-than 2GB</code> which I was very happy with. /// Common filter options for all commands #[derive(clap::Parser, Debug)] pub struct FilterOptions { /// Only include files smaller than this size in bytes #[arg(long, value_parser = parse_human_size)] pub smaller_than: Option<usize>, } fn parse_human_size(input: &str) -> Result<usize, String> { match human_size::Size::from_str(input) { Ok(size) => Ok(size.to_bytes() as usize), Err(_) => input .parse::<usize>() .map_err(|e| format!("Invalid size '{}': {}", input, e)), } }</code></pre> And that’s it, peruse the code</a> if you want to see more of how it came together. This was just a weekend project so I’ll likely refine the interface a bit more as I use it more but for now it’s already useful for me. But Wait</a></h1> Fairly content with myself I wrote up the above and sent it around a bit for feedback. And Predrag dropped a bit of a surprise, most of my queries can just be @filter</code> clauses. Huh, I never knew they existed. Looking it up @filter</code> isn’t part of GraphQL but is apparently common in some things built on top of GraphQL. Currently, if you want to see what a filter can do you can check the ops here: docs.rs</a>. Then couple that with looking up some of the example queries to see how to use them, there’s a Trustfall playground and you can see some queries there</a>. Removing our edge queries and adding filter directives we’d change the following: { Image{ name_matches(substring: "$name_regex") size_in_range(min: $larger_than, max: &smaller_than) name @output size @output created @output } } </code></pre> Into this: { Image{ name @output @filter(op: "regex", value: ["$name_regex"]) size @output @filter(op: "<", value: ["$smaller_than"]) @filter(op: ">", value: ["$larger_than"]) created @output } }</code></pre> There is some difference in how we call this with trustfall::execute_query</code> as well. With the first one we need to replace the arguments in the query string, whereas in the second we need to use the args</code> argument to supply our named arguments. Filter operations in Trustfall won’t work with literals preventing us from placing the values in on query generation. Now comes the time to rework things and see how much code can be deleted. But first this will be my new Trustfall schema with unnecessary edge queries removed: schema { query: Query } type Query { Image: [Image!]! } type Image { name: String, repo: String, tag: String, size: Int!, created: String! # Filtering via edges (with parameters) created_after(timestamp: String!): [Image!]! created_before(timestamp: String!): [Image!]! }</code></pre> I’ve added a name field which will be the string {repo}:{tag}</code>, for convenience. let mut query_args: BTreeMap<Arc<str>, trustfall::FieldValue> = BTreeMap::new(); let mut query_str = "{Image{".to_string(); // SNIP let mut size_filter = String::new(); let mut name_filter = String::new(); if let Some(contains) = &filter.name { name_filter.push_str(r#"@filter(op: "=", value: ["$name_eq"])"#); name_filter.push('\n'); query_args.insert(Arc::from("name_eq".to_string()), contains.into()); } query_str.push_str(&format!( "name @output\n{name_filter}size @output\n{size_filter}created @output\n" )); query_str.push_str("}}"); let adapter = Arc::new(Adapter::new()); let vertices = execute_query(Adapter::schema(), adapter, &query_str, query_args).unwrap();</code></pre> How many lines were deleted, time to see what the commit says: 6 files changed, 55 insertions(+), 148 deletions(-)</code></pre> Not as many lines removed as I would have thought before starting but the need to build up the args map added some extra lines. Conclusion</a></h1> I’ll be refining the interface of this tool to become more useful for my own needs. However, I don’t plan on publishing it or it getting usage from other people. The main hope is that this introduces some people to Trustfall and how they can start to utilise it as a query engine for their own data. Going forward, I’m also going to work more on the Trustfall official documentation, some of which I’ve already started doing. If this has been of interest and you want to learn more check out Predrag’s site</a> for more talks/blogposts and other things. Also, a big thanks to Predrag for taking the time to talk to me about Trustfall and answer some of my questions plus provide feedback on this post. Tarpaulins Week of Speed pt2 2025-05-20T00:00:00+00:00 You might want to read the previous post</a>. This post is going to be much the same stuff, while skipping going over things I talked about before. And with some changes in approach, and different tools than perf</code> and cargo flamegraph</code>! New Tools!</a></h1> Firstly, now that polars doesn’t take 40 minutes for a tarpaulin run, I’ll be using hyperfine</a> to get some statistics on the tarpaulin runs. And from the comments of the last post from /u/Shnatsel: For profiling I’ve moved away from perf + flamegraphs to https://crates.io/crates/samply</a>, which provides the same flamegraphs plus much more, and you even get to share your results to anyone with a web browser in just two clicks. It’s also portable to mac and windows, not just linux. It has become my go-to tool for profiling and I cannot recommend it enough. </blockquote> Strong praise indeed, so I’m finally giving it a try! The First Win</a></h1> Using samply I saw that HashMap::get</code> in generate_subreport</code> was taking most of the time in report generation now. This time I ran it on the llvm-profparsers CLI tool instead of tarpaulin on some files I had lying around from another project I was testing. At this point I was mainly seeing if samply worked. Running samply was as simple as: samply record target/release/cov show --instr-profile benches/data/mapping/profraws/* --object benches/data/mapping/binaries/*</code></pre> Now this code does go over the region expressions and generate count values and add to a list of pending expressions. You can see the full function at this point in time here</a>. But it’s long so I won’t paste it all in but instead explain a bit. There is a map of region IDs to counts and as we go over the expression list, we get elements from the initial version and update values as we resolve expressions. The HashMap::get</code> calls happen in loops as we go over the expression tree. Initially, we do one pass through to build up the pending expressions and then loop over the pending list until we finish it. Reducing the number of times we loop over things should speed things up. I noticed previously that the pending expressions are represented by a flattened tree, which means the deeper nodes are located at indices further into the list. This means that, potentially, we can iterate over the list fewer times if we iterate over it in reverse first. Processing those deeper nodes first will mean that the nodes above them in the tree structure will get resolved quicker. With that in mind I did the following diff: < for (expr_index, expr) in func.expressions.iter().enumerate() --- > for (expr_index, expr) in func.expressions.iter().enumerate().rev()</code></pre> Running the benchmark and… Benchmarking report generation: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. report generation time: [806.43 ms 809.27 ms 812.17 ms] change: [-79.923% -79.433% -78.921%] (p = 0.00 < 0.05) Performance has improved. Benchmarking subreport generation: Warming up for 3.0000 s Warning: Unable to complete 100 samples in 5.0s. subreport generation time: [117.87 ms 119.28 ms 120.84 ms] change: [-22.306% -21.266% -20.132%] (p = 0.00 < 0.05) Performance has improved. Found 13 outliers among 100 measurements (13.00%) 9 (9.00%) high mild 4 (4.00%) high severe</code></pre> This takes report generation from ~3.8s to ~0.8s. I also noticed in report generation there was a HashMap</code> where I could reserve the additional capacity and maybe save some allocations. Running samply again I see the part of the code taking the most time is now the profraw parsing not report generation! Oh dear glancing briefly at the report parsing I see: let name = symtab.names.get(&data.name_ref).cloned(); let (hash, name_hash) = if symtab.contains(data.name_ref) { // SNIP };</code></pre> That can just be a if name.is_some()</code> and avoid another HashMap</code> lookup. When I compared this parsing code to llvm, my initial implementation was significantly faster. With the difficulties in the coverage mapping part of the code and then its performance issues, I didn’t really go back and see if I could speed up the profraw parsing anymore, but this should be an easy win. Benchmarking Results #1</a></h1> With three changes under our belt lets take a look at the impact. This time I’m running on: Polars</a> (commit 5222107)</li> tokio</a> (commit 0cf95f0)</li> datafusion-common</a> (commit 3e30f77)</li> jiff</a> (commit 08abead)</li> ring</a> (commit a041a75)</li> </ul> I do one warmup run and for some crates I’ve slightly adapted the tarpaulin command to either get it working, make it not unreasonably slow or include even more code/features. I’ll also be checking the change in coverage to make sure the iteration order change doesn’t cause a different in the results. For each tarpaulin invocation I share this was ran in hyperfine as: hyperfine --warmup 1 --export-markdown results.md "$TARPAULIN_COMMAND" </code></pre> After running these I realised I could have ran both tarpaulin commands together if I used absolute paths instead of changing my local tarpaulin installation. I’ll keep that in mind for next time. Running polars with the command cargo tarpaulin --engine llvm --skip-clean</code>: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 155.460 ± 1.425</td> 152.717</td> 157.120</td></tr> iteration order</td> 135.417 ± 3.720</td> 129.325</td> 139.286</td></tr> </tbody></table> Running datafusion-common with the command cargo tarpaulin --engine llvm --skip-clean -p datafusion-common</code>: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 10.075 ± 0.069</td> 9.942</td> 10.155</td></tr> iteration order</td> 9.527 ± 0.092</td> 9.407</td> 9.669</td></tr> </tbody></table> Running jiff with cargo tarpaulin --engine llvm --skip-clean --all-features</code>: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 3.241 ± 0.011</td> 3.218</td> 3.260</td></tr> iteration order</td> 3.063 ± 0.064</td> 2.892</td> 3.119</td></tr> </tbody></table> Running ring with cargo tarpaulin --engine llvm --skip-clean --all-features --release</code>: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 77.923 ± 0.954</td> 77.394</td> 80.529</td></tr> iteration order</td> 76.485 ± 0.124</td> 76.211</td> 76.671</td></tr> </tbody></table> Running tokio with cargo tarpaulin --engine llvm --skip-clean --all-features</code>: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 68.236 ± 0.563</td> 67.080</td> 68.992</td></tr> iteration order</td> 59.273 ± 0.477</td> 58.751</td> 60.484</td></tr> </tbody></table> This improvement ranges from 2% to 14% faster than the previous version. Moreover, the coverage never changed so we can be pretty sure this doesn’t introduce any issues! Going for a Second Win</a></h1> Time to generate another flamegraph and look for something to attack. This time here’s a screenshot from samply of how it looks, hopefully nothing unfamiliar: Looking at the flamegraph now for polars I can see reading the object files is a big one. I do use fs::read</code> to read the object files and get the sections for parsing of relevant information. But I can probably avoid reading the entire file and provided those sections are before the executable code hopefully avoid reading some data. Before I go too deeply into this lets just patch the code and see what Criterion says. Right changing this: let binary_data = fs::read(object)?; let object_file = object::File::parse(&*binary_data)?;</code></pre> Into: let binary_data = ReadCache::new(fs::File::open(object)?); let object_file = object::File::parse(&binary_data)?;</code></pre> And the parsing functions need a generic parameter added so that &Section<'_, '_></code> turns into &Section<'data, '_, R></code> where R: ReadRef<'data></code>. An initial run with criterion confirms this assumption: coverage mapping time: [23.541 ms 23.817 ms 24.115 ms] change: [-70.666% -69.879% -69.128%] (p = 0.00 < 0.05) Performance has improved. Found 6 outliers among 100 measurements (6.00%) 6 (6.00%) high mild</code></pre> Next step is to look at the order of the object sections and call my parsing code in that order so we roughly read things in order. That should be more cache friendly and hopefully criterion agrees… Using objdump -h</code> we can list the headers in an ELF file and their offsets. I don’t see a reason for LLVM to be changing the order based on architecture and most users are on Linux anyway, so we’ll stick with the Linux order. Running it on one of the benchmark binaries with some of the output removed for readability: $ objdump -h benches/data/mapping/binaries/test_iter-e3db028e51c53c16 benches/data/mapping/binaries/test_iter-e3db028e51c53c16: file format elf64-x86-64 Sections: Idx Name Size VMA LMA File off Algn // SNIP 29 __llvm_prf_cnts 0004ed10 000000000063f9e8 000000000063f9e8 0063f9e8 2**3 CONTENTS, ALLOC, LOAD, DATA 30 __llvm_prf_data 00109900 000000000068e6f8 000000000068e6f8 0068e6f8 2**3 CONTENTS, ALLOC, LOAD, DATA // SNIP 35 __llvm_covfun 0038cc95 0000000000000000 0000000000000000 0079e198 2**3 CONTENTS, READONLY 36 __llvm_covmap 0000f8d0 0000000000000000 0000000000000000 00b2ae30 2**3 CONTENTS, READONLY // SNIP</code></pre> This is the output limited to the sections I parse. And, there is a difference with my order of parsing. I do llvm_covfun</code>, llvm_covmap</code>, llvm_prf_cnts</code> and then llvm_prf_data</code>. There’s also zero dependencies between the four for parsing so I can safely reorder them as desired. coverage mapping time: [22.903 ms 23.062 ms 23.236 ms] change: [-4.5429% -3.1690% -1.7429%] (p = 0.00 < 0.05) Performance has improved. Found 7 outliers among 100 measurements (7.00%) 4 (4.00%) high mild 3 (3.00%) high severe</code></pre> This seems to back up the hypothesis, although it’s a small change relatively and I wouldn’t trust Criterion that much. Micro-benchmarks tend to be rough indicators rather than sure-fire confirmations. Indeed I sometimes see consecutive Criterion runs differ by up to 10% with no changes to the code. Therefore anything below 10% I take with a grain of salt. However, this change does make logical sense so we’ll keep it. The next step will be running this on our test projects! Hopefully, we’ll see some positive changes. Benchmarking Results #2</a></h1> For a level of consistency, I’ll include the previous table with the new results in as the final row. All commands ran are the same as previously stated. Polars: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 155.460 ± 1.425</td> 152.717</td> 157.120</td></tr> iteration order</td> 135.417 ± 3.720</td> 129.325</td> 139.286</td></tr> file optimisation</td> 93.891 ± 1.646</td> 90.838</td> 96.526</td></tr> </tbody></table> Oh wow another big win, time to see how it does in the other projects! Datafusion-common: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 10.075 ± 0.069</td> 9.942</td> 10.155</td></tr> iteration order</td> 9.527 ± 0.092</td> 9.407</td> 9.669</td></tr> file optimisation</td> 9.300 ± 0.154</td> 8.988</td> 9.491</td></tr> </tbody></table> Jiff: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 3.241 ± 0.011</td> 3.218</td> 3.260</td></tr> iteration order</td> 3.063 ± 0.064</td> 2.892</td> 3.119</td></tr> file optimisation</td> 3.153 ± 0.021</td> 3.112</td> 3.178</td></tr> </tbody></table> Ring: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 77.923 ± 0.954</td> 77.394</td> 80.529</td></tr> iteration order</td> 76.485 ± 0.124</td> 76.211</td> 76.671</td></tr> file optimisation</td> 77.825 ± 0.943</td> 77.140</td> 80.136</td></tr> </tbody></table> Tokio: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 68.236 ± 0.563</td> 67.080</td> 68.992</td></tr> iteration order</td> 59.273 ± 0.477</td> 58.751</td> 60.484</td></tr> file optimisation</td> 60.592 ± 0.800</td> 59.998</td> 62.790</td></tr> </tbody></table> Well aside from polars and datafusion-common everything has gotten a little bit slower. This could be some noise or a sign this approach isn’t perfect. Although, if it’s only ever a small regression it should be a win overall. Thinking about it, if I wrap the File</code> in an std::io::BufReader</code> I might win back some of those losses through larger less frequent file reads… Benchmarking Results #3</a></h1> Time for BufReader</code> to hopefully save the day. Polars: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 155.460 ± 1.425</td> 152.717</td> 157.120</td></tr> iteration order</td> 135.417 ± 3.720</td> 129.325</td> 139.286</td></tr> file optimisation</td> 93.891 ± 1.646</td> 90.838</td> 96.526</td></tr> BufReader</code></td> 87.733 ± 0.300</td> 87.114</td> 88.072</td></tr> </tbody></table> Datafusion-common: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 10.075 ± 0.069</td> 9.942</td> 10.155</td></tr> iteration order</td> 9.527 ± 0.092</td> 9.407</td> 9.669</td></tr> file optimisation</td> 9.300 ± 0.154</td> 8.988</td> 9.491</td></tr> BufReader</code></td> 9.160 ± 0.065</td> 9.055</td> 9.231</td></tr> </tbody></table> Jiff: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 3.241 ± 0.011</td> 3.218</td> 3.260</td></tr> iteration order</td> 3.063 ± 0.064</td> 2.892</td> 3.119</td></tr> file optimisation</td> 3.153 ± 0.021</td> 3.112</td> 3.178</td></tr> BufReader</code></td> 3.118 ± 0.015</td> 3.093</td> 3.139</td></tr> </tbody></table> Ring: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 77.923 ± 0.954</td> 77.394</td> 80.529</td></tr> iteration order</td> 76.485 ± 0.124</td> 76.211</td> 76.671</td></tr> file optimisation</td> 77.825 ± 0.943</td> 77.140</td> 80.136</td></tr> BufReader</code></td> 77.324 ± 0.108</td> 77.213</td> 77.505</td></tr> </tbody></table> Tokio: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 68.236 ± 0.563</td> 67.080</td> 68.992</td></tr> iteration order</td> 59.273 ± 0.477</td> 58.751</td> 60.484</td></tr> file optimisation</td> 60.592 ± 0.800</td> 59.998</td> 62.790</td></tr> BufReader</code></td> 57.703 ± 0.396</td> 56.901</td> 58.145</td></tr> </tbody></table> Okay this looks better again. I could play around with memmap</a>, but it’s not a safe OS level API and I don’t think this is worth adding unsafe into things for. Let’s also take a look at the flamegraph just to see how things have been shaken up Here I actually highlighted the CoverageMapping::new</code> call because whereas before it took up most of the time, now it’s so small you can’t read the symbol name without expanding on the block. I imagine for projects that generate a lot of very large test binaries this change likely impacts the most. This also includes high LoC counts in the crate plus used dependency code as that will increase the symbol table and count metadata in the ELF sections (and other object files of your own choosing). Final Change Today</a></h1> Okay, we’ve done a lot of changes with not a lot of diffs to the project and gotten some big changes in performance. Now there’s one more thing I’d like to try out from this issue</a>. [profile.release] codegen-units = 1 lto = true</code></pre> Link-Time Optimisation (LTO) is a technique that applies optimisations at the link stage taking into account all the crates being linked in and not just optimising crates individually. It should decrease the binary size and hopefully up the performance as the issue states. Let’s try it out: LTO Results</a></h1> Polars: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 155.460 ± 1.425</td> 152.717</td> 157.120</td></tr> iteration order</td> 135.417 ± 3.720</td> 129.325</td> 139.286</td></tr> file optimisation</td> 93.891 ± 1.646</td> 90.838</td> 96.526</td></tr> BufReader</code></td> 87.733 ± 0.300</td> 87.114</td> 88.072</td></tr> LTO</td> 85.787 ± 1.350</td> 82.055</td> 87.070</td></tr> </tbody></table> Datafusion-common: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 10.075 ± 0.069</td> 9.942</td> 10.155</td></tr> iteration order</td> 9.527 ± 0.092</td> 9.407</td> 9.669</td></tr> file optimisation</td> 9.300 ± 0.154</td> 8.988</td> 9.491</td></tr> BufReader</code></td> 9.160 ± 0.065</td> 9.055</td> 9.231</td></tr> LTO</td> 8.944 ± 0.058</td> 8.848</td> 9.020</td></tr> </tbody></table> Jiff: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 3.241 ± 0.011</td> 3.218</td> 3.260</td></tr> iteration order</td> 3.063 ± 0.064</td> 2.892</td> 3.119</td></tr> file optimisation</td> 3.153 ± 0.021</td> 3.112</td> 3.178</td></tr> BufReader</code></td> 3.118 ± 0.015</td> 3.093</td> 3.139</td></tr> LTO</td> 3.065 ± 0.027</td> 3.033</td> 3.109</td></tr> </tbody></table> Ring: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 77.923 ± 0.954</td> 77.394</td> 80.529</td></tr> iteration order</td> 76.485 ± 0.124</td> 76.211</td> 76.671</td></tr> file optimisation</td> 77.825 ± 0.943</td> 77.140</td> 80.136</td></tr> BufReader</code></td> 77.324 ± 0.108</td> 77.213</td> 77.505</td></tr> LTO</td> 77.296 ± 0.129</td> 77.143</td> 77.518</td></tr> </tbody></table> Tokio: Command</th> Mean [s]</th> Min [s]</th> Max [s]</th></tr></thead> baseline</td> 68.236 ± 0.563</td> 67.080</td> 68.992</td></tr> iteration order</td> 59.273 ± 0.477</td> 58.751</td> 60.484</td></tr> file optimisation</td> 60.592 ± 0.800</td> 59.998</td> 62.790</td></tr> BufReader</code></td> 57.703 ± 0.396</td> 56.901</td> 58.145</td></tr> LTO</td> 56.987 ± 0.438</td> 56.427</td> 57.963</td></tr> </tbody></table> Conclusion</a></h1> Well time for the takeaways. I guess generally, people should use samply over perf if it works out of the box. That is if you’re not using any perf features that aren’t supported (if there are any). Doing multiple runs and statistics are nice. Buffering file IO and reading the least amount of data possible does have knock on effects. Preallocate storage when possible and sometimes iteration order matters a lot. Benchmark on a variety of real life workloads as well. There’s not some big magic takeaway here. Performance work is often just trying to get better measurements to judge your changes. Then when you have them trying to reduce allocations, instructions executed and data read/written. Being methodical helps, noting down your results and keeping raw data around for analysis is also very helpful. Now all this is done I should set about the process of releasing it all to the general public. Laters! Tarpaulins Week of Speed 2025-05-08T00:00:00+00:00 This week I’ve dropped two cargo-tarpaulin</a> releases, both of which have significant speed-ups for LLVM coverage reporting. When I say significant, the first one gave me a ~80% speedup on a benchmark, and then the next one was a 90% speedup. Meaning on this benchmark I achieved overall a 98% speedup. Obviously, with getting speed-ups of this significance I was doing something really stupid, that’s true for the first speed-up. For the second one, it wasn’t so much idiocy as a missed opportunity. But before that let’s dive into some context! The Preamble</a></h1> Tarpaulin is a code coverage tool, and as part of coverage it offers two backends: ptrace - UNIX process tracing API, like a debugger. Not fully accurate but more flexible on where you can get coverage data</li> LLVM instrumentation - runtime bundled into a binary which exports some files. Fully accurate, some limitations of what coverage it can report</li> </ol> The LLVM coverage instrumentation was slow, so how does that work? You run your tests</li> It spits out a profraw file</li> Typically you use llvm-profdata (found in the llvm-tools rustup component) to turn a profraw file to a merged (profdata) file</li> Typically you use llvm-cov (also in llvm-tools) to take the executable file and your profdata and make a report</li> </ol> I say typically here because I always find it annoying having to install multiple tools to use one, and when tools go installing other things. This doesn’t matter as much in CI but in my own hubris I implemented the profraw parsing and mapping here</a>. Note I do have a feature planned to optionally use the llvm-tools But anyway, it’s this parsing code that’s the issue. And we can tell this by the fact tarpaulin can go 6 minutes plus on some projects just after printing out Mapping coverage data to source</code> before we see anything else. Of course, the best way to figure things out is benchmarking on realistic input data, and we really want something that will stress tarpaulin so prompted by a users comment on a GitHub issue I’ll be comparing execution times using polars</a>. This isn’t where the benchmark in the intro came from - that’s just the criterion benchmarks in the repo. For development, I looked for a project that a user mentioned. After doing cargo tarpaulin --engine llvm --no-run</code> to avoid measuring the cost of downloading dependencies and building the tests. This is the initial result: Then for testing, I’ll run: time cargo tarpaulin --engine llvm --skip-clean // SNIP tarpaulin output real 41m34.411s user 40m1.784s sys 1m4.368s</code></pre> Looking, I can also see we generated 37 profraws so we’re dealing with at least 37 test binaries and 37 calls to merge the reports. Running polars without tarpaulin but with the coverage instrumentation we get: real 0m3.103s user 0m7.077s sys 0m1.715s</code></pre> So there’s clearly a big gap to close. We can’t get down to 3s but we should try to get as close as possible. Though it is worth mentioning that the instrumentation profiling does seem to balloon the compilation time A LOT. The Idiocy</a></h1> So first off let’s isolate the code that caused the issue. Then once we’ve discussed it I’ll go a bit into how I figured this out and then use that technique again to find the next issue. while !pending_exprs.is_empty() { assert!(tries_left > 0); if index >= pending_exprs.len() { index = 0; tries_left -= 1; } let (expr_index, expr) = pending_exprs[index]; let lhs = region_ids.get(&expr.lhs); let rhs = region_ids.get(&expr.rhs); match (lhs, rhs) { (Some(lhs), Some(rhs)) => { pending_exprs.remove(index); let count = match expr.kind { ExprKind::Subtract => lhs - rhs, ExprKind::Add => lhs + rhs, }; let counter = Counter { kind: CounterType::Expression(expr.kind), id: expr_index as _, }; region_ids.insert(counter, count); if let Some(expr_region) = func.regions.iter().find(|x| { x.count.is_expression() && x.count.id == expr_index as u64 }) { let result = report .files .entry(paths[expr_region.file_id].clone()) .or_default(); result.insert(expr_region.loc.clone(), count as _); } } _ => { index += 1; continue; } } }</code></pre> Here pending_exprs</code> is a Vec<(usize, Expression)></code> where expression is just two counts and a binary operator to apply to them. So pretty small types. This code then goes through a list of expressions that depend on other expressions and evaluates them, stores the result and then passes through until all the expressions are resolved. I remember roughly writing this code, and I remember my main concern was figuring out how the expressions and coverage regions worked. And that’s why I did the big dumb-dumb. And dear reader if you want to take a guess you have until the next paragraph to finish making it before I start revealing. The entire issue is pending_exprs.remove(index);</code>. With this line, I remove an element from potentially the start of the vector, and then have to move every element back filling in the hole. These expressions are also used to work out condition coverage. So the more boolean sub-conditions you have and the more branches in your code the more expressions you have in a function. I haven’t attempted to prove it, but I feel the number of regions should correlate with the cyclometric complexity of a function. That aside there are two “easy” solutions for this: Try to iterate through the list in reverse order so hopefully, you can remove from the back and significantly reduce copies</li> Have a way to mark a vec element as done so you can skip it in the list and do no copies</li> </ol> Intuitively, I feel that 1. has too much risk of copying too much and causing a performance hit. So I went for 2. To do this I make the pending expression type an Option</code> and set it to None</code> instead of removing it. After this, the while statement looks like this: let mut index = 0; let mut cleared_expressions = 0; let mut tries_left = pending_exprs.len() + 1; while pending_exprs.len() != cleared_expressions { assert!(tries_left > 0); if index >= pending_exprs.len() { index = 0; tries_left -= 1; } let (expr_index, expr) = match pending_exprs[index].as_ref() { Some((idx, expr)) => (idx, expr), None => { index += 1; continue; } }; let lhs = region_ids.get(&expr.lhs); let rhs = region_ids.get(&expr.rhs); match (lhs, rhs) { (Some(lhs), Some(rhs)) => { let count = match expr.kind { ExprKind::Subtract => lhs - rhs, ExprKind::Add => lhs + rhs, }; let counter = Counter { kind: CounterType::Expression(expr.kind), id: *expr_index as _, }; region_ids.insert(counter, count); if let Some(expr_region) = func.regions.iter().find(|x| { x.count.is_expression() && x.count.id == *expr_index as u64 }) { let result = report .files .entry(paths[expr_region.file_id].clone()) .or_default(); result.insert(expr_region.loc.clone(), count as _); } pending_exprs[index] = None; cleared_expressions += 1; } _ => { index += 1; continue; } } }</code></pre> After this change, the polars run is: real 2m24.941s user 1m27.586s sys 1m2.708s</code></pre>How to Find Bad Code (Flamegraphs)</a></h1> Now that part is all well and good, a massive win. Pat on the back and go home. But how did I find where to fix my dumb mistake? The relevant part of the parsing code is probably only 500 lines or so. I guess I could have read it or sprinkled some print statements around. But this is serious work so I aim to be a smidgen scientific. And in enter flamegraphs. Just add the following to Tarpaulin’s Cargo.toml so I can get function names instead of addresses and install the local version: [profile.release] debug = true</code></pre> Next up I want to run tarpaulin with perf</code> to get a perf.data</code> file. Perf is a sampling profiler so no instrumentation or changes to your binary - except for the debug symbols to make life easier. I also use inferno</a> by Jon Gjengset to generate the flamegraphs: perf record --call-graph dwarf -- cargo tarpaulin --engine llvm --skip-clean perf script | inferno-collapse-perf | inferno-flamegraph > perf_flamegraph.svg</code></pre> If you want to understand more about perf and getting useful calls for it you probably want to look at anything Brendan Gregg has written link</a>. Also Denis Bakhvalov’s book is great link</a>. Unfortunately, I lost the flamegraph I originally did. But it wasn’t of polars but a smaller project. And serializing the perf.data files afterwards is so slow I didn’t fancy a 1 hour plus wait to create a flamegraph of before the stupid issue was fixed so you’ll have to live with the after flamegraph. After: You should be able to open this SVG in your browser, and click on blocks to zoom in on that execution stack. You can use that to dig in and see what some of the really small thin towers are. Before the trace was massively taken up by calls to Vec<T>::remove</code>, which is why I tackled that first. Why Not Cargo-Flamegraph?</a></h2> I also tried using cargo-flamegraph</a>. However, the flamegraph was mostly dominated by cargo and tarpaulin was only a tiny sliver so something was wrong there. But I’ve used cargo-flamegraph</code> before successfully so if you want to see the command it was: flamegraph -- cargo tarpaulin --engine llvm --skip-clean</code></pre> I’m not sure why the cargo dominance, but I did during this have some faff in trying to stop it rebuilding the tests on the profiling runs. Skipping the build step was mainly so tarpaulin calling out to cargo didn’t dominate the perf data and to keep the size down. Large perf data files take an age to collapse and render so for my own sanity I wanted to avoid that. Still Slow?</a></h1> Well I cut a release and updated the issue and: After updating to Tarpaulin 0.32.4, I see a performance boost of around ~25%, which is already a good improvement. I tried creating a fresh project and adding heavy dependencies to it (like polars-rs), but the performance is still quite fast. So, I don’t know how to provide a minimal example where this step takes too long to execute. </blockquote> And looking at that new flamegraph we’ve got a lot of time spent in find. It’s the new dominant time. This feels like something I can fix. Going back to the code, I can see instantly where the find is: let record = self.profile.records().iter().find(|x| { x.hash == Some(func.header.fn_hash) && Some(func.header.name_hash) == x.name_hash });</code></pre> That’s not great, and I don’t think we can search the records to do things like binary searching for the hashes. But there’s something easier to do… Memoization</a></h2> For previous performance work, some memoization was added for the profraw parsing, an FxHashMap</code> going from the strings to the index in the records array. There’s already a symbol table going from hashes to names so I can look up the name then get the index from the name and get the exact record. This turns one find on a vec to two hashmap lookups and then the array access. So adding this method to my InstrumentationProfile</code> type: pub fn find_record_by_hash(&self, hash: u64) -> Option<&NamedInstrProfRecord> { let name = self.symtab.get(hash)?; self.find_record_by_name(name) }</code></pre> I then change the find call to: let record = self.profile.find_record_by_hash(func.header.name_hash);</code></pre> And now the times for the outputs: real 2m24.562s user 1m25.707s sys 1m3.097s</code></pre> And the flamegraph: The fact you see other stages of the processing pipeline does show it’s more balanced. But given how similar the times are, the first flamegraph might just be some sort of sampling weirdness… Some Thoughts</a></h1> Now, looking at these results I don’t see much change. Which is disappointing, although the flamegraph is a lot different. Running the Criterion benchmarks for my profparser crate I do see an 80% speedup on top of the previous speedup. This indicates that either there’s something a bit screwy with the benchmark for the data is respresentative of a different domain than the polars code. The find scans across all the function records, and remove handles the coverage region expressions. There is some interplay between the amount of functions in your project and the complexity of the coverage regions of those functions. And where do most of the complex functions live? Near the start of the records list or the end? All these things could dramatically change how much impact these different speed-ups give different projects. The user who raised the issue has also confirmed that this lead to a massive speedup for them so mission accomplished! The fact that this change lead to very little change in polars results but a big change in the users project does show that judging tradeoffs in this code can potentially be very tricky in future. More Speed-up?</a></h1> The LLVM profiling data does require (at least for polars), parsing around 1GB of data files. Then after parsing the extra ELF sections added expression evaluation and a lot of merging/resolution work. Looking at the profile I do see a lot of time spent reading files. From the sampling 51% of the tarpaulin running time is handling the instrumentation runs - which will be parsing, merging and extracting relevant data from the profraws and binaries. But ~30% is just reading files. Now this can be thrown off by inaccuracies in sampling, and there is potentially room for some multi-threading to be applied smartly to speed up parsing of a mass of files. But it seems the low-hanging fruit may have all been plucked. While, I can try things, seeing around 96% speed-ups on one project but only 25% on another with the same code being the bottleneck it feels like a more analytic approach is in order. The next steps will be to gather better insights into the shape of the data for different projects. So things like the number of function records, the number of counters in those. The number of expressions and how many steps to resolve them. Start to analyse how these vary and how these variations impact performance. Part of this will also be creating a corpus of test projects where I can stress the code in different ways. But that’s a topic for a future post. Announcing Wiremocket: Wiremock for Websockets 2025-03-03T00:00:00+00:00 This is an announcement blogpost for wiremocket</a> a wiremock</a> influenced library for mocking websocket servers. The Motivation</a></h1> At Dayjob™ we have a number of websocket APIs, this is because they offer bidirectional streaming where the client can send a stream of data and the server at the same time can stream back data. We do this because we work with audio APIs and real time audio/visual systems. When testing these systems it’s too heavy to run multiple docker images with AI models in CI meaning for an application which interacts with multiple websocket connections we have a few options for testing: Split out functionality to allow for testing the meat of it without the client connection</li> Replace the websocket client type with some trait abstraction and in tests inject an in-memory alternative</li> Implement a bespoke websocket server just for this projects tests</li> Don’t test it</li> </ol> Well today I’ve published a 5th option, use a mocking library to help generate a server quickly and ensure requests match what are expected. Why Not PR Wiremock?</a></h1> There’s an issue</a> for this in wiremock opened by a colleague. I’m not saying this won’t go into wiremock eventually in some form, but it is a big change and the two domains dissimilar enough the overlap could confuse existing users. What’s So Difficult?</a></h1> Streaming. Streaming is always difficult. Given a server we may want to match on information available when the connection is established such as: HTTP headers</li> The URL path</li> The URL query parameters</li> </ol> But we may also have things like different input formats available and want to match on the messages coming over the connection. To explain these difficulties perhaps an example will be clearer. A Big Example</a></h1> In addition, with the responses we want the ability for the input messages to impact the responses. As an example of the capabilities let’s look at the process for testing some code using websockets where we have a few requirements: When the client requests to /api/binary_stream</code></li> The first message is valid json</li> The last message before closing is valid json</li> All messages between these are binary data</li> A close frame is sent</li> </ol> We can set this up by implementing our own temporal matcher and using some already provided matchers. Try not to focus too much on the implemented matcher as this is where most of the complexity will lie and the docs explain a lot about how this works. impl Match for BinaryStreamMatcher { fn temporal_match(&self, match_state: &mut MatchState) -> Option<bool> { let json = ValidJsonMatcher; let len = match_state.len(); let last = match_state.last(); if len == 1 && json.unary_match(last).unwrap() { match_state.keep_message(0); Some(true) } else if last.is_binary() { // We won't keep any binary messages! if len > 1 && match_state.get_message(len - 2).is_none() { Some(true) } else { Some(false) } } else if last.is_close() { if len == 1 { None } else { let message = match_state.get_message(len - 2); if let Some(message) = message { json.unary_match(message) } else { Some(false) } } } else if last.is_text() { let res = json.unary_match(last); match_state.keep_message(len - 1); res } else { None } } } #[tokio::test] async fn binary_stream_matcher_passes() { let server = MockServer::start().await; // So our pretend API here will send off a json and then after that every packet will be binary // and then the last one a json followed by a close server .register( Mock::given(path("api/binary_stream")) .add_matcher(BinaryStreamMatcher) .add_matcher(CloseFrameReceivedMatcher) .expect(1); ) .await; println!("connecting to: {}", server.uri()); let (mut stream, response) = connect_async(format!("{}/api/binary_stream", server.uri())) .await .unwrap(); let data: Vec<u8> = vec![0, 1, 2, 3, 4, 5, 6, 7, 8, 1, 2, 3, 4, 5, 6, 7]; let val = json!({"command": "start"}); stream.send(Message::text(val.to_string())).await.unwrap(); stream.send(Message::Binary(data.into())).await.unwrap(); let val = json!({"command": "stop"}); stream.send(Message::text(val.to_string())).await.unwrap(); stream.send(Message::Close(None)).await.unwrap(); std::mem::drop(stream); // Asserts the match conditions were met. server.verify().await; }</code></pre> Ignoring the details of the matcher implementation, the code with-in the test should look familiar to any wiremock users. We add matchers to a mock, an expected number of calls and send off requests and assert on them. But we can also see the extra complexity that matching on streams and looking at sequence based behaviour causes. What’s Next?</a></h1> I’ve been so deep in the sauce I haven’t yet used this in anger. So that will be my next step. However, the code I’ll be using it on is all closed source meaning whether it works well or not is going to be a “trust me bro” until people use it. There’s also a few things I haven’t implemented that are in wiremock. Things like: Storing all the requests received by the server during a test</li> Connection pooling</li> Fancier reports</li> Some ergonomic improvements (my mock construction should really use a builder)</li> More matchers</li> More out-of-the-box responders</li> Stare intensely at grpc and wonder if I want to make a mockery of myself</li> </ul> Streaming Audio APIs: the Axum Server 2025-01-20T00:00:00+00:00 All the code can be found on this branch</a> if you look at main you’ll see future post content there! For the API in this project we’re going to be using Axum</a>, this is largely because of my own familiarity with Axum. Although, there is a small benefit where Axum and Tonic</a> use the same request router so multiplexing between REST and gRPC becomes easier in the future if we want to do it. One important tip when you’re looking at Axum, unlike actix-web and other frameworks there isn’t a website with a tutorial and examples so you have to rely on the docs and Github. When you go on Github view the tag for the version you’re using as the main branch potentially has a number of breaking changes that stop the examples from working with the last released version. At time of writing the latest released Axum is 0.8.1 so this will be added to the Cargo.toml</code> in the dependencies section: axum = { version = "0.8.1", features = ["tracing", "ws"] }</code></pre> With that preamble out of the way let’s get started with defining our function to launch the server. Initially, we’ll just start with a simple health-check, but in future we might want to change the health status if the service needs restarting. We’ll also add a shutdown signal for graceful shutdown so if we get a SIGTERM, the server will stop receiving connections and close after the last request is finished. I’ll also pass in our model’s StreamingContext</code> because we know we’ll need that in future! use tokio::signal; use axum::{extract::Extension, response::Json, routing::get, Router}; async fn health_check() -> Json<Value> { Json(serde_json::json!({"status": "healthy"})) } fn make_service_router(state: Arc<StreamingContext>) -> Router { Router::new() .route("/api/v1/health", get(health_check)) .layer(Extension(app_state)) } pub async fn run_axum_server(app_state: Arc<StreamingContext>) -> anyhow::Result<()> { let app = make_service_router(app_state); // run it with hyper let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?; info!("listening on {}", listener.local_addr().unwrap()); axum::serve(listener, app) .with_graceful_shutdown(shutdown_signal()) .await?; Ok(()) } async fn shutdown_signal() { let ctrl_c = async { signal::ctrl_c() .await .expect("failed to install Ctrl+C handler"); }; #[cfg(unix)] let terminate = async { signal::unix::signal(signal::unix::SignalKind::terminate()) .expect("failed to install signal handler") .recv() .await; }; #[cfg(not(unix))] let terminate = std::future::pending::<()>(); tokio::select! { _ = ctrl_c => {}, _ = terminate => {}, } }</code></pre> Make service router is a separate function for one important reason - testability. This means our tests can easily create a router and test routes etc without binding to a port. The health check API and all of our APIs designed to be used by users will be in a versioned API hence the /api/v1</code> prefix to the health check. Versioned APIs are great because they allow us to make breaking changes to our API and just increase the version. We can then potentially keep a backwards compatible endpoint and give users time to migrate any code relying on our service before we remove legacy APIs. Now we add a launch_server</code> function in the lib.rs</code> which the main function will call to launch everything. It will load a config file with the model settings, create the context and then launch the axum server. pub async fn launch_server() { let config = fs::read("config.json") .await .expect("Couldn't read server config"); let config = serde_json::from_slice(&config).expect("Couldn't deserialize config"); info!(config=?config, "service config loaded"); let ctx = Arc::new(StreamingContext::new_with_config(config)); info!("Launching server"); axum_server::run_axum_server(ctx) .await .expect("Failed to launch server"); info!("Server exiting"); }</code></pre> If we run our server we can now call the health check and see it all working: $ curl localhost:8080/api/v1/health {"status":"healthy"}</code></pre> This is all pretty simple stuff so far and you can find it in the Axum getting started documentation. The next part is where we start to up the complexity because it’s time to add in the websocket handler. Websockets</a></h2> Websockets are a streaming protocol built on top of TCP that allows for bidirectional streaming. A websocket resembles a raw TCP socket more than HTTP as it allows streaming between client and server and before HTTP/2 it was the only way to do streaming where the client streams data into the server. If it’s only the server streaming SSE (Server-Sent Events), can be used with HTTP/1.1. Because of this we need to upgrade the HTTP connection to a websocket connection: use axum::{ extract::{ ws::{Message, WebSocket, WebSocketUpgrade}, Extension, }, response::IntoResponse, }; async fn ws_handler( ws: WebSocketUpgrade, vad_processing: bool, Extension(state): Extension<Arc<StreamingContext>>, ) -> impl IntoResponse { ws.on_upgrade(move |socket| { handle_socket(socket, vad_processing, state, metrics) }) } async fn handle_socket( socket: WebSocket, vad_processing: bool, state: Arc<StreamingContext>, ) { todo!() }</code></pre> Adding this to the router we get: fn make_service_router(state: Arc<StreamingContext>) -> Router { Router::new() .route( "/api/v1/simple", get({ move |ws, app_state| { ws_handler(ws, false, app_state, metrics_enc) } }), ) .route( "/api/v1/segmented", get({ move |ws, app_state| { ws_handler(ws, true, app_state, metrics_enc) } }), ) .route("/api/v1/health", get(health_check)) .layer(Extension(app_state)) }</code></pre> I’m reusing the sample function for both the simple chunked API and the VAD segmented API, this is mainly because the only difference should be the method called on the context but everything else should be reused. In future we might want to separate it for better tracking of metrics per-endpoint, but as a first implementation we always want to strive for simplicity. Well now, take a breath because now we’re finally set to finally write the last parts of the API defined in part 2. With this addition there will be a working API that can be called and audio streams in and responses and events out. Before we start coding lets refamiliarise ourselves with the steps we want to follow: Receive a start message from the client</li> Use this to set up the audio decoding and inference tasks</li> Receive audio and forward it into the decoding</li> Connect audio decoding output to inference tasks (one per channel)</li> Forward inference output to client</li> Handle stop requests and either keep connection or disconnect</li> Wait for another start or data to resume processing</li> </ol> Well waiting for a start message might be a moderately sized block of code that’s repeated. Splitting this into it’s own function used by handle_socket</code> already makes sense. The Websocket</code> type implements Stream</code> and if we split it into a sending and receiving part the receiving part also implements Stream</code>. Because of this I’ll make the function generic on Stream</code> as it can make it easier to test in future and potentially avoids changing the type signature again. Time to implement it as follows: use futures::stream::Stream; async fn handle_initial_start<S, E>(receiver: &mut S) -> Option<StartMessage> where S: Stream<Item = Result<Message, E>> + Unpin, E: Error, { let mut start = None; while let Some(Ok(msg)) = receiver.next().await { if let Ok(text) = msg.into_text() { match serde_json::from_str::<RequestMessage>(&text) { Ok(RequestMessage::Start(start_msg)) => { info!(start=?start, "Initialising streamer"); start = Some(start_msg); break; } Ok(RequestMessage::Stop(_)) => { warn!("Unexpected stop received as first message"); } Err(e) => { error!(json=%text, error=%e, "invalid json"); } } } } start }</code></pre> I’ve popped some tracing</code></a> logs here just to make the process we go through when handshaking clear. This function will ignore any invalid messages and just receive messages from the websocket until it gets a start message or the client disconnects. For invalid start messages we may in future want to return an error to the user but we’ll assume that’s unnecessary for now. The next steps will be to setup the receiver task to forward message to the client from the inference task, and then wait for the first message. I’ll make a small function for encoding the messages just to make the chained calls look tidier as well. fn create_websocket_message(event: ApiResponse) -> Result<Message, axum::Error> { let string = serde_json::to_string(&event).unwrap(); Ok(Message::Text(string)) } async fn handle_socket( socket: WebSocket, vad_processing: bool, state: Arc<StreamingContext>, ) { let (sender, mut receiver) = socket.split(); let (client_sender, client_receiver) = mpsc::channel(8); let client_receiver = ReceiverStream::new(client_receiver); let recv_task = client_receiver .map(create_websocket_message) .forward(sender) .map(|result| { if let Err(e) = result { error!("error sending websocket msg: {}", e); } }); let _ = task::spawn(recv_task); let mut start = match handle_initial_start(&mut receiver).await { Some(start) => start, None => { info!("Exiting with processing any messages, no data received"); return; } }; 'outer: loop { todo!("need to start processing things!"); } }</code></pre> Here we use StreamExt::split</code></a> to split the websocket into a sending and receiving part. This allows us to move sending and receiving into different tasks and do them concurrently instead of having to share one mutable Websocket</code> instance. The eagle-eyed among us may have realised I’ve preemptively put a label on the loop. This is because at the start of the loop we’ll be setting up the audio decoding tasks and inference tasks. Then we will loop through the stream of messages until we get a stop. After we get the stop a new start or more audio data will resume processing but we want the stop to force an end of processing of the existing audio. For that inference and decoding the audio is finished, so if they batch up in anyway they have to know the last audio is potentially a partial batch. Setting up the inference tasks: // Don't forget me from forwarding from inference to client let (client_sender, client_receiver) = mpsc::channel(8); // snip 'outer: loop { info!("Setting up inference loop"); let (audio_bytes_tx, audio_bytes_rx) = mpsc::channel(8); let mut running_inferences = vec![]; let mut senders = vec![]; for channel_id in 0..start.format.channels { let client_sender_clone = client_sender.clone(); let (samples_tx, samples_rx) = mpsc::channel(8); let context = state.clone(); let start_cloned = start.clone(); let inference_task = async move { if vad_processing { context .segmented_runner( start_cloned, channel_id, samples_rx, client_sender_clone, ) .await } else { context .inference_runner(channel_id, samples_rx, client_sender_clone) .await } }; let handle = task::spawn(inference_task); running_inferences.push(handle); senders.push(samples_tx); } // Transcoding and message processing todo!("The rest of the owl"); // Clean up inference task for handle in running_inferences.drain(..) { match handle.await { Ok(Err(e)) => error!("Inference failed: {}", e), Err(e) => error!("Inference task panicked: {}", e), Ok(Ok(_)) => {} } } }</code></pre> The transcoding task is also fairly simple, we just want to spawn the audio decoder with our channels passed in. let transcoding_task = task::spawn( decode_audio(start.format, audio_bytes_rx, senders).in_current_span() );</code></pre> In our current function the last stage is to read the messages from the websocket and send down the appropriate channel. There’s also a small check after we join the running inferences so I’ll include a bit of the code from a previous sample again: 'outer: loop { // Code from setting up the inference and transcoding tasks let mut got_messages = false; let mut disconnect = false; while let Some(Ok(msg)) = receiver.next().await { match msg { Message::Binary(audio) => { got_messages = true; if let Err(e) = audio_bytes_tx.send(audio.into()).await { warn!("Transcoding channel closed, this may indicate that inference has finished: {}", e); break; } } Message::Text(text) => match serde_json::from_str::<RequestMessage>(&text) { Ok(RequestMessage::Start(start_msg)) => { got_messages = true; info!(start=?start, "Reinitialising streamer"); start = start_msg; break; } Ok(RequestMessage::Stop(msg)) => { got_messages = true; info!("Stopping current stream, {:?}", msg); disconnect = msg.disconnect; break; } Err(e) => { error!(json=%text, error=%e, "invalid json"); } }, Message::Close(_frame) => { info!("Finished streaming request"); break 'outer; } _ => {} // We don't care about ping and pong } } std::mem::drop(audio_bytes_tx); for handle in running_inferences.drain(..) { match handle.await { Ok(Err(e)) => error!("Inference failed: {}", e), Err(e) => error!("Inference task panicked: {}", e), Ok(Ok(_)) => {} } } if let Err(e) = transcoding_task.await.unwrap() { error!("Failed from transcoding task: {}", e); } if !got_messages || disconnect { break; } }</code></pre> When we receive binary data this is audio so we send into our transcoding task. If that sending has failed it means one of two things: Transcoding failed with an error</li> We’ve had a single utterance request and the utterances have finished</li> </ol> In this case we break out of the message handling and when we await the other tasks we’ll see if anything went wrong. For the text messages these should either be start or stop, these come when our current stream ends and a new one is starting or the connection will be closed. If we exited the websocket message receiving without receiving anything then the client will have disconnected before we did anything and we want to exit, same with a disconnection request hence the check with a final break at the end. Alongside this I’ve also written a small client to test this. It’s fairly simple using hound</a> to read a WAV file and then streaming it either as fast as possible or in real-time. You can also set the configuration via the CLI. It shouldn’t be too hard to recreate with the tokio-tungstenite</a> docs so I’ve skipped over it currently. But here’s the help text Usage: client [OPTIONS] --input <INPUT> Options: -i, --input <INPUT> Input audio file to stream --chunk-size <CHUNK_SIZE> Size of audio chunks to send to the server [default: 256] -a, --addr <ADDR> Address of the streaming server (/api/v1/segmented or /api/v1/simple for vad or non-vad options) [default: ws://localhost:8080/api/v1/segmented] --real-time Attempts to simulate real time streaming by adding a pause between sending proportional to sample rate --interim-results Return interim results before an endpoint is detected -h, --help Print help</code></pre> If you try with a voiced file you’ll see output for both endpoints. The bare minimum command given an audio called input.wav</code> and the server running locally is: client -i input.wav -a ws://localhost:8080/api/v1/simple client -i input.wav -a ws://localhost:8080/api/v1/segmented</code></pre> One thing to note, tungstenite has issues with proxies as per this issue</a>. If you need proxy support for a client with your own projects you might want to look at some of the solutions in the linked issue. With this we’ve tied together all the building blocks in the previous parts. We now have a working API and can stream audio into a model and get results back. Adding a New Fake to the Fake Crate 2025-01-15T00:00:00+00:00 This post is going to go through my process of PRing something to the fake crate. I’d done it before in a different part of the crate so assumed this would be easy but it took me a bit more time than expected so figured that’s a good reason to write. For anyone who hasn’t seen it before fake</a> is a crate for generating fake values for the purposes of testing. It’s mentioned in zero2prod</a> for generating fake sign-ups to the service and this is where I first heard of it. But, when generating fake API values for testing you sometimes get many more types that need faking than are currently implemented. When this happens there are three choices: Implement it in your crate</li> PR to the fake crate</li> Give up</li> </ol> Option 1 means if we need to fake this type in a different project we have to copy our implementation around which is a bit of a drag. And option 3 is never an option here (or at least for things that should be this easy). This just leaves us with option 2. So what are we faking? This time it is base64 strings. I’m dealing with an API which makes the decision to encode some binary data as base64 in a JSON. Where to put our code</a></h1> Typically, we also don’t want to implement things in fake if there’s crates that do them (or types in crates we want to fake). I’m going to use the base64</a>. Starting off, going into the fake</code> folder in the root of the repo to add to the crate: cd fake cargo add base64 --optional</code></pre> Now lets look at the folder structure in src to see where we might put our code: ├── src │ ├── bin │ ├── faker │ ├── impls │ │ ├── base64 │ │ ├── bigdecimal │ │ ├── bson_oid │ │ ├── chrono │ │ ├── chrono_tz │ │ ├── color │ │ ├── decimal │ │ ├── geo │ │ ├── glam │ │ ├── http │ │ ├── indexmap │ │ ├── semver │ │ ├── serde_json │ │ ├── std │ │ ├── time │ │ ├── ulid │ │ ├── url │ │ ├── uuid │ │ └── zerocopy_byteorder │ └─── locales</code></pre> Looking at where to add our code it can be hard to figure out between the faker</code> and impl</code> modules. But in impls</code> we see there’s folders for a bunch of 3rd party crates. Making that the perfect place to start. Adding a base64</code> module into impls</code> I can start to implement the fake stuff. Implementing it</a></h1> The core of the fake implementations is the Dummy</a> trait. For those who don’t want to leave this post I’ll share the trait definition (minus a method with a default impl). pub trait Dummy<T>: Sized { // Required method fn dummy_with_rng<R: Rng + ?Sized>(config: &T, rng: &mut R) -> Self; }</code></pre> The config type T</code> is often used in a type-state manner. Looking at the uuid</code> fake impl we can see types for UUIDv8</code> etc which are provided like impl Dummy<UUIDv8> for Uuid</code>. There’s not that many variants in base64, looking at the engine the main things we can vary are: The data being encoded as base64</li> Whether we pad the data or not.</li> </ol> To vary these we’ll use the Faker</code> type to generate random data and a random boolean and then encode the base64 using these. My initial errors all boiled down to not using the cheaply constructed Faker</code> type. But once I realised that it felt pretty simple. In the new fake/src/impls/base64/mod.rs</code> the initial implementation looks something like: use crate::{Dummy, Fake, Faker}; use base64::prelude::*; pub struct Base64; impl Dummy<Base64> for String { fn dummy_with_rng<R: rand::Rng + ?Sized>(_: &Base64, rng: &mut R) -> Self { let data: Vec<u8> = Faker.fake_with_rng(rng); let padding = Faker.fake_with_rng(rng); let encoded = if padding { BASE64_STANDARD.encode(&data) } else { BASE64_STANDARD_NO_PAD.encode(&data) }; encoded } }</code></pre> Here we can use this as follows: let fake_base64: String = String::dummy_with_rng(&Base64, rng);</code></pre> Which isn’t super ergonomic but it serves a purpose. We ideally want to implement for Dummy<Faker></code> because then we can use the cheaply constructed Faker</code> type to generate base64 values. If we do this our code can look like: let fake_base64: String = Base64.fake();</code></pre> Right, so we need to implement Dummy<Faker></code> for something, and we can’t implement it for a String</code> directly because fake</code> can already generate fake strings. This means we need some intermediate wrapper type and then all our generics should just flow together. With this in mind I added this to the file after some fiddling: pub struct Base64Value(pub String); impl Dummy<Faker> for Base64Value { fn dummy_with_rng<R: rand::Rng + ?Sized>(config: &Faker, rng: &mut R) -> Self { let s = String::dummy_with_rng(&Base64, rng); Base64Value(s) } }</code></pre> A bunch of the fiddling was whether the previous code should live in this impl and the implementation on String</code> use it or vice versa. I struggled a bit to get that working with the trait bounds and ultimately it doesn’t matter which way round and this way is the path of least resistance. With these two in place we can look at adding a test. The majority of the fake tests are determinism tests ensuring that with the same seed we get the same faked values. There’s a number of macros to define these but if you look at the file you can just copy the pattern. Our test then looks like so: #[cfg(feature = "base64")] mod base64 { use fake::base64::*; use fake::{Fake, Faker}; use rand::SeedableRng as _; check_determinism! { one fake_base64, Base64Value, Faker } check_determinism! { one fake_url_safe_base64, UrlSafeBase64Value, Faker } }</code></pre> Add the public re-exports and mod statements matching the patterns as well and we have a working implementation! The only other thing is to look at the base64</code> crate and see if there’s any other configurations we should look at supporting. I saw there were engines to generate URL safe base64 so implemented them much the same and then called it a day! This is the implementation I opened the PR</a> with which was ultimately merged: use crate::{Dummy, Fake, Faker}; use base64::prelude::*; #[derive(Clone, Debug, PartialEq, Eq, Ord, PartialOrd, Hash)] pub struct UrlSafeBase64Value(pub String); #[derive(Clone, Debug, PartialEq, Eq, Ord, PartialOrd, Hash)] pub struct Base64Value(pub String); pub struct Base64; pub struct UrlSafeBase64; impl Dummy<Base64> for String { fn dummy_with_rng<R: rand::Rng + ?Sized>(_: &Base64, rng: &mut R) -> Self { let data: Vec<u8> = Faker.fake_with_rng(rng); let padding = Faker.fake_with_rng(rng); let encoded = if padding { BASE64_STANDARD.encode(&data) } else { BASE64_STANDARD_NO_PAD.encode(&data) }; encoded } } impl Dummy<UrlSafeBase64> for String { fn dummy_with_rng<R: rand::Rng + ?Sized>(_: &UrlSafeBase64, rng: &mut R) -> Self { let data: Vec<u8> = Faker.fake_with_rng(rng); let padding = Faker.fake_with_rng(rng); let encoded = if padding { BASE64_URL_SAFE.encode(&data) } else { BASE64_URL_SAFE_NO_PAD.encode(&data) }; encoded } } impl Dummy<Faker> for Base64Value { fn dummy_with_rng<R: rand::Rng + ?Sized>(config: &Faker, rng: &mut R) -> Self { let s = String::dummy_with_rng(&Base64, rng); Base64Value(s) } } impl Dummy<Faker> for UrlSafeBase64Value { fn dummy_with_rng<R: rand::Rng + ?Sized>(config: &Faker, rng: &mut R) -> Self { let s = String::dummy_with_rng(&UrlSafeBase64, rng); UrlSafeBase64Value(s) } }</code></pre> And here is a small test program showing it being used including with the derive</code> feature: use fake::{Dummy, Fake, Faker}; use fake::base64::Base64; #[derive(Debug, Dummy)] pub struct FakeMe { #[dummy(faker = "Base64")] base64: String, } fn main() { let f: FakeMe = Faker.fake(); println!("{:?}", f); let fake_string: String = Base64.fake(); println!("{}", fake_string); let fake_string: String = String::dummy(&Base64); println!("{}", fake_string); }</code></pre> 2024 in Review 2025-01-06T00:00:00+00:00 I feel like I’ve managed to do quite a lot this year and also have a lot planned. With that in mind why not reflect a bit on what was planned, done and what I’m going to be doing. When relevant links aren’t mentioned within the text body I’ll pop them at the bottom but I might be inconsistent in this. Sorry (not sorry). rustnation</a></h1> The start of the year was pretty much dominated by my preparation for Rustnation at the end of March. I’d put forwards an ambitious brief for the talk given the time I had. My talk Creating a Text-To-Speech System in Rust involved a mix of open sourced Emotech code and some code written for the talk. As well as a fully working repo there was a repo for SSML parsing and a signal-processing based vocoder with small blog/tutorial provided. And a lot of documentation comments about machine learning in general and things I tried that did and didn’t work. Below is an example doc comment on a neural network model I initially tried to use and abandoned. I kept the source files in the repo and documented them all for anyone really interested. An example of documenting issues with a failed approach: After getting our ONNX file however it uses the loop operator and has dynamically sized inputs within the model. These are two things that proved fatal to running it in a Rust runtime. After finding out about ORT I tried it there, unfortunately pytorch can output ONNX that doesn’t obey the standard (-1 dimension size in expand nodes). There’s an old version of torch used in speedyspeech, one with worse ONNX support. Any changes to the pretrained graph would need to be done via this old version of torch and torch doesn’t come with a clear upgrade path. To run speedyspeech I would have had to port the code to a much newer torch version and retrain a model from scratch. </blockquote> After all is said and done I think I did a good job and I’ve gotten a lot of positive feedback from people during the conference and even months afterwards. I’ve also seen some of my feedback about DX issues in some ML libraries result in an issue on a relatively new project rten to make sure the next generation of libraries are easier to use (the issue</a>). This to me is a sign that struggling through 4 different ML inference frameworks to find I couldn’t get an open source model working was worth it in the end. There’s no plans currently to continue this project but I might be tempted to do some future conference work on more statistical less neural-network based models - but no promises! The repo</a></li> The talk</a></li> </ul> tarpaulin</a></h1> Tarpaulin</a> saw 11 releases this year. I have to admit I’ve had periods of burnout working on it so it’s been in a “soft” maintenance mode. I’ve still been fixing issues, updating docs and helping users who post things but I’ve not maintained any meaningful velocity on it and there’s a number of hard issues still to be tackled. There’s been some improvements with handling toolchains in non-standard places, windows and mac support improved. Logs and error messages also improved to help people debug coverage issues more meaningfully. One thing I also did was redo the item naming for things like lcov, now they spit out names in the same (albeit assumed unstable) form that cargo-mutants</a> uses. With this change it’s possible with either some manual copy and pasting or script glue to use coverage to exclude untested code from cargo-mutants runs saving testing time. You can see my comment about this in this issue</a>. Aside from this I’ve mentored one person through fixing an issue they had in tarpaulin. This is unusual but something I’m happy with, a lot of people’s issues are more in the harder problems and the difficulty cliff is intimidating so I don’t often get to mentor anyone like this successfully. It’s always nice when it works out. You can see the PR here</a>. I did submit a tarpaulin talk to Eurorust but it wasn’t accepted, c’est la vie! After Rustnation I did go for a much less ambitious proposal though so at least I wasn’t too invested! Next year I have some plans to fix some of the lingering ptrace issues that make tests flaky. Unfortunately, the LLVM coverage is limited in what types of things it can collect coverage for and while it’s generally more accurate if you fall into those niches you’re not going to have results you’re happy with. Additionally things like #[tracing::instrument]</code> attributes cause functions to be removed from the results. Given these issues it may be that ptrace and output formats are the main selling point of tarpaulin against other things like cargo-llvm-cov</a>, gcov or rawdogging rustc llvm instrumentation. Two years ago I had started to look at using minicov</a> for embedded on-device coverage. I may go back to those experiments, or I think I see a way to use minicov to fix some issues in the llvm profiling runtime when it comes to things like tests with forks/signals/required-panics when a custom test harness is used. I doubt this will be massively relevant to most people though so it may take a backseat - I may just blog about the ideas but I want to test some assumptions first. dechib</a></h1> I started work on a RDBMS using rocksdb called dechib</a>. Unfortunately, I haven’t had much time to work on it so I just have table creation, and insertion plus some constraints implemented. Hopefully, next year I’ll have time to implement actual queries - part way through the design for querying from inserted data. Not much to write about here but hopefully there will be in future! This is just an educational project working on the inner systems that interest me most. Never use it in production if I get it working. object-trustfall-adapter</a></h1> Inspired by Predrag</a> I wanted to make a trustfall adapter around ELF files to query ELF files like a database for reverse engineering purposes. Nothing really to share yet, but this is me committing to building in public. The repo</a></li> </ul> bidirectional audio streaming blog series</a></h1> In my work we do some audio streaming services for things like live transcription or streaming TTS. However, streaming systems are complicated and it can often be a source of confusion for newcomers to the organisation. Inspired by an actual problem I started a blog series about this which I’ve been posting on the socials™. Four posts are done, one is in late stage draft and another one is in review and another is in early stage draft. Next year I aim to finish all the planned chapter so publish the 3 in-progress posts and add ones on: observability, testing and the gRPC API. I’m excited for when the metrics post is ready to be published (I am trying to keep to a natural order). I’ve put quite a bit of work into it and it covers a lot that I’ve rarely seen written about. Longer term (beyond the year), I want to do some tooling around AsyncAPI</a> in Rust that matches some of what we have for OpenAPI/Swagger. Also, a websocket mocking library which I’m starting to draft. The repo</a></li> websocket mocking library (not properly started)</a></li> </ul> rustybug</a></h1> I bought an early access copy of Building a Debugger - Sy Brand</a> and have been working through it during my christmas holidays as a means of relaxation! I’m doing it in Rust so it will hopefully feedback back into fixing some of tarpaulin’s ptrace code. I’m also going to use this to find any opportunities to add PRs to libc</a> or nix</a> if I spot any APIs that need some love. The first of these is already open: libc PR</a>. Expect some blog posts about this when I have something that works good enough! The repo</a></li> </ul> outside of technology</a></h1> Life isn’t just Rust though (or so I hear). Outside of Rust I’ve made decent progress in learning Japanese (for my partners family not weebery… okay maybe also some weebery). I’ll be spending a number of months in Japan as a result and hope to made even greater strides. Also, being a stereotypical software dev I have some fitness goals related to climbing, I got into board climbing last year and will continue with it. I won’t bore you with those details! I’ve ready a number of Ursula K. Le Guin stories (The Left Hand of Darkness and collected short stories). And the first two Hyperion books by Dan Simmons. I’ll be finishing Hyperion next year, and hopefully also the Wizard of Eathsea series. I’ll also be reading How to Blowup a Pipeline by Andreas Malm, and Abroad in Japan by Chris Broad. This year I’ve kind of read less but the Ursual K. Le Guin short stories book was a chonker. I’ve watched way too much to enumerate and I’ll probably continue to do that. The next movie I’ll likely watch is Decision to Leave</a> and I’m very excited to see the new Nosferatu</a>. For gaming I played a lot last year. This year I plan on playing 1000xRESIST</a>, Chasing Static</a> and Hollowbody</a>. I’ll also finish Half-Life which I’m playing all the way through for the first time (I know I know better late than never). The last year my favourite game I played was SOMA</a>. I might have to play all the amnesia games if they have vibes that impeccable! Well that’s it for my year in review and what’s coming up. Some of this will be blogged some of it you may never hear about again. Laters 👋 Streaming Audio APIs: the Model 2024-12-03T00:00:00+00:00 This entry will lean a lot on types from streaming API design</a> feel free to refer back if any of the types are unexpected. As previously mentioned, this series won’t integrate a real model. There’s plenty of them around and we care more about the API and engine side than the model side. However, we still have a dummy model to help us replicate some behaviour, and we need to discuss it before moving on. In reality, these models are normally the slowest part of the system and a bottleneck we have to get around. So with that in mind, I’ll do a blocking thread sleep. It’s an unrealistic workload as it doesn’t saturate any CPU or GPU resources. But it’s enough to demonstrate a point. Here’s all the code for the model minus creating it: use serde::{Deserialize, Serialize}; use std::sync::atomic::{AtomicBool, Ordering}; use std::thread::sleep; use std::time::Duration; use tracing::{info, instrument}; pub const MODEL_SAMPLE_RATE: usize = 16000; static FIRST_RUN: AtomicBool = AtomicBool::new(true); /// A fake stub model. This will be a model of only hyperparameters #[derive(Clone, Debug, Deserialize)] #[serde(default)] pub struct Model { delay: f32, constant_factor: f32, jitter: f32, failure_rate: f32, panic_rate: f32, warmup_penalty: f32, } #[derive(Debug, Deserialize, Serialize)] pub struct Output { pub count: usize, } impl Model { #[instrument(skip_all)] pub fn infer(&self, data: &[f32]) -> anyhow::Result<Output> { let duration = Duration::from_secs_f32(data.len() as f32 / MODEL_SAMPLE_RATE as f32); let jitter = self.jitter * (fastrand::f32() * 2.0 - 1.0); let mut delay = duration.as_secs_f32() * self.constant_factor + self.delay + jitter; if self.warmup_penalty > 0.0 && FIRST_RUN.load(Ordering::Relaxed) { info!("First inference"); FIRST_RUN.store(false, Ordering::Relaxed); delay += delay * self.warmup_penalty; } let delay = Duration::from_secs_f32(delay); sleep(delay); if fastrand::f32() < self.failure_rate { anyhow::bail!("Unexpected inference failure"); } else { if fastrand::f32() < self.panic_rate { panic!("Inference catastrophically failed"); } else { Ok(Output { count: data.len() }) } } } }</code></pre> The first thing that probably jumps out is that we have this static AtomicBool</code> at the top called FIRST_RUN</code>. Most neural network frameworks lazy-load the model weights or have an optimisation step after some data is processed. Using Tensorflow, I’ve seen the first inference be 10-15x slower than subsequent inferences because of lazy loading. Our calculated delay is roughly: $$ y = mx+c+rand(jitter) $$ This means it should linearly increase as the processed data gets longer with some optional noise and a minimum inference time. Additionally, the first inference time is by default quite long to match behaviour observed in the wild. There’s also, some likelihood of the inference panicking or failing without a panic. Interfacing with neural network runtimes often involves an FFI interface and GPUs. Both of these can cause issues for us, either in the event of resource exhaustion, misconfiguration or woe forbid a dormant issue in the library. One final detail, I’ve derived Deserialize</code> for this so a config file can be used easily changing the behaviour. Passing Audio In</a></h2> In a previous entry, we passed bytes in from the client to an audio decoding module where audio is resampled and sent down another channel to run inference on. Because of this abstraction, everything coming into our type holding the model receives audio for the correct format and sample rate. With this simplicity at our disposal, we can start to look at passing audio in unburdened by earthly concerns. At the top level, we need something to load the model and also pass in the audio received from the API. Often people will give this a super descriptive name like Context</code> or State</code>, but here we’re going to be more descriptive: /// Streaming context. This holds a handle to the model as well as /// potentially some parameters to control usage and how things are split up pub struct StreamingContext { model: Model, max_futures: usize, }</code></pre> Oh that’s not actually much more descriptive, but at least it’ll avoid any potential naming conflicts - anyhow</code> has a Context</code> trait and other crates do as well. Now as previously mentioned, we’ll have two different inference APIs, one that processes all the audio and one which processes only voiced segments. Let’s start with the simple one first. impl StreamingContext { /// This is the simple inference where every part of the audio is processed by the model /// regardless of speech content being present or not #[instrument(skip_all)] pub async fn inference_runner( self: Arc<Self>, channel: usize, mut inference: mpsc::Receiver<Arc<Vec<f32>>>, output: mpsc::Sender<ApiResponse>, ) -> anyhow::Result<()> { let mut runners = FuturesOrdered::new(); let mut still_receiving = true; let mut received_results = 0; let mut received_data = 0; let mut current_start = 0.0; let mut current_end = 0.0; // Need to test and prove this doesn't lose any data! while still_receiving || !runners.is_empty() { tokio::select! { audio = inference.recv(), if still_receiving && runners.len() < self.max_futures => { if let Some(audio) = audio { let temp_model = self.model.clone(); current_end += audio.len() as f32/ MODEL_SAMPLE_RATE as f32; let bound_ms = (current_start, current_end); runners.push_back(task::spawn_blocking(move || { (bound_ms, temp_model.infer(&audio)) }); current_start = current_end; } else { still_receiving = false; } } data = runners.next(), if !runners.is_empty() => { received_results += 1; debug!("Received inference result: {}", received_results); let data = match data { Some(Ok(((start_time, end_time), Ok(output)))) => { let segment = SegmentOutput { start_time, end_time, is_final: None, output }; Event::Segment(segment) }, Some(Ok(((start_time, end_time), Err(e)))) => { error!("Failed inference event {}-{}: {}", start_time, end_time, e); Event::Error { message: e.to_string() } } Some(Err(e)) => { error!(error=$e, "Inference panicked"); Event::Error { message: "Internal server error".to_string() } }, None => { continue; } }; let msg = ApiResponse { channel, data }; output.send(msg).await?; } } } info!("Inference finished"); Ok(()) } }</code></pre> The way this task works is we create two futures in the select which: Receives audio if there’s room in the runners and spawns an inference task</li> Collects the finished inference results and forwards them or errors to the client</li> </ol> Some people might want to exit on an error, but for this initial implementation, we’ll send back an error message and continue processing. Another thing to note is the usage of FuturesOrdered</code>. While there may be more efficient ways to do this for now we’ll be relying on this because it is a fairly simple and convenient way to run multiple futures concurrently and get the responses in order. Additionally, max_futures</code> is only done on a per-request basis, if we wanted it as a global limit we’d have to look to something like a semaphore. When the audio isn’t receiving and there are no running inference futures the while loop terminates and we’re home and dry. This code has been made so simple through the use of channels. These can come with some performance implications, but they should generally perform well enough initially. In a future entry I’ll cover some tips to get the most speed out of them. The next stage will be the voice-segmented runner which is a ton more complex. This might be complex enough to warrant its own subheading! The Segmented API</a></h2> The VAD segmented API will have more places we can call inference, we’ll be calling it on the end of an utterance and also if interim results are desired at regular intervals. Additionally, with how the VAD library works we’ll potentially have an active speech segment at the end and need to run a final inference on it after we get a stop request. We also won’t be running inferences in parallel because we won’t run at a fixed rate. With this in mind, I’ll be creating a new method called spawned_inference</code> which the segmented runner can call every time it wants to perform an inference. async fn spawned_inference( &self, audio: Vec<f32>, bounds_ms: Option<(usize, usize)>, is_final: bool, ) -> Event { let temp_model = self.model.clone(); let result = task::spawn_blocking(move || temp_model.infer(&audio)).await; match result { Ok(Ok(output)) => { if let Some((start, end)) = bounds_ms { let start_time = start as f32 / 1000.0; let end_time = end as f32 / 1000.0; let seg = SegmentOutput { start_time, end_time, is_final: Some(is_final), output, }; Event::Segment(seg) } else { Event::Data(output) } } Ok(Err(e)) => { error!("Failed inference event: {}", e); Event::Error { message: e.to_string(), } } Err(e) => { error!(error=%e, "Inference panicked"); Event::Error { message: "Internal server error".to_string() } } } }</code></pre> There is potential here to use a Duration</code> instead of usize</code> for the time tracking. But as we work it out from samples and our resolution will be limited by the interim response interval a usize</code> works well enough and saves some type conversion effort. We can see this is fairly similar to the simple runner, and while we could work to refactor our simple runner to use this method it wouldn’t be desirable. A future typically only progresses when .await</code> is called on it. And for futures in an FuturesOrdered</code> that will be when we poll the FuturesUnordered</code>. Whereas, the futures returned by spawn</code> and spawn_blocking</code> will start running before being polled. Because of this refactoring would add a delay to when our first task is spawned and add latency into the system. For now, let’s do an initial pass at the implementation this first version won’t be fully featured. Initially, we’ll skip: Events (speech start/end emitting)</li> Partial Inferences</li> </ol> Additionally, for the VAD we’ll be using an opinionated version of silero</a>. This is a project open-sourced by my employer, Emotech and can be found here</a>. The Silero crate will hold onto the audio buffer, you pass it slices and it will push it onto an internal queue. Active speech can be accessed with VadSession::get_current_speech</code> and each process can return a Vec</code> of events containing the speech starts and ends. The ending speech has the samples contained within so they can be removed from the internal buffer as well. With that short introduction to our new dependency here’s the initial code: pub async fn segmented_runner( self: Arc<Self>, _settings: StartMessage, channel: usize, mut inference: mpsc::Receiver<Vec<f32>>, output: mpsc::Sender<ApiResponse>, ) -> anyhow::Result<()> { let mut vad = VadSession::new(VadConfig::default())?; let mut still_receiving = true; // Need to test and prove this doesn't lose any data! while let Some(audio) = inference.recv().await { let mut events = vad.process(&audio)?; for event in events.drain(..) { match event { VadTransition::SpeechStart { timestamp_ms } => { todo!() } VadTransition::SpeechEnd { start_timestamp_ms, end_timestamp_ms, samples, } => { info!(time_ms = end_timestamp_ms, "Detected end of speech"); let data = self .spawned_inference( samples, Some((start_timestamp_ms, end_timestamp_ms)), true, ) .await; let msg = ApiResponse { channel, data }; output .send(msg) .await .context("Failed to send inference result")?; } } } } // If we're speaking then we haven't endpointed so do the final inference if vad.is_speaking() { let audio = vad.get_current_speech().to_vec(); info!(session_time=?vad.session_time(), current_duration=?vad.current_speech_duration(), "vad state"); let current_start = (vad.session_time() - vad.current_speech_duration()).as_millis() as usize; let current_end = session_time.as_millis() as usize; let data = self .spawned_inference(audio, Some((current_start, current_end)), true) .await; let msg = ApiResponse { channel, data }; output .send(msg) .await .context("Failed to send final inference")?; } info!("Inference finished"); Ok(()) }</code></pre> So how do we go about adding events? Simple just emit them when we get our speech start/end! So our match becomes: match event { VadTransition::SpeechStart { timestamp_ms } => { info!(time_ms = timestamp_ms, "Detected start of speech"); let msg = ApiResponse { data: Event::Active { time: timestamp_ms as f32 / 1000.0, }, channel, }; output .send(msg) .await .context("Failed to send vad active event")?; } VadTransition::SpeechEnd { start_timestamp_ms, end_timestamp_ms, samples, } => { info!(time_ms = end_timestamp_ms, "Detected end of speech"); let msg = ApiResponse { data: Event::Inactive { time: end_timestamp_ms as f32 / 1000.0, }, channel, }; // We'll send the inactive message first because it should be faster to // send output .send(msg) .await .context("Failed to send vad inactive event")?; let data = self .spawned_inference( samples, Some((start_timestamp_ms, end_timestamp_ms)), true, ) .await; let msg = ApiResponse { channel, data }; output .send(msg) .await .context("Failed to send inference result")?; } }</code></pre> Also, we can’t forget the final inference of any audio that’s still not finished speaking. // If we're speaking then we haven't endpointed so do the final inference if vad.is_speaking() { let session_time = vad.session_time(); let msg = ApiResponse { data: Event::Inactive { time: session_time.as_secs_f32(), }, channel, }; output .send(msg) .await .context("Failed to send end of audio inactive event")?; let audio = vad.get_current_speech().to_vec(); info!(session_time=?vad.session_time(), current_duration=?vad.current_speech_duration(), "vad state"); let current_start = (vad.session_time() - vad.current_speech_duration()).as_millis() as usize; let current_end = session_time.as_millis() as usize; let data = self .spawned_inference(audio, Some((current_start, current_end)), true) .await; let msg = ApiResponse { channel, data }; output .send(msg) .await .context("Failed to send final inference")?; }</code></pre> Wow, this event stuff is easy. Okay, now onto the interim results and we’ll be done with this code. We’ll add two variables at the top of the function: let mut last_inference_time = Duration::from_millis(0); // So we're not allowing this to be configured via API. Instead we're setting it to the // equivalent of every 500ms. const INTERIM_THRESHOLD: Duration = Duration::from_millis(500);</code></pre> We want to track when we last did an interfence so our interim result doesn’t come too early or too late. We also define our constant interim duration - this could be part of the API but we don’t want to put extra work into making sure users don’t crash our system with insane values. Remember the model is expected to be computationally intense and will take up some amount of compute so setting unreasonably low times to run it is an easy way to bring down a service! We’ll store the value from VadTransition::SpeechEnd::end_timestamp_ms</code> in our last_inference_time</code> then after the code that processes our VAD events we want to put the partial processing: for event in events.drain(..) { // You've seen this code above! } let session_time = vad.session_time(); if vad.is_speaking() && (session_time - last_inference_time) >= INTERIM_THRESHOLD { last_inference_time = vad.session_time(); info!(session_time=?vad.session_time(), current_duration=?vad.current_speech_duration(), "vad state"); let current_start = (vad.session_time() - vad.current_speech_duration()).as_millis() as usize; let current_end = session_time.as_millis() as usize; let data = self .spawned_inference( vad.get_current_speech().to_vec(), Some((current_start, current_end)), false, ) .await; let msg = ApiResponse { channel, data }; output .send(msg) .await .context("Failed to send partial inference")?; }</code></pre> And that’s all there is to it. When all this is put together the function is quite long but I’ve kept it together instead of making some smaller functions that are only used in one place. This might change in future though. Conclusion</a></h2> Well here we are, we’ve got a fake model and inferencing code to call it for our planned APIs. Now we need to cover the Axum API we’ll make and then we’ll have a fully working system and can start to dive into more depth on various parts of the system. Streaming Audio APIs: Audio Decoding 2024-11-04T00:00:00+00:00 In the last entry, it was decided that the audio coming in would be PCM data with a specified sample rate, channel count, and data type. This is quite helpful, as we don’t need to think about audio codecs and setting up decoders. There is one thing we have to think of though: converting from the input sample rate to our model’s desired sample rate. Models typically only work on one sample rate. Rarely, you might see a model take in 2 and do some basic resampling logic at the input. But in these cases giving it the right sample rate first will cause the inference code to do less work so it’s still preferable. I also said previously we’d be using the actor model, if you’re not familiar there’s this blog post by Alice Rhyl actors with tokio</a>. The input to the function will be a channel sending the encoding audio data, we’ll already have the format info when we call it to keep things simpler. The output will be decoded samples sent down a tokio channel specific to the audio-channel. I’ll do my best to try and avoid audio-channel vs tokio channel confusion, but please leave feedback if it’s ambiguous anywhere. The terminology overlap is a real pain at times! use tokio::sync::mpsc; use crate::api_types::AudioFormat; use bytes::Bytes; pub async fn decode_audio( audio_format: AudioFormat, mut rx: mpsc::Receiver<Bytes>, channel_data_tx: Vec<mpsc::Sender<Vec<f32>>>, ) -> anyhow::Result<()> { todo!() }</code></pre> To recap the AudioFormat</code> type is: /// Describes the PCM samples coming in. I could have gone for an enum instead of bit_depth + /// is_float but I only really plan on doing f32, s16 and ignoring everything else including /// compression schemes like mulaw etc. This may change in future. #[derive(Copy, Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub struct AudioFormat { /// Number of channels in the audio pub channels: usize, /// Sample rate of the audio pub sample_rate: usize, /// Number of bits per sample pub bit_depth: u16, /// Whether audio uses floating point samples pub is_float: bool, }</code></pre> Matching on is_float</code> and bit_depth</code> to redirect to the correct decoding code is mildly a pain. Internally, to solve this I’ll create a SampleFormat</code> enum and methods on that enum will handle the decoding of raw bytes to samples, also AudioFormat</code> will have a method to return the version of the enum that does that matching. #[derive(Copy, Clone, Debug, PartialEq, Eq)] enum SampleFormat { /// Signed 16 bit integer Int16, /// 32 bit float Float32 } impl AudioFormat { fn sample_format(&self) -> anyhow::Result<SampleFormat> { match (self.bit_depth, self.is_float) { (16, false) => Ok(SampleFormat::Int16), (32, true) => Ok(SampleFormat::Float32), (bd, float) => { anyhow::bail!("Unsupported format bit_depth: {} is_float: {}", bd, float) } } } }</code></pre> After this, decoding is a case of taking whole samples by their size in bytes, and applying a conversion function to them. Here I’m going to use a boxed function object for the float conversion and there’s going to be some unwrapping. The unwrapping is fine though, because we’ll have a number of checks beforehand to make sure the data is the right size. impl SampleFormat { const fn bytes_per_sample(&self) -> usize { match self { Self::Int16 => 2, Self::Float32 => 4, } } fn to_float_fn(&self) -> Box<dyn Fn(&[u8]) -> f32> { let len = self.bytes_per_sample(); match self { Self::Int16 => Box::new(move |x: &[u8]| { i16::from_le_bytes((&x[..len]).try_into().unwrap()) as f32 / i16::MAX as f32 }), Self::Float32 => { Box::new(move |x: &[u8]| f32::from_le_bytes((&x[..len]).try_into().unwrap())) } } } fn to_float_samples(&self, samples: &[u8]) -> anyhow::Result<Vec<f32>> { let len = self.bytes_per_sample(); if samples.len() % len != 0 { anyhow::bail!("Got a partial sample: {} bytes", samples.len()); } let conversion = self.to_float_fn(); let samples = samples.chunks(len).map(conversion).collect(); Ok(samples) } }</code></pre> To improve this you could have to_float_samples</code> take in an output buffer and reuse allocations. That can be an exercise for the reader (or a future post). But generally speaking, this part of the code is always going to be lightning fast compared to the model code so we’ll hold off on optimising this too much for now. Some readers may be wondering why doesn’t this SampleFormat</code> enum become part of our public API? Well it could, and if there were PCM sample formats we wanted to support not encoded by those constraints (mu-law and a-law) we’d shift to it. This is just a bit of personal preference seeping into the API design, there’s several tools/APIs taking in raw audio with stringly typed formats and users may assume compatibility with something else they use. Keeping to a number and boolean prevents some of these assumptions. Right back to code! Initially, we’ll error out on any formats that aren’t the correct sample rate. This will help me keep things simpler initially. So with some initial error checking, we have everything we need to do a first implementation: pub async fn decode_audio( audio_format: AudioFormat, mut rx: mpsc::Receiver<Bytes>, channel_data_tx: Vec<mpsc::Sender<Vec<f32>>>, ) -> anyhow::Result<()> { if channel_data_tx.is_empty() { anyhow::bail!("No output sinks for channel data"); } if audio_format.sample_rate != MODEL_SAMPLE_RATE { // No need to be more specific in message, we'll rewrite this later anyhow::bail!("Invalid sample rate provided"); } let mut received_samples = 0; let mut sent_samples = 0; let mut current_buffer = vec![]; while let Some(data) = rx.recv().await { let format = audio_format.sample_format()?; let mut samples = format.to_float_samples(&data)?; received_samples += samples.len(); current_buffer.append(&mut samples); let mut channels = vec![ Vec::with_capacity(current_buffer.len() / audio_format.channels); audio_format.channels ]; for (chan, data) in (0..channel_data_tx.len()) .cycle() .zip(current_buffer.drain(..)) { channels[chan].push(data); } for (data, sink) in channels.drain(..).zip(&channel_data_tx) { sink.send(data).await?; } } }</code></pre> Looking at this there are probably a few questions. Could we handle interleaving in the decoding side instead of an extra loop?</li> Could we also not await on the channel send and fire off more futures quicker?</li> </ul> The answer to both of these is yes. However, considering the interleaving if you’re using an existing library like ffmpeg or maybe gstreamer to decode samples you often get your data interleaved and then have to do this extra step anyway. For services with a streaming and non-streaming mode where the non-streaming accepts any potential file format having solid reusable audio code for both is pretty nice and that’s how it exists at my Day Job so that’s what’s replicated here. For the second point, this might be done in the future, but this is “good enough”. At least there are potentially easy optimisations we can work on in the future. Also, in the short-term we know our audio is coming through in the correct ordering and once we have more testing infrastructure in place we can work on harder things. Sample Rates and Human Speech</a></h2> Before jumping into resampling I’ll cover a bit of the surrounding context. It’s not shown in the code, but most speech applications working on wideband data use 16KHz audio, it’s been shown that going higher than this doesn’t improve the performance of speech models. This is in part because of things like the Nyquist-Shannon Sampling Theory (sample rate should be at least 2x the highest frequency). If audio comes in above this and we downsample we don’t lose any voice data and if the rest of the signal is considered noise we’ll just be losing the noise! Looking a bit more on how this works you’ll want to read up on formants - the resonant frequencies from the human mouth and vocal tract. These have been studied and ranges of frequencies of the different formants recorded. Nowadays people typically ignore this and stick to 16KHz unless they’re working with narrowband data such as certain telephony data. So this is purely if you have a deeper interest in the signal processing involved in human speech. Resampling Audio</a></h2> We’ll be implementing our audio resampling via the rubato</a> crate. It seems to be the most thorough crate and provides a few algorithms and parameters to tweak them. And what is software if not the desire for an abundance of knobs to twiddle? Unfortunately, making a choice of technique and how to tune it involves a lot of maths. Because of this I’m going to try and gloss over some of these details as going into this maths is a deep dive series in it’s own right. The resamplers in Rubato are either synchronous or asynchronous, but this is another collision of terminology between domains. An asynchronous resampler allows you to adjust the resampling ratio while the resampler is running. This could be useful for certain codecs or when using an audio protocol that might adjust the sample rate based on the bandwidth available. We’re not going to change the sample rate after initial setup so we don’t have to worry about this. But if you work on an application where this can happen you should think about your maximum possible sample rate and the algorithm used. Changing the sample rate can either be completely fine or change the aliasing effects in resampling so a wrong decision can increase the noise in the output! Think about whether you’ll be increasing or decreasing the sample rate and how your resampler works. Wait what are aliasing effects? Oh right I just threw that term in there didn’t I. Aliasing is when high frequencies are undersampled and low frequencies produced. Visually this is what happens when you see a video of a car driving and the wheels appear to go backwards, or the helicopter videos where the propellers appear to be stationary. Also, in images this can cause a Moiré pattern. In resampling, we can sometimes see high frequency noise turn into lower frequency signal, and the effects/impact of that can differ depending on the algorithm. With that diversion done, looking at Rubato our available types of resamplers are: “Fast”</li> Sinc</li> FFT (Fast Fourier Transform)</li> </ul> Fast and Sinc are both asynchronous and FFT is a synchronous resampler. We can use an asynchronous resampler like a synchronous one. I’ll expand on the reasons why later but for now just accept we’re using a Sinc resampler. We’ll also use a fixed input size resampler, this is because when batching up the input we can more easily see when we hit the necessary input length than checking the resampler on how many more input frames it desires. Within the code, we can more easily trigger when we want to resample and emit our samples. We’ll create our resampler as follows: fn create_resampler(audio_format: &AudioFormat, resampler_size: usize) -> anyhow::Result<SincFixedIn<f32>> { let window = WindowFunction::Blackman; let params = SincInterpolationParameters { sinc_len: 256, f_cutoff: calculate_cutoff(256, window), oversampling_factor: 128, interpolation: SincInterpolationType::Cubic, window, }; // resample_ratio, max_resample_ratio_relative, params, input buffer size, channel count let resampler = SincFixedIn::new( MODEL_SAMPLE_RATE as f64 / audio_format.sample_rate as f64, 1.0, params, resampler_size, audio_format.channels, )?; trace!( input_frames_max = resampler.input_frames_max(), output_frames_max = resampler.output_frames_next(), "Resampler created" ); Ok(resampler) }</code></pre> Most of these parameters come from the documentation or examples, which means they should be a relatively good performance/quality tradeoff. Looking over the docs parameters like window functions are explained along with their impacts on the output. As we don’t plan on changing the sample rate the max_resample_ratio_relative</code> is set to 1.0. Now we change our bailing on incorrect sample rates from: if audio_format.sample_rate != MODEL_SAMPLE_RATE { // No need to be more specific in message, we'll rewrite this later anyhow::bail!("Invalid sample rate provided"); }</code></pre> Into: const RESAMPLER_SIZE: usize = 4096; let mut resampler = if audio_format.sample_rate != MODEL_SAMPLE_RATE { Some(create_resampler(&audio_format, RESAMPLER_SIZE)?) } else { None };</code></pre> Our audio-receiving loop becomes a bit more complicated: l let resample_trigger_len = audio_format.channels * RESAMPLER_SIZE; let mut current_buffer = Vec::with_capacity(resample_trigger_len); while let Some(data) = rx.recv().await { // We could do the sample extraction and uninterleave the samples in one go. But if you're // using an existing library like ffmpeg (or maybe gstreamer) to decode and resample audio // you'll get the audio interleaved and have to split it out so you will get a similar // performance profile anyway. let format = audio_format.sample_format()?; let mut samples = format.to_float_samples(&data)?; received_samples += samples.len(); current_buffer.append(&mut samples); if current_buffer.len() >= resample_trigger_len || resampler.is_none() { let capacity = RESAMPLER_SIZE.min(current_buffer.len() / audio_format.channels); let mut channels = vec![Vec::with_capacity(RESAMPLER_SIZE); audio_format.channels]; for (chan, data) in (0..channel_data_tx.len()) .cycle() .zip(current_buffer.drain(..(capacity * audio_format.channels))) { channels[chan].push(data); } let mut channels = if let Some(resampler) = resampler.as_mut() { resampler.process(&channels, None)? } else { channels }; for (data, sink) in channels.drain(..).zip(&channel_data_tx) { //trace!("Emitting {} samples for channel {}", data.len(), i); sent_samples += data.len(); sink.send(data.into()).await?; } } }</code></pre> The changes here: We accumulate our decoded samples into a buffer</li> When we have enough samples for the input we trigger a resampling</li> We make a Vec<Vec<f32>></code> for our de-interleaved input data</li> Pass it into the resampler if we need to resample, otherwise use it directly</li> Send the samples on</li> </ol> There’s just one last thing we haven’t yet considered, we need the full buffer of samples to trigger a resampling. What if there’s some leftover but not enough for a full buffer? The answer is to largely repeat the same resampling process but use the process_partial</code> function in the resampler to process an incomplete buffer and then truncate the output to the correct length. After the resampling loop, we fill in this code: if !current_buffer.is_empty() { trace!( "Sent out {} expected output {}", sent_samples, received_samples as f64 * resample_ratio ); let new_len = if (sent_samples as f64) < received_samples as f64 * resample_ratio { let per_channel_sample = ((received_samples as f64 * resample_ratio) / audio_format.channels as f64) .round() as usize; Some(per_channel_sample - sent_samples) } else { None }; let mut channels = vec![ Vec::with_capacity(current_buffer.len() / audio_format.channels); audio_format.channels ]; for (chan, data) in (0..channel_data_tx.len()) .cycle() .zip(current_buffer.drain(..)) { channels[chan].push(data); } let mut channels = if let Some(resampler) = resampler.as_mut() { resampler.process_partial(Some(&channels), None)? } else { channels }; for (mut data, sink) in channels.drain(..).zip(&channel_data_tx) { if let Some(new_len) = new_len { trace!( "Downsizing to avoid trailing silence to {} bytes from {}", new_len, data.len() ); data.resize(new_len, 0.0); } sink.send(data.into()).await?; } }</code></pre>Testing It</a></h2> The question now comes of how to test this. We want to make sure we’re resampling things relatively correctly. Given sample rates may not be a factor of one another we can’t easily figure out input and expected output data. Additionally, some of the maths involved means there may be aliasing effects within certain frequency ranges and an element of error. So how do we get past this? One of my preferred techniques is to bring in a crate called dasp</a> for digital audio signal processing. Using dasp I can create a signal and alter things such as the sample rate to create a “perfect” version of the signal at my input and output sample rates and then compare them. Keeping things relatively simple with a sine wave I can make a 1.6KHz sine wave sampled at 16KHz and grab 32000 samples of it as follows: let expected_output = signal::rate(16000.0) .const_hz(1600.0) .sine() .take(32000) .map(|x| x as f32) .collect::<Vec<f32>>();</code></pre> Another thing is resamplers can add noise and amplify or suppress certain components in the signal. They may also impact the phase of the signal. This is why the FFT resampler and fast resampler weren’t chosen. Using the test metric I picked they scored poorly and also doing a direct comparison to the expected dasp output doesn’t work. To test the signal is as expected a metric called cross-correlation is used. This measures the similarity of two signals, and you can apply a delay and find the overlapping point where they are most similar. Our audio and resampled audio won’t have a lag applied it should already exactly overlap so we only need to do the maths at delay=0. And while I vaguely know the maths and can find it in some of my old university notes I wanted to lean on an existing “golden” implementation for discrete instead of continuous time. So using this source</a> we have this function to measure the similarity of two signals: fn xcorr(a: &[f32], b: &[f32]) -> f32 { let mean_a = a.iter().sum::<f32>() / a.len() as f32; let mean_b = b.iter().sum::<f32>() / b.len() as f32; let num = a .iter() .zip(b.iter()) .map(|(a, b)| (a - mean_a) * (b - mean_b)) .sum::<f32>(); let dn_a = a.iter().map(|x| (x - mean_a).powi(2)).sum::<f32>().sqrt(); let dn_b = b.iter().map(|x| (x - mean_b).powi(2)).sum::<f32>().sqrt(); num / (dn_a * dn_b) }</code></pre> The xcorr function I’ve provided doesn’t let you apply a lag or delay to the signals. We can look how how cross-correlation output appears for two similar but slightly delayed sine waves below. Also, everything outside of the visible sine wave will be 0 amplitude in the cross-correlate. At lag 0, this is the value the function should output, the higher the more similar they are. Comparing this to a 180° phase shift the signal scores very poorly: This is better than the very unreliable comparison of each sample and a rougher root-mean-square error or energy calculation which may not work well for changes with no perceptual impact i.e. slight phase offsets. While the xcorr method won’t work for big phase changes small changes can still score reliably above a certain threshold we decide upon. Our inputs are a stream of bytes and the AudioFormat</code> type and the outputs can be grabbed as a Vec<Vec<f32>></code>. Using this a test_audio</code> function can be created that all the tests can utilise: async fn test_audio( format: AudioFormat, mut input: BytesMut, chunk_size: usize, expected: Vec<Vec<f32>>, output_name: &str, ) { let (bytes_tx, bytes_rx) = mpsc::channel(8); let mut output_channels = vec![]; let mut incoming_samples = FuturesOrdered::new(); for _ in 0..format.channels { let (sample_tx, mut sample_rx) = mpsc::channel::<AudioChannel>(8); output_channels.push(sample_tx); incoming_samples.push_back(async move { let mut resampled: Vec<f32> = vec![]; while let Some(samples) = sample_rx.recv().await { resampled.extend_from_slice(&samples); } resampled }); } let decoder = tokio::spawn(decode_audio(format, bytes_rx, output_channels)); let handle = tokio::spawn(async move { while !input.is_empty() { let to_send = if input.remaining() > chunk_size { input.split_to(chunk_size) } else { input.split() }; bytes_tx.send(to_send.freeze()).await.unwrap(); } }); let resampled = incoming_samples.collect::<Vec<_>>().await; let expected_name = format!("{}_expected.wav", output_name); let actual_name = format!("{}_actual.wav", output_name); write_wav(&expected_name, &expected); write_wav(&actual_name, &resampled); decoder.await.unwrap().unwrap(); handle.await.unwrap(); // Save our files for debugging for (channel_index, (expected_channel, actual_channel)) in expected.iter().zip(resampled.iter()).enumerate() { assert_eq!(expected_channel.len(), actual_channel.len()); let similarity = xcorr(&expected_channel, &actual_channel); println!( "Channel {} cross correlation is {}", channel_index, similarity ); assert!(similarity > 0.95); } let _ = fs::remove_file(&expected_name); let _ = fs::remove_file(&actual_name); }</code></pre> The file is saved and at the end of the function deleted. Therefore, if a test fails we have the expected and actual outputs as audio files that can be inspected in Audacity or a similar waveform viewer to figure out what went wrong. With these building blocks in place, a test can be written as so: #[tokio::test] #[traced_test] async fn upsample_s16_audio() { let format = AudioFormat { sample_rate: 8000, channels: 1, bit_depth: 16, is_float: false, }; let expected_output = signal::rate(16000.0) .const_hz(800.0) .sine() .take(64000) .map(|x| x as f32) .collect::<Vec<f32>>(); let input = signal::rate(8000.0) .const_hz(800.0) .sine() .take(32000) .map(|x| x as f32) .flat_map(|x| ((x * i16::MAX as f32) as i16).to_le_bytes()) .collect::<BytesMut>(); test_audio(format, input, 300, vec![expected_output], "upsample_s16").await; }</code></pre> I’m using tracing-test</a> to enable tracing logs in tests for cases on if things fail. If people can see their logs appear when tests fail it makes them less likely to sneak in debug print statements that pass through PR review. Once this was done and the testing approach validated I filled in a few extra tests covering: Upsampling - increasing the sample rate</li> Downsampling - decreasing the sample rate</li> Passthrough - sample rate is unchanged</li> Mono and stereo audio data</li> Some variations on chunk sizes to ensure there are remainders left</li> </ol> With all of this done, it looks like testing is complete! Final Thoughts</a></h2> The world of signal processing is hard and full of maths, and maybe better ways of testing resampled audio for correctness exists. In my experience cross-correlation has addressed a number of shortcomings with other methods and when validating the approach I’ve not seen any samples rejected that should have been accepted. But if anyone has better approaches I’d be interested in hearing about them. For different resamplers a different testing strategy has to be considered, comparing resampled audio can be unreasonably hard! Resamplers can do all manner of things to make audio try to sound better or run faster. Additionally, while upsampling is being allowed in this project, you may want to deny users that ability. If you have models that are trained on audio of a specific quality accepting audio of a lower quality will often result in lower accuracy. In my experience the previously mentioned product manager will have to make the call on whether to accept any input or prevent the users giving input that makes your product look bad. More likely, they’ll just ask you if you can make it perform with equal accuracy on any inputs. So good luck! Debloating Github Actions with Rust 2024-10-26T00:00:00+00:00 Sometimes, you may end up in a situation where you have a big project to test in CI. Maybe you have some data that is processed or loaded or test outputs that clog up a bunch of disk. And github actions starts failing, out of disk space yada yada. Irritated you start a barrage of fix ci</code> commits trying to reduce the amount of data you generate and slim it down but it’s to no avail. Then you look to the base images, the ubuntu-latest that you’re building on top of. Can you slim down some stuff in this? Maybe, what does it come installed with? Android, Haskell, .NET, docker images? 55GB of storage taken up in a 75GB container! Will no one rid me of these meddlesome files? In steps free-disk-space</a>. It looks okay and someones gone to the trouble. You sit back and relax as unprompted it visits all these files you had no idea existed and rids you of them. A few months pass, the seasons change, cherry blossoms bloom and fall as spring turns to summer turns to winter. The days grow shorter, and so should your CI time. Are there any wins to be had here? Looking in the logs there’s alarm free-disk-space takes up 3-5 minutes. Bloody ’ell. So here I am, about to embark on my own voyage. Part shitpost, part CI bill reduction. With free-disk-space lets take a look at the disk stats it prints out: Before running: Filesystem Size Used Avail Use% Mounted on /dev/root 73G 55G 18G 76% /</code></pre> After removing everything: Filesystem Size Used Avail Use% Mounted on /dev/root 73G 33G 41G 45% / Saved 23GiB</code></pre> How well does a simple rm fair? Adding the following command to CI took 1 minute 15 seconds. However, looking at a more recent run it only took 16 seconds which kind of kills my motivation for this tool in retrospect… sudo rm -rf /opt/ghc /usr/local/.ghcup /usr/local/lib/android /usr/share/dotnet</code></pre> Okay, well we can still pretend it takes 1 minute all the time to not make it seem like I’ve wasted my time! Plus, free-disk-space does more than this. It runs a bunch of apt-get remove commands, does a docker image prune and removes swap storage. Sure those rm’s were enough to speed up my CI but you never know… Naturally, as bash has failed us it’s time to use Rust and because it’s my go to language. I can add in the folder paths to delete, delete them in parallel via rayon and do the docker command in a separate thread. But what about those apt-get commands? I probably can’t parallelise them because of the package lock. And here comes a suggestion from a friend: cursed idea for apt, if you really wanted to go zoom. you could run the apt command with strace in your CI to generate a list of files it deletes, and then download this list and delete them in parallel as fast as you can, since i assume apt is going to be pretty slow for a file deletion tool </blockquote> That’s right! apt-get is just a shitty rm in this case. I know, I’ll run a CI job that does the apt-get commands, parse the strace output and build up a file list. This file list will then be committed into the code and a PR opened if it changes. And then my tool will use the list to just push more files and paths into my rayon deleting iterator. I’ve not used strace before so a quick skim of the man page to and I’ve figured out a command that should work and have the following code: fn remove_packages(packages: &[&str]) -> Vec<String> { let strace_out = Command::new("strace") .args(["-e", "trace=%file", "-f", "apt-get", "remove", "-y"]) .args(packages) .arg("--fix-missing") .output() .expect("Failed to strace apt-get"); process_strace_output(&strace_out.stderr) } fn cleanup() -> Vec<String> { let strace_out = Command::new("strace") .args(["-e", "trace=%file", "-f", "apt-get", "autoremove", "-y"]) .output() .expect("Failed to strace apt-get"); process_strace_output(&strace_out.stderr) }</code></pre> process_strace_output</code> is a bit more fiddly. We have to grab just the syscalls we care about. Then the arguments from them and then get the arguments that are files. Some of the syscalls also seem to have optional arguments that get missed off. Also, if a remove doesn’t work we don’t want to list that file. So our hacky start of processing output to get the list of files is: fn process_strace_output(output: &[u8]) -> Vec<String> { let string = String::from_utf8_lossy(output); let mut result = vec![]; for line in string.lines() { if line.ends_with("= 0") && (line.contains("unlink") || line.contains("rmdir")) { let mut tmp = line.to_string(); let mut keep = false; tmp.retain(|c| { if c == '(' { keep = true; false } else if c == ')' { keep = false; false } else { keep } }); for maybe_path in tmp.split(",").filter(|x| x.contains(MAIN_SEPARATOR_STR)) { let tmp = maybe_path .trim() .trim_start_matches("\"") .trim_end_matches("\"") .to_string(); // Do a cheeky filter out of files in directory if line.contains("rmdir") { result.retain(|x: &String| !x.starts_with(&tmp)); } if !tmp.is_empty() { result.push(tmp); } } } } result.dedup(); result }</code></pre> Here I’m trying to be extra-lazy and avoid Regex to get the arguments, by being a bit tricksy with retain. Also, assuming that ,</code> isn’t in a path or other argument is a bit rubbish of me. But it does hold true for ubuntu-latest github actions runner (don’t change this GitHub, it’s now a characteristic of your images people rely on /s). All the paths I saw were absolute, and I saw the syscalls unlink</code>, unlinkat</code> and rmdir</code> are used for deleting things. Try and avoid files that are in paths and bish bash bosh. This list was still really big, so further compression was needed: fn compress_deletions(inputs: Vec<String>) -> Vec<String> { let mut set = BTreeSet::new(); set.insert("/var/cache".to_string()); for file in inputs.iter() { let path = Path::new(&file); if set.iter().any(|x| path.starts_with(x)) { continue; } let mut parent = match path.parent() { Some(s) => s, None => continue, }; if parent.exists() { let parent_is_empty = fs::read_dir(parent).map(|mut x| x.next().is_none()).unwrap_or(false); if parent_is_empty { println!("Removing parent: {}", parent.display()); set.insert(parent.display().to_string()); } else { println!("Removing file: {}", file); set.insert(file.clone()); } } else { while let Some(new_parent) = parent.parent() { if new_parent.exists() { println!("Reduced {} to {}", path.display(), parent.display()); set.insert(parent.display().to_string()); break; } else { println!("Going from {} to {}", parent.display(), new_parent.display()); parent = new_parent } } } } set.into_iter().collect() }</code></pre> I just go up the parents until I find a parent folder that exists and then delete the child as it’s clear apt-get remove deleted the folder! A bit of special /var/cache</code> handling and we’re good to go. Just save it as a RON and piss off. I won’t bother showing the end tools code, but it’s really short like <150 lines. You can read it here</a> if you’re interested. And it takes about 15-20s to run. Publishing It.</a></h2> Now seems like a good time to figure out a way to put this on the Github marketplace. First off doing an actual release of the binary including a github release will be needed. This is because we don’t want to build the code via cargo install</code> in CI because that’ll just be a bit too slow. Instead I plan on using cargo-binstall</code> which will download a binary from a Github release page. Going into settings/actions and changing workflow permissions to allow writing, then adding a crates.io API token to the environment secrets means I can use this simplified release CI from tarpaulin: name: Release on: push: tags: - '[0-9]+.*' jobs: create-release: name: "Create GitHub release" # only publish from the origin repository if: github.repository_owner == 'xd009642' runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - uses: taiki-e/create-gh-release-action@v1 with: changelog: CHANGELOG.md env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} crates: runs-on: ubuntu-latest steps: - uses: actions/checkout@v3 - name: publish package to crates run: | cargo package cargo publish --token ${{ secrets.CARGO_TOKEN }} binaries: name: "Upload release binaries" needs: - create-release strategy: fail-fast: false matrix: include: - target: x86_64-unknown-linux-gnu os: ubuntu-latest runs-on: ${{ matrix.os }} steps: - uses: actions/checkout@v3 - uses: dtolnay/rust-toolchain@stable - uses: taiki-e/upload-rust-binary-action@v1 with: bin: ci-hoover target: ${{ matrix.target }} env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}</code></pre> Next up, Github actions marketplace things have an action.yml</code>, so I’ll just initially pinch the one from free-disk-space and change the running part to install ci-hoover with cargo-binstall and run it. I’ll skip most of the args for some brevity: name: "ci-hoover" description: "A configurable GitHub Action to free up disk space on an Ubuntu GitHub Actions runner. Inspired by free-disk-space." # See: https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#branding branding: icon: "trash-2" color: "orange" inputs: android: description: "Remove Android runtime" required: false default: "true" # More args runs: using: "composite" steps: - uses: cargo-bins/cargo-binstall@main run: | set -eu export ANDROID = ${{ inputs.android }} export DOT_NET = ${{ inputs.dot_net }} export HASKELL = ${{ inputs.haskell }} export LARGE_PACKAGES = ${{ inputs.large_packages }} export DOCKER_IMAGES = ${{ inputs.docker_images }} export TOOLS_CACHE = ${{ inputs.tools_cache }} export SWAP_STORAGE = ${{ inputs.swap_storage }} cargo binstall ci-hoover sudo ci-hoover</code></pre> I’m using envy to get the configuration from environment variables which seemed a bit nicer than doing things like --android true|false</code>. Time to publish to marketplace, but this then involves going into the release page generated and editing it to publish to the marketplace… I should fix that. But it doesn’t look like there’s a way in the action for making release pages and looking at some of the maintained ones I don’t see an option so I’ll just leave it for now. After all this may be my only release this is a bit of a shitpost. Bad Github UI alert, I do the draft release, edit the existing release notes. Then realising I have to accept the terms and conditions go onto that popup and accept. Then all my categories/tag stuff for the marketplace entry are wiped, c’mon Github be chill. Add a lil action just to try this action via the marketplace and it doesn’t work. Error: xd009642/ci-hoover/0.1.0/action.yml (Line: 51, Col: 7): Unexpected value 'run'</code></pre> hmm maybe before the run I need a - shell: bash</code>. Google seems to suggest that’s the syntax to use bash in an action. Gosh if only there was a way to test this which doesn’t involve creating a release page on github and publishing a pre-release… Well I’ll just draft a release and give it a tag that won’t trigger a cargo publish and then just force my way through it like I’m bludgeoning my face with a hammer repeatedly. Because that’s what CI is like. /home/runner/work/_temp/a5d46914-9fa4-4f1f-a508-7b131608f423.sh: line 2: export: `=': not a valid identifier</code></pre> Ah fuck spaces between equals in exports. I don’t know why I did that. If this was a normal tool I would have ran the equivalent bash script on my machine but this deletes a ton of stuff so I didn’t want to. Okay third times the charm right? Yeah third, it didn’t take six goes to do it… This is the final run part of the action: runs: using: "composite" steps: - uses: cargo-bins/cargo-binstall@main - shell: bash run: | set -eu export ANDROID=${{ inputs.android }} export DOT_NET=${{ inputs.dot_net }} export HASKELL=${{ inputs.haskell }} export LARGE_PACKAGES=${{ inputs.large_packages }} export DOCKER_IMAGES=${{ inputs.docker_images }} export TOOLS_CACHE=${{ inputs.tools_cache }} export SWAP_STORAGE=${{ inputs.swap_storage }} cargo binstall -y ci-hoover sudo mv $HOME/.cargo/bin/ci-hoover /bin sudo ci-hoover</code></pre> And this is the action which uses it in an image to clean it up: name: Marketplace on: push: branches: - "main" pull_request: jobs: linux: runs-on: ubuntu-latest steps: - name: ci-hoover uses: xd009642/ci-hoover@0.1.1</code></pre> And some output from ci-hoover: Name Total Used Available "/dev/root" 77.9 GB 55.5 GB 22.3 GB "/dev/sda15" 109.4 MB 6.3 MB 103.1 MB "/dev/sdb1" 78.7 GB 8.3 GB 70.3 GB Deleting things! Name Total Used Available "/dev/root" 77.9 GB 24.2 GB 53.7 GB "/dev/sda15" 109.4 MB 6.3 MB 103.1 MB "/dev/sdb1" 78.7 GB 4.0 GB 74.6 GB Finished in 63.809372 seconds</code></pre> I should really make those ascii tables look a bit nicer… But I have clawed back more than free-disk-space 53.7GB free instead of 41GB. All Done</a></h2> Not quite. For some reason when doing this from the marketplace it always takes about 1 minute. But when installing release in my CI and running it I get times of about 16s. Why is it slower as a Github marketplace action? Actually, no I’m tired and I’ll leave that as a question for the reader. Here’s the project if you want a look ci-hoover</a> as well as the marketplace page</a> Designing a Streaming Audio API 2024-10-20T00:00:00+00:00 This is part of a series on creating a streaming audio service in Rust, with services such as live transcription in mind. All the code is here</a> Our previous entry just initially introduced the system, now it’s time to look outside-in and see what decisions we have to make in terms of our API. Our system is a bidirectional streaming system - meaning that data is streamed both from client to server and server to client. This prevents us from doing a standard REST API so typically you’ll find people use one of two approaches: Websockets</li> gRPC</li> </ol> There’s also a third option of using raw http/2 streams which the AWS transcribe service does. But that definitely feels like a spicier option that’s more unfriendly to people not using any of the languages the official clients are provided in. There’s a fourth option of abusing WebRTC and I won’t say anymore about that. Any other options let me know, this feels fairly comprehensive. for the purpose of this we’ll start by looking at the first two, their pros and cons and then the act of designing the API. Websockets</a></h2> Websockets work pretty much everywhere which is definitely a plus in their favour. You establish a HTTP connection and then upgrade it to a raw TCP socket. This means you get all the pros and cons of raw TCP sockets Easy to use</li> Completely untyped</li> </ol> There’s also an extra negatives if you expect a browser based frontend to interact with your service. You can’t set HTTP headers when opening a websocket! This has implications if you expect to use the Authorization header to validate a user’s allowed to interact with your service. gRPC</a></h2> gRPC works over HTTP/2 meaning you can only use it where you have HTTP/2, which while it may be a lot of places it’s not everywhere. I’ve previously had issues with some cloud providers (Huawei). You also won’t be able to do bidirectional streaming from the browser so using things like gRPC web or one of the auto-proxy generation solutions to expose it to the browser won’t work. However, it does offer some form of typing for your API even if it is Protobuf. Additionally, there’s a gRPC ecosystem you can interact with which can give some benefits of it’s own. Which to Choose?</a></h2> So you don’t actually need to choose just one! You can choose to do both an multiplex your requests. Later on we’ll do this, but for now we’ll choose to do websockets just to keep things simpler and have our API usable from more contexts. Designing the API</a></h2> Now it’s time to think a bit about what we want to do and see what design falls out as a result. We already know we’re sending audio in but how are we sending it in? Audio files? Audio files are simple for the client which is definitely a plus. However, it’s complicated our end and not every file is streamable. If you’ve tried to stream audio files before you may have encountered an issue where there is trailer data at the end of the file which ffmpeg et al insist on reading before they’ll decode any samples. This then ends up removing the streaming part of our streaming service! Also, things like microphones don’t tend to output audio in a file. So what’s our next choice? PCM Data! Pulse Code Modulation (PCM) data is audio data in it’s simplest form. A list of samples (typically little endian), where channels are interleaved. We don’t need to know anything except for the number of channels, the bits per sample and the sample rate and we can perfectly reconstruct the audio file. This is also how Wave files work after their headers (which contain all of that data and some more). The only issue with PCM data is if people have something like an MP3 file they may struggle to use your API, but you can always offer a non-streaming API that handles file decoding if that becomes necessary. Now we could enforce to our users that they only provide data in our desired sample format, channel count and sample rate. But that will go down like a lead balloon and you’ll soon find yourself with product owners knocking down your door to demand a useable API. With this in mind lets add a starting message to our API. We can also use this to propagate other information i.e. any parameters the users can tweak, tracing information and any possible options on the response. Let’s start with something like: #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub struct StartMessage { /// Format information for the audio samples pub format: AudioFormat, } /// Describes the PCM samples coming in. I could have gone for an enum instead of bit_depth + /// is_float but I only really plan on doing f32, s16 and ignoring everything else including /// compression schemes like mulaw etc. This may change in future. #[derive(Copy, Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub struct AudioFormat { /// Number of channels in the audio pub channels: usize, /// Sample rate of the audio pub sample_rate: usize, /// Number of bits per sample pub bit_depth: u16, /// Whether audio uses floating point samples pub is_float: bool, }</code></pre> After, this is sent I’ll assume we’ll be sending the audio data itself. This will just be done via raw bytes because any audio streaming APIs that take base64 data in json are an afront to all that is holy. You have a TCP socket make use of it! When audio is coming in we’re going to be choosing segments from it to run our model on. This means we have some way of identifying the start and end of a segment. But when our client is done they may want any final inferences and if we haven’t detected the end of the segment they need a way to signal to us to stop. Then we can force an end of segment and send the final results back. So for this I define a stop message: #[derive(Clone, Debug, Default, Eq, PartialEq, Serialize, Deserialize)] pub struct StopMessage { /// Whether the server should close the websocket connection after returning final results. pub disconnect: bool, }</code></pre> And then to make deserializing the messages over websocket easier we’ll use a Rust enum like so: #[derive(Serialize, Deserialize)] #[serde(tag = "request", rename_all = "snake_case")] pub enum RequestMessage { Start(StartMessage), Stop(StopMessage), }</code></pre> So as a first pass this is the input stream: [start] [data]... [stop]</code></pre> Of course the client can also instantly disconnect if they don’t care about the final result. And after a stop we can potentially keep the connection open and send another start (though the server may want to timeout imdle connections like that). With the inputs initially decided, lets work out the outputs! The easiest possible output is just sending back the raw model outputs whatever, they are. Some model outputs don’t require any post-processing and for them that can work. However, it’ll only work if the user is able to work out the start and end times automatically based on the index of the message in the stream. An example of a service that could do that might be a voice activity detection service where each output is one frame of data and the frame size is known by the user. We’ll do a “simple” API in this project to cover that usecase. But, often segments may be different sizes and then it’s useful to know the start and end times of the segment - we’ll refer to this as the “segmemnted API”. We also, want a way to report back any errors. With these initial requirements I ended up settling on something like this: /// If we're processing segments of audio we want people to be able to apply the results to the /// actual segments! #[derive(Debug, Serialize, Deserialize)] pub struct SegmentOutput { /// Start time of the segment in seconds pub start_time: f32, /// End time of the segment in seconds pub end_time: f32, /// The output from our ML model #[serde(flatten)] pub output: model::Output, } #[derive(Debug, Serialize, Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] pub enum Event { Data(model::Output), Segment(SegmentOutput), Error{ message: String }, }</code></pre> Of course we could just use segment for both the simple and the segmented routes (and this may end up happening in the actual code). But this works reasonably well as an initial API with minimal complications. But Wait There’s More!</a></h2> Oh of course, there had to be more right! Well let’s get into it, the next subsections will be related to those facets of an API and I’ll show how our request/response times change due to it. Interim Results</a></h3> Looking at the segmented API, we may want our service to be reporting observations at regular intervals not just when a segment start and end have been identified. These are known as interim or partial results. You can see these in effect in live transcriptions on things like Google Meet when works change in the transcript. The transcription service will be returning output for every X seconds added onto the segment. Because of this words may be cut off prematurely, or a lack of context causes errors in the transcript which hopefully are fixed when the model runs on the entire segment. Not every user wants this, if you’re not displaying things to a user or trying to do speculative processing before the full segment is processed they’re not very useful. So we want it to be an opt-in. With this in mind the start message becomes: #[derive(Clone, Debug, Eq, PartialEq, Serialize, Deserialize)] pub struct StartMessage { /// Trace ID for distributed tracing pub trace_id: Option<String>, /// Format information for the audio samples pub format: AudioFormat, /// Whether interim results should be provided. An alternative API to this would be to specify /// the interval at which interim results are returned. #[serde(default)] pub interim_results: bool, }</code></pre> There is a potential option to allow the user to set the interval to return results at, but this is a potentially dangerous API decision. Partial inferences take time and doing them too frequently could cause your real-time system to become not real-time as it falls further and further behind. In which case you have to decide if you want to return an API error for too-small values or round them up to an acceptable value. Naturally, it’s now important for our results to indicate if they’re a final result that won’t change or if they’re an interim result that may be corrected. With this in mind our SegmentOutput</code> turns into: /// If we're processing segments of audio we want people to be able to apply the results to the /// actual segments! #[derive(Debug, Serialize, Deserialize)] pub struct SegmentOutput { /// Start time of the segment in seconds pub start_time: f32, /// End time of the segment in seconds pub end_time: f32, /// Some APIs may do the inverse check of "is_partial" where the last request in an utterance /// would be `false` #[serde(skip_serializing_if = "Option::is_none")] pub is_final: Option<bool>, /// The output from our ML model #[serde(flatten)] pub output: model::Output, }</code></pre>Events</a></h3> For the purpose of debugging or richer interactions you may want to expose events where some processing has started. A semi-common example I’ve seen in Speech To Text systems is an event when speech start and speech end are detected. This means that before we have any transcripts back we know the user is speaking and then can trigger some animation in the frontend or prepare for incoming data. These can also be quite useful for debugging. Did we get no output because our model returned nothing or because we haven’t detected any speech has started? For our events in future API we’ll just be implementing them for the segmented API and not the simple one. Adding in these events our output type becomes: #[derive(Debug, Serialize, Deserialize)] #[serde(tag = "type", rename_all = "snake_case")] pub enum Event { Data(model::Output), Segment(SegmentOutput), Error{ message: String }, Active { time: f32 }, Inactive { time: f32 }, }</code></pre>Conclusion</a></h2> From this we’ve looked at some of the API decisions and started to define some basic types for our API that are generally applicable to any bidirectional streaming audio APIs. In future installments we’ll actually start writing up some code to make this works and breaking apart the things which make it all come together. How to Build Streaming Audio APIs 2024-10-19T00:00:00+00:00 This is the start of a series about implementing bidirectional audio streaming APIs. You can find all the code here</a>. It is a truth universally acknowledged, that a single developer in possession of a good audio processing model, must be in want of a streaming API. Maybe not universally acknowledged but it’s definitely a common request at my place of work, both internally and externally. The existence of the ability to make some observation about audio often leads people to want to do this on a live audio stream. We can see this in live transcription in any meeting applications, existing video is easier we can process it offline. But live, dynamic and non-scripted audio require completely different solutions and trade-offs. In this project I’m not going to provide a model or a solution, those can be found elsewhere. What I will provide however is a general approach to create streaming services where audio is streamed in and observations streamed out. Initially, I’m just doing this with websockets, but in future gRPC may be considered. And maybe even SSE or HTTP/2 streams (who knows). Without further ado the system. The System</a></h2> Here I’ve generally split the system into 3 different concerns: The API received data and forwarding it to tasks, receiving results back and sending them back</li> Audio processing - extracting samples and transcoding to a format our model expects</li> The model itself - some simple segmentation and dealing with blocking inference calls</li> </ol> I’ve coloured these, and we can get a clue on how data flows through the system. Raw bytes and some API messages will come via the API input. The audio extraction with get the audio into a form our model can work with. Finally, we run it through the model, generate an API response and send it back. This is complicated by the fact that these tasks are happening concurrently! While we receive new audio we can be sending out prior responses. And streaming data has not been known to be simple to work with1 As I go deeper into different parts these will be explored at greater and greater depth as well as some common tropes of this genre of system. Well tropes at least as far as I’ve experienced, mainly Speech To Text (STT) systems - sometimes called Automatic Speech Recognition (ASR). This project makes a lot of use of the actor model and tokio channels because they are very convenient. There will be occasional performance tips and tricks, as well as testing and various parts of the ecosystem. I’ll try and fill in a bit of knowledge regarding the engineering concerns of such systems. What Currently Exists?</a></h2> Currently, I do have an actual system, with Voice Activity Detection (VAD) based segmentation. Some performance metrics, Opentelemetry traces and a dummy model that counts the number of samples with an thread::sleep</code> call added so we can pretend we’re a big slow Machine Learning (ML) model. I haven’t wrote about all these components, tested them or refined them so careful if you poke around the code because here be dragons. As a rough roadmap of some planned sections: Basic Introduction (this post)</li> API design</a></li> Audio Decoding and Resampling</a></li> The Model</a></li> Making the Axum API</a></li> Testing the system</li> Improving streaming performance via batching futures</li> Instrumented the service for production monitoring (traces, metrics etc)</li> </ol> If this sounds interesting keep and ear out more posts will drop in future at indeterminable times. Sign Languages, Diglossia and Other Diversions 2024-10-02T00:00:00+00:00 While working on speech/language technology for other languages sometimes I come across linguistic diversions which are endlessly fascinating. I also quite enjoy reading things where people investigate and learn about a topic unfamiliar to them and feel I should get into the habit of writing more things so here we go. I hope you enjoy reading a bit about sign language and related linguistics Disclaimer: I’ve never formally studied linguistics or any sign language so I may make mistakes. Feel free to reach out to me on any communication channel if you have some corrections or comments on what I’ve written! Initial Glances</a></h2> Sometimes customers have asked about automatically generating sign language animations for some text output of an NLP system as an accessibility feature instead of using subtitles. I have a feeling this is more complicated than simply mapping each word to it’s literal sign and playing them back in order. But this isn’t how spoken or signed communication works either. We have the whole range of expressiveness of body language, facial expressions and tone. It only makes sense that sign languages would take advantage of that to make communication simpler and more effective. And from an initial investigation this proves true. Sign languages often use spatial grammars, body language and expressions to communicate. We can see that with Arabic sign languages how the hand positions differ can be used to communicate whether one object is in front of or behind another[^1]. You don’t sign the literal sentence for a person being in front of a car, you do the person sign with a hand in front of a hand doing the car sign. There’s also one interesting definition mentioned in the source, that signed Arabic unlike spoken Arabic isn’t diglossic. And deaf speakers may not be aware of the different forms of Arabic that the non-Deaf community use. Note “not aware” seems to be because the study of Arabic Sign Languages and accommodating the Arabic Deaf community in education and general life seems to be a more recent effort. Diglossia</a></h2> Wait what is diglossia? Well a language is diglossic when there’s two forms of a language a High and Low form. So in Arabic the High form is Modern Standard Arabic, whereas Low forms are the local/regional dialects. This is kind of like register and it’s a bit confusing reading things online on how to make the judgement on when a difference is a change in register or a language being diglossic. Changing register of speech based on context is often called code-switching and we all do it sometimes e.g. when changing between formal and informal settings. I know I don’t speak in a job interview the same way I speak with friends in terms of language but that’s a register shift not English being diglossic. How can I mentally define diglossia then? Well here’s an attempt based on things I read here[^2]: Diglossia is when the gap between the two forms of the language are so wide they are both distinct varieties of the language in their own right. Often one form is used typically for writing and the other form is typically for speaking. </blockquote> This matches well with Arabic, from our work with the language the vast majority of text data is Modern Standard Arabic, whereas speech data you only tend to find it with some TV and Movie data not in casual speech. There are Arabic dialects closer to MSA than others but that’s a bit of a diversion. One thing I saw was some resources saying British Sign Language is diglossic and a suggestion that diglossic languages tend to have non-diglossic sign languages and vice versa. But reading further [^3] it seems this doesn’t hold up. The research that said BSL was diglossic seems flawed/outdated. Village Sign Languages</a></h2> In places where a large Deaf community have gathered either due to a concentration of Deaf people either schools or institutions, or a high prevalence of genetic deafness. There’s likely a number of Arabic Sign Languages which are village sign languages and most definitely undocumented. For a documented one I found out about Mardin Sign Language[^4]. A sign language with ~40 speakers currently, it evolved in a family with a lot of Deaf members and arose as a way for the family to communicate. A lot of resources I wanted to read on this were paywalled so I’ve not really got much other than they exist and I’m curious on how they’ve evolved and any examples where they’ve evolved beyond a village sign language to something more widely spoken. Is BSL an example of this? It seems to have evolved from the 15th Century to the present day with a clear Northern and Southern dialect. Concluding Thoughts</a></h2> So using your whole body to sign and combining all of that together to create meaning I have vague thoughts on detecting the timestamps of sign boundaries. Tokenising the human form and creating systems to aid in sign translation. Identifying when some motion or component is a modifier on a meaning and when it applies seems insanely difficult. I’m probably going to try and read up on what people in computer vision and machine translation communities are doing with sign language. As someone just standing at the entryway peeking curiously in it feels like encoding a sign language in a way it could be an output or input to computer translation systems you’ll encounter a complexity that makes unicode look simple. Also, we’ve seen LLMs are great at taking all written language and creating links and transformations between them via glorified pattern matching. Part of me wonders if similar unsupervised learning on the signed languages of the world with a visual model could lead to interesting insights or exploration of signed language. One thing is clear though. All languages are interesting to study and figure out a bit about how they work. I spent about 4 hours reading up on things which is barely any time and I’ll probably spend more time on it. I just wanted to capture some initial learnings. References</a></h2> Abdel-Fattah, Mahmoud. (2005). Arabic Sign Language: A Perspective. Journal of deaf studies and deaf education. 10. 212-21. 10.1093/deafed/eni007. </a></li> Preston So (2021). Register, diglossia, and why it’s important to distinguish spoken from written conversational interfaces</a></li> Linda Day (2000). British Sign Language in its Social Context</a></li> Ulrike Zeshan. Signing in a ‘deaf family’ – documentation of the Mardin Sign Language, Turkey</a></li> </ol> Snapshot Testing Neural Networks 2024-08-23T00:00:00+00:00 In this post, I’m mainly going to explain a bit about what snapshot testing is some pros and cons and how I approached it for an audio processing project at work. And testing a library that is a wrapper around a neural network with some bonus logic. What Are They?</a></h2> Snapshot testing is when we take a “snapshot” of the output or state of the program and store that alongside the tests. Then when running the test against our code, generate the same output and then compare that output with our previously stored version. You may have seen this used by the Rust compiler if you’ve ever poked into its diagnostic tests. In Rust there’s a handy crate for snapshot testing called insta</a>. Of course, sometimes our tests will fail, when that happens we will either confirm it’s an issue and fix our code or confirm the new snapshot is correct and replace the old one. Sometimes it’s a mixture of the two if we’ve partially broken something when adding a feature. The benefit of this approach is we can hit a large amount of our code with tests on realistic data/scenarios and do this much quicker than crafting test cases manually. A negative is that the more your outputs change or the harder snapshots are to review the greater tendency people might have to just blindly “bless” the new snapshots resulting in tests that aren’t helping you. Introducing the Project</a></h2> So at my work Emotech</a> I spend a lot of time working on audio AI systems. Often, we want to limit these models to just running on speech data and not any background noise or dead air. For that, we use some form of audio segmentation model such as a Voice Activity Detection (VAD) model. We recently open-sourced a Rust library around a popular one silero which you can find here</a>. For explaining all of this we do need to know a bit about how the VAD works internally. So buckle up! Inside the VAD we have a positive and a negative speech threshold. These function together as a hysteresis, so in some Rust-flavoured pseudo-code: let likelihood = vad.run(audio_frame); if last_state == State::Silence { if likelihood > positive_speech_threshold { last_state = State::Speaking; } } else { // We're speaking there're only two states if likelihood < negative_speech_threshold { last_state = State::Silence; } }</code></pre> We also don’t want to trigger too eagerly and catch a lot of vocal filler that can largely be regarded as noise. So there’s a minimum speech duration before we emit an event. But as well as that we don’t want to turn off too eagerly and cut someone talking off! So once we switch to silence we wait for a “redemption” duration and if no speech frames come in that time window we report the last timestamp as the first of those silent frames and emit another event. And if a speech frame comes in we reset the count. So our Rust flavoured pseudo-code will turn into something like: let likelihood = vad.run(audio_frame); if last_state == State::Silence { if likelihood > positive_speech_threshold { last_state = State::Speaking; speech_start = audio_frame.start_time; redemption = 0; is_speaking = false; } } else { // We're speaking there's only two states if likelihood < negative_speech_threshold { redemption += 1; if redemption > threshold { last_state = State::Silence; emit speech_end } } else { redemption = 0; if !is_speaking { duration = audio_frame.end_time.duration_since(speech_start); if duration > min_speech_duration { emit speech_start } } } }</code></pre> There’s some extra complexity with handling the vad frames vs samples vs timestamps, but this is pretty close to all the details! There is some mutable state that is tracked and updated based on more data coming in which is part of what makes testing so important. Also, we currently apply a pre-speech padding which would be an extra saturating subtraction when emitting the speech start event I haven’t shown. Snapshot Testing the VAD</a></h1> Part of making such a library is to test it and make sure it works well so we can have confidence throughout our stack. The model has been ran on a lot of benchmarking data (typically hours) and we have an idea of it’s accuracy. However, there are some nuances in using a VAD model and configuration which may not be tested by that approach. Things like: Different audio chunk sizes going in</li> Different padding values - the silence we allow at the start/end of the audio</li> Switching speed (sometimes you want to limit the max speaking/non-speaking transition)</li> Sensitivity</li> </ol> So an easy way to test this is to generate some test audio and then run all the audio through the VAD with different parameters and chunkings. And utilise snapshot testing to make sure they’re all as expected. Plus by running on representative data we can increase our confidence things work as expected! But because of this, while it’s useful we really can’t use insta. A text diff of our structures isn’t useful, long float arrays of durations and timestamps. These are things that you can’t review as text diffs. And with that, we lose a lot of insta’s DX improvements. Instead, there’s going to be a slightly different approach. Implement a test function that takes in the means to run the model</li> Run through our dataset with the given configuration</li> Generate and save a snapshot in the target directory</li> Load the reference snapshot from our test directory</li> For each file compare the snapshots</li> Highlight areas where they differ and provide a script to plot snapshots and help debug</li> </ol> Replacing snapshots will then be manually done by the user because the investigation might take deeper analysis the workflow is designed for less immediate responses. The snapshots I chose to save in JSON for some human readability and the Rust structs I save the data into are as follows: #[derive(Default, Debug, PartialEq, Deserialize, Serialize)] struct Summary { input_size_ms: usize, config: VadConfig, summary: BTreeMap<PathBuf, Report>, } #[derive(Default, Debug, PartialEq, Eq, Deserialize, Serialize)] struct Report { transitions: Vec<VadTransition>, current_silence_samples: Vec<usize>, current_speech_samples: Vec<usize>, }</code></pre> To explain some of the fields in the Report</code> type: transitions</code> - a vad transition is a speech start/end event emitted by the model</li> current_silence_samples</code> - number of silent frames going from the end of the audio to the last speech frame</li> current_speech_samples</code> - length of the last contiguous chunk of active speech</li> </ol> Current speech samples and current silence samples will grow at the same time until the redemption frame threshold is hit, then speech frames will jump back as the timestamp is corrected whereas silence will continue increasing. This is a bit of an interesting behaviour which isn’t always immediately obvious to people looking at the VAD debugging. In future we want an API for the neural network outputs so we can record them to see how they move above and below the positive and negative thresholds. The actual snapshot testing code is pretty simple: Run model on each audio updating the summary struct</li> Load snapshot</li> Check</li> </ol> The only thing that is slightly atypical is that I don’t just assert_eq</code> on the two Summary</code> objects and call it a day. Aside from the floating point parameters in the VadConfig</code> this doesn’t feel like the right approach. Namely, because the Vec<usize></code> fields can be hundreds of elements long and the debug printout is frankly not useful (though debug is derived in case someone wants to look at it). Instead our checking of the snapshots looks like the below: println!("Checking snapshot is generated with same configuration!"); compare_configs(&summary.config, &expected.config); assert_eq!(summary.input_size_ms, expected.input_size_ms); let mut failing_files = vec![]; println!(); for sample in summary.summary.keys() { let baseline = &summary.summary[sample]; let current = &expected.summary[sample]; if baseline != current { println!("{} is failing", sample.display()); if baseline.transitions != current.transitions { println!("\tDifference in transitions list!"); } if baseline.current_silence_samples != current.current_silence_samples { println!("\tDifference in silence lengths"); } if baseline.current_speech_samples != current.current_speech_samples { println!("\tDifference in speech lengths"); } failing_files.push(sample.to_path_buf()); } } if !failing_files.is_empty() { println!(); println!("You have some failing files and targets. If you get a snapshot file and audio you can plot it via our plot_audio script e.g."); println!(); println!( "python3 scripts/plot_audio.py -a {} -i {}", failing_files[0].display(), current_report.display() ); println!(); panic!("The following files are failing: {:?}", failing_files); }</code></pre> Notice that we print out a Python command you can run to debug the first file with a deviation in the results! After altering a snapshot to make it fail I got the following printout: running 6 tests test chunk_20_default_params_8k ... ok test chunk_20_default_params_16k ... FAILED test chunk_50_default_params_8k ... ok test chunk_50_default_params_16k ... ok test chunk_30_default_params_8k ... ok test chunk_30_default_params_16k ... ok failures: ---- chunk_20_default_params_16k stdout ---- Loading snapshot from: /home/xd009642/work/silero-rs/tests/data/chunk_20_16000Hz_default.json Checking current results target/chunk_20_16000Hz_default.json with snapshot /home/xd009642/work/silero-rs/tests/data/chunk_20_16000Hz_default.json Checking snapshot is generated with same configuration! tests/audio/rooster.wav is failing Difference in speech lengths tests/audio/sample_1.wav is failing Difference in transitions list! You have some failing files and targets. If you get a snapshot file and audio you can plot it via our plot_audio script e.g. python3 scripts/plot_audio.py -a tests/audio/rooster.wav -i target/chunk_20_16000Hz_default.json thread 'chunk_20_default_params_16k' panicked at tests/snapshot.rs:182:9: The following files are failing: ["tests/audio/rooster.wav", "tests/audio/sample_1.wav"] note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace failures: chunk_20_default_params_16k test result: FAILED. 5 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.92s error: test failed, to rerun pass `--test snapshot`</code></pre> And if I run the command: So in the top plot, we can see the audio wave and the shaded areas show where the VAD thinks speech is. The green dashed line shows where the VAD started with the pre-speech padding removed when we’re confident with the location (this is hard to tell at the start of the audio). The red line shows when the redemption threshold was hit and the end event was emitted. In the plot below we track the silence and speech samples with a dotted line for the redemption threshold. We can see the blue line intersecting the dotted line coincides with the redemption frame marker in the top plot as well as the speech frame length decreasing as the correct is applied. Extra post posting edit, this Monday I’ve updated it to include the network outputs into the snapshot tests! Fun Bonus Bug-fix</a></h2> While developing this and testing with chunk sizes below a VAD frame (30ms) I found that none of the samples triggered with 20ms chunk sizes. This is far smaller than anything we’ve tried in prod but there was a little issue with handling remainders that meant that all the samples were ignored. We also found a repeat processing issue on chunk sizes that aren’t a multiple of 30ms which led to the remainder being processed twice. You can see a bit of these in the PRs I’ll link down below: Snapshot Testing PR</a></li> Fixing remainders</a></li> </ol> Conclusion</a></h2> Snapshot testing does have its downsides. If you’re not vetting the snapshots, the data isn’t realistic or they’re treated with a lack of care they could make a project look more mature than it is. If they’re brittle and change often the overhead on developers to push forwards simple changes can be too painful. However, for hard-to-test code particularly code where tests can help drive the development and debugging process more interactively they can prove to be a powerful tool to complement your other testing methods. I hope this post has proven somewhat enlightening and I did enjoy getting to prototype and merge in an idea I’ve been playing around in my head for a while. As a final note on more general snapshot testing of neural networks or machine learning algorithms. Often you will be porting from Python to Rust, so collecting snapshots from the Python code and then using them to test the Rust port is a good approach to be confident in your ported implementation. Thanks to blueridanus for working with me on this and others for proofreading and offering feedback on initial versions of this post. On Coverage and Macros 2024-07-08T00:00:00+00:00 On Coverage and Macros</a></h1> As a maintainer of a code coverage tool</a> one of the common issues I get is that the coverage statistics are incorrect. Users may gate CI on coverage percentages to try and ensure they hit a level of testing and false-negatives can be seen to hamper this. Often there’s some validity to this based on how dead code has been eliminated or code has been optimised sometimes coverage instrumentation is missed and there are false negatives. However, there are cases when it’s correct that the code isn’t covered. Take the following test, what should the coverage be? fn status_check() { println!("All good"); } #[test] fn is_good() { status_check(); }</code></pre> I think we can all agree this should be 100%. But what about this? fn status_check() { tracing::trace!("All good"); } #[test] fn is_good() { status_check(); }</code></pre> Ignoring the lines of test code we have two unambiguously coverable lines - the function signature and the macro call. And with this in mind coverage should be 50%. Why? Maybe this code which was exercised by tests but can cause a segfault shows why: let input = self .format_context .streams() .best(media::Type::Audio) .ok_or(TranscodingError::NoAudio)?; let mut audio_decoder = input .decoder() .map_err(|e| { error!("Unable to create decoder: {}", e); TranscodingError::Unknown("Unable to create decoder".to_string()) })? .audio() .map_err(|_| TranscodingError::NoAudio)?; trace!("Layout: {:?}", audio_decoder.channel_layout()); </code></pre> The answer is that the macro expands into a bunch of code which will semantically be something like (not this exactly): if tracing::level() == tracing::Level::Trace { println!("TRACE: Layout: {:?}", audio_decoder.channel_layout()); }</code></pre> And in that branch there is code with a segfault. Marking the trace macros as covered when they’re not being tested does give a false sense of security. Then when we actually need trace logging to debug an issue the segfault pops up adding to some of the confusion. The Solution</a></h2> Fortunately, there is an answer to this, I use the tracing-test</code></a> crate in projects which also gives me a big log dump when a test fails which is more useful to me than just swallowing all those traces. It also has the side-effect of the tracing calls being covered. Of course, analytically examples like the first one could be removed from the false negatives. But the more assumptions you make when filtering results the more you risk coverage becoming deceptive and not guiding you in any useful way. For what it’s worth my approach is to use coverage as a signal of where I’m maybe not testing adequately. I don’t aim for high coverage numbers and I use it in combination with tools like cargo-mutants</a> to ensure my testing strategy is sound. Every metric becomes useless when it’s a target. The relevant GitHub issue that inspired this short post is here</a> Povception 2023-11-30T00:00:00+00:00 This is a rehosting of a blog originally on www.rs-online.com where our team won a £1,500 prize for the below project. As it’s now been taken offline I’m preserving it here! We also won a single desk calendar to share between 3 people at our employer This project was created by a three-man team comprising of two hardware engineers and a software engineer. From the very beginning, we knew we wanted to do something unique and challenging for a Christmas competition. The idea of trying to get a persistence of vision globe inside another was proposed and after finding no existing implementations of the idea we began the project. We already had some of the necessary materials available when we started the project, including an STM32F469i discovery board and motors borrowed from a quadcopter. We also had tools and software available to aid in the design and manufacture of the POVception. These included a laser cutter, metal lathe, 3D printers, MathWorks Simulink, SolidWorks and a Root3 CNC for milling wood and etching a PCB. When designing the mechanical assembly, the need for a front area where the user would interact and have an unobstructed view of the globes was required. The globes could not be mounted from the bottom alone due to the size. Therefore, transparent frames would be at the back and curve up and mount to the top of the globes to keep it stable and allow people to still view it from the back. Another initial decision was for the rings to rotate in opposite directions meaning two motors were required. There wasn’t an engineering reason for this choice, we felt that it would look impressive when it started spinning up with the LEDs on. With the assembly modelled we started with milling the baseboards and the rings out of 18mm thick softwood plywood. We decided to paint these dark grey to accentuate the lights. We designed the mounts for the boards, motors and power supply for 3D printing and printed them in ABS using 1.75mm RS Pro filament. Red was chosen to keep with the Christmas theme and add a nice accent colour to the finished design. For the transparent frames, we laser cut 3mm acrylic using two arms for each frame for some added rigidity. Mild steel was used for all the metal work on the project. This included the axles mounted to the rings and stand-offs for the two baseboards that sandwiched the electronics. To allow the inner axle to nicely pass through the outer a metal lathe was used to change the bore. For the stand-offs, the lathe was used to cut the rods to size and add a thread to them. The final part of the assembly was the two game controllers, each one would have a single joystick to enable the users to play basic games like Pong. To keep things simple and save time we utilised one of the AdaFruit 2 axis thumb joystick breakout boards and 3D printed a case for it we found on Thingiverse. Two existing cables with DE-9 connectors were used with one end of the cable removed and the wires soldered to the breakout board and the other end remaining intact to plug into the POVception. Initially, when thinking about the software, we had concerns with latency and resolution of the globes. The LED density of the LED strips determines the vertical resolution and the speed the LEDs can be changed combined with the motor speed determines the horizontal resolution. For the LED strip solution, we initially investigated using AdaFruit NeoPixel strips, they were very affordable but had some flaws. Namely, the strips weren’t clocked and had strict timing requirements. Because of this, part of the feasibility was whether the timing could be met with enough slack to perform other software tasks. If timing wasn’t met the image would end up corrupted and there would be no guarantees about what was displayed. Another issue with the NeoPixels was the spacing between the LEDs would restrict the vertical resolution. After some experimentation, the NeoPixels were found unsuitable in terms of throughput, and the SPI driven DotStar LED strips were chosen instead. The DotStar LED strips had an LED density of 144 LEDs per metre and as they were driven by SPI the software could asynchronously drive them via DMA transfers. Looking online we could find no upper limit to the clock speed for the DotStars and after experimentation found they could be driven at the SPIs maximum speed of 10MHz with no issue. Both sides of each ring were also rendered and with only two SPI lines on the board we had to run the strips from the bottom of one ring going upwards then from the top of the opposite face running down. The DotStar SPI also ran at 5V compared to the STM32 running at 3.3V, so to prevent signal integrity issues an additional PCB was designed and etched to drive the LEDs as well as provide power to the motors. On the software side, it was important the microcontroller didn’t spend time on blocking calls. The STM32 was responsible for: motor control for two motors, updating the display frames and rendering the two rings. Additionally, the discovery board had a touchscreen display featuring a simple GUI which had to be polled and updated accordingly. The majority of the code was done in C++ with the STM32 HAL and some ARM Mbed API functions used for interacting with the hardware. MathWorks Simulink was utilised in the creation of the project. Given previous experience with Simulink in control systems, it was decided that it would be used for the motor control with the main C++ code base providing an interface between Simulink and the hardware. Using Simulink for control is a natural fit, however, integrating the generated code into a project involved reading the code to find the names for functions and structs you need to use. It would have been preferable and easier to write the code in C++ like the rest of the project – but then we wouldn’t have met all the requirements. With the hardware and assembly done and software being integrated and refined in the final week some interesting traits were observed. When tracking which column was being rendered initially a modulus operator was used to ensure the index never went out of bounds. However, modulus is quite a slow operator so this would cause the last column to appear wider than the others and cause the display to jitter left and right. Replacing the modulus with an if statement to check the bounds and a subtract to move back into the range removed this effect. Additionally, adding new applications and icons for them in the GUI took up a lot of flash memory. The touchscreen HAL library supports bitmaps but only if they were stored in flash, with the POVception finished around 90% of the flash memory was being used by a mixture of code and resources. The finished POVception had the following applications: Pong – the most important a classic Pong game. Pong on the inner ring, snowfall effect on outer</li> Baubles – show a bauble pattern on both rings</li> Calibration – use the joysticks and set the first column of the displays</li> Globe – showed the globe on the inner ring with the snowfall effect on the outer ring</li> </ul> Of course, no project is ever perfect and there are some obvious things we would have changed and improved – had we had time. The first thing that comes to mind is that the HAL library for the touchscreen display only supported one landscape orientation. We didn’t realise this until after it was fitted and I had to flip all the bitmaps 180 degrees and remove any text printed to the screen. Also, there were some sporadic software bugs that were discovered for the first time on the demonstration day. In terms of the physical side, the DotStar strips got very hot and began to slightly peel off so better thermal management or a better adhesive would have improved the design. Overall, the project was a success and the team was awarded an innovation prize. We’re all proud of what we managed to achieve in such a small amount of time and the project met all its aims. The project is largely open-source with the code currently available on GitHub (link at the end), eventually, the mechanical and hardware designs will also be uploaded to either GitHub or Thingiverse. Here’s the code link</a> Here’s the finished project in action </iframe> Tarpaulin and the Futures 2019-10-02T00:00:00+00:00 Tarpaulin</a> (or cargo-tarpaulin) is a code coverage tool for Rust, and anyone who’s used it might know that until recently it had an issue with code that used futures. This post is going to go into that issue a bit and the eventual fix. For anyone who wants the deeper story there’s also the github issue</a>. This issue has also existed since January and has had a lot of work so if I skirt over anything or don’t provide nice logs it’s genuinely because they’ve been lost in the annals of time and my organisational skills. A bit of background</a></h2> So how does Tarpaulin work? Simply it builds your project with a few special flags to ensure it can get the necessary information to instrument the binaries. Then for each binary it will get the lines it can instrument, filter out some lines based on analysis of the source code and add some extra lines for things other coverage tools miss i.e. unused generic code. It then forks, launches the test via execve</code> so the test starts with the child PID and uses ptrace</code> to add breakpoints to the code and uses waitpid</code> and ptrace</code> to follow what the binary is doing and collect the relevant coverage statistics. After hitting a breakpoint Tarpaulin would count the line, disable the breakpoint so the program counter can progress, single step and then re-enable the breakpoint. This was so we could get the hit-rate for a line. This has all sorts of issues involved, when a test has multiple threads each thread can be in different places. And sometimes they can be in the same place (which is where some of the initial issues started). My very first fix was setting --test-threads 1</code> when the test executables were launched by Tarpaulin. The first threading issue found was much older than this one but is the same general problem. If a thread stops at a breakpoint we know it’s set so we can disable it, count the hit, single step the thread and re-enable the breakpoint so we can get the number of line hits. If two threads hit the same breakpoint at the same time this caused issues initially. Especially, if the next instruction was also instrumented. The main reason why was because when you send a continue, or a step command via ptrace, all the threads in the process react to that signal. Even if you specify which TID you want to move on! The first version of tarpaulin didn’t consider how other threads were causing the instructions to change and would frequently freeze or stall as a result. This was fixed and not counting line hits was also made default to minimise other threading quirks and speed up runs. But there was something else lurking. The issue being fixed</a></h2> So an issue is a lot easier when you have a minimal reproducible example, and for the issue we’re tackling there was this one: #[test] pub fn a() { futures::executor::ThreadPool::new(); } #[test] pub fn b() { futures::executor::ThreadPool::new(); }</code></pre> If we ran this Tarpaulin was guaranteed to report a segfault from the test binary. After spending a while stumped on this I added a --debug</code> flag to tarpaulin that gives any user very detailed logs on all the signals received and Tarpaulin’s response. From this I realised something, if all the threads are stopped and step causes them to all step and execute what they’re doing. If we have a MOV instruction as below with it’s opcode: MOV r/m8,r8 : 88 /r</code></pre> The Opcode would turn into CC /r</code> and as we have to move the program counter back one byte reenable the instruction and step to execute it to execute a valid instruction there’s a potential issue. Any thread that is also stopped on a breakpoint will be 1 byte ahead of it and will execute a potentially illformed instruction. Either way this is definitely undefined behaviour. From this realisation the solution seemed simple, at the start of the Tarpaulin update loop call waitpid</code> as many times as it will deliver signals and create a signal queue. Then for every element of the queue collect the coverage data, update the breakpoints and push a recommended action onto an action list. Another foible with ptrace is that when you get a signal from a PID you need to respond to it in the order you get it. So then I go through the action queue carrying out the recommended action for every signal. This was implemented (src/statemachine/mod.rs</code> and src/statemachine/linux.rs</code> for interested parties). I ran it on the test program 10 times and saw no errors and then ran it on a real world project that had the issue and got the expected coverage. So that’s it fixed then? Wrong. Users started reporting the issue persisted. After running for 100 times I found this occurred 10% of the time on my home machine. Changes in rustc</a></h2> So here’s the point where things slowed down considerably. I was stumped and also I was in the process of changing job and moving so I had a lot on my plate. Additionally, debugging from the logs was tedious and I often found myself drawing long sprawling graphs with the lifetime of a program. To try and speed up my debugging process I took the code for the update loop and removed everything else. Used a toml file to specify addresses to instrument and a path to a binary and made minitarp</a> to generate graphs showing the lifetime of the test. This also let me take the instrumentation points Tarpaulin derived as a starting point and tweak them removing some, adding others from reading the disassembly or DWARF dumps of a program. And here is our failing minimal example (but with 4 tests doing the same thing not 2). Open all images in a new tab, they’re 4K resolution to fit all the information Here the bottom line is the tests PID, all the IDs above are for new threads spawned. When a thread ends in <TID>: EXITED 0</code> it’s exited successfully, otherwise it’s failed. The colours were default from gnuplot so don’t attach any meaning to them. We can also see the printouts show any signals tarpaulin received, actions it received and if it stopped at a breakpoint the breakpoint address. We can see from this image the 8 threads we expect our first ThreadPool</code> to create were created and exited correctly. But then the second test occurs and we catch the threads being created but none of them exit. Some of them we also don’t catch the creation event, others we only catch creation and nothing afterwards. Eventually, after all 8 threads are created we end up getting our segfault and the whole thing comes to a halt. Why is this? Spoiler alert it’s because of --test-threads 1</code>, and I discovered it completely by accident. I thought removing the test thread limit would exacerbate the problem further and help me figure out why I couldn’t see some threads come into existence. I’d noticed from the test PID line that the first line of the second test was hit before the threads all exited - eagle eyed readers can see this point around sample 92 on the timeline. This is the time to note TryContinue</code> as an action exists for two reasons. Sometimes after a thread exits you can get a signal from it which is no longer valid. It’s best to respond to it to keep everything balanced but if it fails it’s not an issue. The other instance is for moments when Tarpaulin has reached a state I didn’t expect so tries to continue to see if it’s a blip. But none of this explains why --test-threads 1</code> would cause this! The answer is naturally in libtest in the rust compiler. if concurrency == 1 { while !remaining.is_empty() { let test = remaining.pop().unwrap(); callback(TeWait(test.desc.clone()))?; run_test(opts, !opts.run_tests, test, run_strategy, tx.clone(), Concurrent::No); let (test, result, exec_time, stdout) = rx.recv().unwrap(); callback(TeResult(test, result, exec_time, stdout))?; } } else { while pending > 0 || !remaining.is_empty() { while pending < concurrency && !remaining.is_empty() { let test = remaining.pop().unwrap(); let timeout = Instant::now() + Duration::from_secs(TEST_WARN_TIMEOUT_S); running_tests.insert(test.desc.clone(), timeout); callback(TeWait(test.desc.clone()))?; //here no pad run_test(opts, !opts.run_tests, test, run_strategy, tx.clone(), Concurrent::Yes); pending += 1; } // More code handling timeouts etc }</code></pre> This code in libtest/lib.rs</code> at line 1201 (existing as of f2023ac). Is different from when I first read libtest to figure out some issues. When I first looked the if concurrency == 1</code> didn’t exist instead the logic in the else</code> statement was used for all the instances. There’s also code further down that launches tests in a subcommand or in-process. I haven’t quite narrowed down the exact reasoning behind why it doesn’t work so if anyone has any insight I’d appreciate it! To avoid an anti-climatic ending I’ll close with the news that Tarpaulin now seems to work perfectly with projects using futures and the removal of --test-threads 1</code> should see a speedup for all projects. I’ll also add a picture of the graph minitarp generated once the issue was fixed: Tarpaulin Past Present Future 2018-06-11T00:00:00+00:00 Tarpaulin</a> (or cargo-tarpaulin) is a code coverage tool for Rust. Last year was pretty busy with the launch of the project and the rush of issues as people started to use it so this is just a chance to look at what’s new with version 0.6.0 and what’s planned for the rest of this year. 2017 A year in review</a></h2> So in 2017 I started tarpaulin, largely because I realised it’s probably the best project name I’ve came up with, but mainly because I found the process of using kcov with Rust frustrating. Language agnostic coverage tools tend to only work properly with C, struggle with abstractions, and setup is fiddly. Whereas building a coverage tool targeted at a specific language means you can utilise tooling in that language to make things easier - Cargo for example. So after my initial May release tarpaulin can now: Send coverage reports to popular coverage sites (coveralls and codecov)</li> Generate cobertural xml reports</li> Include unused templated code in results (kcov doesn’t do this for Rust or C++)</li> Handle unused inlined functions</li> Trim out a lot of false positives that show up such as module imports and using</code> aliases</li> Include or exclude packages</li> Ignore certain tests</li> Work on multithreaded code (with the --no-count</code> option, now a default)</li> Tarpaulin ran on Windows via docker and new releases are now available on docker-hub</a></li> </ul> syntex_syntax to syn</a></h3> As syntex_syntax is unmaintained the introduction of nested import statements wasn’t able to be handled and caused a panic. Because of this tarpaulin had to move to syn for syntax analysis. It also had to make use of semver exempt code to get the source positions from syn. Because of this you may need RUSTFLAGS="--cfg procmacro2_semver_exempt"</code> when running cargo install</code> for tarpaulin 0.6.0. So just type the following: RUSTFLAGS="--cfg procmacro2_semver_exempt" cargo install cargo-tarpaulin</code> Another alternative is to use the install script to download the binary from the github releases for travis or one of the docker images available</a>. 2018 What’s planned</a></h2> So in 2018 there’s a few different features and improvements planned but I’ll focus on the big ones. HTML Reports</a></h3> Tarpaulin will have it’s own HTML report format it can generate for people who aren’t using codecov or coveralls as a web interface to their coverage results. These reports will be able to show coverage at a folder level, individual file level and also let you inspect the source and see it annotated with the results. Branch and condition coverage</a></h3> So a couple of open source tools provide branch coverage, none provide condition coverage. I’m aiming straight for a technical solution that works for condition coverage, this is harder than just branch coverage but has some benefits. It means tarpaulin will be capable of branch coverage and open the door to Modified Condition Decision Coverage (MCDC) something I’ve only seen in expensive closed source tools. MCDC coverage is significantly more useful as a metric and mandated in some safety critical software projects. Currently, I’m deciding between attempting to tackle all three at once or start with branch coverage. Because of the need to analyse boolean subconditions for condition and MCDC coverage they’re harder to implement than branch coverage. However, implementing a system that can provide condition coverage can provide branch and MCDC without extra complexity in the tracing and interpretation of the programs. There may also be a blog post in the future going into the different types of coverage and explaining the pros and cons of them! Performance</a></h3> Instrumenting code with breakpoints and stepping through logging the coverage is obviously a slow process. There are a number of flags in tarpaulin to mitigate the performance issues and by using syn I’ve aimed to remove unnecessary breakpoints to speed up runs. In 2018 I’d like to improve the performance further, potentially by running test executables in parallel or simpler means. Other Improvements</a></h2> As the year goes on I’d like to continue to close issues improving accuracy and consistency across projects of different types and different distros. And a lot of that work will be by closing issues. One issue I’d really like to see closed is #23</a>, although that might be reliant on rust#35061</a>. Likewise, #13</a> coverage of doctests will be another tricky one to close. Community contributions</a></h2> In 2018 the aim is to remove any barriers to entry for people interested in contributing and I’m going to try and start having mentored issues among other initiatives. I’ve already made some significant headway into this by refactoring the code to abstract away some of the details of ptrace and adding more tests and docs. This work will continue as the year goes on. Any help is appreciated and with that in mind I’d like to thank people who have contributed this year. So thanks to: Robinst</li> Andy128k</li> llogiq</li> tafia</li> rep-nop</li> vberger</li> alex-mckenna</li> quodlibetor</li> yodaldevoid</li> mathstuf</li> philipc</li> bbigras</li> jean553</li> apeduru</li> mgeisler</li> </ul> Introducing Tarpaulin 2017-07-20T00:00:00+00:00 Code coverage</a></h1> When people talk about code coverage, they’re referring to metrics which show how much of their source code is “covered” by their tests. Now covering the code is of course only the first step, the tests actually have to test the functionality in a meaningful way, but coverage is a good way of finding areas in your code which you haven’t tested as thoroughly. So how do we define coverage? Well there are numerous metrics, Wikipedia lists these methods as the simplest ways of measuring coverage. Function coverage - how many functions have been executed?</li> Statement (line) coverage - how many statements have been executed?</li> Branch coverage - how many of the possible paths have been executed?</li> Condition coverage - how many boolean subconditions have been executed?</li> </ul> Currently, open source coverage tools (bcov, gcov, kcov) only offer statement coverage. They also require some effort to execute depending on your language and project setup. This is the motivation for tarpaulin, a code coverage tool created specifically for rust as a cargo subcommand. Why tarpaulin?</a></h1> Tarpaulin</a> uses cargo as a library meaning it can automatically identify files in your project. It can also generate and identify the test executables and run coverage on all of them. Compare this to kcov where you need to clean the project area, build the test executables then iterate over the executables in target/debug running kcov on each one and merging the reports at the end. Additionally, kcov will include extern crate definitions and module statements (pub mod foo</code>) in it’s coverage results. Tarpaulin is designed for rust and aims to remove lines which are “uncoverable” from the coverage results. Below is some example output for tarpaulin ran on one of the example projects. cargo tarpaulin Launching test running 1 test test tests::bad_test ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured Coverage Results src/lib.rs: 7/8 src/unused.rs: 0/4 58.33% coverage, 7/12 lines covered</code></pre> Currently, tarpaulin implements line coverage and has coveralls.io</a> integration. It is linux only and only designed with x86_64 support (so 64 bit AMD and Intel processors). Wider support is planned for future expansion, but this should work for the majority of users. By focusing on Rust only and it’s build system tarpaulin can potentially provide better results by being able to extend information derived from the test binaries with information from cargo and the source code itself. By having access to crates like syntex</a>, tarpaulin aims to become the first open source code coverage tool to offer a solution for condition coverage. That’s all for this relatively short introduction, check out the github repository</a> for more information as well as the roadmap! If you find any issues please raise them and I should get back to you fairly sharpish. Hello world! 2017-06-29T00:00:00+00:00 Well here is the obligatory hello world post. My current plan is using this blog to talk about my code coverage tool for the rust language cargo-tarpaulin</a>.

Tarpaulins Week of Speed

Announcing Wiremocket: Wiremock for Websockets

Why Not PR Wiremock?</a></h1>
There’s an issue</a> for this in wiremock opened by a colleague. I’m not saying this won’t go into wiremock eventually in some form, but it is a big change and the two domains dissimilar enough the overlap could confuse existing users.</p>

Streaming Audio APIs: the Axum Server

Adding a New Fake to the Fake Crate

2024 in Review

Streaming Audio APIs: the Model

Conclusion</a></h2>
Well here we are, we’ve got a fake model and inferencing code to call it for our planned APIs. Now we need to cover the Axum API we’ll make and then we’ll have a fully working system and can start to dive into more depth on various parts of the system.</p>

Streaming Audio APIs: Audio Decoding

Debloating Github Actions with Rust

Designing a Streaming Audio API

How to Build Streaming Audio APIs

Sign Languages, Diglossia and Other Diversions

Snapshot Testing Neural Networks

On Coverage and Macros

Povception

Tarpaulin and the Futures

Tarpaulin Past Present Future

Introducing Tarpaulin

Hello world!

xd009642

My Experience as a Rice Farmer

Automatically Running Cargo on Workspace Changes

2025 in Review

Ideas of Crater and Coverage

Making CLI Tools with Trustfall

Tarpaulins Week of Speed pt2

xd009642

My Experience as a Rice Farmer

Automatically Running Cargo on Workspace Changes

Integrating the Current Pieces</a></h1> Given the root of the project, we can now:</p> Get the changes</li>

2025 in Review

Ideas of Crater and Coverage

Making CLI Tools with Trustfall

Tarpaulins Week of Speed pt2

Tarpaulins Week of Speed

The Preamble</a></h1> Tarpaulin is a code coverage tool, and as part of coverage it offers two backends:</p> ptrace - UNIX process tracing API, like a debugger. Not fully accurate but more flexible on where you can get coverage data</li>

Announcing Wiremocket: Wiremock for Websockets

What’s So Difficult?</a></h1> Streaming. Streaming is always difficult.</p> Given a server we may want to match on information available when the connection is established such as:</p> HTTP headers</li> The URL path</li>

Streaming Audio APIs: the Axum Server

Adding a New Fake to the Fake Crate

2024 in Review

Streaming Audio APIs: the Model

Streaming Audio APIs: Audio Decoding

Debloating Github Actions with Rust

Designing a Streaming Audio API

Websockets</a></h2> Websockets work pretty much everywhere which is definitely a plus in their favour. You establish a HTTP connection and then upgrade it to a raw TCP socket. This means you get all the pros and cons of raw TCP sockets</p> Easy to use</li>

How to Build Streaming Audio APIs

The System</a></h2> </p> Here I’ve generally split the system into 3 different concerns:</p> The API received data and forwarding it to tasks, receiving results back and sending them back</li> Audio processing - extracting samples and transcoding to a format our model expects</li>

Sign Languages, Diglossia and Other Diversions

Snapshot Testing Neural Networks

On Coverage and Macros

Povception

Tarpaulin and the Futures

Tarpaulin Past Present Future

Introducing Tarpaulin

Hello world!