⚡ TL;DR

The Default trait can enhance the maintainability of your code. Default values for common types are listed at the end.

A PR Review

Recently, while reviewing a PR¹, I noticed that part of the patch was introducing a new field to a struct:

 1diff --git a/src/lib.rs b/src/lib.rs
 2index eba9a3a..8619e06 100644
 3--- a/src/lib.rs
 4+++ b/src/lib.rs
 5@@ -106,8 +108,9 @@ use std::{
 6 #[derive(Debug, PartialEq, Clone)]
 7 pub struct M<'u> {
 8     up: &'u str,
 9     down: Option<&'u str>,
10+    foreign_key_check: bool,
11 }
12 
13 impl<'u> M<'u> {
14@@ -137,8 +140,9 @@ impl<'u> M<'u> {
15     pub const fn up(sql: &'u str) -> Self {
16         Self {
17             up: sql,
18             down: None,
19+            foreign_key_check: false,
20         }
21     }

That prompted me to reflect on the code I had initially written. Prior to the patch, it looked roughly² like this:

 1#[derive(Debug, PartialEq, Clone)]
 2pub struct M<'u> {
 3    up: &'u str,
 4    down: Option<&'u str>,
 5}
 6
 7impl<'u> M<'u> {
 8    pub const fn up(sql: &'u str) -> Self {
 9        Self {
10            up: sql,
11            down: None,
12        }
13    }
14}

The Default Trait

What if I had used the Default trait here? The code could have looked like this:

 1#[derive(Debug, Default, PartialEq, Clone)]
 2pub struct M<'u> {
 3    up: &'u str,
 4    down: Option<&'u str>,
 5}
 6
 7impl<'u> M<'u> {
 8    pub const fn up(sql: &'u str) -> Self {
 9        Self {
10            up: sql,
11            ..Default::default()
12        }
13    }
14}

On the first line, the #[derive(Default)] attribute makes the structure M implement the Default trait. Thanks to this trait, a call to M::default() will create a struct with default values for its fields: M { up: "", down: None }. Note that when a structure M is expected, Default::default() is equivalent to M::default().

We then need to initialize the two fields of that structure, overriding some defaults:

• up is defined directly as before. That’s the value we want to override.
• down is set by the Default trait. This is done by ..Default::default() on line 11. Default::default() provides the values. Then the .. syntax fills out the fields that were not directly set. down is thus set to the same value as before, None.

The code is just as long as before, when we were not using the Default trait. But then, line 19 of the above patch would have been unnecessary: false is the default for a bool, so the new foreign_key_check field would have been covered by the ..Default::default().

This results in a shorter patch:

 1diff --git a/src/lib.rs b/src/lib.rs
 2index eba9a3a..8619e06 100644
 3--- a/src/lib.rs
 4+++ b/src/lib.rs
 5@@ -106,8 +108,9 @@ use std::{
 6 #[derive(Debug, PartialEq, Clone)]
 7 pub struct M<'u> {
 8     up: &'u str,
 9     down: Option<&'u str>,
10+    foreign_key_check: bool,
11 }
12 
13 impl<'u> M<'u> {

Conclusion

In the example of this post, we are doing only one instantiation of that particular struct, and it has very few fields anyway. But if there were many instantiations of that struct, we would have had to change all of those. Then, using Default would have been quite beneficial.

This pattern were the return type is built with a call to Default::default() seems relatively common in the Rust organization. It can enhance maintainability, much more than I initially thought.

Of course this is a balancing act. For instance, this pattern could be abused by defining custom default values on primitive types for a particular structure. That would lead to Default::default() filling surprising values and the code would be less predictable.

Appendix: Defaults for Some Common Types With derive

Why did I not use the Default trait initially? Part of it might be the fear of introducing incorrect code. It was slightly unclear to me what derive uses as a default value for common primitive types. And you don’t want a field set explicitly to false becoming a true once you use Default, right?

Let’s take a closer look by running the following program:

#[derive(Debug, Default)]
struct D<'a> {
    b: bool,
    c: char,
    o: Option<usize>,

    string: String,
    str: &'a str,

    v: Vec<usize>,
    a: [usize; 10],
    s: &'a [u32],

    f: f64,
    u: (usize, u32, u64)
}

fn main() {
    println!("{:#?}", D::default());
}

Output:

D {
    b: false,
    c: '\0',
    o: None,
    string: "",
    str: "",
    v: [],
    a: [
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
        0,
    ],
    s: [],
    f: 0.0,
    u: (
        0,
        0,
        0,
    ),
}

It turns out that in general, the default for common types in the standard library is a 0 byte in the underlying data structure. Note that arrays are fixed size in rust and thus the default is an array of the right size, filled with defaults for the inner type. That’s quite similar to go.

Quick Reference

Here is a table for future reference:

⚡ TL;DR

In the Linux Git repository:

hyperfine --export-markdown /tmp/tldr.md --warmup 10 'git ls-files' 'find' 'fd --no-ignore'

git ls-files is more than 5 times faster than both fd --no-ignore and find!

Introduction

In my editor I changed my mapping to open files from fd¹ to git ls-files² and I noticed it felt faster after the change. But that’s intriguing, given fd’s goal to be very fast. Git on the other hand is primarily a source code management system (SCM), it’s main business³ is not to help you list your files! Let’s run some benchmarks to make sure.

Benchmarks

Is git ls-files actually faster than fd or is that just an illusion? In our benchmark, we will use:

• fd 8.2.1
• git 2.33.0
• find 4.8.0
• hyperfine 1.11.0

We run the benchmarks with disk-cache filled, we are not measuring the cold cache case. That’s because in your editor, you may use the commands mentioned multiple times and would benefit from cache. The results are similar for an in memory repo, which confirms cache filling.

Also, you work on those files, so they should be in cache to a degree. We also make sure to be on a quiet PC, with CPU power-saving deactivated. Furthermore, the CPU has 8 cores with hyper-threading, so fd uses 8 threads. Last but not least, unless otherwise noted, the files in the repo are only the ones committed, for instance, no build artifacts are present.

A Test Git Repository

We first need a Git repository. I’ve chosen to clone⁴ the Linux kernel repo because it is a fairly big one and a reference for Git performance measurements. This is important to ensure searches take a non-trivial amount of time: as hyperfine rightfully points out, short run times (less than 5 ms) are more difficult to accurately compare.

git clone --depth 1 --recursive ssh://git@github.com/torvalds/linux.git ~/ghq/github.com/torvalds/linux
cd ~/ghq/github.com/torvalds/linux

Choosing the commands

We want to evaluate git ls-files versus fd and find. However, getting exactly the same list of file is not a trivial task:

After some more tries, it turns out that this command gives exactly⁵ the same output as git ls-files:

fd --no-ignore --hidden --exclude .git --type file --type symlink

It is a fairly complicated command, with various criteria on the files to print and that could translate to an unfair advantage to git ls-files. Consequently, we will also use the simpler examples in the table above.

Hyperfine

Hyperfine is a great tool to compare various commands: it has a colored and markdown output, attempts to detect outliers, tunes the number of run… Here is an asciinema⁶ showing its output⁷:

First Results

For our first benchmark, on an SSD with btrfs, with commit ad347abe4a… checked out, we run:

hyperfine --export-markdown /tmp/1.md --warmup 10 'git ls-files' \
    'find' 'fd --no-ignore' 'fd --no-ignore --hidden' 'fd' \
    'fd --no-ignore --hidden --exclude .git --type file --type symlink'

This yields the following results:

As mentioned in the TL;DR, git ls-files is at least 5 times faster than its closest competitor! Let’s find out why that is.

How Does Git Store Files in a Repository

To try to understand where this performance advantage of git ls-files comes from, let’s look into how files are stored in a repository. This is a quick overview, you can find more details about Git’s storage internals in this section of the Pro Git book.

Git Objects

Git builds its own internal representation of the file system tree in the repository:

In the figure above, each tree object contains a list of folder or names and references to these (among other things). This representation is then stored by its hash in the .git folder, like so:

.git/objects
├── 65
│  └── 107a3367b67e7a50788f575f73f70a1e61c1df
├── e6
│  └── 9de29bb2d1d6434b8b29ae775ad8c2e48c5391
├── f0
│  └── f1a67ce36d6d87e09ea711c62e88b135b60411
├── info
└── pack

As a result, to list the content of a folder, it seems Git has to access the corresponding tree object, stored in a file contained in a folder with the beginning of the hash. But doing that for the currently checked out files all the time would be slow, especially for frequently used commands like git status. Fortunately, git also maintains an index for files in the current working directory.

Git Index

This index, lists (among other things) each file in the repository with file-system metadata like last modification time. More details and examples are provided here.

So, it seems that the index has everything ls-files requires. Let’s check it is used by ls-files

Strace

Let’s ensure that ls-files uses only the index, without scanning many files in the repo or the .git folder. That would explain its performance advantage, as reading a file is cheaper than traversing many folders. To this end, we’ll use strace⁸ like so:

strace -e !write git ls-files>/dev/null 2>/tmp/a

It turns out the .git/index is read:

openat(AT_FDCWD, ".git/index", O_RDONLY) = 3

And we are not reading objects in the .git folder or files in the repository. A quick check of Git’s source code confirms this. We now have an explanation for the speed git ls-files displays in our benchmarks!

Other Scenarios

However, listing file in a fully committed repository is not the most common case when you work on your code: as you make changes, a larger portion of the files are changed or added. How does git ls-files compare in these other scenarios?

With Changes

When there are changes to some files, we shouldn’t see any significant performance difference: the index is still usable directly to get the names of the files in the repository, we don’t really care about whether their content changed.

To check this, let’s change all the C files in the kernel sources (using some fish shell scripting):

for f in (fd -e c)
  echo 1 >> $f
end

git status | wc -l
28350

hyperfine --export-markdown /tmp/2.md --warmup 10 'git ls-files' 'find' 'fd --no-ignore' \
  'fd --no-ignore --hidden --exclude .git --type file --type symlink'

We see the same numbers as before and it is again consistent with ls-files source code.

Run git checkout -f @ after this to remove the changes made to the files.

With New Files and -o

With yet uncommitted files, there are two subcases:

• files were created and added (with git add): then the files are in index and reading the index is enough for ls-files, like above,
• files were created but not added: these files are not present in the index, but without the -o flag, ls-files won’t output them either, so it can still use the index, as before.

So the only case that needs further investigations is the use of -o. Since we don’t have baseline results yet for -o, let’s first see how it compares without any unadded new files.

Without any Unadded New Files (Baseline)

When we haven’t added any new files in the repository:

hyperfine --export-markdown /tmp/3.md --warmup 10 'git ls-files' 'git ls-files -o' 'find' \
  'fd --no-ignore' 'fd --no-ignore --hidden --exclude .git --type file --type symlink'

That suggests that git ls-files -o is performing some more work besides “just” reading the index. With strace, we see lines like:

strace -e !write git ls-files -o>/dev/null 2>/tmp/a
…
openat(AT_FDCWD, "Documentation/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=1446, ...}, AT_EMPTY_PATH) = 0
getdents64(4, 0x55df0a6e6890 /* 99 entries */, 32768) = 3032
…

With Unadded New Files

Let’s add some files now:

for f in (seq 1 1000)
  touch $f
end

And compare with our baseline:

hyperfine --export-markdown /tmp/4.md --warmup 10 'git ls-files' 'git ls-files -o' 'find' \
  'fd --no-ignore' 'fd --no-ignore --hidden --exclude .git --type file --type symlink'

There is little to no statically significant difference to our baseline, which highlights that much of the time is spent on things relatively independent of the number of files processed. It’s also worth noting that there is relatively little speed difference between git ls-files -o and fd --no-ignore --hidden --exclude .git --type file --type symlink.

Using strace, we can establish that all commands but git ls-files were reading all files in the repository. By comparing the strace outputs of git ls-files -o and fd --no-ignore --hidden --exclude .git --type file --type symlink (the two commands that print the same file list), we can see that they make similar system calls for each file in the repository. How to explain the (small) time difference between the two? I haven’t found convincing reasons in git source code for this case. It might be that the use of the index gives ls-files a head start.

Conclusions

I’m now using git ls-files in my keyboard driven text editor instead of fd or find. It is faster, although the perceived difference described in the Introduction is probably due to spikes in latency on a cold cache. The selection of files is also narrowed down with ls-files to the ones I care about. That’s said, I’ve still kept the fd-based file listing as a fallback, as sometimes I’m not in a Git repository.

After all, Git is already building an index, why not use it to speed up your jumping from file to file!

 cljoly/readme-in-static-site

This fast script allows you to insert your GitHub README in your static site and apply transformations. For instance, you can read this README on GitHub and on my website.

Why?

The GitHub README of your repo needs to efficiently describe your project to GitHub’s visitor. But featuring your project on your website allows you to (among other things):

• have more control on the theme and layout,
• insert scripts that GitHub would prohibit (like asciinema),
• have your project’s homepage independent of your hosting platform, if you wish to change at some point.

Chances are that for small projects, the page about your project is very similar to the GitHub README. Don’t duplicate efforts, describe the differences! This has been a long-awaited feature, in one form or another.

See this blog post for more details.

Testimonials

RISS made it to the first page of HackerNews and Lobsters and got comments like:

southerntofu

hannu

lifthrasiir

Run It (Nothing to Install)

To try it with Hugo or Zola, run the following in your static-site sources:

wget https://cj.rs/riss.awk
awk -f riss.awk /path/to/my-project/README.md > content/my-project.md

If you don’t use Hugo or Zola, no problem! It should also work with any markdown-based static-site generator. Just put the markdown file where it makes sense for that generator.

To automatically update these files in your static-site sources, see Automate with GitHub Actions below. Since RISS is based on Awk, there is nothing to install, besides copying the script itself!

Example

Add a Front Matter

Most static site generators require a “front matter” at the beginning of a markdown file to attach some metadata. But you don’t want to add this to your GitHub README! Let’s hide this on GitHub and have it in the script’s output.

In you .md file on GitHub, put:

<!-- insert
---
title: "README In Static Site (RISS)"
date: 2021-08-21T10:15:54
---
end_insert -->
<!-- Powered by https://cj.rs/riss -->
<!-- remove -->

# README In Static Site (RISS)
<!-- end_remove -->

The output of the script will be:

---
title: "README In Static Site (RISS)"
date: 2021-08-21T10:15:54
---
<!-- Powered by https://cj.rs/riss -->

and this piece of yaml will be hidden on GitHub!

Replace Asciinema Image

You can’t embed the asciinema player on GitHub for security reasons. So the asciinema documentation suggests using an image there and to link it to a webpage with the player. But on your own website, you can embed this player.

In your .md file, put:

<!-- remove -->
[![Finding the repositories with “telescope” in their name, with the README in the panel on the right](https://asciinema.org/a/427156.svg)](https://asciinema.org/a/427156)
<!-- end_remove -->
<!-- insert
<asciinema-player src="./telescope.cast" poster="npt:0:04"></asciinema-player>
end_insert -->

The output will contain only the asciinema player:

<asciinema-player src="./telescope.cast" poster="npt:0:04"></asciinema-player>

Note: you also need to add the JS/CSS files of the asciinema player somewhere in your theme. This Hugo module makes it easy.

More

See the input file (typically on GitHub) and the output of the script. You can find another real word README converted to a webpage (this gives another example of asciinema conversion using a Hugo shortcode).

With some shell scripting, you could also transform all the markdown files in your repo and put them in a subdirectory of your site, so that your project’s documentation, policy, etc… lives on your site or even on a site of its own.

Your Example!

Have you used this script to transform some markdown (or other) and insert it on your website? Open an issue if you would like a link to your use case from this README!

• telescope-repo.nvim: readme, website; features an Asciinema clip.
• neovide: readme, first iteration of their website, PR setting this up. They have now moved to mdbook and that’s great! RISS makes the first iteration of your website easy and you are free to move to more complete solutions when your project grows.
• Hugo APT Repository: readme, website, PR setting it up.

Transformations Reference

The transformations are driven by HTML comments, so that you can have different results when comments are ignored (e.g. in your GitHub README) and after executing the script on your markdown file.

Escaping

It is important that your comment starts at the beginning of the line: spaces are used for escaping, meaning that if the comment has spaces at the beginning of the line, it is ignored.

So this is escaped

    <!-- insert

but this is not

<!-- insert

Insertion

Use these two lines for text you want to have in your output, but not in the raw .md file.

<!-- insert
end_insert -->

Remove

Use these two comments for text you want to have in your raw .md file, but not in the output

<!-- remove -->
<!-- end_remove -->

Spread the Word

If you find this script useful, please consider inserting the following in your readme:

<!-- Powered by https://cj.rs/riss -->

This will help other people find the script. Thanks for spreading the word!

If you feel especially charitable, you could put this badge somewhere:

with for instance this code:

[![](https://img.shields.io/badge/powered%20by-riss-lightgrey)](https://cj.rs/riss)

Breaking API Changes

We follow semver and any change that change would cause real world READMEs to be converted differently requires a new major version. In particular, the following is a breaking change:

• adding new keywords (like remove or insert), as they may be used in the README prior to their introduction in RISS,
• changing a keywords syntax.

Benchmark

Processes 17600 lines in 10 ms

$ for i in {1..100}; do shuf README.md >>bench.md; done # Create a big md file
$ wc -l README.md
176 README.md
$ wc -l bench.md
17600 bench.md
$ hyperfine 'awk -f riss.awk README.md' 'awk -f riss.awk bench.md'

Automate with GitHub Actions

You can automatically update the markdown file in the sources of your site with GitHub Actions. Add this workflow to, for instance, .github/workflows/readme.yml:

name: Update README files

on:
  schedule:
    - cron: '30 */2 * * *'
  push:
    branches:
    - master
  # To run this workflow manually from GitHub GUI
  workflow_dispatch:

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Check out the repo
      uses: actions/checkout@v2
    - name: Get the latest READMEs
      run: make readme-update
    - name: Commit and push if there are changes
      run: |-
        git diff
        git config --global user.email "bot@example.com"
        git config --global user.name "bot"
        git diff --quiet || (git add -u && git commit -m "Update READMEs")
        git push

and then your Makefile may contain something like:

readme-update:
	curl https://raw.githubusercontent.com/cljoly/readme-in-static-site/main/README.md | awk -f riss.awk >content/readme-in-static-site.md

Alternatively, you might configure your repositories to trigger a website rebuild when committing on your README, for instance using GitHub actions webhooks.

Contributions are Welcome!

Feel free to open an issue to discuss something or to send a PR.

See also the Spread the Word section if you would like to make more folks aware of this script.