• How we built Network Analytics v2

Archived copies:

• WaybackMachine
• Archive.is
• Perma.cc

⚡ TL;DR

As a small start-up time optimization, you can pick the best suited compression algorithm for the initial ramdisk.

The Initial Ramdisk

When a Linux system boots, it needs to mount the root filesystem /. This may be relatively complicated, as it may be on a software RAID, on LVM, encrypted… To keep things manageable, an initial ramdisk can be used to get a small environment that has all the required modules and configuration to load the root filesystem. On Arch Linux, this initial ramdisk is generated using mkinitcpio. It takes multiple parameters to tune various aspects of the system and of the generated ramdisk.

Compression

One such parameter is COMPRESSION. It compresses the ramdisk to make the resulting image smaller. The manpage reads:

Another reason to compress the image is that it may reduce the start-up time. To understand why, imagine that the image is 100 MiB in size and only 20 MiB after compression. Let’s say that the disk reads 10 MiB¹ per second and that the CPU can decompress the full image in 1 second. If we keep the image uncompressed, the disk will need 10 seconds to read the uncompressed image, while it needs only 2 seconds to read the compressed image. Adding the decompression time, the compressed version require only 3 seconds.

Trade-offs

The above example is quite simple, but it illustrates the trade-off between a bigger image that the disk will take longer to read and a smaller image that may take longer to decompress. It is thus more of a spectrum, where more CPU-intensive compression (and decompression) methods could result in a smaller image and less read from the disk but more CPU time:

Then, the question is: is it worth compressing an image more (or at all), to get a faster start-up time?

Protocol

To answer this question on a particular machine², let’s compare the time required to read and decompress various initial ramdisks.

I’m using the linux package in version 5.18.15-arch1-2 from the Arch Linux repository. Then, I generate (sudo mkinitcpio -p linux) various images with the following parameters in /etc/mkinitcpio.conf:

• COMPRESSION="cat"
• COMPRESSION="lz4"
• COMPRESSION="zstd"

Each image is copied in a directory and renamed according to the compression used: cp /boot/initramfs-linux.img initramfs-linux.img.zstd. The result is as follows:

$ file *.img*
initramfs-linux.img:      ASCII cpio archive (SVR4 with no CRC)
initramfs-linux.img.lz4:  LZ4 compressed data (v0.1-v0.9)
initramfs-linux.img.zstd: Zstandard compressed data (v0.8+), Dictionary ID: None

The images are compressing quite well too:

We could compare more algorithms and compression level, but compression levels would need to be passed through COMPRESSION_OPTIONS, which the manpage discourages, as it can result in an unbootable image.

Results

Let’s run some decompression commands and compare their run-time with hyperfine. On a quiet computer:

$ hyperfine \
    --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' \
    'lz4 -d <./initramfs-linux.img.lz4' \
    'zstd -d <initramfs-linux.img.zstd' \
    'cat <initramfs-linux.img'

Note that the command has a --prepare 'sync; echo 3 | sudo tee /proc/sys/vm/drop_caches' argument. This empties the OS file system caches to be closer to start-up conditions: when the computer starts, everything has to be read from the disk as the RAM is basically empty. Without this --prepare argument, we get much shorter times, e.g. 45ms for lz4.

Here are the results:

Lz4 is slightly faster, followed by zstd and no compression at all with cat. If we go back to the sizes table, the trade-off between a smaller image but a slower decompression is clear. Despite a ~30% smaller file size, zstd is still a bit slower to decompress than lz4, while no compression at all is even worse.

Conclusion

The above results are based on runs on a particular machine. As mentioned different machines will yield different results, depending on the relative performance of the disk and the CPU. It’s also a pretty small improvement in the grand scheme of things: only a few tens of milliseconds on a process that takes a couple seconds. But I found it to be a nice example of how compression can make things faster, compared to no compression at all, because CPU nowadays are so fast.

Appendix: Recording of the hyperfine Run

This was done on a different run from the table above, as running the benchmark through Asciinema is sometimes a bit less stable):

⚡ TL;DR

LuaSnip is fast and doesn’t have to be complicated. Give it a try!

Introduction

Snippets are a convenient feature of some text editors to insert and adapt reusable pieces of code. For instance, snippets for for loops are common, to get the tedious bits of the syntax out of the way.

To get this feature in Vim back in the days, I started using UltiSnips. There are default snippets sets for it, and it’s easy to write custom snippets. These custom snippet can call Bash or Python scripts if you need more dynamic replacements. UltiSnips has been very powerful and has served me quite well over the past decade or so, and I have kept it when I migrated to NeoVim a few years ago.

Startup time

Every once in a while though, I run the excellent vim-startuptime command to assess the impact of various configurations and plugins on the startup time of Neovim. With UltiSnips and the corresponding completion plugins in my configuration, the first few lines are:

Extra options: []
Measured: 10 times

Total Average: 104.218300 msec
Total Max:     109.719000 msec
Total Min:     99.533000 msec

  AVERAGE       MAX       MIN
------------------------------
64.218600 70.419000 61.066000: ~/.config/nvim/init.lua
 6.255900  7.238000  5.764000: loading packages
 3.939000  5.338000  3.533000: ~/.local/share/nvim/site/pack/paqs/start/onedark.nvim/colors/onedark.lua
 3.651100  4.003000  3.381000: loading rtp plugins
 2.761600  3.957000  2.474000: expanding arguments
 2.683900  3.035000  2.566000: reading ShaDa
 2.418700  3.610000  2.131000: sourcing vimrc file(s)
 2.389600  2.751000  2.228000: /usr/share/nvim/runtime/filetype.lua
 1.954900  2.293000  1.702000: loading after plugins
 1.842500  2.015000  1.763000: BufEnter autocommands
 1.583350  3.532000  0.018000: ~/.local/share/nvim/site/pack/paqs/start/ultisnips/plugin/UltiSnips.vim
 1.027700  1.170000  0.831000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter/plugin/nvim-treesitter.lua
 0.968200  1.071000  0.860000: ~/.local/share/nvim/site/pack/paqs/start/cmp-nvim-ultisnips/after/plugin/cmp_nvim_ultisnips.lua
 0.859000  1.014000  0.702000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter-textobjects/plugin/nvim-treesitter-textobjects.vim
 0.590700  0.674000  0.524000: ~/.local/share/nvim/site/pack/paqs/start/indent-blankline.nvim/plugin/indent_blankline.vim
 0.557200  0.817000  0.493000: init highlight
 0.553200  0.898000  0.322000: opening buffers

The init.lua line covers a lot of different plugins and mappings set up in that file. Besides the onedark.nvim colorscheme¹, the next biggest contributor is ~/…/ultisnips/plugin/UltiSnips.vim. The script to integrate UltiSnips with cmp, ~/…/cmp-nvim-ultisnips/after/plugin/cmp_nvim_ultisnips.lua, is not far behind. In total, we spend nearly 4 ms of the startup time for UltiSnips-related files, on top of the setup done in init.lua. That feels suboptimal, so I’ve been looking for a possible snippet plugin alternative.

Installing LuaSnip

LuaSnip aims to be a faster² snippet engine, with support of treesitter in the snippets. It can also understand the LSP snippet “format”. Finally, it’s a pure Lua plugin, without any Python requirement, contrary to UltiSnips. That makes installation on various system much easier.

Its drawbacks for me are that it does not support the same snippet definition as UltiSnips and even in its own SnipMate-like syntax, backticks to execute code are not supported. As a result, I’ll have to migrate my (admittedly very small) snippet collection. Conversely, if I want to do more advanced things, I’ll have to learn the relatively complex “VS Studio Code” snippets in JSON or even the pure Lua snippets.

That’s said, I believe I can get the LuaSnip benefits without writing Lua snippets or JSON ones, at least to start. So let’s try and do that, using only the SnipMate-like syntax!

Overview of the Configuration Changes

I’ve made the following changes.

1. Remove UltiSnips and its associated plugins, namely for completion with cmp and the telescope integration.
2. Remove the UltiSnips mappings.
3. Add LuaSnip (following the instructions in the Readme), its cmp integration, a set of snippets and a telescope integration along with some mappings (requires nvim 0.7+³):

   vim.keymap.set({ "i", "s" }, "<C-i>", function() require'luasnip'.jump(1) end, { desc = "LuaSnip forward jump" })
   vim.keymap.se({ "i", "s" }, "<M-i>", function() require'luasnip'.jump(-1) end, { desc = "LuaSnip backward jump" })
   

4. Migrate my existing snippets, see below.
5. Add code to load the snippet set and my own snippet collection:

   require("luasnip.loaders.from_vscode").lazy_load()
   require("luasnip.loaders.from_snipmate").lazy_load({ paths = {"./snippets"} })
   

After

Let’s run vim-startuptime again. Here are the new top contributors:

Extra options: []
Measured: 10 times

Total Average: 98.251400 msec
Total Max:     100.159000 msec
Total Min:     96.519000 msec

  AVERAGE       MAX       MIN
------------------------------
60.469100 62.341000 59.619000: ~/.config/nvim/init.lua
 5.991100  6.219000  5.865000: loading packages
 4.088400  4.162000  3.942000: loading after plugins
 3.635700  3.752000  3.415000: loading rtp plugins
 3.561800  3.936000  3.220000: ~/.local/share/nvim/site/pack/paqs/start/onedark.nvim/colors/onedark.lua
 2.550500  2.628000  2.492000: reading ShaDa
 2.454200  2.525000  2.401000: expanding arguments
 2.363000  2.420000  2.270000: /usr/share/nvim/runtime/filetype.lua
 2.271100  2.337000  2.146000: sourcing vimrc file(s)
 1.701000  1.767000  1.635000: BufEnter autocommands
 1.676700  2.049000  1.366000: opening buffers
 1.034100  1.280000  0.782000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter/plugin/nvim-treesitter.lua
 0.901000  1.021000  0.799000: ~/.local/share/nvim/site/pack/paqs/start/nvim-treesitter-textobjects/plugin/nvim-treesitter-textobjects.vim
 0.590700  0.674000  0.524000: ~/.local/share/nvim/site/pack/paqs/start/indent-blankline.nvim/plugin/indent_blankline.vim
 0.549500  0.789000  0.514000: init highlight

The snippet infrastructure is now absent from that list! More importantly, the overall startup time is down, and we can be confident that the new calls to load the snippets in init.lua are not costlier than UltiSnips settings before, because the init.lua line is down as well.

As a bonus, the snippet expansion feels slightly snappier with LuaSnip, although it might be an illusion and I don’t have hard numbers to back this claim.

Migrate my Snippet Collection

Back to the topic of migrating existing UltiSnips snippets: LuaSnip will loudly complain when given UltiSnips snippets but it’s relatively easy to rewrite those snippets to the SnipMate-like format that LuaSnip understands.

Two Syntaxes

On the one hand, UltiSnips snippets roughly follow this syntax

snippet trigger "Comment" option
snippet content
endsnippet

And so, my markdown snippets would look like this:

priority 10

snippet bjtoday "“Bullet Journal”-styled date for today" b
# `date +'%F %A'`
endsnippet

On the other hand, LuaSnip uses a simplified version of SnipMate snippets:

# Comment
snippet toggle
  snippet content

Simple Snippets

Since I heavily rely on snippet sets, I have only about 30 snippets defined in my own snippet collection. Most of them are really simple, with only a few lines and almost no interactive text. So for most of those, the process has been to simply remove the priority … lines and then a simple substitution command⁴ did the trick.

Advanced Snippets With Environment Variables

However, there is also the case of the snippets calling external commands, like date in the example below:

snippet bjtoday "“Bullet Journal”-styled date for today" b
# `date +'%F %A'`
endsnippet

The problem is, to call arbitrary commands, one needs to define the snippets in Lua.

It turns out though that nearly all my snippets calling external command were actually inserting a date. And luckily LuaSnip defines “environment variables” holding just the values I need like $CURRENT_MONTH. So that snippet becomes:

# “Bullet Journal”-styled date for today
snippet bjtoday
  # ${CURRENT_YEAR}-${CURRENT_MONTH}-${CURRENT_DATE} $CURRENT_DAY_NAME_SHORT

and we get to keep the simple SnipMate-like syntax!

You can find more of those environment variables in the sources.

What’s Next

I now have a slightly faster NeoVim and the snippet syntax that I use the most is simpler. However, there is more to explore!

So far, I’ve steered clear of writing more complicated JSON and Lua snippets. The latter would be necessary to unlock smart snippets using treesitter context. I’ll look into that next, in particular to generate go error handling code.

This is also a very simple configuration, the impact on startup time of LuaSnip might go up slightly as we use more advanced feature, but during my tests, even using all available features, it was much lighter than UltiSnips.

Resources

• Many more details are covered in LuaSnip documentation.
• The SnipMate help contains the SnipMate snippet syntax, if you are unfamiliar with it.
• The LSP snippet documentation is also helpful.

⚡ TL;DR

In the Linux Git repository:

hyperfine --export-markdown /tmp/tldr.md --warmup 10 'git ls-files' 'find' 'fd --no-ignore'

git ls-files is more than 5 times faster than both fd --no-ignore and find!

Introduction

In my editor I changed my mapping to open files from fd¹ to git ls-files² and I noticed it felt faster after the change. But that’s intriguing, given fd’s goal to be very fast. Git on the other hand is primarily a source code management system (SCM), it’s main business³ is not to help you list your files! Let’s run some benchmarks to make sure.

Benchmarks

Is git ls-files actually faster than fd or is that just an illusion? In our benchmark, we will use:

• fd 8.2.1
• git 2.33.0
• find 4.8.0
• hyperfine 1.11.0

We run the benchmarks with disk-cache filled, we are not measuring the cold cache case. That’s because in your editor, you may use the commands mentioned multiple times and would benefit from cache. The results are similar for an in memory repo, which confirms cache filling.

Also, you work on those files, so they should be in cache to a degree. We also make sure to be on a quiet PC, with CPU power-saving deactivated. Furthermore, the CPU has 8 cores with hyper-threading, so fd uses 8 threads. Last but not least, unless otherwise noted, the files in the repo are only the ones committed, for instance, no build artifacts are present.

A Test Git Repository

We first need a Git repository. I’ve chosen to clone⁴ the Linux kernel repo because it is a fairly big one and a reference for Git performance measurements. This is important to ensure searches take a non-trivial amount of time: as hyperfine rightfully points out, short run times (less than 5 ms) are more difficult to accurately compare.

git clone --depth 1 --recursive ssh://git@github.com/torvalds/linux.git ~/ghq/github.com/torvalds/linux
cd ~/ghq/github.com/torvalds/linux

Choosing the commands

We want to evaluate git ls-files versus fd and find. However, getting exactly the same list of file is not a trivial task:

After some more tries, it turns out that this command gives exactly⁵ the same output as git ls-files:

fd --no-ignore --hidden --exclude .git --type file --type symlink

It is a fairly complicated command, with various criteria on the files to print and that could translate to an unfair advantage to git ls-files. Consequently, we will also use the simpler examples in the table above.

Hyperfine

Hyperfine is a great tool to compare various commands: it has a colored and markdown output, attempts to detect outliers, tunes the number of run… Here is an asciinema⁶ showing its output⁷:

First Results

For our first benchmark, on an SSD with btrfs, with commit ad347abe4a… checked out, we run:

hyperfine --export-markdown /tmp/1.md --warmup 10 'git ls-files' \
    'find' 'fd --no-ignore' 'fd --no-ignore --hidden' 'fd' \
    'fd --no-ignore --hidden --exclude .git --type file --type symlink'

This yields the following results:

As mentioned in the TL;DR, git ls-files is at least 5 times faster than its closest competitor! Let’s find out why that is.

How Does Git Store Files in a Repository

To try to understand where this performance advantage of git ls-files comes from, let’s look into how files are stored in a repository. This is a quick overview, you can find more details about Git’s storage internals in this section of the Pro Git book.

Git Objects

Git builds its own internal representation of the file system tree in the repository:

In the figure above, each tree object contains a list of folder or names and references to these (among other things). This representation is then stored by its hash in the .git folder, like so:

.git/objects
├── 65
│  └── 107a3367b67e7a50788f575f73f70a1e61c1df
├── e6
│  └── 9de29bb2d1d6434b8b29ae775ad8c2e48c5391
├── f0
│  └── f1a67ce36d6d87e09ea711c62e88b135b60411
├── info
└── pack

As a result, to list the content of a folder, it seems Git has to access the corresponding tree object, stored in a file contained in a folder with the beginning of the hash. But doing that for the currently checked out files all the time would be slow, especially for frequently used commands like git status. Fortunately, git also maintains an index for files in the current working directory.

Git Index

This index, lists (among other things) each file in the repository with file-system metadata like last modification time. More details and examples are provided here.

So, it seems that the index has everything ls-files requires. Let’s check it is used by ls-files

Strace

Let’s ensure that ls-files uses only the index, without scanning many files in the repo or the .git folder. That would explain its performance advantage, as reading a file is cheaper than traversing many folders. To this end, we’ll use strace⁸ like so:

strace -e !write git ls-files>/dev/null 2>/tmp/a

It turns out the .git/index is read:

openat(AT_FDCWD, ".git/index", O_RDONLY) = 3

And we are not reading objects in the .git folder or files in the repository. A quick check of Git’s source code confirms this. We now have an explanation for the speed git ls-files displays in our benchmarks!

Other Scenarios

However, listing file in a fully committed repository is not the most common case when you work on your code: as you make changes, a larger portion of the files are changed or added. How does git ls-files compare in these other scenarios?

With Changes

When there are changes to some files, we shouldn’t see any significant performance difference: the index is still usable directly to get the names of the files in the repository, we don’t really care about whether their content changed.

To check this, let’s change all the C files in the kernel sources (using some fish shell scripting):

for f in (fd -e c)
  echo 1 >> $f
end

git status | wc -l
28350

hyperfine --export-markdown /tmp/2.md --warmup 10 'git ls-files' 'find' 'fd --no-ignore' \
  'fd --no-ignore --hidden --exclude .git --type file --type symlink'

We see the same numbers as before and it is again consistent with ls-files source code.

Run git checkout -f @ after this to remove the changes made to the files.

With New Files and -o

With yet uncommitted files, there are two subcases:

• files were created and added (with git add): then the files are in index and reading the index is enough for ls-files, like above,
• files were created but not added: these files are not present in the index, but without the -o flag, ls-files won’t output them either, so it can still use the index, as before.

So the only case that needs further investigations is the use of -o. Since we don’t have baseline results yet for -o, let’s first see how it compares without any unadded new files.

Without any Unadded New Files (Baseline)

When we haven’t added any new files in the repository:

hyperfine --export-markdown /tmp/3.md --warmup 10 'git ls-files' 'git ls-files -o' 'find' \
  'fd --no-ignore' 'fd --no-ignore --hidden --exclude .git --type file --type symlink'

That suggests that git ls-files -o is performing some more work besides “just” reading the index. With strace, we see lines like:

strace -e !write git ls-files -o>/dev/null 2>/tmp/a
…
openat(AT_FDCWD, "Documentation/", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 4
newfstatat(4, "", {st_mode=S_IFDIR|0755, st_size=1446, ...}, AT_EMPTY_PATH) = 0
getdents64(4, 0x55df0a6e6890 /* 99 entries */, 32768) = 3032
…

With Unadded New Files

Let’s add some files now:

for f in (seq 1 1000)
  touch $f
end

And compare with our baseline:

hyperfine --export-markdown /tmp/4.md --warmup 10 'git ls-files' 'git ls-files -o' 'find' \
  'fd --no-ignore' 'fd --no-ignore --hidden --exclude .git --type file --type symlink'

There is little to no statically significant difference to our baseline, which highlights that much of the time is spent on things relatively independent of the number of files processed. It’s also worth noting that there is relatively little speed difference between git ls-files -o and fd --no-ignore --hidden --exclude .git --type file --type symlink.

Using strace, we can establish that all commands but git ls-files were reading all files in the repository. By comparing the strace outputs of git ls-files -o and fd --no-ignore --hidden --exclude .git --type file --type symlink (the two commands that print the same file list), we can see that they make similar system calls for each file in the repository. How to explain the (small) time difference between the two? I haven’t found convincing reasons in git source code for this case. It might be that the use of the index gives ls-files a head start.

Conclusions

I’m now using git ls-files in my keyboard driven text editor instead of fd or find. It is faster, although the perceived difference described in the Introduction is probably due to spikes in latency on a cold cache. The selection of files is also narrowed down with ls-files to the ones I care about. That’s said, I’ve still kept the fd-based file listing as a fallback, as sometimes I’m not in a Git repository.

After all, Git is already building an index, why not use it to speed up your jumping from file to file!

Want to insert a short demo of your tool or of a command execution on your website? You could insert a video, but then visitors can’t copy text, and it’ll make your page quite heavy. The Asciinema player solves all of that, by replaying a terminal session stored in a text file.

This Hugo module makes it very easy to use the Asciinema Player on your Hugo-powered static website.

Install

Hugo Module

1. Make sure the go compiler is installed as well as Hugo. To check that, run the commands below. If the programs are installed, they should not return an error. Otherwise, see the go install instructions and Hugo install instructions:

   go version
   

   hugo version
   

2. If you haven’t already, you need to initialize Hugo modules:

   hugo mod init example.com/my-awesome-website
   

   This needs to be done once per Hugo site. Read the Hugo documentation for details and background.

Install This Module

Once Hugo modules are set up (see the previous section), you can install the asciinema module itself.

1. Run:

   hugo mod get -u -v cj.rs/gohugo-asciinema
   

2. Edit your Hugo config to add the module reference.

   For instance, if you use a config.toml config file, add:

   [module]
   [[module.imports]]
   path = "cj.rs/gohugo-asciinema"
   

   For config.yml, add:

   module:
   imports:
       - path: cj.rs/gohugo-asciinema
   

Use

Use this shortcode:

{{< asciicast src="https://joly.pw/blog/cargo-info-in-neovim/demo.json" poster="npt:0:04" autoPlay=true loop=true >}}

It’s rendered like this:

Defaults

You can set default values for the player in your config.toml or params.toml etc.. For example:

config.toml

[params.asciinema.defaults]
theme = "solarized-dark"
loop = true
autoPlay = true
speed = 5.0

params.toml

[asciinema.defaults]
theme = "solarized-dark"
loop = true
autoPlay = true
speed = 5.0

Notes

Advanced Features

Preloading

If you want to instruct the browser to preload the CSS and JS of the Asciinema Player, only when the page contains an asciicast, you can add the following line in the <head> tag of your template:

{{- partial "asciinema_css_js_smart_preload" . -}}

Themes often offer ways to do this easily. For instance with Hugo PaperMod, it’s in this file.

This partial will also cause some services to send Early Hints. That can further improve the page load time.

Features

• Displays a message when JavaScript is disabled in the user browser
• Fingerprinted assets, to improve caching and ultimately your site performance
• Optional preloading of JS and CSS assets
• Easy update with hugo mod get -u cj.rs/gohugo-asciinema

How Are the Sources of the Player Generated?

The Asciinema Player version is fetched from the official repository in the corresponding version. Then, if prebuilt JS/CSS files are provided they are used, so that you can verify that this module is actually distributing the original code. If not, these files are generated following the instructions from Asciinema Player Readme.

Contribute

Contributions (documentation or code improvements in particular) are welcome, see contributing!

To test your code changes locally, you can change your configuration so that your local version is loaded. Here is an example in TOML:

[module]
replacements = "cj.rs/gohugo-asciinema -> /some/path/gohugo-asciinema"
[[module.imports]]
path = "cj.rs/gohugo-asciinema"

Please consider sending a PR with your patches, it’s always appreciated and will save you the trouble of maintaining the changes on your own!

License

The code of this module (everything but the folder assets) is under the Apache License 2.0, like the asciinema player itself.

The assets folder contains the asciinema player. This code is Copyright Marcin Kulik, licensed under the Apache License, Version 2.0. See the upstream repository for the full sources.

⚡ TL;DR

When Opening the DB

PRAGMA journal_mode = wal; -- different implementation of the atomicity properties
PRAGMA synchronous = normal; -- synchronise less often to the filesystem
PRAGMA foreign_keys = on; -- check foreign key reference, slightly worst performance

And check user_version to apply any migrations, for instance with this Rust library.

When Closing the DB

PRAGMA analysis_limit=400; -- make sure pragma optimize does not take too long
PRAGMA optimize; -- gather statistics to improve query optimization

Introduction

SQL pragma are statements (like SELECT … or CREATE TABLE …) that change the database behaviors or call a special functions. This post is a short list of SQLite pragma I use in my projects built on SQLite, to get better performance and more consistency.

Performance

File system Interactions

The following pragma statements set the way SQLite deals with the file system.

journal_mode wal

By default, when applying changes, SQLite uses a rollback journal, which is basically a copy of the data before the changes. Changing the journal mode to “Write-Ahead Log” is known to bring significantly better performance in most cases. It also allows concurrent readers with one writer. To activate it, run:

PRAGMA journal_mode = wal;

Some less common use cases are incompatible with this journal mode, for instance having a database on a network file system. The full list of drawbacks is listed in the documentation¹.

synchronous normal

To ensure integrity of the database, one of the mechanism SQLite uses is the file system synchronization operations. However, these synchronizations are quite costly. With WAL journal mode enabled, your database will still be consistent while synchronizing less often:

PRAGMA synchronous = normal;

Compared to the default synchronous = full, committed transactions could be rolled back if there is a power loss (although not if the application crashes).

Optimize

To execute statement as efficiently as possible, SQLite has a query planner, which tries to read the tables to provide good performance (for instance by evaluating the WHERE clauses that select the fewest rows first).

This query planner sometimes needs to know whether a column has many values or only a few, repeated values (like a boolean column would). To know this, it can’t read the whole table, as that may prove as costly as running the part of the query that is being optimized, so it uses some statistics collected in internal tables.

Using optimize right before closing the database connexion collects the statistics for some table columns. These columns are chosen mainly based on the queries executed during the connexion: if a query had benefited from more accurate statistics, the corresponding columns are analyzed.

For instance:

PRAGMA analysis_limit=400;
PRAGMA optimize;

In the above example, pragma analysis_limit ensures that optimize won’t run for too long, by limiting the number of rows read.

Allow Using More Memory

These two pragma could result in better performance, depending on your hardware and software configuration.

• Keep temporary storage in memory: PRAGMA temp_store = 2. However, setting this pragma does not guarantee that the temporary storage will be held in memory. Please refer to the documentation for other parameters that may change the final outcome.
• Keep more of the database pages in memory: for instance, use 32 MiB of memory for this purpose with PRAGMA cache_size = -32000. Note that the OS cache is already keeping parts of the database file in RAM, so this might end up wasting memory.

Consistency

user_version

The database’s schema can evolve over time. Your application could start with a car table to represent a car but later on, you realize you want to represent bicycles, so you add a bicycle table.

To keep track of the different versions of the schema, some libraries maintain an internal table with a single row with a version number. It then performs migrations as needed. In our example, version 1 would have only the car table and version 2, also the bicycle table. The migration from version 1 to version 2 add the bicycle table.

SQLite offers the user_version pragma, to keep track of these versions. It is an integer at a fixed offset in the database file. It is simpler and more efficient than maintaining a table with versions, in particular because the table has to be found in the database file while the integer is available right away.

The drawback is that this is not portable between database engines. If you only use SQLite though, it is almost always the best option. To use it, you can write some code to check the value of the user_version pragma value right after opening the database. If it is lower than expected, then atomically 1) run the necessary migrations and 2) increment the user_version pragma value. I wrote a library to ease this task in rust and here is an example in python.

foreign_keys on

This may come as a surprise for folks with experience with other database system, but SQLite does not enforce foreign key constraints by default. Consequently, a foreign key can point to a row that does not exist.

This can be fixed with pragma foreign_keys:

PRAGMA foreign_keys = ON;

This comes at a performance cost however, because more checks are performed when inserting values with foreign keys. The cost is usually negligible but your mileage may vary.

Note: pragma foreign_key_check can be used to check a particular table for violated foreign key constraint. This can be useful before enabling foreign_keys on a database with existing data.

STRICT

Another peculiarity of SQLite is dynamic typing through type affinity. When you define a column of type INTEGER in most other database engines, a value of type TEXT inserted in that column will be converted or return an error. But with SQLite, type affinities mean that if the value is not of the expected type, it will be converted if possible or stored with a different type if necessary. That’s handy when prototyping, but you may want a stricter behavior for consistency with other major SQL databases or when working with strictly typed languages.

To this end, SQLite supports the STRICT keyword at table creation. For instance, if a table t is created like so:

CREATE TABLE t(a INTEGER) STRICT;

then

INSERT INTO t VALUES(1);
INSERT INTO t VALUES('1');

are successful but

INSERT INTO t VALUES('f');

is not and returns Runtime error: cannot store TEXT value in INTEGER column t.a (19).

Note: Without the STRICT keyword at table creation, this last INSERT INTO … statement would have been successful and would have just returned a string.

Go deeper

The full list of supported pragma with detailed descriptions is available in the documentation. Some more options can be tweaked for the specific scenario of running a SQLite for a server.