Last updated: 2022-05-23 05:21:51
Channels

How to open a PDF with default PDF viewer? The Rust Programming Language

The Rust Programming Language2022-05-23 05:06:01

This is not a Rust specific question, but I am trying to achieve it in a Rust program.

I generate a PDF file as:

use std::process::Command; let content: String = "\documentclass{article}\begin{document}Hello\end{document}" Command::new("pdflatex") .arg(content) .status() .expect("failed to run pdflatex"); 
  1. How can I open the new PDF file with whatever the default viewer is? Hopefully in Windows, Linux, and Mac.
  2. Is there a more "portable" way to generate the PDF? Or is requesting people to have pdflatex the best I can hope?
submitted by /u/DJDuque
[link] [comments]

rust-analyzer changelog #130 The Rust Programming Language

The Rust Programming Language2022-05-23 05:01:47 submitted by /u/WellMakeItSomehow
[link] [comments]

Can you write for the Commodore 64 in Rust? Why yes, yes you can! The Rust Programming Language

The Rust Programming Language2022-05-23 04:49:27

Using llvm-mos, rust-mos, a lot of time compiling compilers and support from Mariusz (the rust-mos author), I was finally able to program like it was 1982...

I was inspired by Jason Turner's talk where he did this using C++, and I wondered--hey, that was my first computer back in the day... Can I do that with my favorite language?

First test was to see how well rustc and llvm would optimize an idiomatically written 120-line trivial Rust program (which reads the joystick fire button status to set screen border color) to x86 assembly. Turns out the compilers pack it down to three instructions. Yep. 3. See for yourself! (remove the -C opt-level=3 to remove optimizations).

That is just bananas! Modern compilers are amaaazing.

So I wrestled for several days compiling compilers that would target the C-64's MOS-6510 (same instruction set as MOS-6502 also used by Atari, Apple and many, many others back in the day.) It's still a work in progress, but it is working pretty well!

use ufmt_stdio::{println, ufmt}; #[start] pub fn main(_argc: isize, _argv: *const *const u8) -> isize { println!("hELLO, {}-BIT C-64 WORLD, FROM rUST!", size_of::<*const ()>() * u8::BITS); ReturnCode::Ok.into() } 

compiles down to 232 bytes in release mode. (Yes, bytes). The code fits on a single block (256 bytes) of a C-64 floppy disk. And runs beautifully. (Note the inversion of case on this pre-ASCII (!) computer)... :)

The C-64 has an 8-bit data bus and a 16-bit address bus.

submitted by /u/U007D
[link] [comments]

Would you choose Rust for a whiteboard interview? The Rust Programming Language

The Rust Programming Language2022-05-23 04:16:16

Self-explanatory: Would you choose Rust to write code on a whiteboard in a typical coding interview that tests your knowledge of basic algorithms and data structures?

submitted by /u/nomyte
[link] [comments]

[New Learner Question]: What’s some great open source rust libraries that follow modern best practices and patterns for rust? The Rust Programming Language

The Rust Programming Language2022-05-23 04:01:05

Pretty much as the title says. I’ve picked up rust as a new language to try. Thus far I am loving it. What really drew me to rust was the entire philosophy behind memory management. I never gave it much thought because most languages I’ve worked with are GC languages. It’s really changed my perspective on structuring and writing code.

Edit: title should be “What are” not “What’s” 😐

submitted by /u/Slavichh
[link] [comments]

What’s in Which Python Hacker News: Front Page

Hacker News: Front Page2022-05-23 03:42:05

Article URL: https://nedbatchelder.com/text/which-py.html

Comments URL: https://news.ycombinator.com/item?id=31475130

Points: 23

# Comments: 7


STUNner: A Kubernetes ingress gateway for WebRTC Hacker News: Front Page

Hacker News: Front Page2022-05-23 02:27:06

Article URL: https://github.com/l7mp/stunner

Comments URL: https://news.ycombinator.com/item?id=31474810

Points: 11

# Comments: 0


Useless Use of "dd" (2015) Hacker News: Front Page

Hacker News: Front Page2022-05-23 02:09:40

Article URL: https://www.vidarholen.net/contents/blog/?p=479

Comments URL: https://news.ycombinator.com/item?id=31474710

Points: 130

# Comments: 66


Is This a Tank? Hacker News: Front Page

Hacker News: Front Page2022-05-23 01:32:38

Article URL: https://acoup.blog/2022/05/13/collections-ancient-tanks-chariots-scythed-chariots-and-carroballistae/

Comments URL: https://news.ycombinator.com/item?id=31474506

Points: 56

# Comments: 7


Current progress of my platformer game ps1graphics

ps1graphics2022-05-23 01:12:56
Current progress of my platformer game submitted by /u/Afraid_Contact8802
[link] [comments]

Empirical Notes on Kissing Hacker News: Front Page

Hacker News: Front Page2022-05-23 00:05:42

Article URL: https://belkarx.github.io/posts/finished/Empirical%20Notes%20on%20Kissing.html

Comments URL: https://news.ycombinator.com/item?id=31474078

Points: 81

# Comments: 26


Discovered a 63kb tiddlywiki like self saving HTML+JS wiki and its looking cool Hacker News: Front Page

Hacker News: Front Page2022-05-23 00:02:02

Article URL: https://feather.wiki/

Comments URL: https://news.ycombinator.com/item?id=31474062

Points: 131

# Comments: 23


Blinkenlights: PC Binary Emulating Visualizer Hacker News: Front Page

Hacker News: Front Page2022-05-22 23:29:23

Article URL: https://justine.lol/blinkenlights/

Comments URL: https://news.ycombinator.com/item?id=31473846

Points: 74

# Comments: 9


MasterCard to launch biometric checkout tool for retailers Hacker News: Front Page

Hacker News: Front Page2022-05-22 22:57:14

Article URL: https://www.retaildive.com/news/mastercard-biometric-checkout-retailers/624056/

Comments URL: https://news.ycombinator.com/item?id=31473632

Points: 32

# Comments: 35


Suffocation ps1graphics

ps1graphics2022-05-22 22:30:49
Suffocation submitted by /u/Theguywholikestf2
[link] [comments]

BonsaiDb performance update: A deep-dive on file synchronization The Rust Programming Language

The Rust Programming Language2022-05-22 22:25:55 submitted by /u/ectonDev
[link] [comments]

Case Studies in USB-C Debugging – Analog and digital compliance of a PD-enable Hacker News: Front Page

Hacker News: Front Page2022-05-22 22:18:23

Article URL: https://medium.com/@kolluru.nathan/case-studies-in-usb-c-debugging-93cc3e8bbc0c

Comments URL: https://news.ycombinator.com/item?id=31473369

Points: 24

# Comments: 2


Apple Cash Hacker News: Front Page

Hacker News: Front Page2022-05-22 22:08:36

Article URL: https://www.apple.com/apple-cash/

Comments URL: https://news.ycombinator.com/item?id=31473293

Points: 49

# Comments: 16


How to properly execute this workflow with PDFs The Rust Programming Language

The Rust Programming Language2022-05-22 21:37:31

I recently finished reading the book and I was looking to make a pet project and I think today I found the idea. The project I want to do is about dealing with pdf files.

I did some research and it turned out pdf files are hard to work with and it's better to use OCR,

My initial idea is to go like this:

  1. use OCR to convert pdf file to text format
  2. apply the changes
  3. replace the old text with the new, preserving images and original file structure

I'm wondering if this is the proper way to execute this, and if it's doable? I never worked with pdf files before I'm afraid that I may not able to get pretty results. What tools do you suggest? Any feedback will be welcome, thanks in advance

submitted by /u/AlexRodger
[link] [comments]

“What if it changes?” Hacker News: Front Page

Hacker News: Front Page2022-05-22 21:31:24

Article URL: https://chriskiehl.com/article/the-tyranny-of-what-if-it-changes

Comments URL: https://news.ycombinator.com/item?id=31472997

Points: 266

# Comments: 174


What is the reason behind things being private by default? I am a little confused about the benefits. The Rust Programming Language

The Rust Programming Language2022-05-22 21:29:41 submitted by /u/HSCorp666
[link] [comments]

If we wrote a lot of code depending on the linkme/inventory/typetag crates, are we stuck on legacy Rust versions forever? Or is there a path forward? The Rust Programming Language

The Rust Programming Language2022-05-22 21:27:07

These crates (with around 20 million combined downloads) are no longer maintained and have stopped working correctly on new Rust versions (https://github.com/rust-lang/rust/issues/47384). If we have a codebase where rewriting all of our code is infeasible, what other options do we have? Is there a realistic world where this functionality will ever exist again?

submitted by /u/thurn
[link] [comments]

Details on AMD's Quirky Chipset Solutions for AM5 Hacker News: Front Page

Hacker News: Front Page2022-05-22 20:59:21

Article URL: https://angstronomics.substack.com/p/site-launch-exclusive-all-the-juicy

Comments URL: https://news.ycombinator.com/item?id=31472716

Points: 103

# Comments: 55


Create height map mask / gradient from hex map? Procedural generation

Procedural generation2022-05-22 20:51:53

I have an old school D&D style hex map and I want to create a mask from it for Simplex Noise. I played with an idea of generating elevation ranges to interpolate between and increasing the range going farther from the coast and changing the rate of increase depending on the type of land formation. I'm curious if people have other ideas of how to constrain noise to match an old school map?

submitted by /u/pspeter3
[link] [comments]

Whats up with wasm? The Rust Programming Language

The Rust Programming Language2022-05-22 20:48:21

I played arround for 2 hours today and reading trough the official docs in wasm with rust they give you an example of compiling rust to wasm and then running it in the browser as functions trough js. Isn't the point of wasm to get rid of js or am i missing something? Why would you run wasm trough js?

submitted by /u/varto111
[link] [comments]

Map Compass Generator [Dragons Abound] Procedural generation

Procedural generation2022-05-22 20:46:22 submitted by /u/srt19170
[link] [comments]

Print numbers in arbitrary base in rust, or at least in seximal or niftimal The Rust Programming Language

The Rust Programming Language2022-05-22 20:44:04 submitted by /u/porky11
[link] [comments]

Simple software things that are actually very complicated Hacker News: Front Page

Hacker News: Front Page2022-05-22 19:56:54

Article URL: https://www.construct.net/en/blogs/ashleys-blog-2/simple-software-things-1587

Comments URL: https://news.ycombinator.com/item?id=31472093

Points: 115

# Comments: 52


Programming with Nothing Hacker News: Front Page

Hacker News: Front Page2022-05-22 19:27:58

Article URL: https://tomstu.art/programming-with-nothing

Comments URL: https://news.ycombinator.com/item?id=31471804

Points: 90

# Comments: 23


Current guidelines for sun exposure are unhealthy and unscientific – research (2019) Hacker News: Front Page

Hacker News: Front Page2022-05-22 18:47:39

Article URL: https://www.outsideonline.com/health/wellness/sunscreen-sun-exposure-skin-cancer-science/

Comments URL: https://news.ycombinator.com/item?id=31471416

Points: 367

# Comments: 283


Procedural low-poly geometry Procedural generation

Procedural generation2022-05-22 18:34:22
Procedural low-poly geometry submitted by /u/Fennyon
[link] [comments]

Smoke Trail ps1graphics

ps1graphics2022-05-22 18:32:08
Smoke Trail submitted by /u/MMMarcis
[link] [comments]

claui: Automatically create GUIs for your command-line applications The Rust Programming Language

The Rust Programming Language2022-05-22 18:13:57 submitted by /u/Dear_Spring7657
[link] [comments]

Lofi.co – Relax and Focus Hacker News: Front Page

Hacker News: Front Page2022-05-22 17:57:24

Article URL: https://lofi.co/

Comments URL: https://news.ycombinator.com/item?id=31470928

Points: 211

# Comments: 88


Operational IS-2M ps1graphics

ps1graphics2022-05-22 17:45:57
Operational IS-2M submitted by /u/Drelias
[link] [comments]

Rust format does not work with a very specific code structure The Rust Programming Language

The Rust Programming Language2022-05-22 17:34:22

I've noticed rustfmt stopped working in my file. After countless trial and error, I've managed to boil down my code to a minimum reproducible example.

Note the code below must have the white spaces exactly as I laid out, with the nested if, thread spawn, match etc. This seems to cause rust format to stop formatting the document.

```` use std::thread;

enum Switch1 { Option1, Option2, }

enum MassiveSwitch { Cmd1, }

struct Worker { id: i32, }

impl Worker { pub fn start(self) { let switch: MassiveSwitch = MassiveSwitch::Cmd1; let switch1 = Switch1::Option1;

 let _ = thread::Builder::new() .name(format!("Worker {}", self.id)) .spawn(move || { if true { match switch1 { Switch1::Option1 => match switch { MassiveSwitch::Cmd1 => { // TODO: the white lines here should be deleted by rustfmt on save (or format command) // TODO: removing this line causes rustfmt to work again. Commenting it out doesn't work, have to delete the line complete to get rust fmt to work println!("Shard worker {}: received shutdown signal, exiting loop...", 5); let test = "test"; } }, Switch1::Option2 => todo!(), } } }) .expect("Can't spawn"); } 

}

fn main() { let worker = Worker { id: 3 };

worker.start(); 

} ````

Here is a gist of the file: https://gist.github.com/l3utterfly/08e4b62ab57e69f6c8bb33c48655d206

If the println marked with TODO comment is removed, rust format starts to work again.

I am using VS Code, rust-lang.rust extension: v0.3.1051

Does anyone know why this specific structure of code stops rustfmt from working?

EDIT: also the content of the println must be exactly the text I wrote it seems. I tried with a few different other text, rustfmt works.

submitted by /u/Tasty-Lobster-8915
[link] [comments]

Constraint-based geometry (CAD) sketcher for Blender Hacker News: Front Page

Hacker News: Front Page2022-05-22 17:17:11

Article URL: https://github.com/hlorus/CAD_Sketcher

Comments URL: https://news.ycombinator.com/item?id=31470482

Points: 86

# Comments: 20


Grandpa’s Basement House Hacker News: Front Page

Hacker News: Front Page2022-05-22 17:09:37

Article URL: https://www.granolashotgun.com/granolashotguncom/hp5pmb0n95ut9hyeatewotgd2n1ebr

Comments URL: https://news.ycombinator.com/item?id=31470400

Points: 269

# Comments: 188


Kable (YC W22) Is Hiring Founding Software Engineers (US/Remote) Hacker News: Front Page

Hacker News: Front Page2022-05-22 17:00:54

Article URL: https://www.ycombinator.com/companies/kable/jobs/h3wKq6F-founding-backend-software-engineer

Comments URL: https://news.ycombinator.com/item?id=31470304

Points: 1

# Comments: 0


Show HN: Remult – a CRUD framework for full-stack TypeScript Hacker News: Front Page

Hacker News: Front Page2022-05-22 15:52:42

Article URL: https://github.com/remult/remult

Comments URL: https://news.ycombinator.com/item?id=31469481

Points: 111

# Comments: 46


Lilbits: Apple’s Self Service Repair isn’t all it’s cracked up to be Liliputing

Liliputing2022-05-22 15:30:12

Apple launched its Self Service Repair program in April in what appeared to be a major win for the right to repair movement. Folks who want to buy a replacement screen, battery, or other components without taking their phone to a repair shop can now do that. But actually fixing a phone by yourself? That’s […]

The post Lilbits: Apple’s Self Service Repair isn’t all it’s cracked up to be appeared first on Liliputing.


Jump Smoke Simulation in Houdini | Houdini Tutorial + Project File Procedural generation

Procedural generation2022-05-22 15:07:09
Jump Smoke Simulation in Houdini | Houdini Tutorial + Project File submitted by /u/Fxguru_yt14
[link] [comments]

Parca Agent rewrites eBPF in-kernel C code in Rust (using Aya-rs) The Rust Programming Language

The Rust Programming Language2022-05-22 14:42:24 submitted by /u/mdaverde
[link] [comments]

Serde support for esoteric configuration language The Rust Programming Language

The Rust Programming Language2022-05-22 14:12:00 submitted by /u/sameisshark
[link] [comments]

Trunk – Build, bundle and ship your rust WASM application to the web The Rust Programming Language

The Rust Programming Language2022-05-22 14:01:27 submitted by /u/ur_mum_goes_to_uni
[link] [comments]

is there a rust Async best practices book? The Rust Programming Language

The Rust Programming Language2022-05-22 13:28:53

I'm doing two real life projects in Tokio. I've learnt a lot. I've come across some problems where I hacked out a solution that works without fully understanding the tools at my disposal.

High level tools like channels, spawning tasks, streamext and sinkext. Low level tools like implementing streams and sinks and futures.

My "go to" solution seems to be, spawn ana sync block that drives a channel, then subscribe to that channel as a stream elsewhere.

I use mpsc channels with a single producer a lot. It feels like I'm doing something wrong, but it hacks together a solution.

In one solution, because order of things didn't matter, I have an exponentially growing swarm of tasks that spawn hundreds of new tasks each and doesn't keep track of them. It probably reaches ten million plus at any given point in time.

I know tasks are supposed to be light weight, but I feel like a spoilt person consuming the planets resources. I tried it with less spawning and more .awaiting and it was about 30% slower.

So my question is, because these tools, rust futures, and high level preemptive programming are quite new and unique, has anyone done a deep dive and published a best practices book?

submitted by /u/matiu2
[link] [comments]

Procedural Barrel Generator Procedural generation

Procedural generation2022-05-22 13:22:23
Procedural Barrel Generator submitted by /u/mishka__
[link] [comments]

Beach whale ps1graphics

ps1graphics2022-05-22 13:01:08
Beach whale submitted by /u/_SereneMango
[link] [comments]

Procedurally generated swamp biome for my roguelike game Procedural generation

Procedural generation2022-05-22 12:12:11
Procedurally generated swamp biome for my roguelike game submitted by /u/KCFOS
[link] [comments]

Is there a study comparing the memory usage of using cellular automata vs poisson disk sampling in generating objects? Procedural generation

Procedural generation2022-05-22 10:46:35

I'm very new to PCG. I'm still learning and I'm trying to see the pros and cons of both cellular automata and poisson disk sampling in generating objects, in my case trees. I was thinking of comparing them based on how much RAM they consume but I'm not sure if it's a good idea. Should I use Big O notation?
If you can share studies or papers regarding the pros and cons of the two algorithm it would be greatly appreciated. I'm so sorry if it's a dumb question. Thank you in advance

submitted by /u/desugunn
[link] [comments]

process-stream v0.1.3: streamable and long running tokio::Command The Rust Programming Language

The Rust Programming Language2022-05-22 10:19:19

Hello there, I'm happy to announce a simple crate that simplify dealing with long running process in a simple way. In addition it support converting from Command or vector like vec!["ls", "~/"]

https://github.com/tami5/process-stream

 use process_stream::Process; use process_stream::StreamExt; use std::io; #[tokio::main] async fn main() -> io::Result<()> { let mut long_process = Process::new("/bin/app"); long_process.current_dir("/"); long_process.args(&["--debug"]) let mut stream = long_process.stream()?; tokio::spawn(async move { while let Some(output) = stream.next().await { println!("{output}") } }); // process some outputs tokio::time::sleep(std::time::Duration::new(10, 0)).await; // close the process long_process.kill().await; Ok(()) } 

Here another example with vector

 #[tokio::main] async fn main() -> io::Result<()> { let ls_home: Process = vec!["/bin/ls", "."].into(); let mut stream = ls_home.stream()?; while let Some(output) = stream.next().await { println!("{output}") } Ok(()) } 
submitted by /u/tami5
[link] [comments]

Let's Code Snake with Rust and WASM The Rust Programming Language

The Rust Programming Language2022-05-22 10:10:01 submitted by /u/yishn
[link] [comments]

Rust's problematic reliance on GitHub The Rust Programming Language

The Rust Programming Language2022-05-22 10:02:38

Is anyone else worried that you basically must have a GitHub account to publish a crate to crates.io? Or that GitHub must be available in your country for you to download your dependencies?

This means that anyone who's in a country that has blocked GitHub or where GitHub has been blocked due to sanctions is unable to participate in the Rust ecosystem. Or maybe your account has been blocked due to sanctions? Boom, all your work gone. You can't access your account now.

And even if GitHub was freely available everywhere for everyone, this would still be problematic, because the requirement of a GitHub profile to publish a crate only further incentivises centralisation of all code on GitHub. I hope i don't have to explain why it's problematic.

I love crates.io and cargo, and I see the convenience of having a package manager, but if it comes with complete control of an entire ecosystem by a single american company with one of the worst track records among technology companies... I'd rather just download everything manually tbh

submitted by /u/whyvitamins
[link] [comments]

Procedural generated backrooms-type maze Procedural generation

Procedural generation2022-05-22 08:50:20

i was trying to make a backrooms game in unity, i searched everywhere to make a 3d procedural generation, but i found just 22d videos.
my best option right now is just to make 2d tilemaps into 3d.

https://keesiemeijer.github.io/maze-generator/#generate

i wanted to do something like this. Im new to coding so help would be apreciated.

submitted by /u/FippiOmega
[link] [comments]

Metal Gear Mk2, little animation test ps1graphics

ps1graphics2022-05-22 05:39:31
Metal Gear Mk2, little animation test submitted by /u/JaxiPup
[link] [comments]

Bevy How To UserInput Programming Games in Rust

Programming Games in Rust2022-05-22 04:17:42

I Have Made the first episode of a 5 part mini-series on how to get and use user input with the bevy engine.

I would love any feedback;

https://youtu.be/pB3ERI5JtrA

submitted by /u/PhaestusFox
[link] [comments]

Bevy How To Get User Input The Rust Programming Language

The Rust Programming Language2022-05-22 04:17:10

I Have Made the first episode of a 5 part mini-series on how to get and use user input with the bevy engine.

I would love any feedback;

https://youtu.be/pB3ERI5JtrA

submitted by /u/PhaestusFox
[link] [comments]

[Gitoxide in April] 2.2x more speed for `onefetch` and `git-sec` baked in The Rust Programming Language

The Rust Programming Language2022-05-22 03:23:03 submitted by /u/ByronBates
[link] [comments]

soldier ready ps1graphics

ps1graphics2022-05-22 02:36:07
soldier ready submitted by /u/RisingForce3D
[link] [comments]

Soldier (soon on my patreon page) ps1graphics

ps1graphics2022-05-22 02:34:55

Lowpoly ps1 style chracter, fully rigged.

Near to finish him , will be available on my patreon soon.

submitted by /u/RisingForce3D
[link] [comments]

Robyn turns 1 today! 🎂 The Rust Programming Language

The Rust Programming Language2022-05-21 19:25:36

Robyn, a fast and extensible async Python web server with a Rust runtime, turns 1 today! 🎂

Never did I think that a project made as a getaway from my college curriculum, would receive such love from the community. Thank you for all the support! 💖
In the past year Robyn has over 1.3k stars on GitHub and over 250k installs on PyPi.

Robyn now has the following features:

  • A multithreaded Runtime
  • Extensible Plugin Ecosystem
  • A simple API
  • Strong Type definitions
  • Sync and Async Function Support
  • Dynamic URL Routing
  • Multi Core Scaling
  • WebSockets
  • Middlewares
  • Hot Reloading
  • A great community support!

Hoping that this year is even better!
Onwards and upwards! 🚀

Project Source: https://github.com/sansyrox/robyn

https://i.redd.it/fzafc019rv091.gif

submitted by /u/stealthanthrax
[link] [comments]

Be careful folks! rust-analyzer is currently (accidentally) breaking VSCode extension updates. The Rust Programming Language

The Rust Programming Language2022-05-21 18:47:04

Here's the issue: https://github.com/microsoft/vscode/issues/149518

Basically, if you have rust-analyzer installed, you won't get any VSCode extension updates, which could be a security concern. I was able to uninstall rust-analyzer, update all my extensions, and then re-install it.

Please let's be kind to both VSCode and Rust, software is hard and sometimes mistakes get made. Props to both communities for already figuring out what's wrong, and having plans to solve it.

submitted by /u/last_account_promise
[link] [comments]

Procedural spaceship generator that I made for my new game Procedural generation

Procedural generation2022-05-21 18:24:20
Procedural spaceship generator that I made for my new game submitted by /u/apseren
[link] [comments]

cemetary ps1graphics

ps1graphics2022-05-21 18:21:30
cemetary submitted by /u/kaypathy
[link] [comments]

Just finished texturing it. A spacefighter thing named "Mouse86" lul ps1graphics

ps1graphics2022-05-21 17:55:12
Just finished texturing it. A spacefighter thing named "Mouse86" lul submitted by /u/dovedels
[link] [comments]

Transport bots improved, so they can equally distribute resources for you! Procedural generation

Procedural generation2022-05-21 15:11:15
Transport bots improved, so they can equally distribute resources for you! submitted by /u/Tefel
[link] [comments]

HP Dev One laptop with Pop!_OS Linux coming soon for $1099 Liliputing

Liliputing2022-05-21 14:45:06

Most HP computers ship with Windows, but from time to time the company has dabbled in Linux by offering models with Ubuntu or Red Hat Enterprise Linux. HP’s next Linux laptop is a little different. The upcoming HP Dev One is a 14 inch laptop with an AMD Ryzen processor that will ship with Pop!_OS […]

The post HP Dev One laptop with Pop!_OS Linux coming soon for $1099 appeared first on Liliputing.


made a dog monster ps1graphics

ps1graphics2022-05-21 14:19:13
made a dog monster submitted by /u/Nash_Dash
[link] [comments]

Is there a planet generator that generates planets with cities? Procedural generation

Procedural generation2022-05-21 13:44:07

I've honestly been searching for a long time for one of these but, I just haven't found any.

submitted by /u/Rastaferiaiscool
[link] [comments]

The Backrooms - Level 974 (Missing Persons) Procedural generation

Procedural generation2022-05-21 13:05:37
The Backrooms - Level 974 (Missing Persons) submitted by /u/BassamGaad
[link] [comments]

What are legitimate problems with Rust? The Rust Programming Language

The Rust Programming Language2022-05-21 12:31:56

As a huge fan of Rust, I firmly believe that rust is easily the best programming language I have worked with to date. Most of us here love Rust, and know all the reasons why it's amazing. But I wonder, if I take off my rose-colored glasses, what issues might reveal themselves. What do you all think? What are the things in rust that are genuinely bad, especially in regards to the language itself?

submitted by /u/deerangle
[link] [comments]

New song "I wish I could come back to Earth now" generated by my TuneStar computer program Procedural generation

Procedural generation2022-05-21 00:50:52
New song "I wish I could come back to Earth now" generated by my TuneStar computer program submitted by /u/tunestar2018
[link] [comments]

Hiking With Your Dog Posts on elder.dev

Posts on elder.dev2022-05-21 00:00:00 Bobby loves hiking

Screen 13 joins the ray tracing club with release v0.3! Programming Games in Rust

Programming Games in Rust2022-05-20 23:17:47
Screen 13 joins the ray tracing club with release v0.3!

The easy-to-use Vulkan render graph library can now record ray trace passes! A new example shows how in just 1,000 lines; this is compared to 100 lines for the triangle example.

https://github.com/attackgoat/screen-13

https://preview.redd.it/i4k9yap0qp091.png?width=802&format=png&auto=webp&s=767b57d3de397391c00c2dabf8665a7fa6a43678

submitted by /u/attackgoat_official
[link] [comments]

All the art and environment is procedurally generated! Procedural generation

Procedural generation2022-05-20 21:47:32
All the art and environment is procedurally generated! submitted by /u/lorddeus369
[link] [comments]

Lilbits: OSOM V1 will be Snapdragon 8+ Gen 1 powered, ONEXPLAYER working on a handheld with Ryzen 6000U Liliputing

Liliputing2022-05-20 19:00:42

Qualcomm’s new Snapdragon 8+ Gen 1 processor is expected to bring a 10% boost in CPU and graphics performance and a 20% improvement in AI performance when compared with the Snapdragon 8 Gen 1, while also lowing power consumption by as much as 30%. So it’s no surprise that a bunch of phone makers plan […]

The post Lilbits: OSOM V1 will be Snapdragon 8+ Gen 1 powered, ONEXPLAYER working on a handheld with Ryzen 6000U appeared first on Liliputing.


Microsoft is bringing Android 12.1 to the Windows Subsystem for Android for Windows 11 Liliputing

Liliputing2022-05-20 17:33:52

The Windows Subsystem for Android is an optional component of Windows 11 that allows you to run Android apps on a Windows PC. Basically Microsoft achieves this by installing a complete Android operating system in a virtual machine and configuring it so that Android you can launch and use Android apps almost as if they […]

The post Microsoft is bringing Android 12.1 to the Windows Subsystem for Android for Windows 11 appeared first on Liliputing.


Daily Deals (5-20-2022) Liliputing

Liliputing2022-05-20 17:00:35

Hulu is running a deal that lets you subscribe to the company’s ad-supported tier for $1 per month for the first three months, which means you can save as much as $18 in that time (although you’ll have to put up with ads). Meanwhile Sling TV is slashing the prices of its TV-over-the-internet service in […]

The post Daily Deals (5-20-2022) appeared first on Liliputing.


"Ketchup" - my in development retro fps - Coven ps1graphics

ps1graphics2022-05-20 16:55:20
"Ketchup" - my in development retro fps - Coven submitted by /u/imaethan
[link] [comments]

Qualcomm introduces Snapdragon 8+ Gen 1 and Snapdragon 7 Gen 1 chips for premium & mid-range devices Liliputing

Liliputing2022-05-20 15:43:20

Qualcomm’s new Snapdragon 8+ Gen 1 processor is coming soon to smartphones and other mobile devices, with the company promising up to 10% faster performance than the original Snapdragon 8 Gen 1 and up to 30% lower power consumption. You’ll likely have to wait until this fall to get your hands on a phone with the new […]

The post Qualcomm introduces Snapdragon 8+ Gen 1 and Snapdragon 7 Gen 1 chips for premium & mid-range devices appeared first on Liliputing.


MINISFORUM Neptune HX90G is a compact desktop with AMD Radeon RX 6650M discrete graphics Liliputing

Liliputing2022-05-20 14:58:42

MINISFORUM has been making small form-factor desktop computers for a few years at this point, but the company has recently started cranking up the horsepower, first by launching models with high-performance processors, and more recently introducing mini PCs with support for discrete graphics, although the first MINISFORUM PC with that option requires an external graphics […]

The post MINISFORUM Neptune HX90G is a compact desktop with AMD Radeon RX 6650M discrete graphics appeared first on Liliputing.


WIP: Generating natural paths in terrain with hills Procedural generation

Procedural generation2022-05-20 14:55:30
WIP: Generating natural paths in terrain with hills submitted by /u/spidergrisen
[link] [comments]

Mixtile Blade 3 is a stackable single-board PC with RK3588 processor, support for cluster computing (crowdfunding) Liliputing

Liliputing2022-05-20 14:24:33

The Mixtile Blade 3 is a single-board computer with a Rockchip RK3588 processor at its heart, a Pico-ITX form factor, and a design that allows you to stack multiple boards on top of one another to build a cluster. First unveiled in April, the Mixtile Blade 3 is now up for pre-order through a Crowd Supply […]

The post Mixtile Blade 3 is a stackable single-board PC with RK3588 processor, support for cluster computing (crowdfunding) appeared first on Liliputing.


One Netbook T1 is a 13 inch tablet with Intel Core i7-1260P Liliputing

Liliputing2022-05-20 13:23:43

One Netbook, a Chinese company best known for making mini-laptops and handheld gaming computers is preparing to launch its first 2-in-1 tablet. The One Netbook T1 is a 13 inch Windows tablet with a detachable keyboard, a built-in kickstand, and support for a pressure-sensitive pen. It’s also one of the first tablets powered by a […]

The post One Netbook T1 is a 13 inch tablet with Intel Core i7-1260P appeared first on Liliputing.


Proceduraly generate Plants & Trees yourself easly Procedural generation

Procedural generation2022-05-20 09:26:28
Proceduraly generate Plants & Trees yourself easly submitted by /u/Consistent_Reveal_53
[link] [comments]

Liminal Space PS1 music video ps1graphics

ps1graphics2022-05-20 08:34:06
Liminal Space PS1 music video submitted by /u/CyberOstrich
[link] [comments]

New programmer, looking for some advice with Rust as my first language and gamedev as my main hobby. Programming Games in Rust

Programming Games in Rust2022-05-20 07:44:55

I made a post over at r/Rust, but I figured here works as well.

Basically, I think the best way to learn things in general is to pick something you're passionate about and do your best, even if it's not optimal.

For some reason, Rust really appeals to me. I like the way the language looks, I like the community, I like the underdog story lol. But because of how much it appeals to me, I believe it will be the best for motivation to learn Rust.

I also want to develop games. Ideally, a game with an isometric view like Stardew Valley (not the artstyle and not the gameplay, just that view/2d looking characters).

I'm looking for resources dedicated to beginner programmers. No history with Python, C, C++, nothing. A new programmer who happens to want to learn Rust!

Even better if there are ones targeted toward making a game.

And lastly, I'd like some information on engines (I know that Unreal uses C++) to create a game entirely with Rust. I've seen some Rust game engine names floating around, just curious to get more info.

Thanks everyone in advance.

submitted by /u/directedgraphs
[link] [comments]

mapgen - a small collection of procedural dungeon generation algorithms Procedural generation

Procedural generation2022-05-20 05:12:46
mapgen - a small collection of procedural dungeon generation algorithms

Lately I've been having fun with implementing some procedural dungeon generation algorithms.

Here's the repository with the code and a list of the works I took inspiration from. You can try the WASM version here.

The algorithm implementations are far from perfect, some are downright dumbed down compared to the originals; the code is far from great quality. Still, I hope that it can serve as a somewhat helpful example to some of you :)

rooms and mazes

submitted by /u/_gothgoat
[link] [comments]

Character Trait Generator Procedural generation

Procedural generation2022-05-20 03:19:24

Not sure if this counts, but I made my first generator today!

Character Trait Generator

It outputs positive, negative, and neutral traits in one roll!
Would love feedback!

submitted by /u/Zezty-Lemon
[link] [comments]

Epstein's Private Prison on the Moon - For our VR Journalist Sim ps1graphics

ps1graphics2022-05-19 22:12:58
Epstein's Private Prison on the Moon - For our VR Journalist Sim submitted by /u/Buce1
[link] [comments]

Mineral generator Procedural generation

Procedural generation2022-05-19 21:10:11

I've been looking for a procedural mineral, gem, stone, etc. generator on the web and have had no luck finding one.
Anyone have links?

submitted by /u/Zezty-Lemon
[link] [comments]

Lilbits: OnePlus Nord 2T, Moto G71, and GeForce NOW brings Fortnite to iOS (and Android, and anything with a web browser) Liliputing

Liliputing2022-05-19 20:00:16

Fortnite was famously booted from the App Store and Google Play Store a few years ago when Epic Games intentionally flouted Apple and Google’s rules in order to kick off a legal challenge. While it’s still fairly easy to install Fortnite on an Android phone by sideloading it, the game has only recently returned to […]

The post Lilbits: OnePlus Nord 2T, Moto G71, and GeForce NOW brings Fortnite to iOS (and Android, and anything with a web browser) appeared first on Liliputing.


A gaming laptop with Intel Arc A730M graphics is up for pre-order in China Liliputing

Liliputing2022-05-19 18:33:33

The first laptops with Intel Arc 3 discrete graphics began shipping earlier this year, although notebooks with Intel’s entry-level GPU still aren’t exactly common. Even less common? Notebooks with Intel’s higher-performance Arc 5 and Arc 7 discrete GPUs. The chip maker said we’d have to wait until this summer for those models to arrive. But […]

The post A gaming laptop with Intel Arc A730M graphics is up for pre-order in China appeared first on Liliputing.


Daily Deals (5-19-2022) Liliputing

Liliputing2022-05-19 17:18:35

The Epic Games Store is giving away Borderlands 3 for free this week. Meanwhile, Epic has also launched a MEGA Sale on PC games that runs from today through June 16th. Many games are on sale, with some titles going for as much as 75% off. And that’s before you apply a coupon good for […]

The post Daily Deals (5-19-2022) appeared first on Liliputing.


Rust Arcade Cabinet at RustConf in Portland: An opportunity to show your games to the broader Rust community Programming Games in Rust

Programming Games in Rust2022-05-19 17:03:37

Hello Rustaceans!

I announced this project last week on twitter here, but I wanted to make sure to mention on this subreddit as well to get the word out.

I'm building a custom arcade cabinet that will be at RustConf in Portland. I'm going to load it with games made with Rust by people in the Rust game development community. My goal is to promote people's games and show off this cool space within Rust.

If you want your game to be on the cabinet please fill out this form. It's very short, you aren't committing to anything by filling it out, but it will allow me to contact you to inform you of steps to take to get your game on the cabinet.

The arcade cabinet is going to be made out of 18 gauge cold rolled carbon steel that will be laser cut and bent to shape. Then the parts will be welded together into the cabinet.

The current plan is for the cabinet to run Windows 10 with a GTX 1080 graphics card and Ryzen 7 processor. The inputs are going to be handled using an IPAC 2 game controller interface. I plan on writing some guides that will help developers get their games compatible with the controls.

This week I submitted the custom metal parts for the cabinet to be fabricated and I'm currently working with an artist to create some cool vinyl stickers featuring Ferris to be put on the visible surfaces of the cabinet. I'm also starting to work on a way to control the LEDs on the buttons from people's games.

As you can probably guess, this is an expensive project, so if you want to support this idea and are in the position to donate, I have setup a patreon where you can do so: https://www.patreon.com/metalmancy

I also plan on posting some videos about the project to my youtube channel, so look out for those here: https://www.youtube.com/channel/UC6cKxhWxtWCgKy9_-T8xJAg

I have a discord server where people can further discuss this project and share ideas (not super active now, but hopefully that will change soon): https://discord.gg/Q52Ksvv

Lastly, follow me on twitter for more frequent updates about the development of the project: https://twitter.com/carlosupina

I hope to see your submissions!

submitted by /u/FetusBoBetus
[link] [comments]

Microsoft is experimenting with a Search widget on the Windows 11 desktop Liliputing

Liliputing2022-05-19 16:53:13

Windows 11 brought the return of widgets to Microsoft’s desktop operating system, but for the most part they’ve been confined to a Widgets board that slides out from the left side of the screen when you want to use it and hides away when you don’t. Now Microsoft has introduced an experimental search box widget […]

The post Microsoft is experimenting with a Search widget on the Windows 11 desktop appeared first on Liliputing.


Framework’s modular laptop now available with Intel Alder Lake (upgrade kits for Tiger Lake models coming soon) Liliputing

Liliputing2022-05-19 15:11:45

The Framework Laptop is a thin and light notebook with premium components including a 13.5 inch, 2256 x 1504 pixel LCD display and support for up to an Intel Core i7 processor, 32GB of RAM, and 1TB of storage. It’s also unique in that it has a modular design: you can easily swap out ports. […]

The post Framework’s modular laptop now available with Intel Alder Lake (upgrade kits for Tiger Lake models coming soon) appeared first on Liliputing.


Saturn Procedural generation

Procedural generation2022-05-19 13:25:22
Saturn submitted by /u/Comfortable-Talk-514
[link] [comments]

HP Envy 2022 laptops come in many sizes, but they all have 5MP webcams Liliputing

Liliputing2022-05-19 12:00:35

Laptop webcams have largely been horrible for years. But now that video conferencing is becoming a way of life for many computer users, HP is making the upgraded cameras in this year’s HP Envy laptops a key selling point. The new HP Envy x360 13.3″, HP Envy x360 15.6″, HP Envy 16, and HP Envy 17.3″ laptops all feature HP True […]

The post HP Envy 2022 laptops come in many sizes, but they all have 5MP webcams appeared first on Liliputing.


HP Spectre x360 13.5 inch convertible packs a 3:2 display and a 5MP webcam Liliputing

Liliputing2022-05-19 12:00:34

The new HP Spectre x360 13.5″ laptop is a 3 pound convertible notebook with a 12th-gen Intel Core U-series processor, support for up to 32GB of RAM and 2TB of storage, and a 66 Wh battery. As part of HP’s Spectre line of premium notebooks, it’s got a sleek design and some nice extras like quad speakers […]

The post HP Spectre x360 13.5 inch convertible packs a 3:2 display and a 5MP webcam appeared first on Liliputing.


Completed The Witcher 3 animation, PS1 style ps1graphics

ps1graphics2022-05-19 11:51:40
Completed The Witcher 3 animation, PS1 style submitted by /u/Magni7icent
[link] [comments]

would it be possible to procedurally generate chunks according to the terrain in them? Procedural generation

Procedural generation2022-05-19 09:37:37

Like I was thinking, if it was a mountain, don't load things behind the mountain, so make a chunk that is the mountain itself and boom, leave it at that, then for plains, you'd look far into the distance so it would be a very big chunk section. Then you could I guess split up the chunk according to poly density??? Idk, I've never dipped into procedural but I have experience in programming, just not games and was wondering if any of you have insight into this. Thank you for your time

submitted by /u/ModellingArtsYT
[link] [comments]

What do you think about using SDL2 bindings for a simple game? Programming Games in Rust

Programming Games in Rust2022-05-19 08:51:58

Hi, I am trying to make a book to learn Rust for total beginners that teaches coding using Rust, assuming zero knowledge.

At some point I need to have small projects, and gamedev is one of the best tools to make people learn, because tweaking + playing is a great way to get people invested.

But it seems we lack something similar to PyGame. I've been looking into Bevy and Amethyst. They make usage of too many advanced concepts in Rust. And they seem to be changing fast. A beginner is going to have problems following traits, lifetimes and such.

After a quick look onto the SDL2 bindings, https://github.com/Rust-SDL2/rust-sdl2, it seems quite mature and the sample code I'm reading uses very basic Rust features, nothing too fancy. Which makes sense, since we're binding to a C library.

Do you think SDL2 would be appropriate for a beginner to create a simple pong game?

submitted by /u/deavidsedice
[link] [comments]

🦀 Dims dev log - Now you can make a landscape complete with nature objects in a minute! Here's three examples we made in about 3 minutes. Next up is enabling you to script gameplay on top of these. Programming Games in Rust

Programming Games in Rust2022-05-19 08:43:48
🦀 Dims dev log - Now you can make a landscape complete with nature objects in a minute! Here's three examples we made in about 3 minutes. Next up is enabling you to script gameplay on top of these. submitted by /u/5dims
[link] [comments]

This Week in Rust #443 The Rust Programming Language

The Rust Programming Language2022-05-19 05:00:59 submitted by /u/seino_chan
[link] [comments]

Diablo (PS1) on Unreal Engine ps1graphics

ps1graphics2022-05-19 04:14:49
Diablo (PS1) on Unreal Engine submitted by /u/AlucardieBR
[link] [comments]

Creating traversable 3D game levels using Wave Function Collapse Procedural generation

Procedural generation2022-05-19 00:22:35

I'm working on a procedural cave level generator and I got the base WFC generation working, but the problem is that a lot of the pieces are disconnected and you cannot call the terrain "playable". I'm having trouble figuring out how to enforce at least one path that goes through all of the big chunks and then probably just discard the very small ones. How can I achieve this feat in a 3D generated artefact? Most of the pathing options I've seen seem to be directed at 2D level generation

submitted by /u/vladutelu
[link] [comments]

The second Rust Graphics Meetup is happening this Saturday! More details in the blog post Programming Games in Rust

Programming Games in Rust2022-05-18 23:48:45
The second Rust Graphics Meetup is happening this Saturday! More details in the blog post submitted by /u/_AngelOnFira_
[link] [comments]

Need advice on whether I should use the unreal mannequin skeleton or not? More details in the comments ps1graphics

ps1graphics2022-05-18 23:31:03
Need advice on whether I should use the unreal mannequin skeleton or not? More details in the comments submitted by /u/Afraid_Contact8802
[link] [comments]

Daily Deals (5-18-2022) Liliputing

Liliputing2022-05-18 18:04:31

Google’s Pixel 6a is coming in July for $449. But you know what you can get for $449 right now? a Google Pixel 5. The previous-gen flagship doesn’t quite have the same processing power as the Pixel 6a, but it’s got the same cameras and more RAM. And it’s just one of several Pixel phones […]

The post Daily Deals (5-18-2022) appeared first on Liliputing.


Amazon Fire OS 8 debuts this summer (Android 11-based software for Fire devices) Liliputing

Liliputing2022-05-18 15:48:10

The new Amazon Fire 7 (2022) tablet brings a better processor, more memory, and longer battery life to Amazon’s entry-level tablet. The $60 tablet will also be the first to ship with Fire OS 8 when the new Fire 7 begins shipping June 29, 2022. Fire OS is the Android-based operating system Amazon uses for […]

The post Amazon Fire OS 8 debuts this summer (Android 11-based software for Fire devices) appeared first on Liliputing.


Acer brings glasses-free 3D to gaming laptops, portable monitors Liliputing

Liliputing2022-05-18 15:21:43

Acer’s SpatialLabs technology is designed to let you view 3D content on a 2D screen without the need to put on a special pair of glasses. Last year the company launched a high-end content creation laptop with support for the feature. Now Acer is bringing SpatialLabs to gaming laptops and portable displays. The new Acer […]

The post Acer brings glasses-free 3D to gaming laptops, portable monitors appeared first on Liliputing.


Compare Amazon Fire tablet specs: Fire 7, Fire HD 8, and Fire 10 tablets Liliputing

Liliputing2022-05-18 14:18:33

Amazon’s Fire tablets are some of the most cheapest tablets worth buying thanks to a combination of decent screens, acceptable performance (for some tasks) and really low starting prices. Normally you can pick up an Amazon Fire tablet for between $60 and $180. From time to time, they go on sale at deep discounts. But […]

The post Compare Amazon Fire tablet specs: Fire 7, Fire HD 8, and Fire 10 tablets appeared first on Liliputing.


Arcade Mode ps1graphics

ps1graphics2022-05-18 13:38:30
Arcade Mode submitted by /u/TG__Stig
[link] [comments]

Looking for a blocky noise map. Any recommendations? Procedural generation

Procedural generation2022-05-18 11:01:27
Looking for a blocky noise map. Any recommendations? submitted by /u/getrasa
[link] [comments]

Voxel space and WGPU troubles Programming Games in Rust

Programming Games in Rust2022-05-18 01:43:50 submitted by /u/im_alone_and_alive
[link] [comments]

Terraformation in Antimatter Procedural generation

Procedural generation2022-05-17 21:43:50
Terraformation in Antimatter submitted by /u/geoffroypir
[link] [comments]

On the Way to Work ps1graphics

ps1graphics2022-05-17 20:57:05
On the Way to Work submitted by /u/CannedPeas_1
[link] [comments]

Website to generate 3D plants Procedural generation

Procedural generation2022-05-17 15:48:54
Website to generate 3D plants submitted by /u/Ok-Air-238
[link] [comments]

Making more progress - onto rigging before another pass on these textures. The old man is 726 tris and a 256² px texture. ps1graphics

ps1graphics2022-05-17 15:07:25
Making more progress - onto rigging before another pass on these textures. The old man is 726 tris and a 256² px texture. submitted by /u/DistributionBrave580
[link] [comments]

Bevy Basics #8 Events Programming Games in Rust

Programming Games in Rust2022-05-17 09:02:10
Bevy Basics #8 Events submitted by /u/PhaestusFox
[link] [comments]

I can definitely beat Ramiel in a fight... probably? ps1graphics

ps1graphics2022-05-17 01:49:20
I can definitely beat Ramiel in a fight... probably? submitted by /u/SwagSoul
[link] [comments]

Audio Libraries Considered Challenging Programming Games in Rust

Programming Games in Rust2022-05-16 23:31:53 submitted by /u/tesselode
[link] [comments]

Hey Rustaceans! Got a question? Ask here! (20/2022)! The Rust Programming Language

The Rust Programming Language2022-05-16 06:35:59

Mystified about strings? Borrow checker have you in a headlock? Seek help here! There are no stupid questions, only docs that haven't been written yet.

If you have a StackOverflow account, consider asking it there instead! StackOverflow shows up much higher in search results, so having your question there also helps future Rust users (be sure to give it the "Rust" tag for maximum visibility). Note that this site is very interested in question quality. I've been asked to read a RFC I authored once. If you want your code reviewed or review other's code, there's a codereview stackexchange, too. If you need to test your code, maybe the Rust playground is for you.

Here are some other venues where help may be found:

/r/learnrust is a subreddit to share your questions and epiphanies learning Rust programming.

The official Rust user forums: https://users.rust-lang.org/.

The official Rust Programming Language Discord: https://discord.gg/rust-lang

The unofficial Rust community Discord: https://bit.ly/rust-community

Also check out last weeks' thread with many good questions and answers. And if you believe your question to be either very complex or worthy of larger dissemination, feel free to create a text post.

Also if you want to be mentored by experienced Rustaceans, tell us the area of expertise that you seek. Finally, if you are looking for Rust jobs, the most recent thread is here.

submitted by /u/llogiq
[link] [comments]

Is there a better way to do this? (Drawing a lot of rectangles with GGEZ) Programming Games in Rust

Programming Games in Rust2022-05-15 21:44:41

Hello! I am learning Rust and I wanted to make Game of Life using the GGEZ framework. The code all works, but the draw method is relatively "slow". After a little bit of research I found that the likely cause is my use of the let rect_mesh = graphics::Mesh::new_rectangle(context, graphics::DrawMode::fill(), rect, color)?; and graphics::draw(context, &rect_mesh, graphics::DrawParam::default())?; being called in the nested for loop.

The basic flow of my draw is:

  1. Create a graphics::Rect with the correct dimensions of the cells
  2. Iterate through the 2d array of cells:
    1. Move the rectangle to the correct location based on the x and y of the for loops
    2. For every cell, call the draw on the cell (ex: cells[x][y].draw(context, rect, draw_mode)?;)
    3. Within the cell "class", the program uses the draw_mode to figure out the color of the cells (the draw_mode is either normal, or it colors the cells based on neighbors and the differences from the previous frame)
    4. If the color is different from the background (i.e. not alive or black so there is no point)
      1. Create the mesh from the passed in rect and call the graphics::draw() with the mesh

Is the any other way to do this? I don't think there is a way to avoid the iteration of the double for loops, but is there a way to reduce the amount of times I create a mesh and call draw?

Link to the full code: https://github.com/LelsersLasers/GameOfLife/blob/main/ggez/game_of_life/src/main.rs

Below are cut outs of the relevant draw code:

In impl event::EventHandler for Controller

fn draw(&mut self, context: &mut Context) -> GameResult { graphics::clear(context, graphics::Color::BLACK); self.draw_cells(context)?; graphics::present(context)?; Ok(()) } 

In my Controller "class":

fn draw_cells(&self, context: &mut Context) -> GameResult { let mut rect = graphics::Rect::new(-1.0, -1.0, SIZE, SIZE); for x in 0..WIDTH as usize { for y in 0..HEIGHT as usize { rect.move_to(ggez::mint::Point2 { x: x as f32 * (SIZE + SPACER) + SPACER, y: y as f32 * (SIZE + SPACER) + SPACER }); self.cells[x][y].draw(context, rect, self.draw_mode)?; } } Ok(()) } 

In my Cell "class"

fn draw(&self, context: &mut Context, rect: graphics::Rect, draw_mode: u8) -> GameResult { let mut color = graphics::Color::BLACK; if draw_mode == 0 { if self.alive { color = graphics::Color::new(218.0/255.0, 165.0/255.0, 32.0/255.0, 1.0); // "GOLDENROD" } } else if draw_mode == 1 { let colors = [ graphics::Color::BLACK, graphics::Color::GREEN, graphics::Color::RED, graphics::Color::BLUE ]; color = colors[self.status]; } else { if self.alive { color = graphics::Color::YELLOW; } else { let brightness = self.neighbors as f32/8.0; color = graphics::Color::new(brightness, brightness, brightness, 1.0); } } if color != graphics::Color::BLACK { let rect_mesh = graphics::Mesh::new_rectangle(context, graphics::DrawMode::fill(), rect, color)?; graphics::draw(context, &rect_mesh, graphics::DrawParam::default())?; if draw_mode == 3 && self.neighbors > 0 { let text = graphics::Text::new(self.neighbors.to_string()); graphics::draw(context, &text, graphics::DrawParam::default().dest(rect.point()))?; } } Ok(()) } 

Thanks in advance!

submitted by /u/LelsersLasers
[link] [comments]

Released Notan 0.4.0 and some info about future releases Programming Games in Rust

Programming Games in Rust2022-05-15 20:26:29 submitted by /u/Nazariglez
[link] [comments]

Press I To Inspect B̵̩̖͖̪̓̐̎̄͆̆̒̎͋̂̍̽̓̀̒͑͐̕̕l̷̨̨̜͓͚̼͓͇͎͕͚̖̲̹͎̠̩͈̱̳͍͎̰͕̟͎̫̃̌̀̊̍͐̓̓̇̅̋̒͌͋̏͋̓̊̚͜͝ỏ̷̧̡̭̦̭̺͔̼̳̠̭̭̹̭͉̬̱̭̬̖̽̒̔͂̋́̓͛̓̅̈́̋̓̇̃̈͠͝ͅợ̸̢̢̨̡̨̙̩͕̖̳͚̼͈̫̙͔̭͉͚͉͕́̍͝d̸̡̨̢̠̲̤̞̤̹̠̱͉̻͚̙̪͇̲̟͉̺̩͚̰̹͚̗̝̾͛͆͌̃͝ͅ ps1graphics

ps1graphics2022-05-15 18:38:37
Press I To Inspect B̵̩̖͖̪̓̐̎̄͆̆̒̎͋̂̍̽̓̀̒͑͐̕̕l̷̨̨̜͓͚̼͓͇͎͕͚̖̲̹͎̠̩͈̱̳͍͎̰͕̟͎̫̃̌̀̊̍͐̓̓̇̅̋̒͌͋̏͋̓̊̚͜͝ỏ̷̧̡̭̦̭̺͔̼̳̠̭̭̹̭͉̬̱̭̬̖̽̒̔͂̋́̓͛̓̅̈́̋̓̇̃̈͠͝ͅợ̸̢̢̨̡̨̙̩͕̖̳͚̼͈̫̙͔̭͉͚͉͕́̍͝d̸̡̨̢̠̲̤̞̤̹̠̱͉̻͚̙̪͇̲̟͉̺̩͚̰̹͚̗̝̾͛͆͌̃͝ͅ submitted by /u/Theguywholikestf2
[link] [comments]

Does any one know how to fix this screen issue on rust Programming Games in Rust

Programming Games in Rust2022-05-15 18:32:30
Does any one know how to fix this screen issue on rust submitted by /u/Typical1011
[link] [comments]

I'm writing a detailed devlog about my automation game Combine And Conquer written in Rust Programming Games in Rust

Programming Games in Rust2022-05-15 18:04:36 submitted by /u/i3ck
[link] [comments]

I'm happy to announce that Fyrox Game Engine 0.25 was released! This release adds static plugins and scripting support, improves prefab property inheritance, adds animation blending state machine, integrates sound entities to the scene graph. Programming Games in Rust

Programming Games in Rust2022-05-15 11:01:21
I'm happy to announce that Fyrox Game Engine 0.25 was released! This release adds static plugins and scripting support, improves prefab property inheritance, adds animation blending state machine, integrates sound entities to the scene graph. submitted by /u/_v1al_
[link] [comments]

rust game for web, gltf question Programming Games in Rust

Programming Games in Rust2022-05-15 01:57:57

I am a c++ gamedev and i made game engine on it, i am interested in learning rust and discovered that we can also make web games from it as well, though i havent started my study and r&d on it yet as i am completing a project now and plan to do it soon after, so i am throwing my questions in advance, my questions are

  1. when building a web game in rust, can we do webgl/opengl as well? what package/cargo to use?
  2. if the answer to above is yes, can we load animated gltf as well?

thank in advance and looking forward to joining you guys in making games in rust

submitted by /u/AgentCooderX
[link] [comments]

This Month in Rust GameDev #33 - April 2022 Programming Games in Rust

Programming Games in Rust2022-05-14 20:19:13
This Month in Rust GameDev #33 - April 2022 submitted by /u/_AngelOnFira_
[link] [comments]

Locker Room - WIP ps1graphics

ps1graphics2022-05-14 20:01:21
Locker Room - WIP submitted by /u/awhiskin
[link] [comments]

Epstein's Mansion on the moon in our MGS inspired game ps1graphics

ps1graphics2022-05-14 19:12:48
Epstein's Mansion on the moon in our MGS inspired game submitted by /u/Buce1
[link] [comments]

Kanpeki, our psychological horror game has now a demo out now on Itchio. Check it out here :https://wiloux.itch.io/kanpeki ps1graphics

ps1graphics2022-05-14 13:02:59
Kanpeki, our psychological horror game has now a demo out now on Itchio. Check it out here :https://wiloux.itch.io/kanpeki submitted by /u/StreetLightTeam
[link] [comments]

Gryazevichki - vehicle simulation prototype using Bevy and Rapier Programming Games in Rust

Programming Games in Rust2022-05-13 11:10:24
Gryazevichki - vehicle simulation prototype using Bevy and Rapier

cargo run

wheel sizes

Full Video

GitHub

Hello everyone! This is a prototype of vehicle simulator based on Rapier and Bevy written on Rust. inspired by Godot-6DOF-Vehicle-Demo.

i've been itching to try Rust for a long time and here's the outcome! Given both Bevy and Rapier clearly state that they are not ready for production i didn't expect much and yet for their age they are quite mature already!

State of Rapier v0.12.0-alpha.0 is surprisingly robust, the goal was to get a somewhat working vehicle without any tweaks from game code over the results of simulation using just rigid bodies, joints and motors (just like in 6DOF-Vehicle-Demo). Unlike Bullet (physics engine used in said demo), Rapier doesn't support 6 degrees of freedom joints yet, but with two chained "revolute" joints: one for wheel rotation(x-axis) and one for steering(y-axis), a stable and functional wheel was made! Making four of those and attaching them to a box scaled by z-axis made a nice wagon. Using motors for accelerating and steering was also quite intuitive, though i must admit that it took me some time to figure out all the axis and how to orient them in both frames of a joint.

Had to fix a couple of things in rapier: contact with kinematic didn't wake dynamic bodies and the same issue was with joint motors, they didn't wake attached bodies when enabled. i haven't made PRs for fixes yet, so for now gryazevichki uses my forks in Cargo.toml.

What you see on gifs and video is from branch gryazevichki_rapier_v0.12.0-alpha.0. Right by the time i achieved that behavior a new version of rapier and bevy_rapier got released so i bit the bullet and decided to update my dependencies too. i'd say it was a good decision, i refactored av0.12.0-alpha.0 lot of c++-like code into more rusty approach, but unfortunately something has got broken or changed in rapier and the same values i used in v0.12.0-alpha.0 don't work anymore, so i'll have to do an investigation and find out what went wrong.

Since it was my "vacation project" and the vacation is over i'll make some conclusions. As a c++ programmer by day i had my share of pain with borrow checker, with syntax a little bit (i was hoping there would be less boilerplate code, but hey, coming from c++ i have no right to bitch about that). The most painful process probably was changing paradigm of organizing memory. My first take was to make a struct that keeps all the info about vehicle within (aka config), but borrow checker made sure i don't do that. A good example would be gui code where i wanted to access config from multiple places. So after surviving the butthurt i compartmentalized my configs, made components from them and attached respective entities. After doing so both me and borrow checker felt better and app started building without errors. And to be honest i felt in love with bevy's ecs, i liked the ease of making new components and systems, it eliminates so much pain of figuring out the inner workings of an engine and every gameplay system you intend to meddle with. i had a good look of rapier's insides because i wanted to fix bugs(insides are pretty btw), can't say i had an urge to look inside bevy's sources, because there was no need for that. Only one bug i encountered, it's in legacy branch when you open body parameters camera loses entity, but after refactor it's gone so it could be just me doing something wrong.

submitted by /u/gavlig
[link] [comments]

Crates to build a cross platform multiplayer game? (+on the browser) Programming Games in Rust

Programming Games in Rust2022-05-12 23:13:48

these are what I have in mind for now:

winit - windowing + inputs

wgpu - graphics

quinn - Networking (waiting for webtransport for browser)

cpal - Audio

egui - Ui

parry2d/parry3d - collision + physics

SlotMap/bevy-ECS - logic layer

submitted by /u/CyberSoulWriter
[link] [comments]

my unban request Programming Games in Rust

Programming Games in Rust2022-05-12 16:37:01

hey game devs I understand that i was vac banned but i must state that I haven't logged in to this acc in a year so if you guys could check the ban date and if it's under a year or so give or take could you guys unban me rust has been my favorite game since i got it and i got locked out of my acc due to personal details i had to give up for a bit but i finally got in and first thing i did was download rust just to be banned it's a sad day for me but i just hope we could resolve things

submitted by /u/Aggravating_Cod_2368
[link] [comments]

The May Rust Gamedev meetup is this Saturday! Fill out the form in the comments if you have something neat to show off :) Programming Games in Rust

Programming Games in Rust2022-05-12 13:46:55
The May Rust Gamedev meetup is this Saturday! Fill out the form in the comments if you have something neat to show off :) submitted by /u/_AngelOnFira_
[link] [comments]

the difference between a CPU renderer and WebGpu Programming Games in Rust

Programming Games in Rust2022-05-11 19:58:58

my CPU renderer: https://www.youtube.com/watch?v=0IxnxlFDttE (1 spiral mesh at 10 fps)

WGPU renderer: https://www.youtube.com/watch?v=JuwBvYEBr1c (400 spirals at 60 fps)

each spiral has 80 000 triangles!

submitted by /u/CyberSoulWriter
[link] [comments]

Simple crate for 3d rendering and transforms Programming Games in Rust

Programming Games in Rust2022-05-07 09:21:21

tldr: what's the easiest way to render rectangular prisms with arbitrary layered rotations and translations but no textures and only basic shading?

I don't have any 3d game experience beyond some experimenting with VPython back in college, but I've got a decent understanding of linear algebra and rotation/translation matrices.

I'm working on a small project where I want to generate some geometry (rectangular prisms) and render them on screen with basic shading, similar to VPython but only prisms.

I just want to be able to create shapes with dimensions I choose then apply successive rotations and translations. This isn't grid based like minecraft, so I want to be able to define prisms in terms of w/l/h then rotate around the origin, translate, and rotate around the origin again.

I tried Nannou since I've used it before, but it looks like once you want more than basic 2d, you need to drop all the way down to wgpu, which is much lower level than I'd like.

Are there any Rust libraries or engines that would make this relatively easy? Or are there crates I could use to relatively easily translate/rotate points and triangles to export with obj_export and just view my shape in Blender or another program?

If you're curious about context, I've got a 3d PCB sculpture in mind and I want to generate the PCB shapes in Rust and make sure nothing will intersect and I get the geometry right before ordering circuit boards. The PCBs won't be orthagonal so the tabs won't line up nicely. I can easily generate the 2d board outlines, export them to KiCAD, and break them up into overlapping rectangles (to be extruded to 1mm thick prisms), and I'll be able to figure out what series of rotations and translations I need to orient my PCB shapes relative to each other, but I'm hoping I can find a relatively straightforward way to render these transformed rectangles (or at least check collision, but I'd rather do it visually).

submitted by /u/nagromo
[link] [comments]

Choosing a networking library for my game Programming Games in Rust

Programming Games in Rust2022-05-06 10:06:22

Hello,

I am currently developing a voxel based (raytraced) game in which the client is tightly coupled with the game server, even playing single player requires spinning up internal server (similar to minecraft).

For early development I just used TCP, but now I want to finally switch to UDP. I need to send unreliable packets (entity positions), big reliable packets (entity voxel data), reliable ordered packets (chat messages).

So far I found and checked out these libraries:

enet
- Golden standard, tested and reliable
- Not native rust
- Does not seem to be popular in rust

turbulence
- Readme says it is not stable, but last meaningful commit was 1 year ago
- Lacking documentation and examples
- Not very popular

laminar
- Last meaningful release was 3 years ago (ignoring changes that fix typos etc.)
- Despite this, everywhere (book, readme) there are mentions that it is under "active development"
- Created for Amethyst, which is dead. I am kinda fearful that the same thing will happen to this lib.

Tachyon
- New and not tested in the battle
- Many features that other libraries have planned, Tachyon actually has implemented
- Lacking documentation (except for one big readme file), examples

Quinn
- Big, under active development (daily commits), very popular because web development.
- Modular, ability to just use core implemetation: quinn-proto
- Do i really need TLS certificates and cryptography for my playing with friends game server?
- Stream based, I would need to implement recv/send messages on top of it (not that hard tbh)
- Only reliable stream and "unreliable" messages.

In conclusion, the more I read the more I don't know :) Currently I am at the point of deciding to go minecraft way: Just use TCP and deal with the problems later, even though I know this is a bad idea.

submitted by /u/SkewPL
[link] [comments]

The start of a tool that lets you tweak how and where trees and foliage automatically populate, what do you think? New Dims dev log 🦀 Programming Games in Rust

Programming Games in Rust2022-05-06 07:07:36
The start of a tool that lets you tweak how and where trees and foliage automatically populate, what do you think? New Dims dev log 🦀 submitted by /u/5dims
[link] [comments]

New to Game Dev Programming Games in Rust

Programming Games in Rust2022-05-03 12:49:26

I've just recently finished an introductory course in Rust at my university and I can honestly see myself working with Rust as a career, having come from a Java/C++ background. This summer I am working on a Master's project and I had an idea to develop a few simple 2D games in Rust. Just classic games everyone is familiar with, like Tetris, Snake, Battleship, etc.

I've started looking into game engines available, but I'm not sure which one is best for my situation, since the project's development needs to start in a couple weeks and I only just now find enough time to sit down and look into how I'm going to do it. I've made games before, graphical and text-based, in Java and C++. I consider the above game examples to be "simple" 2D games, so can anyone recommend a good engine for me to use? I would prefer something with good documentation.

submitted by /u/Frigid-Inferno
[link] [comments]

I won free load testing fasterthanli.me

fasterthanli.me2022-05-02 01:20:00

Long story short: a couple of my articles got really popular on a bunch of sites, and someone, somewhere, went "well, let's see how much traffic that smart-ass can handle", and suddenly I was on the receiving end of a couple DDoS attacks.


[piston] How can I get mouse coordinates from Event while not moving the mouse? (beginner) Programming Games in Rust

Programming Games in Rust2022-05-01 15:21:54

I've made a simple Conway's GOL and wanted to add some functions on mouse click and I need its coordinates. I found mouse_cursor_args, but it keeps returning None unless I move the mouse, which is not ideal.

submitted by /u/Thers_VV
[link] [comments]

Lies we tell ourselves to keep using Golang fasterthanli.me

fasterthanli.me2022-04-29 13:40:00

In the two years since I've posted I want off Mr Golang's Wild Ride, it's made the rounds time and time again, on Reddit, on Lobste.rs, on HackerNews, and elsewhere.

And every time, it elicits the same responses:

  • You talk about Windows: that's not what Go is good at! (Also, who cares?)
  • This is very one-sided: you're not talking about the good sides of Go!
  • You don't understand the compromises Go makes.
  • Large companies use Go, so it can't be that bad!
  • Modelling problems "correctly" is too costly, so caring about correctness is moot.
  • Correctness is a spectrum, Go lets you trade some for development speed.
  • Your go-to is Rust, which also has shortcomings, so your argument is invalid.
  • etc.

Getting good at SNES games through DLL injection fasterthanli.me

fasterthanli.me2022-04-25 00:00:00

Are you ever confronted with a problem and then think to yourself "wait a minute, I know how to code?" — that's exactly what happened there.

In this video, we keep dying in games emulated with Snes9X, and decide it's too much work to take our hands from the controller to the keyboard and back to save and load the game state.


I'm in ur address space fasterthanli.me

fasterthanli.me2022-04-11 00:00:00

Hey Notepad! Nice process you got there. Would be a shame if someone were to... butt in.

In this video, we learn about applications and processes and threads, and use Win32 APIs to create a remote thread in another process, running into all kinds of complications on the way there.


Futures Nostalgia fasterthanli.me

fasterthanli.me2022-04-03 00:03:00

Up until recently, hyper was my favorite Rust HTTP framework. It's low-level, but that gives you a lot of control over what happens.

Here's what a sample hyper application would look like:

Shell session
$ cargo new nostalgia
     Created binary (application) `nostalgia` package
Shell session
$ cd nostalgia
$ cargo add hyper@0.14 --features "http1 tcp server"
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding hyper v0.14 to dependencies with features: ["http1", "tcp", "server"]
$ cargo add tokio@1 --features "full"
    Updating 'https://github.com/rust-lang/crates.io-index' index
      Adding tokio v1 to dependencies with features: ["full"]
Rust code
use std::{
    convert::Infallible,
    future::{ready, Ready},
    task::{Context, Poll},
};

use hyper::{server::conn::AddrStream, service::Service, Body, Request, Response, Server};

#[tokio::main]
async fn main() {
    Server::bind(&([127, 0, 0, 1], 1025).into())
        .serve(MyServiceFactory)
        .await
        .unwrap();
}

struct MyServiceFactory;

impl Service<&AddrStream> for MyServiceFactory {
    type Response = MyService;
    type Error = Infallible;
    type Future = Ready<Result<Self::Response, Self::Error>>;

    fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        Ok(()).into()
    }

    fn call(&mut self, req: &AddrStream) -> Self::Future {
        println!("Accepted connection from {}", req.remote_addr());
        ready(Ok(MyService))
    }
}

struct MyService;

impl Service<Request<Body>> for MyService {
    type Response = Response<Body>;
    type Error = Infallible;
    type Future = Ready<Result<Self::Response, Self::Error>>;

    fn poll_ready(&mut self, _cx: &mut Context<'_>) -> Poll<Result<(), Self::Error>> {
        Ok(()).into()
    }

    fn call(&mut self, req: Request<Body>) -> Self::Future {
        println!("Handling {req:?}");
        ready(Ok(Response::builder().body("Hello World!\n".into()).unwrap()))
    }
}
Shell session
$ cargo run
cargo run
   Compiling nostalgia v0.1.0 (/home/amos/bearcove/nostalgia)
    Finished dev [unoptimized + debuginfo] target(s) in 1.20s
     Running `target/debug/nostalgia`
(server runs peacefully in the background)

Request coalescing in async Rust fasterthanli.me

fasterthanli.me2022-03-06 18:00:00

As the popular saying goes, there are only two hard problems in computer science: caching, off-by-one errors, and getting a Rust job that isn't cryptocurrency-related.

Today, we'll discuss caching! Or rather, we'll discuss... "request coalescing", or "request deduplication", or "single-flighting" - there's many names for that concept, which we'll get into fairly soon.


A Rust match made in hell fasterthanli.me

fasterthanli.me2022-02-11 07:30:00

I often write pieces that showcase how well Rust can work for you, and how it can let you build powerful abstractions, and prevents you from making a bunch of mistakes.

If you read something like Some mistakes Rust doesn't catch in isolation, it could seem as if I had only nice things to say about Rust, and it's a perfect little fantasy land where nothing ever goes wrong.


Some mistakes Rust doesn't catch fasterthanli.me

fasterthanli.me2022-02-07 07:30:00

I still get excited about programming languages. But these days, it's not so much because of what they let me do, but rather what they don't let me do.

Ultimately, what you can with a programming language is seldom limited by the language itself: there's nothing you can do in C++ that you can't do in C, given infinite time.


Computers as a social construct fasterthanli.me

fasterthanli.me2022-02-05 07:30:00

Those who learn from history are doomed to repeat that fucking proverb.


Messing with the recipe fasterthanli.me

fasterthanli.me2022-01-10 07:30:00

What if we learned just enough to be a little dangerous?


Profiling linkers fasterthanli.me

fasterthanli.me2022-01-03 07:30:00

In the wake of Why is my Rust build so slow?, developers from the mold and lld linkers reached out, wondering why using their linker didn't make a big difference.

Of course the answer was "there's just not that much linking to do", and so any difference between mold and lld was within a second. GNU ld was lagging way behind, at four seconds or so.


One funny way to bundle assets fasterthanli.me

fasterthanli.me2022-01-02 07:30:00

There's one thing that bothers me. In part 1, why are we using hyper-staticfile? Couldn't we just use file:/// URLs?

Well, first off: showing off how easy it is to serve some static files, even in a "scary" language like Rust, is just not something I could pass up.


The rest of the fucking owl fasterthanli.me

fasterthanli.me2021-12-31 07:37:00

NO! No no no.

What?

WE WERE DONE!

Well... yes! But also no. We still shell out to a bunch of tools:

Shell session
$ rg 'Command::new'
src/commands/mod.rs
126:        let variant = if let Ok(output) = run_command(Command::new("wslpath").arg("-m").arg("/")) {

src/commands/cavif.rs
29:            Command::new("cavif")

src/commands/imagemagick.rs
25:            Command::new(&self.bin)

src/commands/cwebp.rs
25:            Command::new("cwebp")

src/commands/svgo.rs
25:            Command::new("svgo")
Cool bear's hot tip

Productionizing our poppler build fasterthanli.me

fasterthanli.me2021-12-31 07:36:00

I was a bit anxious about running our poppler meson build in CI, because it's the real test, you know? "Works on my machine" only goes so far, things have a tendency to break once you try to make them reproducible.


Porting poppler to meson fasterthanli.me

fasterthanli.me2021-12-31 07:35:00

It took a hot minute.

Try several weeks.

Well, yeah. I got to contribute to a bunch of open-source projects in the meantime though, so I'm fairly pleased with it!

  • libffi (for static linking)
  • cairo (more static linking!)
  • proxy-libintl (more static linking!)
  • expat (static linking strikes again)
  • poppler (for file descriptor stuff not properly gated on Windows, closed in favor of a similar MR)
  • glib (work in progress, involves thread-local storage and dynamic linker/loader cursedness)

Building poppler for Windows fasterthanli.me

fasterthanli.me2021-12-31 07:34:00

I know what you're thinking: haven't we strayed from the whole "content pipeline" theme in this series?

Well... fair. But compiling and distributing software is part of software engineering, and unless you're in specific circles, I see that taught a lot less than the "just write code and stuff happens" part.


A static poppler build: the easy way fasterthanli.me

fasterthanli.me2021-12-31 07:33:00

So! Now our asset processing pipeline is almost complete. But we've just traded dependencies against CLI tools, for dependencies against dynamic libraries:

Shell session
$ ldd ./target/debug/pdftocairo
        linux-vdso.so.1 (0x00007ffd615be000)
        libpoppler-glib.so.8 => /lib64/libpoppler-glib.so.8 (0x00007f2ba1bb4000)
        libgobject-2.0.so.0 => /lib64/libgobject-2.0.so.0 (0x00007f2ba1b59000)
        libglib-2.0.so.0 => /lib64/libglib-2.0.so.0 (0x00007f2ba1a1e000)
        libcairo.so.2 => /lib64/libcairo.so.2 (0x00007f2ba1902000)
        libcairo-gobject.so.2 => /lib64/libcairo-gobject.so.2 (0x00007f2ba18f6000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f2ba18dc000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f2ba17fe000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f2ba15f4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f2ba216c000)
        libpoppler.so.112 => /lib64/libpoppler.so.112 (0x00007f2ba1288000)
        libfreetype.so.6 => /lib64/libfreetype.so.6 (0x00007f2ba11bd000)
        libgio-2.0.so.0 => /lib64/libgio-2.0.so.0 (0x00007f2ba0fe4000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f2ba0dc5000)
        libffi.so.6 => /lib64/libffi.so.6 (0x00007f2ba0db8000)
        libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f2ba0d40000)
        libpixman-1.so.0 => /lib64/libpixman-1.so.0 (0x00007f2ba0c94000)
        libfontconfig.so.1 => /lib64/libfontconfig.so.1 (0x00007f2ba0c45000)
        libpng16.so.16 => /lib64/libpng16.so.16 (0x00007f2ba0c0c000)
        libxcb-shm.so.0 => /lib64/libxcb-shm.so.0 (0x00007f2ba0c07000)
        libxcb.so.1 => /lib64/libxcb.so.1 (0x00007f2ba0bda000)
        libxcb-render.so.0 => /lib64/libxcb-render.so.0 (0x00007f2ba0bca000)
        libXrender.so.1 => /lib64/libXrender.so.1 (0x00007f2ba0bbd000)
        libX11.so.6 => /lib64/libX11.so.6 (0x00007f2ba0a75000)
        libXext.so.6 => /lib64/libXext.so.6 (0x00007f2ba0a60000)
        libz.so.1 => /lib64/libz.so.1 (0x00007f2ba0a46000)
        libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f2ba09c2000)
        libopenjp2.so.7 => /lib64/libopenjp2.so.7 (0x00007f2ba0968000)
        liblcms2.so.2 => /lib64/liblcms2.so.2 (0x00007f2ba0903000)
        libtiff.so.5 => /lib64/libtiff.so.5 (0x00007f2ba087c000)
        libsmime3.so => /lib64/libsmime3.so (0x00007f2ba0850000)
        libnss3.so => /lib64/libnss3.so (0x00007f2ba0712000)
        libplc4.so => /lib64/libplc4.so (0x00007f2ba0709000)
        libnspr4.so => /lib64/libnspr4.so (0x00007f2ba06c6000)
        libbz2.so.1 => /lib64/libbz2.so.1 (0x00007f2ba06b3000)
        libharfbuzz.so.0 => /lib64/libharfbuzz.so.0 (0x00007f2ba05dd000)
        libbrotlidec.so.1 => /lib64/libbrotlidec.so.1 (0x00007f2ba05cf000)
        libgmodule-2.0.so.0 => /lib64/libgmodule-2.0.so.0 (0x00007f2ba05c8000)
        libmount.so.1 => /lib64/libmount.so.1 (0x00007f2ba0581000)
        libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f2ba0556000)
        libxml2.so.2 => /lib64/libxml2.so.2 (0x00007f2ba03cd000)
        libXau.so.6 => /lib64/libXau.so.6 (0x00007f2ba03c7000)
        libwebp.so.7 => /lib64/libwebp.so.7 (0x00007f2ba0358000)
        libzstd.so.1 => /lib64/libzstd.so.1 (0x00007f2ba0260000)
        libjbig.so.2.1 => /lib64/libjbig.so.2.1 (0x00007f2ba0252000)
        libnssutil3.so => /lib64/libnssutil3.so (0x00007f2ba021f000)
        libplds4.so => /lib64/libplds4.so (0x00007f2ba021a000)
        libgraphite2.so.3 => /lib64/libgraphite2.so.3 (0x00007f2ba01f9000)
        libbrotlicommon.so.1 => /lib64/libbrotlicommon.so.1 (0x00007f2ba01d4000)
        libblkid.so.1 => /lib64/libblkid.so.1 (0x00007f2ba019c000)
        libpcre2-8.so.0 => /lib64/libpcre2-8.so.0 (0x00007f2ba0105000)
        liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f2ba00d9000)

From Inkscape to poppler fasterthanli.me

fasterthanli.me2021-12-31 07:32:00

What's next? Well... poppler is the library Inkscape uses to import PDFs.

Cool bear's hot tip

Yes, the name comes from Futurama.

Turns out, poppler comes with a bunch of CLI tools, including pdftocairo!


Truly headless draw.io exports fasterthanli.me

fasterthanli.me2021-12-31 07:31:00

I love diagrams. I love them so much! In fact, I have fairly poor visualization skills, so making a diagram is extremely helpful to me: I'll have some vague idea of how different things are connected, and then I'll make a diagram, and suddenly there's a tangible thing I can look at and talk about.


Why is my Rust build so slow? fasterthanli.me

fasterthanli.me2021-12-30 21:50:00

I've recently come back to an older project of mine (that powers this website), and as I did some maintenance work: upgrade to newer crates, upgrade to a newer rustc, I noticed that my build was taking too damn long!


This Year in Wgpu - 2021 gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-12-25 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. Our main projects are:

  • wgpu is built on top of wgpu-hal and naga. It provides safety, accessibility, and portability for graphics applications.
  • naga translates shader programs between languages, including WGSL. It also provides shader validation and transformation, ensuring user code running on the GPU is safe and efficient.

As 2021 comes to an end, let’s look back at everything that has been accomplished.

Fredrik Norén's terrain with trees

Wgpu

We moved from gfx-hal to the newly created wgpu-hal and restructured the repository to keep everything together. At the same time, we dropped SPIRV-Cross in favor of naga, reaching the pure-rust tech stack. Read more in the 0.10 release post. Credit goes to @kvark.

At the same time, @cwfitzgerald has revamped our testing infrastructure with Rust integration tests and example snapshots. On top of that, wgpu has tightly integrated with Deno (thanks to the effort of Deno team!), opening up the road to testing on a real CTS, which is available in CI now.

One shiny highlight of the year was the WebGL port, which became practically usable. Getting it ready was truly a collaborative effort, kicked off by @zicklag. Today, wgpu-rs examples can be run online with WebGL.

In terms of correctness and portability, @Wumpf landed the titanic work of ensuring all our resources are properly zero-initialized. This has proven to be much more involved than it seems, and now users will get consistent behavior across platforms.

Finally, we just released version 0.12 with the fresh and good stuff!

Naga

Naga grew more backends (HLSL, WGSL) and greatly improved support all around the table. It went from an experimental prototype in 0.3 to production, shipping in Firefox Nightly. It proved to be 4x faster than SPIRV-Cross at SPV->MSL translation.

One notable improvement, led by @JCapucho with some help from @jimblandy, is the rewrite of SPIR-V control flow processing. This has been a very problematic and complicated area in past, and now it’s mostly solved.

Things have been busy on GLSL frontend as well. It got a completely new parser thanks to @JCapucho, which made it easier to improve and maintain.

Validation grew to cover all the expressions and types and everything. For some time, it was annoying to see rough validation errors without any reference to the source. But @ElectronicRU saved the day by making our errors really nice, similar to how WGSL parser errors were made pretty by @grovesNL work earlier.

Last but not the least, SPIR-V and MSL backends have been bullet-proofed by @jimblandy. This includes guarding against out-of-bounds accesses on arrays, buffers, and textures.

Future Work

One big project that hasn’t landed is the removal of “hubs”. This is a purely internal change, but a grand one. It would streamline our policy of locking internal data and allow the whole infrastructure to scale better with more elaborate user workloads. We hope to see it coming in 2022.

Another missing piece is DX11 backend. We know it’s much needed, and it was the only regression from the wgpu-hal port. This becomes especially important now as Intel stopped supporting DX12 on its Haswell GPUs.

Overall, there’s been a lot of good quality contributions, and this list by no means can describe the depth of it. We greatly appreciate all the improvements and would love to shout out about your work at the earliest opportunity. Big thanks for everybody involved!


PSA about NFT's Procedural generation

Procedural generation2021-11-29 23:32:20

We are really, really casual about the content we allow here. The rules are pretty loose because procgen comes in many shapes and forms and is often in the eye of the beholder. We love to see your ideas and content.

NFT's are not procedural generation. They might point to something you generated using techniques we all know and love here, but they themselves are not.

This post is not for a debate about the merit, value, utility or otherwise of NFT's. It's just an announcement that this subreddit is for the content that they may point to.

Do share the content if you generated it, do tell use how you made it, do be excited about the work you put into it.

Do not share links to places where NFT's of your work can be bought.
Do not tell us how much you sold it for.

In the same way we would remove a post saying "Hey guys my procgen game is doing mad numbers on steam" we will also remove posts talking about how much money people paid for an NFT of your work.

Please report any posts you see to help us out.

submitted by /u/Bergasms
[link] [comments]

My ideal Rust workflow fasterthanli.me

fasterthanli.me2021-10-26 18:30:00

Writing Rust is pretty neat. But you know what's even neater? Continuously testing Rust, releasing Rust, and eventually, shipping Rust to production. And for that, we want more than plug-in for a code editor.


PGC - Aquarium/Terrarium Voting + October Spooky Challenge Procedural generation

Procedural generation2021-10-09 10:27:40

Wheee, sorry this is a bit late. Consider this a dual voting and challenge brief. First up, the previous challenge submissions are as follows. Link to challenge thread

User Thread Example
watawatabou Comment Final Submission With Examples
Orenog Comment Final Submission With Examples
simiansays Comment See comment
Paulluuk Comment See comment
Bergasms Comment Final Submission comment
Gogodinosaur Comment See comment
datta_sid Comment See comment
MetaGlitch Comment Final comment

I will not open voting yet as I suspect I might have missed some submissions that were done as top level posts recently, so I will wait a day or so and then open voting, I'll sticky the voting link as well as a reminder in another post.

Finally, on to the quick and cheeky next challenge that was determined by previous challenges winners. Generate something spoooooky for halloween. This could be a halloween costume, a spooky scary skeleton, a front yard of a house with scary things in it, or something else to terrify us all. Deadline is the end of this month.

voting link

submitted by /u/Bergasms
[link] [comments]

Release of wgpu v0.11 and naga v0.7 gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-10-07 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. Our main projects are:

  • wgpu is built on top of wgpu-hal and naga. It provides safety, accessibility, and portability for graphics applications.
  • naga translates shader programs between languages, including WGSL. It also provides shader validation and transformation, ensuring user code running on the GPU is safe and efficient.

Following our release cadence of every few months, we rolled out v0.11 through all of the gfx-rs projects! See wgpu v0.11 changelog and naga v0.7 changelog for the details.

This is our second release using our pure rust graphics stack. We’ve made a significant progress with shader translation and squashed many bugs in both wgpu and the underlying abstraction layer.

WebGL2

Thanks to the help of @Zicklag for spearheading the work on the WebGL2 backend. Through modifying the use of our OpenGL ES backend, they got WebGL2 working on the web. The backend is still in beta, so please test it out and file bugs! See the guide to running on the web for more information.

The following shows one of Bevy’s PBR examples running on the web.

bevy running on webgl2

Explicit Presentation

A long standing point of confusion when using wgpu was that dropping the surface frame caused presentation. This was confusing and often happened implicitly. With this new version, presentation is now marked explicitly by calling frame.present(). This makes very clear where the important action of presentation takes place.

More Robust Shader Translation

naga has made progress on all frontends and backends.

The most notable change was that @JCapucho, with the help of @jimb, completely rewrote the parsing of spirv’s control flow. spirv has notably complex control flow which has a large number of complicated edge cases. After multiple reworks, we have settled on this new style of control flow graph parsing. If you input spirv into wgpu, this will mean that even more spirv, especially optimized spirv, will properly validate and convert.

See the changelog for all the other awesome editions to naga.

Thank You!

Thanks to the countless contributors that helped out with this release! wgpu and naga’s momentum is truely incredible due to everyone’s contributions and we look forward to seeing the amazing places wgpu and naga will go as projects. If you are interested in helping, take a look at our good-first-issues, our issues with help wanted, or contact us on our matrix chat, we are always willing to help mentor first time and returning contributors.

Additionally, thank you to all the users who report new issues, ask for enhancements, or test the git version of wgpu. Keep it coming!

Happy rendering!


A terminal case of Linux fasterthanli.me

fasterthanli.me2021-09-24 18:45:00

Has this ever happened to you?

You want to look at a JSON file in your terminal, so you pipe it into jq so you can look at it with colors and stuff.

Cool bear's hot tip

wgpu alliance with Deno gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-09-16 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. Our main projects are:

  • wgpu is built on top of wgpu-hal and naga. It provides safety, accessibility, and portability for graphics applications.
  • naga translates shader programs between languages, including WGSL. It also provides shader validation and transformation, ensuring user code running on the GPU is safe and efficient.

wgpu works over native APIs, such as Vulkan, D3D12, Metal, and others. This involves a layer of translation to these APIs, which is generally straightforward. It promises safety and portability, so it’s critical for this library to be well tested. To this date, our testing was a mix of unit tests, examples, and a small number of integration tests. Is this going to be enough? Definitely no!

Fortunately, WebGPU is developed with a proper Conformance Test Suite (CTS), largely contributed by Google to date. It’s a modern test suite covering all of the API parts: API correctness, validation messages, shader functionality, feature support, etc. The only complication is that it’s written in TypeScript against the web-facing WebGPU API, while wgpu exposes a Rust API.

Deno

We want to be sure that the parts working today will keep working tomorrow, and ideally enforce this in continuous integration, so that offending pull requests are instantly detected. Thus, we were looking for the simplest way to bridge wgpu with TS-based CTS, and we found it.

Back in March Deno 1.8 shipped with initial WebGPU support, using wgpu for implementing it. Deno is a secure JS/TS runtime written in Rust. Using Rust from Rust is :heart:! Deno team walked the extra mile to hook up the CTS to Deno WebGPU and run it, and they reported first CTS results/issues ever on wgpu.

Thanks to Deno’s modular architecture, the WebGPU implementation is one of the pluggable components. We figured that it can live right inside wgpu repository, together with the CTS harness. This way, our team has full control of the plugin, and can update the JS bindings together with the API changes we bring from the spec.

Today, WebGPU CTS is fully hooked up to wgpu CI. We are able to run the white-listed tests by the virtue of adding “needs testing” tag to any PR. We are looking to expand the list of passing tests and eventually cover the full CTS. The GPU tests actually run on github CI, using D3D12’s WARP software adapter. In the future, we’ll enable Linux testing with lavapipe for Vulkan and llvmpipe for GLES as well. We are also dreaming of a way to run daemons on our working (and idle) machines that would pull revisions and run the test suite on real GPUs. Please reach out if you are interested in helping with any of this :wink:.

Note that Gecko is also going to be running WebGPU CTS on its testing infrastructure, independently. The expectation is that Gecko’s runs will not show any failures on tests enabled on our CI based on Deno, unless the failures are related to Gecko-specific code, thus making the process of updating wgpu in Gecko painless.

We love the work Deno is doing, and greatly appreciate the contribution to wgpu infrastructure and ecosystem! Special thanks to Luca Casonato and Leo K for leading the effort :medal_military:.


Release of a Pure-Rust v0.10 and a Call For Testing gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-08-18 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. Our main projects are:

  • wgpu is built on top of wgpu-hal and naga. It provides safety, accessibility, and portability for graphics applications.
  • naga translates shader programs between languages, including WGSL. It also provides shader validation and transformation, ensuring user code running on the GPU is safe and efficient.

If you’ve been following these releases you’ll notice that gfx-hal is absent from this list. gfx-hal has now been deprecated in favor of a new abstraction layer inside of wgpu called wgpu-hal. To see more information about the deprecation, see the 0.9 release post.

Following our release cadence every few months, we rolled out 0.10 through all of the gfx-rs projects! See wgpu v0.10 changelog and naga v0.6 changelog for the details.

Pure-Rust Graphics

wgpu has had many new changes, the most notible of which is the switch to our new Hardware Abstraction Layer wgpu-hal. This includes completely rebuilt backends which are more efficient, easier to maintain, and signifigantly leaner. As part of this, we have shed our last C/C++ dependency spirv-cross. We now are entirely based on naga for all of our shader translation. This is not only a marked achievement for rust graphics, but has made wgpu safer and more robust.

The new wgpu-hal:

  • Supports Vulkan, D3D12, Metal, and OpenGL ES with D3D11 to come soon.
  • Has 60% fewer lines of code than gfx-hal (22k LOC vs 55k)
  • Maps better to the wide variety of backends we need to support.

Other notable changes within wgpu:

  • Many api improvements and bug fixes.
  • New automated testing infrastructure.

naga has continued to matured significantly since the last release:

  • hlsl output is now supported and working well.
  • wgsl parsing has had numerous bugs fixed.
  • spirv parsing support continues to be very difficult but improving steadily.
  • With wgpu-hal now dependending on naga, all code paths have gotten signifigant testing.
  • Validation has gotten more complete and correct.

Call For Testing

This is an extremely big release for us. While we have confidence in our code and we have tested it extensively, we need everyone’s help in testing this new release! As such we ask if people can update to the latest wgpu and report to us any problems or issues you face.

If you aren’t sure if something is an issue, feel free to hop on our matrix chat to discuss.

Thank You!

Thanks to the countless contributors that helped out with this massive release! wgpu’s momentum is truely incredible due to everyone’s contributions and we look forward to seeing the amazing places wgpu will go as a project. If you are interested in helping, take a look at our good-first-issues, our issues with help wanted, or contact us on our matrix chat, we are always willing to help mentor first time and returning contributors.

Additionally, thank you to all the users who report new issues, ask for enhancements, or test the git version of wgpu. Keep it coming!

Happy rendering!


Understanding Rust futures by going way too deep fasterthanli.me

fasterthanli.me2021-07-25 22:00:00

So! Rust futures! Easy peasy lemon squeezy. Until it's not. So let's do the easy thing, and then instead of waiting for the hard thing to sneak up on us, we'll go for it intentionally.

Cool bear's hot tip

Release of v0.9 and the Future of wgpu gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-07-16 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. Our current main projects are:

  • gfx-rs makes low-level GPU programming portable with low overhead. It’s a single Vulkan-like Rust API with multiple backends that implement it: Direct3D 12/11, Metal, Vulkan, and even OpenGL ES.
  • naga translates the shaders between languages, including WGSL. Also provides validation and processing facilities on the intermediate representation.
  • wgpu is built on top of gfx-rs and gpu-alloc/gpu-descriptor. It provides safety, accessibility, and strong portability of applications.

Following our release cadence every few months, we rolled out the 0.9 version through all of the gfx projects! See gfx-rs changelog, wgpu changelog, and naga changelog for the details.

naga has matured significantly since the last release.

  • wgsl parsing has improved incredibly, targeting an up-to-date spec.
  • spirv parsing support has had numerous bugs fixed.
  • glsl support is starting to take shape, though still in an alpha state.
  • Validation has gotten more complete and correct.

wgpu validation has continued to improve. Many validation holes were plugged with the last release. Through the combined work in wgpu and naga, validation holes have been sured up, and new features have been implemented. One such feature is getting the array length of runtime-sized arrays, which is now properly implemented on metal.

wgpu performance is still a vital target for us, so we have done work on improving the overhead of resource tracking. We’ve reduced unnecessary overhead through only doing stateful tracking for resources that have complex states. These changes were made from benchmarks of Gecko’s WebGPU implementation which showed that tracking was a bottleneck. You can read more about it #1413.

wgpu Family Reunion, Relicense, and the Future

wgpu has had a number of large internal changes which are laying the future for wgpu to be a safe, efficient, and portable api for doing cross-platform graphics.

wgpu has been relicensed from MPL-2.0 to MIT/Apache-2.0. Thank you to all 142 people who replied to the issue and made this happen. This relicense is an important change because it allows the possibility of adding backends targeting APIs which are behind NDAs.

For a while, we acknowledged that having different essential parts of the project living in different repositories was hurting developers productivity. There were objective reasons for this, but the time has come to change that. Feedback from our friends at the Bevy game engine gave us the final push and we launched an initiative to make wgpu easier to contribute to. We moved wgpu-rs back into the wgpu repo. This means that PRs that touch both the core crate and the rust bindings no longer need multiple PRs that need to be synchronized. We have already heard from collaborators how much easier the contribution is now that there is less coordination to do. Read more about the family reunion.

As a part of our family reunion, 0.9 is going to be the last release that will use gfx-hal as its hardware abstraction layer. While it has served us well, it has proved to not be at the exact level of abstraction we need. We have started work on a new abstraction layer called wgpu-hal. This new abstraction has already had Vulkan, Metal, and GLES ported, with DX12 landed in an incomplete state, and DX11 to come soon. To learn more about this transition, you can read the whole discussion.

Finally, we have brand new testing infrastructure that allows us to automatically test across all backends and all adapters in the system. Included in our tests are image comparison tests for all of our examples and the beginnings of feature tests. We hope to expand this to cover a wide variety of features and use cases. We will be able to run these tests in CI on software adapters and our future goal is to setting up a distributed testing network so that we can automatically test on a wide range of adapters. This will be one important layer of our in-depth defences, ensuring that wgpu is actually portable and safe. Numerous bugs have already been caught by this new infrastructure thus far and it will help us prevent regressions in the future. Read more about our testing infrastructure.

Thank You!

Thank you for the countless contributors that helped out with this release! wgpu’s momentum is only increasing due to everyone’s contributions and we look forward to seeing the amazing places wgpu will go as a project. If you are interested in helping, take a look at our good-first-issues, our issues with help wanted, or contact us on our matrix chat, we are always willing to help mentor first time and returning contributors.

Additionally, thank you to all the users who report new issues, ask for enhancements, or test the git version of wgpu. Keep it coming!

Happy rendering!


The Uber of Poland Matthias Endler

Matthias Endler2021-06-14 00:00:00

A few years ago I visited a friend in Gdańsk, Poland. As we explored the city, one thing I noticed was that cabs were relatively expensive and there was no Uber. Instead, most (young) people used a community-organized service called Night Riders.

I couldn't find anything about that service on the web, so I decided to write about it to preserve its history.

Delightfully Low-Tech

What fascinated me about Night Riders was the way the service operated — completely via WhatsApp: you post a message in a group chat and one of the free riders would reply with a 👍 emoji. With that, your ride was scheduled. You'd pay through PayPal or cash.

In these days of venture-backed startups that need millions in capital before they turn a profit, this approach is decidedly antagonistic. Basically, Night Riders built on top of existing infrastructure instead of maintaining their own ride-hailing platform, sign-up process, or even website.

The service would grow solely by word of mouth. Using existing infrastructure meant that it was extremely cheap to run and there were almost zero upfront costs without a single line of code to write.

It simply solved the customer's problem in the most straightforward way possible. Of course, there are legal issues regarding data protection, labor law or payment processing, but the important bit is that they had paying customers from day one. The rest is easier to solve than a lack of product market fit.

In Defense of Clones

Uber and Lyft can't be everywhere from the start. While they expand their businesses, others have the ability to outpace them. There's an Uber clone in China (DiDi), one in Africa and the Middle East (Careem) and basically one for every country in the world. The tech industry rarely talks about these Ubers of X, but they serve millions of customers. While they start as exact copies of their well-known counterparts, some of them end up offering better service thanks to their understanding of the local market.

People always find a way

With creativity, you can provide great service even without a big budget. The important part is to know which corners you can cut while staying true to your mission. If there's a market, there's a way. The Cubans have a word for it: resolver, which means "we'll figure it out".


How Does The Unix `history` Command Work? Matthias Endler

Matthias Endler2021-05-31 00:00:00
Cozy attic created by [vectorpouch](https://www.freepik.com/vectors/poster) and tux created by [catalyststuff](https://www.freepik.com/vectors/baby) &mdash; freepik.com
Source: Cozy attic created by vectorpouch and tux created by catalyststuff — freepik.com

As the day is winding down, I have a good hour just to myself. Perfect time to listen to some Billie Joel (it's either Billie Joel or Billie Eilish for me these days) and learn how the Unix history command works. Life is good.

Learning what makes Unix tick is a bit of a hobby of mine.
I covered yes, ls, and cat before. Don't judge.

How does history even work?

Every command is tracked, so I see the last few commands on my machine when I run history.

❯❯❯ history
8680  cd endler.dev
8682  cd content/2021
8683  mkdir history
8684  cd history
8685  vim index.md

Yeah, but how does it do that?

The manpage on my mac is not really helpful — I also couldn't find much in the first place.

I found this article (it's good etiquette nowadays to warn you that this is a Medium link) and it describes a bit of what's going on.

Every command is stored in $HISTFILE, which points to ~/.zsh_history for me.

❯❯❯ tail $HISTFILE
: 1586007759:0;cd endler.dev
: 1586007763:0;cd content/2021
: 1586007771:0;mkdir history
: 1586007772:0;cd history
: 1586007777:0;vim index.md
...

So let's see. We got a : followed by a timestamp followed by :0, then a separator (;) and finally the command itself. Each new command gets appended to the end of the file. Not too hard to recreate.

Hold on, what's that 0 about!?

It turns out it's the command duration, and the entire thing is called the extended history format:

: <beginning time>:<elapsed seconds>;<command>

(Depending on your settings, your file might look different.)

Hooking into history

But still, how does history really work.

It must run some code whenever I execute a command — a hook of some sort!

💥 Swoooooosh 💥

Matthias from the future steps out of a blinding ball of light: Waaait! That's not really how it works!

It turns out that shells like bash and zsh don't actually call a hook for history. Why should they? When history is a shell builtin, they can just track the commands internally.

Thankfully my editor-in-chief and resident Unix neckbeard Simon Brüggen explained that to me — but only after I sent him the first draft for this article. 😓

As such, the next section is a bit like Lord of the Rings: a sympathetic but naive fellow on a questionable mission with no clue of what he's getting himself into.

In my defense, Lord of the Rings is also enjoyed primarily for its entertainment value, not its historical accuracy.... and just like in this epic story, I promise we'll get to the bottom of things in the end.

I found add-zsh-hook and a usage example in atuin's source code.

I might not fully comprehend all of that is written there, but I'm a man of action, and I can take a solid piece of work and tear it apart.

It's not much, but here's what I got:

# Source this in your ~/.zshrc
autoload -U add-zsh-hook

_past_preexec(){
    echo "preexec"
}

_past_precmd(){
    echo "precmd"
}

add-zsh-hook preexec _past_preexec
add-zsh-hook precmd _past_precmd

This sets up two hooks: the first one gets called right before a command gets executed and the second one directly after. (I decided to call my little history replacement past. I like short names.)

Okay, let's tell zsh to totally run this file whenever we execute a command:

source src/shell/past.zsh

...aaaaaand

❯❯❯ date
preexec
Fri May 28 18:53:55 CEST 2021
precmd

It works! ✨ How exciting!

Actually, I just remember now that I did the same thing for my little environment settings manager envy over two years ago, but hey!

So what to do with our newly acquired power?

Let's Run Some Rust Code

Here's the thing: only preexec gets the "real" command. precmd gets nothing:

_past_preexec(){
    echo "preexec $@"
}

_past_precmd(){
    echo "precmd $@"
}

$@ means "show me what you got" and here's what it got:

❯❯❯ date
preexec date date date
Fri May 28 19:02:11 CEST 2021
precmd

Shouldn't one "date" be enough?
Hum... let's look at the zsh documentation for preexec:

If the history mechanism is active [...], the string that the user typed is passed as the first argument, otherwise it is an empty string. The actual command that will be executed (including expanded aliases) is passed in two different forms: the second argument is a single-line, size-limited version of the command (with things like function bodies elided); the third argument contains the full text that is being executed.

I don't know about you, but the third argument should be all we ever need? 🤨

Checking...

❯❯❯ ls -l
preexec ls -l lsd -l lsd -l

(Shout out to lsd, the next-gen ls command )

Alright, good enough. Let's parse $3 with some Rust code and write it to our own history file.

use std::env;
use std::error::Error;
use std::fs::OpenOptions;
use std::io::Write;

const HISTORY_FILE: &str = "lol";

fn main() -> Result<(), Box<dyn Error>> {
    let mut history = OpenOptions::new()
        .create(true)
        .append(true)
        .open(HISTORY_FILE)?;

    if let Some(command) = env::args().nth(3) {
        writeln!(history, "{}", command)?;
    };
    Ok(())
}
❯❯❯ cargo run -- dummy dummy hello
❯❯❯ cargo run -- dummy dummy world
❯❯❯ cat lol
hello
world

We're almost done — at least if we're willing to cheat a bit. 😏 Let's hardcode that format string:

use std::env;
use std::error::Error;
use std::fs::OpenOptions;
use std::io::Write;
use std::time::SystemTime;

const HISTORY_FILE: &str = "lol";

fn timestamp() -> Result<u64, Box<dyn Error>> {
    let n = SystemTime::now().duration_since(SystemTime::UNIX_EPOCH)?;
    Ok(n.as_secs())
}

fn main() -> Result<(), Box<dyn Error>> {
    let mut history = OpenOptions::new()
        .create(true)
        .append(true)
        .open(HISTORY_FILE)?;

    if let Some(command) = env::args().nth(3) {
        writeln!(history, ": {}:0;{}", timestamp()?, command)?;
    };
    Ok(())
}

Now, if we squint a little, it sorta kinda writes our command in my history format. (That part about the Unix timestamp was taken straight from the docs. Zero regrets.)

Remember when I said that precmd gets nothing?

I lied.

In reality, you can read the exit code of the executed command (from $?). That's very helpful, but we just agree to ignore that and never talk about it again.

With this out of the way, our final past.zsh hooks file looks like that:

autoload -U add-zsh-hook

_past_preexec(){
    past $@
}

add-zsh-hook preexec _past_preexec

Now here comes the dangerous part! Step back while I replace the original history command with my own. Never try this at home. (Actually I'm exaggerating a bit. Feel free to try it. Worst thing that will happen is that you'll lose a bit of history, but don't sue me.)

First, let's change the path to the history file to my real one:

// You should read the ${HISTFILE} env var instead ;)
const HISTORY_FILE: &str = "/Users/mendler/.zhistory";

Then let's install past:

❯❯❯ cargo install --path .
# bleep bloop...

After that, it's ready to use. Let's add that bad boy to my ~/.zshrc:

source "/Users/mendler/Code/private/past/src/shell/past.zsh"

And FINALLY we can test it.

We open a new shell and run a few commands followed by history:

❯❯❯  date
...
❯❯❯ ls
...
❯❯❯ it works
...
❯❯❯ history
 1011  date
 1012  ls
 1013  it works

Yay.The source code for past is on Github.

How it really really works

Our experiment was a great success, but I since learned that reality is a bit different.

"In early versions of Unix the history command was a separate program", but most modern shells have history builtin.

zsh tracks the history in its main run loop. Here are the important bits. (Assume all types are in scope.)

Eprog prog;

/* Main zsh run loop */
for (;;)
{
    /* Init history */
    hbegin(1);
    if (!(prog = parse_event(ENDINPUT)))
    {
        /* Couldn't parse command. Stop history */
        hend(NULL);
        continue;
    }
    /* Store command in history */
    if (hend(prog))
    {
        LinkList args;
        args = newlinklist();
        addlinknode(args, hist_ring->node.nam);
        addlinknode(args, dupstring(getjobtext(prog, NULL)));
        addlinknode(args, cmdstr = getpermtext(prog, NULL, 0));

        /* Here's the preexec hook that we used.
        * It gets passed all the args we saw earlier.
        */
        callhookfunc("preexec", args, 1, NULL);

        /* Main routine for executing a command */
        execode(prog);
    }
}

The history lines are kept in a hash, and also in a ring-buffer to prevent the history from getting too big. (See here.)

That's smart! Without the ring-buffer, a malicious user could just thrash the history with random commands until a buffer overflow is triggered. I never thought of that.

History time (see what I did there?)

The original history command was added to the Unix C shell (csh) in 1978. Here's a link to the paper by Bill Joy (hey, another Bill!). He took inspiration from the REDO command in Interlisp. You can find its specification in the original Interlisp manual in section 8.7.

Lessons learned

  • Rebuild what you don't understand.
  • The history file is human-readable and pretty straightforward.
  • The history command is a shell builtin, but we can use hooks to write our own.
  • Fun fact: Did you know that in zsh, history is actually just an alias for fc -l? More info here or check out the source code.

“What I cannot create, I do not understand” — Richard Feynman


Shader translation benchmark on Dota2/Metal gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-05-09 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. See The Big Picture for the overview, and release-0.8 for the latest progress. In this post, we are going to share the first performance metrics of our new pure-Rust shader translation library Naga, which is integrated into gfx-rs. Check the Javelin announcement, which was the original name of this project, for the background.

gfx-portability is a Vulkan Portability implementation in Rust, based on gfx-rs. Previous Dota2 benchmarks showed good potential in our implementation. However, it couldn’t be truly called an alternative to MoltenVK if it relies on SPIRV-Cross. Today, we are able to run Dota2 with a purely rust Vulkan Portability implementation, thanks to Naga.

Test

Testing was done on MacBook Pro (13-inch, 2016), which has a humble dual-core Intel CPU running at 3.3GHz. We created an alias to libMoltenVK.dylib and pointed DYLD_LIBRARY_PATH to it for Dota2 to pick up on boot, thus running on gfx-portability. It was build from naga-bench-dota tag in release. The SPIRV-Cross path was enabled by uncommenting features = ["cross"] line in libportability-gfx/Cargo.toml.

In-game steps:

  1. launch make dota-release
  2. skip the intro videos
  3. proceed to “Heroes” menu
  4. select “Tide Hunter”
  5. and click on “Demo Hero”
  6. walk the center lane, enable the 2nd and 3rd abilities
  7. use the 3rd ability, then quit

Hero selection screen with Naga (low settings)

The point of this short run is to get a bulk of shaders loaded (about 600 graphics pipelines). We are only interested in the CPU cost for loading shaders and creating pipelines. This isn’t a test for the GPU time executing the shaders. The only fact about GPU that matters here is that the picture looks identical. We don’t expect any architectural changes for potential visual issues to be discovered.

Times were collected using profiling instrumentation, which is integrated into gfx-backend-metal. We added this as a temporary dependency to gfx-portability with “profile-with-tracy” feature enabled in order to capture the times in Tracy.

In tracy profiles, we’d find the relevant chunks and click on the “Statistics” for them. We are interested in the mean (μ) time and the standard deviation (σ).

Results

Function Cross μ Cross σ Naga μ Naga σ
SPIR-V parsing 0.34ms 0.15ms 0.45ms 0.50ms
MSL generation 3.94ms 3.5ms 0.56ms 0.38ms
Total per stage 4.27ms   1.01ms  
         
create_shader_module 0.005ms 0.01ms 0.53ms 0.57ms
create_shader_library 5.19ms 6.19ms 0.89ms 1.23ms
create_graphics_pipeline 10.94ms 12.05ms 2.24ms 5.13ms

The results are split in 2 groups: one for the time spent purely in the shader translation code of SPIRV-Cross (or just “Cross”) and Naga. And the other group shows combined times of the translation + Metal runtime doing its part. The latter very much depends on the driver caches of the shaders, which we don’t have any control of. We made sure to run the same test multiple times, and only take the last result, giving the opportunity for caches to warm up. Interestingly, the number of outliers (shaders that ended up missing the cache) was still higher in the “Cross” path. This may be just noise, or improperly warmed up caches, but there is a chance it’s also indicative of the fact “Cross” generates more of different shaders, and/or being non-deterministic.

The total time spent in shader module or pipeline creation is 7s with Cross path and just 1.29s with Naga. So we basically shaved 6 seconds off the user (single-core) time just to get into the game.

In neither case there was any pipeline caching involved. One could argue that pipeline caches, when loaded from disk, would essentially solve this problem, regardless of the translation times. We have the support for caching implemented for Naga path, and we don’t want to make it unfair to Cross, so we excluded the caches from the benchmark. We will definitely include them in any full games runs of gfx-portability versus MoltenVK in the future.

Conclusions

This benchmark shows Naga being roughly 4x faster than SPIRV-Cross in shader translation from SPIR-V to MSL. It’s still early days for Naga, and we want to optimize the SPIR-V control-flow graph processing, which can be seen in the numbers taking time. We assume SPIRV-Cross also has a lot of low-hanging fruits to optimize, and are looking forward to see its situation improving.

Previously, we heard multiple requests to allow MSL generation to happen off-line. We are hoping that the lightning fast translation times (1ms per stage) coupled with pipeline caching would resolve this need.

The quality and read-ability of generated MSL code in Naga is improving, but it’s still not at the level of SPIRV-Cross results. It also doesn’t have the same feature coverage. We are constantly adding new things in Naga, such as interpolation qualifiers, atomics, etc.

Finally, Naga is architectured for shader module re-use. It does a lot of work up-front, and can produce target-specific shaders quickly, so it works best when there are many pipelines created using fewer shader modules. Dota2’s ratio appears to be 2 pipelines per 1 shader module. We expect that applications using multiple entry points in SPIR-V modules, or creating more variations of pipeline states, would see even bigger gains.


Fine, we'll relocate our own binary! fasterthanli.me

fasterthanli.me2021-05-01 07:30:00

Welcome back to the eighteenth and final part of "Making our own executable packer".

In the last article, we had a lot of fun. We already had a "packer" executable, minipak, which joined together stage1 (a launcher), and a compressed version of whichever executable we wanted to pack.


Release of v0.8 gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-04-30 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. The main projects are:

  • gfx-rs makes low-level GPU programming portable with low overhead. It’s a single Vulkan-like Rust API with multiple backends that implement it: Direct3D 12/11, Metal, Vulkan, and even OpenGL ES.
  • naga translates the shaders between languages, including WGSL. Also provides validation and processing facilities on the intermediate representation.
  • wgpu-rs is built on top of gfx-rs and gpu-alloc/gpu-descriptor. It provides safety, accessibility, and strong portability of applications.

Following the regular schedule of releasing once in a few month, we just rolled out 0.8 versions across gfx/wgpu projects! See gfx-rs changelist, wgpu changelist, and naga changelist for the details.

tree

Naga-based shader infrastructure has been growing and capturing more ground. It has reached an important point where SPIRV-Cross is not just optional on some platforms, but even not enabled by default. This is now the case for Metal and OpenGL backends. Naga path is easier to integrate, share types with, compile, and it’s much faster to run. Early benchmarks suggest about 2.5x perf improvement over SPIRV-Cross for us.

The work on HLSL and WGSL backends is underway. The former will allow us to deprecate SPIRV-Cross on Direct3D 12/11 and eventually remove this C dependency. The latter will help users port the existing shaders to WGSL.

Another big theme of the release is enhanced wgpu validation. The host API side is mostly covered, with occasional small holes discovered by testing. The shader side is now validating both statements and expressions. Programming shaders with wgpu starts getting closer to Rust than C: most of the time you fight the validator to pass, and then it just works, portably. The error messages are still a bit cryptic though, hopefully we’ll improve it in the next release. Hitting a driver panic/crash becomes rare, and we are working on eliminating these outcomes entirely. In addition, wgpu now knows when to zero-initialize buffers automatically, bringing the strong portability story a bit closer to reality.

We also integrated profiling into wgpu and gfx-backend-metal. The author was receptive to our needs and ideas, and we are very happy with the results so far. Gathering CPU performance profiles from your applications today can’t be any simpler:

profiling

In Naga internals, the main internal improvement was about establishing an association of expressions to statements. It allows backends to know exactly if expression results can be re-used, and when they need to be evaluated. Overall, the boundary between statements and expressions became well defined and easy to understand. We also converged to a model, at high level, where the intermediate representation is compact, but there is a bag of derived information. It is produced by the validator, and is required for backends to function. Finally, entry points are real functions now: they can accept parameters from the previous pipeline stages and return results.

Finally, we added a few experimental graphics features for wgpu on native-only:

  • Buffer descriptor indexing
  • Conservative rasterization

P.S. overall, we are in the middle of a grand project that builds the modern graphics infrastructure in pure Rust, and we appreciate anybody willing to join the fight!


What's in the box? fasterthanli.me

fasterthanli.me2021-04-18 16:30:00

Here's a sentence I find myself saying several times a week:

...or we could just box it.

There's two remarkable things about this sentence.

The first, is that the advice is very rarely heeded, and instead, whoever I just said it to disappears for two days, emerging victorious, basking in the knowledge that, YES, the compiler could inline that, if it wanted to.


Your First Business Should Be A Spreadsheet Matthias Endler

Matthias Endler2021-03-10 00:00:00

One of the best decisions I made in 2020 was to open my calendar to everyone. People book appointments to chat about open-source projects, content creation, and business ideas.

When we talk about business ideas, people usually gravitate towards problems suitable for startups. Things like using artificial intelligence to find clothes that fit or building a crowdfunding platform on the blockchain.

These are all great ideas, but they require lots of persistence and deep pockets. It's probably easier and less risky to just join a startup in that field.

In reality, most people just want something cool to work on and happy customers. It turns out, you don't have to run a startup for that (and you probably shouldn't). Instead, starting a side project is less risky and can grow organically into a business over time.

Usually, the solution is right in front of them: in some Excel spreadsheet on their computer.

I Hate Excel

I never spend more time in Excel than I absolutely have to. Normally, I need to get something done quickly. I don't care about the layout or the design; I care about getting over it. Probably I'd pay someone to do that work for me — and that's my point!

The spreadsheets and lists you create for your own solve a real problem: yours; and chances are, someone else has the same problem. That's a business opportunity!

This approach ticks all the boxes:

  • 💪 Solves a real problem.
  • 🥱 It's boring, so people might pay for not having to do it themselves.
  • ⚡️ No time wasted on design or infrastructure. It's the ultimate MVP.
  • 🐢 Low tech: no programming required. You can start with Notion + Super.so.
  • 🐜 Niche market: if an established service existed, you'd use it. BigCorp ain't gonna eat your lunch.
  • 🚀 You spend less time building and more time talking to potential customers.

Examples

A few years ago, I looked for static code analysis tools. I did some boring research, started a list, pushed it to Github, and that's that. Fast forward a few years, and that side project is pulling in money from sponsors and attracts consulting gigs.

Another example: a guy built a spreadsheet for places to work from remotely. He shared the spreadsheet on Twitter. People added more places, and he created a website from it. The website is NomadList, and the guy is Pieter Levels. He pulls in 300k/year from that website.

Instead of building a site first, I simply made [a] public Google spreadsheet to collect the first data and see if there’d be interest for this at all. — Pieter Levels on how he created NomadList.

I left a spot for your story here. Now brush up that spreadsheet (or that list), share it with your friends, iterate on their feedback, and build your first business.


Release of v0.7 gfx-rs nuts and bolts

gfx-rs nuts and bolts2021-02-02 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. It governs a wide range of projects:

  • gfx-rs makes low-level GPU programming portable with low overhead. It’s a single Vulkan-like Rust API with multiple backends that implement it: Direct3D 12/11, Metal, Vulkan, and even OpenGL.
  • naga translates the shaders between languages, including WGSL. Also provides validation and processing facilities on the intermediate representation.
  • wgpu-rs is built on top of gfx-rs and gfx-extras. It provides safety, accessibility, and even stronger portability of applications.
  • metal-rs and d3d12-rs wrap native graphics APIs on macOS and Windows 10 in Rust.

Today, we are happy to announce the release of 0.7 versions across gfx/wgpu projects!

gfx-hal-0.7

Overall theme of this release is simplification. We cut off a lot of experimental cruft that accumulated over the years, cleaned up the dependencies, and upgraded the API to be more modern.

For example, last release we made a step towards more generic bounds with ExactSizeIterator on our APIs. In this release, we are taking two steps back by removing not just ExactSizeIterator, but also Borrow from the iterator API. We figured a way to do the stack allocation without extra bounds, using inplace_it.

Having two distinct swapchain models has also come to an end. We removed the old Vulkan-like model, but also upgraded the new model to match “VK_KHR_imageless_framebuffer”, getting the best of both worlds. It maps to the backends even better than before, and we can expose it directly in gfx-portability now.

There is also a lot of API fixes and improvements, one particularly interesting one is aligning to Vulkan’s “external synchronization” requirements. This allows us to do less locking in the backends, making them more efficient.

Another highlight of the show is the OpenGL ES backend. It’s finally taking off based on EGL context and window system integration. There is still a lot of work to do on the logic, but the API is finally aligned to the rest of the backends (see 3.5 year old issue). We are targeting Linux/Android GLES3 and WebGL2 only.

See the full changelog for details.

wgpu-0.7

spaceship cheese

The list of libraries and applications has grown solidly since the last release. A lot of exciting projects and creative people joined our community.

Our goals were to bring the API closer to the stable point and improve validation. There is quite a bit of API changes, in particular with the pipeline descriptors and bind group layouts, but nothing architectural. We also got much nicer validation errors now, hopefully allowing users to iterate without always being confused :)

The highlight of wgpu work is support for WGSL shaders. It’s the emerging new shading language developed by WebGPU group, designed to be modern, safe, and writable by hands. Most of our examples are already using the new shaders, check them out! We are excited to finally be able to throw the C dependencies (spirv-cross, shaderc, etc) out of our projects, and build and deploy more easily.

See the core changelog and the rust API changelog for details.

naga-0.3

Naga has seen intensive development in all areas. SPIR-V frontend and backend, WGSL frontent, GLSL frontent and backend, intermediate layer, validation - all got a lot of improvements. It’s still not fully robust, but Naga has crossed the threshold of being actually usable, and we are taking advantage of it in wgpu-rs.

We experimented on the testing infrastructure and settled on cargo-insta. This boosted our ability to detect regressions, and allowed us to move forward more boldly.

The next steps for us are completing the validation, adding out-of-bounds checks, and replacing SPIRV-Cross completely in applications that have known shaders.

See the changelog for details.

P.S. overall, we are in the middle of a grand project that builds the modern graphics infrastructure in pure Rust, and we’d appreciate anybody willing to join the fight!


Starting A Print-On-Demand Business As A Software Engineer Matthias Endler

Matthias Endler2021-01-22 00:00:00

One day I had the idea to make a print of my Github timeline. I liked the thought of bringing something "virtual" into the real world. 😄

So I called up my friend Wolfgang and we built codeprints. It's my first "physical" product, so I decided to share my learnings.

[Felix Krause](https://krausefx.com/) of [fastlane](https://fastlane.tools/) fame was one
of our first customers and we are very thankful for this tweet promoting our
service, which gave us a huge traffic boost.
Felix Krause of fastlane fame was one of our first customers and we are very thankful for this tweet promoting our service, which gave us a huge traffic boost.

Launching Is Hard, So Launch Early

Even though I knew that launching early was vital, I still didn't want to "commit" to the final design shortly before the planned go-live. There was always that last bug to fix or that little extra feature to implement. For example, I wanted to offer two designs/layouts: the classic Github contribution timeline and a graph-based design for repositories. In cases like that, it helps to have a co-founder. Wolfgang convinced me that multiple layouts were not needed for the MVP and that whatever we'd come up with would probably be wrong anyway without getting early user feedback. He was right. Without Wolfgang, the shop would probably still not be live today. We have a much clearer vision now of what people want to see, thanks to launching early. Turns out users were not really interested in the graph-based design after all, and it would have been a waste of time to create it.

Lesson learned: Even if you know all the rules for building products, it's different when applying them in practice for the first time. We'll probably never be completely happy with the shop functionality, but it's better to launch early and make incremental improvements later.

Software Development Is Easy

When we started, my main concern was software development. The frontend and the backend needed to be coded and work together. We didn't want to run into Github rate-limiting issues in case there were many users on the site. I was also thinking a lot about which web frontend to use. Should we build it in Rust using Yew or better go with Gatsby?

Turns out writing the code is the easy part.

Being software engineers, it didn't take us too long to implement the backend API and we quickly found a decent template for the frontend. Most of our time was spent thinking about the product, the user experience, financing, taxes, the shipping process, marketing, and integrating customer feedback. These were all things I had (and still have) little experience in.

Wolfgang suggested to "just use Shopify and the default template" to get started quickly. In hindsight, it was the absolute right decision. I always thought Shopify was for simple mom-and-pop stores, but it turns out it's highly customizable, integrates well with pretty much anything, and offers excellent tooling like themekit. Payments, refunds, discounts, customer analytics: it's all built into the platform. It saved us sooo much development time.

Lesson learned: There are many unknown unknowns — things we are neither aware of nor understand — when starting a project. Try to get to the root of the problem as soon as possible to save time and avoid the sunk cost fallacy.

Users Expect Great UI/UX

Giants like Amazon, Facebook, and Netflix have raised customer expectations for great UX. They spend millions polishing their websites and getting every detail right. As a result, their sites work just right for millions of customers and on every device.

An indie shop does not have these resources. Nevertheless, many customers expect the same quality user experience as on other sites they use. Being on the other side of the fence for the first time, I learned how hard it is to build a user interface that works for 90% of the people. Every little detail — like the order of form fields — makes a huge difference. Get too many details wrong, and you lose a customer.

Those things can only be found by watching real users use your product. I promise you, it will be eye-opening!

Lesson learned: Watch potential customers use your service. It will be painful at first, but will improve the quality of your product. Use standard frameworks for shops if you can because they get many UI/UX details right out of the box. WooCommerce or Shopify come to mind.

Building Products Means Being Pragmatic

We have many ideas for future products. Many friends and customers tell us about potential features all the time, but the problem is how to prioritize them. Most ideas won't work at scale: It's tricky to find a supplier that has a product on offer, is cheap, ships worldwide, and has a working integration with your shop-system. So we have to regularly scrap product ideas, simply because our suppliers' support is not there. On top of that, we run the business next to our day job and other responsibilities, so we need to make use of our time as efficiently as possible.

Lesson learned: Making services look effortless is hard work. Time is your biggest constraint. You'll have to say "no" more often than you can say "yes".

Due to the pandemic, codeprints was
entirely built remotely. More people should give [whereby](https://whereby.com/)
a try.
Due to the pandemic, codeprints was entirely built remotely. More people should give whereby a try.

Getting Traction As A Small Business

It has never been easier to launch a shop. Services like Shopify, Stripe, and a host of suppliers make starting out a breeze. On the other hand, there is a lot more competition now that the barrier to entry is so low.

Thousands of services are constantly competing for our attention. On top of that, most customers just default to big platforms like Amazon, AliExpress, or eBay for their shopping needs these days, and search engines send a big chunk of the traffic there.

Since our product is custom-made, we can not offer it on those bigger platforms. As an indie shop, we get most visitors through word of mouth, exceptional customer support, and advertising where developers hang out: Twitter, Reddit, HackerNews, Lobste.rs, and friends. It's essential to focus on providing value on those platforms; a plain marketing post won't get you any attention. Other platforms like LinkedIn, Facebook, ProductHunt, or IndieHackers could also work, but our target audience (OSS developers with an active Github profile) doesn't hang out there that much.

Lesson learned: Always know where your customers are and understand their needs.

Finding A Niche Is Only Half The Job

Common market wisdom is to find niche and grow from within. With codeprints we definitely found our niche: the audience is very narrow but interested in our geeky products. There are 56 million developers on Github today; that's a big target audience. Most profiles are not very active, though. To make a print look attractive, you'd have to consistently commit code over a long period of time — many years. If we assume that only 1% of devs are active, that limits our target audience to 560.000 users. That's still a big but much smaller market. Now, if only 1% of these people find the shop and order something (which would be quite a good ratio), we're looking at 5.600 orders total. Not that much!

In order to extend that audience, one could either increase the number of potential customers or focus on getting more of the existing potential customers on the page. In our case, we expanded by offering a one-year layout, reducing the required level of Github activity for a cool print. We are also working on making emptier profiles look more interesting and highlighting the value-producing part of open source contribution. Every contribution counts — no matter how tiny.

Lesson learned: Make sure that your niche market is not too narrow so that you can make a sustainable business out of it.

Early adopters like [Orta
Therox](https://orta.io/) are incredibly precious when starting out. Not
everybody has a rockstar profile like that, though (and that's fine).
Early adopters like Orta Therox are incredibly precious when starting out. Not everybody has a rockstar profile like that, though (and that's fine).

Make User Feedback Actionable

Initial customer feedback is precious. You should focus on every word these customers say as they believe in your product and want you to win. (They voted with their wallet after all.) Feedback from friends is helpful, too, but I usually apply a bigger filter to that. Not all of my friends are software developers, and while they all mean well, what they tell me might be different from what they mean. It's like they are asking for faster horses when what they really want is a car. Feedback on social media can be... snarky at times; be prepared for that! Your job is to find the grain of truth in every statement and focus on constructive advice.

For example, take this feedback we got:

How lazy can someone be to pay €36 for this.

You could turn it around to make it constructive:

Can I get a cheaper version to print myself?

And that is some valuable feedback. We could provide a downloadable version in the future!

Lesson learned: It takes practice to extract actionable feedback from user input and make it fit your product vision.

Summary

2020 was a crazy year. I helped launch two small side-businesses, codeprints and analysis-tools.dev.

Both have an entirely different revenue model, but they have one thing in common: they were super fun to build! 🤩 It's motivating to look back at those achievements sometimes... That print of 2020 pretty much encapsulates those feelings for me. (Note the greener spots in August and September, which is when we launched analysis-tools and the days in December when we built codeprints.)

My coding year in review using our new
vertical layout.<br />Here's to
building more products in 2021.
My coding year in review using our new vertical layout.
Here's to building more products in 2021.

Let me know if you found that post helpful and reach out if you have questions. Oh and if you're looking for a unique way to decorate your home office, why not get your own print from codeprints? 😊

P.S.: If you're a product owner and you're looking for a unique present for your team, get in contact and be the first to get an invite to a private beta.


So You Want To Earn Money With Open Source Matthias Endler

Matthias Endler2021-01-04 00:00:00

I earned 0 Euros from maintaining OSS software for years, and I thought that's the way things are. I finally looked into ways to monetize my projects last year and in this talk I want to share what I learned so far. It didn't make me rich (yet!), but I built my first sustainable side-project with analysis-tools.dev ✨.

I'll talk about this and other projects and the mistakes I made on the road towards sustainability.

Related links and resources:

Find a full transcript of the talk below. (Sorry for the wall of text.)


This is my talk about earning money with Open Source, which I gave at the Web Engineering Meetup Aachen at the end of 2020. The organizers gladly allowed me to share it on my YouTube channel. I'm basically trying to answer the question: "Why am I not making 100k on Github?". I'm talking about finding corporate sponsors for myself and the long road towards sustainability of open-source maintenance.

You might not even want to start. This is a talk for those people that have the mindset that it's probably not worth it to spend that much effort on Open Source if it takes so long until you find success. Now, this talk turned out to be a little grim. I had this very motivational talk in mind, but in reality, it's hard, and by hard, I mean it's really hard.

I just want to get this point across and maybe still motivate you to do it but first: why am I entitled to talk about this? I've been doing Open Source for 10 years over 10 years now. This is a talk dedicated to my former self maybe 15 years ago. I work at trivago, which is a hotel search company based in Düsseldorf. I have a blog at endler.dev. Like everyone and their mom, I also have a YouTube channel. It's called Hello, Rust! and I'm extremely active with one video every two years. Hence, you definitely want to subscribe to not miss any updates. But today, I want to talk about Open Source, and I have a very sophisticated outline with two points my journey and revenue models.

Let's go back all the way to 2010. The world definitely looked a bit different back then.

Github in 2010
Github in 2010

This was Github, and I was a bit late to the game. I joined in January 2010, and by then, Github was already two years old, so my username was taken. I usually go by the handle mre on platforms, and I noticed that this handle was not used by anyone, so I just sent a mail to support and asked if I could have it, and then I got an answer from this guy saying "go for it." It was Chris Wanstrath, who goes by the handle defunct, and he's the former CEO of Github, and at this point in time, I was hooked. I really liked the platform. I really liked how they worked very hands-on with Open Source. I used it for some projects of mine; you can see in the screenshot that I uploaded my blog, for example, because they host it for free. It was built with Jekyll, and you just push it to their site. Then they statically generate it, and it's done. It goes without saying that nothing has changed in the last 10 years because my blog more or less still looks like that. It's not built with jQuery and Jekyll anymore, but with zola and Cloudflare Worker Sites, but it's more or less the same thing. For preparing for this talk, I wanted to take a step back and see where I was coming from and where I am right now, and probably the best way to do it is to look up some statistics and see if the number of repositories over time would give me some insights. So I queried the Github API for that.

You can see it's pretty much a linear graph from 2010 all the way to 2020. Except for 2018, where I reached peak productivity, it seems, but oh well. In the end, it's more or less a linear thing, and you might say you put some work in you get some feedback out, but in reality, it's different. There is a compound effect. If we look at my number of stars over time, you can see that more or less it started very slowly, and now it's sort of growing exponentially, so right now, we are at 25.000 stars across all projects. Another way to look at it would be the number of followers. That's kind of a new metric to me, but I did look up some statistics from archive.org (because Github doesn't have that information through their API), and again, it's more or less exponential growth.

You put some work in, but you get a compound effect of your work plus some interest out. This is not luck; it's work. It means you know what you're doing. At the same time, there's the elephant in the room, and that is it's just a pat on the back. We have earned zero dollars until now, and one question you might have is how do you monetize this effort.

First off, is it an effort?

Well, I don't know about you, but I probably spend two or three hours on average per day on Open Source: thinking about Open Source and creating new projects, but also maintaining and code review, so it really is work, and it's a lot of work, and you more or less do that for free.

There's nothing wrong with doing things for free and doing it as a hobby, but in this case, you are supposed to be working on whatever you like. Open Source is not like that; sometimes you have obligations, and you feel responsible for maybe helping people out, which is a big part of it. You do that next to your regular work, so it can really be a burden. If you don't know by now, making this somehow valuable is hard, it's really hard. I want to talk about some ways to build a proper revenue model from Open Source. It goes without saying that this should probably not be your first focus if you saw the graphs before, but once you reach a point where you want to get some revenue, you have a couple of options. I don't want to talk about doing Open Source as part of your business, and I don't want to talk about bigger companies and more significant support here. I want to focus on a couple things that everyone can do. Sponsoring [on Github] is one. Offer paid learning materials on top of your normal documentation. For example, you might have a video series that you ask for money. Sell merchandising like Mozilla does. Consulting next to your Open Source business Services and plugins like writing an ADFS plugin or high availability functionality are very common examples for paid features targeting enterprises.

But let's start with the basics. Let's start with point number one, sponsoring. There are two types of sponsoring: the first one is individual donations. Individual sponsoring is what Github Sponsors is all about. If you want to earn money [with that model], you have to think about the funnel, and you have to think about how you capture people's attention and how you monetize that. It starts with a product, [which] can be anything. From there, you generate interest, and this interest creates an audience, and that audience eventually might pay for your service, and this is actually the entire secret. It's how you earn money with any product, and with Open Source, if you want to attract sponsors, you build a product people want.

If you transfer that to Open Source, building a project is maybe a repository, and the stars indicate the interest of the audience. The audience itself is made out of followers (personal followers or followers of a company), and those followers might or might not become sponsors in the end. Now, I know stars are a terrible metric for popularity because some people use stars differently than others. For example, some use it as bookmarks to check out projects later, others want to thank the developers for maybe putting in a lot of effort, and so on, but it's a good first estimation.

Now, think about the following. Think about the number of stars I have and the followers and the number of sponsors. Think about my "funnel" right now. I told you that I have 25.000 stars and roughly 1000 followers, and out of those, I have three sponsors, so the ratio between the stars and sponsors is 0.01. That looks pretty grim. It means you need around 8.000 stars to attract a single supporter. I was wondering: "maybe it's just me?". Maybe the top 1000 Github maintainers did not have that problem. Well, it turns out it's exactly the same schema. If you take the top 1000 Github maintainers and look at their sponsors, it's again a pretty grim picture. For example, looking at the median, you look at 3421 followers per person and a median of zero sponsors. That's zero percent if my math is correct, and if you look at the average, you even have 5430 followers (because Linus Torvalds pushes that number up). You have 2.8 sponsors out of that on average, and that is 0.5%, which is a bit more than I have, but it's roughly in the same ballpark. Now think about this: Github has 40 million users, so that means the top 1000 maintainers make up 0.0025% of the entire community. The median income of those maintainers on Github is basically zero.

That in and on itself is maybe not the biggest problem, but keep in mind that the Github revenue of 2019 was 300 million dollars. I read that comment on Hacker News yesterday:

I have sponsors on Github and rake in a cool two dollars per month. It's obviously less after taxes, so I have to have a day job.

So this is clearly not working. You have to think of different ways to monetize Open Source, or you just wait until Github Sponsors becomes more popular -- whatever happens first. One way I just want to quickly touch on is the notion of sponsorware. It's kind of a new concept, and some people haven't heard of it before. I honestly really like it. Generally speaking, you create a project, and you keep it private. You talk about it on Twitter, though or any other platform, and you say: "hey, I'm building this, but if you want early access, you have to become a sponsor," and once you reach a certain threshold of sponsored sponsors, or income or whatever. Then you make a project public. This initial example that I showed you, where someone was earning 100k on Open Source, is from someone doing just that. He's building products and services, talks about them, and then makes them open for everyone in the end.

This has some advantages: first of you get early feedback from people that really believe in your mission. Second, you don't have to work for free all the time, and third, you might also create an audience and hype from your projects. The disadvantage is that if you are a hardcore Open Source or free software believer, this goes against your ethic. You want the software to be open, to begin with, without any additional requirements. So you really have to make up your own mind about that. I tried, and I have an early access program, which I only share with sponsors. [My first sponsorware was a] tool for getting Github statistics. [The statistics from this talk were] created with that tool. I think you need a big audience to pull that off. The question is if you want to put that much effort in, or you just want to make it open in the first place and think about other revenue models. However, I think still it's a very interesting concept, and we might see that [more] in the future, so you know how it looks like now, and you have a name for it.

Another one is corporate sponsoring. This is a double-edged sword because corporate sponsoring means that a company gives you money and sometimes wants something. They might want additional support, or they want the bug to be fixed, and more or less it feels like you are somehow beginning to work for them, but nevertheless, those companies put in quite a big amount of money into Open Source these days. Looking at two big companies, Facebook and Google, they invested 177k and 845k respectively into Open Source over their lifetime on Open Collective, a platform for collecting those donations. That's really great. We need more companies doing that, but also, as a little side note and maybe as a little rant, I believe that those companies are doing way too little.

Facebook's revenue last year was 70 billion, and Google had 160 billion, which is nothing to be ashamed of, so I wonder really if this is the most they can do. Of course, Google, for example, also donated to other projects like Mozilla, and they also organize meetups and so on. But do you really think that Facebook and Google would exist today if there was no Python or web server or Linux back in the day when two Stanford students tried to build a search engine? Sometimes I feel that Fortune 500 companies really don't understand how much they depend on Open Source and how many people depend on a few people who maintain critical parts of our infrastructure.

I don't think they invest nearly enough into Open Source. What a lot of people think is that Open Source works like the panel on the left where you have a full room of engineers trying to figure out the best way to build a project, and in reality, it's more or less someone working late at night to fix bugs and doing it because they believe in it. The public perception is probably wrong, and a really small group of people who maintain critical infrastructure. Sometimes that can lead to very tricky situations. Two of my childhood heroes talked about it openly: Kenneth Reitz is the core maintainer of requests for Python and antirez is the creator of Redis, a key-value store. So one is front-end development and the other one from backend-end. They both talk about burnout here because the burden of becoming an Open Source maintainer on a big scale can very much and very quickly lead to burnout. The internet never sleeps. You never go to sleep. You always get a ticket, a feature request, a pull request, an issue. You always have something to work on, and on top of that, you have to do all your other responsibilities, so that can lead to burnout really quickly. There was one guy who I also respect deeply. His name is Mark Pilgrim. He is the author of Dive Into Python, and he once pulled a 410 for deleting everything [about him] on the internet. There's actually a term for it: infocide for "information suicide." He got fed up with the ecosystem, and if you think about the Ruby community, you might remember _why, the author of the Poignant Guide to Ruby. He did kind of the same thing. Focusing on what antirez has said, "once I started to receive money to work at Redis, it was no longer possible for my ethics to have my past pattern, so I started to force myself to work on the normal schedules. This, for me, is a huge struggle for many years. At this point, moreover, I'm sure I'm doing less than I could, because of that, but this is how things work", so it feels like he feels guilty for maybe being forced into that work schedule and maybe not performing well enough. There are some signs of burnout for me somehow, and it's that love-hate relationship of Open Source and money. If you accept money, it becomes a job, but you're not writing code most of the time. You're writing the talks, reviewing pull requests, you're looking at issues, you're answering questions on StackOverflow, you're discussing on Discord, you're marketing on YouTube or conferences. When you become popular with Open Source, then it feels like you have a choice between two options: one is depression and the other one is burnout. If your project does not become successful, then suddenly you think you're a failure, you're a mistake. It has zero stars; nobody likes it. But if it becomes a success, then everyone likes it, and you get hugged to death. That's a really unfortunate situation to be in, and you want to stop being overwhelmed with those responsibilities. You have to set clear boundaries and pick your poison. You have to be careful if you accept companies as sponsors. I want to show you one example of how it can work and [point out] some risks. Earlier this year, I started working on a real project that I had been putting off for many years before.

You see, in December 2015, I started a list of static analysis tools on Github. Static analysis tools are just tools that help you improve your code, and it turns out that there's a lot of those tools. Just starting to collect them was the first step. I didn't think much about it, but over time that became really popular. And you can see that this graph is more or less a linear increase in stars over time. In 2018, I started really thinking hard about whether there was more than just a Github project here. I talked to many people that I had this idea of building something more from that. It really took someone else to maybe push me over the finishing line and convinced me that this was worth it, and that is Jakub. He said, "why not build a website from it?" and over the course of maybe two weekends or so, we built a website. It's built with Gatsby, but it really doesn't matter. We just did it, and then we saw what happened to it. We render 500 tools right now, and the initial feedback was really great. People really seem to like that. We got a cool 720.000 requests on the first day, and over the next week or so, it more or less hit 1.5 million. That was great because suddenly people started getting interested in that project. So we started finding some sponsors. Those companies are special because they believe in your mission, but they also know how Open Source works. They don't really expect you to advertise their tool. They want to sell to developers, so they want to be in the developers' minds, saying: "Hey! You are a developer. We built this amazing tool you might want to check it out!" but they also get seen as an Open Source company. I think that's a win-win. I have to say it doesn't always go as easily. sometimes companies expect you to just have cheap advertising space. Then they jump off the moment they see you don't get that many clicks, but others understand that they invest into something that maybe pays off in a year or two from now. So I'm really thankful that some companies understand that mission. However, what companies want is different than what individuals want. Companies want an invoice. Companies want something tax-deductible. Companies want someone that keeps the lights on and is responsive via email, so you really have those obligations, and one platform that helps with that is Open Collective. They have a 501c6 program for Open Source projects that acts as a fiscal host, which means they will do all the invoicing and officially be the maintainers. If you, as an Open Source maintainer or a contributor to a project, want to get [reimbursed for your work], you have to send an invoice to open collective.

I think that's the best of both worlds. Again, because it's a very transparent process, companies are in the loop and don't have to deal with all the financial stuff. But it also means that you have to really polish your public perception. Companies really want to know what they can get out of sponsoring you, and you have to make that very clear. Probably the most important site that you have is not your website, but it's your sponsors page on Github where you describe the different tiers and what those tiers mean, so we have three tiers: One is targeted at smaller companies and freelancers. They just get exposure, and they get seen as an Open Source friendly tech company. That is a hundred dollars a month. We have a middle-tier, a company sponsor that maybe is a bigger company. They get the batch, too, but they also get a blog post about a static analysis tool that they want to promote, but we make it transparent that this is really a sponsored content. Finally, if you want to go all the way, you go to full content creation, which might be a video workshop, but we don't have video workshop sponsors yet, so I cannot talk about that yet. I have to say I really would like to try though and it's cheap really for what you get.

Anyway, those are things that you can do today. Without really changing how you work on Open Source, you can set that up, and you just see how it goes. Maybe no one reacts, and that's fine. Everything else on that list is kind of advanced. You need an audience, and so you should start with that.

Paid learning material is something that we are doing with analysis tools in the future with a video course. There are companies like tailwind that do that impressively well, so you can learn from them. For merchandising, you have to have a brand. Hence, it's not something that I could do, but someone like Mozilla or the Coding Train on YouTube could definitely do something like that. Consulting is always an option. Still, it's also a lot more work and probably takes you away from what you really love, so it really becomes a job. You have to think about whether you want to do that or not. Enterprise services are very advanced [and interesting] for maybe the one percent of projects that can be run in a business and where you have special requirements. I have to say start from the top and work your way down. Start to create an audience. It's probably easier to build an audience on Twitter and then funnel it back to Github than the other way around. Oh, by the way, did I tell you it's hard? I really don't want to end on a low note. I really want to emphasize that I would do it again, all of that if I started today. I think there's no better time to contribute to Open Source than today. Probably tomorrow will even be a better time because suddenly, way more people are interested, it's way easier to set up projects, you have all those free tools like VSCode and Github actions, free hosting. It's just amazing how much you can pull off with very little money involved. So you can try it. What's the worst thing that can happen? No one cares? Well, okay, then you're as good as me. But I have some tips for you if you want to start today. My first tip is: "do your homework." Many people start with learning, and then they build things, and then they close the circle, but there's one key piece missing here. Some people hate the word, but you learn to love it eventually. It's called marketing. Marketing means a lot of things to a lot of people, but what it means to me is getting the word out because someone else will if you don't, and you are awesome; you just have to realize that. Maybe not everyone knows [about your project] right away, so you should really talk about it more. Maybe at conferences, maybe on Twitter, maybe you can just tell your friends. Maybe you can ask people to contribute and to support you. Somehow it's frowned upon in the community that if you do marketing, you're not doing it for real, but I think that's not true. I think that if smart people and patient and passionate people did marketing, then the world would be a better place; because I'm pretty sure the evil guys do marketing. So do your homework, but rest assured that being an Open Source maintainer means running a business, and you are the product. You have to think about why someone would want to sponsor you because if you don't come up with an answer for that, how should they know. Also, think about the funnel. How will people find you, for example? The best way for people to find you is probably starting a YouTube channel.

There are easier ways, though.

[First,] you can always help out in a different project, and you don't even have to be a coder. If you are good with design, then I can tell you there are so many Open Source projects that need designers. It's crazy. Maybe start creating a logo for a small project and start getting some visibility. Another one is having fun. If you know that earning money is hard in Open Source, then that can also be liberating because it means you can experiment and you can be creative, and yeah, having fun is the most important thing, I guess.

Second, build things you love because it's your free time in the end. The chances that someone will find the project is pretty low, so it better be something that you're really interested in. If you don't believe in that, just move on to the next thing. It's fine if you drop a project that you don't believe in anymore. No one will hold you accountable for that unless they are jerks, and you don't want to be surrounded by jerks.

Third, find friendly people because you really grow with your community. You want people that support your project and maybe eventually become maintainers to ease the burden, and that takes a lot of time, sometimes years, until you find one maintainer, so always be friendly, try to put yourself in their perspective. Go the extra mile if you can. For example, reintegrate the master branch into their pull request. Just do it for them. Say thanks twice if you're unsure.

Fourth is to grow an audience.
Radical marketing is one way, but being approachable and being inclusive is another way. You want to be the guy or the girl that people go to when they have a tough question, or they want to know how to get into Open Source. You want to be the person that helps them out on their first pull request. They will pay it back a thousand times. The most exciting people I have met so far are available for questions, and they don't really ask for anything in return. You hold them very close and dear to your heart. When the time comes, you will remember those people. We will say, like, "this is an amazing person to work with; I can highly recommend them," which is called a lead.

Finally, be in it for the long run. Good things take time. You see, it took me 10 years. Maybe it takes you five or maybe even less, but it's probably not an overnight success. It's really a long-term investment.


Lazy Loading YouTube Videos Posts on elder.dev

Posts on elder.dev2020-12-12 00:00:00 ⓘ Note First, let me acknowledge up-front that this is neither a novel problem nor a novel solution. This is simply what I cobbled together to fit my own needs, I thought I’d share about how this went / works. Why Lazy Load? YouTube is a pretty ubiquitous for video hosting and very easy to embed. For most videos you can just open the video on youtube.com, click “share”, click “embed”, and finally copy + paste the generated <iframe> into your page source.

The Big Picture gfx-rs nuts and bolts

gfx-rs nuts and bolts2020-11-16 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. In this post, we are going to provide an overview of the projects we work on, how they are connected, and what the future holds for us:

big picture

Full diagram link.

Legend:

  • parallelogram: Rust node
  • rectangle: C/C++ node
  • hexagon: node in Rust that exposes an external API

Colors correspond roughly to the areas: gfx is blue, wgpu is green, JS stuff is yellow, etc.

Nodes

ash - Vulkan bindings we use in gfx-rs, external to us.

metal-rs - Metal bindings crate we use in gfx-rs, also used outside.

d3d12-rs - simple D3D12 convenience wrapper we use in gfx-rs.

winapi - WinAPI bindings we use in gfx-rs for both DX11 and DX12 (where d3d12-rs has gaps), external to us.

glow - OpenGL (including ES and WebGL) bindings we use in gfx-rs, also used outside.

gfx-rs - our portable graphics API in Rust, tightly following Vulkan.

gfx-portability - a Vulkan Portability API wrapper around gfx-rs. Can be used as a drop-in Vulkan driver on platforms without native Vulkan support.

SPIRV-Cross - shader translation library in C++, taking SPIR-V as an input. It’s developed by a few Khronos members. We currently use it for generating platform-specific shaders in gfx-rs. We are phasing it away, replacing by naga gradually (the striped fill signifies deprecation in the diagram).

naga - our new shader translation library in Rust, it has a number of front-ends (SPIR-V, GLSL, and WGSL so far), various backends (SPIR-V, MSL, GLSL so far), and a solid intermediate representation (IR) in the middle. It’s a young project, and we are slowly rolling it out to replace the C++ blocks around the ecosystem, such as SPIRV-Cross and glsl-to-spirv/shaderc (used by many gfx/wgpu/Vulkan devs).

wgpu - our implementation of WebGPU API in Rust. It’s safe, portable, and fast (in this order).

  • Uses gfx-rs to reach the hardware.
  • Uses naga to parse WGSL shaders, as well as introspect both SPIR-V and WGSL shaders. This allows us to validate the shaders, also to derive the implicit bind group layouts.
  • able to record and replay API traces across platforms.

wgpu-rs - idiomatic Rust bindings to wgpu, meant for Rust end-users. Has many dependent applications and libraries, including Nannou and Bevy.

  • Currently able to target wgpu as well as the WebGPU/WASM (experimental in browsers).
  • Will be able to link to an implementation behind the portable WebGPU C header, such as wgpu-native.

wgpu-native - a wrapper around wgpu that exposes a WebGPU C API conforming to the shared header. The plan is to have it accessible by non-Rust libraries, as a drop-in replacement for Dawn, also accessible via NAPI.

Deno - JS/TS runtime in Rust. There is a mostly complete PR #7977 delivering support for WebGPU via wgpu. It should allow native JS/TS applications (such as TensorFlow.js) using WebGPU to run directly on top of wgpu in the future.

Servo - an experimental browser engine in Rust. It has WebGPU support via wgpu.

Gecko - the browser engine powering Firefox, developed by Mozilla. It also implements WebGPU via wgpu.


My Blog Just Got Faster: Cloudflare Workers and AVIF Support Matthias Endler

Matthias Endler2020-09-14 00:00:00

Did I mention that this website is fast? Oh yeah, I did, multiple times.

Few reasons (from ordinary to the first signs of creeping insanity):

  • 📄 Static site
  • ☁️ Cached on Cloudflare CDN
  • 🔗 ️HTTP/2 and HTTP/3 support
  • 🚫 No web fonts (sadly)
  • Edge-worker powered analytics (no Google Analytics)
  • 🌸 Avoiding JavaScript whenever possible; CSS covers 90% of my use-cases.
  • 🖼️ Image width and height specified in HTML to avoid page reflows.
  • 👍🏻 Inlined, optimized SVG graphics and hand-rolled CSS
  • 🚅 Static WASM search (lazy loaded)
  • 🏎️ The entire homepage is <10K (brotli-compressed), including graphics, thus should fit into the first HTTP round-trip.
  • 💟 Heck, even the favicon is optimized for size. Update: I'm using an SVG icon now thanks to this article.

Then again, it's 2020: everyone is optimizing their favicons, right? ...right!?

Well, it turns out most other sites don't think about their user's data plans as much as I do. Actually, that's an understatement: they don't care at all. But to me, lean is beautiful!

Wait, What About Images?

I prefer SVG for diagrams and illustrations. Only if it's a photo, I'll use JPEG or WebP.

To be honest with you, I never really liked WebP. The gist is that it might not even be smaller than JPEGs compressed with MozJPEG. There is a lengthy debate on the Mozilla bug tracker if you want to read more. To this day, Safari doesn't support WebP.

Hello AVIF 👋

Meet AVIF, the new next-gen image compression format. Check this out:

[ReachLightSpeed.com](https://reachlightspeed.com/blog/using-the-new-high-performance-avif-image-format-on-the-web-today/)
Source: ReachLightSpeed.com

It's already supported by Chrome 85 and Firefox 80.
Then it hit me like a hurricane 🌪️:

😲 Holy smokes, AVIF is supported by major browsers now!?
I want this for my blog!

Yes and no.

I'm using Zola for my blog, and AVIF support for Zola is not yet there, but I want it now! So I whipped up an ugly Rust script (as you do) that creates AVIF images from my old JPEG and PNG images. I keep the original raw files around just in case.

Under the hood, it calls cavif by Kornel Lesiński.

Data Savings

The results of AVIF on the blog were nothing short of impressive:

Total image size for [endler.dev/2020/sponsors](https://endler.dev/2020/sponsors)
Total image size for endler.dev/2020/sponsors

Check Your Browser

But hold on for a sec... is your browser even capable of showing AVIF?

If that reads "yup," you're all set.
If that reads "nope," then you have a few options:

  • On Firefox: Open about:config from the address bar and search for avif.
  • On Chrome: Make sure to update to the latest version.
  • On Safari: I'm not sure what you're doing with your life. Try a real browser instead. 😏

Workaround I: Fallback For Older Browsers

HTML is great in that your browser ignores unknown new syntax. So I can use the <picture> element to serve the right format to you. (Look ma, no JavaScript!)

<picture>
  <source srcset="fancy_browser.avif" />
  <source srcset="decent_browser.webp" />
  <img src="meh_browser.jpg" />
</picture>

The real thing is a bit more convoluted, but you get the idea.

Workaround II: Wrong Content-Type On Github Pages

There was one ugly problem with Github and AVIF, though: Their server returned a Content-Type: application/octet-stream header.

This meant that the images did not load on Firefox.

There is no way to fix that on my side as Github is hosting my page. Until now! I wanted to try Cloudflare's Workers Sites for a long time, and this bug finally made me switch. Basically, I run the full website as an edge worker right on the CDN; no own web server is needed. What's great about it is that the site is fast everywhere now — even in remote locations — no more roundtrips to a server.

By running an edge worker, I also gained full control over the request- and response objects. I added this gem of a snippet to intercept the worker response:

if (/\.avif$/.test(url)) {
  response.headers.set("Content-Type", "image/avif");
  response.headers.set("Content-Disposition", "inline");
}

And bam, Bob's your uncle. Firefox is happy. You can read more about modifying response objects here.

Another side-effect of Workers Sites is that a production deployment takes one minute now.

Performance Results After Moving To Cloudflare

Website response time before
Website response time before
Source: KeyCDN
Website response time after
Website response time after
Source: KeyCDN

Page size and rating before
Page size and rating before
Source: Pingdom.com
Page size and rating after
Page size and rating after
Source: Pingdom.com

I don't have to hide from a comparison with well-known sites either:

Comparison with some other blogs I read
Comparison with some other blogs I read
Source: Speedcurve

Further reading


Launching a Side Project Backed by Github Sponsors Matthias Endler

Matthias Endler2020-08-21 00:00:00

Yesterday we launched analysis-tools.dev, and boy had I underestimated the response.

It's a side project about comparing static code analysis tools. Static analysis helps improve code quality by detecting bugs in source code without even running it.

What's best about the project is that it's completely open-source. We wanted to build a product that wouldn't depend on showing ads or tracking users. Instead, we were asking for sponsors on Github — that's it. We learned a lot in the process, and if you like to do the same, keep reading!

First, Some Stats

Everyone likes business metrics. Here are some of ours:

  • The project started as an awesome list on Github in December 2015.
  • We're currently listing 470 static analysis tools.
  • Traffic grew continuously. Counting 7.5k stars and over 190 contributors at the moment.
  • 500-1000 unique users per week.
  • I had the idea to build a website for years now, but my coworker Jakub joined in May 2020 to finally make it a reality.
Github stars over time. That graph screams BUSINESS OPPORTUNITY.
Github stars over time. That graph screams BUSINESS OPPORTUNITY.
Source: star-history.t9t.io

"Why did it take five years to build a website!?", I hear you ask. Because I thought the idea was so obvious that others must have tried before and failed.

I put it off, even though nobody stepped in to fill this niche.
I put it off, even though I kept the list up-to-date for five years, just to learn about the tools out there.
You get the gist: don't put things off for too long. When ideas sound obvious, it's probably because they are.

Revenue Model

It took a while to figure out how to support the project financially. We knew what we didn't want: an SEO landfill backed by AdWords. Neither did we want to "sell user data" to trackers.

We owe it to the contributors on Github to keep all data free for everyone. How could we still build a service around it? Initially, we thought about swallowing the infrastructure costs ourselves, but we'd have no incentive to maintain the site or extend it with new features.

Github Sponsors was still quite new at that time. Yet, as soon as we realized that it was an option, it suddenly clicked: Companies that are not afraid of a comparison with the competition have an incentive to support an open platform that facilitates that. Furthermore, we could avoid bias and build a product that makes comparing objective and accessible.

Sponsoring could be the antidote to soulless growth and instead allow us to build a lean, sustainable side business. We don't expect analysis-tools.dev ever to be a full-time job. The market might be too small for that — and that's fine.

Tech

Once we had a revenue model, we could focus on the tech. We're both engineers, which helps with iterating quickly.

Initially, I wanted to build something fancy with Yew. It's a Rust/Webassembly framework and your boy likes Rust/Webassembly...

I'm glad Jakub suggested something else: Gatsby. Now, let me be honest with you: I couldn't care less about Gatsby. And that's what I said to Jakub: "I couldn't care less about Gatsby." But that's precisely the point: not being emotionally attached to something makes us focus on the job and not the tool. We get more stuff done!

From there on, it was pretty much easy going: we used a starter template, Jakub showed me how the GraphQL integration worked, and we even got to use some Rust! The site runs on Cloudflare as an edge worker built on top of Rust. (Yeah, I cheated a bit.)

Count to three, MVP!

Finding Sponsors

So we had our prototype but zero sponsors so far. What started now was (and still is) by far the hardest part: convincing people to support us.

We were smart enough not to send cold e-mails because most companies ignore them. Instead, we turned to our network and realized that developers reached out before to add their company's projects to the old static analysis list on Github.

These were the people we contacted first. We tried to keep the messages short and personal.

What worked best was a medium-sized e-mail with some context and a reminder that they contributed to the project before. We included a link to our sponsors page.

Businesses want reliable partners and a reasonable value proposal, so a prerequisite is that the sponsor page has to be meticulously polished.

Our Github Sponsors page
Our Github Sponsors page

Just like Star Wars Episode IX, we received mixed reviews: many people never replied, others passed the message on to their managers, which in turn never replied, while others again had no interest in sponsoring open-source projects in general. That's all fair game: people are busy, and sponsorware is quite a new concept.

A little rant: I'm of the opinion that tech businesses don't nearly sponsor enough compared to all the value they get from Open Source. Would your company exist if there hadn't been a free operating system like Linux or a web server like Nginx or Apache when it was founded?

There was, however, a rare breed of respondents, which expressed interest but needed some guidance. For many, it is the first step towards sponsoring any developer through Github Sponsors / OpenCollective.

It helped that we use OpenCollective as our fiscal host, which handles invoicing and donation transfers. Their docs helped us a lot when getting started.

The task of finding sponsors is never done, but it was very reassuring to hear from DeepCode - an AI-based semantic analysis service, that they were willing to take a chance on us.

Thanks to them, we could push product over the finishing line. Because of them, we can keep the site free for everybody. It also means the website is kept free from ads and trackers.

In turn, DeepCode gets exposed to many great developers that care about code quality and might become loyal customers. Also, they get recognized as an open-source-friendly tech company, which is more important than ever if you're trying to sell dev tools. Win-win!

Marketing

Jakub and I both had started businesses before, but this was the first truly open product we would build.

Phase 1: Ship early 🚀

We decided for a soft launch: deploy the site as early as possible and let the crawlers index it. The fact that the page is statically rendered and follows some basic SEO guidelines sure helped with improving our search engine rankings over time.

Phase 2: Ask for feedback from your target audience 💬

After we got some organic traffic and our first votes, we reached out to our developer friends to test the page and vote on tools they know and love. This served as an early validation, and we got some honest feedback, which helped us catch the most blatant flaws.

Phase 3: Prepare announcement post 📝

We wrote a blog post which, even if clickbaity, got the job done: Static Analysis is Broken — Let's Fix It! It pretty much captures our frustration about the space and why building an open platform is important. We could have done a better job explaining the technical differences between the different analysis tools, but that's for another day.

Phase 4: Announce on social media 🔥

Shortly before the official announcement, we noticed that the search functionality was broken (of course). Turns out, we hit the free quota limit on Algolia a biiit earlier than expected. 😅 No biggie: quick exchange with Algolia's customer support, and they moved us over to the open-source plan (which we didn't know existed). We were back on track!

Site note: Algolia customer support is top-notch. Responsive, tech-savvy, and helpful. Using Algolia turned out to be a great fit for our product. Response times are consistently in the low milliseconds and the integration with Gatsby was quick and easy.

We got quite a bit of buzz from that
tweet: 63 retweets, 86 likes and counting
We got quite a bit of buzz from that tweet: 63 retweets, 86 likes and counting

Clearly, everyone knew that we were asking for support here, but we are thankful for every single one that liked and retweeted. It's one of these situations where having a network of like-minded people can help.

As soon as we were confident that the site wasn't completely broken, we set off to announce it on Lobste.rs (2 downvotes), /r/SideProject (3 upvotes) and Hacker News (173 upvotes, 57 comments). Social media is kind of unpredictable. It helps to cater the message to each audience and stay humble, though.

The response from all of that marketing effort was nuts:

Traffic on launch day
Traffic on launch day

Perhaps unsurprisingly, the Cloudflare edge workers didn't break a sweat.

Edge worker CPU time on Cloudflare
Edge worker CPU time on Cloudflare

My boss Xoan Vilas even did a quick performance analysis and he approved. (Thanks boss!)

High fives all around!

Now what?

Of course, we'll add new features; of course, we have more plans for the future, yada yada yada. Instead, let's reflect on that milestone: a healthy little business with no ads or trackers, solely carried by sponsors. 🎉

Finally, I want you to look deep inside yourself and find your own little product to work on. It's probably right in front of your nose, and like myself, you've been putting it off for too long. Well, not anymore! The next success story is yours. So go out and build things.

Oh wait! ...before you leave, would you mind checking out analysis-tools.dev and smashing that upvote button for a few tools you like? Hey, and if you feel super generous today (or you have a fabulous employer that cares about open-source), why not check out our sponsorship page?

Jakub and me in Vienna, Austria. I'm not actually that small.
Jakub and me in Vienna, Austria. I'm not actually that small.

Release of v0.6 gfx-rs nuts and bolts

gfx-rs nuts and bolts2020-08-18 00:00:00

gfx-rs community’s goal is to make graphics programming in Rust easy, fast, and reliable. It governs a wide range of projects:

  • gfx-rs makes low-level GPU programming portable with low overhead. It’s a single Vulkan-like Rust API with multiple backends that implement it: Direct3D 12/11, Metal, Vulkan, and even OpenGL.
  • gfx-extras provides ready-to-use allocators for memory and descriptors in gfx-rs.
  • wgpu-rs is built on top of gfx-rs and gfx-extras. It provides safety, accessibility, and even stronger portability of applications.
  • metal-rs and d3d12-rs wrap native graphics APIs on macOS and Windows 10 in Rust.

Today, we are happy to announce the release of 0.6 versions across gfx/wgpu projects!

gfx-hal-0.6

On gfx-rs project side, the biggest change is converting the incoming lists to be ExactSizeIterator-bound. That allows our backends to stack-allocate any derived data, lowering the overhead.

Other changes are split between incremental improvements, like unbound ranges for mipmap levels and array layers, and totally experimental features, like the mesh shaders.

In the backend space, Vulkan now uses stack allocation everywhere, and D3D12 has seen a number of critical correctness fixes. Hopefully, the latter will get more use and testing in this release, as it’s approaching maturity. At the same time, we had to temporarily disable OpenGL backend, as it’s going through migration to surfman, and it’s not clear now what the future is for this library, given the latest devastating news about Mozilla.

In gfx-memory (of extras), the API is brought closer to gfx-hal, and heuristics are improved, especially for the outlier cases. We are now fuzzy-testing both memory and descriptor allocators.

wgpu-0.6

vangers wgpu shadows

wgpu is rapidly maturing and growing the ecosystem around it. In wgpu-rs on the Web we listed many projects taking advantage of wgpu. During the past 4 months this list more than doubled, and we couldn’t contain all the best parts in the README, so we moved it to a wiki page. We are also featuring a show case on wgpu.rs with some of the interesting projects.

Most important changes in wgpu were about us trying to figure out the proper components, targets, and the API boundaries between them. We ended up with wgpu-core crate having a safe pure-Rust internal API, implementing WebGPU specification. Everything else is built on top of it:

  • wgpu-rs - the idiomatic Rust wrapper
  • wgpu-native - the C API wrapper, aiming to bbe compatible with Dawn
  • Gecko and Servo - for implementing WebGPU API in the browsers

The path to this model was hard and full of unexpected turns. We experienced the bovine invasion, with Cows taking a stronghold inside wgpu repository. We’ve gone through non-exhaustive structs with builders, and then back to plain structs. The best part about it was the unprecedented initiative by our contributors, who weren’t afraid to do large changes, experiment, and scrap intermediate results. Thank you all for surviving this!

In infrastructure, we got an ability to record API traces, replay them on a different machine, and run data-driven tests. Read more on kvark’s blog about this feature! It allows us to easily share workloads that exhibit issues, and fix them promptly, improving the experience for users and developers alike.

On the API side, the most notable additions are write_buffer and write_texture, which allow users to update their GPU data without intermediate staging buffers or encoders. We’ve also got the staging belt implemented for those seeking more control.

We now support a number of powerful native-only extensions, such as descriptor indexing, as well as web-compatible extensions like depth-clamping. Targeting the web is supported, but it’s not useful for the published crate, because the WebGPU API is still evolving and so does browser support for it. Therefore, we recommend anyone who wants to test the Web to stick to the gecko branch.

Finally, the work is ongoing to validate all the use of the API and make it truly safe and portable. Most of the remaining work is related to the shader interface matching, as well as securing the resource accesses from shaders. This largely depends on the progress in next section.

naga-0.2

Naga is our emerging pure-Rust shader infrastructure to convert anything to anything, as well as transform the code in the middle. It aims to be fast and robust, also compiling for WASM target to be used on the Web. It was previously announced as Javelin project but now restarted. We are no longer basing it on rspirv, and the name had to change because it appeared to be reseved on crates.

Naga has modular structure that consists of:

  • front-ends (like WGSL or SPIR-V)
  • intermediate representation (similar to WGSL AST)
  • backends (SPIR-V, HLSL, MSL, etc)

There are many scenarios which Naga wants to help with. One goal is to be able to load SPIR-V or WGSL and reflect it, exposing the shader requirements and the interface to wgpu for validation. Another goal is to parse WGSL, insert bound checks, and produce SPIR-V - this would allow wgpu to accept WGSL like a proper WebGPU implementation, and then feed the SPIR-V output into gfx-rs as usual.

We also want it to parse GLSL and generate SPIR-V, thus replacing the disaster of glsl-to-spirv converters that the Rust ecosystem has been suffering from. Although in the longer run we expect more people choosing WGSL instead of GLSL for their projects.

Finally, but perhaps most importantly, Naga needs to be able to generate the platform-dependent code for all target platforms. This will essentially allow it to replace SPIRV-Cross in gfx-rs, removing this big and problematic native dependency, making our shader translation faster and more robust, and opening the doors to innovation on this front.

We released version 0.2, but this is still heavily a work-in-progress. We invite people interested in graphics and compiler design - collaborating on Naga should be easy given the modular structure, and honestly a joy to develop with the power of Rust ;)


What Happened To Programming In The 2010s? Matthias Endler

Matthias Endler2020-07-02 00:00:00

A while ago, I read an article titled "What Happened In The 2010s" by Fred Wilson. The post highlights key changes in technology and business during the last ten years. This inspired me to think about a much more narrow topic: What Happened To Programming In The 2010s?

🚓 I probably forgot like 90% of what actually happened. Please don't sue me. My goal is to reflect on the past so that you can better predict the future.

Where To Start?

From a mile-high perspective, programming is still the same as a decade ago:

  1. Punch program into editor
  2. Feed to compiler (or interpreter)
  3. Bleep Boop 🤖
  4. Receive output

But if we take a closer look, a lot has changed around us. Many things we take for granted today didn't exist a decade ago.

What Happened Before?

Back in 2009, we wrote jQuery plugins, ran websites on shared hosting services, and uploaded content via FTP. Sometimes code was copy-pasted from dubious forums, tutorials on blogs, or even hand-transcribed from books. Stack Overflow (which launched on 15th of September 2008) was still in its infancy. Version control was done with CVS or SVN — or not at all. I signed up for Github on 3rd of January 2010. Nobody had even heard of a Raspberry Pi (which only got released in 2012).

<a href='https://xkcd.com/2324/'>xkcd #2324</a>
Source: xkcd #2324

An Explosion Of New Programming Languages

The last decade saw the creation of a vast number of new and exciting programming languages.

Crystal, Dart, Elixir, Elm, Go, Julia, Kotlin, Nim, Rust, Swift, TypeScript all released their first stable version!

Even more exciting: all of the above languages are developed in the open now, and the source code is freely available on Github. That means, everyone can contribute to their development — a big testament to Open Source.

Each of those languages introduced new ideas that were not widespread before:

  • Strong Type Systems: Kotlin and Swift made optional null types mainstream, TypeScript brought types to JavaScript, Algebraic datatypes are common in Kotlin, Swift, TypeScript, and Rust.
  • Interoperability: Dart compiles to JavaScript, Elixir interfaces with Erlang, Kotlin with Java, and Swift with Objective-C.
  • Better Performance: Go promoted Goroutines and channels for easier concurrency and impressed with a sub-millisecond Garbage Collector, while Rust avoids Garbage Collector overhead altogether thanks to ownership and borrowing.

This is just a short list, but innovation in the programming language field has greatly accelerated.

More Innovation in Older Languages

Established languages didn't stand still either. A few examples:

C++ woke up from its long winter sleep and released C++11 after its last major release in 1998. It introduced numerous new features like Lambdas, auto pointers, and range-based loops to the language.

At the beginning of the last decade, the latest PHP version was 5.3. We're at 7.4 now. (We skipped 6.0, but I'm not ready to talk about it yet.) Along the way, it got over twice as fast. PHP is a truly modern programming language now with a thriving ecosystem.

Heck, even Visual Basic has tuples now. (Sorry, I couldn't resist.)

Faster Release Cycles

Most languages adopted a quicker release cycle. Here's a list for some popular languages:

LanguageCurrent release cycle
Cirregular
C#~ 12 months
C++~ 3 years
Go6 months
Java6 months
JavaScript (ECMAScript)12 months
PHP12 months
Python12 months
Ruby12 months
Rust6 weeks (!)
Swift6 months
Visual Basic .NET~ 24 months

The Slow Death Of Null

Close to the end of the last decade, in a talk from 25thof August 2009, Tony Hoare described the null pointer as his Billion Dollar Mistake.

A study by the Chromium project found that 70% of their serious security bugs were memory safety problems (same for Microsoft). Fortunately, the notion that our memory safety problem isn't bad coders has finally gained some traction.
Many mainstream languages embraced safer alternatives to null: nullable types, Option, and Result types. Languages like Haskell had these features before, but they only gained popularity in the 2010s.

Revenge of the Type System

Closely related is the debate about type systems. The past decade has seen type systems make their stage comeback; TypeScript, Python, and PHP (just to name a few) started to embrace type systems.

The trend goes towards type inference: add types to make your intent clearer for other humans and in the face of ambiguity — otherwise, skip them. Java, C++, Go, Kotlin, Swift, and Rust are popular examples with type inference support. I can only speak for myself, but I think writing Java has become a lot more ergonomic in the last few years.

Exponential Growth Of Libraries and Frameworks

As of today, npm hosts 1,330,634 packages. That's over a million packages that somebody else is maintaining for you. Add another 160,488 Ruby gems, 243,984 Python projects, and top it off with 42,547 Rust crates.

Number of packages for popular programming languages.<br /> Don't ask me what happened to npm in 2019.
Number of packages for popular programming languages.
Don't ask me what happened to npm in 2019.
Source: Module Counts

Of course, there's the occasional leftpad, but it also means that we have to write less library code ourselves and can focus on business value instead. On the other hand, there are more potential points of failure, and auditing is difficult. There is also a large number of outdated packages. For a more in-depth discussion, I recommend the Census II report by the Linux Foundation & Harvard [PDF].

We also went a bit crazy on frontend frameworks:

No Free Lunch

A review like this wouldn't be complete without taking a peek at Moore's Law. It has held up surprisingly well in the last decade:

<a href='https://en.wikipedia.org/wiki/Moore%27s_law'>Wikipedia</a>
Source: Wikipedia

There's a catch, though. Looking at single-core performance, the curve is flattening:

<a href='https://www.youtube.com/watch?v=Azt8Nc-mtKM&'>Standford University: The Future of Computing (video)</a>
Source: Standford University: The Future of Computing (video)

The new transistors prophesied by Moore don’t make our CPUs faster but instead add other kinds of processing capabilities like more parallelism or hardware encryption. There is no free lunch anymore. Engineers have to find new ways of making their applications faster, e.g. by embracing concurrent execution.

Callbacks, coroutines, and eventually async/await are becoming industry standards.

GPUs (Graphical Processing Units) became very powerful, allowing for massively parallel computations, which caused a renaissance of Machine Learning for practical use-cases:

Deep learning becomes feasible, which leads to machine learning becoming integral to many widely used software services and applications. — Timeline of Machine Learning on Wikipedia

Compute is ubiquitous, so in most cases, energy efficiency plays a more prominent role now than raw performance (at least for consumer devices).

Unlikely Twists Of Fate

Learnings

If you're now thinking: Matthias, you totally forgot X, then I brought that point home. This is not even close to everything that happened. You'd roughly need a decade to talk about all of it.

Personally, I'm excited about the next ten years. Software is eating the world — at an ever-faster pace.


Tips for Faster Rust Compile Times Matthias Endler

Matthias Endler2020-06-21 00:00:00

When it comes to runtime performance, Rust is one of the fastest guns in the west. 🔫 It is on par with the likes of C and C++ and sometimes even surpasses those. Compile times, however? That's another story.

Below is a list of tips and tricks on how to make your Rust project compile faster today. They are roughly ordered by practicality, so start at the top and work your way down until you're happy and your compiler goes brrrrrrr.

Table of Contents

Why Is Rust Compilation Slow?

Wait a sec, slow in comparison to what? That is, if you compare Rust with Go, the Go compiler is doing a lot less work in general. For example, it lacks support for generics and macros. On top of that, the Go compiler was built from scratch as a monolithic toolchain consisting of both, the frontend and the backend (rather than relying on, say, LLVM to take over the backend part, which is the case for Rust or Swift). This has advantages (more flexibility when tweaking the entire compilation process, yay) and disadvantages (higher overall maintenance cost and fewer supported architectures).

In general, comparing across different programming languages makes little sense and overall, the Rust compiler is legitimately doing a great job. That said, above a certain project size, the compile times are... let's just say they could be better.

Why Bother?

According to the Rust 2019 survey, improving compile times is #4 on the Rust wishlist:

Rust Survey results 2019. (<a href='https://xkcd.com/303/'>Obligatory xkcd</a>.)
Rust Survey results 2019. (Obligatory xkcd.)

Compile-Time vs Runtime Performance

As is often cautioned in debates among their designers, programming language design is full of tradeoffs. One of those fundamental tradeoffs is runtime performance vs. compile-time performance, and the Rust team nearly always (if not always) chose runtime over compile-time.
Brian Anderson

Overall, there are a few features and design decisions that limit Rust compilation speed:

  • Macros: Code generation with macros can be quite expensive.
  • Type checking
  • Monomorphization: this is the process of generating specialized versions of generic functions. E.g., a function that takes an Into<String> gets converted into one that takes a String and one that takes a &str.
  • LLVM: that's the default compiler backend for Rust, where a lot of the heavy-lifting (like code-optimizations) takes place. LLVM is notorious for being slow.
  • Linking: Strictly speaking, this is not part of compiling but happens right after. It "connects" your Rust binary with the system libraries. cargo does not explicitly mark the linking step, so many people add it to the overall compilation time.

If you're interested in all the gory details, check out this blog post by Brian Anderson.

Update The Rust Compiler And Toolchain

Making the Rust compiler faster is an ongoing process, and many fearless people are working on it. Thanks to their hard work, compiler speed has improved 30-40% across the board year-to-date, with some projects seeing up to 45%+ improvements.

So make sure you use the latest Rust version:

rustup update

On top of that, Rust tracks compile regressions on a website dedicated to performance. Work is also put into optimizing the LLVM backend. Rumor has it that there's still a lot of low-hanging fruit. 🍇

Use cargo check Instead Of cargo build

Most of the time, you don't even have to compile your project at all; you just want to know if you messed up somewhere. Whenever you can, skip compilation altogether. What you need instead is laser-fast code linting, type- and borrow-checking.

For that, cargo has a special treat for you: ✨ cargo check ✨. Consider the differences in the number of instructions between cargo check on the left and cargo debug in the middle. (Pay attention to the different scales.)

Speedup factors: check 1, debug 5, opt 20
Speedup factors: check 1, debug 5, opt 20

A sweet trick I use is to run it in the background with cargo watch. This way, it will cargo check whenever you change a file.

Pro-tip: Use cargo watch -c to clear the screen before every run.

Use Rust Analyzer Instead Of Rust Language Server (RLS)

Another quick way to check if you set the codebase on fire is to use a "language server". That's basically a "linter as a service", that runs next to your editor.

For a long time, the default choice here was RLS, but lately, folks moved over to rust-analyzer, because it's more feature-complete and way more snappy. It supports all major IDEs. Switching to that alone might save your day.

Remove Unused Dependencies

So let's say you tried all of the above and find that compilation is still slow. What now?

Dependencies sometimes become obsolete thanks to refactoring. From time to time it helps to check if all of them are still needed to save compile time.

If this is your own project (or a project you like to contribute to), do a quick check if you can toss anything with cargo-udeps:

cargo install cargo-udeps && cargo +nightly udeps

Update Remaining Dependencies

Next, update your dependencies, because they themselves could have tidied up their dependency tree lately.

Take a deep dive with cargo-outdated or cargo tree (built right into cargo itself) to find any outdated dependencies. On top of that, use cargo audit to get notified about any vulnerabilities which need to be addressed, or deprecated crates which need a replacement.

Here's a nice workflow that I learned from /u/oherrala on Reddit:

  1. Run cargo update to update to the latest semver compatible version.
  2. Run cargo outdated -wR to find newer, possibly incompatible dependencies. Update those and fix code as needed.
  3. Find duplicate versions of a dependency and figure out where they come from: cargo tree --duplicate shows dependencies which come in multiple versions.
    (Thanks to /u/dbdr for pointing this out.)

Pro-tip: Step 3 is a great way to contribute back to the community! Clone the repository and execute steps 1 and 2. Finally, send a pull request to the maintainers.

Replace Heavy Dependencies

From time to time, it helps to shop around for more lightweight alternatives to popular crates.

Again, cargo tree is your friend here to help you understand which of your dependencies are quite heavy: they require many other crates, cause excessive network I/O and slow down your build. Then search for lighter alternatives.

Also, cargo-bloat has a --time flag that shows you the per-crate build time. Very handy!

Here are a few examples:

Here's an example where switching crates reduced compile times from 2:22min to 26 seconds.

Use Cargo Workspaces

Cargo has that neat feature called workspaces, which allow you to split one big crate into multiple smaller ones. This code-splitting is great for avoiding repetitive compilation because only crates with changes have to be recompiled. Bigger projects like servo and vector are using workspaces heavily to slim down compile times. Learn more about workspaces here.

Combine All Integration Tests In A Single Binary

Have any integration tests? (These are the ones in your tests folder.) Did you know that the Rust compiler will create a binary for every single one of them? And every binary will have to be linked individually. This can take most of your build time because linking is slooow. 🐢 The reason is that many system linkers (like ld) are single threaded.

👨‍🍳️💡‍️ A linker is a tool that combines the output of a compiler and mashes that into one executable you can run.

To make the linker's job a little easier, you can put all your tests in one crate. (Basically create a main.rs in your test folder and add your test files as mod in there.)

Then the linker will go ahead and build a single binary only. Sounds nice, but careful: it's still a trade-off as you'll need to expose your internal types and functions (i.e. make them pub).

Might be worth a try, though because a recent benchmark revealed a 1.9x speedup for one project.

This tip was brought to you by Luca Palmieri, Lucio Franco, and Azriel Hoh. Thanks!

Disable Unused Features Of Crate Dependencies

Check the feature flags of your dependencies. A lot of library maintainers take the effort to split their crate into separate features that can be toggled off on demand. Maybe you don't need all the default functionality from every crate?

For example, tokio has a ton of features that you can disable if not needed.

Another example is bindgen, which enables clap support by default for its binary usage. This isn't needed for library usage, which is the common use-case. Disabling that feature improved compile time of rust-rocksdb by ~13s and ~9s for debug and release builds respectively. Thanks to reader Lilian Anatolie Moraru for mentioning this.

⚠️ Fair warning: it seems that switching off features doesn't always improve compile time. (See tikv's experiences here.) It may still be a good idea for improving security be reducing the code's attack surface.

A quick way to list all features of a crate is cargo-feature-set.

Admittedly, features are not very discoverable at the moment because there is no standard way to document them, but we'll get there eventually.

Use A Ramdisk For Compilation

When starting to compile heavy projects, I noticed that I was throttled on I/O. The reason was that I kept my projects on a measly HDD. A more performant alternative would be SSDs, but if that's not an option, don't throw in the sponge just yet.

Ramdisks to the rescue! These are like "virtual harddisks" that live in system memory.

User moschroe_de shared the following snippet over on Reddit, which creates a ramdisk for your current Rust project (on Linux):

mkdir -p target && \
sudo mount -t tmpfs none ./target && \
cat /proc/mounts | grep "$(pwd)" | sudo tee -a /etc/fstab

On macOS, you could probably do something similar with this script. I haven't tried that myself, though.

Cache Dependencies With sccache

Another neat project is sccache by Mozilla, which caches compiled crates to avoid repeated compilation.

I had this running on my laptop for a while, but the benefit was rather negligible, to be honest. It works best if you work on a lot of independent projects that share dependencies (in the same version). A common use-case is shared build servers.

Cranelift – The Alternative Rust Compiler

Lately, I was excited to hear that the Rust project is using an alternative compiler that runs in parallel with rustc for every CI build: Cranelift, also called CG_CLIF.

Here is a comparison between rustc and Cranelift for some popular crates (blue means better):

LLVM compile time comparison between rustc and cranelift in favor of cranelift
LLVM compile time comparison between rustc and cranelift in favor of cranelift

Somewhat unbelieving, I tried to compile vector with both compilers.

The results were astonishing:

  • Rustc: 5m 45s
  • Cranelift: 3m 13s

I could really notice the difference! What's cool about this is that it creates fully working executable binaries. They won't be optimized as much, but they are great for local development.

A more detailed write-up is on Jason Williams' page, and the project code is on Github.

Switch To A Faster Linker

  • 🐧 Linux users: Try mold
  • 🍎 Apple users: Try zld
  • 🪟 Windows users: 🤷

The thing that nobody seems to target is linking time. For me, when using something with a big dependency tree like Amethyst, for example linking time on my fairly recent Ryzen 7 1700 is ~10s each time, even if I change only some minute detail only in my code. — /u/Almindor on Reddit

You can check how long your linker takes by running the following commands:

cargo clean
cargo +nightly rustc --bin <your_binary_name> -- -Z time-passes

It will output the timings of each step, including link time:

...
time:   0.000   llvm_dump_timing_file
time:   0.001   serialize_work_products
time:   0.002   incr_comp_finalize_session_directory
time:   0.004   link_binary_check_files_are_writeable
time:   0.614   run_linker
time:   0.000   link_binary_remove_temps
time:   0.620   link_binary
time:   0.622   link_crate
time:   0.757   link
time:   3.836   total
    Finished dev [unoptimized + debuginfo] target(s) in 42.75s

If the link steps account for a big percentage of the build time, consider switching over to a different linker. There are quite a few options.

According to the official documentation, "LLD is a linker from the LLVM project that is a drop-in replacement for system linkers and runs much faster than them. [..] When you link a large program on a multicore machine, you can expect that LLD runs more than twice as fast as the GNU gold linker. Your mileage may vary, though."

If you're on Linux you can switch to lld like so:

[target.x86_64-unknown-linux-gnu]
rustflags = [
    "-C", "link-arg=-fuse-ld=lld",
]

A word of caution: lld might not be working on all platforms yet. At least on macOS, Rust support seems to be broken at the moment, and the work on fixing it has stalled (see rust-lang/rust#39915).

Update: I recently learned about another linker called mold, which claims a massive 12x performance bump over lld. Compared to GNU gold, it's said to be more than 50x. Would be great if anyone could verify and send me a message.

Update II: Aaand another one called zld, which is a drop-in replacement for Apple's ld linker and is targeting debug builds. [Source]

The zld benchmarks are quite impressive.
The zld benchmarks are quite impressive.

Which one you want to choose depends on your requirements. Which platforms do you need to support? Is it just for local testing or for production usage?

mold is optimized for Linux, zld only works on macOS. For production use, lld might be the most mature option.

Faster Incremental Debug Builds On Macos

Rust 1.51 added an interesting flag for faster incremental debug builds on macOS. It can make debug builds up to seconds faster (depending on your use-case). Just add this to your Cargo.toml:

[profile.dev]
split-debuginfo = "unpacked"

Some engineers report that this flag alone reduces compilation times on macOS by 70%.

The flag might become the standard for macOS soon. It is already the default on nightly.

Tweak More Codegen Options / Compiler Flags

Rust comes with a huge set of settings for code generation. It can help to look through the list and tweak the parameters for your project.

There are many gems in the full list of codegen options. For inspiration, here's bevy's config for faster compilation.

Profile Compile Times

If you like to dig deeper, Rust compilation can be profiled with cargo rustc -- -Zself-profile. The resulting trace file can be visualized with a flamegraph or the Chromium profiler:

Image of Chrome profiler with all crates
Image of Chrome profiler with all crates
Source: Rust Lang Blog

There's also a cargo -Z timings feature that gives some information about how long each compilation step takes, and tracks concurrency information over time.

You might have to run it using the nightly compiler:

cargo +nightly build -Z timings

Another golden one is cargo-llvm-lines, which shows the number of lines generated and objects copied in the LLVM backend:

$ cargo llvm-lines | head -20

  Lines        Copies         Function name
  -----        ------         -------------
  30737 (100%)   1107 (100%)  (TOTAL)
   1395 (4.5%)     83 (7.5%)  core::ptr::drop_in_place
    760 (2.5%)      2 (0.2%)  alloc::slice::merge_sort
    734 (2.4%)      2 (0.2%)  alloc::raw_vec::RawVec<T,A>::reserve_internal
    666 (2.2%)      1 (0.1%)  cargo_llvm_lines::count_lines
    490 (1.6%)      1 (0.1%)  <std::process::Command as cargo_llvm_lines::PipeTo>::pipe_to
    476 (1.5%)      6 (0.5%)  core::result::Result<T,E>::map
    440 (1.4%)      1 (0.1%)  cargo_llvm_lines::read_llvm_ir
    422 (1.4%)      2 (0.2%)  alloc::slice::merge
    399 (1.3%)      4 (0.4%)  alloc::vec::Vec<T>::extend_desugared
    388 (1.3%)      2 (0.2%)  alloc::slice::insert_head
    366 (1.2%)      5 (0.5%)  core::option::Option<T>::map
    304 (1.0%)      6 (0.5%)  alloc::alloc::box_free
    296 (1.0%)      4 (0.4%)  core::result::Result<T,E>::map_err
    295 (1.0%)      1 (0.1%)  cargo_llvm_lines::wrap_args
    291 (0.9%)      1 (0.1%)  core::char::methods::<impl char>::encode_utf8
    286 (0.9%)      1 (0.1%)  cargo_llvm_lines::run_cargo_rustc
    284 (0.9%)      4 (0.4%)  core::option::Option<T>::ok_or_else

Avoid Procedural Macro Crates

Procedural macros are the hot sauce of Rust development: they burn through CPU cycles so use with care (keyword: monomorphization).

Update: Over on Twitter Manish pointed out that "the reason proc macros are slow is that the (excellent) proc macro infrastructure – syn and friends – are slow to compile. Using proc macros themselves does not have a huge impact on compile times." (This might change in the future.)

Manish goes on to say

This basically means that if you use one proc macro, the marginal compile time cost of adding additional proc macros is insignificant. A lot of people end up needing serde in their deptree anyway, so if you are forced to use serde, you should not care about proc macros.

If you are not forced to use serde, one thing a lot of folks do is have serde be an optional dependency so that their types are still serializable if necessary.

If you heavily use procedural macros in your project (e.g., if you use serde), you can try to sidestep their impact on compile times with watt, a tool that offloads macro compilation to Webassembly.

From the docs:

By compiling macros ahead-of-time to Wasm, we save all downstream users of the macro from having to compile the macro logic or its dependencies themselves.

Instead, what they compile is a small self-contained Wasm runtime (~3 seconds, shared by all macros) and a tiny proc macro shim for each macro crate to hand off Wasm bytecode into the Watt runtime (~0.3 seconds per proc-macro crate you depend on). This is much less than the 20+ seconds it can take to compile complex procedural macros and their dependencies.

Note that this crate is still experimental.

(Oh, and did I mention that both, watt and cargo-llvm-lines were built by David Tolnay, who is a frickin' steamroller of an engineer?)

Get Dedicated Hardware

If you reached this point, the easiest way to improve compile times even more is probably to spend money on top-of-the-line hardware.

Perhaps a bit surprisingly, the fastest machines for Rust compiles seem to be Apple machines with an M1 chip:

Rik Arends on Twitter
Rik Arends on Twitter

The benchmarks for the new Macbook Pro with M1 Max are absolutely ridiculous — even in comparison to the already fast M1:

ProjectM1 MaxM1 Air
Deno6m11s11m15s
MeiliSearch1m28s3m36s
bat43s1m23s
hyperfine23s42s
ripgrep16s37s

That's a solid 2x performance improvement compared to an already fast M1.

But if you rather like to stick to Linux, people also had great success with a multicore CPU like an AMD Ryzen Threadripper and 32 GB of RAM.

On portable devices, compiling can drain your battery and be slow. To avoid that, I'm using my machine at home, a 6-core AMD FX 6300 with 12GB RAM, as a build machine. I can use it in combination with Visual Studio Code Remote Development.

Compile in the Cloud

If you don't have a dedicated machine yourself, you can offload the compilation process to the cloud instead.
Gitpod.io is superb for testing a cloud build as they provide you with a beefy machine (currently 16 core Intel Xeon 2.80GHz, 60GB RAM) for free during a limited period. Simply add https://gitpod.io/# in front of any Github URL. Here is an example for one of my Hello Rust episodes.

Gitpod has a neat feature called prebuilds. From their docs:

Whenever your code changes (e.g. when new commits are pushed to your repository), Gitpod can prebuild workspaces. Then, when you do create a new workspace on a branch, or Pull/Merge Request, for which a prebuild exists, this workspace will load much faster, because all dependencies will have been already downloaded ahead of time, and your code will be already compiled.

Especially when reviewing pull requests, this could give you a nice speedup. Prebuilds are quite customizable; take a look at the .gitpod.yml config of nushell to get an idea.

Download ALL The Crates

If you have a slow internet connection, a big part of the initial build process is fetching all those shiny crates from crates.io. To mitigate that, you can download all crates in advance to have them cached locally. criner does just that:

git clone https://github.com/the-lean-crate/criner
cd criner
cargo run --release -- mine

The archive size is surprisingly reasonable, with roughly 50GB of required disk space (as of today).

Bonus: Speed Up Rust Docker Builds 🐳

Building Docker images from your Rust code? These can be notoriously slow, because cargo doesn't support building only a project's dependencies yet, invalidating the Docker cache with every build if you don't pay attention. cargo-chef to the rescue! ⚡

cargo-chef can be used to fully leverage Docker layer caching, therefore massively speeding up Docker builds for Rust projects. On our commercial codebase (~14k lines of code, ~500 dependencies) we measured a 5x speed-up: we cut Docker build times from ~10 minutes to ~2 minutes.

Here is an example Dockerfile if you're interested:

# Step 1: Compute a recipe file
FROM rust as planner
WORKDIR app
RUN cargo install cargo-chef
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

# Step 2: Cache project dependencies
FROM rust as cacher
WORKDIR app
RUN cargo install cargo-chef
COPY --from=planner /app/recipe.json recipe.json
RUN cargo chef cook --release --recipe-path recipe.json

# Step 3: Build the binary
FROM rust as builder
WORKDIR app
COPY . .
# Copy over the cached dependencies from above
COPY --from=cacher /app/target target
COPY --from=cacher /usr/local/cargo /usr/local/cargo
RUN cargo build --release --bin app

# Step 4:
# Create a tiny output image.
# It only contains our final binary.
FROM rust as runtime
WORKDIR app
COPY --from=builder /app/target/release/app /usr/local/bin
ENTRYPOINT ["./usr/local/bin/app"]

cargo-chef can help speed up your continuous integration with Github Actions or your deployment process to Google Cloud.

Drastic Measures: Overclock Your CPU? 🔥

⚠️ Warning: You can damage your hardware if you don't know what you are doing. Proceed at your own risk.

Here's an idea for the desperate. Now I don't recommend that to everyone, but if you have a standalone desktop computer with a decent CPU, this might be a way to squeeze out the last bits of performance.

Even though the Rust compiler executes a lot of steps in parallel, single-threaded performance is still quite relevant.

As a somewhat drastic measure, you can try to overclock your CPU. Here's a tutorial for my processor. (I owe you some benchmarks from my machine.)

Speeding Up Your CI Builds

If you collaborate with others on a Rust project, chances are you use some sort of continuous integration like Github Actions. Optimizing a CI build processes is a whole subject on its own. Thankfully Aleksey Kladov (matklad) collected a few tips on his blog. He touches on bors, caching, splitting build steps, disabling compiler features like incremental compilation or debug output, and more. It's a great read and you can find it here.

Upstream Work

Making the Rust compiler faster is an ongoing process, and many fearless people are working on it. Thanks to their hard work, compiler speed has improved 30-40% across the board year-to-date, with some projects seeing up to 45%+ improvements. On top of that, Rust tracks compile regressions on a website dedicated to performance

Work is also put into optimizing the LLVM backend. Rumor has it that there's still a lot of low-hanging fruit. 🍇

The Rust team is also continuously working to make the compiler faster. Here's an extract of the 2020 survey:

One continuing topic of importance to the Rust community and the Rust team is improving compile times. Progress has already been made with 50.5% of respondents saying they felt compile times have improved. This improvement was particularly pronounced with respondents with large codebases (10,000 lines of code or more) where 62.6% citing improvement and only 2.9% saying they have gotten worse. Improving compile times is likely to be the source of significant effort in 2021, so stay tuned!

Help Others: Upload Leaner Crates For Faster Build Times

cargo-diet helps you build lean crates that significantly reduce download size (sometimes by 98%). It might not directly affect your own build time, but your users will surely be thankful. 😊

More Resources

What's Next?

Phew! That was a long list. 😅 If you have any additional tips, please let me know.

If compiler performance is something you're interested in, why not collaborate on a tool to see what user code is causing rustc to use lots of time?

Also, once you're done optimizing your build times, how about optimizing runtimes next? My friend Pascal Hertleif has a nice article on that.


Gravity Matthias Endler

Matthias Endler2020-05-29 00:00:00

Here's a test to show your age:

Do you still remember that funny JavaScript gravity effect, which Google used on their homepage ten years ago? This one?

I wanted to have some fun and integrated it into a website I was building. Unfortunately, it didn't work out-of-the-box. It choked on some DOM elements that were not strictly classes (like SVG elements). So, in good hacker fashion, I quickly patched up the script (it's just a three-line change), and now it's back to its former glory.

Test it here! (Caution: you'll have to reload the page after that. 😏)

Apply Gravity

Anyway, feel free to add it to your own sites and have some fun. It's also great to prank your friends. Simply add that single line to any website and weeee!

<script
  type="text/javascript"
  src="https://endler.dev/2020/gravity/gravity.js"
></script>

Sometimes I miss those simple times of the early web...


Hacker Folklore Matthias Endler

Matthias Endler2020-04-24 00:00:00

Some computer terms have a surprising legacy. Many of them are derived from long-obsolete technologies. This post tries to dust off the exciting history of some of these terms that we use every day but aren't quite sure about their origins. Let's jump right in!

Bike-Shedding

Today's meaning: A pointless discussion about trivial issues.

The term bike-shed effect or bike-shedding was coined as a metaphor to illuminate the law of triviality; it was popularised in the Berkeley Software Distribution community by the Danish computer developer Poul-Henning Kamp in 1999 on the FreeBSD mailing list and has spread from there to the whole software industry.

The concept was first presented as a corollary of his broader "Parkinson's law" spoof of management. He dramatizes this "law of triviality" with the example of a committee's deliberations on an atomic reactor, contrasting it to deliberations on a bicycle shed. As he put it: "The time spent on any item of the agenda will be in inverse proportion to the sum of money involved."

A reactor is so vastly expensive and complicated that an average person cannot understand it, so one assumes that those who work on it understand it. On the other hand, everyone can visualize a cheap, simple bicycle shed, so planning one can result in endless discussions because everyone involved wants to add a touch and show personal contribution.
Reference - Wikipedia: Law of Triviality

Boilerplate

An old machine that bended steel
plates to water boilers.
An old machine that bended steel plates to water boilers.
Source: Wikimedia Commons

Today's meaning: A chunk of code that is copied over and over again with little or no changes made to it in the process.

Boiler plate originally referred to the rolled steel used to make water boilers but is used in the media to refer to hackneyed or unoriginal writing. The term refers to the metal printing plates of pre-prepared text such as advertisements or syndicated columns that were distributed to small, local newspapers. These printing plates came to be known as 'boilerplates' by analogy. One large supplier to newspapers of this kind of boilerplate was the Western Newspaper Union, which supplied "ready-to-print stories [which] contained national or international news" to papers with smaller geographic footprints, which could include advertisements pre-printed next to the conventional content.

References:

The man in the foreground is holding
a rounded printing plate. Plates like this were provided by companies such as
Western Newspaper Union to many smaller newspapers.
The man in the foreground is holding a rounded printing plate. Plates like this were provided by companies such as Western Newspaper Union to many smaller newspapers.
Source: Wikimedia Commons

Bug

Today's meaning: A defect in a piece of code or hardware.

The origins are unknown!

Contrary to popular belief it predates the bug found by Grace Hopper in the Mark II computer.

The term was used by engineers way before that; at least since the 1870s. It predates electronic computers and computer software. Thomas Edison used the term "bug" in his notes. Reference

Carriage Return and Line Feed

Today's meaning: Set the cursor to the beginning of the next line.

These two terms were adopted from typewriters.

The carriage holds the paper and is moving from left to right to advance the typing position as the keys are pressed. It "carries" the paper with it. The carriage return is the operation when the carriage gets moved into its original position on the very left end side of the paper.

Simply returning the carriage to the left is not enough to start with a new line, however. The carriage would still be on the same line than before — just at the beginning of the line. To go to a new line, a line feed was needed. It would move the paper inside the typewriter up by one line.

These two operations — carriage return (CR) and line feed (LF) — were commonly done at once by pushing the carriage return lever.

A mechanical typewriter. The lever for the carriage return is
on the outer left side.
A mechanical typewriter. The lever for the carriage return is on the outer left side.
Source: Source: piqsels
  • On Unix systems (like Linux or macOS), a \n still stands for a
    line feed (ASCII symbol: LF) or newline.
  • On CP/M, DOS, and Windows, \r\n is used, where \r stands for carriage return and \n stands for line feed (CR+LF).
  • Reference

Here is an old video that shows the basic mechanics of carriage return and line-feed:

Command key symbol (⌘)

Today's meaning: A meta-key available on Apple computers to provide additional keyboard combinations.

Directly quoting Wikipedia (emphasis mine):

The ⌘ symbol came into the Macintosh project at a late stage. The development team originally went for their old Apple key, but Steve Jobs found it frustrating when "apples" filled up the Mac's menus next to the key commands, because he felt that this was an over-use of the company logo. He then opted for a different key symbol. With only a few days left before deadline, the team's bitmap artist Susan Kare started researching for the Apple logo's successor. She was browsing through a symbol dictionary when she came across the cloverleaf-like symbol, commonly used in Nordic countries as an indicator of cultural locations and places of interest (it is the official road sign for tourist attraction in Denmark, Finland, Iceland, Norway, and Sweden and the computer key has often been called Fornminne — ancient monument — by Swedish Mac users and Seværdighedstegn by Danish users). When she showed it to the rest of the team, everyone liked it, and so it became the symbol of the 1984 Macintosh command key. Susan Kare states that it has since been told to her that the symbol had been picked for its Scandinavian usage due to its resembling the shape of a square castle with round corner towers as seen from above looking down, notably Borgholm Castle.

Norwegian Severdighet road sign
Norwegian Severdighet road sign
Source: Wikimedia Commons
Aearial view of Borgholm Castle, which could have been the model for the symbol
Aearial view of Borgholm Castle, which could have been the model for the symbol
Source: Wikimedia Commons

References:

Core Dump

Today's meaning: Retrieving a snapshot of a (crashed) program's state by storing all of its memory for offline analysis.

The name comes from magnetic core memory, which is an early storage mechanism based on a grid of toroid magnets. It has since become obsolete, but the term is still used today for getting a snapshot of a computer process. Reference

A 32 x 32 core memory plane storing
1024 bits (or 128 bytes) of data. The first core dumps were printed on paper, which sounds reasonable given these small amounts of bytes.
A 32 x 32 core memory plane storing 1024 bits (or 128 bytes) of data. The first core dumps were printed on paper, which sounds reasonable given these small amounts of bytes.
Source: Wikimedia Commons

Cursor

Today's meaning: a visual cue (such as a flashing vertical line) on a video display that indicates position (as for data entry). Merriam-Webster

Cursor is Latin for runner. A cursor is the name given to the transparent slide engraved with a hairline that is used for marking a point on a slide rule. The term was then transferred to computers through analogy. Reference

A December 1951 advertisement for the
IBM 604 Electronic Calculating Punch that was first produced in 1948. The
advertisement claims the IBM 604 can do the work of 150 engineers with slide
rules. The cursor (or runner) is the transparent part in the middle of the
slide.
Source: A December 1951 advertisement for the IBM 604 Electronic Calculating Punch that was first produced in 1948. The advertisement claims the IBM 604 can do the work of 150 engineers with slide rules. The cursor (or runner) is the transparent part in the middle of the slide.

Dashboard

Today's meaning: A user interface that provides a quick overview of a system's status.

Originally a plank of wood at the front of a horse-drawn carriage to protect the driver from mud 'dashed' backward by a horses hooves.

When automobiles were manufactured, the board in front of the driver was given the same name. That was the logical place to put the necessary gauges so the driver could see them easily. In time, the term became more associated with the readouts than the protection it offered. Reference

A dashboard of a horse carriage.
A dashboard of a horse carriage.
Source: Wikimedia Commons

Firewall

Today's meaning: A network security system that establishes a barrier between a trusted internal network and an untrusted external network, such as the Internet.

Fire walls are used mainly in terraced houses, but also in individual residential buildings. They prevent fire and smoke from spreading to another part of the building in the event of a fire. Large fires can thus be prevented. The term is used in computing since the 80s. Reference

Firewall residential construction, separating the building into two separate residential units, and fire areas.
Firewall residential construction, separating the building into two separate residential units, and fire areas.
Source: Wikimedia Commons

Firmware

Today's meaning: A class of computer software that provides the low-level control for the device's specific hardware and closely tied to the hardware it runs on.

Ascher Opler coined the term firmware in a 1967 Datamation article. As originally used, firmware contrasted with hardware (the CPU itself) and software (normal instructions executing on a CPU). It existed on the boundary between hardware and software; thus the name "firmware". The original article is available on the Internet Archive. Reference

Foo and Bar

Today's meaning: Common placeholder variable names.

Originally the term might come from the military term FUBAR. There are a few variations, but a common meaning is FUBAR: "f***ed up beyond all recognition".

The use of foo in a programming context is generally credited to the Tech Model Railroad Club (TMRC) of MIT from circa 1960. In the complex model system, there were scram switches located at numerous places around the room that could be thrown if something undesirable was about to occur, such as a train going full-bore at an obstruction.

The way I understood it was that they literally had emergency buttons labeled foo for lack of a better name. Maybe related to the original military meaning of FUBAR to indicate that something is going very very wrong.

A scram switch (button), that could be
pressed to prevent inadvertent operation. Maybe the TMRC had buttons labeled `foo` instead
A scram switch (button), that could be pressed to prevent inadvertent operation. Maybe the TMRC had buttons labeled foo instead
Source: Source Wikimedia Commons

References:

Freelancer

Today's meaning: A self-employed person, which is not committed to a particular employer long-term.

The term first appears in the novel Ivanhoe by Sir Walter Scott. (The novel also had a lasting influence on the Robin Hood legend.)

Cover of a Classic Comics book
Cover of a Classic Comics book
Source: Wikimedia Commons

In it, a Lord offers his paid army of 'free lances' to King Richard:

I offered Richard the service of my Free Lances, and he refused them — I will lead them to Hull, seize on shipping, and embark for Flanders; thanks to the bustling times, a man of action will always find employment.

Therefore, a "free lancer" is someone who fights for whoever pays the most. Free does not mean "without pay", but refers to the additional freedom to work for any employer. Reference

Log / Logfile

Today's meaning: A file that records events of a computer program or system.

Sailors used so-called log lines to measure the speed of their ship. A flat piece of wood (the log) was attached to a long rope. The log had regularly spaced knots in it. As the log would drift away, the sailors would count the number of knots that went out in a fixed time interval, and this would be the ship's speed — in knots.

The ship's speed was important for navigation, so the sailors noted it down in a book, aptly called the log book, together with other information to establish the position of the ship more accurately, like landmark sightings and weather events. Later, additional information, more generally concerning the ship, was added — or logged — such as harbor fees and abnormal provision depletion.

Reference.

Sailors measuring ship speed with a
log line
Sailors measuring ship speed with a log line
Source: The Pilgrims & Plymouth Colony:1620 by Duane A. Cline
The parts of a log-line
The parts of a log-line
Source: The Pilgrims & Plymouth Colony:1620 by Duane A. Cline
Page from the log-file of the British
Winchelsea. The second column denotes the number of knots measured with the
log-line, which indicates the ship's speed
Page from the log-file of the British Winchelsea. The second column denotes the number of knots measured with the log-line, which indicates the ship's speed
Source: Navigation and Logbooks in the Age of Sail by Peter Reaveley

Patch

Today's meaning: A piece of code that can be applied to fix or improve a computer program.

In the early days of computing history, if you made a programming mistake, you'd have to fix a paper tape or a punched card by putting a patch on top of a hole.

A program tape with physical patches used
to correct punched holes by covering them.
A program tape with physical patches used to correct punched holes by covering them.
Source: Smithsonian Archives Center

Ping

Today's meaning: A way to check the availability and response time of a computer over the network.

Ping is a terminal program originally written by Mike Muuss in 1983 that is included in every version of UNIX, Windows, and macOS. He named it "after the sound that a sonar makes, inspired by the whole principle of echo-location. [...] ping uses timed IP/ICMP ECHO_REQUEST and ECHO_REPLY packets to probe the "distance" to the target machine." The reference is well worth a read.

Pixel

Today's meaning: The smallest controllable element of a picture represented on the screen.

The word pixel is a combination of pix (from "pictures", shortened to "pics") and el (for "element"). Similarly, voxel is a volume element and texel is a texture element. Reference

Shell

Today's meaning: An interactive, commonly text-based runtime to interact with a computer system.

The inventor of the term, Louis Pouzin, does not give an explanation for the name in his essay The Origins of the Shell. It can however be traced back to Unix' predecessor Multics. It is described in the Multics glossary like so:

[The shell] is passed a command line for execution by the listener.

The The New Hacker's Dictionary, (also known as the Jargon File) by Eric S. Raymond contains the following:

Historical note: Apparently, the original Multics shell (sense 1) was so called because it was a shell (sense 3);

where sense 3 refers to

A skeleton program, created by hand or by another program (like, say, a parser generator), which provides the necessary incantations to set up some task and the control flow to drive it (the term driver is sometimes used synonymously). The user is meant to fill in whatever code is needed to get real work done. This usage is common in the AI and Microsoft Windows worlds, and confuses Unix hackers.

Unfortunately, the book does not provide any evidence to back up this claim.

I like the (possibly historically incorrect) analogy to a nut with the shell being on the outside, protecting the kernel.

Reference

Slab allocator

Today's meaning: An efficient memory allocation technique, which reuses previous allocations.

Slab allocation was invented by John Bonwick (Note: PDF file) in 1994 and has since been used by services like Memcached and the Linux Kernel.

With slab allocation, a cache for a certain type or size of data object has a number of pre-allocated "slabs" of memory; within each slab there are memory chunks of fixed size suitable for the objects. (Wikpedia)

The name slab comes from a teenage friend of Bonwick. He tells the story on the Oracle blog:

While watching TV together, a commercial by Kellogg's came on with the tag line, "Can you pinch an inch?"

The implication was that you were overweight if you could pinch more than an inch of fat on your waist — and that hoovering a bowl of corn flakes would help.

Without missing a beat, Tommy, who weighed about 250 pounds, reached for his midsection and offered his response: "Hell, I can grab a slab!"

A decade later, Bonwick remembered that term when he was looking for a word to describe the allocation of a larger chunk of memory.

Here is the original Kellogg's advertisement:

Spam

Today's meaning: Unsolicited electronic communications, for example by sending mass-emails or posting in forums and chats.

The term goes back to a sketch by the British comedy group Monty Python from 1970. In the sketch, a cafe is including Spam (a brand of canned cooked pork) in almost every dish. The excessive amount of Spam mentioned is a reference to the ubiquity of it and other imported canned meat products in the UK after World War II (a period of rationing in the UK) as the country struggled to rebuild its agricultural base. Reference

Vintage Ad: Look What You Can Do With One
Can of Spam
Vintage Ad: Look What You Can Do With One Can of Spam
Source: By user Jamie (jbcurio) on flickr.com

Monty Pythons Flying Circus (1974) - SPAM from Testing Tester on Vimeo.

Radio Button

Today's meaning: A UI element that allows to choose from a predefined set of mutually exclusive options

"Radio buttons" are named after the analogous pendant of mechanical buttons that were used in radios. The UI concept has later been used in tape recorders, cassette recorders and wearable audio players (the famous "Walkman" and similar). And later in VCRs and video cameras. Reference

An old car radio (left) and CSS
radio buttons (right). Only a single option can be selected at any point in
time. As a kid, I would push two buttons at once so they would interlock. Good
times.
An old car radio (left) and CSS radio buttons (right). Only a single option can be selected at any point in time. As a kid, I would push two buttons at once so they would interlock. Good times.
Source: Images by Matt Coady

Uppercase and lowercase

Today's meaning: Distinction between capital letters and small letters on a keyboard.

Back when typesetting was a manual process where single letters made of led were "type set" to form words and sentences, upper- and lowercase letters were kept in separate containers — or cases — to make this rather tedious process a little faster.

A set of printers cases
A set of printers cases
Source: From the book 'Printing types, their history, forms, and use; a study in survivals' by Updike, Daniel Berkeley, 1860-1941. Freely available on archive.org.

Honorable mentions

404

Today's meaning: HTTP Status Code for "File not found".

There is a story that the number comes from the server room where the World Wide Web's central database was located. In there, administrators would manually locate the requested files and transfer them, over the network, to the person who made that request. If a file didn't exist, they'd return an error message: "Room 404: file not found".

This, however, seems to be a myth and the status code was chosen rather arbitrarily based on the then well-established FTP status codes. Reference

Programming languages and Abbreviations

The etymology of programming language names and common abbreviations would probably warrant its own article, but I've decided to note down some of my favorites for the time being.

C++

C++ is a programming language based on C by Bjarne Stroustrup. The name is a programmer pun by Rick Mascitti, a coworker of Stroustrup. The ++ refers to the post-increment operator, that is common in many C-like languages. It increases the value of a variable by 1. In that sense, C++ can be seen as the spiritual "successor" of C. Reference

C Sharp

Similarly to C++, C# is a C-like programming language. The name again refers to "incremental" improvements on top of C++. The # in the name looks like four plus signs. Hence C# == (C++)++. But on top of that, the name was also inspired by the musical notation where a sharp indicates that the written note should be made a semitone higher in pitch. Reference

A C-Sharp note.
A C-Sharp note.
Source: Wikimedia Commons

PNG

Officially, PNG stands for Portable Network Graphics. It was born out of frustration over a CompuServe announcement in 1994 that programs supporting GIF would have to pay licensing fees from now on. A working group lead by hacker Thomas Boutell created the .webp file format, a patent-free replacement for GIF. Therefore I prefer the format's unofficial name: PNG's Not GIF. Here's a great article on PNG's history. Reference

Credits

Most of the content comes from sources like Wikipedia (with reference where appropriate), but the explanations are difficult to hunt down if you don't know what you're looking for.
This is a living document, and I'm planning to update it in case of reader submissions.

Conclusion

You have to know the past to understand the present.
— Dr. Carl Sagan (1980)

I hope you enjoyed this trip down memory lane. Now it's your turn!
👉 Do you know any other stories? Send me a message, and I'll add them here.


Open Source Virtual Background Posts on elder.dev

Posts on elder.dev2020-04-09 00:00:00 With many of us around the globe under shelter in place due to COVID-19 video calls have become a lot more common. In particular, ZOOM has controversially become very popular. Arguably Zoom’s most interesting feature is the “Virtual Background” support which allows users to replace the background behind them in their webcam video feed with any image (or video). I’ve been using Zoom for a long time at work for Kubernetes open source meetings, usually from my company laptop.

A Timelapse of Timelapse Matthias Endler

Matthias Endler2020-02-04 00:00:00

Timelapse is a little open-source screen recorder for macOS. It takes a screenshot every second and creates a movie in the end.

To celebrate its unlikely 1.0 release today, I present here a "timelapse" of this project's journey. It just took ten years to get here.

2011 - How it all began

To be honest, I don't remember why I initially wrote the tool. I must have had a personal need for a screen recorder, I guess...

In May 2011, when I started the project, I was doing my Masters Degree in Computer Science. I might have needed the tool for University; most likely, however, I was just trying to find an excuse for not working on an assignment.

During that time, I wrote a lot of tools like that. Mainly to scratch a personal itch, learn a new programming language, or just have fun.

Among them are tools like a random sandwich generator for Subway (the American fast-food chain), DrawRoom, a keyboard-driven drawing app inspired by WriteRoom, and the obligatory CMS software, that I sold to clients. Surprisingly, none of them were a great success.

DrawRoom, a tool that I wrote around the same time, is a real piece of art. To this day it has five commits and a single Github star (by myself, don't judge...).
DrawRoom, a tool that I wrote around the same time, is a real piece of art. To this day it has five commits and a single Github star (by myself, don't judge...).

What I do know for sure is that I was unhappy with all existing screen recorders. They could roughly be categorized into these three groups:

  • Proprietary solutions that cost money or could call home.
  • Tools that didn't work on macOS.
  • Small, fragile, one-off scripts that people passed around in forums or as Github gists. They rarely worked as advertised.

Among the remaining tools were none that provided any timelapse functionality; so I set out to write my own.

This all sounds very epic, but in reality, I worked on it for a day. After five heroic commits on May 11, 2011, it sat there, idle, for seven years...

2018

A lot of time elapsed before anything exciting happened.

In January '18, seemingly out of nowhere, the first user filed a bug report. It was titled hung when creating the avi 😱. Turns out that a game developer from Canada, juul1a, was trying to use the tool to track her progress on an indie game — how cool is that?

To help her out, I decided to do some general cleanup, finally write down some instructions on how to even use the program, add a requirements.txt, and port the tool from mencoder to ffmpeg.

After that, timelapse was ready for prime-time. 🎬 Here is some live action from her videos featuring timelapses:

At that point, the tool was still very wobbly and could only be used from the commandline, but I began to see some potential for building a proper app from it; I just never found the time.

In October '18, I decided to ask for support during Hacktoberfest. I created a few tickets and labeled them with hacktoberfest to try and find contributors.

And then, I waited.

First, Shreya V Prabhu fixed an issue where a new recording was overwriting the previous one by adding a timestamp to the video name. Then Abner Campanha and Shane Creedon created a basic test structure. Gbenro Selere added a CI pipeline for Travis CI. It really worked, and the project was in much better shape after that!

2019

One year passes by, and Kyle Jones adds some contribution guidelines, while I move the CI pipeline to the newly released Github actions.

Chaitanya fixed a bug where the program would hang when the recording stopped by moving the video creation from threads to a separate process. He continued to make the codebase more robust and became a core contributor, reviewing pull requests and handling releases.

Thanks to orcutt989, the app now made use of type hints in Python 3.6.

gkpln3 added support for multi-monitor configurations. The screen captured will always be the one with the mouse on it.

2020

Fast forward to today, and after almost ten years, we finally created a true macOS app using the awesome py2app bundler. This should make the tool usable by non-developers.

Back to the Future

We reached the end of our little journey.

A long time has passed until 1.0. This project is a testament to the wonders of open source collaboration, and I am proud to work on it with contributors from around the world. It doesn't have to be a life-changing project to bring people together who have fun building things. If this were the end of the story, I'd be okay with that. I doubt it, though. Here's to the next ten years!

🎬 Download timelapse on Github.

Bonus

The video at the beginning is a timelapse of how I finish this article.
How meta.


Github Stars Matthias Endler

Matthias Endler2020-01-01 00:00:00
RepositoryStars
analysis-tools-dev/static-analysis9823 ★
mre/idiomatic-rust3517 ★
tinysearch/tinysearch1936 ★
mre/the-coding-interview1474 ★
ReceiptManager/receipt-parser-legacy692 ★
analysis-tools-dev/dynamic-analysis550 ★
mre/hyperjson450 ★
lycheeverse/lychee400 ★
mre/cargo-inspect370 ★
hello-rust/show293 ★
mre/fcat228 ★
mre/kafka-influxdb216 ★
mre/prettyprint203 ★
mre/timelapse186 ★
mre/vscode-snippet148 ★
ReceiptManager/receipt-manager-app103 ★
lycheeverse/lychee-action90 ★
ReceiptManager/receipt-parser-server57 ★
mre/envy53 ★
mre/PHPench52 ★
hello-rust/hello51 ★
mre/futures-batch51 ★

A Note on Mask Registers Performance Matters

Performance Matters2019-12-05 16:30:00

AVX-512 introduced eight so-called mask registers1, k02 through k7, which apply to most ALU operations and allow you to apply a zero-masking or merging3 operation on a per-element basis, speeding up code that would otherwise require extra blending operations in AVX2 and earlier.

If that single sentence doesn’t immediately indoctrinate you into the mask register religion, here’s a copy and paste from Wikipedia that should fill in the gaps and close the deal:

Most AVX-512 instructions may indicate one of 8 opmask registers (k0–k7). For instructions which use a mask register as an opmask, register k0 is special: a hardcoded constant used to indicate unmasked operations. For other operations, such as those that write to an opmask register or perform arithmetic or logical operations, k0 is a functioning, valid register. In most instructions, the opmask is used to control which values are written to the destination. A flag controls the opmask behavior, which can either be “zero”, which zeros everything not selected by the mask, or “merge”, which leaves everything not selected untouched. The merge behavior is identical to the blend instructions.

So mask registers4 are important, but are not household names unlike say general purpose registers (eax, rsi and friends) or SIMD registers (xmm0, ymm5, etc). They certainly aren’t going to show up on Intel slides disclosing the size of uarch resources, like these:

Intel Slide


In particular, I don’t think the size of the mask register physical register file (PRF) has ever been reported. Let’s fix that today.

We use an updated version of the ROB size probing tool originally authored and described by Henry Wong5 (hereafter simply Henry), who used it to probe the size of various documented and undocumented out-of-order structures on earlier architecture. If you haven’t already read that post, stop now and do it. This post will be here when you get back.

You’ve already read Henry’s blog for a full description (right?), but for the naughty among you here’s the fast food version:

Fast Food Method of Operation

We separate two cache miss load instructions6 by a variable number of filler instructions which vary based on the CPU resource we are probing. When the number of filler instructions is small enough, the two cache misses execute in parallel and their latencies are overlapped so the total execution time is roughly7 as long as a single miss.

However, once the number of filler instructions reaches a critical threshold, all of the targeted resource are consumed and instruction allocation stalls before the second miss is issued and so the cache misses can no longer run in parallel. This causes the runtime to spike to about twice the baseline cache miss latency.

Finally, we ensure that each filler instruction consumes exactly one of the resource we are interested in, so that the location of the spike indicates the size of the underlying resource. For example, regular GP instructions usually consume one physical register from the GP PRF so are a good choice to measure the size of that resource.

Mask Register PRF Size

Here, we use instructions that write a mask register, so can measure the size of the mask register PRF.

To start, we use a series of kaddd k1, k2, k3 instructions, as such (shown for 16 filler instructions):

mov    rcx,QWORD PTR [rcx]  ; first cache miss load
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
kaddd  k1,k2,k3
mov    rdx,QWORD PTR [rdx]  ; second cache miss load
lfence                      ; stop issue until the above block completes
; this block is repeated 16 more times

Each kaddd instruction consumes one physical mask register. If number of filler instructions is equal to or less than the number of mask registers, we expect the misses to happen in parallel, otherwise the misses will be resolved serially. So we expect at that point to see a large spike in the running time.

That’s exactly what we see:

Test 27 kaddd instructions

Let’s zoom in on the critical region, where the spike occurs:

Test 27 zoomed

Here we clearly see that the transition isn’t sharp – when the filler instruction count is between 130 and 134, we the runtime is intermediate: falling between the low and high levels. Henry calls this non ideal behavior and I have seen it repeatedly across many but not all of these resource size tests. The idea is that the hardware implementation doesn’t always allow all of the resources to be used as you approach the limit8 - sometimes you get to use every last resource, but in other cases you may hit the limit a few filler instructions before the theoretical limit.

Under this assumption, we want to look at the last (rightmost) point which is still faster than the slow performance level, since it indicates that sometimes that many resources are available, implying that at least that many are physically present. Here, we see that final point occurs at 134 filler instructions.

So we conclude that SKX has 134 physical registers available to hold speculative mask register values. As Henry indicates on the original post, it is likely that there are 8 physical registers dedicated to holding the non-speculative architectural state of the 8 mask registers, so our best guess at the total size of the mask register PRF is 142. That’s somewhat smaller than the GP PRF (180 entires) or the SIMD PRF (168 entries), but still quite large (see this table of out of order resource sizes for sizes on other platforms).

In particular, it is definitely large enough that you aren’t likely to run into this limit in practical code: it’s hard to imagine non-contrived code where almost 60%9 of the instructions write10 to mask registers, because that’s what you’d need to hit this limit.

Are They Distinct PRFs?

You may have noticed that so far I’m simply assuming that the mask register PRF is distinct from the others. I think this is highly likely, given the way mask registers are used and since they are part of a disjoint renaming domain11. It is also supported by the fact that that apparent mask register PFR size doesn’t match either the GP or SIMD PRF sizes, but we can go further and actually test it!

To do that, we use a similar test to the above, but with the filler instructions alternating between the same kaddd instruction as the original test and an instruction that uses either a GP or SIMD register. If the register file is shared, we expect to hit a limit at size of the PRF. If the PRFs are not shared, we expect that neither PRF limit will be hit, and we will instead hit a different limit such as the ROB size.

Test 29 alternates kaddd and scalar add instructions, like this:

mov    rcx,QWORD PTR [rcx]
add    ebx,ebx
kaddd  k1,k2,k3
add    esi,esi
kaddd  k1,k2,k3
add    ebx,ebx
kaddd  k1,k2,k3
add    esi,esi
kaddd  k1,k2,k3
add    ebx,ebx
kaddd  k1,k2,k3
add    esi,esi
kaddd  k1,k2,k3
add    ebx,ebx
kaddd  k1,k2,k3
mov    rdx,QWORD PTR [rdx]
lfence

Here’s the chart:

Test 29: alternating kaddd and scalar add

We see that the spike is at a filler count larger than the GP and PRF sizes. So we can conclude that the mask and GP PRFs are not shared.

Maybe the mask register is shared with the SIMD PRF? After all, mask registers are more closely associated with SIMD instructions than general purpose ones, so maybe there is some synergy there.

To check, here’s Test 35, which is similar to 29 except that it alternates between kaddd and vxorps, like so:

mov    rcx,QWORD PTR [rcx]
vxorps ymm0,ymm0,ymm1
kaddd  k1,k2,k3
vxorps ymm2,ymm2,ymm3
kaddd  k1,k2,k3
vxorps ymm4,ymm4,ymm5
kaddd  k1,k2,k3
vxorps ymm6,ymm6,ymm7
kaddd  k1,k2,k3
vxorps ymm0,ymm0,ymm1
kaddd  k1,k2,k3
vxorps ymm2,ymm2,ymm3
kaddd  k1,k2,k3
vxorps ymm4,ymm4,ymm5
kaddd  k1,k2,k3
mov    rdx,QWORD PTR [rdx]
lfence

Here’s the corresponding chart:

Test 35: alternating kaddd and SIMD xor

The behavior is basically identical to the prior test, so we conclude that there is no direct sharing between the mask register and SIMD PRFs either.

This turned out not to be the end of the story. The mask registers are shared, just not with the general purpose or SSE/AVX register file. For all the details, see this follow up post.

An Unresolved Puzzle

Something we notice in both of the above tests, however, is that the spike seems to finish around 212 filler instructions. However, the ROB size for this microarchtiecture is 224. Is this just non ideal behavior as we saw earlier? Well we can test this by comparing against Test 4, which just uses nop instructions as the filler: these shouldn’t consume almost any resources beyond ROB entries. Here’s Test 4 (nop filer) versus Test 29 (alternating kaddd and scalar add):

Test 4 vs 29

The nop-using Test 4 nails the ROB size at exactly 224 (these charts are SVG so feel free to “View Image” and zoom in confirm). So it seems that we hit some other limit around 212 when we mix mask and GP registers, or when we mix mask and SIMD registers. In fact the same limit applies even between GP and SIMD registers, if we compare Test 4 and Test 21 (which mixes GP adds with SIMD vxorps):

Test 4 vs 21

Henry mentions a more extreme version of the same thing in the original blog entry, in the section also headed Unresolved Puzzle:

Sandy Bridge AVX or SSE interleaved with integer instructions seems to be limited to looking ahead ~147 instructions by something other than the ROB. Having tried other combinations (e.g., varying the ordering and proportion of AVX vs. integer instructions, inserting some NOPs into the mix), it seems as though both SSE/AVX and integer instructions consume registers from some form of shared pool, as the instruction window is always limited to around 147 regardless of how many of each type of instruction are used, as long as neither type exhausts its own PRF supply on its own.

Read the full section for all the details. The effect is similar here but smaller: we at least get 95% of the way to the ROB size, but still stop before it. It is possible the shared resource is related to register reclamation, e.g., the PRRT12 - a table which keeps track of which registers can be reclaimed when a given instruction retires.

Finally, we finish this party off with a few miscellaneous notes on mask registers, checking for parity with some features available to GP and SIMD registers.

Move Elimination

Both GP and SIMD registers are eligible for so-called move elimination. This means that a register to register move like mov eax, edx or vmovdqu ymm1, ymm2 can be eliminated at rename by “simply”13 pointing the destination register entry in the RAT to the same physical register as the source, without involving the ALU.

Let’s check if something like kmov k1, k2 also qualifies for move elimination. First, we check the chart for Test 28, where the filler instruction is kmovd k1, k2:

Test 28

It looks exactly like Test 27 we saw earlier with kaddd. So we would suspect that physical registers are being consumed, unless we have happened to hit a different move-elimination related limit with exactly the same size and limiting behavior14.

Additional confirmation comes from uops.info which shows that all variants of mask to mask register kmov take one uop dispatched to p0. If the move is eliminated, we wouldn’t see any dispatched uops.

Therefore I conclude that register to register15 moves involving mask registers are not eliminated.

Dependency Breaking Idioms

The best way to set a GP register to zero in x86 is via the xor zeroing idiom: xor reg, reg. This works because any value xored with itself is zero. This is smaller (fewer instruction bytes) than the more obvious mov eax, 0, and also faster since the processor recognizes it as a zeroing idiom and performs the necessary work at rename16, so no ALU is involved and no uop is dispatched.

Furthermore, the idiom is dependency breaking: although xor reg1, reg2 in general depends on the value of both reg1 and reg2, in the special case that reg1 and reg2 are the same, there is no dependency as the result is zero regardless of the inputs. All modern x86 CPUs recognize this17 special case for xor. The same applies to SIMD versions of xor such as integer vpxor and floating point vxorps and vxorpd.

That background out of the way, a curious person might wonder if the kxor variants are treated the same way. Is kxorb k1, k1, k118 treated as a zeroing idiom?

This is actually two separate questions, since there are two aspects to zeroing idioms:

  • Zero latency execution with no execution unit (elimination)
  • Dependency breaking

Let’s look at each in turn.

Execution Elimination

So are zeroing xors like kxorb k1, k1, k1 executed at rename without latency and without needing an execution unit?

No.

Here, I don’t even have to do any work: uops.info has our back because they’ve performed this exact test and report a latency of 1 cycle and one p0 uop used. So we can conclude that zeroing xors of mask registers are not eliminated.

Dependency Breaking

Well maybe zeroing kxors are dependency breaking, even though they require an execution unit?

In this case, we can’t simply check uops.info. kxor is a one cycle latency instruction that runs only on a single execution port (p0), so we hit the interesting (?) case where a chain of kxor runs at the same speed regardless of whether the are dependent or independent: the throughput bottleneck of 1/cycle is the same as the latency bottleneck of 1/cycle!

Don’t worry, we’ve got other tricks up our sleeve. We can test this by constructing a tests which involve a kxor in a carried dependency chain with enough total latency so that the chain latency is the bottleneck. If the kxor carries a dependency, the runtime will be equal to the sum of the latencies in the chain. If the instruction is dependency breaking, the chain is broken and the different disconnected chains can overlap and performance will likely be limited by some throughput restriction (e.g., port contention). This could use a good diagram, but I’m not good at diagrams.

All the tests are in uarch bench, but I’ll show the key parts here.

First we get a baseline measurement for the latency of moving from a mask register to a GP register and back:

kmovb k0, eax
kmovb eax, k0
; repeated 127 more times

This pair clocks in19 at 4 cycles. It’s hard to know how to partition the latency between the two instructions: are they both 2 cycles or is there a 3-1 split one way or the other20, but for our purposes it doesn’t matter because we just care about the latency of the round-trip. Importantly, the post-based throughput limit of this sequence is 1/cycle, 4x faster than the latency limit, because each instruction goes to a different port (p5 and p0, respectively). This means we will be able to tease out latency effects independent of throughput.

Next, we throw a kxor into the chain that we know is not zeroing:

kmovb k0, eax
kxorb k0, k0, k1
kmovb eax, k0
; repeated 127 more times

Since we know kxorb has 1 cycle of latency, we expect to increase the latency to 5 cycles and that’s exactly what we measure (the first two tests shown):

** Running group avx512 : AVX512 stuff **
                               Benchmark    Cycles     Nanos
                kreg-GP rountrip latency      4.00      1.25
    kreg-GP roundtrip + nonzeroing kxorb      5.00      1.57

Finally, the key test:

kmovb k0, eax
kxorb k0, k0, k0
kmovb eax, k0
; repeated 127 more times

This has a zeroing kxorb k0, k0, k0. If it breaks the dependency on k0, it would mean that the kmovb eax, k0 no longer depends on the earlier kmovb k0, eax, and the carried chain is broken and we’d see a lower cycle time.

Drumroll…

We measure this at the exact same 5.0 cycles as the prior example:

** Running group avx512 : AVX512 stuff **
                               Benchmark    Cycles     Nanos
                kreg-GP rountrip latency      4.00      1.25
    kreg-GP roundtrip + nonzeroing kxorb      5.00      1.57
       kreg-GP roundtrip + zeroing kxorb      5.00      1.57

So we tentatively conclude that zeroing idioms aren’t recognized at all when they involve mask registers.

Finally, as a check on our logic, we use the following test which replaces the kxor with a kmov which we know is always dependency breaking:

kmovb k0, eax
kmovb k0, ecx
kmovb eax, k0
; repeated 127 more times

This is the final result shown in the output above, and it runs much more quickly at 2 cycles, bottlenecked on p5 (the two kmov k, r32 instructions both go only to p5):

** Running group avx512 : AVX512 stuff **
                               Benchmark    Cycles     Nanos
                kreg-GP rountrip latency      4.00      1.25
    kreg-GP roundtrip + nonzeroing kxorb      5.00      1.57
       kreg-GP roundtrip + zeroing kxorb      5.00      1.57
         kreg-GP roundtrip + mov from GP      2.00      0.63

So our experiment seems to check out.

Reproduction

You can reproduce these results yourself with the robsize binary on Linux or Windows (using WSL). The specific results for this article are also available as are the scripts used to collect them and generate the plots.

Summary

  • SKX has a separate PRF for mask registers with a speculative size of 134 and an estimated total size of 142
  • This is large enough compared to the other PRF size and the ROB to make it unlikely to be a bottleneck
  • Mask registers are not eligible for move elimination
  • Zeroing idioms21 in mask registers are not recognized for execution elimination or dependency breaking

Part II

I didn’t expect it to happen, but it did: there is a follow up post about mask registers, where we (roughly) confirm the register file size by looking at an image of a SKX CPU captured via microcope, and make an interesting discovery regarding sharing.

Comments

Discussion on Hacker News, Reddit (r/asm and r/programming) or Twitter.

Direct feedback also welcomed by email or as a GitHub issue.

Thanks

Daniel Lemire who provided access to the AVX-512 system I used for testing.

Henry Wong who wrote the original article which introduced me to this technique and graciously shared the code for his tool, which I now host on github.

Jeff Baker, Wojciech Muła for reporting typos.

Image credit: Kellogg’s Special K by Like_the_Grand_Canyon is licensed under CC BY 2.0.

If you liked this post, check out the homepage for others you might enjoy.




  1. These mask registers are often called k registers or simply kregs based on their naming scheme. Rumor has it that this letter was chosen randomly only after a long and bloody naming battle between MFs. 

  2. There is sometimes a misconception (until recently even on the AVX-512 wikipedia article) that k0 is not a normal mask register, but just a hardcoded indicator that no masking should be used. That’s not true: k0 is a valid mask register and you can read and write to it with the k-prefixed instructions and SIMD instructions that write mask registers (e.g., any AVX-512 comparison. However, the encoding that would normally be used for k0 as a writemask register in a SIMD operation indicates instead “no masking”, so the contents of k0 cannot be used for that purpose. 

  3. The distinction being that a zero-masking operation results in zeroed destination elements at positions not selected by the mask, while merging leaves the existing elements in the destination register unchanged at those positions. As as side-effect this means that with merging, the destination register becomes a type of destructive source-destination register and there is an input dependency on this register. 

  4. I’ll try to use the full term mask register here, but I may also use kreg a common nickname based on the labels k0, k1, etc. So just mentally swap kreg for mask register if and when you see it (or vice-versa). 

  5. H. Wong, Measuring Reorder Buffer Capacity, May, 2013. [Online]. Available: http://blog.stuffedcow.net/2013/05/measuring-rob-capacity/ 

  6. Generally taking 100 to 300 cycles each (latency-wise). The wide range is because the cache miss wall clock time varies by a factor of about 2x, generally between 50 and 100 naneseconds, depending on platform and uarch details, and the CPU frequency varies by a factor of about 2.5x (say from 2 GHz to 5 GHz). However, on a given host, with equivalent TLB miss/hit behavior, we expect the time to be roughly constant. 

  7. The reason I have to add roughly as a weasel word here is itself interesting. A glance at the charts shows that they are certainly not totally flat in either the fast or slow regions surrounding the spike. Rather there are various noticeable regions with distinct behavior and other artifacts: e.g., in Test 29 a very flat region up to about 104 filler instructions, followed by a bump and then a linearly ramping region up to the spike somewhat after 200 instructions. Some of those features are explicable by mentally (or actually) simulating the pipeline, which reveals that at some point the filler instructions will contribute (although only a cycle or so) to the runtime, but some features are still unexplained (for now). 

  8. For example, a given rename slot may only be able to write a subset of all the RAT entries, and uses the first available. When the RAT is almost full, it is possible that none of the allowed entries are empty, so it is as if the structure is full even though some free entries remain, but accessible only to other uops. Since the allowed entries may be essentially random across iterations, this ends up with a more-or-less linear ramp between the low and high performance levels in the non-ideal region. 

  9. The “60 percent” comes from 134 / 224, i.e., the speculative mask register PRF size, divided by the ROB size. The idea is that if you’ll hit the ROB size limit no matter what once you have 224 instructions in flight, so you’d need to have 60% of those instructions be mask register writes10 in order to hit the 134 limit first. Of course, you might also hit some other limit first, so even 60% might not be enough, but the ROB size puts a lower bound on this figure since it always applies. 

  10. Importantly, only instructions which write a mask register consume a physical register. Instructions that simply read a mask register (e.g,. SIMD instructions using a writemask) do not consume a new physical mask register.  2

  11. More renaming domains makes things easier on the renamer for a given number of input registers. That is, it is easier to rename 2 GP and 2 SIMD input registers (separate domains) than 4 GP registers. 

  12. This is either the Physical Register Reclaim Table or Post Retirement Reclaim Table depending on who you ask. 

  13. Of course, it is not actually so simple. For one, you now need to track these “move elimination sets” (sets of registers all pointing to the same physical register) in order to know when the physical register can be released (once the set is empty), and these sets are themselves a limited resource which must be tracked. Flags introduce another complication since flags are apparently stored along with the destination register, so the presence and liveness of the flags must be tracked as well. 

  14. In particular, in the corresponding test for GP registers (Test 7), the chart looks very different as move elimination reduce the PRF demand down to almost zero and we get to the ROB limit. 

  15. Note that I am not restricting my statement to moves between two mask registers only, but any registers. That is, moves between a GP registers and a mask registers are also not eliminated (the latter fact is obvious if consider than they use distinct register files, so move elimination seems impossible). 

  16. Probably by pointing the entry in the RAT to a fixed, shared zero register, or setting a flag in the RAT that indicates it is zero. 

  17. Although xor is the most reliable, other idioms may be recognized as zeroing or dependency breaking idioms by some CPUs as well, e.g., sub reg,reg and even sbb reg, reg which is not a zeroing idiom, but rather sets the value of reg to zero or -1 (all bits set) depending on the value of the carry flag. This doesn’t depend on the value of reg but only the carry flag, and some CPUs recognize that and break the dependency. Agner’s microarchitecture guide covers the uarch-dependent support for these idioms very well. 

  18. Note that only the two source registers really need to be the same: if kxorb k1, k1, k1 is treated as zeroing, I would expect the same for kxorb k1, k2, k2

  19. Run all the tests in this section using ./uarch-bench.sh --test-name=avx512/*

  20. This is why uops.info reports the latency for both kmov r32, k and kmov k, 32 as <= 3. They know the pair takes 4 cycles in total and under the assumption that each instruction must take at least one cycle the only thing you can really say is that each instruction takes at most 3 cycles. 

  21. Technically, I only tested the xor zeroing idiom, but since that’s the groud-zero, most basic idiom we can pretty sure nothing else will be recognized as zeroing. I’m open to being proven wrong: the code is public and easy to modify to test whatever idiom you want. 


Digging Into etcd Posts on elder.dev

Posts on elder.dev2019-12-01 00:00:00 What Is etcd? etcd per the official site is: A distributed, reliable key-value store for the most critical data of a distributed system I presume etcd is a play on /etc and the long history of nameing daemons with a d suffix, a daemon for your /etc config, though I’ve not yet found proof of this. Kubernetes uses etcd as the backing store for cluster data, which drove my own interest in collecting the information in this post.

Self-Driving Debian Posts on elder.dev

Posts on elder.dev2019-12-01 00:00:00 For my home server I’ve come to appreciate using it rather than maintaining it 😏 After replacing some parts starting over I really wanted it to be fully “self-driving” to the extent possible – primarily meaning totally unattended and automatic updates. No manual maintenance. Automated Updates Debian 10 “Buster” 🐶 ships with the unattended-upgrades package installed out of the box, but it needs a little configuring to achieve what we want.

A Tiny, Static, Full-Text Search Engine using Rust and WebAssembly Matthias Endler

Matthias Endler2019-10-17 00:00:00

I wrote a basic search module that you can add to a static website. It's very lightweight (50kB-100kB gzipped) and works with Hugo, Zola, and Jekyll. Only searching for entire words is supported. Try the search box on the left for a demo. The code is on Github.

Static site generators are magical. They combine the best of both worlds: dynamic content without sacrificing performance.

Over the years, this blog has been running on Jekyll, Cobalt, and, lately, Zola.

One thing I always disliked, however, was the fact that static websites don't come with "static" search engines, too. Instead, people resort to custom Google searches, external search engines like Algolia, or pure JavaScript-based solutions like lunr.js or elasticlunr.

All of these work fine for most sites, but it never felt like the final answer.

I didn't want to add yet another dependency on Google; neither did I want to use a stand-alone web-backend like Algolia, which adds latency and is proprietary.

On the other side, I'm not a huge fan of JavaScript-heavy websites. For example, just the search indices that lunr creates can be multiple megabytes in size. That feels lavish - even by today's bandwidth standards. On top of that, parsing JavaScript is still time-consuming.

I wanted some simple, lean, and self-contained search, that could be deployed next to my other static content.

As a consequence, I refrained from adding search functionality to my blog at all. That's unfortunate because, with a growing number of articles, it gets harder and harder to find relevant content.

The Idea

Many years ago, in 2013, I read "Writing a full-text search engine using Bloom filters" — and it was a revelation.

The idea was simple: Let's run all my blog articles through a generator that creates a tiny, self-contained search index using this magical data structure called a ✨Bloom Filter ✨.

Wait, what's a Bloom Filter?

A Bloom filter is a space-efficient way to check if an element is in a set.

The trick is that it doesn't store the elements themselves; it just knows with some confidence that they were stored before. In our case, it can say with a certain error rate that a word is in an article.

A Bloom filter stores a
'fingerprint' (a number of hash values) of all input values instead of the raw
input. The result is a low-memory-footprint data structure. This is an example
of 'hello' as an input.
A Bloom filter stores a 'fingerprint' (a number of hash values) of all input values instead of the raw input. The result is a low-memory-footprint data structure. This is an example of 'hello' as an input.

Here's the Python code from the original article that generates the Bloom filters for each post (courtesy of Stavros Korokithakis):

filters = {}
for name, words in split_posts.items():
  filters[name] = BloomFilter(capacity=len(words), error_rate=0.1)
  for word in words:
    filters[name].add(word)

The memory footprint is extremely small, thanks to error_rate, which allows for a negligible number of false positives.

I immediately knew that I wanted something like this for my homepage. My idea was to directly ship the Bloom filters and the search engine to the browser. I could finally have a small, static search without the need for a backend!

Headaches

Disillusionment came quickly.

I had no idea how to bundle and minimize the generated Bloom filters, let alone run them on clients. The original article briefly touches on this:

You need to implement a Bloom filter algorithm on the client-side. This will probably not be much longer than the inverted index search algorithm, but it’s still probably a bit more complicated.

I didn't feel confident enough in my JavaScript skills to pull this off. Back in 2013, NPM was a mere three years old, and WebPack just turned one, so I also didn't know where to look for existing solutions.

Unsure what to do next, my idea remained a pipe dream.

A New Hope

Five years later, in 2018, the web had become a different place. Bundlers were ubiquitous, and the Node ecosystem was flourishing. One thing, in particular, revived my dreams about the tiny static search engine: WebAssembly.

WebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications. [source]

This meant that I could use a language that I was familiar with to write the client-side code — Rust! 🎉

My journey started with a prototype back in January 2018. It was just a direct port of the Python version from above:

let mut filters = HashMap::new();
for (name, words) in articles {
  let mut filter = BloomFilter::with_rate(0.1, words.len() as u32);
  for word in words {
    filter.insert(&word);
  }
  filters.insert(name, filter);
}

While I managed to create the Bloom filters for every article, I still had no clue how to package it for the web... until wasm-pack came along in February 2018.

Whoops! I Shipped Some Rust Code To Your Browser.

Now I had all the pieces of the puzzle:

  • Rust — A language I was comfortable with
  • wasm-pack — A bundler for WebAssembly modules
  • A working prototype that served as a proof-of-concept

The search box you see on the left side of this page is the outcome. It fully runs on Rust using WebAssembly (a.k.a the RAW stack). Try it now if you like.

There were quite a few obstacles along the way.

Bloom Filter Crates

I looked into a few Rust libraries (crates) that implement Bloom filters.

First, I tried jedisct1's rust-bloom-filter, but the types didn't implement Serialize/Deserialize. This meant that I could not store my generated Bloom filters inside the binary and load them on the client-side.

After trying a few others, I found the cuckoofilter crate, which supported serialization. The behavior is similar to Bloom filters, but if you're interested in the differences, you can look at this summary.

Here's how to use it:

let mut cf = cuckoofilter::new();

// Add data to the filter
let value: &str = "hello world";
let success = cf.add(value)?;

// Lookup if data was added before
let success = cf.contains(value);
// success ==> true

Let's check the output size when bundling the filters for ten articles on my blog using cuckoo filters:

~/C/p/tinysearch ❯❯❯ l storage
Permissions Size User    Date Modified Name
.rw-r--r--   44k mendler 24 Mar 15:42  storage

44kB doesn't sound too shabby, but these are just the cuckoo filters for ten articles, serialized as a Rust binary. On top of that, we have to add the search functionality and the helper code. In total, the client-side code weighed in at 216kB using vanilla wasm-pack. Too much.

Trimming Binary Size

After the sobering first result of 216kB for our initial prototype, we have a few options to bring the binary size down.

The first is following johnthagen's advice on minimizing Rust binary size.

By setting a few options in our Cargo.toml, we can shave off quite a few bytes:

"opt-level = 'z'" => 249665 bytes
"lto = true"      => 202516 bytes
"opt-level = 's'" => 195950 bytes

Setting opt-level to s means we trade size for speed, but we're preliminarily interested in minimal size anyway. After all, a small download size also improves performance.

Next, we can try wee_alloc, an alternative Rust allocator producing a small .wasm code size.

It is geared towards code that makes a handful of initial dynamically sized allocations, and then performs its heavy lifting without any further allocations. This scenario requires some allocator to exist, but we are more than happy to trade allocation performance for small code size.

Exactly what we want. Let's try!

"wee_alloc and nightly" => 187560 bytes

We shaved off another 4% from our binary.

Out of curiosity, I tried to set codegen-units to 1, meaning we only use a single thread for code generation. Surprisingly, this resulted in a slightly smaller binary size.

"codegen-units = 1" => 183294 bytes

Then I got word of a Wasm optimizer called binaryen. On macOS, it's available through homebrew:

brew install binaryen

It ships a binary called wasm-opt and that shaved off another 15%:

"wasm-opt -Oz" => 154413 bytes

Then I removed web-sys as we don't have to bind to the DOM: 152858 bytes.

There's a tool called twiggy to profile the code size of Wasm binaries. It printed the following output:

twiggy top -n 20 pkg/tinysearch_bg.wasm
 Shallow Bytes │ Shallow % │ Item
─────────────┼───────────┼────────────────────────────────
         79256 ┊    44.37% ┊ data[0]
         13886 ┊     7.77% ┊ "function names" subsection
          7289 ┊     4.08% ┊ data[1]
          6888 ┊     3.86% ┊ core::fmt::float::float_to_decimal_common_shortest::hdd201d50dffd0509
          6080 ┊     3.40% ┊ core::fmt::float::float_to_decimal_common_exact::hcb5f56a54ebe7361
          5972 ┊     3.34% ┊ std::sync::once::Once::call_once::{{closure}}::ha520deb2caa7e231
          5869 ┊     3.29% ┊ search

From what I can tell, the biggest chunk of our binary is occupied by the raw data section for our articles. Next up, we got the function headers and some float to decimal helper functions, that most likely come from deserialization.

Finally, I tried wasm-snip, which replaces a WebAssembly function's body with an unreachable like so, but it didn't reduce code size:

wasm-snip --snip-rust-fmt-code --snip-rust-panicking-code -o pkg/tinysearch_bg_snip.wasm pkg/tinysearch_bg_opt.wasm

After tweaking with the parameters of the cuckoo filters a bit and removing stop words from the articles, I arrived at 121kB (51kB gzipped) — not bad considering the average image size on the web is around 900kB. On top of that, the search functionality only gets loaded when a user clicks into the search field.

Update

Recently I moved the project from cuckoofilters to XOR filters. I used the awesome xorf project, which comes with built-in serde serialization. which allowed me to remove a lot of custom code.

With that, I could reduce the payload size by another 20-25% percent. I'm down to 99kB (49kB gzipped) on my blog now. 🎉

The new version is released on crates.io already, if you want to give it a try.

Frontend- and Glue Code

wasm-pack will auto-generate the JavaScript code to talk to Wasm.

For the search UI, I customized a few JavaScript and CSS bits from w3schools. It even has keyboard support! Now when a user enters a search query, we go through the cuckoo filter of each article and try to match the words. The results are scored by the number of hits. Thanks to my dear colleague Jorge Luis Betancourt for adding that part.

Video of the search functionality

(Fun fact: this animation is about the same size as the uncompressed Wasm search itself.)

Caveats

Only whole words are matched. I would love to add prefix-search, but the binary became too big when I tried.

Usage

The standalone binary to create the Wasm file is called tinysearch. It expects a single path to a JSON file as an input:

tinysearch path/to/corpus.json

This corpus.json contains the text you would like to index. The format is pretty straightforward:

[
  {
    "title": "Article 1",
    "url": "https://example.com/article1",
    "body": "This is the body of article 1."
  },
  {
    "title": "Article 2",
    "url": "https://example.com/article2",
    "body": "This is the body of article 2."
  }
]

You can generate this JSON file with any static site generator. Here's my version for Zola:

{% set section = get_section(path="_index.md") %}

[
  {%- for post in section.pages -%}
    {% if not post.draft %}
      {
        "title": {{ post.title | striptags | json_encode | safe }},
        "url": {{ post.permalink | json_encode | safe }},
        "body": {{ post.content | striptags | json_encode | safe }}
      }
      {% if not loop.last %},{% endif %}
    {% endif %}
  {%- endfor -%}
]

I'm pretty sure that the Jekyll version looks quite similar. Here's a starting point. If you get something working for your static site generator, please let me know.

Observations

  • This is still the wild west: unstable features, nightly Rust, documentation gets outdated almost every day.
    Bring your thinking cap!
  • Creating a product out of a good idea is a lot of work. One has to pay attention to many factors: ease-of-use, generality, maintainability, documentation, and so on.
  • Rust is very good at removing dead code, so you usually don't pay for what you don't use. I would still advise you to be very conservative about the dependencies you add to a Wasm binary because it's tempting to add features that you don't need and which will add to the binary size. For example, I used StructOpt during testing, and I had a main() function that was parsing these command-line arguments. This was not necessary for Wasm, so I removed it later.
  • I understand that not everyone wants to write Rust code. It's complicated to get started with, but the cool thing is that you can use almost any other language, too. For example, you can write Go code and transpile to Wasm, or maybe you prefer PHP or Haskell. There is support for many languages already.
  • A lot of people dismiss WebAssembly as a toy technology. They couldn't be further from the truth. In my opinion, WebAssembly will revolutionize the way we build products for the web and beyond. What was very hard just two years ago is now easy: shipping code in any language to every browser. I'm super excited about its future.
  • If you're looking for a standalone, self-hosted search index for your company website, check out sonic. Also check out stork as an alternative.

WOW! This tool getting quite a bit of traction lately.✨‍

I don't run ads on this website, but if you like these kind of experiments, please consider sponsoring me on Github. This allows me to write more tools like this in the future.

Also, if you're interested in hands-on Rust consulting, pick a date from my calendar and we can talk about how I can help .

Try it!

The code for tinysearch is on Github.

Please be aware of these limitations:

  • Only searches for entire words. There are no search suggestions. The reason is that prefix search blows up binary size like Mentos and Diet Coke.
  • Since we bundle all search indices for all articles into one static binary, I only recommend to use it for low- to medium-sized websites. Expect around 4kB (non-compressed) per article.
  • The compile times are abysmal at the moment (around 1.5 minutes after a fresh install on my machine), mainly because we're compiling the Rust crate from scratch every time we rebuild the index.
    Update: This is mostly fixed thanks to the awesome work of CephalonRho in PR #13. Thanks again!

The final Wasm code is laser-fast because we save the roundtrips to a search-server. The instant feedback loop feels more like filtering a list than searching through posts. It can even work fully offline, which might be nice if you like to bundle it with an app.


California is Beautiful Posts on elder.dev

Posts on elder.dev2019-08-14 00:00:00 Just a few select photos from a short trip away from it all …

Writing Safer Bash Posts on elder.dev

Posts on elder.dev2019-04-08 00:00:00 Bash scripts are a really convenient way to write simple utilities. Unfortunately many bash scripts in the wild are littered with bugs. Writing reliable bash can be hard. I’ve been reviewing and fixing a lot of bash while working on cleaning up the Kubernetes project’s scripts and wanted to collect some tips for writing more reliable scripts. Use ShellCheck ShellCheck is an excellent open source linter for shell capable of detecting many errors.

Maybe You Don't Need Kubernetes Matthias Endler

Matthias Endler2019-03-21 00:00:00
A woman riding a scooter
A woman riding a scooter
Source: Illustration created by freepik, Nomad logo by HashiCorp.

Kubernetes is the 800-pound gorilla of container orchestration.
It powers some of the biggest deployments worldwide, but it comes with a price tag.

Especially for smaller teams, it can be time-consuming to maintain and has a steep learning curve. For what our team of four wanted to achieve at trivago, it added too much overhead. So we looked into alternatives — and fell in love with Nomad.

The Wishlist

Our team runs a number of typical services for monitoring and performance analysis: API endpoints for metrics written in Go, Prometheus exporters, log parsers like Logstash or Gollum, and databases like InfluxDB or Elasticsearch. Each of these services run in their own container. We needed a simple system to keep those jobs running.

We started with a list of requirements for container orchestration:

  • Run a fleet of services across many machines.
  • Provide an overview of running services.
  • Allow for communication between services.
  • Restart them automatically when they die.
  • Be manageable by a small team.

On top of that, the following things were nice to have but not strictly required:

  • Tag machines by their capabilities (e.g., label machines with fast disks for I/O heavy services.)
  • Be able to run these services independently of any orchestrator (e.g. in development).
  • Have a common place to share configurations and secrets.
  • Provide an endpoint for metrics and logging.

Why Kubernetes Was Not A Good Fit For Us

When creating a prototype with Kubernetes, we noticed that we started adding ever-more complex layers of logic to operate our services. Logic on which we implicitly relied on.

As an example, Kubernetes allows embedding service configurations using ConfigMaps. Especially when merging multiple config files or adding more services to a pod, this can get quite confusing quickly. Kubernetes - or helm, for that matter - allows injecting external configs dynamically to ensure separation of concerns. But this can lead to tight, implicit coupling between your project and Kubernetes. Helm and ConfigMaps are optional features so you don’t have to use them. You might as well just copy the config into the Docker image. However, it’s tempting to go down that path and build unnecessary abstractions that can later bite you.

On top of that, the Kubernetes ecosystem is still rapidly evolving. It takes a fair amount of time and energy to stay up-to-date with the best practices and latest tooling. Kubectl, minikube, kubeadm, helm, tiller, kops, oc - the list goes on and on. Not all tools are necessary to get started with Kubernetes, but it’s hard to know which ones are, so you have to be at least aware of them. Because of that, the learning curve is quite steep.

When To Use Kubernetes

At trivago specifically, many teams use Kubernetes and are quite happy with it. These instances are managed by Google or Amazon however, which have the capacity to do so.

Kubernetes comes with amazing features, that make container orchestration at scale more manageable:

  • Fine-grained rights management
  • Custom controllers allow getting logic into the cluster. These are just programs that talk to the Kubernetes API.
  • Autoscaling! Kubernetes can scale your services up and down on demand. It uses service metrics to do this without manual intervention.

The question is if you really need all those features. You can't rely on these abstractions to just work; you'll have to learn what's going on under the hood.

Especially in our team, which runs most services on-premise (because of its close connection to trivago's core infrastructure), we didn't want to afford running our own Kubernetes cluster; we wanted to ship services instead.

Nuclear hot take: nobody will care about Kubernetes in five years. -A tweet by Corey Quinn

Batteries Not Included

Nomad is the 20% of service orchestration that gets you 80% of the way. All it does is manage deployments. It takes care of your rollouts and restarts your containers in case of errors, and that's about it.

The entire point of Nomad is that it does less: it doesn’t include fine-grained rights management or advanced network policies, and that’s by design. Those components are provided as enterprise services, by a third-party — or not at all.

I think Nomad hit a sweet-spot between ease of use and expressiveness. It's good for small, mostly independent services. If you need more control, you'll have to build it yourself or use a different approach. Nomad is just an orchestrator.

The best part about Nomad is that it's easy to replace. There is little to no vendor lock-in because the functionality it provides can easily be integrated into any other system that manages services. It just runs as a plain old single binary on every machine in your cluster; that's it!

The Nomad Ecosystem Of Loosely Coupled Components

The real power of Nomad lies within its ecosystem. It integrates very well with other - completely optional - products like Consul (a key-value store) or Vault (for secrets handling). Inside your Nomad file, you can have sections for fetching data from those services:

template {
  data = <<EOH
LOG_LEVEL="{{key "service/geo-api/log-verbosity"}}"
API_KEY="{{with secret "secret/geo-api-key"}}{{.Data.value}}{{end}}"
EOH

  destination = "secrets/file.env"
  env         = true
}

This will read the service/geo-api/log-verbosity key from Consul and expose it as a LOG_LEVEL environment variable inside your job. It's also exposing secret/geo-api-key from Vault as API_KEY. Simple, but powerful!

Because it's so simple, Nomad can also be easily extended with other services through its API. For example, jobs can be tagged for service discovery. At trivago, we tag all services, which expose metrics, with trv-metrics. This way, Prometheus finds the services via Consul and periodically scrapes the /metrics endpoint for new data. The same can be done for logs by integrating Loki for example.

There are many other examples for extensibility:

  • Trigger a Jenkins job using a webhook and Consul watches to redeploy your Nomad job on service config changes.
  • Use Ceph to add a distributed file system to Nomad.
  • Use fabio for load balancing.

All of this allowed us to grow our infrastructure organically without too much up-front commitment.

Fair Warning

No system is perfect. I advise you not to use any fancy new features in production right now. There are bugs and missing features of course - but that's also the case for Kubernetes.

Compared to Kubernetes, there is far less momentum behind Nomad. Kubernetes has seen around 75.000 commits and 2000 contributors so far, while Nomad sports about 14.000 commits and 300 contributors. It will be hard for Nomad to keep up with the velocity of Kubernetes, but maybe it doesn’t have to! The scope is much more narrow and the smaller community could also mean that it'll be easier to get your pull request accepted, in comparison to Kubernetes.

Summary

The takeaway is: don't use Kubernetes just because everyone else does. Carefully evaluate your requirements and check which tool fits the bill.

If you're planning to deploy a fleet of homogenous services on large-scale infrastructure, Kubernetes might be the way to go. Just be aware of the additional complexity and operational costs. Some of these costs can be avoided by using a managed Kubernetes environment like Google Kubernetes Engine or Amazon EKS.

If you're just looking for a reliable orchestrator that is easy to maintain and extendable, why not give Nomad a try? You might be surprised by how far it'll get you.

If Kubernetes were a car, Nomad would be a scooter. Sometimes you prefer one and sometimes the other. Both have their right to exist.


Avoiding Burnout in Open Source Posts on elder.dev

Posts on elder.dev2019-03-17 00:00:00 This post may come off a bit ironic, coming from someone who burned out pretty hard recently, but I received some really good advice and I hope it can help someone else. Some of the advice I received: Set boundaries, reserve time for yourself Don’t feel guilty for not responding right away. Even if you work on Open Source fulltime, don’t let it become a “second job”, take time for yourself.

What Is Rust Doing Behind the Curtains? Matthias Endler

Matthias Endler2018-12-02 00:00:00

Rust allows for a lot of syntactic sugar, that makes it a pleasure to write. It is sometimes hard, however, to look behind the curtain and see what the compiler is really doing with our code.


The Unreasonable Effectiveness of Excel Macros Matthias Endler

Matthias Endler2018-11-05 00:00:00

I never was a big fan of internships, partially because all the exciting companies were far away from my little village in Bavaria and partially because I was too shy to apply.

Only once I applied for an internship in Ireland as part of a school program. Our teacher assigned the jobs and so my friend got one at Apple and I ended up at a medium-sized IT distributor — let's call them PcGo.


Mapping Appalachia Posts on elder.dev

Posts on elder.dev2018-10-31 00:00:00 ✎ Update It’s worth noting that I did not wind up playing Fallout 76 much more. After the B.E.T.A. my interest fell off quickly as the locations and quests failed to be as engaging for me as previous Fallout games. I do not recommend Fallout 76 to anyone. October 30th B.E.T.A. (Break-It Early Test Application) During the first B.E.T.A. some places I discovered along my travels were:

GitOps All The Things! Posts on elder.dev

Posts on elder.dev2018-09-23 00:00:00 You should use GitOps for everything. Everything. GitOps is a recent-ish term for: use declarative configuration for your infrastructure (e.g. Kubernetes) version all of your configuration in source control (I.E. Git) use your source control to drive your infrastructure (I.E. use CI/CD = Ops) GitOps: versioned CI/CD on top of declarative infrastructure. Stop scripting and start shipping. https://t.co/SgUlHgNrnY — Kelsey Hightower (@kelseyhightower) January 17, 2018 Why? - Well, do you like the sound of:

Slackmoji Anywhere Posts on elder.dev

Posts on elder.dev2018-09-09 00:00:00 I use slack a lot to communicate with other Kubernetes contributors, and I’m a fan of the emoji reaction feature for reacting to posts without notifying everyone in the group. Positive emoji responses in particular are a simple way to acknowledge messages and make discussions more friendly and welcoming. slack emoji reaction example (thanks dims !) A particularly fun part of this feature is custom emoji support, commonly known as “slackmoji”, which allows adding arbitrary images (and even gifs!

Switching from a German to a US Keyboard Layout - Is It Worth It? Matthias Endler

Matthias Endler2018-09-02 00:00:00

For the first three decades of my life, I've exclusively used a German keyboard layout for programming. In 2018, I finally switched to a US layout. This post summarizes my thoughts around the topic. I was looking for a similar article before jumping the gun, but I couldn't find one — so I wrote it.

My current keyboard (as of April 2021), the low-profile, tenkeyless Keychron K1 is close to my favorite input device. Yes, I got the RGB version. &mdash; [Amazon referral link](https://amzn.to/3tRatjU).
My current keyboard (as of April 2021), the low-profile, tenkeyless Keychron K1 is close to my favorite input device. Yes, I got the RGB version. — Amazon referral link.

Why Switch To the US Layout?

I was reasonably efficient when writing prose, but felt like a lemur on a piano when programming: lots of finger-stretching while trying to reach the special keys like {, ;, or /.

German Keyboard Layout
German Keyboard Layout
Source: Image by Wikipedia

Here's Wikipedia's polite explanation why the German keyboard sucks for programming:

Like many other non-American keyboards, German keyboards change the right Alt key into an Alt Gr key to access a third level of key assignments. This is necessary because the umlauts and some other special characters leave no room to have all the special symbols of ASCII, needed by programmers among others, available on the first or second (shifted) levels without unduly increasing the size of the keyboard.

But Why Switch Now?

After many years of using a rubber-dome Logitech Cordless Desktop Wave, I had to get a mechanical keyboard again.

Those rubber domes just feel too mushy to me now. In addition to that, I enjoy the clicky sound of a mechanical keyboard and the noticeable tactile bump. (I'm using Cherry MX Brown Keys with O-Ring dampeners to contain the anger of my coworkers.)

Most mechanical keyboards come with an ANSI US layout only, so I figured, I'd finally make the switch.

My first mechanical keyboard &mdash; [Durgod Taurus K320](https://www.amazon.de/gp/product/B07QK16RDQ/ref=as_li_tl?ie=UTF8&tag=matthiasendle-21&camp=1638&creative=6742&linkCode=as2&creativeASIN=B07QK16RDQ&linkId=fb0a782ecbc713f8266b90b941375a5f) (referral link). They also have a fancy [white-pink](https://www.amazon.de/gp/product/B081LZV2QM?ie=UTF8&tag=matthiasendle-21&camp=1638&linkCode=xm2&creativeASIN=B081LZV2QM) ISO version now.
My first mechanical keyboard — Durgod Taurus K320 (referral link). They also have a fancy white-pink ISO version now.

How Long Did It Take To Get Accustomed To The New Layout?

Working as a Software Engineer, my biggest fear was, that the switch would slow down my daily work. This turned out not to be true. I was reasonably productive from day one, and nobody even noticed any difference. (That's a good thing, right?)

At first, I didn't like the bar-shaped US-Return key. I preferred the European layout with a vertical enter key. I was afraid that I would hit the key by accident. After a while, I find that the US return key to be even more convenient. I never hit it by accident, and it's easy to reach with my pinky from the home position.

Within two weeks, I was back to 100% typing speed.

Did My Programming Speed Improve Noticeably?

Yup. I'd say I can type programs about 30% faster now.

Especially when using special characters (/, ;, {, and so on) I'm much faster now; partly because the key locations feel more intuitive, but mainly because my fingers stay at their dedicated positions now.

Somehow the position of special characters feels just right. I can now understand the reason why Vim is using / for search or why the pipe symbol is |: both are easy to reach! It all makes sense now! (For a fun time, try that on a German keyboard!)

I now understand why Mircosoft chose \ as a directory separator: it's easily accessible from a US keyboard. On the German layout, it's… just… awful (Alt Gr+ß on Windows, Shift + Option + 7 on Mac).

The opening curly brace on a German layout Mac is produced with Alt+8, which always made me leave the home row and break my typing flow. Now there are dedicated keys for parentheses. Such a relief!

Update: It also helps greatly when looking up hotkeys for IDEs, text editors, photo editors, etc. because some programs remap shortcuts for the German market, which means that all the English documentation is totally worthless. Now I can just use the shortcuts mentioned and move on with my life.

Am I Slower When Writing German Texts Now?

In the beginning, I was.

Somehow my brain associated the German layout with German texts. First, I used the macOS layout switcher. This turned out to be cumbersome and take time.

Then I found the "US with Umlauts via Option Key Layout". It works perfectly fine for me. It allows me to use a single Keyboard layout but insert German umlauts at will (e.g. ö is Option+o). There is probably a similar layout for other language combinations.

Is Switching Between Keyboards Painful?

US keyboard layout
US keyboard layout
Source: Wikipedia

My built-in MacBook Pro keyboard layout is still German. I was afraid, that switching between the internal German and the external English keyboard would confuse me. This turned out not to be a problem. I rarely look at the print anyway. (Update: can't remember when I last looked at the print.)

How Often Do You Switch Back To A German Layout Now?

Never. My Girlfriend has a German keyboard and ever time I have to use it, I switch to the US layout. It makes her very happy when I do this and forget to switch back to German when I'm done.

Summary

If you consider switching, just do it! I don't look back at all and apart from the initial transition period, I still couldn't find any downsides.

Since posting this article, many of my friends made the switch as well and had similar experiences:


fastcat - A Faster `cat` Implementation Using Splice Matthias Endler

Matthias Endler2018-07-31 00:00:00

Lots of people asked me to write another piece about the internals of well-known Unix commands. Well, actually, nobody asked me, but it makes for a good intro. I'm sure you’ve read the previous parts about yes and ls — they are epic.

Anyway, today we talk about cat, which is used to concatenate files - or, more commonly, abused to print a file's contents to the screen.

# Concatenate files, the intended purpose
cat input1.txt input2.txt input3.txt > output.txt

# Print file to screen, the most common use case
cat myfile

Implementing cat

Here's a naive cat in Ruby:

#!/usr/bin/env ruby

def cat(args)
  args.each do |arg|
    IO.foreach(arg) do |line|
      puts line
    end
  end
end

cat(ARGV)

This program goes through each file and prints its contents line by line. Easy peasy! But wait, how fast is this tool?

I quickly created a random 2 GB file for the benchmark.

Let's compare the speed of our naive implementation with the system one using the awesome pv (Pipe Viewer) tool. All tests are averaged over five runs on a warm cache (file in memory).

# Ruby 2.5.1
> ./rubycat myfile | pv -r > /dev/null
[196MiB/s]

Not bad, I guess? How does it compare with my system's cat?

cat myfile | pv -r > /dev/null
[1.90GiB/s]

Uh oh, GNU cat is ten times faster than our little Ruby cat. 🐌

Making our Ruby cat a little faster

Our naive Ruby code can be tweaked a bit. Turns out line buffering hurts performance in the end1:

#!/usr/bin/env ruby

def cat(args)
  args.each do |arg|
    IO.copy_stream(arg, STDOUT)
  end
end

cat(ARGV)
rubycat myfile | pv -r > /dev/null
[1.81GiB/s]

Wow... we didn't really try hard, and we're already approaching the speed of a tool that gets optimized since 1971. 🎉

But before we celebrate too much, let's see if we can go even faster.

Splice

What initially motivated me to write about cat was this comment by user wahern on Hacker News:

I'm surprised that neither GNU yes nor GNU cat uses splice(2).

Could this splice thing make printing files even faster? — I was intrigued.

Splice was first introduced to the Linux Kernel in 2006, and there is a nice summary from Linus Torvalds himself, but I prefer the description from the manpage:

splice() moves data between two file descriptors without copying between kernel address space and user address space. It transfers up to len bytes of data from the file descriptor fd_in to the file descriptor fd_out, where one of the file descriptors must refer to a pipe.

If you really want to dig deeper, here's the corresponding source code from the Linux Kernel, but we don't need to know all the nitty-gritty details for now. Instead, we can just inspect the header from the C implementation:

#include <fcntl.h>

ssize_t splice (int fd_in, loff_t *off_in, int fd_out,
                loff_t *off_out, size_t len,
                unsigned int flags);

To break it down even more, here's how we would copy the entire src file to dst:

const ssize_t r = splice (src, NULL, dst, NULL, size, 0);

The cool thing about this is that all of it happens inside the Linux kernel, which means we won't copy a single byte to userspace (where our program runs). Ideally, splice works by remapping pages and does not actually copy any data, which may improve I/O performance (reference).

File icon by Aleksandr Vector from the Noun Project. Terminal icon by useiconic.com from the Noun Project.
Source: File icon by Aleksandr Vector from the Noun Project. Terminal icon by useiconic.com from the Noun Project.

Using splice from Rust

I have to say I'm not a C programmer and I prefer Rust because it offers a safer interface. Here's the same thing in Rust:

#[cfg(any(target_os = "linux", target_os = "android"))]
pub fn splice(
    fd_in: RawFd,
    off_in: Option<&mut libc::loff_t>,
    fd_out: RawFd,
    off_out: Option<&mut libc::loff_t>,
    len: usize,
    flags: SpliceFFlags,
) -> Result<usize>

Now I didn't implement the Linux bindings myself. Instead, I just used a library called nix, which provides Rust friendly bindings to *nix APIs.

There is one caveat, though: We cannot really copy the file directly to standard out, because splice requires one file descriptor to be a pipe. The way around that is to create a pipe, which consists of a reader and a writer (rd and wr). We pipe the file into the writer, and then we read from the pipe and push the data to stdout.

You can see that I use a relatively big buffer of 16384 bytes (214) to improve performance.

extern crate nix;

use std::env;
use std::fs::File;
use std::io;
use std::os::unix::io::AsRawFd;

use nix::fcntl::{splice, SpliceFFlags};
use nix::unistd::pipe;

const BUF_SIZE: usize = 16384;

fn main() {
    for path in env::args().skip(1) {
        let input = File::open(&path).expect(&format!("fcat: {}: No such file or directory", path));
        let (rd, wr) = pipe().unwrap();
        let stdout = io::stdout();
        let _handle = stdout.lock();

        loop {
            let res = splice(
                input.as_raw_fd(),
                None,
                wr,
                None,
                BUF_SIZE,
                SpliceFFlags::empty(),
            ).unwrap();

            if res == 0 {
                // We read 0 bytes from the input,
                // which means we're done copying.
                break;
            }

            let _res = splice(
                rd,
                None,
                stdout.as_raw_fd(),
                None,
                BUF_SIZE,
                SpliceFFlags::empty(),
            ).unwrap();
        }
    }
}

So, how fast is this?

fcat myfile | pv -r > /dev/null
[5.90GiB/s]

Holy guacamole. That's over three times as fast as system cat.

Operating System support

  • Linux and Android are fully supported.
  • OpenBSD also has some sort of splice implementation called sosplice. I haven't tested that, though.
  • On macOS, the closest thing to splice is its bigger brother, sendfile, which can send a file to a socket within the Kernel. Unfortunately, it does not support sending from file to file.2 There's also copyfile, which has a similar interface, but unfortunately, it is not zero-copy. (I thought so in the beginning, but I was wrong.)
  • Windows doesn't provide zero-copy file-to-file transfer (only file-to-socket transfer using the TransmitFile API).

Nevertheless, in a production-grade implementation, the splice support could be activated on systems that support it, while using a generic implementation as a fallback.

Nice, but why on earth would I want that?

I have no idea. Probably you don't, because your bottleneck is somewhere else. That said, many people use cat for piping data into another process like

# Count all lines in C files
cat *.c | wc -l

or

cat kittens.txt | grep "dog"

In this case, if you notice that cat is the bottleneck try fcat (but first, try to avoid cat altogether).

With some more work, fcat could also be used to directly route packets from one network card to another, similar to netcat.

Lessons learned

  • The closer we get to bare metal, the more our hard-won abstractions fall apart, and we are back to low-level systems programming.
  • Apart from a fast cat, there's also a use-case for a slow cat: old computers. For that purpose, there's... well.. slowcat.

That said, I still have no idea why GNU cat does not use splice on Linux. 🤔 The source code for fcat is on Github. Contributions welcome!

Footnotes

1. Thanks to reader Freeky for making this code more idiomatic.
2. Thanks to reader masklinn for the hint.


That Octocat on the Wall Matthias Endler

Matthias Endler2018-06-09 00:00:00
Photo of my office with Github's octocat on the wall over my couch
Photo of my office with Github's octocat on the wall over my couch

So I'm in a bit of a sentimental mood lately. Github got acquired by Microsoft. While I think the acquisition was well-deserved, I still wish it didn't happen. Let me explain.

My early days

I joined Github on 3rd of January 2010. Since I was a bit late to the game, my usual handle (mre) was already taken. So I naively sent a mail to Github, asking if I could bag the name as it seemed to be abandoned. To my surprise, I got an answer. The response came from a guy named Chris Wanstrath.

All he wrote was "it's yours."

That was the moment I fell in love with Github. I felt encouraged to collaborate on projects, that everybody could contribute something valuable. Only later I found out that Chris was one of the founders and the CEO of the company.

Living on Github

Before Github, there was SourceForge, and I only went there to download binaries. Github showed me, that there was an entire community of like-minded people out there, who ❤️ to work on code in their free-time. To me, Github is much more than a git interface; it's a social network. While other people browse Facebook or Instagram, I browse Github.

I can still vividly remember getting my first star and my first issue on one of my projects coming from a real (!) person other than myself.

After so many years, a pull-request still feels like the most personal gift anyone could give to me.

Github - the culture

After a while, I started to admire some Github employees deeply:

All three developers have since left the company. I can't help but notice that Github has changed. The harassment accusations and letting Zach Holman go are only part of the story.

It has become a company like any other, maintaining a mature product. It doesn't excite me anymore.

An alternative reality

There's still a bitter taste in my mouth when I think that Github has fallen prey to one of the tech giants. I loved Github while it was a small, friendly community of passionate developers. Could this have been sustainable?

Maybe through paid features for project maintainers.

You see, if you do Open Source every day, it can be a lot of work. People start depending on your projects, and you feel responsible for keeping the lights on.

To ease the burden, I'd love to have deeper insights into my project usage: visitor statistics for longer than two weeks, a front page where you could filter and search for events, a better way to handle discussions (which can get out of hand quickly), better CI integration à la Gitlab.

These features would be targeted at the top 10% of Github users, a group of 3 million people. Would this be enough to pay the bills? Probably. Would it be enough to grow? Probably not.

So what?

I don't think the acquisition will kill the culture. Microsoft is a strong partner and Nat Friedman is one of us. On the other side, I'm not as enthusiastic as I used to be. There's room for competitors now and I'm beginning to wonder what will be the next Github. That said, I will keep the Octocat on my office wall, in the hope that the excitement comes back.


Ten Years of Vim Matthias Endler

Matthias Endler2018-05-20 00:00:00

When I opened Vim by accident for the first time, I thought it was broken. My keystrokes changed the screen in unpredictable ways, and I wanted to undo things and quit. Needless to say, it was an unpleasant experience. There was something about it though, that kept me coming back and it became my main editor.

Fast forward ten years (!) and I still use Vim. After all the Textmates and Atoms and PhpStorms I tried, I still find myself at home in Vim. People keep asking me: Why is that?

Why Vim?

Before Vim, I had used many other editors like notepad or nano. They all behaved more or less as expected: you insert text, you move your cursor with the arrow keys or your mouse, and you save with Control + S or by using the menu bar. VI (and Vim, its spiritual successor) is different.

EVERYTHING in Vim is different, and that's why it's so highly effective. Let me explain.

The Zen of Vim

The philosophy behind Vim takes a while to sink in: While other editors focus on writing as the central part of working with text, Vim thinks it's editing.

You see, most of the time I don't spend writing new text; instead, I edit existing text.
I mold text, form it, turn it upside down. Writing text is craftsmanship and hard work. You have to shape your thoughts with your cold, bare hands until they somewhat form a coherent whole. This painful process is what Vim tries to make at least bearable. It helps you keep control. It does that, by providing you sharp, effective tools to modify text. The core of Vim is a language for editing text.

Vim, The Language

The Vim commands are not cryptic, you already know them.

  • To undo, type u.
  • To find the next t, type ft.
  • To delete a word, type daw.
  • To change a sentence, type cas.

More often than not, you can guess the correct command by thinking of an operation you want to execute and an object to execute it on. Then just take the first character of every word. Try it! If anything goes wrong, you can always hit ESC and type u for undo.

Operations: delete, find, change, back, insert, append,...
Objects: word, sentence, parentheses, (html) tag,... (see :help text-objects)

Inserting text is just another editing operation, which can be triggered with i. That's why, by default, you are in normal mode — also called command mode — where all those operations work.

Once you know this, Vim makes a lot more sense, and that's when you start to be productive.

How My Workflow Changed Over The Years

When I was a beginner, I was very interested in how people with more Vim experience would use the editor. Now that I'm a long-time user, here's my answer: there's no secret sauce. I certainly feel less exhausted after editing text for a day, but 90% of the commands I use fit on a post-it note.

That said, throughout the years, my Vim habits changed.
I went through several phases:

Year 1: I'm happy if I can insert text and quit again.
Year 2: That's cool, let's learn more shortcuts.
Year 3-5: Let's add all the features!!!
Year 6-10: My .vimrc is five lines long.

Year three is when I started to learn the Vim ecosystem for real. I tried all sorts of flavors like MacVim and distributions like janus. For a while, I even maintained my own Vim configuration , which was almost 400 lines long.

All of that certainly helped me learn what's out there, but I'm not sure if I would recommend that to a Vim beginner. After all, you don't really need all of that. Start with a vanilla Vim editor which works just fine!

My current Vim setup is pretty minimalistic. I don't use plugins anymore, mostly out of laziness and because built-in Vim commands or macros can replace them.

Here are three concrete examples of how my workflow changed over the years:

  1. In the beginning, I used a lot of "number powered movements". That is, if you have a command like b, which goes back one word in the text, you can also say 5b to go back five words. Nowadays I mostly use / to move to a matching word because it's quicker.

  2. I don't use arrow keys to move around in text anymore but forced myself to use h, j, k, l. Many people say that this is faster. After trying this for a few years, I don't think that is true (at least for me). I now just stick to it out of habit.

  3. On my main working machine I use Vim for quick text editing and Visual Studio Code plus the awesome Vim plugin for projects. This way, I get the best of both worlds.

Workflow Issues I Still Struggle With

After all these years I'm still not a Vim master — far from it. As every other Vim user will tell you, we're all still learning.

Here are a few things I wish I could do better:

  • Jumping around in longer texts: I know the basics, like searching (/), jumping to a matching bracket (%) or jumping to specific lines (for line 10, type 10G), but I still could use symbols more often for navigation.
  • Using visual mode for moving text around: Sometimes it can be quite complicated to type the right combination of letters to cut (delete) the text I want to move around. That's where visual mode (v) shines. It highlights the selected text. I should use it more often.
  • Multiple registers for copy and paste: Right now I only use one register (like a pastebin) for copying text, but Vim supports multiple registers. That's cool if you want to move around more than one thing at the same time. Let's use more of those!
  • Tabs: I know how tabs work, but all the typing feels clunky. That's why I never extensively used them. Instead, I mostly use multiple terminal tabs or an IDE with Vim bindings for bigger projects.

Would I learn Vim again?

That's a tough question to answer.

On one side, I would say no. There's a steep learning curve in Vim and seeing all those modern IDEs become better at understanding the user's intent, editing text became way easier and faster in general.

On the other side, Vim is the fastest way for me to write down my thoughts and code. As a bonus, it runs on every machine and might well be around for decades to come. In contrast, I don't know if the IntelliJ shortcuts will be relevant in ten years (note: if you read this in the future and ask yourself "What is IntelliJ?", the answer might be no).

Takeaways

If I can give you one tip, don't learn Vim by memorizing commands. Instead, look at your current workflow and try to make it better, then see how Vim can make that easier. It helps to look at other people using Vim to get inspired (Youtube link with sound).

You will spend a lot of time writing text, so it's well worth the time investment to learn one editor really well — especially if you are a programmer.

After ten years, Vim is somehow ingrained in my mind. I think Vim when I'm editing text. It has become yet another natural language to me. I'm looking forward to the next ten years.


Refactoring Go Code to Avoid File I/O in Unit Tests Matthias Endler

Matthias Endler2018-03-22 00:00:00

At work today, I refactored some simple Go code to make it more testable. The idea was to avoid file handling in unit tests without mocking or using temporary files by separating data input/output and data manipulation.


A Tiny `ls` Clone Written in Rust Matthias Endler

Matthias Endler2018-03-09 00:00:00

In my series of useless Unix tools rewritten in Rust, today I'm going to be covering one of my all-time favorites: ls.

First off, let me say that you probably don't want to use this code as a replacement for ls on your local machine (although you could!). As we will find out, ls is actually quite a powerful tool under the hood. I'm not going to come up with a full rewrite, but instead only cover the very basic output that you would expect from calling ls -l on your command line. What is this output? I'm glad you asked.

Expected output

> ls -l
drwxr-xr-x 2 mendler  staff    13468 Feb  4 11:19 Top Secret
-rwxr--r-- 1 mendler  staff  6323935 Mar  8 21:56 Never Gonna Give You Up - Rick Astley.mp3
-rw-r--r-- 1 mendler  staff        0 Feb 18 23:55 Thoughts on Chess Boxing.doc
-rw-r--r-- 1 mendler  staff   380434 Dec 24 16:00 nobel-prize-speech.txt

Your output may vary, but generally, there are a couple of notable things going on. From left to right, we've got the following fields:

  • The drwx things in the beginning are the file permissions (also called the file mode). If d is set, it's a directory. r means read, w means write and x execute. This rwx pattern gets repeated three times for the current user, the group, and other computer users respectively.
  • Next we got the hardlink count when referring to a file, or the number of contained directory entries when referring to a directory. (Reference)
  • Owner name
  • Group name
  • Number of bytes in the file
  • Date when the file was last modified
  • Finally, the path name

For more in-depth information, I can recommend reading the manpage of ls from the GNU coreutils used in most Linux distributions and the one from Darwin (which powers MacOS).

Whew, that's a lot of information for such a tiny tool. But then again, it can't be so hard to port that to Rust, right? Let's get started!

A very basic ls in Rust

Here is the most bare-bones version of ls, which just prints all files in the current directory:

use std::fs;
use std::path::Path;
use std::error::Error;
use std::process;

fn main() {
	if let Err(ref e) = run(Path::new(".")) {
		println!("{}", e);
		process::exit(1);
	}
}

fn run(dir: &Path) -> Result<(), Box<Error>> {
	if dir.is_dir() {
		for entry in fs::read_dir(dir)? {
				let entry = entry?;
				let file_name = entry
						.file_name()
						.into_string()
						.or_else(|f| Err(format!("Invalid entry: {:?}", f)))?;
				println!("{}", file_name);
		}
	}
	Ok(())
}

We can copy that straight out of the documentation. When we run it, we get the expected output:

> cargo run
Cargo.lock
Cargo.toml
src
target

It prints the files and exits. Simple enough.

We should stop for a moment and celebrate our success, knowing that we just wrote our first little Unix utility from scratch. Pro Tip: You can install the binary with cargo install and call it like any other binary from now on.

But we have higher goals, so let's continue.

Adding a parameter to specify the directory

Usually, if we type ls mydir, we expect to get the file listing of no other directory than mydir. We should add the same functionality to our version.

To do this, we need to accept command line parameters. One Rust crate that I love to use in this case is structopt. It makes argument parsing very easy.

Add it to your Cargo.toml. (You need cargo-edit for the following command).

cargo add structopt

Now we can import it and use it in our project:

#[macro_use]
extern crate structopt;

// use std::...
use structopt::StructOpt;

#[derive(StructOpt, Debug)]
struct Opt {
	/// Output file
	#[structopt(default_value = ".", parse(from_os_str))]
	path: PathBuf,
}

fn main() {
	let opt = Opt::from_args();
	if let Err(ref e) = run(&opt.path) {
			println!("{}", e);
			process::exit(1);
	}
}

fn run(dir: &PathBuf) -> Result<(), Box<Error>> {
	// Same as before
}

By adding the Opt struct, we can define the command line flags, input parameters, and the help output super easily. There are tons of configuration options, so it's worth checking out the project homepage.

Also note, that we changed the type of the path variable from Path to PathBuf. The difference is, that PathBuf owns the inner path string, while Path simply provides a reference to it. The relationship is similar to String and &str.

Reading the modification time

Now let's deal with the metadata. First, we try to retrieve the modification time from the file. A quick look at the documentation shows us how to do it:

use std::fs;

let metadata = fs::metadata("foo.txt")?;

if let Ok(time) = metadata.modified() {
	println!("{:?}", time);
}

The output might not be what you expect: we receive a SystemTime object, which represents the measurement of the system clock. E.g. this code

println!("{:?}", SystemTime::now());
// Prints: SystemTime { tv_sec: 1520554933, tv_nsec: 610406401 }

But the format that we would like to have is something like this:

Mar  9 01:24

Thankfully, there is a library called chrono, which can read this format and convert it into any human readable output we like:

let current: DateTime<Local> = DateTime::from(SystemTime::now());
println!("{}", current.format("%_d %b %H:%M").to_string());

this prints

9 Mar 01:29

(Yeah, I know it's getting late.)

Armed with that knowledge, we can now read our file modification time.

cargo add chrono
use chrono::{DateTime, Local};

fn run(dir: &PathBuf) -> Result<(), Box<Error>> {
	if dir.is_dir() {
		for entry in fs::read_dir(dir)? {
			let entry = entry?;
			let file_name = ...

			let metadata = entry.metadata()?;
			let size = metadata.len();
			let modified: DateTime<Local> = DateTime::from(metadata.modified()?);

			println!(
				"{:>5} {} {}",
				size,
				modified.format("%_d %b %H:%M").to_string(),
				file_name
			);
		}
	}
	Ok(())
}

This {:>5} might look weird. It's a formatting directive provided by std::fmt. It means "right align this field with a space padding of 5" - just like our bigger brother ls -l is doing it.

Similarly, we retrieved the size in bytes with metadata.len().

Unix file permissions are a zoo

Reading the file permissions is a bit more tricky. While the rwx notation is very common in Unix derivatives such as *BSD or GNU/Linux, many other operating systems ship their own permission management. There are even differences between the Unix derivatives.

Wikipedia lists a few extensions to the file permissions that you might encounter:

That just goes to show, that there are a lot of important details to be considered when implementing this in real life.

Implementing very basic file mode

For now, we just stick to the basics and assume we are on a platform that supports the rwx file mode.

Behind the r, the w and the x are in reality octal numbers. That's easier for computers to work with and many hardcore users even prefer to type the numbers over the symbols. The ruleset behind those octals is as follows. I took that from the chmod manpage.

	Modes may be absolute or symbolic.
	An absolute mode is an octal number constructed
	from the sum of one or more of the following values

	 0400    Allow read by owner.
	 0200    Allow write by owner.
	 0100    For files, allow execution by owner.
	 0040    Allow read by group members.
	 0020    Allow write by group members.
	 0010    For files, allow execution by group members.
	 0004    Allow read by others.
	 0002    Allow write by others.
	 0001    For files, allow execution by others.

For example, to set the permissions for a file so that the owner can read, write and execute it and nobody else can do anything would be 700 (400 + 200 +100).

Granted, those numbers are the same since the 70s and are not going to change soon, but it's still a bad idea to compare our file permissions directly with the values; if not for compatibility reasons, then for readability and to avoid magic numbers in our code.

Therefore, we use the libc crate, which provides constants for those magic numbers. As mentioned above, these file permissions are Unix specific, so we need to import a Unix-only library named std::os::unix::fs::PermissionsExt; for that.

extern crate libc;

// Examples:
// * `S_IRGRP` stands for "read permission for group",
// * `S_IXUSR` stands for "execution permission for user"
use libc::{S_IRGRP, S_IROTH, S_IRUSR, S_IWGRP, S_IWOTH, S_IWUSR, S_IXGRP, S_IXOTH, S_IXUSR};
use std::os::unix::fs::PermissionsExt;

We can now get the file permissions like so:

let metadata = entry.metadata()?;
let mode = metadata.permissions().mode();
parse_permissions(mode as u16);

parse_permissions() is a little helper function defined as follows:

fn parse_permissions(mode: u16) -> String {
	let user = triplet(mode, S_IRUSR, S_IWUSR, S_IXUSR);
	let group = triplet(mode, S_IRGRP, S_IWGRP, S_IXGRP);
	let other = triplet(mode, S_IROTH, S_IWOTH, S_IXOTH);
	[user, group, other].join("")
}

It takes the file mode as a u16 (simply because the libc constants are u16) and calls triplet on it. For each flag read, write, and execute, it runs a binary & operation on mode. The output is matched exhaustively against all possible permission patterns.

fn triplet(mode: u16, read: u16, write: u16, execute: u16) -> String {
	match (mode & read, mode & write, mode & execute) {
		(0, 0, 0) => "---",
		(_, 0, 0) => "r--",
		(0, _, 0) => "-w-",
		(0, 0, _) => "--x",
		(_, 0, _) => "r-x",
		(_, _, 0) => "rw-",
		(0, _, _) => "-wx",
		(_, _, _) => "rwx",
	}.to_string()
}

Wrapping up

The final output looks like this. Close enough.

> cargo run
rw-r--r--     7  6 Mar 23:10 .gitignore
rw-r--r-- 15618  8 Mar 00:41 Cargo.lock
rw-r--r--   185  8 Mar 00:41 Cargo.toml
rwxr-xr-x   102  5 Mar 21:31 src
rwxr-xr-x   136  6 Mar 23:07 target

That's it! You can find the final version of our toy ls on Github. We are still far away from a full-fledged ls replacement, but at least we learned a thing or two about its internals.

If you're looking for a proper ls replacement written in Rust, go check out lsd. If, instead, you want to read another blog post from the same series, check out A Little Story About the yes Unix Command.


Brewing With Kubernetes Posts on elder.dev

Posts on elder.dev2018-03-04 00:00:00 My coffee pot is now a node in my home Kubernetes cluster, and it’s awesome. More specifically the Raspberry Pi wired to my CoffeePot controller now runs on Kubernetes thanks to kubeadm in a cluster with the node running my site. I set up a public live status page displaying all of the sensor data as well as the last update time, with control restricted to users on my local network.

Migrating My Site to Kubernetes Posts on elder.dev

Posts on elder.dev2018-03-04 00:00:00 Previously when I brought my my site back online I briefly mentioned the simple setup I threw together with Caddy running on a tiny GCE VM with a few scripts — Since then I’ve had plenty of time to experience the awesomeness that is managing services with Kubernetes at work while developing Kubernetes’s testing infrastructure (which we run on GKE). So I decided, of course, that it was only natural to migrate my own service(s) to Kubernetes for maximum dog-fooding.

Rust in 2018 Matthias Endler

Matthias Endler2018-01-09 00:00:00

I wrote about the future of Rust before and it seems like nobody stops me from doing it again! Quite the contrary: this time the Rust core team even asked for it. I'm a bit late to the party, but here are my 2 cents about the priorities for Rust in 2018.


Functional Programming for Mathematical Computing Matthias Endler

Matthias Endler2018-01-02 00:00:00

Programming languages help us describe general solutions for problems; the result just happens to be executable by machines. Every programming language comes with a different set of strengths and weaknesses, one reason being that its syntax and semantics heavily influence the range of problems which can easily be tackled with it.

tl;dr: I think that functional programming is better suited for mathematical computations than the more common imperative approach.

Using built-in abstractions for Mathematics

The ideas behind a language (the underlying programming paradigms) are distinctive for the community that builds around it. The developers create a unique ecosystem of ready-to-use libraries and frameworks around the language core. As a consequence, some languages are stronger in areas such as business applications (one could think of Cobol), others work great for systems programming (like C or Rust).

When it comes to solving mathematical and numerical problems with computers, Fortran might come to mind. Although Fortran is a general-purpose language, it is mostly known for scientific computing. Of course, the language was created with that purpose in mind – hence the name, Formula Translation.

One reason for its popularity in this area is that it offers some built-in domain-specific keywords to express mathematical concepts, while keeping an eye on performance. For instance, it has a dedicated datatype for complex numbers – COMPLEX – and a keyword named DIMENSION which is quite similar to the mathematical term and can be used to create arrays and vectors.

Imperative vs functional style

Built-in keywords can help expand the expressiveness of a language into a specific problem space, but this approach is severly limited. It’s not feasible to extend the language core ad infinitum; it would just be harder to maintain and take longer to learn. Therefore, most languages provide other ways of abstraction – like functions, subroutines, classes and objects – to split a routine into smaller, more manageable parts. These mechanisms might help to control the complexity of a program, but especially when dealing with mathematical problems, one has to be careful not to obfuscate the solution with boilerplate code.

Specimen I - Factorial

As an example, the stated problem might be to translate the following formula, which calculates the factorial of a positive number n, into program code:

The mathematical definition of a faculty: n! = 1 * 2 * 3 ... * n

An implementation of the above formula using imperative style Java might look like this:

public static long fact(final int n) {
    if (n < 0) {
        // Negative numbers not allowed
        return 0;
    }
    long prod = 1;
    for (int i = 1; i <= n; ++i) {
        prod *= i;
    }
    return prod;
}

This is quite a long solution for such a short problem definition. (Note that writing a version with an explicit loop from 1 to n was on purpose; a recursive function would be shorter, but uses a concept which was not introduced by the mathematical formula.)

Also, the program contains many language-specific keywords, such as public, static, and System.err.println(). On top of that, the programmer must explicitly provide all data types for the variables in use – a tiresome obligation.

All of this obfuscates the mathematical definition.

Compare this with the following version written in a functional language, like Haskell.

fact n = product [1..n]

This is an almost direct translation from the problem definition into code. It needs no explicit types, no temporary variables and no access modifiers (such as public).

Specimen II - Dot product

One could argue that the above Haskell program owes its brevity to the fact, that the language provides just the right abstractions (namely the product keyword and the [1..n] range syntax) for that specific task. Therfore let’s examine a simple function which is neither available in Haskell nor in Java: The dot product of two vectors. The mathematical definition is as follows:

The mathematical definition of a vector dot product: a·b= aibi =a1b1+a2b2+···+anbn =abT

For vectors with three dimensions, it can be written as

Vector dot product for three dimentsions: a·b = a1 * b1 + a2 * b2 + a3* b3

First, a Haskell implementation:

type Scalar a = a
data Vector a = Vector a a a deriving (Show)
dot :: (Num a) => Vector a -> Vector a -> Scalar a
(Vector a1 a2 a3) `dot` (Vector b1 b2 b3) = a1*b1 + a2*b2 + a3*b3

Note, that the mathematical types can be defined in one line each. Further note, that we define the dot function in infix notation, that is, we place the first argument of dot in front of the function name and the second argument behind it. This way, the code looks more like its mathematical equivalent. An example call of the above function would be

(Vector 1 2 3) ’dot’ (Vector 3 2 1)

which is short, precise and readable.

Now, a similar implementation in Java.

public static class Vector<T extends Number> {
    private T x, y, z;

    public Vector(T x, T y, T z) {
        this.x = x;
        this.y = y;
        this.z = z;
    }

    public double dot(Vector<?> v) {
        return (x.doubleValue() * v.x.doubleValue() +
                y.doubleValue() * v.y.doubleValue() +
                z.doubleValue() * v.z.doubleValue());
        }
    }

    public static void main(String[] args) {
        Vector<Integer> a = new Vector<Integer>(3, 2, 1);
        Vector<Integer> b = new Vector<Integer>(1, 2, 3);
        System.out.println(a.dot(b));
    }
}

For a proper textual representation of Vectors, the toString() Method would also need to be overwritten. In Haskell, one can simply derive from the Show typeclass as shown in the code.

Creating new abstractions

If functions and types are not sufficient to write straightforward programs, Haskell also offers simple constructs to create new operators and keywords which extend the language core itself. This makes domain-specific-languages feasible and enables the developer to work more directly on the actual problem instead of working around peculiarities of the programming language itself (such as memory management or array iteration). Haskell embraces this concept; Java has no such functionality.

Conclusion

I'm not trying to bash Java or worship Haskell here. Both languages have their place. I merely picked Java, because lots of programmers can read it.

The comparison is more between a functional and an imperative approach for numerical and symbolical programming; and for that, I prefer a functional approach every day. It removes clutter and yields elegant solutions. It provides convenient methods to work on a high level of abstraction and speak in mathematical terms and still, these strengths are disregarded by many programmers.

Abraham H. Maslow’s observation in his 1966 book The Psychology of Science seems fitting:

“I suppose it is tempting, if the only tool you have is a hammer, to treat everything as if it were a nail.”


Prow Posts on elder.dev

Posts on elder.dev2017-12-26 00:00:00 Prow - extended nautical metaphor. Go Gopher originally by Renee French, SVG version by Takuya Ueda, modified under the CC BY 3.0 license. Ship's wheel from Kubernetes logo by Tim Hockin. The Kubernetes project does a lot of testing, on the order of 10000 jobs per day covering everything from build and unit tests, to end-to-end testing on real clusters deployed from source all the way up to ~5000 node scalability and performance tests.

Rust for Rubyists Matthias Endler

Matthias Endler2017-12-17 00:00:00

Recently I came across a delightful article on idiomatic Ruby. I'm not a good Ruby developer by any means, but I realized, that a lot of the patterns are also quite common in Rust. What follows is a side-by-side comparison of idiomatic code in both languages.

The Ruby code samples are from the original article.

Map and Higher-Order Functions

The first example is a pretty basic iteration over elements of a container using map.

user_ids = users.map { |user| user.id }

The map concept is also pretty standard in Rust. Compared to Ruby, we need to be a little more explicit here: If users is a vector of User objects, we first need to create an iterator from it:

let user_ids = users.iter().map(|user| user.id);

You might say that's quite verbose, but this additional abstraction allows us to express an important concept: will the iterator take ownership of the vector, or will it not?

  • With iter(), you get a "read-only view" into the vector. After the iteration, it will be unchanged.
  • With into_iter(), you take ownership over the vector. After the iteration, the vector will be gone. In Rust terminology, it will have moved.
  • Read some more about the difference between iter() and into_iter() here.

The above Ruby code can be simplified like this:

user_ids = users.map(&:id)

In Ruby, higher-order functions (like map) take blocks or procs as an argument and the language provides a convenient shortcut for method invocation — &:id is the same as {|o| o.id()}.

Something similar could be done in Rust:

let id = |u: &User| u.id;
let user_ids = users.iter().map(id);

This is probably not the most idiomatic way to do it, though. What you will see more often is the use of Universal Function Call Syntax in this case:1

let user_ids = users.iter().map(User::id);

In Rust, higher-order functions take functions as an argument. Therefore users.iter().map(Users::id) is more or less equivalent to users.iter().map(|u| u.id()).2

Also, map() in Rust returns another iterator and not a collection. If you want a collection, you would have to run collect() on that, as we'll see later.

Iteration with Each

Speaking of iteration, one pattern that I see a lot in Ruby code is this:

["Ruby", "Rust", "Python", "Cobol"].each do |lang|
  puts "Hello #{lang}!"
end

Since Rust 1.21, this is now also possible:

["Ruby", "Rust", "Python", "Cobol"]
    .iter()
    .for_each(|lang| println!("Hello {lang}!", lang = lang));

Although, more commonly one would write that as a normal for-loop in Rust:

for lang in ["Ruby", "Rust", "Python", "Cobol"].iter() {
    println!("Hello {lang}!", lang = lang);
}

Select and filter

Let's say you want to extract only even numbers from a collection in Ruby.

even_numbers = [1, 2, 3, 4, 5].map { |element| element if element.even? } # [ni, 2, nil, 4, nil]
even_numbers = even_numbers.compact # [2, 4]

In this example, before calling compact, our even_numbers array had nil entries. Well, in Rust there is no concept of nil or Null. You don't need a compact. Also, map doesn't take predicates. You would use filter for that:

let even_numbers = vec![1, 2, 3, 4, 5]
    .iter()
    .filter(|&element| element % 2 == 0);

or, to make a vector out of the result

// Result: [2, 4]
let even_numbers: Vec<i64> = vec![1, 2, 3, 4, 5]
    .into_iter()
    .filter(|element| element % 2 == 0).collect();

Some hints:

  • I'm using the type hint Vec<i64> here because, without it, Rust does not know what collection I want to build when calling collect.
  • vec! is a macro for creating a vector.
  • Instead of iter, I use into_iter. This way, I take ownership of the elements in the vector. With iter() I would get a Vec<&i64> instead.

In Rust, there is no even method on numbers, but that doesn't keep us from defining one!

let even = |x: &i64| x % 2 == 0;
let even_numbers = vec![1, 2, 3, 4, 5].into_iter().filter(even);

In a real-world scenario, you would probably use a third-party package (crate) like num for numerical mathematics:

extern crate num;
use num::Integer;

fn main() {
    let even_numbers: Vec<i64> = vec![1, 2, 3, 4, 5]
        .into_iter()
        .filter(|x| x.is_even()).collect();
}

In general, it's quite common to use crates in Rust for functionality that is not in the standard lib. Part of the reason why this is so well accepted is that cargo is such a rad package manager. (Maybe because it was built by no other than Yehuda Katz of Ruby fame. 😉)

As mentioned before, Rust does not have nil. However, there is still the concept of operations that can fail. The canonical type to express that is called Result.

Let's say you want to convert a vector of strings to integers.

let maybe_numbers = vec!["1", "2", "nah", "nope", "3"];
let numbers: Vec<_> = maybe_numbers
    .into_iter()
    .map(|i| i.parse::<u64>())
    .collect();

That looks nice, but maybe the output is a little unexpected. numbers will also contain the parsing errors:

[Ok(1), Ok(2), Err(ParseIntError { kind: InvalidDigit }), Err(ParseIntError { kind: InvalidDigit }), Ok(3)]

Sometimes you're just interested in the successful operations. An easy way to filter out the errors is to use filter_map:

let maybe_numbers = vec!["1", "2", "nah", "nope", "3"];
let numbers: Vec<_> = maybe_numbers
    .into_iter()
    .filter_map(|i| i.parse::<u64>().ok())
    .collect();

I changed two things here:

  • Instead of map, I'm now using filter_map.
  • parse returns a Result, but filter_map expects an Option. We can convert a Result into an Option by calling ok() on it3.

The return value contains all successfully converted strings:

[1, 2, 3]

The filter_map is similar to the select method in Ruby:

[1, 2, 3, 4, 5].select { |element| element.even? }

Random numbers

Here's how to get a random number from an array in Ruby:

[1, 2, 3].sample

That's quite nice and idiomatic! Compare that to Rust:

let mut rng = thread_rng();
rng.choose(&[1, 2, 3, 4, 5])

For the code to work, you need the rand crate. Click on the snippet for a running example.

There are some differences to Ruby. Namely, we need to be more explicit about what random number generator we want exactly. We decide for a lazily-initialized thread-local random number generator, seeded by the system. In this case, I'm using a slice instead of a vector. The main difference is that the slice has a fixed size while the vector does not.

Within the standard library, Rust doesn't have a sample or choose method on the slice itself. That's a design decision: the core of the language is kept small to allow evolving the language in the future.

This doesn't mean that you cannot have a nicer implementation today. For instance, you could define a Choose trait and implement it for [T].

extern crate rand;
use rand::{thread_rng, Rng};

trait Choose<T> {
    fn choose(&self) -> Option<&T>;
}

impl<T> Choose<T> for [T] {
    fn choose(&self) -> Option<&T> {
        let mut rng = thread_rng();
        rng.choose(&self)
    }
}

This boilerplate could be put into a crate to make it reusable for others. With that, we arrive at a solution that rivals Ruby's elegance.

[1, 2, 4, 8, 16, 32].choose()

Implicit returns and expressions

Ruby methods automatically return the result of the last statement.

def get_user_ids(users)
  users.map(&:id)
end

Same for Rust. Note the missing semicolon.

fn get_user_ids(users: &[User]) -> Vec<u64> {
    users.iter().map(|user| user.id).collect()
}

But in Rust, this is just the beginning, because everything is an expression. The following block splits a string into characters, removes the h, and returns the result as a HashSet. This HashSet will be assigned to x.

let x: HashSet<_> = {
    // Get unique chars of a word {'h', 'e', 'l', 'o'}
    let unique = "hello".chars();
    // filter out the 'h'
    unique.filter(|&char| char != 'h').collect()
};

Same works for conditions:

let x = if 1 > 0 { "absolutely!" } else { "no seriously" };

Since a match statement is also an expression, you can assign the result to a variable, too!

enum Unit {
    Meter,
    Yard,
    Angstroem,
    Lightyear,
}

let length_in_meters = match unit {
    Unit::Meter => 1.0,
    Unit::Yard => 0.91,
    Unit::Angstroem => 0.0000000001,
    Unit::Lightyear => 9.461e+15,
};

Multiple Assignments

In Ruby you can assign multiple values to variables in one step:

def values
  [1, 2, 3]
end

one, two, three = values

In Rust, you can only decompose tuples into tuples, but not a vector into a tuple for example. So this will work:

let (one, two, three) = (1, 2, 3);

But this won't:

let (one, two, three) = [1, 2, 3];
//    ^^^^^^^^^^^^^^^^^ expected array of 3 elements, found tuple

Neither will this:

let (one, two, three) = [1, 2, 3].iter().collect();
// a collection of type `(_, _, _)` cannot be built from an iterator over elements of type `&{integer}`

But with nightly Rust, you can now do this:

let [one, two, three] = [1, 2, 3];

On the other hand, there's a lot more you can do with destructuring apart from multiple assignments. You can write beautiful, ergonomic code using pattern syntax.

let x = 4;
let y = false;

match x {
    4 | 5 | 6 if y => println!("yes"),
    _ => println!("no"),
}

To quote The Book:

This prints no since the if condition applies to the whole pattern 4 | 5 | 6, not only to the last value 6.

String interpolation

Ruby has extensive string interpolation support.

programming_language = "Ruby"
"#{programming_language} is a beautiful programming language"

This can be translated like so:

let programming_language = "Rust";
format!("{} is also a beautiful programming language", programming_language);

Named arguments are also possible, albeit much less common:

println!("{language} is also a beautiful programming language", language="Rust");

Rust's println!() syntax is even more extensive than Ruby's. Check the docs if you're curious about what else you can do.

That’s it!

Ruby comes with syntactic sugar for many common usage patterns, which allows for very elegant code. Low-level programming and raw performance are no primary goals of the language.

If you do need that, Rust might be a good fit, because it provides fine-grained hardware control with comparable ergonomics. If in doubt, Rust favors explicitness, though; it eschews magic.

Did I whet your appetite for idiomatic Rust? Have a look at this Github project. I'd be thankful for contributions.

Footnotes

1. Thanks to Florian Gilcher for the hint.
2. Thanks to masklin for pointing out multiple inaccuracies.
3. In the first version, I sait that ok() would convert a Result into a boolean, which was wrong. Thanks to isaacg for the correction.


Making Myself Obsolete Matthias Endler

Matthias Endler2017-12-10 00:00:00
The Stegosaurus had better days 150 million years ago.
The Stegosaurus had better days 150 million years ago.
Source: Paleontologists once thought it had a brain in its butt.

In December 2015 I was looking for static analysis tools to integrate into trivago's CI process. The idea was to detect typical programming mistakes automatically. That's quite a common thing, and there are lots of helpful tools out there which fit the bill.

So I looked for a list of tools...

To my surprise, the only list I found was on Wikipedia — and it was outdated. There was no such project on Github, where most modern static analysis tools were hosted.

Without overthinking it, I opened up my editor and wrote down a few tools I found through my initial research. After that, I pushed the list to Github.

I called the project Awesome Static Analysis.

Fast forward two years and the list has grown quite a bit. So far, it has 75 contributors, 277 forks and received over 2000 stars. (Thanks for all the support!) (Update May 2018: 91 contributors, 363 forks, over 3000 stars)

Around 1000 unique visitors find the list every week. Not much by any means, but I feel obliged to keep it up-to-date because it has become an essential source of information for many people.

It now lists around 300 tools for static analysis. Everything from Ada to TypeScript is on there. What I find particularly motivating is, that now the authors themselves create pull requests to add their tools!

There was one problem though: The list of pull requests got longer and longer, as I was busy doing other things.

The list of Github Pull requests for awesome-static-analysis

Adding contributors

I always try to make team members out of regular contributors. My friend and colleague Andy Grunwald as well as Ouroboros Chrysopoeia are both valuable collaborators. They help me weed out new PRs whenever they find the time.

But let's face it: checking the pull requests is a dull, manual task. What needs to be checked for each new tool can be summarized like this:

  • Formatting rules are satisfied
  • Project URL is reachable
  • License annotation is correct
  • Tools of each section are alphabetically ordered
  • Description is not too long

I guess it's obvious what we should do with that checklist: automate it!

A linter for linting linters

So why not write an analysis tool, which checks our list of analysis tools! What sounds pretty meta, is actually pretty straightforward.

With every pull request, we trigger our bot, which checks the above rules and responds with a result.

The first step was to read the Github documentation about building a CI server.

Just for fun, I wanted to create the bot in Rust. The two most popular Github clients for Rust were github-rs and hubcaps. Both looked pretty neat, but then I found afterparty, a "Github webhook server".

The example looked fabulous:

#[macro_use]
extern crate log;
extern crate env_logger;
extern crate afterparty;
extern crate hyper;

use afterparty::{Delivery, Hub};

use hyper::Server;

pub fn main() {
    env_logger::init().unwrap();
    let addr = format!("0.0.0.0:{}", 4567);
    let mut hub = Hub::new();
    hub.handle("pull_request", |delivery: &Delivery| {
        match delivery.payload {
            Event::PullRequest { ref action, ref sender, .. } => {
                // TODO: My code here!
                println!("sender {} action {}", sender.login, action)
            }
            _ => (),
        }
    });
    let srvc = Server::http(&addr[..])
                   .unwrap()
                   .handle(hub);
    println!("listening on {}", addr);
    srvc.unwrap();
}

This allowed me to focus on the actual analysis code, which makes for a pretty boring read. It mechanically checks for the things mentioned above and could be written in any language. If you want to have a look (or even contribute!), check out the repo.

Talking to Github

After the analysis code was done, I had a bot, running locally, waiting for incoming pull requests.

But how could I talk to Github?
I found out, that I should use the Status API and send a POST request to /repos/mre/awesome-static-analysis/statuses/:sha
(:sha is the commit ID that points to the HEAD of the pull request):

{
  "state": "success",
  "description": "The build succeeded!"
}

I could have used one of the existing Rust Github clients, but I decided to write a simple function to update the pull request status code.

fn set_status(status: Status, desc: String, repo: &str, sha: &str) -> Result<reqwest::Response> {
    let token = env::var("GITHUB_TOKEN")?;
    let client = reqwest::Client::new();
    let mut params = HashMap::new();
    params.insert("state", format!("{}", status));
    params.insert("description", desc);
    println!("Sending status: {:#?}", params);

    let status_url = format!("https://api.github.com/repos/{}/statuses/{}", repo, sha);
    println!("Status url: {}", status_url);
    Ok(client
        .request(
            reqwest::Method::Post,
            &format!(
                "{}?access_token={}",
                status_url,
                token,
            ),
        )
        .json(&params)
        .send()?)
}

You can see that I pass in a Github token from the environment and then I send the JSON payload as a post request using the reqwest library.

That turned out to become a problem in the end: while afterparty was using version 0.9 of hyper, reqwest was using 0.11. Unfortunately, these two versions depend on a different build of the openssl-sys bindings. That's a well-known problem and the only way to fix it, is to resolve the conflict.

I was stuck for a while, but then I saw, that there was an open pull request to upgrade afterparty to hyper 0.10.

So inside my Cargo.toml, I locked the version of afterparty to the version of the pull request:

[dependencies]
afterparty = { git = "https://github.com/ms705/afterparty" }

This fixed the build, and I could finally move on.

Deployment

I needed a place to host the bot.

Preferably for free, as it was a non-profit Open Source project. Also, the provider would have to run binaries.

For quite some time, I was following a product named zeit. It runs any Docker container using an intuitive command line interface called now.

I fell in love the first time I saw their demo on the site, so I wanted to give it a try.

So I added a multi-stage Dockerfile to my project:

FROM rust as builder
COPY . /usr/src/app
WORKDIR /usr/src/app
RUN cargo build --release

FROM debian:stretch
RUN apt update \
    && apt install -y libssl1.1 ca-certificates \
    && apt clean -y \
    && apt autoclean -y \
    && apt autoremove -y
COPY --from=builder target/release/check .
EXPOSE 4567
ENTRYPOINT ["./check"]
CMD ["--help"]

The first part would build a static binary, the second part would run it at container startup. Well, that didn't work, because zeit does not support multi-stage builds yet.

The workaround was to split up the Dockerfile into two and connect them both with a Makefile. Makefiles are pretty powerful, you know?

With that, I had all the parts for deployment together.

# Build Rust binary for Linux
docker run --rm -v $(CURDIR):/usr/src/ci -w /usr/src/ci rust cargo build --release

# Deploy Docker images built from the local Dockerfile
now deploy --force --public -e GITHUB_TOKEN=${GITHUB_TOKEN}

# Set domain name of new build to `check.now.sh`
# (The deployment URL was copied to the clipboard and is retrieved with pbpaste on macOS)
now alias `pbpaste` check.now.sh

Here's the output of the deploy using now:

> Deploying ~/Code/private/awesome-static-analysis-ci/deploy
> Ready! https://deploy-sjbiykfvtx.now.sh (copied to clipboard) [2s]
> Initializing…
> Initializing…
> Building
> ▲ docker build
Sending build context to Docker daemon 2.048 kBkB
> Step 1 : FROM mre0/ci:latest
> latest: Pulling from mre0/ci
> ...
> Digest: sha256:5ad07c12184755b84ca1b587e91b97c30f7d547e76628645a2c23dc1d9d3fd4b
> Status: Downloaded newer image for mre0/ci:latest
>  ---> 8ee1b20de28b
> Successfully built 8ee1b20de28b
> ▲ Storing image
> ▲ Deploying image
> ▲ Container started
> listening on 0.0.0.0:4567
> Deployment complete!

The last step was to add check.now.sh as a webhook inside the awesome-static-analysis project settings.

Now, whenever a new pull request is coming in, you see that little bot getting active!

A successful pull request, which was checked by the bot

Outcome and future plans

I am very pleased with my choice of tools: afterparty saved me from a lot of manual work, while zeit made deployment really easy.
It feels like Amazon Lambda on steroids.

If you look at the code and the commits for my bot, you can see all my little missteps, until I got everything just right. Turns out, parsing human-readable text is tedious.
Therefore I was thinking about turning the list of analysis tools into a structured format like YAML. This would greatly simplify the parsing and have the added benefit of having a machine-readable list of tools that can be used for other projects.

Update May 2018

While attending the WeAreDevelopers conference in Vienna (can recommend that), I moved the CI pipeline from zeit.co to Travis CI. The reason was, that I wanted the linting code next to the project, which greatly simplified things. First and foremost I don't need the web request handling code anymore, because travis takes care of that. If you like, you can compare the old and the new version.


Modern Day Annoyances - Digital Clocks Matthias Endler

Matthias Endler2017-11-07 00:00:00

This morning I woke up to the beeping noise of our oven's alarm clock. The reason was that I tried to correct the oven's local time the day before — and I pushed the wrong buttons. As a result I didn't set the correct time, instead, I set a cooking timer... and that's what woke me up today.


Learn Some Rust During Hacktoberfest Matthias Endler

Matthias Endler2017-10-15 00:00:00
Dirndl, Lederhose, Brezn, Beer, Rust
Dirndl, Lederhose, Brezn, Beer, Rust
Source: Designed by Freepik

October is the perfect time to contribute to Open Source — at least according to Github and DigitalOcean. Because that's when they organize Hacktoberfest, a global event where you get a free shirt and lots of street cred for creating pull requests. Read the official announcement here.

Some people think they cannot contribute anything of value. Either because they lack the programming skills or because they don't know where to start.

This guide is trying to change that!

Let me show you, how everybody can contribute code to Rust, a safe systems programming language. I was inspired to write this by a tweet from llogiq.

1. Find a great Rust project to work on

We all want our work to be appreciated.
Therefore I suggest to start contributing to medium-sized projects, because they gained some momentum but are still driven by a small number of maintainers, so help is always welcome. By contrast, tiny projects are mostly useful to the original author only, while large projects can be intimidating at first and have stricter guidelines.

For now, let's look at repositories with 5-100 stars, which were updated within this year. Github supports advanced search options based on Lucene syntax.

language:Rust stars:5..100 pushed:>2017-01-01

Here's a list of projects, which match this filter.

2. Install the Rust toolchain

To start contributing, we need a working Rust compiler and the cargo package manager. Fortunately, the installation should be straightforward. I recommend rustup for that.

Run the following command in your terminal, then follow the onscreen instructions.

curl https://sh.rustup.rs -sSf | sh

If you're unsure, just accept the defaults. After the installation is done, we also need to get the nightly version of the compiler for later.

rustup install nightly

Questions so far? Find more detailed installation instructions here.

3. Fork the project and clone it to your computer

First, click on the little fork button on the top right of the Github project page. Then clone your fork to your computer.

git clone git@github.com:yourusername/project.git

For more detailed instructions, go here.

4. Does it build?

Before we start modifying the codebase, we should make sure that it is in a workable state. The following commands should work right away from inside the project folder.

cargo build
cargo test

If not, you might want to consult the README for further instructions. (But feel free to choose another project.)

5. The magic sauce

Here's the trick: we use a linter called clippy to show us improvement areas in any Rust codebase.

To get clippy, install it like so:

cargo +nightly install clippy

Afterwards, run it from the project root as often as you like.

rustup run nightly cargo clippy

This should give you actionable information on how to improve the codebase.

Here's some sample output:

warning: useless use of `format!`
   --> src/mach/header.rs:420:49
    |
420 |             let error = error::Error::Malformed(format!("bytes size is smaller than an Mach-o header"));
    |                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |
    = note: #[warn(useless_format)] on by default
    = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.165/index.html#useless_format

warning: this expression borrows a reference that is immediately dereferenced by the compiler
   --> src/mach/header.rs:423:36
    |
423 |             let magic = mach::peek(&bytes, 0)?;
    |                                    ^^^^^^ help: change this to: `bytes`
    |
    = help: for further information visit https://rust-lang-nursery.github.io/rust-clippy/v0.0.165/index.html#needless_borrow

Just try some of the suggestions and see if the project still compiles and the tests still pass. Check out the links to the documentation in the help section to learn more. Start small to make your changes easier to review.

6. Creating a Pull Request

If you're happy with your changes, now is the time to publish them! It's best to create a new branch for your changes and then push it to your fork.

git checkout -b codestyle
git commit -am "Minor codestyle fixes"
git push --set-upstream origin codestyle

Afterwards, go to the homepage of your fork on Github. There should be a button titled Compare & pull request. Please add a meaningful description and then submit the pull request.

Congratulations! You've contributed to the Rust ecosystem. Thank you! 🎉

Trophy case

Bonus!

If all of the manual fixing and checking sounds too dull, you can automate step number 5 using rustfix by Pascal Hertleif (@killercup):

rustfix --yolo && cargo check

A Little Story About the `yes` Unix Command Matthias Endler

Matthias Endler2017-10-10 00:00:00

What's the simplest Unix command you know?
There's echo, which prints a string to stdout and true, which always terminates with an exit code of 0.

Among the series of simple Unix commands, there's also yes. If you execute it without arguments, you get an infinite stream of y's, separated by a newline:

y
y
y
y
(...you get the idea)

What seems to be pointless in the beginning turns out to be pretty helpful :

yes | sh boring_installation.sh

Ever installed a program, which required you to type "y" and hit enter to keep going? yes to the rescue! It will carefully fulfill its duty, so you can keep watching Pootie Tang.

Writing yes

Here's a basic version in... uhm... BASIC.

10 PRINT "y"
20 GOTO 10

And here's the same thing in Python:

while True:
    print("y")

Simple, eh? Not so quick!
Turns out, that program is quite slow.

python yes.py | pv -r > /dev/null
[4.17MiB/s]

Compare that with the built-in version on my Mac:

yes | pv -r > /dev/null
[34.2MiB/s]

So I tried to write a quicker version in Rust. Here's my first attempt:

use std::env;

fn main() {
  let expletive = env::args().nth(1).unwrap_or("y".into());
  loop {
    println!("{}", expletive);
  }
}

Some explanations:

  • The string we want to print in a loop is the first command line parameter and is named expletive. I learned this word from the yes manpage.
  • I use unwrap_or to get the expletive from the parameters. In case the parameter is not set, we use "y" as a default.
  • The default parameter gets converted from a string slice (&str) into an owned string on the heap (String) using into().

Let's test it.

cargo run --release | pv -r > /dev/null
   Compiling yes v0.1.0
    Finished release [optimized] target(s) in 1.0 secs
     Running `target/release/yes`
[2.35MiB/s]

Whoops, that doesn't look any better. It's even slower than the Python version! That caught my attention, so I looked around for the source code of a C implementation.

Here's the very first version of the program, released with Version 7 Unix and famously authored by Ken Thompson on Jan 10, 1979:

main(argc, argv)
char **argv;
{
  for (;;)
    printf("%s\n", argc>1? argv[1]: "y");
}

No magic here.

Compare that to the 128-line-version from the GNU coreutils, which is mirrored on Github. After 25 years, it is still under active development! The last code change happened around a year ago. That's quite fast:

# brew install coreutils
gyes | pv -r > /dev/null
[854MiB/s]

The important part is at the end:

/* Repeatedly output the buffer until there is a write error; then fail.  */
while (full_write (STDOUT_FILENO, buf, bufused) == bufused)
  continue;

Aha! So they simply use a buffer to make write operations faster. The buffer size is defined by a constant named BUFSIZ, which gets chosen on each system so as to make I/O efficient (see here). On my system, that was defined as 1024 bytes. I actually had better performance with 8192 bytes.

I've extended my Rust program:

use std::env;
use std::io::{self, BufWriter, Write};

const BUFSIZE: usize = 8192;

fn main() {
    let expletive = env::args().nth(1).unwrap_or("y".into());
    let mut writer = BufWriter::with_capacity(BUFSIZE, io::stdout());
    loop {
        writeln!(writer, "{}", expletive).unwrap();
    }
}

The important part is, that the buffer size is a multiple of four, to ensure memory alignment.

Running that gave me 51.3MiB/s. Faster than the version, which comes with my system, but still way slower than the results from this Reddit post that I found, where the author talks about 10.2GiB/s.

Update

Once again, the Rust community did not disappoint.
As soon as this post hit the Rust subreddit, user nwydo pointed out a previous discussion on the same topic. Here's their optimized code, that breaks the 3GB/s mark on my machine:

use std::env;
use std::io<