The Tao of Cal - Cal Newport

With the end of year rapidly approaching, and people finding themselves with some spare thinking time as work winds down for the holidays, I thought it might be fun to try to summarize essentially every major idea I discuss in one short primer.

I’m a big fan of Cal Newport’s work. I’d like to quote the entire post, but I’ll just post this one sentence. Read the rest of this article, then buy his books and read those too. Follow his advice and live a better life.

Godot Isn’t Making it

Outside of a miracle, we are about to enter an era of desperation in the generative AI space. We’re two years in, and we have no killer apps — no industry-defining products — other than ChatGPT, a product that burns billions of dollars and nobody can really describe. Neither Microsoft, nor Meta, nor Google or Amazon seem to be able to come up with a profitable use case, let alone one their users actually like, nor have any of the people that have raised billions of dollars in venture capital for anything with “AI” taped to the side — and investor interest in AI is cooling.

Edward Zitron seems to be a rare voice of frustrated reason in the tech industry. He’s very critical of AI, and, more and more, I’m thinking rightfully so. OpenAI is spending over $2 to make $1, burning through billions with no path to profitability.

Couple that with the environmental cost of AI (and its just plain awful cousin, crypto currency) and the unreliability of the generated answers, and I’m wondering just where all of this goes in the next year or so.

I’m not sure what’s going on over in Cupertino for them to think that any of the recent Apple Intelligence ads they’ve been running are a good idea. They’re cringy at best, and honestly just flat out insulting.

In one a schlub writes an email to his boss and uses AI to make it sound ‘more professional’, in another a young woman uses it to lie about remembering an acquaintance’s name. In another the same young woman again uses it to lie about reading an email from a college, to her face, while she’s sitting with her. In yet another, linked to recently by Scott McNulty, a woman uses AI to lie to her husband about getting him something for his birthday.

If this is what Apple thinks their AI is for, I honestly don’t know that I want any part of it.

Compare and contrast with the video I posted yesterday, and with this beautiful animation from Canonical.

I’ve watched that little animation several times, and they tell a better story in a minute twenty-five than all of Apple’s AI commercials combined.

I’m rooting for these guys. If they can pull off this truck at the $50-$60k mark, I think they are going to have a winner. I’ve been looking at electric trucks for a while, and I’m excited to see another entry in the market. And what a fantastic video:

A proper body-on-frame truck, 10,000 lb towing capacity, all electric, made in the USA. Count me in.

Love this guy. Patrick Rhone calls him folksy, I agree.

What a difference a couple of lines of code can make.

I recognize that databases have always been a weak point for me, so I’ve been trying to correct that lately. I have a lot of experience with management of the database engines, failover, filesystems, and networking, but too little working with the internals of the databases themselves. Early this morning I decided I didn’t know enough about how database indexes worked. So I did some reading, got to the point where I had a good mental model for them, and decided I’d like to do some testing myself. I figured 40 million records was a nice round number, so I used fakedata to generate 40 million SQL inserts that looked something like this:

INSERT INTO contacts (name,email,country) VALUES ("Milo Morris","pmeissner@test.tienda","Italy");
INSERT INTO contacts (name,email,country) VALUES ("Hosea Burgess","kolage@example.walmart","Dominica");
INSERT INTO contacts (name,email,country) VALUES ("Adaline Frank","shaneIxD@example.talk","Slovenia");

I saved this as fakedata.sql and piped it into sqlite3 and figured I’d just let it run in the background. After about six hours I realized this was taking a ridiculously long time, and I estimated I’d only loaded about a quarter of the data. I believe that’s because SQLite was treating each INSERT as a separate transaction.

A transaction in SQLite is a unit of work. SQLite ensures that the write to the database is Atomic, Consistent, Isolated, and Durable, which means that for each of the 40 million lines I was piping into sqlite3, the engine was ensuring that every line was fully committed to the database before moving on to the next line. That’s a lot of work for a very, very small amount of data. So, I did some more reading and found one recommendation of explicitly wrapping the entire load into a single transaction, so my file now looked like:

BEGIN TRANSACTION;

INSERT INTO contacts (name,email,country) VALUES ("Milo Morris","pmeissner@test.tienda","Italy");
INSERT INTO contacts (name,email,country) VALUES ("Hosea Burgess","kolage@example.walmart","Dominica");
INSERT INTO contacts (name,email,country) VALUES ("Adaline Frank","shaneIxD@example.talk","Slovenia");

COMMIT;

I set a timer and ran the import again:

➜  var time cat fakedata.sql| sqlite3 test.db
cat fakedata.sql  0.07s user 0.90s system 1% cpu 1:13.66 total
sqlite3 test.db  70.81s user 2.19s system 98% cpu 1:13.79 total

So, that went from 6+ hours to about 71 seconds. And I imagine if I did some more optimization (possibly using the Write Ahead Log?) I might be able to get that import faster still. But a little over a minute is good enough for some local curiosity testing.

Indexes

So… back to indexes.

Indexing is a way of sorting a number of records on multiple fields. Creating an index on a field in a table creates another data structure that holds the field values and a pointer to the record it relates to. Once the index is created it is sorted. This allows binary searches to be performed on the new data structure.

One good analogy is the index of a physical book. Imagine that a book has ten chapters and each chapter has 100 pages. Now imagine you’d like to find all instances of the word “continuum” in the book. If the book doesn’t have an index, you’d have to read through every page in every chapter to find the word.

However, if the book is already indexed, you can find the word in the alphabetical list, which will then have a pointer to the page numbers where the word can be found.

The downside to the index is that it does take additional space. In the book analogy, while the book itself is 1000 pages, we’d need another ten or so for the index, bringing up the total size to 1010 pages. Same with a database, the additional index data structure requires more space to hold both the original data field being indexed, and a small (4-byte, for example) pointer to the record.

Oh, and the results of creating the index are below.

SELECT * from contacts WHERE name is 'Hank Perry';
Run Time: real 2.124 user 1.771679 sys 0.322396


CREATE INDEX IF NOT EXISTS name_index on contacts (name);
Run Time: real 22.129 user 16.048308 sys 2.274184


SELECT * from contacts WHERE name is 'Hank Perry';
Run Time: real 0.003 user 0.001287 sys 0.001598

That’s a massive improvement. And now I know a little more than I did.

If you spend all day in the terminal like I do, you come to appreciate it’s speed and efficiency. I often find myself in Terminal for mundane tasks like navigating to a folder and opening a file; it’s just faster to type where I want to go than it is to click in the Finder, scan the folders for the one I want, double-click that one, scan again… small improvements to the speed of my work build up over time. The speed is increased exponentially with the correct configuration for your shell, in my case, zsh.

zsh is powerful and flexible, which means that it can also be intimidating to try to configure yourself. Doubly-so when there are multiple ‘frameworks’ available that will do the bulk of the configuration for you. I used Oh My Zsh for years, but I recently abandoned it in favor of maintaining my own configuration using only the settings that I need for the perfect configuration for my use.

I’ve split my configuration into five files:

• apple.zsh-theme
• zshenv
• zshrc
• zsh_alias
• zsh_functions

I have all five files in a dotfiles git repository, pushed to a private Github repository.

The zshenv file is read first by zsh when starting a new shell. It contains a collection of environmental variables I’ve set, mainly for development. For example:

export PIP_REQUIRE_VIRTUALENV=true
export PIP_DOWNLOAD_CACHE=$HOME/.pip/cache
export VIRTUALENV_DISTRIBUTE=true

The next file is zshrc, which contains the main bulk of the configurations. My file is 113 lines, so let’s take it a section at a time.

source /Users/jonathanbuys/Unix/etc/dotfiles/apple.zsh-theme
source /Users/jonathanbuys/Unix/etc/dotfiles/zsh_alias
source /Users/jonathanbuys/Unix/etc/dotfiles/zsh_functions

The first thing I do is source the other three files. The first is my prompt, which is cribbed entirely from Oh My Zsh. It’s nothing fancy, but I consider it to be elegant and functional. I don’t like the massive multi-line prompts. I find them to be far too distracting for what they are supposed to do.

My prompt looks like this:

 ~/Unix/etc/dotfiles/ [master*] 

It gives me my current path, what git branch I’ve checked out, and if that branch has been modified since the last commit.

The next two files, as their names suggest, contain aliases and functions. I have three functions and 16 aliases. I won’t go into each of them here, as they are fairly mundane and only specific for my setup. The three functions are to print the current path of the open Finder window, to use Quicklook to preview a file, and to generate a uuid string.

The next few lines establish some basic settings.

autoload -U colors && colors
autoload -U zmv

setopt AUTO_CD
setopt NOCLOBBER
setopt SHARE_HISTORY
setopt HIST_IGNORE_DUPS
setopt HIST_IGNORE_SPACE

The autoload lines setup zsh to use pretty colors, and to enable the extremely useful zmv command for batch file renaming. The interesting parts of the setopt settings are the ones dealing with command history. These three commands allow the sharing of command line history between open windows or tabs. So if I have multiple Terminal windows open, I can browse the history of both from either window. I find myself thinking that the environment is broken if this is not present.

Next, I setup some bindings:

  # start typing + [Up-Arrow] - fuzzy find history forward
  bindkey '^[[A' up-line-or-search
  bindkey '^[[B' down-line-or-search
  
  # Use option as meta
  bindkey "^[f" forward-word
  bindkey "^[b" backward-word
  
  # Use option+backspace to delete words
  x-bash-backward-kill-word(){
      WORDCHARS='' zle backward-kill-word
  
  }
  zle -N x-bash-backward-kill-word
  bindkey '^W' x-bash-backward-kill-word
  
  x-backward-kill-word(){
      WORDCHARS='*?_-[]~\!#$%^(){}<>|`@#$%^*()+:?' zle backward-kill-word
  }
  zle -N x-backward-kill-word
  bindkey '\e^?' x-backward-kill-word

These settings let me use the arrow keys to browse history, and to use option + arrow keys to move one word at a time through the current command, or to use option + delete to delete one word at a time. Incredibly useful, use it all the time. Importantly, this also lets me do incremental searching through my command history with the arrow keys. So, if I type aws, then arrow up, I can browse all of my previous commands that start with aws. And when you have to remember commands that have 15 arguments, this is absolutely invaluable.

The next section has to do with autocompletion.

# Better autocomplete for file names
WORDCHARS=''

unsetopt menu_complete   # do not autoselect the first completion entry
unsetopt flowcontrol
setopt auto_menu         # show completion menu on successive tab press
setopt complete_in_word
setopt always_to_end

zstyle ':completion:*:*:*:*:*' menu select

# case insensitive (all), partial-word and substring completion
if [[ "$CASE_SENSITIVE" = true ]]; then
  zstyle ':completion:*' matcher-list 'r:|=*' 'l:|=* r:|=*'
else
  if [[ "$HYPHEN_INSENSITIVE" = true ]]; then
    zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]-_}={[:upper:][:lower:]_-}' 'r:|=*' 'l:|=* r:|=*'
  else
    zstyle ':completion:*' matcher-list 'm:{[:lower:][:upper:]}={[:upper:][:lower:]}' 'r:|=*' 'l:|=* r:|=*'
  fi
fi

unset CASE_SENSITIVE HYPHEN_INSENSITIVE
# Complete . and .. special directories
zstyle ':completion:*' special-dirs true

zstyle ':completion:*' list-colors ''
zstyle ':completion:*:*:kill:*:processes' list-colors '=(#b) #([0-9]#) ([0-9a-z-]#)*=01;34=0=01'
zstyle ':completion:*:*:*:*:processes' command "ps -u $USERNAME -o pid,user,comm -w -w"
# disable named-directories autocompletion
zstyle ':completion:*:cd:*' tag-order local-directories directory-stack path-directories

# Use caching so that commands like apt and dpkg complete are useable
zstyle ':completion:*' use-cache yes
zstyle ':completion:*' cache-path $ZSH_CACHE_DIR

zstyle ':completion:*:*:*:users' ignored-patterns \
        adm amanda apache at avahi avahi-autoipd beaglidx bin cacti canna \
        clamav daemon dbus distcache dnsmasq dovecot fax ftp games gdm \
        gkrellmd gopher hacluster haldaemon halt hsqldb ident junkbust kdm \
        ldap lp mail mailman mailnull man messagebus  mldonkey mysql nagios \
        named netdump news nfsnobody nobody nscd ntp nut nx obsrun openvpn \
        operator pcap polkitd postfix postgres privoxy pulse pvm quagga radvd \
        rpc rpcuser rpm rtkit scard shutdown squid sshd statd svn sync tftp \
        usbmux uucp vcsa wwwrun xfs '_*'

if [[ ${COMPLETION_WAITING_DOTS:-false} != false ]]; then
  expand-or-complete-with-dots() {
    # use $COMPLETION_WAITING_DOTS either as toggle or as the sequence to show
    [[ $COMPLETION_WAITING_DOTS = true ]] && COMPLETION_WAITING_DOTS="%F{red}…%f"
    # turn off line wrapping and print prompt-expanded "dot" sequence
    printf '\e[?7l%s\e[?7h' "${(%)COMPLETION_WAITING_DOTS}"
    zle expand-or-complete
    zle redisplay
  }
  zle -N expand-or-complete-with-dots
  # Set the function as the default tab completion widget
  bindkey -M emacs "^I" expand-or-complete-with-dots
  bindkey -M viins "^I" expand-or-complete-with-dots
  bindkey -M vicmd "^I" expand-or-complete-with-dots
fi

# automatically load bash completion functions
autoload -U +X bashcompinit && bashcompinit

That’s a long section, but in a nutshell this lets me type one character, then hit tab, and be offered a menu of all the possible completions of that character. It is case-insensitive, so b would match both boring.txt and Baseball.txt. I can continue to hit tab to cycle through the options, and hit enter when I’ve found the one I want.

The last section sources a few other files:

[ -f ~/.fzf.zsh ] && source ~/.fzf.zsh
[ -f "/Users/jonathanbuys/.ghcup/env" ] && source "/Users/jonathanbuys/.ghcup/env" # ghcup-env
[ -s "/Users/jonathanbuys/.bun/_bun" ] && source "/Users/jonathanbuys/.bun/_bun"
source /Users/jonathanbuys/Unix/src/zsh-autosuggestions/zsh-autosuggestions.zsh
source /Users/jonathanbuys/Unix/src/zsh-syntax-highlighting/zsh-syntax-highlighting.zsh

If I’m experimenting with Haskell, I’d like to load the ghcup-env variables. If I have bun installed (a way, way faster npm), than use that. The final two sources are for even more enhanced autosuggestions and command line syntax highlighting. So, typos or commands that don’t exist will be red, good commands where zsh can find the executable will be green. The autosuggestions take commands from my history and suggest them, I can type right-arrow to accept the suggestion, or keep typing to ignore it.

Taken together, I’ve been able to remove Oh My Zsh, but keep all of the functionality. My shell configuration is constantly evolving as I find ways to make things faster and more efficient. I don’t consider myself a command line zealot, but I do appreciate how this setup gets out of my way and helps me work as fast as I can think.

p.s. A lot of this configuration was taken from other sources shared around the internet, as well as the zsh documentation. I regret that I haven’t kept references to the origins of some of these configs. If I can find the links I’ll post them here.

I’ve been trying to wrap my small monkey brain around what ChatGPT will mean in the long run. I’m going to try to think this through here. In many ways the advances we’ve seen in AI this past year perpetuate the automation trend that’s existed since… well, since humans started creating technology. I’ve seen arguments that seem to be on two ends of a spectrum, that the AI is often wrong and unreliable, and we shouldn’t use it for anything important, to AI is so good that it’s going to put us all out of jobs. As with most truths, I think the reality is somewhere in between.

It’s my opinion that jobs that AI can replace, it probably will replace a lot of. But not all. Referring back to our discussion about the current state of Apple news sites, if the site is a content farm pumping out low-value articles for hit counts and views, I can see AI handling that. If your site is well thought out opinions and reviews about things around the Apple ecosystem, that I think will be safe. Because it’s the person’s opinion that gives the site value.

For more enterprise-y jobs, I could see fewer low and mid-level developers. Fewer managers, fewer secretaries, fewer creatives. Not all gone, but certainly less than before. If your job is to create stock photos and put together slide shows, you might want to expand your skill set a bit.

I think… the kind of jobs that will survive are the type that bring real value. The kind of value that can’t be replicated by a computer. Not just the generation of some text or code, but coming up with the why. What needs to be made, and why does it need to be made?

Maybe AI will help free us up to concentrate on solving really hard problems. Poverty, clean water, famine, climate change. Then again, maybe it’ll make things worse. I suppose in the end that’s up to us.

The Prairie City Monroe district school board invited state senator Ken Rozenboom to their board meeting Monday, January 16th to discuss his support of the “school choice” bill currently making its way through Iowa legislature. The superintendent started off asking some tough questions of the Senator, who is the chair of the education committee. As he puts it, “school choice is in his lap”. He’s worked on different forms of the bill for seven years.

A lot was said about numbers and funds, percentages and estimates. The superintendent fully expects that the bill will negatively impact the PCM school district, and that the funds to cover the loss will have to come out of school programs. The senator was unconvinced… or at least he couldn’t admit he was convinced. I think the super is right.

But…

None of that matters. None of the numbers or dollars or percentages or estimates actually matter. At one point the Senator said something along the lines of asking us if we knew what was happening in some of the public schools in the state, calling out Lynnville-Sully in particular, where he claimed middle school girls were forced to shower with transgender students¹. He also called out another school who spent a week on Black Lives Matter², with a look of pure disgust. He said that public schools brought this on themselves.

He brushed off any and all criticism as being union talking points, with a smug grin and wave of his hand.

After the meeting was over, a few of us met with him in the hall to ask a few more questions. Again he said that you’d have to have your head in the sand to not know what was going on in public schools “out there”. I raised my hand and said “my heads in the sand. Seriously, I don’t read the news, I have no idea what’s going on.” I was hoping that he would explain or give a few more examples, but somehow the conversation moved on. I suspect he had no concrete evedence to provide.

What I got from the Senator was that it’s not really about kids, it’s not about education, it’s about the right-wing culture war. That’s what he said, but also what he couldn’t really admit. He said it’s not the republicans, it’s the left, it’s the teachers who are shoving this progressive ideology down our throats. He also said that the median household income across the state of Iowa is $61,000, he followed that up by saying that the median teacher income in Iowa is $61,000. He asked me if I knew how many teachers there were out there making $100,000 a year. I asked him if he knew how many there were in PCM, and said zero.

I was flabbergasted at the number of times the Senator would say something, and immediately claim that he didn’t say it.

In the Senators mind, schools are funded appropriately, or possibly overfunded. Public school teachers are overpaid and actively poisoning our kids minds. He said that the social contract has been irrevocably broken, that we can’t put the genie back in the bottle. I’m guessing what he means by that and the “school choice” program is that public schools should be relegated to serving only those whom the private schools turn away. Student in need, students with disabilities or behaviors, students with non-Christian parents. Creating an evermore stratified society of haves and have-nots.

We ended the night with me trying to explain to the Senator that what I heard was that instead of investing in our public schools, we are taking funds away from the schools. Even though we spent the past hour talking about exactly that, the Senator claimed that wasn’t at all accurate. We shook hands and parted ways.

On my way towards completing another project I needed to setup gpg public key infrastructure. There are many tutorials and explanations about gpg on the web, so I won’t try to explain what it is here. My goal is to simply record how I went about setting it up for myself to securely sign my Git commits.

Most everything here I gathered from this tutorial on dev.to, but since I’m sure I’ll never be able to find it again after today, I’m going to document it here.

First, install gpg with Homebrew:

brew install gpg

Next, generate a new Ed25519 key:

gpg --full-generate-key --expert

We pick option (9) for the first prompt, Elliptic Curve Cryptography, and option (1) for the second, Curve 25519. Pick the defaults for the rest of the prompts, giving the key a descriptive name.

Once finished you should be able to see your key by running:

gpg --list-keys --keyid-format short

The tutorial recommends using a second subkey generated from the first key to actually do the signing. So, we edit the master key by running:

gpg --expert --edit-key XXXXXXX

Replacing XXXXX with the ID of your newly generated key. Once in the gpg command line, enter addkey, and again select ECC and Curve 25519 for the options. Finally, enter save to save the key and exit the command line.

Now when we run gpg --list-keys --keyid-format short we should be able to see a second key listed with the designation [S] after it. The ID will look similar to this:

sub   ed25519/599D272D 2021-01-02 [S]

We will need the part after ed25519/, in this case 599D272D. Add that to your global Git configuration file by running:

git config --global user.signingkey 599D272D

If you’d like git to sign every commit, you can add this to your config file:

git config --global commit.gpgsign true

Otherwise, pass the -S flag to your git command to sign individual commits. I’d never remember to do that, so I just sign all of them.

Make sure that gpg is unlocked and ready to use by running:

echo "test"  | gpg --clearsign

If that fails, run export GPG_TTY=$(tty) and try again. You should be prompted to unlock GPG with the passphrase set during creation of the key. Enter the export command in your ~/.zshrc to fix this issue.

Finally, Github has a simple way to add gpg keys, but first we’ll need to export the public key:

gpg --armor --export 599D272D

Copy the entire output of that command and enter it into the Github console under Settings, “SSH and GPG keys”, and click on “New GPG key”. Once that’s finished, you should start seeing nice green “Verified” icons next to your commits.