Inspiration:

I started writing this after looking at some GitHub API docs, and not understanding how to read files from a private repo. I ended up looking around, realizing the answer after looking at a GitHub gist, and thought I was an idiot for not realizing it sooner. But I realized that even if I was being an idiot, I had read the page and the OAuth page, and knew they needed to fit together somehow: all I needed was an example.

Do we really need examples?

When writing documentation for our own programs, we're biased. Very biased. Not only do we know how to write code or use our software efficiently, but we also know exactly how it works the way it does and the processes it takes.

We sometimes even assume that our software is meant to be used in a specific way and will never be used in inefficient/impractical method X. Why? Because X is insecure, or X would never happen if documentation was followed, or there are more efficient tools than the one we've made, that are actually built for doing X.

But that's not always the case.

Sometimes, or more often, almost always, the user won't be reading all of the documentation. Maybe barely any of it. Or maybe the docs you wrote were bad, or incomplete, or assumed the user knew things they didn't.

That's what examples are for

Examples at their most basic level are beginner friendly. For beginners, writing a program, and walking through it, explaining as you go, is more important and valuable than providing each function call along with a detailed desicription of what it does.

We all learn by being given examples. Like babies learn from hearing adults talk, we learn how to use a language, library, or framework, from seing well-structured, thought-out examples and their given or implied use cases. And babies babble before they learn to talk, just like we understand small bits of code before we understand a program as a whole. But sentence structure, proper word usage, etc., take much longer.

It's clear that the solution is more frequent and more thorough examples. You don't need to give five examples for every new concept you introduce. But you could.

As an example...

Since I'm writing about giving examples, it's only fair that I give an example on how to give examples. I started out learning JS, and no other resource was as consistently thorough and complete in its documentation as the Mozilla Developer Network.

Consider for example, the ternary operator. It's simple enough, if you're used it multiple times. It's just an if/else statement with a different return value on true/false. It's dead simple, right? Not to all beginners. Ask a friend who's just started programming if they can understand it and use it properly just by you explaining it to them. Chances are, they won't.

If you look at the MDN docs on it though, it looks like 2/3 of the relevant words are examples alone. And half of the other words are explaining exactly what the examples are doing. This is great documentation, and it's not because of just the description of the functional requirements, because you probably included something along those lines when you explained it. It's the examples, the walkthrough of what the example does, the thoroughness of the examples, and the sheer number of them, which helps them cover so many use cases.

What are they?

So you're going to a hackathon. You've signed up, and you don't know quite what it is, but you're interested, maybe because you've heard good things about them or they sounded like fun. They're all pretty similar in concept, but when I think of hackathons, I think of coroporate hackathons and collegiate hackathons.

Corporate hackathons

Corporate hackathons are generally smaller than collegiate hackathons. Both in size, and in time spent working. That's not to say they're bad, some of the ones I've been to have been better than some of the collegiate hackathons that I've been to, despite their size. These hackathons will generally be 50 people or less, and all the ones I've been to have been less than 24 hours, and they've also had participants go home for the night and come back in the morning. They also usually have no problem with providing enough food and snacks.

They're also much different from collegiate events in that professionals are allowed to these events, so you'll most likely end up working with people from companies with local branches. If you're around the Dallas area, the AT&T Foundry usually hosts a few of these every year. There are also larger corporate hackathons, which I can't say too much about, as I haven't been to more than a couple with much more than 50 people.

Collegiate hackathons

Collegiate hackathons usually only allow current students to participate in them. If you're in high school, they might require you to be above 18 as well. They usually give out more swag (read: goodies, so be prepared to carry lots back), and are generally either 24 hours or 48 hours. If you want to sleep, you'll generally sleep on the floor. They usually have lots of sponsors, lots of prizes, and generally are much larger events overall.

Another difference is that while corporate hackathons usually have a theme (IoT, mobile apps, big data, etc.), collegiate hackathons usually don't. They're also almost always fully organized by college students, which is pretty impressive considering the huge amount of money that goes into them.

The rest of this post is about collegiate hackathons, but some of it can be applied to corporate hackathons as well.

What to bring?

I'm a hardware hacker, so I usually bring my own hardware, which I've written about more over here. I'll also bring a day's worth of clothes, my phone, wallet, device chargers, plane tickets, headphones (nice for listening on long flights), deodorant, shampoo, and toothpaste/toothbrush, all in one duffel bag. I'll also bring a backpack with my hardware kit, laptop, and free space for free stuff. Feel free to bring resumes, as well, recruiters are there for a reason. It's also a good idea to bring a blanket, or clothes warm enough comfortably to sleep in. I usually go with jeans/sweatpants and a hoodie. (Side note: bring and use those toiletries. After a couple days without them you're going to reek and that's no fun. Also, shoutout to hackathons with showers, y'all are the best)

What to do?

Before you go

So, from the start, book your plane tickets early, arrive early (get there early and you get good spots), and make sure you have all your stuff packed the night before. It also helps to have a team and idea of what you're going to work on before you get to the event. If you don't, that's fine too, there are almost always team-building events beforehand to help people who don't have teammates connect and form a team.

When you get there

Talk to the recruiters, don't be afraid to ask questions or take the free stuff they're giving out, a lot of the goodies they give out will be gone quickly. If you're interested in a company, you can take time to talk to their recruiter, and give out a copy of your resume. For large hackathons I usually spend almost an hour at the beginning for just going through the booths, looking around, asking questions, and talking. Then, after the opening ceremony, it's time to start working on your idea. At most collegiate hackathons, hardware checkout will open after the opening ceremony. If you want a big-ticket item to borrow from the hardware checkout, it's best to get there early before it gets taken.

While working on your hack, be sure to grab snacks and take some short breaks. Hackathons are usually very well stocked as far as snacks go, and they'll often have events like smash tournaments, either set up by the hackathon organizers themselves or sponsoring companies. And while some people like to brag about how much sleep they didn't get, getting a few hours of sleep can definitely help refresh you and give you a new mindset when tackling coding errors. On that note, don't be afraid to ask for mentors if you're having trouble.

In the final few hours before presenting, you'll usually want to make sure that you upload your submission to whatever website the organizers are using, as well as make some finishing touches. You'll usually be presenting expo-style, where you have an assigned table and judges come and walk through and look at everyone's project like they would at a science fair. You'll also usually have to pack up all of your stuff before you go to your table, and after presenting, head to the closing ceremony.

Where to find them?

That's all great, but where do you find out about hackathons? Look no further, hackalist is my personal favorite for finding out about new hackathons I mean, you can sort by travel reimbursement, and it has them in order by month. This really only includes collegiate hackathons, though. If you want to find out about corporate hackathons, the best way I've found is by looking through EventBrite, asking around, or Googling.

What you'll want:

Basics:

Here are some things that it'd be hard to really make a hardware project without:

• Dev Board (an Arduino, Raspberry Pi, Photon, NodeMCU, etc.)
• Cable (whichever your board needs)
• Breadboard
• Jumper wires (optionally include alligator clips)
• Resistors

Tools:

Some things that'll make working with hardware a lot easier:

• Needle nose pliers (with built in wire cutters)
• Screwdrivers
• Superglue
• Electric tape
• Multimeter (this might be too bulky, I don't always bring one)

Input and Output:

You can't do much with just the list of above items. You'll need some of the following, depending on what you're working on, to get a solid project.

Small components:

These are smaller, more basic components, which aren't mounted on a board of their own.

• LEDs
• Photoresistors
• Reflective IR
• IR Transmitter/Reciever LEDs
• Transistors
• Capacitors
• Piezos
• Humidity sensor
• Temperature sensor
• Hall sensors
• Vibration Motors
• Tilt/vibration sensors
• DC Motors
• Push button switches
• Flex sensors (a bit more expensive, I'd only get one)
• Force sensitive resistors (same as above)

Larger Components:

• Servos
• Pulse sensor
• Ultrasonic range sensors
• Gyro/Accelerometer (usually have both on the same breakout board)
• LCD screen (preferably with I2C for less wires)
• Bluetooth module (you'll need one with BLE support for iPhones)

Additional Power:

For powering your board or components that require more voltage

• Relay module
• Breadboard psu
• 9v Batteries (not for your dev board, for powering a motor that needs more than it can handle. Use with transistor to switch on/off)
• Rechargeable power bank (for when you want to power your dev board without having it hooked to your laptop)

Where to get them?

So now that you know what to get, where do you get it? If you're looking for reliability, use Amazon, Adafruit, or my favorite of the three (because of their nice guides and docs), Sparkfun. If you're looking to save money (and it does cost a decent amount), check out Aliexpress for components, and your local dollar tree for superglue/electric tape/pliers/etc.

Which board to use?

So, each board is useful for different things. For this, I'm just going to be comparing the three most popular: Arduino UNO, Raspberry Pi 2, and Particle Photon.

There are a few factors that I consider when picking a board:

• Setup time
• Usage
• Language

Setup Time:

• A Raspi has a decent setup time. You need to be able to set it up to ssh into it, which means you'll need an OS on an SD card, the necessary cables, and a monitor/keyboard/mouse to be able to use it. If you've already set it up before, though, you can just bring it with an ethernet cable (you'll need an ethernet to thunderbolt adapter for macs though). If you're renting one out, though, that's not really an option.
• An Uno is generally plug and play, so setup time is nil
• Photons are pretty nice, if you aren't in a saturated wireless airspace. The problem is, a lot of hackathons are. There's a solution to this, though, which is using serial setup. Particle uses the node-serial package though, so if that gets a bit funky (like it did with the El Capitan update on OS X), then you won't be having any fun.

Basically, for setup time, Uno > Photon > Raspi.

Usage:

• A Raspi can do a lot of stuff, should be able to handle a lot, but it has some trouble with analog input. You can solve that with an analog to digital converter, or by having an arduino as a go-between. I'd use Raspi for anything where you want a small, local server, where you can directly interface with hardware. However, in a hackathon setup, an Uno or Photon can do almost everything, if not more, with the right libraries, so it generally comes down to preference.
• An Uno can do everything a Photon can do, but it lacks wireless connectivity. However, you can use a node package such as node-serial or johnny-five (the latter also uses node-serial, but check it out as well) to interface with your computer while it's connected over USB, which opens up so many options (and also makes the Uno my go-to board). This does, however, make your setup a bit more complicated, if you're using it for the first time.
• A Photon can do everything an arduino can do, but with less ports, less libraries, and less guides. It makes up for that, though, with its incredibly easy setup (when it works), and Particle's servers, which you can use as a limited backend. It also has a smaller form factor, if that matters in your application.

Language:

• A Raspi is basically a full computer, soooo you can do pretty much anything. Most people perfer python on it, though.
• An Uno can be programmed with C/C++, however, if you use something like pySerial or node-serial, you can write python or javascript programs to communicate with the arduino on your computer, and even just use it as a data-collector if you want (which, out of the three devices, it's the best at). There's also johnny-five, if you want to use literally nothing but javascript (these options require the arduino to be tethered to the computer via USB for them to work).
• A Photon can be programmed with C/C++, however, you can just send and recieve the data to and from Particle's servers, have all of the benefits of node-serial/pyserial, except just using REST and any server, so it doesn't even need to be local.

Some cool hardware projects!

If you're looking for inspiration, here are a few hardware projects to check out:

• Sesame, which lets you remotely unlock your door via a smartwatch voice command.
• A simple tetris game, amazingly small, powered with a LiPo battery, and controlled with push buttons.
• FifthSense, a tool to for blind people to send and recieve data. Uses buttons for input, vibration motors for output.

How does it work?

For measurement, we used a reflective IR LED. More detail on this later, but basically I'm using it here as a close-range distance measurement tool.

In order to measure a 3d surface, two other components come into play, each adding a separate dimension to the scan. The first is a vertical shaft which raises and lowers the reflective IR LED. The second is a DC motor, which rotates the scanned object, allowing the scanner to measure its sides.

Scan accuracy

You're probably wondering how accurate this project's scans actually were. It's literally a ten-cent RIR LED on a stick, held together by cardboard.

I wondered that too, as we were building it. I hadn't seen examples of reflective IR LEDs being used for measuring distance before. And even though it showed promising results when I tested it before starting, I couldn't be sure how accurate it was, given that the data was definitely not a linear function of distance.

A scan of a paper cone

The same paper cone, but a judge we were demoing to wanted to wave his hand between the cone and the scanner for a bit while it was running.

A scan of a slightly deformed paper cylinder.

Some example scans generated when plotting the raw data through Wolfram

As it turns out, the scans were pretty darn accurate. In fact they were so clean, we had a judge doubt that the data was live. He waved his hand in between to see, and you can see the results above (middle picture, compared to the original on left).

Is it really under $20?

Yup. I source most of my electronics from Aliexpress, so that helps. Here's the breakdown:

• Reflective IR LED - 10 cents
• DC motor - $3
• Breadboard - $3
• Continuous rotation servo - $7
• Arduino clone - $3 to $10
• USB A Male <-> USB B Male adapter - Free (usually included with the Arduino, even cheap ones)
• Plastic gears - $3 for a bag, probably $0.20 for parts used
• Cardboard - Free
• Electrical tape, wires, resistors - around $2 worth of parts

And this isn't even pricing at scale, or with efficient use of parts (e.g. using a PCB instead of breadboard + wires).

Ideation process

Akhilesh and I flew into Philadelphia early, so we went to a Public Library to kill time before the hackathon started. He goes to the bathroom, and comes out wondering how hard it'd be to scan his poop. I look at my kit and see if we have the parts. And here we are.

Prototype process

After some fairly basic tests, I decided that a Reflective IR LED was probably our best bet as a scanner. Our alternative was an ultrasonic rangefinder, and I didn't think it'd have enough granularity at low distances to work as a close-range 3d scanner.

Next, we needed to move the object or the scanner such that the entire object could be scanned. We focused on the vertical dimension first. I tried using a pulley-like system with a DC motor, and of course without a smooth pulley it'd occasionally slow or snag, and we couldn't get a consistent speed.

Looking for an alternative, I looked in my tackle box of hardware and found a rack to use in a rack and pinion gearing, and that actually worked perfectly, once we secured it into place. I attached the RIR LED, and also put a cylinder of electrical tape on it to try and make it pick up less noise, which actually worked pretty well.

Then we tried making a flat base and attaching it to a dc motor. We first tried taping a few pieces of cardboard together and using the hole in a zip tie to attach it to the dc motor and hold it in place. It didn't work; it ended up too wobbly to actually get any decent measurements on, though.

We were lucky, though: the dc motor I used had basically the exact same diameter as one of the gears in the gear set, so we were able to tape cardboard circles to the gear in order to provide a larger base, and this gave us less wobble than our earlier design.

Once we got this working, we also needed to hold the rack and pinion together using a bunch of zip ties, cardboard, and duct tape. The motor also broke apart a couple times, requiring the metal inside to be repositioned. And of course, we needed to calibrate everything, to get near-usable results!

Calibration process

In this process, we figured out a function to convert the Reflective IR LED's raw analog output to units of distance. We also figured out the function to get vertical distance (vertical scanner movement) and degrees of rotation (horizontal base rotation) as a function of time.

Figuring out position of the z-axis was simple, we just measured the distance/time to get the speed of the motor when moving vertically, which given the initial value of 0 (measuring starting at the object's base), gives the z position as a function of time.

To find the angle of rotation, I measured the length of time it took for 10 rotations (to reduce error from starting/stopping time) and divided by 10.

To find the distance from the object to the scanner, we needed to fit the data given by the reflective IR LEDs. And for that we needed some data points to get a curve best fit. Using Chrome dev tools as a substitute for a ruler, I recorded the scanner's analog output values, while moving a piece of cardboard away, centimeter by centimeter.

With this data, we were able to estimate the function best fit using Wolfram alpha:

$$$ f(x) = -600 + \frac{19169 \times x}{30} + \frac{30433 x^2}{360} - \frac{1751 x^3}{16} + \frac{4009 x^4}{144} - \frac{727 x^5}{240} + \frac{89 x^6}{720} $$$

Glad we didn't have to calculate that by hand. Unfortunately Wolfram didn't give us the inverse, which we needed (in a scan, we have raw analog values and want estimated distance). So I just plugged in the range of input values, which is a limited range of ints (0-2048), and made a lookup table instead.

Combining these functions, we were able to determine y/z coordinates given distance and time. (Remember, our scanner rotates and moves up at the same time, so it's moving in a spiral shape.)

And with this, we could generate point clouds (seen above in earlier sections)! From this, you can estimate the shape of the object, and generate a STL file to send to your 3d printer, in order to print the object you just scanned!

Want to build it yourself?

Caveats

Want to build this on your own? It's very cool, and very cheap! But if you're planning on using it for anything serious, note a couple caveats:

You'll probably need 3d-printed parts for it to be calibrated well. (Though it's fun project to make by hand though, and the experience and results were pretty rewarding.) It's also not that great at scanning anything too glossy (shiny plastic/metal).

Details you might find useful

All data is passed via USB from the Arduino to the laptop, where all of the processing is done in Node. It's much faster that way, and you can do whatever you want with it from there.

Data is sent from the Node instance to the Wolfram notebook through.... the wondrous OS X clipboard (last minute features, what can I say). It takes a long time for Wolfram Notebook to graph the data, but I think it's just slow frontend javascript, not optimized for large datasets. It might be faster on other software.

I don't have any good pictures of the circuit, but the only complicated part is wiring the Reflective IR LEDs, which you can do like this:

The circuit we used was something like this one

Thankfully no signal amplifiers are needed (I didn't have any), and the wiring for the servos and dc motor are also pretty standard. Have fun!

What we used

• Frontend:
  • React
  • Babel
  • SASS
• Backend:
  • Go
  • gin-gonic/gin
  • tarm/serial
• Hardware:
  • Electronics:
    • Arduino Uno x2
    • White LED x12
    • 1k Resistors
  • Materials:
    • Wire (lots)
    • Paper clips (brass plated)
    • Paper plates
    • Cardboard
    • Springs
    • Foam

There's not a whole lot of complicated electronics here, and realistically you could probably use one arduino mega instead of two unos. Most of the time I spent on this was painstakingly soldering everything together, placing all of the layers down precisely, hardware debugging (encountered the weirdest hardware bug I've seen yet), and figuring how to orient everything properly (springs, clips, wires). The backend code itself was written pretty last minute, which looking back on it, could have gone pretty badly, but thankfully it ended up working out.

How we made it

I worked on the hardware and backend, and had very little to do with the frontend, which Benny made, but here are the steps I took for the hardware:

First, it was important to have an idea for how the fretboard worked. There are a few ways to do a fretboard which immediately come to mind, but the way we finally implemented it was by far the best way to do it in my opinion. I have a lot of projects that I've done with reflective IR LEDs, so at first I thought of using those as a range finder since we could use those to find the range on small instruments (7cm from a uke or banjo head is a decent amount for demo purposes) or to use a system with force or light sensitive resistors under each fret, but these methods all have issues with emulating how a real string instrument would have its audio affected.

I ended up going with a system with much less complicated components. Simply using 4 strings with 5v out and different resistances on each string (not grounded), and having each metal fret piece connected to an analog input pin on the arduino means that you get lifelike and accurate strings. Based on which analog pin you're reading from, you can figure out the fret number, and based on what the voltage reading is, you can figure out which combination of strings is being pressed. It's also very accurate in how it works compared to a real string instrument because the way this reads string/fret combos is by the string being touched to the fret, which (aside from a few techniques), is how notes are normally played on string instruments.

The wiring for the neck—each string has a different resistance, and each fret measures voltage independently, allowing multiple string-fret combos to be read without conflict.

So after hashing out how I wanted the frets to work (since without the frets, there's not a whole lot you can do) I moved on to making the strings for detecting strumming. I first tried reading analog input on all of the strings and putting 5v through a wristband on my hand, but it's much easier to reverse that and put 5v through all of the strings and ground my hand, since my hand isn't very conductive. This is pretty similar to how a makey-makey works, actually. I used a 1k pulldown resistor and analog input, since I was going to need two arduinos for this project anyways, but you could change the resistor value and use digital pins instead.

The wiring for the strumming side. This one's a bit more simple, it just checks if I've touched a string.

The fretboard was made by first hollowing out the space for the LEDs on two layers of cardboard, inserting them, and placing a cut sheet of paper plate on top (Benny did this), then the LEDs had to be wired up, cut, soldered, covered with electrical tape, and the next layer (frets) had to be added, cut, soldered, wired, then the strings on top of those had to be added, cut and soldered and wired to the arduino as well. I'm not the quickest, so at one point I spent 4 hrs straight just cutting, taping, and soldering wires.

It took hours to solder LEDs, layer frets, and connect it all to the arduino without breaking any wiring.

During this time, I was also testing the wires to make sure I didn't miswire something before soldering. At one point I taped all of the thin LED wires (that took up less space on the fretboard) to the thicker ones that would go inside the arduino pins after twisting them together, and had to untape, solder, and tape again, since they would lose connections at some angles after being taped. So there were a few other things that took up some time.

The code

I didn't write any of the frontend code, but basically how it works is the frontend continuously sends a GET request to the server to get which strings/frets to highlight on the webapp, and sends a POST request telling the server what LEDs to light up when it wants to light up the fretboard. This allows you to do things like select different chords and have them displayed on the banjo in realtime, as well as showing the tab for the chord on the screen. There was also a feature for playing songs and having the webapp show the user which chords and lyrics to use in realtime.

Benny's design and frontend code were nice and polished; the backend on the other hand, was incredibly rushed. The arduino files were written as I went and weren't too bad since I had to write them to test the code. The server itself however, was written pretty quickly, and I would reccomend restructuring large amounts of it if you're going to actually use it. It basically opens up one serial connection to the fretboard uke, one to the string uke, and starts reading in data and parsing it as JSON. The server's running at the same time in a goroutine and basically just serves up the last read data, formatted properly into JSON for GET requests, and for POST requests, simply writes the correct LED to light up to the fretboard arduino over serial.

The wonky part is the math used to calculate which frets are being played. It was getting pretty close to presentations, so I used some hackathon quality logic to go with my hackathon quality code, and didn't bother using math to calculate which string combination would create which analog value on the fretboard. I ended up just getting a few combinations of frets (e.g. 0, 1, 2, 3, 01, 12, 23, 012, 123, 0123), manually pressing those down, getting an approximate value, and giving it a short range around that, and just checking which range a fret read in, and seeing if it was in one of those ranges. As a consequence, since I forgot 03 as a combo, holding down the top and bottom strings on a single fret worked, but I don't think we even thought to test that, it was more like testing the first fret on the first string and the third fret on the fourth string, so it worked out all right.

The results

We presented pretty well to the judges prior to the closing ceremony, and HackPrinceton actually had a private judging prior to the closing ceremony, so the results were decided before we went on stage, and we did well in that. For the closing ceremony though, we had 2 minutes to present, went on stage, immediately tangled wires with the hdmi, usb, etc., wires on stage, took about a minute to plug in the hdmi, and fumbled through that presentation so badly that if we were judged based off of that, we wouldn't be finalists. We ended up doing pretty well though, Benny won us the best design prize, I won us the best DIY prize, and together we got best overall. A great ending to HackPrinceton part 2: Electric Boogaloo.

The weirdest bug ever

So while I was working on wiring the fretboard, I was also testing the wires to make sure I didn't have to undo the whole thing after I was done and realized I made an error. After I got to the stage where I put the LEDs on, connected them to a single ground, wired the positive ends up to individual thin wires into individual thin wires into the digital ports on the fretboard arduino, I moved on to the frets themselves. I placed a layer of electrical tape on the wire, and added the paper clips, which went on the front of the neck and wrapped around to the back, and actually looked pretty solid as frets. When reading input from the frets, however, it turned out that all three frets gave nearly the same reading... (nearly the same reading as opposed to exactly the same reading, I assume because of the built in delay on analogread).

After some lots of hardware debugging, it turned out that the three frets were grounded together... through the wire that grounded the LEDs. That wouldn't be so crazy, except each fret had a layer of electrical tape covering it entirely from the LEDs and their wires. Adding two more layers of tape did the trick, so I can only assume that a combination of the tape being on the thinner side, being stretched, and firmly pressed (the paper clips held on pretty tightly to the neck) caused a leak.

Future Ideas

Here are a few ideas that I had thought of while making this project. Keep in mind that these are untested, and I'm just a CS major who does hardware at hackathons, so these might not be 100% accurate in implementation. These were ideas I discarded because I knew we didn't have the materials/time, and they were plans for muting the strings and using springs for analog strums.

Muting strings is a pretty big thing in playing guitar, and probably also in banjo playing, and it's where you put your finger on top of a string and press gently but not hard enough to touch the string to the fret. As a result, you dampen the strings by causing almost all of the vibrations to travel through you. This would be checked for by checking the voltage that comes through the other side of the string. You could probably check for this by grounding yourself, and also reading in voltage from the end of the strings on the neck of the banjo. Then the strings you were touching would have different values than the ones that didn't. You'd also be grounded if you were touching the strings to push them down to the frets, but you'd know that they were pushed to the frets, and there wouldn't be any scenarios where they'd be pushed to a fret but not touched.

For using analog strums (e.g. to know how loud to play an mp3 based on the strength of the strumming), using something similar to how an electric guitar works would probably be the best method. Placing pickups under strings would allow you to see if the string was being pulled up or down, and how much, based on the motion of the magnetic field generated, and the resulting current that would go to the amp (or in this case the voltage-reading analog inputs on the arduino, since v = ir). You could alternatively attach springs to the very end of the strings, and use a hall sensor to determine the change in magnetic field caused by the spring's expansion and contraction. Again, these are untested, so they may not work exactly how I describe.

Misdiagnosing the root issue

To decide what I "needed" to build, I started by comparing workflows: I looked at the way I wanted to write, and contrasted it to the way I did write.

My previous workflow had me spend a lot of time in my text editor. Which I thought was bad for writing, because when I'd start writing, I'd try live previewing my blog, and only an hour later realize I was knee deep in minor CSS changes.

So I thought it was the text editor that made it easy to get off-track. I wanted a simpler workflow. And I was actually so fed up with my previous workflow, I wanted to cut my laptop out entirely. I wanted to write and publish posts just using my phone.

The two-cable SSH

Here's how to do it with a macOS host:

1. Start with a copy of Raspbian on your SD card. You can install the Raspberry Pi imager, which makes it a painless process.
2. Re-insert the SD Card, then create an empty file named "ssh" into its boot partition. This should be the partition that auto-mounts, so you can simply cd into it, then touch ssh.
3. Insert the SD card into your Pi, and connect the ethernet and microUSB cables to your laptop.
4. Open System Preferences and go to the Sharing tab. Enable internet sharing through your ethernet port. If you're using an adapter, your ethernet port may be called by the adapter name, such as "USB 10/100 LAN".
5. Open System Preferences, and go to the Network tab. Enable DHCP or DHCP with manual address on your ethernet port. Wait for it to get connected.
6. In your terminal, run ifconfig, then find your bridge interface, bridge100 in my case, and get its network address under inet.
7. Usually, you can just assume the Pi's IP address will be allocated to the next IP bridge interface's IP + 1. In my case, this is 192.168.2.1 + 1, so 192.168.2.2.
8. Or to be really sure, you can find the Pi's IP by scanning the range of IP addresses in your subnet.
   • In my case, my bridge interface address is 192.168.2.1 and mask is 0xffffff00, which is a /24 block.
   • So sudo nmap -sS -p 22 192.168.2.1/24 will list out all devices under this block.
9. Simply SSH in! In my case, ssh pi@192.168.2.2, with default password raspberry.

Permanent tasks cause fatigue

Before starting Tandem, my manager Tim previously co-founded Astrid, a personal task manager, which he later sold to Yahoo.

At Astrid, Tim tried an interesting re-engagement email. Users who had been inactive for 6 weeks were asked: "want to delete all your tasks and start fresh?"

The hypothesis was that these inactive users had tasks that overwhelmed them. Either the amount, the complexity, or the low importance of backlogged tasks, caused users to put tasks off. So every time these users visited Astrid, they saw tasks they couldn't do, felt bad, and eventually stopped visiting—procrastination in action. Astrid's re-engagement email offered the users a chance to start fresh.

It was impressively compelling. Most re-engagement emails get a 1% response rate. This email had about 5%.

I imagine those users had the same feeling you get when you move from one browser to another, or from one task manager to another, and leave your bookmarks, open tabs, backlogged tasks, to start fresh in the name of increased productivity.

But simply deleting the backlog doesn't end the cycle.

Nowadays, Tim uses a simpler task manager—a notebook. He uses a page a day, adds some tasks from JIRA, and when tasks come in during the day, he writes them down and works on them when he's available. These are mostly small tasks, things that are easy to forget. If he gets a bug (important to track) or a task he can delegate, he puts it in JIRA.

The notebook isn't meant to store issues forever. The pages get turned, and previous information becomes less accessible. So the next day, if he needs to carry over something that hasn't been completed, he needs to make the conscious decision to do so.

This solution solves task-management fatigue. And incidentally, it solves another problem:

Not all tasks are created the same

There is a gap in task management. When we look at the tools used for tracking tasks and goals, there's something for every level of the company:

Except at the personal day-level, where you're working on tickets, getting requests for feature updates from your PM, bug discoveries from QA, and breaking down your actual tickets into different tasks, which may require subtasks that weren't anticipated.

One proposal is that you should track all of these tasks, no matter how small, in JIRA. But the teams I see successful at implementing this are those that can afford to work at a slower, more "enterprise" pace. Which isn't a problem—at least for engineering—but it's often not an option at smaller startups, when learning cycles must be fast.

Even if you work at a larger company, at a slower pace, you may relate when it comes to personal life: tasks come from a variety of directions, at all times. You can't toss every minor task in a task manager.

You can't fix every task before the next one comes in. If you only use JIRA, you choose between possibly forgetting these small tasks, or breaking your train of thought to search, categorize, and fill out a form every time you get a small task.

On the flip side, quickly writing these down in a temporary place means that you don't need to:

• Spend mental effort remembering minor tasks,
• or take significant time to file minor issues.
• Have your task manager filled with minor issues you've forgotten to close,
• or leave hastily-written issues for later, when you forget what the vaguely titled issue is for.

Correcting for the mistakes

The solution is not complicated: use a temporary, personal task manager.

Don't go out and buy something new. You already have something you can use at home. Tim's notebook is a good example. I personally message myself on Slack (and iMessage, for personal use).

An example of how I Slack myself

Using Slack for daily personal tasks

When I'm working, and get a non-urgent task that would otherwise interrupt me, it goes in my Slack DMs. This is what it looks like:

Some days are quiet

And some are hectic

In the morning, to re-build yesterday's context, I just look at yesterday's list. If there are unfinished items, I can carry them over. But it's a conscious decision. I don't need to carry over optional, low-priority tasks.

And if you've got a bad memory, there's a huge benefit for team stand-ups. I used to write JQL filters to search JIRA (doesn't work—and misses tasks), now I just take a look at Slack.

The important requirements are just that your task manager:

1. Requires effort to carry over old tasks.
2. Makes writing down tasks quick.
3. Isn't used for notes (anecdotally, this adds noise, and forces you to parse notes from tasks).

If a task is critical, long term, or requires cooperation, file it in your normal issue tracker. Otherwise, it's probably a daily, personal task, and you can file it in here.

So take out a notebook, open Slack, pick what works for you. And see if you don't feel calmer and more in control of your tasks.

Relevant tools

I think these tools have interesting ideas to help individuals manage tasks.

Amna

Ravi, my old college roommate and friend since high school, left Microsoft to build Amna, a personal task manager with tight browser integration.

I resonate with the thesis: recently I switched away from Firefox Tree Style Tabs, because I noticed that by organizing my tabs (which were part of tasks), it inadvertently became a de-facto task manager. But it wasn't built for that!

Amna solves that exact problem: each task you do spawns a browser that keeps all of your relevant tabs.

All the tabs in this browser window are related to the same task, and Amna keeps their state when I close the browser.

After I'm done with a task, I can go back and look at the tabs I had open.

It helps prevent:

• Muscle-memory opening HN, clicking an interesting blog post, and leaving it open "to read later" for several weeks
• The pain of trawling through bookmarks and browser history, when you realize you forgot something
• The clutter of leaving tabs open to prevent forgetting something

If you've forgotten a time when you had less than five tabs open, try using Amna.

Jam

Two of my ex-Cloudflare friends, Dani and Irtefa, started Jam. It's probably the fastest way for PMs/designers to directly make comments on a website. So you get less minor interruptions, and more clarity on the requested change.

All you do is type jam on the page, click an element, and type the comment.

But I don't include Jam here just for that. Jam also allows you to offload the actual copy changes to your PM, which flat-out decreases your minor tasks. If this sounds exciting, get on the waitlist here.

A big thank you to Calvin French-Owen, Zack Bloom, Tim Su, and Tejas Manohar for their thoughtful feedback on earlier drafts of this post.

How does it work?

When you use a hotspot on iOS, your device treats your hotspot's network packets differently than the network data that apps use. Hotspot packets get routed differently, and they get metered. And if your hotspot plan isn't activated, your device won't even let you hotspot at all.

Connecting through my built-in iOS hotspot doesn't route traffic through my iOS VPN

In order to prevent that, this custom hotspot tunnels your laptop's network packets through an app (in this case, iSH). This way, your laptop's traffic gets treated as if it's just an app making network connections.

Here's a quick visual overview of the process:

After step 7, the custom hotspot is configured. Data from this hotspot is treated just like normal app data!

Setting it up

Here's how I got this working.

1. Download iSH, an interactive shell, to proxy connections through an iOS app
   • If you previously had an old version of iSH installed, you will need to manually enable its built in apk for package management. New downloads have it enabled by default.
2. Create a new adhoc wifi network from your laptop, connect to it from your phone, and get your phone's IP
   • you can quickly verify that the connection works using something simple like python3 -m http.server
3. Send your SSH public key from laptop → phone
   This step was harder without wifi. Click here to expand and see the quick-and-dirty way I did it.

   On my laptop, I ran cat publickey | base64 | pbcopy, then pasted the result into http://$PHONE_IP/$RESULT. This hit the server on my phone (run using python3 -m http.server), which showed it in the URL path of the server logs on my phone.
   Then on my phone, taking that result from phone logs, I copy → pasted it into a shell command, echo "[paste here]" | base64 -d > publickey.
4. Install sshd on iSH
   • Add your public key from the previous step to the /.ssh/authorized_keys file
   • Configure the ssh server settings. The changes I made were:

   Port 2222 # Custom port, 22 won't work
   ListenAddress 0.0.0.0 # Allow the laptop to connect via wifi
   PermitRootLogin prohibit-password # Allow root login
   PubkeyAuthentication yes # Allow public key auth
   AuthorizedKeysFile /.ssh/authorized_keys # Allow access from your public key
   PasswordAuthentication no # Disable passwords
   PermitEmptyPasswords no
   Compression no # CPU is a concern more than bandwidth, when we're doing this emulated + local network
   AllowTCPForwarding yes # Allow port forwarding
   GatewayPorts yes # Allow port forwarding
   UseDNS no # Not useful
   PermitTunnel yes # Allow our reverse tunnel
   

5. Test an SSH connection to the app from the laptop
   • You should be able to get your phone's IP from the settings page
   • SSH in from your laptop with ssh root@$PHONE_IP -p 2222

Now, on your laptop you should be able to run ssh -N -D 1080 root@$PHONE_IP -p 2222 to create a SOCKS5 proxy server on port 1080! You can open your browser (Firefox, Chrome), configure this as a proxy, and use this to route traffic!

If you only need to use your browser, you're good! But you might also want to use other tools, and many of these tools don't respect system proxy settings. For that, we can use proxychains.

Using Proxychains

Proxychains and proxychains-ng are tools that allow you to use a proxy with software that isn't configured for it.

Since I'm on macOS, and proxychains-ng has better support, I used it here. Here's the setup:

1. Install with brew install proxychains-ng
2. Edit the file /usr/local/etc/proxychains.conf to use socks5 127.0.0.1 1080 as the proxy
3. Run the program you want to use with proxychains4 program_name [arguments]

Limitations

Though this is a great backup internet solution, it's not as good as a normal carrier-enabled hotspot. Here's a little more depth on what its drawbacks are and why they occur.

Desktop limitations

On an unrooted iPhone, you can't modify iptables or truly run as root, so setting up a VPN server on the phone and connecting to it on your laptop doesn't work. You'll have to route the traffic through SSH or another protocol whose server can be run in iSH.

I spent a couple hours, but for whatever reason, I couldn't get sshuttle working. Proxychains works fine, but it means you need to manually launch every program you'd like proxied, and UDP traffic isn't supported, just TCP.

Note that you may be able to work around these limitations on Linux, where you have more control over the network stack than on macOS.

App limitations

There are a couple of limitations to this method from the app side, as well. Though they're both bearable as-is.

1. Backgrounding

When your phone locks, or you switch apps, iSH will get backgrounded, and your connection to the server will be lost.

To allow iSH to run into the background, you can run cat/dev/location > dev/null &, which polls your location, though it's still a bit flaky. You could also disable auto-lock screen, though remember to re-enable it after you're done.

2. Heat

After an hour or two of constantly running sshd on iSH, your phone gets warm and toasty. This is because iSH emulates all of its commands—but for good reason, App Store approval!

However, with effort, a custom app could solve the above limitations by:

1. Allowing more control over backgrounding behavior, including abusing background music for unlimited backgrounding.
2. Running arm64 sshd without requiring x86 emulation, reducing CPU usage.
3. And doing both of the above without requiring App-Store approval, since you can load custom apps locally.

Until then, this is a proof-of-concept that leaves productionization as an exercise to the reader.

Enjoy!

Structure

Portal Windows are made up of a three-layer system:

• Integration with React, allowing you to treat a window as a component in your tree, change its props, render elements inside it (thanks to React Portals)
• A window management store, which keeps track of window and screen state, allowing relative positioning of windows, bounds correction, and multi-screen rendering support
• Desktop-side code that enables the frontend to apply and receive updates, and create and address windows

What does this let you do? Great things!

Examples

If you enjoy using Tandem, you might be familiar with these windows, and their use cases:

Being able to use portal windows means that the "End Screenshare" window ended up being only ~100 lines of code total, and took <1hr to build.

Using Portal Windows

Remember the swarm of windows in the video at the top? Here's the code behind it, and here's a simpler version that only has one portal window.

You can follow the steps to get it integrated in your existing Electron app here, or copy a fresh project to play around with.

This tech has cut down on the time it takes to ship new features drastically, and we’re open-sourcing it as a gift to the community. Check out the source code here! (licensed MIT)

Does this interest you? Let us know, we're hiring! jobs@tandem.chat!

Teachers are Mentors

My first internship was exceptional. It shaped my career in the same way that Mrs. Hill's fifth grade class shaped my educational path.

That summer I had a great time, learned a lot, and grew as a person. I left that internship better-equipped to solve problems.

Not just because the tech was robust, or the team was skilled, or because I had real responsibilities. But also because of who I sat next to.

My manager, Zack, was more of a mentor than anything else. When we had discussions, we had them as peers. When we had 1-1s, we would talk about life, and I knew I could ask difficult questions. The type you would ask a friend, when you needed advice.

To some people, I'm describing their current manager-direct relationship. To others, I'm describing something unfathomably impossible. For me, it was my first internship, and I kind of figured I had found something good.

Zack mentored as a peer, and by that, I mean imagine you had time-traveled to see yourself 10 years in the future. Would you expect them to look down on you? Hand you quick answers, instead of expanding your understanding? Or would you imagine them to do everything they could in their power to increase your knowledge and independence, assuming they would be better for it?

Paying it forward

Nowadays, when I need to be a mentor, I try to imagine how Zack would approach it. Last winter, I helped my sister as she built and programmed a RC car with an arduino, and she actually had a good time. This summer, I mentored two of our interns at Tandem, and I actually did well.

I say actually, because I historically haven't been a great mentor. Back when I was in college, I set expectations too high, and I looked down on people I mentored if I didn't feel they met them. I probably had an overall discouraging effect on my sister, or friends I tried to teach programming to, instead of accelerating their learning.

If I hadn't been other side of the table, and learned how to be a mentor from my mentors, I don't know how I would've ever improved.

A grateful thanks

I feel incredibly lucky to have these people in my life. People with no prior connection to me, who still care that I learn, and do well.

To Mrs. Hill, to Zack, and to all my other teachers:

Thank you for instilling a learning spirit in me.

What patch-package does.

When working with javascript, you naturally have file access to the contents of your imported NPM packages. So you can modify them locally. If your code needs a dependency changed, you can make that change, rebuild, and try it out.

What patch-package does then, is it lets you check those patches into version control. So when your teammates or CI systems pull the repo, as part of running yarn to update deps, they reapply those patches. That way you can modify your dependencies, without having to take on the weighty burden of forking them.

Why make a patch?

So here's the context. I work at a company called Jam, and we make the best QA tool for websites and webapps. We care about our users' experience, so we want to keep track of unexpected errors that users hit. We use Datadog for this.

The problem is that a portion of our Chrome¹ extension is a content-script, which gets loaded alongside pages that users report bugs on. When our content script sends updates to Datadog, these updates live in the context of the host page. So they pollute the user's network requests tab in devtools. And when a user closes the page, requests that were buffered could be lost!

So that's the issue. We don't want Datadog requests to go be sent by the pages that our users browse. But Datadog's browser-logs and browser-rum packages don't let you proxy messages with a proxy function! You can set a URL to proxy them through, but that doesn't help in our case. What do? Fork it?

No! Patch-package to the rescue!

The patch itself

The Datadog packages we use have all of their requests go through one file:

note: in this case, this file lived in a third-package, @datadog/browser-core, which is the package I patched directly, after elevating it to a top-level dependency.

What we need to do now is modify this file to allow us to proxy these requests. After that, we'll be able to proxy these requests over postMessage, from our content script to our background script.

But first, we need to figure out what version of the file we're modifying! For this file, there's a cjs and an esm version. Easiest way to figure out which one we're importing? Just sprinkle some log statements in, clear webpack's cache, and reload!

THIS IS ESM!

We've found our file! Next step: modify it!

Since Chrome extension content scripts live in a separate context from their host page (window objects aren't shared), we can simply have our patched package read from the window object, to figure out where should send the message data, without worrying about intereference with/from the host page.

And if we don't add a 'DATADOG_REQUEST_PROXY' function to the window, our logger falls back gracefully. This is useful, because we'll be using this dependency in our background script as well, where we want to use default behavior!

Amazing! Last step, proxy the messages and execute them on the background script.

in the content script, it looks like this

And in the background script, we execute the messages as regular network requests:

That HttpRequest? Imported from the same file we patched, which thankfully already exported it.

The end result: Datadog logs are smoothly sent through our background script!

Now that we're satsified with our changes, we run yarn patch-package to generate a patch file, and commit it to version control:

Try it at home!

Next time you consider forking a package to change a few lines, don't forget you have another option!

Patch-package even makes it easy to contribute back, by providing an easy way to submit your patch as a PR!

One more thing: care as much about user experience as we do? Hate how long it takes to file bugs?

Try Jam today! Our users file bugs in seconds, directly to their issue tracker, with all the details engineers need. Learn more here!

Seriously, give your engineers something to be excited about, when you hand them a bug.

How does it work?

In order to understand the magic, let's first go over the flow:

1. When a user visits the site, they can click a card to have it copied to clipboard
2. They paste it in an input field to verify its value
3. They open a link to Google docs
4. They paste in their Google doc, and they get a Joker card instead, which tells them to try pasting in their URL bar
5. Pasting in their URL bar gives them a link directly to my Twitter account

And now, the details.

This trick was made possible with the ClipboardItem constructor, part of the Clipboard API in Javascript. This constructor allows you to copy text programatically. But it's not only plain text—you can also copy rich text, the kind with styles and formatting that you might find in a fancy word doc.

Of course, when you're pasting rich text into an input field that doesn't support it (like a URL bar), you want to fall back to plain text. This works by the receiving application pulling the requested MIME-type from your pasteboard—and not simply by converting copied rich text to plain text.

This means that we can populate both a rich text value and a plain text value in the pasteboard, and they don't have to have the same data!

So, walking through the steps in detail:

• When a user clicks the card to copy it to clipboard, we copy a regular card to clipboard (in plain text), and a joker (in rich text). The joker is only shown if a user pastes into another application without going to a later step (this is unexpected).
• When the user pastes into the input field, they paste the plain text data, which is the card they copied—a digital equivalent to when magicians say "now, memorize your card".
• When the user clicks the link to navigate to Google Docs, we perform a sleight-of-hand trick, and actually populate the pasteboard again — this time, we copy my twitter URL to clipboard in plain text, and a joker which tells the user to paste into their URL bar, in rich text.
• When the user pastes in Google Docs to verify their card, they see the joker, telling them to paste in their URL bar
• When the user pastes in their URL bar, they get my twitter link, a URL they don't remember copying! This part feels unexpectedly surprising, as it seems like your clipboard was updated while you were on a trusted site, Google docs.
• Now that the user is convinced that I post interesting content, and is already on my twitter page with the follow button a single click away, they follow me. (I gained 1/3 of my twitter followers in the past from this magic trick alone, over the past few days!)

It's not so magic anymore, but I hope you enjoyed the trick, and the explanation!

What you should do when you join a company

1. First, found out what your job is
   • Have a general idea before you join. In fact, you should learn as much of this before you start, in order to align your expectations with reality
   • What are you hired for? (Where do you fit in to the team)
   • What is the composition of the team (Where do others fit in to the team)
   • Find out what your manager thinks their job is (do they take an active involvement in the team? Do they view 1-1s as a chore?) This will help you understand how to treat them as a resource. And unfortunately, when you cannot rely on them (a situation I've seen too often with friends at larger companies).
2. Find out how to do your job (high-level)
   • Find out where your team sits in the org chart (and what are your adjacent teams)
   • Find out which people you'll need to talk to at those relevant teams (and of course, your teammates should also have some relations with these people to lubricate communications)
   • Find out the collaboration process with other roles, especially new roles you might not have interacted with
     • For example, at Tandem, I switched from a primarily-backend role (DNS work at cloudflare, systems work at Deviceplane) to frontend/full-stack. My mistake was instead of asking how to work with our designer, I assumed design handoffs were a one-way street. I was wrong, of course, and ended up having a great handoff process with our designer, Bernat. I would give product and design feedback as I was handed designs, as well as suggestions for minor changes that would improve product velocity. But in order for us to make this relationship effective, I first had to understand it.
3. Find out how to do your job (low-level)
   • What are the systems you'll be building
   • What are the repos you'll be touching
   • What are the connected systems (upstream, downstream)
   • What's oncall like, what metrics, tools, dashboards you need
     • You'll probably forget this—that's okay. Better to have a vague idea of what you need to do, than no idea at all
   • Ask an experienced coworker for a rundown of how your main area of focus operates (this'll be the main repo you'll be touching), including a walkthrough of the code. After this, you should feel ready to jump into the code
4. Engage with excitement, push boundaries, do not become jaded
   • One of my founder friends remarked that part of the reason they hired interns was because interns bring energy. As a new hire, you too have this energy, excitement, and something to prove. Express it soon, or it will fade as work becomes routine.
   • Read all the interesting docs, especially notes the founders have written
     • At Segment, the founders had referenced [author's] post on [commoditizing your complement], which was eye-opening perspective to hear on business strategy, as an engineering intern
   • Schedule coffee chats with people on your team, and new interesting people you meet—this is one of the best ways to learn about another role
     • At Linkedin, I scheduled a 30m coffee chat with a director, who ended up extending it to an hour after he realized I was based in SF and had trekked down from SF to Sunnyvale for the meeting. People often want to reward you for making an effort to get to know them.
   • Browse all the channels
   • Read all the interesting docs (and figure out what the travel policy is ;))
     • At Cloudflare, there was an almost-hidden clause in the travel policy doc that noted you could get up to $500 for staying with a friend on a business trip, instead of booking a hotel. As a result, every time I visited Austin for work, I stayed with a coworker and earned $500 (and did them a favor by skipping a lunch and expensing them a good dinner) :)
   • Offer your opinions — sometimes a fresh set of eyes is the most useful thing you might have. Other times, an energetic response is a useful tool for leaders trying to energize or braisntorm with their team
     • At LinkedIn, our extended team had an extended all-hands with a VP, and for what felt like minutes, nobody asked a question. I noticed he had pink laces on his gray Allbirds, asked him where he got them. Having my dumb (though well-received) question to set the bar low broke the ice enough for engineers with better questions than me to start talking.
   • Ask questions

Now, as you prepare to make the most of your first week and following month, remember: you're off to a good start!

Show me the code!

Alright! Note that this won't work in Docker without privilege escalation, so you'll have to run this after your Dockerfile is built.

After running this, an application that would otherwise have 2GB of free disk space will use that disk space for memory allocation.

# Allocate 2048MB (2GiB) to a swapfile
fallocate -l 2048M /swapfile
chmod 0600 /swapfile
mkswap /swapfile

# Optionally modify swappiness (the likelihood to use swap)
# (default is 60 on most Linux systems)
# More info: https://unix.stackexchange.com/a/88820
echo 90 > /proc/sys/vm/swappiness

# Enable the swap file
swapon /swapfile

You'll want to call this from your application code, as your app starts up. As noted by a Fly maintainer, Fly automatically does this for Rails projects. This should prevent high/bursty-memory programs from being killed by the system OOM killer!

Verify it works

After running, you can run free —mega to see if it's taken effect:

# free —mega
              total        used        free      shared  buff/cache   available
Mem:            231          75          95           0          60         148
Swap:          2147          65        2081

Now we see that we've got about 2gb of memory from swap thanks to our changes!

Querying your own Gmail with IMAP

Here's how you can use IMAP to query your own Gmail account.

First, you'll want to make an app password. This is a password that isn't your real password, but you can give to an app to use for certain functionality on your account. They're nice to have for testing. You can either follow the instructions here or create one directly at this link.

Now, you can use this app password to connect to Gmail! You can use a variety of programs that will let you create a TLS client. We're writing IMAP by hand, after all.

For example, you could connect with ncat using: ncat —ssl imap.gmail.com 993 —crlf. Or you could connect with socat using: socat ssl:imap.gmail.com:993,crlf readline. Essentially, these commands connect to Gmail's IMAP server over SSL, and replace your stdin \n with \r\n, which is required for IMAP commands.

After opening a connection, you can login with your password:

a LOGIN youremail@gmail.com yourapppassword

Which should have the server respond with:

* CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH UTF8=ACCEPT LIST-EXTENDED LIST-STATUS LITERAL- SPECIAL-USE APPENDLIMIT=35651584
a OK youremail@gmail.com authenticated (Success)

Great, now select a mailbox, such as your inbox:

a SELECT INBOX

And you can query away! Try running a FETCH 1 FAST to see how old your first email in your inbox is.

Now that you're at the bottom of this post, you might be interested in a project I'm working on to automagically sort your emails. If that sounds interesting, you can sign up for the waitlist here.

And how does that feel?

Well, I just finished week 5, and it's been like a plunge in cold water! Remember on a hot summer day, you might hop into a swimming pool, and even if the water was cold, it felt refreshing? But first it felt cold?

So first you're eyeing the pool, and it looks great. Then you jump in, and it's cold. But then you start swimming, and heat up. As you reach the surface again to breathe, it's starting to get refreshing.

Here's how it went:

On my first week, I was still wrapping up a feature and some bugfixes unrelated to devtools. At the same time, I was doing market research: thinking of ideas, asking friends/developers/our eng team, looking at customer feedback, and looking at other products (mostly Chrome devtools), to create and pick features we might build.

This felt relatively simple. I'd already done market research for half of my eng/product internship at LinkedIn. I'd compare it to looking at the pool.

Then, the second week, I scoped out the cost and the expected impact of the potential devtools features, and decided which ones we'd prioritize for our 6 week sprint. Once I had that list, I began writing Product Requirement Documents (PRDs) for the features we were going to build. These are documents that should contain everything an engineer needs to build the feature, including handling edge cases, design mocks, and text copy.

This felt adventurous but doable. I'd have to express my opinions and be the authority on product decisions here—previously I was always able to ask the PM if I felt unsure. I'd compare this to jumping into the pool, cannonball style.

Then, the third week, I continued writing PRDs, and started pairing with our designer to mock designs for them. This is a lot more involved than your normal product + design collaboration, because devtools are very opinionated and intricate. As a technical PM, you need to be there to assert those opinions that engineers (users) will have, but at the same time, you're not a designer. So it's a tight collaboration.

This started to become a little overwhelming, as the sheer amount of meetings began to balloon. I'd compare it to jumping in the pool, and feeling the cold shock of the water.

Next, the fourth week, I continued writing PRDs in tandem with designs, and finished designs for these PRDs with our designer. I also worked with our PM-founder, Dani, to decide how to allocate engineers onto these project.

This felt a little tricky: meetings decreased, but decisionmaking and managing my own time increased. I'd compare this to swimming up from the bottom of the pool.

And most recently, the fifth week, I did a couple of engineering kickoffs to hand off two PRDs to the engineers working on them. Then, I quickly finished a small feature to unblock them—they'll hook into the code in it as part of the ticket they're currently working on. After that, I continued writing PRDs, creating the final PRD (but not finishing designs for it).

This again felt overwhelming and tricky. meetings increased, decisionmaking increased, and I had to manage my own time better than before. But I knew it'd get easier soon. I'd compare this to swimming, almost at the surface.

At this point, I have to kick off two more features with engineers next week, work on the last two unfinished PRDs (one has design assets, one needs both product and design work). Then, I can work on the next feature that I've allocated myself onto as an engineer.

I expect the following weeks to be calmer (starting 2wks from now), as most of the product planning will have been done, and I'll be mostly in engineer-mode. Then, when other engineers wrap up their projects, I'll kick off another project with them and work with our marketer to launch updates.

So that's the overview.

My mistakes

As usual, I made mistakes. Nothing crazy, but I'll be unlikely to make them next time.

Chronologically from the top, they are:

• As I did market research, I didn't consider that after I do all of the research on what exists (thinking as a developer, reading customer feedback, sourcing developer opinions, looking at existing products), I'd still have more thoughts that I would personally come up with at the end. So I ended up rushing near the end, and I don't think I did a good job capturing (and thus prioritizing) some of my more experimental ideas.
  • Giving me a hybrid PM role for a whole 6wk eng sprint is already experimental (for context, this is also the first time we've tried 6wk eng sprints at work), so maybe it's better to not add too much variance with experimental ideas. But I do wish I had more time to flesh these ideas out, they seem interesting.
• As I was writing PRDs in week 2 and week 3, I suddenly realized that while a PRD communicates product requirements to engineers (and that had been all I'd experienced as an engineer), there was something more to a PRD.
  
  Turns out, a designer also needs to know the product requirements. So I went back and typed out the full user flow from the perspective of what the user needed to do, for PRDs with complicated flows. Then, for PRDs with specific cases that needed to be designed, I went back, and typed out the cases that needed to be covered.
  • This wasted both product time and design time. Design time was more important, as it was more limited.
• Next, as I allocated engineers, I realized that I didn't chart out the allocation of engineers earlier, I just kinda winged it and estimated in my head. It wasn't a huge mistake—I hadn't overpromised any customers. But I had gone through the design process for most of our features, and it'd be a waste of design and product time if we had to cut a fully-scoped feature.
  • Luckily, we managed to fit everything in (or I just made another mistake and overpromised my own capacity as an engineer, we'll see).
• Finally, for the fifth week, I didn't think to show to the team what our sprint would look like on Monday at standup (to get them excited), to involve marketing (to give them heads-up), or to pipeline the work I was doing (to make sure people are unblocked in the future).
  • Luckily our PM-founder Dani thought of these things, and mentioned them to me, so I managed to benefit from her foresight 😛

The biggest mistake

But the biggest source of pressure was my last mistake: the unexpected time-cost of things I needed to do that didn't produce deliverables.

For example, a few sources of time pressure:

1. I've been getting my wrists checked out, and going to Occupational Therapy (OT) to get this sorted out, which takes 2.5hrs a week on average
2. Two of our engineers recently joined, and as engineers onboard, they need help with:
   • dealing with our tooling, CI, local dev env
   • architectural overviews of dataflows/parts of our codebase
   • pair programming and/or debugging
3. Having regular 1-1s with teammates, which used to be a nice forced-break from programming, lost this benefit, given I was already frqeuently context-switching.
4. Being the engineer with the most experience on our devtools codebase (due to my earlier chance to improve them) and devtools-adjacent data flows (coincidentally, due to other feature work I'd done), I was the go-to person for background on them.
   • This meant that after I kicked off devtools features with an engineer, I'd spend additional time in calls/DMs giving background for architecture or technical decisions

What I've learned

These 5 weeks have given me more respect for PMs. It's easy to see a good PM do their job, without realizing all the parts they have to keep coordinated in lockstep.

I already knew how important communication was, but I didn't realize how hard it was. It's easy for things to slip between the cracks, yet optional communication (like giving an overview of the sprint to the whole team in standup) often has an outsized impact (like getting everyone excited).

Most importantly, I learned how important excitement is. Most of us would agree:

• When work feels meaningless, I feel demotivated.
• When work gives me potential to do good work, learn, and make a product people enjoy, I feel excited.

So, when kicking off features with engineers in last week, I tried to group features together for each engineer, allocate enough time to do features well, and be flexible when engineers want to make product and design changes. The idea is to let engineers own the features they're working on—just like when I upgraded devtools myself, last time.

It ended up being helpful to me, too! It turns out that when an engineer feels that they're allowed to make product and design decisions, they might make better decisions than you—the PM! Think about it: they're focusing deeply on one feature, while you're spread out over multiple features at a time. That means they have time to focus on the nuance of tiny behaviors that they implement, but you might not think to cover in your PRD.

Obligatory callout

Anyway, if you enjoyed reading this journal-style post, and thought you might have fun working here—check out Jam! To be clear, my engineering weeks are much more relaxing than the hybrid role I've written about here ;)

Simple transform functions

Just to understand what some common transforms might look like, I tried plotting them out.

This histogram, which comes from plotting Math.pow(Math.random(), 2) samples, looks like a power law distribution:

(About half of the total samples in this distribution are less than 0.25.)

The distribution of these Math.sqrt(Math.random()) samples feels fairly linear:

(This is the same as Math.pow(Math.random(), 1/2))

Interesting, but how do we get a normal distribution?

Getting a normal distribution

Combining the two above functions, we get closer. Their combination looks like an attempt at building a sandcastle with dry sand:

This distribution comes from Math.sqrt(Math.random()) + Math.pow(Math.random(), 2).

Remember the linear-looking Math.sqrt(Math.random()) function? When sampled three times and added together, it gives us a normal-looking distribution:

It's a little misshapen, but Math.sqrt(Math.random()) + Math.sqrt(Math.random()) + Math.sqrt(Math.random()) looks normal enough.

You might've guessed the pattern by now.

If we add Math.random() on itself a few times, we get an even-more normal looking distribution. The more times we do it, the more-normal the graph starts to look.

Math.random() + Math.random() + Math.random()

Great, but why does this work?

Why does adding samples of a uniform distribution give us a normal distribution?

This may seem confusing. So let's break down the possibilities. Imagine instead a uniform distribution that gives integers in the range [1, 6].

One provider of this distribution function is the common six-sided die.

When we have one die, we have an equal 1/6 chance of getting any value.

When we have two dice, each die still has an equal 1/6 chance of getting any value. However, if you've played any dice-rolling game, you might know that the chance of rolling a 12 is much lower than the chance of rolling a 7.

Why is this?

1. To roll a two, you must roll two 1's. We'll call this a (1, 1).
2. To roll a three, could roll:
   • (1, 2)
   • (2, 1)
3. To roll a four, you could roll:
   • (1, 3)
   • (2, 2)
   • (3, 1)
4. To roll a five, you could roll:
   • (1, 4)
   • (2, 3)
   • (3, 2)
   • (4, 1)
5. And so on, with permutation count increasing until 7, and then falling down after..
6. To roll an 11, you could roll:
   • (5, 6)
   • (6, 5)
7. And to roll a 12, you must roll a (6, 6).

Since each die roll is as likely as any other die value, each ordered die permutation (an ordered combination) is as likely as any other ordered die permutation.

That is, rolling a (1, 1) is as likely as rolling a (5, 6) (ordered). In the former, you have a 1/6 chance of rolling a 1, then another 1/6 of rolling the next one. In the latter, you have a 1/6 chance of rolling a five, then a 1/6 chance of rolling a 6.

That means that each permutation is equally likely, so the likelihood of rolling a specific sum is proportional to the number of permutations that can create it.

In other words, when you roll two dice, you have six ways to roll a 7, but only one way to roll a 2. Then, it would make sense that our distribution curve of combined dice values show 7 being six times as common as a 2.

The more dice you combine, the more you shift the distribution more closely to a normal distribution. Why does this approximate the normal curve shape specifically? It's explained by the central limit theorem, though honestly I don't quite understand it.

In lieu of that, here is a live demo I've created for you to simulate the probability of outcomes. It generates all possible sums of N rolls of an M-sided die, and plots the count of each sum:

Moving back to our real-world application, how can we create a bimodal distribution, with our normal probability distribution?

Creating a bimodal distribution

Well, the easiest way is to simply alternate between picking samples from two transformed normal probabilities:

// Where randNormal is a random-number-generating function
// with a normal probability distribution

const PEAK_DISTANCE = 0.7;
const PEAK_WEIGHT = 0.5;
function bimodalSample() {
	return Math.random() < PEAK_WEIGHT
		? randNormal(Math.random) * (1 - PEAK_DISTANCE)
		: randNormal(Math.random) * (1 - PEAK_DISTANCE) + PEAK_DISTANCE;
}

If this seems too easy, note that bimodal distributions are frequently caused by just this: sampling two different groups into the same dataset.

Sampling the above function gives us this plot.

You can change the PEAK_DISTANCE and PEAK_WEIGHT parameters to change the distribution shape:

Here, we set PEAK_DISTANCE to 0.6, and PEAK_WEIGHT to 0.2.

For my use case, I kept the distribution even between the two sides. You can see its effects live on my blog homepage. Note that post angles won't change between updates or page refreshes. This consistency is achieved by using a Math.random()-like function which can be seeded with the post's ID. Seeding the function this way gives us the same angle for each post across refreshes.

And that's it! You now have a bimodal distribution using only a uniform distribution function as your randomness source.

So, what does Jam do, anyway?

At Jam, our engineering team builds a browser extension for reporting bugs.

Our extension automatically captures bug data, such as:

• console logs
• network requests
• screenshot/video/DOM instant replay
• page metadata
• repro steps
• device info, timestamp, viewport size, etc.,

So that when a Product Manager, QA specialist, Designer, etc., files a bug, the bug automatically contains all the info an engineering team needs to debug the bug.

Engineers like this because they don’t need to hop on call. Bug reporters (PMs, QAs) like Jam because it integrates with issue trackers (like Jira and Linear) so they can rapidly fire off a bug report with all the data embedded inside it.

That’s the gist. Underneath the surface, there are a lot of moving pieces that make this possible.

What do the components of a browser extension look like?

Note: at time of writing, we’re using the Manifest v2 (MV2) architecture, supported by Firefox and Chrome/Chromium-based browsers.

Within a browser extension, there are a handful of different pieces (all of which are optional):

1. One background script, which is persistent in the background
2. One popup window per browser window
3. One or more content scripts per browser tab
4. One or more host scripts per browser tab

Note: This is a simplified model. For example, if your extension is running in split mode (which is a non-default behavior), you can have multiple background script instances. But generally, this structure applies for nearly all browser extensions.

Take for example a Chrome instance with two windows open, with each window having two tabs open. A browser extension running on this Chrome instance could be arranged like so:

How can the components of a browser extension communicate?

Across the different components of the browser extension, there are a few different messaging APIs. Each API is used to communicate between different components.

These APIs bridge the following components:

1. We can use chrome.runtime.sendMessage between:
   • popup and background (bidirectional)
   • content script to popup (one way)
2. We can use chrome.tabs.sendMessage between:
   • background to content script (one way)
   • popup to content script (one way)
3. We can use window.postMessage between:
   • content script and host script (bidirectional)

In essence, all of the components are connected, like so:

But of course, if sending messages was as easy as this diagram seems, this blog post wouldn’t exist.

What messaging difficulties does this setup have?

For a long time, we used the native APIs directly.

However, with native APIs alone, we have the following issues:

1. Different underlying messaging APIs: Each component has its unique way of sending messages.
2. Messages have size constraints (on some APIs): Some messages are too big to fit through the API pipeline.
3. Some components cannot communicate directly: Certain components can't talk directly, requiring a 'middleman'.
4. Multiple Tabs, Popups can exist independently: Keeping track of multiple, independent components can be tricky.
5. Messages can be dropped: e.g. Popup is closed, Tab gets closed while receiving, Tab’s thread is processing so much that the event loop can’t pull queued messages + respond/ack them before they time out.
6. Sharing object URLs not allowed to/from an incognito context: sending object URLs (a trick for circumventing message size constraints) from your background script to a content script won’t work, if the content script is in an incognito tab.

This last point was what finally forced us to build a network stack.

Hitting our breaking point: Incognito

So, let me provide the business context behind this feature.

Some of Jam’s biggest power-users do Quality Assurance (QA). They aim to test and break an app as fast as possible, going through dozens of flows over the course of a day, doing manual regression tests, and putting new features under a stress test before release.

When you’re testing so many flows, Incognito Mode is an essential part of your toolkit. It lets you instantly access a logged-out instance of your web app, which makes testing new user flows much easier. Plus, some apps have different behavior in incognito mode, which requires additional testing (looking at you Netflix, why won’t you let me use incognito mode).

All that to say, adding incognito support to Jam was important.

Why did we need to rebuild?

In order to understand our problem with our system up until now, we need to understand the flow for a user creating a bug report in incognito.

Before a user can submit a bug report, we need to display a preview of the bug report in the tab they’re filing the bug from.

This preview shows the user their network errors, lets them trim videos/instant replays, annotate screenshots, write a title/description, and directly file bug reports to an issue tracker or Slack channel, if they wish:

To create this preview, data is sent from the background script to the content script. Some elements of preview data (e.g. DOM Instant Replay) can’t fit in a single message: they exceed the size constraint of the chrome.tabs.sendMessage API.

To get around this on non-incognito tabs, we use a neat trick: you can create a URL for a JavaScript object using window.URL.createObjectURL on one end, and fetch the URL on the other end to transfer large objects. Using this method, both processes can share large objects that wouldn’t fit in a single message.

Except this method won’t work between the background script and an incognito tab. Incognito tabs have different object URL contexts. Without this trick, we can’t circumvent message size constraints. Message chunking must be implemented.

But our existing system is a simple layer directly above native messaging APIs, so adding chunking requires a rebuild.

Do our problems seem familiar?

Remember our messaging difficulties from before? Those problems are actually …general networking problems.

1. Different underlying messaging APIs? ← Different link layer protocols (Ethernet vs Wifi)
2. Messages have size constraints (on some APIs)? ← Wifi/Ethernet packet-size limit
3. Some components cannot communicate directly? ← Routing can require hops
4. Multiple Tabs, Popups can exist independently? ← Multiple independent devices
5. Messages can be dropped? ← Packet loss
6. Sharing object URLs not allowed to/from an incognito context? ← Separate machines, data must be serialized

Looks like our browser extension’s components have similar communication problems to devices accessing the Internet. The solution to our shared problem set? Create a network stack.

What is a network stack?

A network stack is a set of layered protocols for communication. Each protocol solves a specific problem. Each protocol layer builds on guarantees offered by previous layers.

Take the TCP/IP stack, for example. This stack is used by Internet applications, such as websites or video games:

At the bottom, we have the link layer, connecting physical devices.

Next, the Internet layer builds on top of the link layer, allowing routing and forwarding across devices, so messages can be sent between devices that aren’t physically connected.

The internet layer makes use of a routing table. This table is used to determine where packets should be sent, based on their IP address. Since each machine isn’t directly connected to every other machine on the internet, a per-machine routing table keeps track of which addresses should be used for packet forwarding.

This routing table can be populated by a few sources. At the highest level, routing can be determined by a protocol called BGP (Border Gateway Protocol), the communication protocol between Internet Service Providers (ISPs), such as Xfinity and Verizon. From there, your ISP can populate the routing table for your router, which can populate the routing table for your individual machine.

Above the internet layer sits the transport layer, which can be as simple as UDP (a simple wrapper and checksum of an IP packet), but in the Internet’s case is typically TCP, which provides more reliability. TCP allows streaming, retries, and prevents double-delivery.

Finally, the application layer gives a well-featured layer for developers to build applications on. This layer gives us cookies, headers, encryption, status codes, and user-readable domain names. It’s much easier to build software on this layer than to attempt to use TCP directly.

These are the same abstractions we used for our extension’s messaging system.

Jam’s extension network stack

At the bottom of our extension component network stack, we have our link layer, which handles communication between directly-connected components.

Above that, our packet layer handles routing, forwarding, and dropping packets, much like the internet layer from the TCP/IP stack.

The packet layer makes use of a routing table, which is used for routing packets to the correct extension component. Since our extension components will always have the same configuration (host script is connected to content script, content script is connected to background, etc.,), we can use a static routing table, no BGP-like dynamic routing algorithm required.

Above the packet layer, we have the datagram layer. This layer was essential for implementing incognito mode support on Jam, because it allows large messages to/from incognito tabs to be automatically chunked and sent in pieces. If we need to implement message streaming in the future, we would implement it at this layer.

Finally, we have the application layer. This layer gives us a comfortable send/receive pattern that automatically creates response listeners and ties a message request back to a message response. (Imagine if HTTP requests didn’t have responses, and you’ll understand what using this stack would feel like without this layer).

Did we solve our problems?

Do you remember our earlier messaging difficulties from before? Well, here are their solutions.

1. Different underlying messaging APIs? ← Abstracted by the link layer
2. Messages have size constraints (on some APIs)? ← Chunking at the datagram layer
3. Some components cannot communicate directly? ← Packet forwarding at packet layer
4. Multiple Tabs, Popups can exist independently? ← Independent addressing at packet layer
5. Messages can be dropped? ← Packet retries at datagram layer
6. Sharing object URLs not allowed to/from an incognito context? ← Automatically switch to chunking if the sender/receiver is incognito (and message size is too large to fit in a single packet), at the datagram layer

So, yes! These problems feel solved. And if this system worked for the internet, it will probably scale for any other complicated communication scenarios you may imagine, where we can’t use HTTP (a second extension, a native desktop app, etc.,).

Outcomes of this project

This extension component network stack has been in production and stable for around 9 months at time of writing.

Here are a few choice benefits that our approach gave us:

Smooth rollout

Prior to this network stack we had a simpler setup, with an application layer built on top of native messaging APIs. By keeping our final application layer the same for this project, we were able to simply cutover to the new network stack, and have Incognito Mode just work.

Not only that, but as this stack was built, the rest of our team was able to continue writing code that introduced new application message types, and used more application message calls. Switching to this new system didn’t impact developer velocity.

Simpler debugging

Separation of concerns across each layer makes it easier to narrow down where an issue is happening. Tests at every layer (with simulated native message APIs) give us confidence on every layer and the system overall.

Focus on building features

Developers doing feature-work no longer need to bother with message length constraints, forwarding messages over the content-script-to-host-script-hop. Our network stack abstracts away these problems. This cuts down the amount of code we need to make a cross-component feature work, and lets our engineers focus on building product features.

Background

After roughly four years at Jam.dev, I felt it was time for a change.

Four years ago, I simply wanted to improve my Product and Marketing skills, while continuing to do Engineering work.

Coincidentally Dani and Irtefa—the two best PMs I knew from Cloudflare—had been calling me to join their new startup for a year. Their marketing was strong, their product ability was stronger, and we had a proven trust in each other from our days launching zero-to-one products at Cloudflare. It was a great fit.

At Jam, I accomplished my initial goals (Product and Marketing), and then some (Eng leadership).

But over those four years, some things had changed. The pandemic had ended, I had proposed to my then-girlfriend, moved from SF to NYC, bought a condo, and married my now-wife.

There was a tension between my desires and my reality: my financial risk profile had changed, my desired learnings had changed, and covid lockdowns no longer restricted me from working from an office—but Jam was fundamentally a remote company.

Leaving this mismatch unaddressed was painful, so I decided to make the switch.

Goal: Qualifying Success

When starting any process, it's useful to approach it with a goal in mind. Naturally, in a job search, we want to find the best role and secure an offer.

But what "best" means is hard to define. I believe it's not simply the total compensation. So in this stage, I came up with a qualifier to determine which role offers I was most likely to accept.

My qualifier here was personal and likely different from yours. So while I'll share my qualifier (less relevant to you), I'll also share the process I used to derive it, which you can use for yourself.

My qualifier:

I decided that my ideal role would have NYC engineering presence, series C or later fundraising stage, good role fit (people, work environment, learnings — more detail below), and competitive comp.

In the unlikely event that I completed the process with two strong offers that I truly couldn't decide between, I planned to use a long-tail of minor selection criteria to decide (e.g. commute path, office's distance to nearest gym location, etc.).

Determining your qualifier:

I had a two-part process to decide this qualifier. First, I had already introspected to determine what was pushing me to search for a new role.

Second, I had the fortune to bring in a wonderful external consultant who kindly did some pro-bono work:

Caption: A consultant-turned-designer, Martin briefed me on strategy live in FigJam

Before joining Jam, Martin had spent 7 years at BCG, a prominent consulting firm. Martin has a well-structured thinking process, and I was generally curious on his advice, so I reached out to chat with him before taking sabbatical.

During our call, Martin created 6 cards to represent attributes of a role:

1. Work location: In-person/hybrid (you could change this to "remote", if that's what you prefer)
2. People and work environment
3. Learnings
4. Compensation
5. Career trajectory
6. Nature of the work

The above is the ordering that I selected. You can try reordering these to reflect your own preferences. The hard part is that there can be no ties.

I took the editorial liberty to cut and condense this down into three bullet points:

1. NYC engineering presence (hybrid)
2. "Fit" (a combination of people, work environment, learnings)
3. Compensation

In order to more-effectively screen companies, I added "Series C or later" as a requirement. This was a proxy indicator for both my learning and comp requirements.

We now have our qualifier. This will come in handy for both qualifying leads (apply this qualifier loosely) and selecting an offer (apply this qualifier strictly).

Architecture: the Pipeline

Imagine a minimal interview funnel:

1. Find a role
2. Submit an application
3. Interview
4. Receive the offer
5. Sign the offer

It works in theory, but has obvious failure points: What if your application is rejected? What if you fail a round? What if the offer is uncompetitive?

The naive solution is to flood the funnel with more applications, and hope a strong offer appears.

But the response rate of direct job applications is not high. If your qualifier is strict, you may have difficulty finding qualified roles. And as a mere human, you do not have infinite time or energy. I believe there is a better solution.

Pipeline Overview

The pipeline

This is the pipeline I gradually refined over the course of my search.

Its scope is broad:

• Prospecting, qualifying, closing, and negotiation — roles traditionally across: BDR/SDR (junior sales), AE (senior sales), and Deal Desk (sales ops), in a standard sales pipeline
• SEO (a form of marketing)
• Due diligence (finance)
• Contract review (legal)
• And more...

On top of actually interviewing, of course.

That said, every stage within this pipeline either provided a significant benefit, or was outright necessary to secure the comp and role I was targeting.

Prep Stage

The first stage in the pipeline is preparation.

For both inbound and outbound opportunities, three things are essential:

1. An intro you can comfortably and confidently rattle off.
2. A resume that showcases your work.
3. A LinkedIn profile that makes you look like a good candidate.

If you are rusty with leetcode style problems, this is also the time to sharpen up. I set up a weekly practice call with my younger sister, who also works in tech. Most candidates have trouble clearly communicating while coding, so I'd highly recommend pairing rather than solo nonverbal problem solving.

Optimizing your narrative

Do you feel awkward or unrehearsed when asked to introduce yourself? A weak intro can give a bad first impression and start the interview with lower confidence, in turn affecting your performance.

A strong intro, on the other hand, helps the interviewer contextualize your background and better assess fit.

In general, a good intro can cover the following areas:

• Brief technical overview
• Key career choices and the reasoning behind them
• Big wins or standout projects, with relevant business context
• Why you're interested in this company

To come up with this, look at your resume and connect the dots into a coherent story. If you know someone in consulting, they can be helpful. Consultants are trained to structure narratives effectively.

Here's what my intro looked like:

1. Broad overview
   • Positioned myself as a "product-leaning full-stack engineer".
2. Career moves
   • Highlighted my major project at Cloudflare (role + business context), briefly mentioned my 2 intermediate (failed) startups, and explained my transition to Jam — outlining the reasoning behind each move.
3. Jam context
   • Explained Jam's core product at a high-level, walked through the multi-year roadmap, and described my role in shaping it.
4. Major project
   • Briefly described and framed the business need behind my most recent large initiative: TLM'ing our new product, Jam for Customer Support.

I didn't memorize this intro word-for-word. This kept it natural, and let me adjust on-the-fly. Since it was on the longer side, I trimmed details if I sensed an interviewer was impatient or less-likely to be interested.

Updating your resume

Plenty has been written about resumes. I'll add just two points I found useful.

First, you can bold text to improve readability. My resume is quite dense. To counteract this, I bolded key points to draw the eye.

Feel free to judge this technique's effectiveness for yourself.

Second, provide context. The weakest part of most resumes is the lack of context.

If you worked at a startup: what did the company actually do? If you drove a change within a large org: what was its business impact? When possible, reinforce this with metrics. Answering just these two questions helps readers understand both your experience and its significance.

Assume a critical reader will interpret ambiguity against you. Providing context increases your odds of passing a resume screen.

Optimizing your LinkedIn

In today's world, role listings are overwhelmed with applications. Optimizing your inbound funnel is arguably your highest-impact lever — and like it or not, LinkedIn is the most effective tool.

Consider this anecdote:

• With a referral, I applied to a well-known public ridesharing company 3 times (Staff, Staff, Senior). Rejected. Then again twice more (Senior, Senior), all rejected at resume screen.
• Later, that same company's recruiter instead reached out to me on LinkedIn about a separate Staff-level role.

In my search, referrals were not more effective than cold applications, whereas inbound consistently surfaced strong opportunities. My take is that during the ZIRP-era tech bubble, rapid growth and low-signal referrals (amplified by Blind users) diluted applicant quality. From my experience, larger companies in particular no longer view referrals as strong signal.

To optimize your LinkedIn, I recommend two key steps:

1. Think like a recruiter
   • Recruiters use LinkedIn's Recruiter tool to search by company, keywords, job titles, and years of experience. In that sense, I treated my profile as an SEO problem.
2. Think like a human reader
   • Next, when a recruiter, HM, or interviewer views your profile, what do they see? Profile photo, header, tagline, description, and role description should be clear, compelling, and human-readable. Avoid simply stuffing with keywords.

For tangible LinkedIn optimizations, I recommend:

1. Profile photo — professional, approachable, and intelligent-looking
2. Cover photo — visually reinforces your work or industry
3. Tagline — brief and role-specific
4. "About" section — balances keywords with clear human-readable content
5. Experience descriptions — similar to resume entries, concise and achievement-oriented
6. Skills section — crammed with relevant skills (these are essentially keywords that don't take up space on your profile)
7. Featured posts — showcase work you are proud of (ideally with media assets, which increase clickthrough rates)
8. Activity section — professional, or hidden (no dumb comments on friends' posts)

Finally, enable "Open to Work" with settings you're comfortable with. I recommend avoiding the green profile picture banner. Fair or not, it signals desperation, which is negative signal for a candidate.

One caution: after enabling "Open to Work", I noticed a sharp rise in spam calls. To minimize this, remove your phone number from LinkedIn, or disable the setting that allows user lookup by phone number.

Inbound Stage

With your prep complete, you are ready for inbound.

When a recruiter reaches out, their first goal is simply to confirm fit. In most cases, this means scheduling an initial call to cover your background and collect your resume, which will then be screened again by the hiring manager.

If you're working with a third-party recruiter (rather than an internal one), expect a few key differences:

1. They'll try to match you with multiple companies.
2. After their initial screen, they'll hand you off to an internal recruiter so the company can integrate you into its pipeline.
3. Most third-party recruiters function mainly as sourcers — they generally can't answer detailed questions about the company, compensation, or process. (An exception in my own experience was Lewis Fegan, who supported my Ramp process and was excellent.)

Two things to keep in mind before talking with any recruiter:

1. Screen inbound carefully
   • The biggest mistake is wasting time on low-quality inbound that won't convert. Just as BDRs and SDRs qualify leads before handing them to AEs, you should qualify recruiter outreach before investing time. If you're unsure how, don't worry — after a couple dozen calls, you'll develop an instinct for which messages lead to quality opportunities and which do not.
2. Present yourself well on the call
   • When you do take a call, speak comfortably and confidently. Share your background without going too deep in the weeds — they'll ask if they want more detail. Be prepared to answer common questions like "What are you looking for?" and "Why are you leaving your current role?" (I cover my answers to these in the section on qualifiers above).

Practical tip: Save the recruiter's number as a contact. This helps distinguish scheduled calls from spam, and reminds you of their name and company when you pick up — especially useful when calls are back-to-back.

Sourcing Stage

The purpose of sourcing is to gather intellilgence needed to enable outbound.

Outbound is the act of reaching out: crafting the pitch and sending the email. Sourcing, by contrast, is the preparation: identifying the right person to contact and the right context to frame the outreach.

There are three main types of sourcing, best tackled in this order:

1. Role listings
2. Relevant cold contacts
3. Relevant 1st-degree connections

Sourcing Role Listings

In my experience, sourcing role listings is still a surprisingly manual process.

A couple reasons stand out:

1. No single aggregator for all roles (LinkedIn does well, but doesn't have everything)
2. Poor search filters and discovery (even LinkedIn struggles with filtering its own content)

Despite this, it's worth the effort. Sourcing roles feeds directly into the next two sourcing steps: cold contacts and first-degree connections. Without a qualified role, you don't have a reason to run outbound in the first place.

Today, LinkedIn Jobs is probably the best broad aggregator. Beyond that, think of specific companies and check their careers pages directly. You can also borrow sourcing tools from other domains — for example, Crunchbase (venture) or Apollo (sales) — to expand your search. I'll cover more on that in the cold-contact sourcing section below.

As part of role sourcing, the next step is to apply your qualifier to decide whether a role is worth pursuing. This doesn't require the full rigor of your final criteria. A lighter version is enough (e.g., assume more comp flexibility, or allow for a weaker NYC engineering presence). Casting a slightly wider net both increases options and provides better market signal.

As you begin identifying relevant roles, it's also useful to start tracking them systematically. I cover this in more detail in the "Monitoring System" section.

Sourcing Relevant 1st-degree connections

With a list of roles in hand, the next step is to see if you know someone to reach out to. On LinkedIn, this is straightforward: search the company and see if anyone in your network works there.

Sourcing Relevant Cold Contacts

But first, why do we even bother with cold contacts?

To explain why sourcing cold contacts matters, we also have to acknowledge its counterpart: cold outbound. Unfortunately, the idea of sending cold outbound makes many engineers recoil.

The good news is, you don't need to do much. In my entire search, I sent just two cold emails.

One email went poorly. Everything from targeting (director without an open Staff role), to addressing (mail bounceback on first attempt), to the actual contents (uncompelling copy) were bad. Predictably, it received no response. It was a low-effort cold email, and it was a waste of time.

The other email got me into the pipeline for one of my top choice companies (one of the two leading AI startups), and matched me with the exact team I was interested in. How? I cold emailed the HM with compelling pitch for a role he was hiring for. It was a high-effort, well-targeted email, and it was very effective.

So, cold outbound can be very effective. But it also needs to be high-quality.

The most important part of sourcing is finding the right person. Who's more likely to respond to a cold email: a random engineer, or the hiring manager actively looking to fill the exact role you want?

Additonally, why are you emailing them? Why is their team interesting? Why did their company catch your eye? Targeted cold outbound comes off as genuine interest, while untargeted cold outbound comes off as spam.

On tools: aside from LinkedIn, I recommend Apollo over Crunchbase. Though built for sales, Apollo is excellent for sourcing both companies and contacts, and even role listings. A few killer features:

• Office-level filtering: Crunchbase limits you to headquarters, but Apollo lets you filter by any office location. This is invaluable when targeting NYC offices for SF-headquartered companies.
• Role-level filtering: Apollo can filter for companies with open roles by title and location. In fact, I found LinkedIn postings through Apollo that LinkedIn itself failed to surface in search.
• Contact discovery: Apollo also surfaces company contacts and associated work emails.

Outbound Stage

With roles and contacts in hand, it's time to run outbound.

Direct application

The simplest form of outbound is filling out an application form and hitting submit. Because the barrier to entry is so low, these channels are flooded, so success rates are generally low.

That said, here's one useful LinkedIn-specific tip: even without Premium, LinkedIn shows how closely your profile matches a role. While recruiters may not see this score, it's a reasonable proxy for how your profile ranks in their searches. Conveniently, the match score updates when you add skills. This means that every job listing can be treated as training data to fine-tune your profile and increase inbound.

Before and after: adding a "prospecting" skill to your profile, and checking the role fit score.

Cold outbound

Cold outbound is extensively covered in the sales world, and there are resources written by people far more knowledgeable than myself. My simple advice is this:

• Provide context: why are you reaching out to them?
• Provide background: why you?
• Make a clear ask: define the next step, and keep it low-effort.

Then... send the email. Don't overthink it, and keep expectations low — after all, how many cold emails do you reply to?

Warm outbound

Warm outbound should feel like asking a friend for a small favor. Pop in, say hello, request a referral, and keep them updated as you move forward. Keep it light: not transactional, but not an essay either. They're likely busy, and this is still a work-related request.

The primary goal of warm outbound is usually a referral. In my case, though, referral success rates were about the same as cold applications (see the Sankey diagram below).

That said, reach out anyway. And once you're later in the process, call them, catch up, and ask for their perspective. It's always useful to get insider context on the team and company. I cover this further in the "backchannel diligence" section.

Company Interview Subpipeline

Each company runs its own interview process, which I'll call the company interview subpipeline: this spans everything from the initial recruiter screen to team matching.

Naturally, this means the diagram above will vary in practice. A few examples I encountered:

1. Async coding screens before live coding screens (ranging from leetcode-style automated assessments to manually-reviewed CTF submissions)
2. Hiring manager calls before technical screens
3. System Design interviews as the first technical round, before the onsite
4. Informal team matching prior to onsite (not uncommon, often with the hiring manager conducting the onsite leadership round)

With that variance noted, let's now walk through the standard interview pipeline.

Internal Recruiter Screen

The internal recruiter is your first real company contact. They begin the process with a screening call.

Recruiters aren't exactly your friends, but you should be friendly with them. They prep you on what to expect, are your point of contact for urgent updates or scheduling changes, and serve as your first-line advocate in negotiation.

An internal recruiter screen is similar to a third-party recruiter call (covered earlier), but here you can ask substantive questions about the company, process, or team.

Use this call to gather details that matter to you. For example, if you're screening companies based on equity liquidity, you'll want clarity on structure:

• Public stock (e.g. Google, Meta, Uber)
• Standard 7-year RSUs (e.g. Ramp)
• ISOs with a long post-termination exercise window (e.g. Hightouch)

Recruiters often confuse PTEW with vesting schedules, so you may need to push for specifics. In my case, I discounted equity value by 50% for companies offering ISOs with <7-year PTEW.

Sometimes this call is run by a junior recruiter or coordinator, who may later hand you off to a senior recruiter. Junior recruiters often can't answer detailed questions, so save those for the next stage.

Recruiters may also ask for your salary range, or share theirs and ask for feedback. At this stage, I answered honestly and said I was prioritizing fit over comp, and would look at numbers later. This helped keep my options open and gave me more market data. You can always decline to proceed later. To pre-screen roles with obviously below-market comp, check salary ranges on job postings or use sources such as levels.fyi.

You may get rejected after this stage for two common reasons:

1. Salary expectations too high (if you've stated them)
2. "Not a good fit": typically after call, recruiter passes resume to EM, who decides to pass. (Especially if headcount is limited)

Core interviews

One tip for everything after the recruiter screen: do light background research on your interviewers.

Before each call, I'd skim my interviewer's LinkedIn profile. This gave me a sense of their focus areas, but the main benefit was in shaping my post-interview questions. I found it valuable to hear their personal anecdotes — especially around role transitions — as it helped me understand their perspective and how they viewed their own role.

Tech Screen

The tech screen is usually a straightforward coding round, typically one or two medium-level leetcode style questions. That said, my tech screens and coding rounds often leaned towards practical problems.

I recommend the following flow:

1. Clarify the problem
   • Ask questions, get details on the problem, confirm your understanding.
2. Walk through examples
   • Test understanding of normal cases, and confirm expected behavior for edge cases.
3. Talk through your solution verbally
   • I recommend commenting this out as you walk through, so you can implement code from your comments.
4. Implement the solution
   • After confirming your interviewer understands and agrees, you can code out the solution.
5. Validate the test cases to confirm correct behavior
   • You can either do this manually, or by writing tests. Check with your interviewer on what they prefer. On top of any pre-supplied test cases, add your own test cases here to ensure confidence.
6. Refine and optimize
   • Often, there are still optimizations left (code quality, runtime performance, memory, etc,). You can ask your interviewer what they'd prefer you to dig into.
   • Alternatively, you may have new follow-up requirement to implement, for which you can repeat the above steps until completion.

Anecdotally, I found tech screens and coding rounds to be the most relaxed interviews. The structure is clear, and once the solution and optimizations are complete, the remaining is free for you to chat and ask questions.

Onsite

In most processes, once you pass the tech screen you move to the onsite. Scheduling the onsite, however, involves strategy of its own.

Negotiation depends on having viable alternatives — that's what gives you leverage. Onsite scheduling affects this in two ways:

1. Offers expire — you can't hold them open indefinitely.
2. Leverage peaks right after an offer — the best time to negotiate is immediately upon receiving one.

We'll cover negotiation in more detail later. For now, let's talk about scheduling.

Onsite scheduling

When scheduling onsites, I considered two approaches:

1. Clustered onsites
   • All onsites are grouped within a 1–2 week window.
   • The main advantage is efficiency: you can take a single PTO block and complete multiple onsites without disrupting your day job.
   • The tradeoff is that you only get one "snapshot" of the market, with limited room to adjust if better opportunities appear later.
2. Rolling onsites
   • The pipeline never stops: sourcing, inbound responses, and tech screens continue while you move through onsite rounds.
   • Early onsites are clustered, but additional onsites are scheduled as opportunities arise. This keeps multiple offers in play, collects more data, and retains ongoing leverage for negotiation.
   • Lower-priority roles are often scheduled first, giving you time to practice before interviewing with your top choices.
   • Conceptually, this is similar to the secretary problem: you keep sampling until you're confident you've seen enough of the market. This strategy is especially useful if you're unfamiliar with current conditions.

I chose rolling onsites, the safer but more-demanding option.

Unfortunately, a series of life events cut into my timeline (more on that later). To stay on track with my three-month timeline, I condensed the schedule: some days included two onsites, or an onsite plus recruiter and tech screen calls. My instant coffee consumption probably went up tenfold for a month or so. Still, no regrets: the results were worth the sleep debt.

Coding Round

The coding round is usually the first onsite round. For whatever reason, it often gets scheduled earlier in the day.

It's similar to the tech screen, with one key difference: you may have a shadow interviewer. This slightly changes the dynamic during post-interview questions.

For deeper conversations, direct your questions to a specific interviewer based on their background. But you can also ask quick, open-ended questions that invite both to respond. For example: "In terms of resourcing, would you say your current team is bottlenecked on engineering, product, design, or something else?"

System Design Round

System Design interviews vary widely. In my search, I encountered differences in:

• High-level format
  • live system design vs project retrospective deep-dives
• Low-level format
  • whiteboard vs written document
• Area of focus
  • fullstack product engineering vs backend infrastructure

Because of this variation, System Design is one of the harder rounds to prepare for. I recommend the following:

• Speak calmly and confidently, this has a strong impact in qualitative rounds.
• Explain tradeoffs when selecing an option.
• For product-leaning interviews (e.g. Meta Product Eng), start by defining the product domain, outlining business requirements, and clarifying constraints and upstream APIs. These often have flexibility, which lets you simplify your design.
• Surface risk explicitly. When adding a component that introduces risk, point it out and explain mitigation.
• Review your past projects. Know not just the architecture but also the business drivers, timeline, and impact.
• Refresh scaling fundamentals, e.g. Kafka and similar tools: review internals, failure modes, and usage patterns.

For structured practice, I recommend hellointerview — both for their interactive practice problems and their YouTube walkthroughs.

Leadership or Collaboration Round

This round is meant to judge your leadership and collaboration skills. It is typically conducted by your HM or an equivalent engineering leader.

The STAR format (Situation, Task, Action, Result) is a common recommendation for answering questions. Think of it as a guideline rather than a strict rule. What matters most is providing clear situational context and walking through your structured thought process. The goal is to leave the interviewer confident you'd be a capable teammate or leader, can influence without authority, and make pragmatic, well-evaluated decisions.

Anecdotally, I've found it useful to talk with people from finance or operations backgrounds: they often think in more process-oriented ways than early-stage engineers, which can help you verbalize your own processes.

For Staff-level interviews, it also helps to have well-formed views on leadership and management philosophy. Personally, I've always enjoyed organizational theory and human dynamics, so these were fun interviews for me.

Team Matching Call

Team matching is usually the final step before an offer. By this point, you've already cleared the interviews. The purpose here is for both you and the hiring manager to confirm mutual fit.

Treat this call as your opportunity to interview the HM. Come prepared with questions on topics like team and org structure, role expectations, past challenges, and the roadmap ahead. This is your chance to understand what it would actually be like to join the team.

To make the most of it:

• Prepare questions in advance.
• Keep a notes doc open to capture details.
• Approach the call with neutral expectations.
• Do deep background research on your HM. (For example, before my Ramp team-matching call, I looked into my HM's previous startup, their past role transitions, and carefully read several posts on their personal blog.)

Verbal offer number call

The verbal offer call is unique: it's part of the company's interview subpipeline, but it also marks the start of your diligence and negotiation stages.

From the company's perspective, the call is a quick touchpoint to check your flexibility on numbers, build excitement, lightly sell you, and answer surface-level questions.

From your perspective, two things matter:

1. Negotiation
   • The numbers presented act as an anchor. Your response signals how much flexibility you have.
   • My approach: take the numbers, write them down, and thank the recruiter. If pressed on an opinion, I'd say I was waiting on additional data points for comparison, which was true.
   • The energy you show in your response should be balanced:
     • Show too little enthusiasm, and they may see you as a lost deal—not worth fighting comp committee for.
     • Show too much, and it signals emotional overinvestment and lack of options, weakening your leverage at the negotiation table.
     • The reason you state for your excitement is a key lens into your decision-making framework. Give excitement grounded in reason, rather than under-explained emotion, if you want to signal seniority and build trust.
2. Direct diligence
   • The recruiter is a resource. Use this call to gather information that matters to you.
   • My focus was on equity, since most offers I received were late-stage private equity. I recommend asking for, at minimum:
     • Past 5 years of ARR data
     • Equity details (RSU vs. ISO, vesting schedule, PTEW duration)
     • Anecdotally, companies with strong financials share them readily. Companies with weaker numbers often hedge and pivot to "future growth story" narratives instead.
   • This is a good time to mention scheduling calls with the rest of the team, part of your direct diligence covered below.

Direct Diligence

After compensation discussions, my next step was meeting the team I'd be working with. Since this stage takes about as much time as an onsite, I prioritized it only for offers I was most likely to accept.

Who to meet (in order of priority, especially the first two):

1. Engineering leadership (CTO, Head of Eng, or VP)
   • Purpose: understand how engineering is run as an org, and get a read on your leadership chain.
   • If you don't respect your leadership chain, you'll have a bad time. This call lets you gauge whether leadership feels out of touch, or conversely too in-the-weeds.
2. Product counterpart
   • Purpose: get the most realistic view of how the business is run day to day, at the ground level.
   • As a Senior or Staff product engineer, your PM is often your closest collaborator, so their perspective is critical.
3. Engineering peer
   • Purpose: understand what the role feels like from the inside.
   • Ask about what they enjoy, what challenges they face, and what you should watch out for.
4. Engineering junior
   • Purpose: get an unfiltered sense of how the org treats new talent.
   • If juniors are under-resourced or poorly treated, that's a red flag — the canary in the coal mine.

Since scheduling these conversations isn't automatic, you'll likely need to ask your recruiter or set expectations during the verbal offer call. Don't skip this step — it's the data you need to make a confident decision.

Backchannel Diligence

To build confidence in a company's valuation, I recommend speaking with two final personas:

1. An investor
   • Ideally a friend in VC (or, if not, other late-stage founders).
   • The goal isn't a detailed model — just an honest, high-level perspective on the company's financial trajectory.
2. A customer
   • Customers reveal where a company may be cutting corners.
   • Downstream complaints often signal upstream issues.

In Ramp's case, I spoke with both VC friends and a former skip manager who had worked with them as an enterprise customer. The feedback was strong: some literally said "I love Ramp", while others said Ramp was great but also pitched one of their portfolio startups (also a good sign).

Taken together, these conversations were a useful gut-check to confirm I wasn't missing any red flags from either the customer or investor perspective.

Negotiation

Negotiate comp

Once you're confident in the company and role, it's time to lock down your numbers.

Candidates love the idea of a bidding war between companies. HMs and recruiters dread it. In my case, I had other strong offers, but with Ramp I wanted to maintain excitement on both sides — it felt like a great fit, and their offer was already competitive.

My strategy was to define the numbers I'd be comfortable signing even without every offer in hand, and present those as my "yes numbers". I added the caveat that I was awaiting another hiring committee decision for a top-choice role the following week, so my stance might change by Monday. Ramp moved quickly: same-day approval came through within hours.

Negotiate contract

Once you have the contract, read it carefully.

Even at this stage, you still have negotiation leverage: the hiring manager and recruiter want to close the process as much as you do. That said, any new ask typically involves legal, which adds friction and can slow things down.

In my case, my contract came in on Friday. Everyone wanted to close before the weekend. I quickly reviewed it and decided to use my leverage on a single ask. However, in my rush to close (and with ChatGPT's half-baked contract advice), I asked for an equity clause I didn't actually need... The contract already had better terms. Because I framed it as my "will sign" ask, I lost some opportunity to negotiate further. That said, the fairness of the equity clause and smoothness of the email thread with Legal gave me more confidence in the company.

And with that, I signed! Officially a "Rampling".

Monitoring and Incident Management

Visibility into your process is essential. Many applicants rely only on their email inbox, but a smart candidate builds a dashboard.

Your dashboard should answer key questions such as:

• Which companies am I actively interviewing with right now?
• Where is each company in the process?
• When was the last time I heard from them?
• What are my top 5 active companies, ranked by expected total comp or base pay?
• Which companies do I plan to get referrals for?

A simple spreadsheet works well. Recommended columns:

• Company name
• Expected level
• Process stage (see picture below)
• Date of latest update (last email, application, or interview)
• Compensation numbers
  • Early: expected TC and base
  • Later: final TC and base for comparison
• NYC office (or whatever your key location criterion is)
• Outbound method: referral, direct apply, cold outbound — as well as contact name, if applicable.
• Notes

Each time you source a company or receive inbound, add a row. Update it whenever the status changes, e.g. a new interview scheduled or an email from a recruiter.

I started with a simple Google Sheet, one company per row. It allowed for fast entry, easy sorting, and a quick at-a-glance view of my pipeline.

Numeric prefixes: Adding a numeric prefix to the process stage column enables ordered sorting.

Conditional formatting: Applying formatting to the "date of latest update" column gives you instant visibility into which processes are moving and which ones may need a follow-up.

A couple useful spreadsheet tricks.

Later in my process, I switched to Notion. It was slower to update, but had one killer feature: a proper notes document could be embedded directly in each row. The ease of organized note-taking justified the switch: for every recruiter call, interview Q&A, or HM conversation, I could quickly open the linked doc to take notes or reference pre-written questions.

Incident playbook

During my three-month search, I faced both planned disruptions (a months-long home renovation) and unplanned crises (medical issues, bathroom leak into a neighbor apartment, and a major family emergency abroad).

The small silver lining was that none of these incidents caused me to lose offers. By proactively communicating updates and setting expectations, I retained trust, and recruiters remained flexible and friendly, even when quickly pivoting.

When communicating a process change, I recommend the following:

1. Give context
   • Explain why a process is being paused, without oversharing.
2. Give expected timeline for resolution
   • Even a placeholder like "I'll update by EOD tomorrow, nothing resolved yet" maintains trust.
3. Set expectations around responsiveness
   • If replies will be delayed (e.g., different timezone, other priorities), state it upfront.
4. Track status
   • Mark paused items on your dashboard so you can unpause quickly once things stabilize.

I applied this approach across both professional and personal stakeholders to keep everyone aligned. For example, when I flew unexpectedly to Vietnam, I paused more than seven active company processes and notified all parties. On return to JFK, I immediately restarted: scheduling three onsites and several calls in the first week. For slow processes, I was able to apply timeline leverage using existing offers and an ongoing team-matching process with a major social media company.

Numbers and Market commentary

Below is a Sankey diagram summarizing the stats across stages of my search:

Note: for simplicity, nonstandard stages have been normalized.

Second note: the "Paused" stage corresponds to stages left in a paused state until offer signing, all other processes were unpaused.

Some market commentary based on my experience:

• Large orgs are slowing hiring, while smaller orgs remain more nimble.
  • Public companies seem more exposed to recent macroeconomic shifts than startups.
• Hiring is consolidating around core offices and proven models.
  • Just as investors retreat from risk, orgs appear to pull back to their strongest locations (often in-person, HQ offices).
  • Anecdotally, one of my largest orgs stopped ingesting new NYC-based candidates — fortunately after I had entered the pipeline.
• Physical onsites have mostly disappeared, except where relocation isn't required.
  • Across dozens of processes, only one strongly pushed for in-person onsite, and only a handful even offered it as an option.
• Base salary and equity strength tend to correlate.
  • Successful startups tend to outperform on both cash and equity, likely because less funding pressure frees budget for hiring.
• Companies that are hiring may be struggling to fill roles.
  • Many roles had 3+ separate independent recruiters reach out over the span of my search, a handful had 5+. This suggests difficulty closing candidates.
• AI is slowly entering more processes, but is not very common.
  • Only a couple companies allowed or encouraged AI as part of their process.
  • Ironically, none of the foundational AI companies permitted AI in their own processes.

Afterword

The best advice I received was simple: protect your mental clarity. Find an activity that restores your energy and make it routine. For me, it was long walks in the park, often while calling a friend or mentor. Martin used mountain biking to the same effect. Anything that combines light cardio with a flow state seems to work well.

Don't forget your support system. I leaned heavily on my wife, friends, mentors, and former colleagues for advice and sanity. Though the outcome was positive, there were plenty of ups and downs from both life and the process itself.

If you enjoyed this post, consider sending it to a friend.