The first thing you'll always do when starting a new #bpftrace script is selecting a probe. In this thread we will discuss how to select the probes that will give you the information you desire. 1/

2/ This is arguably the most difficult step. First, we need to know the landmarks. Use the very basic command of "bpftrace -l" to list all the probes. On my system, that produces 40032 lines

3/ Each line consists of N-tuples, some 2-, and some 3-tuples. The tuples are colon-separated and can be thought of as pathnames to events that fire to inform you something of interest has occurred

4/ The landmarks to find here are the top-level probe elements. Let's count them to see how bpftrace is organized:

$ sudo bpftrace -l | sed -e 's/:.*//' | sort | uniq -c
10 hardware
38527 kprobe
11 software
1484 tracepoint

5/ The two big landmarks here are kprobes and tracepoints. Nearly every event you will be interested in will fall into one of these two categories.

6/ Finding the landmarks are important because unlike DTrace, where you can blindly hook on to, say, all the probes, you can't do that with bpftrace. You're limited to 512 probes you can attach to in one-go

7/ Next, before we do an exploratory trace, we need a command that does what we want to observe. For our immediate needs (writing network observation tools), let's go with:

curl -sLo- http://google.com 

Make TCP/80 request for index and dump to stdout

8/ Now we need to come up with a probe glob that selects a wide swath of what we are interested in but not more than 512 probes. I came up with:

$ sudo bpftrace -l 'kprobe:*tcp*' | wc -l
381

9/ Now we have our program to run and a set of probes we are interested in, we can use the following syntax to do an exploratory trace:

bpftrace -c "command to run" -e "bpftrace code"

bpftrace exits when your command completes

10/ Next, it is important to know two bpftrace code landmarks. Just like awk, it supports BEGIN {} and END {}

I find it helps to throw some silly printf()'s in one of these each to help separate the output

Here's the whole recipe with output https://pastebin.com/WJLccCU5 

From this output we can select the few probes that we are interested in. The ones that stand out like a sore thumb are:

18 curl[4405]: kprobe:tcp_connect
39 …tcp_sendmsg
68 …tcp_recvmsg
110 …tcp_sendmsg
141 …tcp_recvmsg
150 …tcp_recvmsg
162 …tcp_recvmsg

/11 End thread.