The last post was about the development toolbar Muru adds to each HTML page under muructl run, with the record Muru keeps of each request. That record can say a lot about how a request was served, how long it waited for a worker, how long the application took, what it logged and what went wrong. What it can't say is what the application did with its time, because Muru only sees the application from the outside.For that, Muru uses wrapture, the package I wrote for monkey patching, testing and tracing, which I covered in a series of posts earlier this year. With tracing turned on, wrapture's instrumentation records the calls the application makes while it serves a request, such as its database queries and calls to other services, and the toolbar gains panels showing them.Turning tracing onThe shop from the last post is a Flask application which uses sqlite3 for its database, and urllib.request to call its stock service. Tracing needs wrapture and its instrumentation packages added to the project:$ uv add wrapture wrapture-instrumentationthen tracing = true, and a wrapture configuration saying what to instrument, which can go in the same pyproject.toml:[tool.muru]entrypoint = "shop:app"tracing = true[tool.muru.run]toolbar = true[[tool.wrapture.instrument]]name = "flask"[[tool.wrapture.instrument]]name = "sqlite3"statement = true[[tool.wrapture.instrument]]name = "urllib.request"The flask instrumentation records each request with the view that handled it, sqlite3 records each query, with statement = true so the SQL itself is kept, and urllib.request records calls made with it. The configuration can also be in a wrapture.toml file beside pyproject.toml, or a file named by WRAPTURE_CONFIG, which is where wrapture itself would look for it. I have also put toolbar = true in the [tool.muru.run] table this time, so the toolbar is on without needing --toolbar each time.When the run starts, the host says where the configuration came from and what it instruments:muru.host[36012]: tracing on: wrapture configuration from [tool.wrapture] in ./pyproject.toml, instrumenting flask, sqlite3, urllib.requestNot everyone working on a project may want tracing on, since it adds work to every request. tracing = true in the muru.local.toml beside the project turns it on for my own runs only, with the configuration still coming from the project. Tracing needs Python 3.12 or later, and if wrapture is missing, or its configuration can't be found or applied, the run logs a one-line warning saying why and serves the application untraced, rather than failing to start.What the front page doesWith tracing on, the front page's badge now shows two hints, and the toolbar opens on them:The toolbar also has new tabs, Trace, SQL and Outbound. The Trace panel is the request's tree of operations, on the same clock as the Timing panel from the last post:The view ran one query for the books, and then for each book, a query for its author and a call to the stock service. This is the code which did that:@app.route("/")def books(): rows = database().execute("SELECT id, title, author_id FROM book ORDER BY id").fetchall() books = [] for book_id, title, author_id in rows: query = "SELECT name FROM author WHERE id = ?" (author,) = database().execute(query, (author_id,)).fetchone() stock = in_stock(book_id) if stock == 0: log.warning("%s is out of stock", title) books.append({"id": book_id, "line": f"{title} by {author}, {stock} in stock"}) return render_template_string(PAGE, title="Books", books=books)def in_stock(book_id): with urllib.request.urlopen(f"{request.host_url}stock/{book_id}") as response: return int(response.read())Nothing about it looks wrong when reading it, and with eight books, the whole page took under ten milliseconds. It is a different story with a thousand books, a database on another machine, and a stock service on the other side of a network.Queries and calls, groupedThe Trace panel shows the pattern when there are only a few books, but with hundreds it would be a very long list. The SQL panel groups the request's queries by statement, with how many times each ran and their total time, and then lists each query in order:Statements which differ only in the values written into them group together, so the author query shows up as one statement run eight times. This is the N+1 pattern, one query for the list and then one more for each row of it, and it is one of the most common performance problems in web applications which use a database. It is also easy to write without noticing, since an ORM will happily generate it from code that looks perfectly reasonable.The Outbound panel does the same for calls to other services, grouped by method, host and path:A path segment which names a row rather than the endpoint, such as a number, a UUID or a long hexadecimal string, is grouped as {id}, so the eight calls to /stock/1 through /stock/8 are one endpoint called eight times. Calls made with requests, httpx, urllib3, aiohttp and http.client are recorded the same way, given the instrumentation for them, as are gRPC calls.The fix for both is to ask for everything at once. A join gets the authors with the books, and the stock service can answer for several books in one call:@app.route("/")def books(): query = ( "SELECT book.id, book.title, author.name FROM book" " JOIN author ON author.id = book.author_id ORDER BY book.id" ) rows = database().execute(query).fetchall() stock = in_stock([book_id for book_id, _, _ in rows]) books = [] for book_id, title, author in rows: if stock[book_id] == 0: log.warning("%s is out of stock", title) books.append({"id": book_id, "line": f"{title} by {author}, {stock[book_id]} in stock"}) return render_template_string(PAGE, title="Books", books=books)def in_stock(book_ids): ids = ",".join(str(book_id) for book_id in book_ids) with urllib.request.urlopen(f"{request.host_url}stock?ids={ids}") as response: return {int(book_id): count for book_id, count in json.load(response).items()}Saving the file was enough for the run to start a new release with the change:muru: reloading, /Users/graham/shop/shop.py changedmuru: switched to release r2and the front page's record now has one query and one call:$ muructl debug request 01M7PBFZP78AN9K8YRZ0DC3AMY...SQL By statement SELECT book.id, book.title, author.name FROM book JOIN author ON author.id = book.author_id ORDER BY book.id 1 0.098 Each query 0.419 0.098 SELECT book.id, book.title, author.name FROM book JOIN author ON author.id = book.author_id ORDER BY book.idOutbound By endpoint GET 127.0.0.1:8080 /stock 1 0.991 Each call 0.604 0.991 GET http://127.0.0.1:8080/stock 200The two hints went with them. Hints are where I would start with any page, since they apply the same thresholds to every request without me having to remember what to look for. A statement run five or more times in one request, or an endpoint called five or more times, is flagged, as is a single query or call taking a quarter or more of the request's time and at least 10 milliseconds.The failing page againThe author page from the last post still divides by zero for an author with no books. With tracing on, its Trace panel shows where the exception went:The view, author, is marked as failed, since the exception escaped it. The request is marked as well, because Flask's instrumentation notes the exception Flask's error handling caught against the request. The Exceptions panel lists both, the one Flask handled being marked as such:GET /authors/4 ZeroDivisionError: division by zero (handled)author ZeroDivisionError: division by zeroThe bars also show something the Errors panel didn't. The view itself was done in about a third of a millisecond, out of a request which took about three. Most of the rest went on Flask handling the exception, which includes formatting and logging the traceback.Applications without framework instrumentationThe shop gets its request spans from the flask instrumentation, and there is the same for Django and other frameworks. An application without such instrumentation, such as a plain WSGI or ASGI callable, can have Muru wrap its entry point in wrapture's request middleware instead:[tool.muru]entrypoint = "api:application"tracing = truetracing_middleware = trueThe same key in a mount's table wraps just that mount, for an application which mixes WSGI and ASGI. It shouldn't be used where a framework's instrumentation already records requests, since the middleware would then record them instead, and the framework's own request settings would no longer apply.Tracing and the workersHow tracing is set up follows from how Muru starts an application, as covered in the import once, fork many post. The host process applies the wrapture configuration before it imports the application, so the instrumentation's patches are in place for the application's own imports. What the configuration sends events to, its sinks, are only started in each worker after it is forked, so the host holds no threads or open files of wrapture's when it forks.The toolbar's panels don't need a sink at all, since Muru collects what wrapture records for each request itself. Any sinks the configuration does declare still work as they would without Muru. Adding wrapture's printer sink, for example, prints each request's tree in the run's output, from the worker which served it:muru.worker[0]: GET /authors/1 (shop.wsgi_app)muru.worker[0]: author(author_id=1)muru.worker[0]: sqlite3:connect('')muru.worker[0]: sqlite3:connect -> '' [220us]muru.worker[0]: sqlite3:Connection.execute(sql='', parameters='')muru.worker[0]: sqlite3:Connection.execute -> '' [136us]muru.worker[0]: sqlite3:Connection.execute(sql='', parameters='')muru.worker[0]: sqlite3:Connection.execute -> '' [26us]muru.worker[0]: flask:render_template_string(source='\n\n