Music streaming DJ - Langfuse

Music streaming DJ

This is an example to illustrate the concepts of the Langfuse Academy.

Context

A music streaming app adds a DJ feature in beta. The DJ keeps a queue going based on what the user has played before and what the algorithm thinks the user will like. Every few songs it says a short word about what's coming up and why it chose it. Listeners can also optionally interact with the DJ to steer it, by clicking a microphone button and talking to it.

Good evening

Alex
Your DJ
Mixing songs for your evening

It picks the tracks, talks you through them, and listens when you talk back.

Now playing

DJ · why this song

A deep cut after that album you ran on repeat last week — easing you in before I pick the pace up.

Slow Tide

Marlowe

0:48 3:36

Talk to DJ

Listening…
Play something with a bit more energy
Try saying
Who is this? Skip this one More like this
Tap the orb to stop

Two characteristics drive how to approach the AI engineering setup:

The risk of a bad output is low. The worst case is a skipped song or an awkward DJ comment, and the feature is labeled beta.
This is a new feature, so there is no historical data to start from, and the team will need to learn from live listener behavior.

Because of this, it makes sense to go live as soon as possible and iterate based on live feedback from the beginning.

Tracing the DJ

There are two different kinds of traces that will together form a session:

Trace name	Details
`plan-next-set`	Plans the next set of tracks and writes the commentary, triggered by the DJ itself every few songs or by a DJ request.
input:	listening context and any instructions
output:	next tracks queued, plus the commentary line
`handle-dj-request`	Handles a voice request from the listener, triggered by the microphone button.
input:	voice clip and the current queue
output:	a short reply, plus instructions for the `plan-next-set` run it triggers

A listening session and both trace types up close:

Example session
Example plan-next-set trace
Example handle-dj-request trace

Session: listen_7f3e · user: u_8841

plan-next-set: "Kicking off with two favorites from your week."
2.8s
plan-next-set: "Staying in this lane with some mellow electronica."
2.4s
handle-dj-request: "Play something calmer."
1.6s
plan-next-set: "Calming it down, here is some ambient piano."
triggered by: dj-request, 1.5s

Capturing user behavior

In order to learn from users, we will log some interesting behavior on the applicable traces.

track_skipped: The listener skips a DJ-picked track shortly after it starts
dj_replaced: The listener turns the DJ off but keeps listening
message_type: Classifies each voice request as steering, correction, repeated_instruction, positive_reaction, or negative_reaction

In principle, the team could already iterate with only this in place. Tracing and monitoring form a small loop of their own. In the beginning, this is probably enough for the team to quickly improve the DJ feature.

Structured testing

With only the live signals, testing a change means shipping it and watching the scores. The team can add two more deliberate ways of testing: experiments on datasets, and A/B tests on live users.

Experiments on datasets

In this use case, testing end to end is very hard offline: whether a session was good only shows in live listening behavior, and taste differs per user, so no expected output holds for everyone. A single step like select-tracks can be tested, with the expectation describing the direction of the set rather than exact tracks:

selection-directions: Listening contexts paired with good and bad directions for the next set.
direction_match: Does the selected set go in one of the good directions and avoid the bad ones?

A/B tests on live users

Some changes are hard to grade offline, like the tone of voice of the commentary. Since the risk is low, the team can give a small group of listeners the new version and compare the signal scores between the groups, like the skip rate and the message_type distribution. If the new version does better, they can roll it out to everyone.

With this in place, the full loop is running:

Deploy
Trace every set and every request
Monitor skips, dj_replaced, message_type, A/B comparisons
Build datasets from what we see while monitoring production
Experiment selection algorithm, DJ prompt, ...
Evaluate grade step outputs against their expectations

Conclusion

This is an example of a feature that's low risk and there is no historical data to learn from. The best thing you can do in such cases is getting traces in as soon as possible and start monitoring them to learn from. Everything else, datasets, experiments, A/B tests, can be built on top of that over time.