swarnim's blog

Some thoughts on harnesses and the agents they shape

Sun, 13 Sep 2026 00:00:00 GMT

Harnesses today are a snapshot of the constraints and invariants of the current generation of models. As models improve and what we use them for becomes increasingly more ambitious, some of those assumptions can become unnecessary constraints.

The folks building harnesses do a very good job of making design decisions that maximize the model's ability to drive meaningful outcomes, the best harnesses can also sometimes provide a glimpse into the future by building capabilities that the model can yet only enable crudely, but even so, if they are useful enough, (and sufficiently verifiable), the models are post-trained to be more effective at that capability.

This is much of what the agent improvement loop looks like, at least for now.

I want to explore how much of it can become an optimization problem for the agent itself, shaping its working methods around the task and learning from the results.

Now, agents are already quite involved in this process, the people building harnesses, and training models are all quite ai-pilled and heavily leverage AI in their work.

I can't speak to the research end of that spectrum, but if you've spent any amount of time actually trying to get agents to build harnesses, you would know that they are not very good at it...

Models have terrible intuitions for context management, tool design, and being thoughtful about not busting the prompt cache.

I can't be certain as to why that is, maybe it's because good harness engineering patterns are out of distribution relative to the model's training data because of how novel the category is. Maybe it's hard to RL on, since the "reward function" for good harness design is largely confounded and signals are full of noise, and can only truly be evaluated long-term.

What is more certain is that models can accomplish great things by composing simple, strong primitives together, and with iterative improvement over a measurable goal.

So then, another way to frame the problem of a recursively self-improving harness perhaps is figuring out which parts of the harness are immutable aspects of its runtime, so things like the agent loop, context management primitives, etc, vs which parts should the agent be able to modify by itself.

Harness Design <> Model Intelligence

It is hard to forecast exactly how much the future of recursive self-improvement (RSI) will rely on harness engineering, but it seems unlikely to me that the near term path to RSI starts as a model directly rewriting its weights.

I suspect thoughtful harness design will take on more of the responsibility for making that improvement process possible in the near term.

This means improving the machinery for getting better answers rather than just the answer.

Harness engineering as a discipline will evolve more toward building general systems for letting the agent improve its own methods, we build general mechanisms that let it try different approaches, learn from the results, and change how it works, with fewer heuristic rules prescribing the approach.

As those systems mature, they can support autoresearch workflows for improving the models themselves. More capable models may then need less guidance from the harness, letting us remove some of the complexity that helped earlier generations work reliably.

Harness Design Patterns for Continual Learning

quick plug: Noesis is my attempt at reifying some of these ideas into a self-evolving agent that's delightful to use.

The "harness" is the runtime the agent operates within. It defines how the model observes, acts, memorizes, checks itself, and improves.

The design of this runtime should be deliberately simple and generic to enable generalization, referencing existing software engineering patterns is also helpful, because it gives the model something to draw on from pretraining, so it can spend more effort on the task and less on figuring out how its environment works.

I also find operating systems and the UNIX philosophy to be a useful analogy. Similar to an OS, a harness should encapsulate complexity while keeping the interface simple.

Dynamic Workflows

Code is the universal language for defining programs and systems. When several tool calls contribute to the same task, I think code is the natural place to express how they fit together.

The model could fetch data from a few MCP servers, transform the results, and pass the relevant parts into another operation, all within the abstraction of a "program". It can decide which operations run concurrently and how to recover from failures.

Doing this through code means the foreground model doesn't need another inference round just to move data between tools. Intermediate results can remain in the execution environment until there is something worth bringing into context. The tool calls still happen, but the model gets to decide how much of that machinery it needs to see.

And whenever a step requires intelligence, the program should be able to invoke a model. Intelligence becomes a function you can call as part of a larger computation.

A program can initially be written as a script for a particular task, and then if the agent thinks it could be useful again, the agent should be able to give it an interface, save it, and keep iterating on it as it continues using it across sessions.

Longer, more involved workflows may need explicit phases and enough recorded state to resume after an interruption or retry a failed step. The runtime should provide those mechanics so the agent can define a workflow without having to rely on external durable execution mechanisms or worse, unnecessarily reinvent a bespoke one.

In Noesis, these are Programs, which can either take the form of scripts or durable phased workflows. Programs can be bundled alongside a skill for easy portability, and they can just as well be invoked either by the agent, or by the human.

Context Engineering

I think it helps to distinguish between what the agent has access to and what it is attending to right now.

A long running session might produce far more material than belongs in any one model request, but that material can still be useful later. We should be able to remove it from the immediate context without making it inaccessible.

Files are a useful primitive here. Large tool outputs and other artifacts created by the agent can live outside the context window, where the agent can inspect them programmatically. The agent can then grep, sed, awk its way through any large corpus of text to get the insight it needs in any subsequent turn.

The same principle applies to the session transcripts (both current and past). Noesis for example, makes transcripts from prior sessions accessible through codemode, so an agent can semantically search relevant chunks from prior sessions, slice parts of its context window, do transformations on them, use it as context for a subagent, etc.

Compaction

Compaction is super important for long running agents, as the agent works on a long horizon goal and makes its way through various smaller tasks. It accumulates a lot of context that isn't necessarily relevant going forward.

The context window is a scarce resource, you only have so many tokens to play with until you find yourself contending with the effects of attention degradation.

The principal goal of context engineering is to reduce noise, and ensure that only the tokens most relevant to the task at every point remain in the context window.

The way Noesis implements compaction lets it carry notes from one context window into the next, preserving details it has gathered without repeatedly compressing those notes into a single summary. Earlier messages and tool outputs remain searchable, so if something relevant wasn't captured in the notes, the agent can go back and find it from the original session transcript.

Orchestration

A harness should be able to spawn multiple subagents to work in parallel and the main agent should have the ability to launch, monitor, inspect, steer, and reintegrate subagent outputs in the main thread.

This is useful when the main agent needs to test multiple hypotheses, run experiments concurrently, or delegate isolated subtasks without polluting the main context.

Much of this should be expressible through code. The agent can decide how it wants to compose and orchestrate subagents and define the dependencies between them.

For this to work, subagents need stable addressable identities. The main agent should be able to continue its own work, then come back to a subagent it spawned earlier with more context or a follow up. Other subagents should also be able to communicate with one another if they deem it necessary.

Delegation requires observability to be most effective. A reasonable conclusion may depend on an assumption that doesn't make sense in the context of the broader task. The parent should be able to examine the relevant parts of the subagent's trace and artifacts, then follow up with, or steer it. The same programmatic context management principles apply here too.

Closing Thoughts

A true fully RSI solution would also improve the model, the compute underneath it, and the power feeding it all. Perhaps an RSI harness is a step on that path.

I don't know how far that goes. But it seems worth building environments where we can find out through continued use, and where useful discoveries can carry forward into increasingly ambitious work.

If you find all of this as interesting as I do, I'd love for you to give Noesis a swing. It's the self-evolving agent I'm building to explore these ideas and make a general-purpose harness I'd enjoy using.

Tools and Curiosities

Sun, 24 May 2026 00:00:00 GMT

"Don't Follow Your Dreams, Follow Your Tools"

<p>The omniscient YouTube algorithm recently resurfaced a Hank Green video on my feed, with that very title.</p> And I have been thinking about it.

I think it is more generous than "follow your dreams," because dreams are often just old pictures of yourself. They are formed from the vocabulary you had at the time. They inherit the prestige objects of the culture around you. They assume the world will keep offering the same destinations it offered when the dream first appeared.

Tools are more honest. They are a current account of what you can do, what you can see, what you care about, who trusts you, what you have learned, and what kinds of problems you keep finding yourself most drawn to.

A toolkit includes the obvious things: technical skills, writing ability, taste in interfaces, knowledge of a domain, the ability to model complex systems in your head. But it also includes the less obvious things: relationships, accumulated scars, values, recurring irritations, the specific kind of friction you notice before other people do.

It includes you.

A tool is not just a thing outside the self. It is any capacity that changes what you can do next.

You see this most clearly when you make something.

Before you know how to build software, many problems remain complaints. A workflow is slow. Some information lives in the wrong place. A person is copying data between systems because nobody has given the work a better shape.

After you know how to build, the same problem looks different. It becomes material. You can see where a script would help, where an interface is missing, where a system boundary is wrong. The world did not change. Your handle on it did.

Learning a tool changes the board.

This is why "follow your tools" is better than it first sounds. It is not saying: only do what you already know how to do. That would be a recipe for stagnation. It is saying: begin from actual contact with your current capacities, then use projects to stretch them.

The usual dream-frame asks: what do I want to become?

The tool-frame asks: given what I can now see and do, what is the next live problem?

That second question is much better.

Curiosity is the pointer

Tools by themselves are inert. They need a search policy.

Curiosity is one of the better ones.

By curiosity I do not mean novelty addiction. I do not mean opening twenty tabs and calling the feeling of motion research. I mean the thing that keeps snagging your attention. The broken workflow you keep complaining about. The question you keep trying to explain to someone else. The topic where the available answers feel fake. The capability that feels newly possible but not yet well-shaped. The area where everyone else seems strangely resigned.

Curiosity usually arrives before a thesis. This is why it is easy to distrust. It does not yet have the dignity of a plan. It is just a tug.

But a tug is information.

A lot of good work starts as an inability to leave something alone. You notice a gap. You notice that people are using clumsy language for a real thing. You notice that a system works, but only because some human is absorbing the mess in their head. You notice that everyone is arguing about the wrong part of the problem.

At that stage, the job is not to write the grand plan. It is to build enough contact with the problem that the next question becomes sharper.

Curiosity is a pointer. It is not proof. It has to be tethered.

The tether is a project.

Projects are probes

A project is curiosity made concrete.

You do not need the full plan. In fact, if you already know exactly what the thing will become, the project is probably less interesting. The project is where the plan earns detail.

The point of a project is not only to produce the artifact. It is to send a probe into reality.

The probe asks:

Is the problem real?
Does anyone care?
Was the hard part technical, social, distributional, aesthetic, or organizational?
Does my current toolkit transfer?
What tool am I missing?
Do I still care after the first burst of interest?
Can this become useful to someone besides me?

A serious project should leave behind at least one of three things: value, taste, or tools.

Value means the thing helped someone. Taste means your predictive model improved. You now understand some class of decision better than before. Tools means you acquired a reusable capacity: a library, a pattern, a contact, a distribution surface, a sharper concept, a scar.

This is why failed projects are not always failures. Some projects fail as products but succeed as probes. They teach you where the hard part really was. They reveal that the market did not care, or that the interface was wrong, or that the technical problem was easy and the coordination problem was the whole game. They show you that your toolkit was missing something.

Then you learn the missing thing.

And that new tool changes what projects are available next.

This loop is much more useful than the dream-frame. The dream-frame treats deviation as distraction. The tool-frame treats deviation as exploration, as long as it makes contact with reality.

That last clause matters.

It is very easy to fake this loop. You can start projects that never leave your private world. You can collect beginnings. You can confuse the feeling of expanded possibility with the reality of expanded capacity. You can stay in the pleasant phase where the project is still mostly a story about who you might become.

Shipping is what closes the loop.

Not always shipping as in a polished public launch. Sometimes the right unit is a prototype shown to one user, a script used by a teammate, a memo that changes a decision, a small tool that saves someone ten minutes every week. But there has to be some surface where the world can answer back.

You can improve something bad. You cannot improve nothing.

Work with gravity

There is a kind of work that feels like falling downhill.

This is not the same as easy work. It is work where your tools, values, interests, and energy are already arranged in the direction of the problem.

Ambitious people are trained to be suspicious of this. If it feels easy, surely it cannot count. Work is supposed to feel like swimming upstream. The difficulty becomes part of the proof.

Sometimes that is right. Some important work is hard because the problem is hard. Some obligations have to be met whether or not they feel alive. Some stretches of any real project are just unpleasant.

But some work is hard because it is badly matched to your tools. Or because it is mostly prestige. Or because you are trying to become a person you once imagined instead of looking at the person, toolkit, and world that actually exist now.

There is no virtue in that kind of difficulty.

Working with gravity is what it feels like when your tools, taste, curiosity, and the problem all point in roughly the same direction. You still work. Sometimes you work a lot. But the work carries you forward instead of asking you to constantly manufacture motion.

The test is whether the downhill path compounds or merely soothes.

A compounding downhill path leaves you with stronger tools, sharper taste, more useful artifacts, better relationships, or better problems. A soothing downhill path leaves you with the same self, just better entertained.

This is the guardrail. "Do what you want" is not enough. Wants are noisy. Some wants are signals. Some are avoidance. The difference is contact with reality.

If the work keeps making the world answer back, trust the pull more.

Usefulness disciplines curiosity

One of the best ways to keep curiosity honest is to aim it at something useful.

Useful does not have to mean profitable. It does not have to mean large. It means there is a counterparty. Someone has a problem. Someone can use the thing, reject it, misunderstand it, depend on it, ask for it again, or tell you it did not matter.

Other people's problems resist your story about the project.

This is why tools are such good projects. A tool has to meet another mind at the interface. It has to be legible enough to use. It has to encode some opinion about the work. If the opinion is wrong, the user will feel it quickly.

A useful tool also creates leverage beyond the original act of making it. It lets someone else move. It removes you from the loop, or at least reduces the number of times you have to personally carry the same context across the same gap.

This is one reason composability matters. A good tool solves a case and creates an affordance. It becomes part of someone else's toolkit. They can combine it with capacities you did not predict, in situations you did not design for.

Hamming has a line that the essence of science is cumulative. You should do your work so others can build on top of it. This is true outside science too. A project becomes more valuable when it is not only an answer, but a surface for future work.

Useful tools are a way of making curiosity cumulative.

The toolkit compounds

The visible output of a project is the artifact. The durable output is often the toolkit that remains after the artifact succeeds or fails.

This is why small projects can matter more than they look. They plant acorns. You learn a domain. You meet a collaborator. You acquire a distribution channel. You develop a reusable pattern. You get one painful lesson about what users actually do. You write one script that becomes the seed of a system. You learn that the elegant part was irrelevant and the boring part was load-bearing.

Then the next project starts from a different place.

This is the loop:

Tools let you see or do something.
Curiosity points you at a live edge.
A project creates contact with reality.
Reality contact expands the toolkit.
The expanded toolkit reveals better curiosities.

This may feel a bit scary. It does not give you the comfort of a fixed destination. It asks you to keep moving while the map is changing.

But this is how a lot of real careers seem to work. With the compounding of tools and judgment across projects.

The particular thing you do may involve luck. That you keep becoming the kind of person who can do interesting things is less luck.

But you still need an arena.

An arena can be a user, a production system, a deadline, or a peer whose taste you respect. What matters is that the work has a surface where being wrong costs something and being right changes what happens next.

Without that surface, the loop can collapse inward. Curiosity becomes stimulation. Tools become accumulation. Projects become beginnings.

Better maps

I do not think the alternative to dreams is cynicism.

The alternative is attention.

Pay attention to what you can do. Pay attention to what keeps pulling on you. Pay attention to the problems that become visible only because of the tools you already have. Pay attention to where the world pushes back. Build something small enough to touch reality and serious enough to change your tools.

Dreams can still be useful. They reveal values. They tell you something about what you once found beautiful or urgent. But they should not be allowed to harden into old maps.

The better aim is to keep producing better maps.

Follow your tools. Follow your curiosities. Take on projects before you fully know how they will work. Let them teach you. Let some of them fail. But make them real enough that the world can answer.

Don't follow your dreams into a picture of who you used to want to be.

Follow your tools and curiosities into an arena where the work can change what you are capable of seeing next.

The Discerning Machine

Mon, 18 May 2026 00:00:00 GMT

<aside class="!my-0 font-serif italic text-base leading-snug text-neutral-500"> Originally published on <a href="https://devfolio.co/blog/the-discerning-machine/" target="_blank" rel="noopener" class="text-neutral-300 underline decoration-neutral-700 underline-offset-[3px] transition-colors duration-200 hover:decoration-neutral-400">Devfolio's blog</a>. </aside>

What we learned from scaling hackathon judging by separating machine thoroughness from human taste.

We recently ran an experiment for the Push to Prod hackathon to see if an AI system could help scale hackathon judging while preserving human judgment and taste.

Hackathon judging is usually described as a question of quality: which projects are best? In practice, it is also a question of attention. By the time submissions close, a small panel has to read descriptions, watch demos, inspect repositories, compare projects against a rubric, and decide which teams deserve glory and prizes. The work is not only subjective, A lot of it is plain inspection: what was actually built, which claims are supported by the code, and where the pitch and the artifact come apart.

That distinction became the design frame for the system. Some parts of judging are thoroughness-bound: reading code, checking claims, citing evidence, applying the same rubric across a long queue. Other parts are taste-bound: deciding how much the evidence matters, whether the hard part was actually solved, and which project deserves to win. We built for the first part so humans could spend more of their attention on the second.

The result is not an AI judge in the usual sense. It is closer to an evidence layer. An agent reads the team's submission, walks the repository with tools, writes a cited audit, and turns that audit into structured scores a human can inspect, override, or ignore. The decision stays with the human panel. The machine's job is to make sure that decision begins closer to the work itself.

Here are the kinds of audits the system produces:

Credited AI Depth at 9/10. The submission describes the product as a meeting summarizer. The code implements a more specific workflow: lib/agents/briefing.ts generates different artifacts for the chair, the scribe, and the operations lead; lib/schema/action-items.ts preserves owner, deadline, confidence, and source span; app/review/page.tsx makes those extracted obligations editable before publish. The implemented system turns messy discussion into accountable work rather than stopping at generic summarization.

It also catches gaps hidden deep in the codebase:

Reduced Technical Execution from 8 → 5. The submission claims "multi-tenant isolation with row-level security," but db/policies.sql only checks that a user is authenticated, and lib/db/server.ts performs writes through a service-role client that bypasses those policies. The app may still work as a demo, but the claimed isolation is not implemented.

These examples are composites from a few real samples, but they show the shape of the assessment, the system reads the pitch against the code and leaves behind an audit a human can check. The system uses a human-written rubric to localize the judge's attention; the decision about who deserves to win stays with the human panel.

We built a system that produces audits like this for every published project in a hackathon. An agent runs over the repository with the team's submission text alongside it. It uses read, grep, and ls to walk the codebase claim by claim and check what the code supports. The output is a markdown report. A cheaper model turns the report into structured scores. The aggregate score is computed deterministically in code. And in the end, a human reads what matters and decides.

The first production version ran on Push to Prod submissions. It was useful enough to keep building: it surfaced implementation details and claim mismatches that would have been easy to miss in a high-volume manual review, and it produced audits that a human could inspect rather than opaque scores they had to accept or reject wholesale.

This is what we learned from treating judging as an attention-allocation problem with a constrained human compute budget.

Thoroughness and taste are different jobs

Some parts of judging are thoroughness-bound. Other parts are taste-bound.

	Thoroughness	Taste
Core question	What is actually present in the artifact?	How much does it matter?
Unit of work	Inspect files, verify claims, cite evidence	Compare tradeoffs, infer depth, judge core value proposition
Main constraint	Coverage under limited attention	Domain priors built from experience
Best delegated to	Systems that can apply the same policy repeatedly	Humans who understand context, ambition, and what the rubric misses
Useful output	An audit: claims, mismatches, citations	A decision: ranking, override, or prize call
Typical failure mode	Search error: the system looked in the wrong place	Rubric overreach: the score hides a judgment call

A pipeline that routes each step to the resource with the better cost and error profile is resource allocation across humans and machines. The design follows from there.

Taste is the judgment you build after seeing many projects succeed, fail, and evolve. A strong judge is not only asking whether a submission satisfies the rubric. They are also asking: does this implementation have depth? Is the hard part actually solved? If this team kept going for another month, would the project compound or collapse? Those calls come from experience with similar artifacts. A rubric can describe some of that judgment. A model can approximate some of it. Neither carries the full set of priors a strong human judge brings to the artifact.

So yes, the machine is discerning in a narrow sense. It is good at checking evidence before it is good at judging quality. It can notice that a team claimed a custom model and shipped a wrapper. It can notice that a README describes a backend the repo does not contain. It can notice that a criterion has no cited evidence. That narrower form of discernment is valuable precisely because it leaves taste to the human layer.

What the machine can discern

The usual "LLM as a judge" setup asks the model: given this artifact, what is its quality? That framing invites a familiar set of problems. The model may reward confident prose. It may prefer its own style. It may be swayed by order, verbosity, or polish. It is being asked to compress a complex artifact into a judgment it cannot reliably ground.

We ended up asking a different question.

The agent reads the team's pitch: name, tagline, description, tracks, demo URL, submission answers. Then it enters the repository with tools. Its job is to walk the codebase and check, claim by claim, what is and is not supported. The output is a report whose spine is claims_mismatches: places where what the team said and what the code does come apart, with file:line citations.

The hierarchy matters. The repository is the evidence. The markdown audit is the agent's reading of that evidence. The structured JSON is an extraction from the audit. The aggregate score is deterministic compression: score × weight, computed in code, not by the model. The human decision sits on top. We trust each layer less as it gets farther from the code, so the lower layers stay inspectable.

Under this frame, a polished README becomes a longer list of claims to check. A blank README has less to disprove. Bias has not disappeared; the risk has moved. The main failure modes shift from presentation bias toward search error, incomplete exploration, and bad evidence retrieval. Those are still serious failure modes, but they are easier to inspect and easier to repair.

One kind of reward-hacking also gets harder: winning on confident prose alone. A judge that estimates quality can be coaxed toward a high score by an artifact that sounds complete. An auditor has a fixed reference to return to. If the pitch is inflated, the repository can push back. The hard cases remain hard: impressive-looking but trivial code, copied work, generated filler, hidden dependencies, or a demo path that does not match the repo. The auditor reduces the part of the scoring surface where prose alone can dominate.

The output tends toward binary questions. Is this claim supported? Is this file present? Does this implementation match the described architecture? That is a better primitive than is this a 7 or an 8? We still produce 0–10 scores per criterion because hackathon rubrics expect them, but the useful object underneath is the cited claim check. Nobody really knows what separates a 3 from a 4 in the abstract. A cited mismatch is easier to argue with.

That is why the transcript matters. When the model returns only a number, the human has nothing to react to. Accepting or rejecting the score becomes ceremony. When the model returns "docked Originality from 7→4 because src/model.ts:42 shows the 'custom architecture' is transformers.AutoModel.from_pretrained('bert-base-uncased')", the human can verify or override in thirty seconds with the primary source in front of them. The audit gives the human a concrete object to appeal against: a claim, a citation, and a local reason for the score change.

This hierarchy explains the two-phase pipeline too. The expensive agent does the open-ended work: clone the repo, explore, read, reason, and write the audit. A cheaper extraction model turns that markdown into schema-shaped JSON. The extractor is not allowed to invent claim mismatches, omit criteria, duplicate criteria, change max scores, or return an overall score. Validation catches those failures, canonicalizes criterion names back to the rubric, retries parse-level errors with a retry note, and leaves the aggregate score to code.

The model does the open-ended reading; deterministic code handles schema validation, canonicalization, weighting, and aggregation.

Floor quality over peak accuracy

Is the AI actually as good as a human judge? Probably not. That comparison uses the wrong baseline.

The easy benchmark compares the model against a careful, well-rested expert with unlimited time. That person is not the system we are replacing. The real comparison is the human process under workload: a small panel, hundreds of submissions, uneven energy, uneven context.

The important question has two parts: how good can the best review be? and how many projects get a real review at all?

One way to think about this is the difference between peak quality and floor quality. A panel of humans has a high ceiling. On a project that catches the right judge at the right moment, the review can be excellent. The floor drops as the queue grows. The system becomes uneven. Some projects get deep attention. Others get triage.

The auditor has a lower ceiling than a great human judge and a more consistent floor. It does not get curious or develop taste in the way a human expert does. Its advantage is invariance under queue position: later projects receive the same exploration policy as earlier ones, subject to the same budget and harness constraints.

That trade off can be worth making even if the model is worse than an expert on any single project. The practical baseline is variance in review depth across the queue. At the prize-money end of a hackathon, that variance matters.

Agentic evaluators are a different category

Once the artifact is an audit rather than a score, the engineering problem changes. The evaluator is no longer a prompt that maps input to verdict. It is a search system over an evidence base: submission text, repository files, demo metadata, rubric criteria, tool traces, and the intermediate claims the agent writes down along the way.

That makes the harness the main object. The harness decides what evidence the agent can reach, which tools it can call, how much exploration budget it gets, what counts as a citation, how failures are retried, and which parts of the output are allowed to influence the final score. The model is one component inside that system. The evaluator is the whole loop.

Rubrics matter a lot. A vague rubric gives the agent too much freedom to substitute its own notion of quality. A good rubric is closer to a specification: criteria, weights, examples, disallowed assumptions, evidence requirements, and the boundary between claim verification and human judgment. If the rubric says "technical execution," the agent has to know whether that means code completeness, architectural depth, deployed functionality, reliability, or some weighted combination of those. Otherwise the audit becomes fluent but under-specified.

Profiles are how the same evaluator adapts inside production. A catalog pass can prioritize coverage: inspect every project enough to surface obvious claim mismatches and route human attention. A sponsor-track profile can spend more budget on the files and flows relevant to that track's stated criteria. A finalist profile can rerun the same project with deeper repository traversal, stricter citation requirements, and a stronger model. An appeal profile can focus only on disputed criteria and force the agent to re-check the evidence behind a specific score change. These are operating points on the same evaluation system, not environment switches.

This changes the cost model. With a single model call, the main knobs are model choice and context size. With an agentic evaluator, the knobs include tool-call budget, repository traversal policy, timeout, model choice, extraction strictness, rubric specificity, and retry strategy. You can spend more compute where the decision is high-stakes or where the first pass found uncertainty. You can also cap exploration when the audit only needs to route attention.

The failure mode changes too. A truncated prompt cannot see what was cut. An under-explored agent saw what it chose to look at, and the transcript tells you where it looked. That does not make the system automatically reliable. It makes the unreliability more diagnosable: search error, missing context, weak rubric, bad extraction, schema drift, timeout, or an unsupported inference from evidence.

The output shape also becomes part of the protocol. An agent in a sandbox can decide, mid-run, to write its report to a file instead of returning it inline. Or write half inline and half to disk. Or crash after streaming 90% of a useful report. A normal model call either returns the requested object or fails. An agent leaves traces, so the system has to recover from traces: streamed events as the source of truth, filesystem fallback, a targeted re-prompt for inline output, partial-text recovery on late-session crash.

This is the surface area you inherit the moment the model becomes a process instead of a function call. "Agentic" is mostly a marketing word, but in this system it has a concrete meaning: evaluation quality depends on the harness, the rubric, the exploration policy, and the recovery path, not only on the base model.

The same is true for observability. We ended up reading the agent's own JSONL session logs from inside the sandbox when the vendor SDK did not surface token usage consistently. The system also persists phase latencies as it moves through fetching context, sandbox acquisition, cloning, review, extraction, and persistence. This is boring accounting. It is also how you notice that an evaluator is getting slower, more expensive, or less complete before the scores start feeling wrong. The alternative is paying for runs you cannot attribute and trusting evaluations you cannot inspect.

Disagreement is the product boundary

The disagreement between an automated judge and an expert is often treated as noise to eliminate. Some of it is. If the system misses a file, misreads a claim, or invents a mismatch, that is a bug.

The final band of disagreement is where the expert's knowledge lives.

In the auditor frame, that is exactly where the human should spend time. The pipeline's job is not to make the model and the human agree on everything. It is to make sure they agree on what is present, what is missing, and what has evidence. Then the human can spend attention on what it means.

That reverses the usual alignment target. If the system and the human disagree on whether a claimed feature exists, the system needs work. If they disagree on whether the implementation is prize-worthy, that may be the product working as intended. High agreement on thoroughness is good. High agreement on taste is suspicious if it means the rubric has crowded out judgment or the human has stopped looking.

A healthy pipeline converges on evidence and leaves room for human disagreement over significance.

Looking ahead

The Push to Prod experiment gave us a glimpse into what an agentic evaluator could look like. The next step is turning that experiment into a measurable, repeatable part of Devfolio's broader judging product stack.

The audit should live where judging already happens: attached to the project, broken down by rubric criterion, visible beside the judge's scoring/prize assignment flow, and backed by citations into the submission and repository. A judge should be able to inspect the evidence, accept or override the AI's assessment, and continue with the normal judging flow. For organizers and sponsors, this enables configuring when audits run, which profiles apply to which tracks or prizes, and how much of the audit is shown to judges. We treat human overrides and disagreements as first-class product data.

The evaluator itself can change shape. We have run the single-judge version. A small panel of cheaper models may beat a single expensive one, especially on prize-deciding runs. Whether you want one general agent or several specialized ones is also open. We do not have enough runs to take a side yet.

Hackathon judging is the clearest version of the problem, and the same constraint appears elsewhere in Devfolio's product surface. Devfolio sits between builders and scarce attention. Hackathons have many teams, a short window, a small judging panel, and real consequences for who gets noticed. The same pattern shows up elsewhere in our world: curators deciding which projects to feature, ecosystem teams looking for promising builders, sponsors trying to understand what actually came out of a track, internal teams comparing many artifacts without reducing them to who wrote the best pitch.

These workflows differ from judging in their risk profile. A talent-search workflow has different failure modes. A showcase page is not a prize decision. A sponsor report is not an evaluation rubric. They share one operational constraint: expert attention runs out before the artifacts do.

The useful system is one that splits thoroughness from taste, routes each operation to the resource with the right cost and error profile, and makes the boundary between machine audit and human authority explicit.

That is the sense in which the machine can be discerning. Not because it has taste. Because it can learn to tell the difference between a claim and evidence for that claim; between a working implementation and a well-written description of one; between a score that should move automatically and a judgment call that belongs with a person.

Every serious submission still needs a human decision. The point of the system is to make sure that decision begins from contact with the artifact: the repo, the demo, the cited source, the thing the team actually built. A discerning machine does not replace the judge. It helps ensure the judge is looking at the work itself.

References

Optimizing Information Retrieval for RAG Systems

Sat, 04 Jan 2025 00:00:00 GMT

The following are notes from some of my learnings from studying and building retrieval systems for RAG applications.

Full-Text Search vs Vector Search

Vector search excels at capturing semantic similarity, making it great for understanding the general meaning and context of queries. However, it has limitations, particularly when dealing with specific keyword-based queries or named entities.

For instance, when users search for names acronyms, or IDs (e.g., claude-3.5-sonnet).

This is where a hybrid approach shines;

You can create a more robust and accurate retrieval system by combining:

Traditional full-text search for precise keyword matching
Vector search for semantic understanding
Metadata filtering to scope the search space

Hybrid Search

The decision to use hybrid search depends on your use case; For technical documentation where users need to find specific API endpoints or error codes, keyword search excels at exact matches.

For a research paper database where users explore related concepts and methodologies, semantic search might better capture topical relevance.

For a knowledge base where users might search for specific terms but also need contextually related information, a hybrid approach combines precise matching with semantic understanding.

Combining Search Techniques

One straightforward approach to implementing hybrid search is to combine traditional keyword matching with semantic search through score fusion.

In this method, we first run both searches in parallel: a keyword search for exact matches and a vector search for semantic similarity. Each returns a scored list of results. These scores are then normalized and combined using a weighted sum:

final_score = α * keyword_score + (1 - α) * semantic_score

Where α is a tunable parameter which prioritizes certain documents based on factors like their relevance to the search query, their ranking in the individual lists, or other criteria.

The result is a final list that integrates the strengths of both keyword and semantic search methods.

Reciprocal Rank Fusion (RRF)

Another common approach to combine results is Reciprocal Rank Fusion (RRF). RRF assigns higher weights to items that appear near the top of multiple result lists.

For each document d, the RRF score is calculated as:

RRF(d) = Σ(r ∈ R) 1 / (k + r(d))

Where:

d is a document
R is the set of rankers (retrievers)
k is a smoothing constant that prevents disproportionate influence of top-ranked items (typically 60)
r(d) is the rank of document d in ranker r

For example, if a document ranks 3rd in keyword search and 9th in semantic search with k=60:

RRF = 1/(60 + 3) + 1/(60 + 9) = 0.0317

Documents appearing in only one list get zero contribution from the other list. The final results are sorted by descending RRF score.

This method effectively balances results that perform well across multiple search strategies while dampening the impact of outlier rankings.

Reference implementation in Typescript:

function reciprocalRankFusion(
  allResults: Document[][],
  options: RRFOptions = {}
): DocumentWithRanks[] {
  const { k = 60, maxResults = 10, scoreThreshold = 0 } = options;

  const rrfScores = new Map<string, number>();
  const docDetails = new Map<
    string,
    Document & { ranks: Map<number, number> }
  >();

  // Process each result list
  allResults.forEach((resultList, retrieverId) => {
    resultList.forEach((doc, rank) => {
      const rrf = 1 / (k + (rank + 1));
      const currentScore = rrfScores.get(doc.id) || 0;
      rrfScores.set(doc.id, currentScore + rrf);

      // Store document details from first occurrence
      if (!docDetails.has(doc.id)) {
        docDetails.set(doc.id, {
          ...doc,
          ranks: new Map(),
        });
      }

      // Track rank in each retriever
      docDetails.get(doc.id)?.ranks.set(retrieverId, rank + 1);
    });
  });

  // Build final results
  const fusedResults: DocumentWithRanks[] = Array.from(rrfScores.entries())
    .filter(([_, score]) => score >= scoreThreshold)
    .map(([id, score]) => {
      const document = docDetails.get(id)!;

      return {
        ...document,
        rrf_score: score,
        ranks: Object.fromEntries(document.ranks) as {
          [retrieverId: number]: number;
        },
      };
    })
    .sort((a, b) => (b.rrf_score ?? 0) - (a.rrf_score ?? 0))
    .slice(0, maxResults);

Query Reformulation

Raw user queries often lack context or contain ambiguous references. Rather than taking these queries at face value, query reformulation acts as a preprocessing step that transforms them into more complete, self-contained expressions.

The goal is to add implicit context that users typically omit, resolve ambiguous references, and expand queries with relevant terms.

For instance, a query like "How did we do this quarter?" might be reformulated to "What is the revenue for Q1 2025?" - adding both the specific metric (revenue) and temporal context (Q1 2025) that was implicit in the original query.

One way to implement this is to maintain a context window containing recent conversation history, currently visible information, and relevant system state.

An LLM can then be used to reformulate queries by identifying contextual references, resolving pronouns, and expanding abbreviated expressions. For example, "Find similar docs" might become "Find documentation similar to the current deployment guide" when reformulated with appropriate context.

The key is ensuring that reformulated queries are self-contained while preserving the original intent. This makes them more effective for downstream search operations, whether using keyword, semantic, or hybrid approaches.

Feature and Entity Extraction

Many search queries contain specific attributes that are better handled through structured queries. By extracting these features first, we can leverage traditional search infrastructure more effectively.

We can use LLMs to extract various features from the reformulated query. Each feature extractor can run in parallel and should focus on specific attributes:

Temporal expressions ("last week", "Q1", "2024")
Named entities (people, organizations, products)
Numerical ranges ("over 500", "between 10-20")
Status indicators ("completed", "pending", "failed")
Technical identifiers (API endpoints, error codes)

For example, given the query "Show me failed deployments from last week for the auth service", feature extraction might yield:

{
  "deployed_at": {
    "range": "2024-12-28/2025-01-03",
    "confidence": 0.95
  },
  "status": {
    "value": "failed",
    "confidence": 0.98
  },
  "service": {
    "value": "auth",
    "confidence": 0.99
  }
}

Re-ranking

Effective re-ranking strategies are helpful for surfacing the most relevant documents. Here are two approaches,

These strategies are helpful independently but most powerful when combined. The result should be a more robust ordering that considers both semantic relevance and metadata compatibility.

Heuristic Re-ranking

Heuristic re-ranking employs efficient algorithms that leverage readily available signals. With metadata signals through simple, fast heuristics, we can quickly improve ranking quality:

Recency Bias: For queries requiring current information ("current status", "ongoing issues"), apply time-decay functions to favor newer documents.
Entity Matching: Boost documents that reference entities (people, organizations, locations) mentioned in the query. If someone searches for "budget proposals from marketing team", documents containing marketing team members get prioritized.
Category Alignment: Leverage document categories or tags when queries suggest specific content types ("technical documentation", "financial reports").

This heuristic re-ranking is fast, but can sometimes miss subtle nuances.

Re-ranking with Cross-Encoders

Cross-encoders look at queries and documents together, unlike traditional embedding models that process them separately. This joint analysis leads to more nuanced relevance judgments — they can better understand context, catch subtle relationships, and make more accurate relevance decisions.

For model selection, a general-purpose models like ms-marco-MiniLM will be good enough in most cases. But consider fine-tuning if your use-case truly warrants it.

Closing Thoughts

The most robust retrieval solutions often combine multiple approaches. The goal is to build systems that leverage both the semantic understanding of embeddings and the precision of traditional search methods.

The key is choosing the right combination of techniques based on your specific use case, and gradually add complexity only where it demonstrably improves results.

References

https://supabase.com/docs/guides/ai/hybrid-search
https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking#how-rrf-ranking-works
https://en.wikipedia.org/wiki/Query_understanding
https://jxnl.co/writing/2023/09/17/rag-is-more-than-embeddings/#improving-the-rag-model-with-query-understanding
https://haystack.deepset.ai/blog/query-expansion
https://www.sbert.net/examples/applications/cross-encoder/README.html
https://www.sbert.net/examples/applications/retrieve_rerank/README.html

The Socratic Method

Sat, 10 Sep 2022 00:00:00 GMT

To put it into action, follow this general structure:

Start with open-ended questions.
Propose ideas based on these questions.
Probe these ideas with progressive questioning.
Repeat 2/3 until the best ideas are developed.

Here's a process of how you might apply Socratic Questioning:

Start asking questions: What's the problem you are trying to solve? We often waste time and energy trying to solve the "wrong" problem. Identify the “right” problem before you try to solve it.
Propose your current thinking on the problem: What is your hypothesis? What are the origins of that thinking?
Open the floor for targeted questioning: Why do you think this? Is the thinking too vague? What is it based upon?
Challenge the assumptions underlying the original thinking: Why do you believe this to be true? How do you know it’s true? How would you know if you were wrong? Identify the source of beliefs on a problem. Be ruthless in evaluating their integrity and validity.
Evaluate the evidence used to support the thinking: What concrete evidence do I have? How credible is it? What “hidden evidence” may exist?
Understand the consequences of being wrong: Can an error be quickly fixed? How costly is this mistake? Always understand the stakes.
Evaluate potential alternatives: What alternative beliefs or viewpoints might exist? Why might they be superior? Why do others believe them to be true? What do they know that I don’t? Evaluate them on their merits and ask these same fundamental questions about them.
After zooming in, zoom out: What was my original thinking? Was it correct? If not, where did I err? What conclusions can I draw from the process about systemic errors in my thinking?

https://www.sahilbloom.com/newsletter/superhuman-effort-socratic-questioning-more

Monads in Functional Programming

Sat, 25 Dec 2021 00:00:00 GMT

Monads in Haskell

In functional programming, Monads are an abstraction used to structure programs

Help reduce complicated sequences of functions into “a pipeline” of actions
Abstract away control flow
Facilitate side-effects
Manage external data interactions

class Monad m where
  return :: a					-> m a
  (>>=)  :: m a -> (a -> m b) 	-> m b

Monads are abstraction used to help structure programs and help easily achieve some functionality which would be difficult to achieve otherwise.For example, they help achieve side-effects which would be required in the real world.

Monads in Haskell are defined as a Typeclass.

We make things “monadic” by making them an instance of this typeclass. 2 main operations defined by the typeclass:

Lifting: take a non-monadic value and turn it into a monadic value
Binding: monadic value and function that return a monadic value
- The bind operator can have different semantics for different monads

Functors and Aplicatives

Functors are things that can be mapped over

fmap::(a -> b) -> f a -> f b

Applicatives are functors that can be applied

pure  :: a -> f a
(<*>) :: f(a -> b) -> f a -> f b

A monad on category C consists of an endofunctor (a functor mapping a category to itself), T: C -> C along with two natural transformations:

1_C -> T where 1C denotes the identity functor on C, and
T^2 -> T where T^2 is the functor T to T from C to C

These are required to fulfill coherence conditions

class Monad m where
	return :: a					-> m a
	(>>=) :: m a -> (a->m b)	-> m b
	(>>) :: m a -> m b			-> m b

Coherence Conditions

Left identity: The first monad law states that if we take a value, put it in a default context with return and then feed it to a function by using >>=, it’s the same as just taking the value and applying the function to it. return a >>= f ≡ f a

Right identity: The second law states that if we have a monadic value and we use >>= to feed it to return, the result is our original monadic value. m >>= return ≡ m

Associativity: The final monad law says that when we have a chain of monadic function applications with >>=, it shouldn’t matter how they’re nested. (m >>= f) >>= g ≡ m >>= (\x -> f x >>= g)

Monads for Side-Effects

Eg. IO Monads in Haskell can function as "containers" that carry "extra information" apart from the value inside that functions need not worry about. Here, the "information" can be used as the action that performs IO

instance Monad IO where
	return :: a -> IO a
	(>>=) :: IO a -> a (a -> IO b) -> IO b

Example as a REPL reading/writing to a terminal

flushStr :: String -> IO ()
readPrompt :: String -> IO String
evalString :: String -> IO String
until_ :: Monad m => (a -> Bool) -> m a -> (a -> m ()) -> m ()
runRepl :: IO ()
main :: IO ()

Monads for Control Flow

Eg. Error Handling We define all types of errors we want to catch and throw as MonadicError We define a type for functions that may throw a MonadicError

type ThrowsError = Either MondaicError

Either is another instance of a monad, The "extra information" in this case is whether the error occurred.

instance (Error e) => Monad (Either e) where
	return x = Right x
	Right x >> f = f x
	Left err >>= f = Left err

If (>==) sees an error it simply passes that error through without subsequent computations, else passes the value along

Monad Transformers

Take our 2 Error Handling and IO Monads for example, say we need to use their behavior functionality simultaneously. We use monad transformers to combine functionality of multiple monads We use ExceptT, a monad transformer that adds exceptions to other monads

newtype ExceptT e m a :: * -> (* -> *) -> * -> *

The combined Monad would then be:

type IOThrowsError = ExceptT MonadicError IO

Monads for Environment Management

Haskell has no notion of mutable state. Each function has an environment storing values for each of its args and vars.

We use a feature called IORef that helps us hold the environment state within the IO monad We then simply access this environment mutating its state as required, and keep passing it around on each evaluation cycle.

type Env = IORef[(String, IORef SomeVal)]

eval :: Env -> SomeVal -> IOThrowsError SomeVal

The Essence of Computer Science

Wed, 09 Jun 2021 00:00:00 GMT

From -> https://youtu.be/-J_xL4IGhJA?si=NHglUsKKpbT_8jLY

“I think that it’s extraordinarily important that we in computer science keep fun in computing. When it started out, it was an awful lot of fun.

Of course, the paying customers got shafted every now and then, and after a while we began to take their complaints seriously. We began to feel as if we really were responsible for the successful, error-free perfect use of these machines. I don’t think we are. I think we’re responsible for stretching them, setting them off in new directions, and keeping fun in the house.

I hope the field of computer science never loses its sense of fun. Above all, I hope we don’t become missionaries. Don’t feel as if you’re Bible salesmen. The world has too many of those already. What you know about computing other people will learn. Don’t feel as if the key to successful computing is only in your hands. What’s in your hands, I think and hope, is intelligence: the ability to see the machine as more than when you were first led up to it, that you can make it more.”

—Alan J. Perlis (April 1, 1922 – February 7, 1990)

Computer Science is a terrible name

First, Its not a science, And its also not about computers
Computer Science has a lot in common with magic than it does with science
Computer Science is no more about computers than Physics is about particle accelerators and Geometry about surveying instruments

When a field is in its infancy and you don't really understand it really well, its really easy to confuse the essence of what you're doing with the tools that you use.

The Essence of Computer Science

Its really about formalizing an intuition about process; about developing a way to talk precisely about how to do things

Declarative Knowledge vs Imperative Knowledge

Declarative Knowledge is knowledge that talks about what is true. It makes statements of fact that one can use to try to reason about things.

Imperative Knowledge talks about "how to" knowledge. It tends to describe a specific sequence of steps that characterize the evolution of a process by which one can deduce information, transforming one set of facts into a new set.

Distributed Systems

Mon, 24 May 2021 00:00:00 GMT

A distributed system is a collection of autonomous computing elements that appears to its users as a single coherent system.

Features of distributed systems:

A collection of computing elements (nodes) each able to behave differently
Clients believe they're dealing with a single system. The Independent nodes have to collaborate

Autonomous Compute Units

Fundamentally, nodes act Independently from each other
Nodes communicate by exchanging messages (cf. event driven systems)
Nodes have their own notion of time, ie. No global clock
Relationship between nodes in a collection need to be managed (register which nodes do or do not belong in the system)
- open group: any node can join the system, it can communicate with any other node
- closed group: only members of a group can communicate with each other, separate mechanism is needed for a node to leave or join a group

Distributed systems are often organized as an overlay network, where a node is a process with a list of other nodes it can directly send messages to. There are roughly two typed of overlay networks

Structured Overlay: Each node has a well-defined set of neighbors with whom it can communicate. The nodes are organized in a tree or logical ring.

Unstructured Overlay: Each node has a number of references to randomly selected other nodes.

An overlay network should, in principle, always be connected, meaning that between any two nodes there is always a communication path allowing those nodes to route messages from one to the other.

cf: Peer-To-Peer (P2P) networks.

Single Coherent System

A distributed system should appear as a single coherent system.
A distributed system is coherent if it behaves according to the expectations of its users.
In a single coherent system the collection of nodes as a whole operates the same, no matter where, when, and how interaction between a user and the system takes place.

Design Goals

Performance

Distributing computation
- Cons: network and latency bottlenecks - Tricks that help: caching, async programming
- Pros: parallelism, client can pick geographically proximate server
  - Scalability:
    - ideally, performance would linearly increase with the addition of new nodes N x nodes -> N x throughput
- Need to balance based on some factor N
  - divide the state by N (In the example of a file system)
    - split by user
    - split by file name
    - Sharding or Partitioning
  - Wont always work -> can only scale so far (potential drawbacks)
    - Global operations, eg search
    - Load Imbalance
      - One very active user
      - One very popular file
        
        -> one particular server takes all the load, other remain idle
        
        = N x nodes -> 1 x throughput

Fault Tolerance

Availability The system keeps on serving as it normally would under certain kinds of failures
- fail-over
  - active-passive
    - heartbeats are sent between 2 servers (active and passive)
    - If the heartbeat is interrupted, passive server takes over the active one
    - downtime length can vary depending of whether the passive service was in "hot" standby or has to do a "cold" startup
    - Only the active server ever handles traffic
    - Also refereed to as master-slave failover.
- replication
  - Master-Slave Replication
    - The master serves reads and writes, replicating writes to one or more slaves, which serve only reads.
    - Slaves can also replicate to additional slaves in a tree-like fashion.
    - If the master goes offline, the system can continue to operate in read-only mode until a slave is promoted to a master or a new master is provisioned.
  - Master-Master Replication
    - Both masters serve reads and writes and coordinate with each other on writes
    - If either master goes down, the system can continue to operate with both reads and writes
Recoverability
- The system gracefully recovers from failures and continues working correctly
  - store state on disk

Consistency

CAP Theorem
- Consistency - Every read receives the most recent write or an error
- Availability - Every request receives a response, without guarantee that it contains the most recent version of the information
- Partition Tolerance - The system continues to operate despite arbitrary partitioning due to network failures

Networks aren't reliable, so you'll need to support partition tolerance. You'll need to make a software tradeoff between consistency and availability.

CP - consistency and partition tolerance Waiting for a response from the partitioned node might result in a timeout error. CP is a good choice if your business needs require atomic reads and writes.
AP - availability and partition tolerance Responses return the most readily available version of the data available on any node, which might not be the latest. Writes might take some time to propagate when the partition is resolved.

AP is a good choice if the business needs allow for eventual consistency or when the system needs to continue working despite external errors.

swarnim's blog

Some thoughts on harnesses and the agents they shape

Harness Design <> Model Intelligence

Harness Design Patterns for Continual Learning

Dynamic Workflows

Context Engineering

Compaction

Orchestration

Closing Thoughts

Tools and Curiosities

Curiosity is the pointer

Projects are probes

Work with gravity

Usefulness disciplines curiosity

The toolkit compounds

Better maps

The Discerning Machine

Thoroughness and taste are different jobs

What the machine can discern

Floor quality over peak accuracy

Agentic evaluators are a different category

Disagreement is the product boundary

Looking ahead

References

Optimizing Information Retrieval for RAG Systems

Full-Text Search vs Vector Search

Hybrid Search

Combining Search Techniques

Reciprocal Rank Fusion (RRF)

Query Reformulation

Feature and Entity Extraction

Re-ranking

Heuristic Re-ranking

Re-ranking with Cross-Encoders

Closing Thoughts

References

The Socratic Method

Monads in Functional Programming

Monads in Haskell

Functors and Aplicatives

Coherence Conditions

Monads for Side-Effects

Monads for Control Flow

Monad Transformers

Monads for Environment Management

The Essence of Computer Science

Computer Science is a terrible name

The Essence of Computer Science

Declarative Knowledge vs Imperative Knowledge

Distributed Systems

Features of distributed systems:

Autonomous Compute Units

Single Coherent System

Design Goals

Performance

Fault Tolerance

Consistency

Topics

Networking

CDNs

Caching

Communication Protocols