# Vlad Temian's Blog - Full Content > I build agentic programming tools. 15+ years of systems infrastructure experience, now focused on making AI agents reliable and accessible for developers. Former CTO, open-source builder, community organizer at agentic.tm. This file contains the complete content of all blog posts for LLMs and AI assistants. --- ## About the Author **Vlad Temian** I build agentic programming tools. 15+ years of systems infrastructure experience, now focused on making AI agents reliable and accessible for developers. Former CTO, open-source builder, community organizer at agentic.tm. - Website: https://blog.vtemian.com/ - GitHub: https://github.com/vtemian --- ## Blog Posts ### Claude Code Isn't About Code - **URL**: https://blog.vtemian.com/post/claude-code-isnt-about-code/ - **Published**: June 2, 2026 - **Word Count**: 1430 words - **Tags**: ai tools, software economics, agentic-programming, ai - **Summary**: The people getting the most out of Claude Code aren't writing code. Some notes on where the value moved, picking up where the disposable tools manifesto left off. Claude Code isn't a coding tool. The people using it best right now barely touch code. They run factory floors, legal teams, HR, finance, operations. They don't think of it as software. They think of it as *the thing that does the work*. That took me two months to see. In April I wrote that I'd shipped 20+ tools, made zero revenue, and watched the value move from tools humans use to [tools agents use](/post/disposable-tools-manifesto/). I ended on three words: I don't have the answer. Then YC published theirs. ## YC stopped building tools In ["Inside YC's AI Playbook"](https://www.youtube.com/watch?v=B246K_G7mHU), Pete Koomen, Garry Tan, and Jared walk through how YC rebuilt itself internally. YC has always run on its own software, sitting on a single Postgres database that holds everything that matters: every company, every founder, every financial transaction, every note in their internal CRM. A year ago, the loop looked like every company you've worked at. The finance team would describe a workflow (booking journal entries, logging priced rounds) to software engineers. The engineers would go build purpose-built, deterministic software encoding that workflow. Then they'd hand it back. Koomen's word for it: *inefficient*. So they killed the loop. Instead of building finance software, they gave the finance team agents and a registry of tools. The first tool that changed everything was almost embarrassingly simple: read-only SQL access to the production database. Jared built it, felt like he was breaking the rules, and pushed it out late at night. It worked extremely well. {{< diagram-loop >}} They started with about 20 tools. **At the time of the talk, there were more than 350.** Every team adds their own. Finance books journal entries. Others manage office hours and events. And the same registry feeds both YC's internal agents and Claude Code running on individual laptops. The effect was a textbook case of [Jevons paradox](https://en.wikipedia.org/wiki/Jevons_paradox). When asking a complex question stopped costing several hours of someone else's SQL time, people didn't just answer the old questions faster. They asked far more questions, and far harder ones, because *asking became cheap*. ## The inversion that makes it work The reason this is a real shift and not just internal tooling is captured in Koomen's earlier essay, ["AI Horseless Carriages"](https://koomen.dev/essays/horseless-carriages/). His critique was of AI bolted onto existing software as a feature, like Gmail's email writer, where the developer keeps all the control and hides it from the user. The crux is one sentence, and it's worth reading slowly. **AI-native software is the agent wrapping deterministic tools, not deterministic software wrapping an AI.** Control shifts from the developer to the user. {{< diagram-inversion >}} Flip that around and the strategy falls out of it. You stop building the interface. You stop building another finance SaaS, another ERP screen, another internal operations dashboard with its carefully designed flows. Those are horseless carriages: the old shape, with an engine bolted on. Instead you build small, deterministic, well-scoped tools, you put them in a registry, and you let humans drive the agent that composes them. The plant manager, the lawyer, the HR lead. They get to build whatever they need in the moment, in plain language, without waiting on an engineering backlog. The interface was the constraint. *Removing it is the product.* This is the answer I was groping for in April. The environment where disposable tools thrive is not hosting and previews and billing dashboards, or not only that. It's a layer of unified context plus a shared tool registry plus a harness that lets agents compose them. The tools inside that layer are disposable by design. The registry and the context are the durable thing. YC built a private one and called it raising the floor: a new hire absorbs the institutional knowledge of the best people in the org through the tools and transcripts, instead of six months of ramp. There's a price of admission, and the talk is honest about it. You have to be willing to spend somewhere between $10,000 and $100,000 a year on tokens, sometimes more. Garry Tan's framing is that this buys you a one-time time warp: you get to live in 2028 now, and in two years it costs a hundredth as much and everyone does it. That's the optimistic read. Now for the other one. ## Why your economy won't feel this for a decade If software is this transformative, where's the growth? A talk that dropped the same week, from a very different room, has the answer. In ["A.I. and Our Economic Future"](https://www.youtube.com/watch?v=xBpGn3BDcOY), Stanford economist Chad Jones offers a model built on a single idea: weak links. A chain is only as strong as its weakest link. Business success requires completing many tasks, and if you make 17 of 20 links incredibly strong, the chain is still limited by the three you didn't touch. That insight produces an elegant and slightly deflating formula. Infinite automation of a task raises GDP by roughly that task's share of GDP. Software is about 2% of GDP. So if software became infinitely abundant and free tomorrow, we'd be about 2% richer. Not 100 times richer. *Two percent.* Everything else stays a bottleneck. {{< diagram-weak-links >}} His killer chart makes it concrete. We carry computers with 100 million times the transistors we had in the 1970s. Yet the share of GDP paid to computing power peaked at about 4.5% in 2000 and has since fallen by a third, to 3%. The price collapse dominates the quantity explosion. **The abundant link loses its share. Value flows to whatever stays scarce.** That is the mechanism underneath my manifesto, stated more rigorously than I managed. Disposable tools don't become businesses because software is now the cheap, plentiful link. The returns accrue to the scarce weak links: the context, the judgment, the orchestration layer, the human deciding what to ask. I felt that the value had moved. Jones gives you the equation for where it moves and why. He has the receipts too. In 2016, Geoffrey Hinton said we should stop training radiologists because AI would replace them within five years. A decade later we have more radiologists, and they're paid more. Jobs are bundles of tasks. Automate 75% of them and the remaining 25% become the scarce, valuable, well-paid part. The weak link is wherever the human still has to stand. ## Both talks are right Here is the tension, and I think it's the whole point. YC says you can live in 2028 today. Jones says the explosion takes 30 years, not three. They're both correct, at different scales. Inside a single high-trust organization that is willing to tear down its own weak links (locked-down context, command-and-control approvals, siloed tools, the engineering bottleneck between a question and its answer), the 10x is real and available *right now*. YC is the proof. Across the whole economy, the weak links are everywhere and mostly human, and strengthening all of them takes *decades*. Both things are true at once. That gap is the opportunity, and it has an expiration date. The org that removes its own weak links now gets to operate years ahead of competitors who are still commissioning another internal dashboard. Not because they have a better model. Everyone has the same models. Because they stopped building interfaces and started building a registry. ## The answer, finally In April I wrote that the tool is not the end, and I left it there because I couldn't name what was. I can now. Stop building specialized dashboards, interfaces, and flows. Stop building another SaaS for finance, another internal tool for operations, another ERP nobody enjoys. Build small deterministic tools, put them in a registry, give them to agents, and let people drive. The interface was scaffolding. The agent wraps the tools. The human holds the leash. The tool was never the product. The loop is: a person, an agent, and a registry of tools, composing whatever the moment needs and throwing it away when the moment passes. Claude Code isn't about code. *It never was.* Stay curious ☕ --- ### Fight Slop with Clarity - **URL**: https://blog.vtemian.com/post/fight-slop-with-clarity/ - **Published**: May 14, 2026 - **Word Count**: 1165 words - **Tags**: ai tools, vibe-coding, software economics - **Summary**: AI made slop cheap. We made more of it, faster. Clarity is the only input that doesn't degrade when you multiply it. AI made slop cheap. We made more of it, faster. I made my share. Here's the part I missed. **Slop isn't what AI produces. Slop is what any multiplier produces when there's no clarity behind it.** Simon Willison, who [popularized *AI slop*](https://simonwillison.net/2024/May/8/slop/), drew the parallel directly: slop is to AI-generated content what spam became for email. The model isn't the problem. *The clarity behind it is.* The model is a multiplier. Whatever clarity you bring, it scales. Whatever fog you bring, it scales the same. The output looks impressive in both cases. Only one of them is signal. The fight isn't against AI. It isn't against slop either, not directly. The fight is for clarity. Clarity is the only input that, when multiplied at industrial speed, doesn't degrade. Everything else turns into noise. ## The AI Productivity Gap The faster AI helps you build, the less you ship that matters. *Volume goes up. Signal goes down.* [METR's 2025 randomized trial](https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/) caught the pattern at scale: experienced developers using AI tools were **19% slower** at completing real issues in their own codebases. The strange part wasn't the slowdown. It was that, even after shipping the work, they still believed they'd been **20% faster**. A forty-point gap between perception and reality, and the perception held even with the evidence in front of them. I lived a smaller version of that and called it progress. ## I've Seen This Before. So Have You. Every company I've worked at hit the same wall. Things break. Someone proposes a procedure. The procedure works for a month. Then a use case appears that the procedure didn't anticipate. Someone improvises. Someone else writes a new procedure to cover the gap. The procedure stack grows. Compliance drops. The procedures stop describing what people *do* and start describing what people *pretend* to do. [Goodhart's Law](https://en.wikipedia.org/wiki/Goodhart%27s_law) in office form: when a measure becomes a target, it ceases to be a good measure. The work is the same as before, just buried under more documentation. The procedure wasn't the fix. Clarity about what we were trying to protect was the fix. Without that, the procedure was a layer of fog dressed up as control. **There was no AI in this room.** We have been making slop by hand for decades, in meetings, in procedures, in code. AI didn't invent the pattern. It just made the multiplier cheap enough to use everywhere. If you've played Factorio, you know this feeling at the level of muscle memory. You build a base. It works for a while. You unlock a new tier of technology. The new tech demands more inputs, faster throughput, different geometries. Your old base can't handle it. You bolt new belts onto the old layout. The base survives, technically, but moving anything through it takes ten times longer. You unlock another tier. You bolt more belts. At some point you realize the only fix is to tear the whole thing down and design it again from the start. Nobody who has played Factorio thinks the next technology tier solves the spaghetti. The spaghetti is a clarity problem. **Tier 5 just makes the spaghetti more expensive.** That's exactly what I did to my own work. I unlocked AI as the new tier. I bolted it onto a base that was never clear in the first place. What was I building? For whom? Solving what? I didn't have answers. I had *energy*. So I shipped. The AI multiplied the shipping. The shipping multiplied the unclarity. The base got louder, busier, and no closer to a thing anyone needed. This is what slop looks like from the inside. Not bad code. Not low-effort prompts. Genuine work, technically correct, multiplied without a question to anchor it. ## Clarity is Infrastructure, Not Aesthetic We talk about clarity like it's a writing virtue. Use shorter sentences. Cut adjectives. That's not what I mean. Clarity is the answer to a small set of unsexy questions. What problem am I actually solving? Whose problem is it? What does success look like, concretely, that I'd recognize when I see it? What am I willing to not build to keep this clear? These questions are slow. They don't ship anything. They feel like procrastination next to the warm dopamine of generating another tool. But they are the only input that, when fed into a multiplier, produces signal instead of noise. Kentaro Toyama, who spent two decades watching technology interventions either work or [amplify dysfunction](https://www.si.umich.edu/about-umsi/news/toyama-ai-amplifies-good-and-bad-society) in international development, formalized this as the Law of Amplification: technology's primary effect is to amplify human forces. *"ChatGPT helps honest writers brainstorm and helps bad students cheat."* The model is neutral. The forces it scales aren't. **Without clarity, AI is the most expensive way ever invented to be busy.** ## A Test Here's the operational version, the one you can use today on any tool you're considering, building, or shipping. **If a tool sells itself as a replacement for thinking clearly, it's a tarpit.** The more you use it, the deeper the dysfunction it's covering for, and the less likely you are to ever climb out. **If a tool sells itself as an amplifier for someone who already thinks clearly, it works.** The clearer the user, the better the output. The multiplier finally has something worth multiplying. (I wrote about how to build this kind of [amplifier in Harness Engineering](/post/harness-engineering/)). [Andrej Karpathy coined *vibe coding*](https://x.com/karpathy/status/1886192184808149383) in February 2025: *"I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works."* A year later he [refined the framing](https://thenewstack.io/vibe-coding-is-passe/), introducing *agentic engineering* for production work. I documented my own transition from "vibing" to systematic learning in my [vibe-infer case study](/post/vibe-infer/). The slop-friendly version was for throwaways. The serious version requires thinking first. The person who named the vibe ended up at the same place: clarity is the part you can't outsource. This applies to AI. It applies to project management software. It applies to the procedures at every company you've ever worked at. The version of any thesis that survives is the one built for users who already know what problem they're solving. Not for users who reach for AI to figure out what their problem is. ## The Only Input The fight isn't against AI. It's against everything that lets us avoid clarity. Including the multiplier we love most. Tools come and go. Multipliers get bigger. *Clarity is the only input that doesn't decay when scaled.* Bring it, or watch the slop scale instead. ### Key Takeaways * **AI is a Multiplier:** It scales whatever you feed it. Feed it fog, you get slop. Feed it clarity, you get signal. * **The Perception Gap:** Developers using AI often *feel* 20% faster while actually being 19% slower due to the overhead of managing unvetted output. * **Clarity as Infrastructure:** Thinking is not procrastination. It is the only input that doesn't degrade at industrial speeds. * **The Tarpit Test:** If a tool tries to replace your thinking, avoid it. If it amplifies your existing clarity, use it. Stay curious ☕ --- ### The Disposable Tools Manifesto - **URL**: https://blog.vtemian.com/post/disposable-tools-manifesto/ - **Published**: April 5, 2026 - **Word Count**: 1257 words - **Tags**: ai tools, saas, software economics, vibe-coding - **Summary**: I built 20+ tools in 6 months and made zero revenue. Here's what I learned about why SaaS is dying and what's replacing it. I built 20+ developer tools in six months. All of them work. None of them matter. Not because they're bad. Because the world changed underneath them while I was shipping. A friend put it bluntly: "I don't work on the tool. The tool is a means to an end, not the end. Disposable tooling." He doesn't build polished tool repositories. He builds what he needs, uses it, and moves on. The tool is scaffolding, not the building. ## 20+ products. 0 revenue. In the past six months, I've shipped roughly 8,000 GitHub contributions across 20+ projects. MCP servers, browser extensions, CLI plugins, observation libraries, desktop apps. Here's a partial list: - **[claudebin](https://claudebin.com/)** ([write-up](/project/claudebin/)): a session-sharing tool for Claude Code. - **[agentprobe](https://www.npmjs.com/package/agentprobe)** ([write-up](/project/agentprobe/)): an observation library for AI coding agents. Four provider integrations, published on npm. - **micode** ([write-up](/project/micode/)): a structured AI coding workflow plugin for OpenCode. Version 0.9.1. - **octto** ([write-up](/project/octto/)): an interactive browser-based brainstorming tool. Version 0.3.1. - **[claude-notes](https://github.com/vtemian/claude-notes)** ([write-up](/project/claude-notes/)): a CLI tool to transform Claude Code JSONL transcripts into readable, searchable HTML. - **[wunderlabs](https://wunderlabs.dev/)**: Lovable for existing codebases. - **[cursouls](https://cursouls.xyz/)** ([write-up](/project/cursouls/)): a pixel-art agent visualization extension for VS Code. Every single one of these works. They're tested, documented, published. And every single one of them generates zero revenue. Here's the part that stings: I built most of these with AI agents. They didn't take months. They took days. I wasn't slowly hand-crafting artisan software. I was shipping fast, with modern tools, into a void. Take agentprobe. It gives you real-time observability for AI coding agents. Four provider integrations, clean API, published on npm. And here's the thing: if you actually need real-time notifications from your agents, you don't install agentprobe. You just ask Claude to implement that specific piece for your specific setup. It takes 20 minutes. Why would you add a dependency when the solution tailored to your exact problem is just as fast to build? That's the uncomfortable realization. **The problem isn't build speed. The problem is that generic solutions lose to specific ones when specific is just as cheap.** I wasn't competing with other developers. I was competing with the user's own Claude session. ## The IDE is now Claude Here's what changed. A year ago, if you wanted a script to parse your bank statements into a spreadsheet, you had two options: find a SaaS that does it (and pay $15/month), or spend an evening writing a parser. Today, you open Claude, paste a sample statement, and have a working parser in 10 minutes. Not a prototype. A working tool, tailored to your exact bank format. That's a trivial example, but the pattern scales. Every tool I built solves a real problem. The issue is that the user can now solve their exact version of that problem, with their exact constraints, faster than they can evaluate whether my generic version fits. **If your tool survives longer than the problem it solves, you over-engineered it.** Software became so cheap to produce that paying a monthly subscription for a simple tool makes less and less sense. Andrej Karpathy [called it "vibe coding"](https://x.com/karpathy/status/1886192184808149383): you describe what you want, the AI builds it, and you move on. Why subscribe to a SaaS that sort-of does what you need when you can generate exactly what you need, tailored to your specific problem, in minutes? I did the research. I went looking for new SaaS products built with [Lovable](https://lovable.dev), [Base44](https://base44.com), or [Bolt](https://bolt.new) that have meaningful revenue at scale. I found micro-wins (a few paying customers here, a handful of lifetime deals there), but nothing approaching a real SaaS business. Yet the platforms themselves are printing money: Lovable hit [$300M ARR](https://sacra.com/c/lovable/) and a $6.6B valuation. Bolt.new [reached $40M ARR](https://www.nocode.mba/articles/bolt-vs-lovable) in its first six months. The platforms that help you build disposable tools are thriving. The disposable tools themselves are not businesses. ## From dev tools to agent-ready tools But something else is happening that's more interesting than "SaaS is dead." The value isn't gone. It moved. It shifted from developer tools (things humans use directly) to agent-ready tools (things AI agents use on behalf of humans). The difference is subtle but important. A developer tool has a UI, documentation, onboarding. An agent-ready tool has an API, clear inputs and outputs, and can be called programmatically. The user never touches it. They say "Claude, deploy this" or "Claude, use this style" and the agent figures out which tools to invoke. MCP is one implementation of this, and it's growing fast: [97 million monthly SDK downloads](https://www.pento.ai/blog/a-year-of-mcp-2025-review), 10,000+ public servers, and [governance under the Linux Foundation](https://www.anthropic.com/news/donating-the-model-context-protocol-and-establishing-of-the-agentic-ai-foundation) with backing from Anthropic, OpenAI, Google, Microsoft, and AWS. Projects like [OpenClaw](https://github.com/openclaw/openclaw) are attempting to be distribution layers for agent tools, routing MCP-compatible skills to millions of users across WhatsApp, Telegram, and Slack. But the pattern is broader than any single protocol or runtime. Any tool that an AI agent can discover, understand, and call autonomously is an agent-ready tool. The value proposition changed: you're not selling to a human who evaluates your landing page. You're making your capability available to an agent that evaluates your interface programmatically. ## The real opportunity: the environment If disposable tools are the new normal, then the opportunity isn't in building better tools. It's in building the environment where disposable tools thrive. Think about it. You generate a tool in Claude in 20 minutes. Great. Now what? - Where do you run it? Your laptop? A server? - How do you connect it to your other tools? - How do you share it with your team? - How do you preview what it does before trusting it? - How do you keep costs down when you're spinning up tools daily? Hosting, integrations, previews, cost reduction. The infrastructure layer for disposable tools. Some companies are starting to build pieces of this. [E2B](https://e2b.dev) raised [$35M to build secure sandboxes](https://e2b.dev/blog/series-a) for AI-generated code, using Firecracker microVMs that spin up in under 200ms. They're [used by 88% of Fortune 100 companies](https://venturebeat.com/ai/how-e2b-became-essential-to-88-of-fortune-100-companies-and-raised-21-million). [Val Town](https://www.val.town/) lets you deploy small functions without worrying about infrastructure. But nobody has put the full picture together yet. The analogy: containers didn't win because Docker was a great tool. Containers won because Kubernetes, AWS ECS, and the entire ecosystem around them made it trivial to run containers at scale. The container itself is disposable. The orchestration layer is the business. Docker popularized containers but [failed to capture the orchestration layer](https://techcrunch.com/2019/11/13/mirantis-acquires-docker-enterprise/), had to sell its enterprise business, and nearly died. The value accrued to the platform, not the tool. Disposable tools need their Kubernetes. ## I don't have the answer I'm not going to wrap this up with a neat conclusion, because I don't have one. I spent six months building tools that the world no longer needs in their polished, permanent form. That's a hard thing to admit when you've poured real effort into them. But here's what I keep coming back to: The cost of producing software is approaching zero. Google reported that [25% of its new code](https://blog.google/technology/ai/google-gemini-ai-update-december-2024/) is now AI-generated. GitHub's research showed developers [complete tasks 55% faster](https://github.blog/news-insights/research/research-quantifying-github-copilots-impact-on-developer-productivity-and-happiness/) with Copilot. That changes everything about how we think about tools, products, and what's worth paying for. The SaaS model built on "we'll maintain this tool so you don't have to" loses its value when maintaining it yourself takes minutes, not months. Something new is forming. Agent-ready tools, disposable by design, running on infrastructure that doesn't exist yet. I don't know exactly what it looks like. But I know that building another polished developer tool isn't it. Reusable tools are a tax on speed. My friend was right. The tool is not the end. It never was. --- ### Harness Engineering: Why Your AI Coding Setup Matters More Than the Model - **URL**: https://blog.vtemian.com/post/harness-engineering/ - **Published**: March 28, 2026 - **Word Count**: 3009 words - **Tags**: ai, agentic-programming, deep-dive - **Summary**: My tech friends tried AI coding tools and said they don't work. We use the same models. The difference is the harness. Here's my full setup. I have tech friends, good developers, who tried Claude Code or Cursor or Codex. They gave it a task, it wrote some code, the code was wrong, and they told me "AI coding doesn't work." I hear this a lot. But I use Claude Code every day. It writes code, runs tests, manages my git workflow, plans my day, helps me [prepare debates](/talk/why-i-think-software-development-will-be-automated-by-2030/). We're using the same models. So why are our experiences so different? The difference is what OpenAI calls "harness engineering" [1]: the discipline of designing environments, boundaries, and feedback loops that let AI agents do reliable work. The first step of harness engineering is a single file: `CLAUDE.md`. ## The Problem Out of the box, Claude Code is a capable model with generic defaults. It doesn't know your coding style. It doesn't know your project structure. It doesn't know that you hate mocks in tests, that you use TDD, or that you never want it to disable a pre-commit hook. So it guesses. And guessing produces mediocre results. The fix isn't a better model. It's giving the model the right context and the right constraints. That's what `CLAUDE.md` does. ```mermaid flowchart LR A[You] -->|generic prompt| B[Claude Code] B -->|guesses style| C[Mediocre Code] A2[You] -->|generic prompt| D[Claude Code + CLAUDE.md] D -->|knows your rules| E[Code That Fits] ``` ## The Mental Shift Before we get into configuration, there's a mindset change that everything else depends on. Most people use AI coding tools like a search engine. They ask a question, get an answer, copy it into their project. Or they describe a task, get code back, paste it in, and debug it themselves. The AI generates. The human operates. Flip it. Your job is to provide workflows, tools, and boundaries. The AI's job is to do everything else: write code, run the project, read logs, query the database, check production, deploy, fix bugs. You don't operate. You direct. This means giving Claude access to everything you have access to. Logs. Database. Local environment. Production. Deployment pipeline. GitHub. Monitoring. If you can see it, Claude should be able to see it. Why? Because context is everything. When Claude can start the project, see the startup logs, hit an error, read the stack trace, check the database state, and trace the issue end-to-end, it builds the correct context by itself. When you copy-paste a log snippet into the chat, you're filtering. You're deciding what's relevant. And you might be wrong. The difference isn't capability. The model is the same. The difference is context. And context comes from access. ```mermaid flowchart LR subgraph before["Before: You are the middleman"] direction TB P1[Project] -->|you run| Y1[You] Y1 -->|copy logs| C1[Claude] C1 -->|code snippet| Y1 Y1 -->|paste + debug| P1 end subgraph after["After: Claude operates directly"] direction TB C2[Claude] -->|make dev| P2[Project] P2 -->|logs, errors| C2 C2 -->|fix + verify| P2 end style before fill:#f4e8e8,stroke:#5a2d2d style after fill:#e8f4e8,stroke:#2d5a2d ``` ## Part 1: User-Level CLAUDE.md Your user-level `CLAUDE.md` lives at `~/.claude/CLAUDE.md`. It loads into every conversation, every project. This is where you define who you are, how you work, and what you won't tolerate. ### Define the Relationship This is the most important section. Without it, Claude defaults to agreeable assistant mode. It tells you everything is great and produces mediocre results. ```markdown ## Our relationship - We're colleagues working together - no formal hierarchy. - Don't glaze me. NEVER be agreeable just to be nice. - YOU MUST call out bad ideas and mistakes - I depend on this - When you disagree, push back. Cite technical reasons if you have them. If it's just a gut feeling, say so. ``` With this, Claude tells me when my architecture is wrong, when I'm over-engineering, when my test doesn't test anything. It pushes back. That's the whole point. ### Set Your Engineering Principles ```markdown ## Writing code - YAGNI. The best code is no code. - Make the SMALLEST reasonable changes - Simple, clean, maintainable over clever or complex - NEVER throw away or rewrite implementations without EXPLICIT permission - MATCH the style of surrounding code ``` Notice the language: `YOU MUST`, `NEVER`, `ALWAYS`. This matters. I ran 150+ trials measuring how well Claude follows instructions [2]. Terse, direct rules achieve 94.8% compliance. Verbose explanations achieve 86.6%. Academic research backs this up: Jaroslawicz et al. found that even frontier models only achieve 68% accuracy at 500 instructions, with linear compliance decay as instruction count grows [4]. Strong language, fewer words, better results. ### Enforce Process ```markdown ## Test Driven Development - FOR EVERY NEW FEATURE OR BUGFIX, follow TDD ## Version Control - NEVER SKIP, EVADE OR DISABLE A PRE-COMMIT HOOK - Commit frequently throughout development - NEVER use `git add -A` unless you've just done a `git status` ## Testing - NEVER write tests that "test" mocked behavior - NEVER implement mocks in end to end tests - Test output MUST BE PRISTINE TO PASS ``` Without these rules, Claude takes shortcuts. It disables hooks when they fail. It writes tests that mock everything and test nothing. It skips commits. The rules make it disciplined. ### Control Proactiveness ```markdown ## Proactiveness **Do as much as you can, always.** Don't ask "should I do X?" - just do it. Only pause for confirmation when: - Multiple valid approaches exist and the choice matters - The action would delete or restructure existing code - You genuinely don't understand what's being asked ``` This is the difference between an assistant that asks permission for everything and one that gets work done. Here's how the CLAUDE.md hierarchy stacks: ```mermaid flowchart TD G["~/.claude/CLAUDE.md
Global: personality, principles, process"] --> P P["project/CLAUDE.md
Project: architecture, commands, boundaries"] --> F1 P --> F2 F1["src/web/CLAUDE.md
Frontend: React rules, design system"] F2["src/tauri/CLAUDE.md
Backend: Rust safety, async patterns"] style G fill:#e8f4e8,stroke:#2d5a2d style P fill:#e8e8f4,stroke:#2d2d5a style F1 fill:#f4e8e8,stroke:#5a2d2d style F2 fill:#f4e8e8,stroke:#5a2d2d ``` Each level adds specificity. Global sets the baseline. Project overrides or extends. Folder narrows further. ## Part 2: Project-Level CLAUDE.md User-level sets your personality. Project-level sets the boundaries for your codebase. This is where harness engineering really starts. The key insight from OpenAI's experiment: "give Codex a map, not a 1,000-page instruction manual." Your project `CLAUDE.md` should be a concise entry point that tells AI what it's working with and where to find more. ### Describe the Architecture ```markdown # Opnble: Project Rules ## What is Opnble? Native desktop app (macOS/Linux/Windows) that lets non-technical users run React/Next.js projects locally. ## Architecture ### Three Codebases 1. Frontend (src/web/) - React 19 + TypeScript + Tailwind 2. Backend (src/tauri/) - Rust (Tauri 2.x) 3. VM Agent (src/agent/) - Rust gRPC server ### Data Flow 1. User adds repo URL -> React hook calls Tauri command 2. Rust clones via libgit2 3. Container manager starts Node.js with repo mounted 4. Logs stream via Tauri events 5. Frontend renders in embedded webview ``` Now when I say "fix the auth flow," Claude knows to check both the React frontend and the Rust backend. Without this, it guesses which codebase to modify. ### Give It the Tooling Commands ```markdown ## Tooling make install # install all dependencies make dev-all # start both Vite + Tauri make check # full quality gate (lint + test) make restart # kill all, reset state, start fresh Always use `make restart` from a clean state. ``` This is critical. Claude needs to know how to build, run, and verify the project. If it can't run `make check` to validate its own work, it's flying blind. ### Make Every Workflow Scriptable Here's something most people miss entirely: if you do something manually, the AI can't do it. Every workflow that lives in your head or in a wiki page is invisible to Claude. Think about what you do daily as an engineer: start the project, restart after a crash, run tests, check logs, query the database, deploy to staging, debug a failing service. If any of these requires you to "just know" the right sequence of commands, Claude can't help. The fix: put everything in a Makefile or scripts directory. Not for you. For the AI. ```makefile # Every workflow the AI needs, one command away make install # install all dependencies make dev # start dev environment make restart # kill everything, reset state, start fresh make status # show what's running make test # run all tests make test-unit # just unit tests make test-e2e # just e2e tests make lint # static analysis + formatting make check # full quality gate (lint + test + type check) make logs # tail aggregated logs make db-shell # open database REPL make db-migrate # run pending migrations make db-seed # seed with test data make deploy-staging # deploy to staging make deploy-prod # deploy to production (requires confirmation) ``` Then reference them in your `CLAUDE.md`: ```markdown ## Tooling Always use Makefile targets. Never run raw commands. - `make restart` to start from clean state - `make check` to verify all changes before committing - `make logs` to debug runtime issues - `make db-shell` to inspect database state ``` But here's the mindset shift that matters: don't just make these scripts available. Let Claude actually use them. Instead of you starting the project in a separate terminal, let Claude start it in the background. Instead of you reading logs and copy-pasting errors into the chat, let Claude tail the logs itself. Instead of you taking screenshots of the UI, let Claude drive the browser. Why? Because when Claude runs the project itself, it has all the context: the startup sequence, the logs, the errors, the stack traces. It sees the problem firsthand. When you copy-paste a log snippet, you're filtering. You're deciding what's relevant. You might miss the line above the error that actually explains the cause. I used to do this: run the project, see an error, copy the log, paste it to Claude, explain what I was trying to do. Now I just say "start the project and fix whatever's broken." Claude runs it, sees the error in context, traces the cause, and fixes it. Faster, more accurate, less back-and-forth. This is the real shift. You're not just giving AI access to your tools. You're letting it operate the way you operate. Start the project. See what happens. Fix what's broken. Verify the fix. Repeat. The same loop you do, but automated. OpenAI learned this the hard way building their internal product with Codex: "Early progress was slower than we expected, not because Codex was incapable, but because the environment was underspecified. The agent lacked the tools, abstractions, and internal structure required to make progress toward high-level goals" [1]. They built their app bootable per git worktree, wired Chrome DevTools into the agent, and exposed an entire observability stack (logs, metrics, traces) that Codex could query directly. The result: single Codex runs working on a task for 6+ hours, autonomously. You don't need to go that far. But every script you add is a capability the AI gains. Start with the basics: start, stop, test, lint, logs, database. ### Set the Feedback Loop This is the part most people miss. AI code generation is only half the equation. The other half is **verification**. Claude needs a way to check if what it generated is correct. Locally. Before committing. The feedback loop looks like this: ```mermaid flowchart LR W[Write Code] --> L[Run Linter] L -->|pass| T[Run Tests] L -->|fail| W T -->|pass| O[Check Output] T -->|fail| W O -->|clean| C[Commit] O -->|errors in logs| W style C fill:#e8f4e8,stroke:#2d5a2d style W fill:#f4f4e8,stroke:#5a5a2d ``` In your project `CLAUDE.md`: ```markdown ## Verification After ANY code change: 1. Run `make lint` - must pass with zero warnings 2. Run `make test` - all tests must pass 3. Check test output is clean - no unexpected errors in logs 4. If lint or tests fail, fix before committing. NEVER skip. ``` OpenAI's team learned the same thing: "When something failed, the fix was almost never 'try harder.' It was always 'what capability is missing to let the agent verify its own work?'" [1]. ## Part 3: Boundaries and Guardrails Harness engineering isn't about giving AI freedom. It's about giving it freedom *within strict boundaries*. ### Tell It How to Write Code Your `CLAUDE.md` defines the rules. Your linter enforces them. ```markdown ## Code Style - No classes for business logic. Use factory functions. - No nesting beyond 2 levels inside a function body - Max function length: 40 lines - No `any` types, no type assertions - Use `unknown` at system boundaries, normalize with Zod ``` But here's the key: these rules also exist in your linter config. ESLint, Biome, Clippy, ruff. The `CLAUDE.md` tells Claude what to aim for. The linter catches what it misses. Two layers of enforcement. OpenAI encoded this as custom linters with error messages designed to inject remediation instructions directly into agent context [1]. Same principle: the tool tells the agent what went wrong and how to fix it. Why two layers? Because AI-generated code carries 1.7x more defects per PR than human-written code [5]. But teams that pair AI with verification loops see 3.5x better code quality [6]. The `CLAUDE.md` is guidance. The linter is enforcement. You need both. As Addy Osmani puts it: "Without checks, an agent might merrily introduce bugs or failing builds while thinking it succeeded" [7]. CodeScene's research adds: "AI performs best in healthy code. Enabling AI acceleration requires more rigor, more structure, not less" [8]. You can see this two-layer approach in my open source projects: - [lint-claude](https://github.com/vtemian/lint-claude): enforces your CLAUDE.md guidelines using Claude AI — the automated enforcement layer this article describes - [agentprobe](https://github.com/vtemian/agentprobe/blob/main/CLAUDE.md): strict TypeScript rules (no `any`, max 40 lines, factory functions) enforced by Biome - [blog.vtemian.com](https://github.com/vtemian/blog.vtemian.com/blob/content/CLAUDE.md): Hugo build commands + content conventions - [claudebin.com](https://github.com/wunderlabs-dev/claudebin.com/blob/main/CLAUDE.md): session sharing platform for Claude Code transcripts - [cursouls](https://github.com/wunderlabs-dev/cursouls/blob/main/CLAUDE.md): pixel art agent visualization for VS Code - [claude-notes](https://github.com/vtemian/claude-notes/blob/main/CLAUDE.md): TypeScript CLI for transforming Claude Code transcripts - [blueprints.md](https://github.com/vtemian/blueprints.md/blob/main/CLAUDE.md): markdown-to-code generator with strict architecture boundaries ### Set the Guardrails Defining what Claude should NOT do is as important as defining what it should do. ```markdown ## Boundaries - NEVER throw away or rewrite existing implementations without explicit permission - NEVER delete a test because it's failing - NEVER skip a pre-commit hook - NEVER use `git add -A` without checking `git status` first - NEVER implement backward compatibility without approval ``` These aren't restrictions on Claude's capability. They're guardrails that prevent the most common failure modes. Every one of these rules exists because Claude did the wrong thing without them. I added each one after a real incident. ### Folder-Level Scoping For larger projects, different parts of the codebase need different rules. In Opnble, the Rust backend and React frontend have separate `CLAUDE.md` files: **`src/web/CLAUDE.md`** (frontend): ```markdown - Max function: 40 lines, max cognitive complexity: 10 - No classes, hooks + composition only - No `any` types, no type assertions - Terminal aesthetic: monospace, uppercase buttons ``` **`src/tauri/CLAUDE.md`** (backend): ```markdown - unwrap_used deny: use .expect("reason") or ? - todo!(), dbg_macro, print statements deny: use tracing - Exhaustive enum matching - Never hold Tokio Mutex across .await ``` The frontend enforces React patterns. The backend enforces Rust safety. Same repo, completely different constraints, both enforced automatically. ## What the Data Says I ran 150+ trials to measure Claude's compliance with `CLAUDE.md` instructions [2]. The findings: | Finding | Implication | |---------|------------| | Compliance drops to 0% beyond ~128K chars | Keep total CLAUDE.md content under 80K characters | | 200+ rules at 98% compliance | Rule count isn't the bottleneck, total size is | | Terse rules: 94.8% compliance | One sentence per rule. No explanations needed. | | Verbose rules: 86.6% compliance | Explanations waste 5x context for 8% worse results | | 4 levels of nesting works fine | Total size matters, not depth | HumanLayer's engineering team reached the same conclusion independently [3]: target under 300 lines, don't outsource linting to the LLM, and use progressive disclosure. Arize AI ran a controlled experiment and measured **+5.19% accuracy on unseen repos** and **+10.87% on the same repo** purely from refined CLAUDE.md configuration [9]. Boris Cherny, the creator of Claude Code, keeps his CLAUDE.md at ~100 lines and updates it multiple times a week [10]. Anthropic's own best practices docs say it directly: "For each line, ask: 'Would removing this cause Claude to make mistakes?' If not, cut it" [11]. ## The Full Picture Put it all together and this is what harness engineering looks like: ```mermaid flowchart TD subgraph context["Context Layer"] CM["CLAUDE.md
(rules, architecture, boundaries)"] MF["Makefile / Scripts
(workflows, commands)"] AC["Access
(logs, DB, deploy, monitoring)"] end subgraph agent["Agent Loop"] direction LR WRITE[Write Code] --> LINT[Lint] LINT --> TEST[Test] TEST --> VERIFY[Verify Output] VERIFY -->|fail| WRITE VERIFY -->|pass| HOOK[Pre-commit Hook] HOOK -->|fail| WRITE HOOK -->|pass| COMMIT[Commit] end context --> agent YOU[You: intent + review] -.->|direct| context style context fill:#e8e8f4,stroke:#2d2d5a style agent fill:#f4f4e8,stroke:#5a5a2d style YOU fill:#e8f4e8,stroke:#2d5a2d style COMMIT fill:#e8f4e8,stroke:#2d5a2d ``` You provide the context layer: rules, workflows, access. Claude runs the agent loop: write, lint, test, verify, commit. If any step fails, it fixes and retries. You direct. Claude operates. The model is the same for everyone. The harness is what makes it work. ## Getting Started 1. Create `~/.claude/CLAUDE.md` with your relationship rules, engineering principles, and process. 2. Add a `CLAUDE.md` to each project with architecture, tooling commands, and verification steps. 3. Make sure your linter and test suite run locally. Claude needs the feedback loop. 4. Use terse, direct language. One sentence per rule. 5. Add boundary rules after every incident. Each rule is a lesson learned. 6. Iterate. The file evolves with you. My full global config is public at [github.com/vtemian/.claude](https://github.com/vtemian/.claude/blob/main/CLAUDE.md). My compliance research is at [github.com/vtemian/claude-context](https://github.com/vtemian/claude-context). Fork them, customize them, make them yours. Stay curious ☕ --- [1] OpenAI, "Harness engineering: leveraging Codex in an agent-first world." [openai.com/index/harness-engineering](https://openai.com/index/harness-engineering/) [2] Vlad Temian, "CLAUDE.md Compliance Research." [github.com/vtemian/claude-context](https://github.com/vtemian/claude-context) [3] HumanLayer, "Writing a Good CLAUDE.md." [humanlayer.dev/blog/writing-a-good-claude-md](https://www.humanlayer.dev/blog/writing-a-good-claude-md) [4] Jaroslawicz et al. (2025), "How Many Instructions Can LLMs Follow at Once?" [arxiv.org/abs/2507.11538](https://arxiv.org/abs/2507.11538) [5] IQ Source, "AI code carries 1.7x more defects per PR." [iqsource.ai](https://www.iqsource.ai/en/blog/ai-code-review-quality-governance/) [6] Qodo, "State of AI Code Quality 2025." [qodo.ai/reports/state-of-ai-code-quality](https://www.qodo.ai/reports/state-of-ai-code-quality/) [7] Addy Osmani, "Self-Improving Coding Agents." [addyosmani.com/blog/self-improving-agents](https://addyosmani.com/blog/self-improving-agents/) [8] CodeScene, "Agentic AI Coding: Best Practice Patterns." [codescene.com/blog/agentic-ai-coding-best-practice-patterns](https://codescene.com/blog/agentic-ai-coding-best-practice-patterns-for-speed-with-quality) [9] Arize AI, "CLAUDE.md Best Practices from Optimizing Claude Code with Prompt Learning." [arize.com/blog/claude-md-best-practices](https://arize.com/blog/claude-md-best-practices-learned-from-optimizing-claude-code-with-prompt-learning/) [10] MindWiredAI, "Claude Code Best Practices: Inside the Creator's 100-Line Workflow." [mindwiredai.com](https://mindwiredai.com/2026/03/25/claude-code-creator-workflow-claudemd/) [11] Anthropic, "Best Practices for Claude Code." [code.claude.com/docs/en/best-practices](https://code.claude.com/docs/en/best-practices) **Further reading:** - Martin Fowler, [Harness Engineering](https://martinfowler.com/articles/exploring-gen-ai/harness-engineering.html) - Anthropic, [Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) - Armin Ronacher, [Agentic Coding Recommendations](https://lucumr.pocoo.org/2025/6/12/agentic-coding/) - Philipp Schmid, [The importance of Agent Harness in 2026](https://www.philschmid.de/agent-harness-2026) --- ### MCP Is Great for Tools. Terrible for Agents. - **URL**: https://blog.vtemian.com/post/mcp-is-great-for-tools-terrible-for-agents/ - **Published**: February 27, 2026 - **Word Count**: 1868 words - **Tags**: mcp, agentic-programming, typescript, deep-dive - **Summary**: Building claudebin as an MCP server for Claude Code. JSON-RPC protocol, session extraction, device auth, and an honest comparison with OpenCode's plugin model. We (Balaj Marius and I) built [Claudebin](https://claudebin.com) because we were tired of losing the "messy middle" of our Claude Code sessions. You debug a race condition, the agent rewrites three files, and the transcript is gone the moment you close the terminal. Claudebin turns those ephemeral sessions into shareable URLs. Run `/claudebin:share`, and you get a permanent link to the exact moment things clicked. But this post isn't just about the tool. It's about what we learned while building an MCP server from scratch, the hidden frustrations of the JSON-RPC stream, and a fundamental realization: the Model Context Protocol (MCP) is the right model for tools, but it might be the wrong one for agents. **TL;DR** - MCP is JSON-RPC 2.0 over stdio. Your plugin is a child process that Claude Code pipes JSON to. Simple, isolated, and language-agnostic. - Claude Code's slash commands are non-deterministic. A markdown file instructs the LLM to call your tool. The dispatch goes through model reasoning. It works. Usually. - **The Stdio Trap:** A single `console.log` in an MCP server corrupts the JSON-RPC stream. I spent an hour debugging a "server failure" only to realize my own debug logs were blowing up the pipe. - Claude Code plugins can't observe, intercept, or modify agent behavior. You get a pipe and a schema. No lifecycle hooks. No shared state. - OpenCode plugins run in-process with 11+ hooks. The DX is better for complex systems, but the portability is worse. - **My Verdict:** If you need more than a simple tool, you're fighting the architecture. ## MCP: JSON-RPC 2.0 Over stdio The [Model Context Protocol](https://modelcontextprotocol.io/) sits on top of JSON-RPC 2.0 [1]. Claude Code spawns your server as a child process, pipes stdin/stdout, and exchanges newline-delimited JSON [2]. No HTTP, no WebSockets, no port allocation. One child process per MCP server, lifecycle tied to the parent [2]. ```mermaid flowchart LR CC[Claude Code] -->|stdin: JSON-RPC| MCP[MCP Server process] MCP -->|stdout: JSON-RPC| CC CC -->|stdin: JSON-RPC| MCP2[Another MCP Server] MCP2 -->|stdout: JSON-RPC| CC ``` When Claude Code starts your server, it sends an `initialize` request: ```json → {"jsonrpc":"2.0","id":1,"method":"initialize","params":{ "protocolVersion":"2025-11-25", "capabilities":{"roots":{"listChanged":true}}, "clientInfo":{"name":"claude-code","version":"1.0.0"}}} ← {"jsonrpc":"2.0","id":1,"result":{ "protocolVersion":"2025-11-25", "capabilities":{"tools":{}}, "serverInfo":{"name":"claudebin","version":"1.0.0"}}} → {"jsonrpc":"2.0","method":"notifications/initialized"} ``` Capability negotiation [3]. The client declares what it supports, the server responds with its own. Claudebin only declares `tools`. The third message is a notification (no `id`, no response expected), signaling the handshake is complete. Then Claude discovers tools via `tools/list` and invokes them via `tools/call` [4]. The tool response uses MCP's content block format: ```json ← {"jsonrpc":"2.0","id":3,"result":{ "content":[{"type":"text","text":"https://claudebin.com/threads/abc123"}]}} ``` MCP separates *protocol errors* (unknown tools, malformed requests, server failures) from *tool errors* (your code ran but failed). Tool errors return a successful response with `isError: true` [4]. The distinction matters because Claude can reason about tool errors and retry. Protocol errors rarely lead to successful recovery. One gotcha with the long-lived process model: `console.log` goes to stdout and corrupts the JSON-RPC stream [2]. This bites every new MCP developer. Use `console.error` for debugging. ## Session Extraction The genuinely hard part of building Claudebin wasn't the protocol—it was the archaeology. I had to figure out exactly where Claude Code hides its session data and how to extract it without breaking anything. Sessions live as `.jsonl` files under `~/.claude/projects/`. But the directory names aren't human-readable; they're normalized project paths. I found that `/Users/vlad/projects/my-app` becomes `-Users-vlad-projects-my-app`. Every non-alphanumeric character (including that leading slash) maps to a dash. I didn't find this in any documentation. I found it by digging through the Claude Code source and verifying it against my own filesystem. One regex, one pass: ```typescript const normalizeProjectPath = (projectPath: string): string => { return projectPath.replace(/[^a-zA-Z0-9]/g, "-"); }; ``` Getting this wrong means a silent "session not found" error, which is a terrible user experience. Claudebin has to handle this normalization perfectly to find the right project folder. Inside, things get even messier: ``` ~/.claude/projects/Users-vlad-projects-my-app/ ├── default.jsonl # Main conversation ├── agent-a1b2c3d4.jsonl # Subagent session (Task tool) ├── agent-e5f6g7h8.jsonl # Another subagent ``` I had to write logic to filter out those `agent-*` files and pick the most recent one by `mtime`. Each line is a self-contained JSON object. I read the whole file as a UTF-8 string and ship it to the backend. **The Memory Reality Check:** Node.js strings are UTF-16 internally, so a 50MB session suddenly eats ~100MB of heap. By the time you JSON-serialize the HTTP body, you're hitting a 200MB peak. I considered a streaming upload, but for the typical 5MB session, the complexity wasn't worth the trade-off. I settled on a 50MB client-side safety valve to keep things fast and predictable. ## Device Authorization MCP servers run in a terminal. No browser window, no OAuth redirect URI. Claudebin uses a pattern similar to RFC 8628 (Device Authorization Grant) [7]: ```mermaid sequenceDiagram participant P as Plugin participant B as Backend participant Br as Browser P->>B: POST /api/auth/start B-->>P: { code, authUrl } P->>Br: exec("open", authUrl) Br->>B: User logs in via OAuth B->>B: Mark code as verified loop Every 2s, up to 5min P->>B: GET /api/auth/poll?code=abc end B-->>P: { accessToken, refreshToken, expiresAt } ``` The one-time code correlates the CLI and browser session. Tokens are cached in `~/.claudebin/config.json` with `0o600` permissions [5] (same pattern as `~/.ssh/`). Before every share, claudebin checks the cached token. If it's within 5 minutes of expiry, it refreshes proactively rather than failing mid-upload. Both auth and session publishing use async polling. Rather than duplicate the retry logic, claudebin has a generic `poll()` with configurable interval, timeout, and a type guard callback. Auth polls for 5 minutes (user might be slow). Session processing polls for 2 minutes. No backoff, fixed 2-second interval. For a single-user CLI tool generating 30 req/min during a poll, this is fine. Why not WebSockets? Connection lifecycle management, stateful load balancing, maintaining a persistent socket alongside the stdio event loop. The polling implementation is 20 lines. A robust WebSocket client is 200+. For a 5-15 second wait, 2 seconds of overhead is acceptable. ## Two Plugin Models, Two Philosophies I've also built [micode](https://github.com/vtemian/micode), a plugin for [OpenCode](https://opencode.ai/) with 26 specialized agents. The experience of building for both platforms exposed a fundamental design disagreement about what a "plugin" should be. In Claude Code, a plugin is a separate process that the LLM talks to over a protocol. In OpenCode, a plugin is code that runs inside the agent runtime. This isn't a minor implementation detail. It shapes everything. ```mermaid flowchart TD U1["User: /share"] --> MD["Markdown file"] --> LLM["LLM interprets"] --> RPC["JSON-RPC over stdio"] --> ZOD["Zod validates"] --> FN1["Handler runs"] U2["User: /init"] --> CFG["Config lookup"] --> AGT["Agent receives prompt"] --> FN2["Tool executes"] style U1 fill:#f9f,stroke:#333 style U2 fill:#bbf,stroke:#333 style FN1 fill:#f9f,stroke:#333 style FN2 fill:#bbf,stroke:#333 ``` The left chain (pink) is Claude Code: 6 hops. The right chain (blue) is OpenCode: 4 hops. ### Dispatch: Deterministic vs Probabilistic When you type `/claudebin:share` in Claude Code, the slash command is a markdown file. Claude reads it, interprets the natural language instructions, and *decides* to call the MCP tool. The dispatch goes through model reasoning. It's probabilistic. Claude usually gets it right. Usually. When you type `/init` in OpenCode, a config object maps the command to an agent: ```typescript config.command = { init: { agent: "project-initializer", template: `Initialize this project. $ARGUMENTS`, }, }; ``` String substitution. Hash map lookup. Deterministic. `/init` always routes to `project-initializer`. There is no interpretation step. For claudebin, a single-tool plugin, this means Claude spends 1-2 seconds reasoning about which tool to call when there's only one option. That's model inference time burned on a decision with exactly one valid answer. ### Hooks: Observing vs Hoping This is where the gap gets wide. OpenCode plugins have lifecycle hooks. You can intercept and modify behavior at 11+ points: before/after tool execution, on message receive, on context compaction, on permission prompts. In micode, the `tool.execute.after` hook truncates tool output to stay within context limits. The `chat.params` hook injects project context files before each LLM call. The `experimental.chat.system.transform` hook modifies the system prompt dynamically based on what the agent is doing. ```typescript return { "tool.execute.after": async (input, output) => { await tokenAwareTruncation(input, output); await artifactAutoIndex(input, output); }, "chat.params": async (input, output) => { await injectProjectContext(input, output); await loadContinuityLedger(input, output); }, }; ``` Claude Code plugins can't do any of this. Your MCP server is a black box behind a pipe. You get a request, you return a response. You can't observe what Claude is doing, intercept other tools, modify the system prompt, or react to session events. You can't even know how much context is left. This matters in practice. Micode tracks file operations across tool calls and auto-indexes artifacts. It detects when context is running low and triggers a ledger dump. It enforces code style patterns by injecting constraints into the system prompt. None of this is possible through MCP. You'd have to rely on Claude's built-in behavior and hope it does the right thing. ### State: Shared vs Isolated OpenCode plugins share the runtime. Tools access a `ToolContext` with session ID and abort signals. Shared state lives in closures over the plugin scope. One tool can read what another tool wrote. Agents spawn subagents via `spawn_agent`, which creates fresh sessions but receives context explicitly through the prompt. MCP servers are isolated processes. If two tools need shared state, they go through the filesystem. There's no session context, no abort propagation, no shared memory. Claudebin manages its own auth state in `~/.claudebin/config.json` because there's nowhere else to put it. For simple tools like claudebin, isolation is fine. You call one function, get one result. But for a system like micode where 26 agents coordinate, spawn subagents in parallel, share continuity ledgers, and track progress across tasks, process isolation would be a straitjacket. ### So Which One Wins? Neither. They solve different problems. | | MCP (Claude Code) | Direct Plugin (OpenCode) | |-----------------------|----------------------------|---------------------------------| | Tool dispatch | LLM inference | Hash map lookup | | Lifecycle hooks | None | 11+ hook points | | State sharing | Filesystem only | In-process context | | Language support | Any (JSON-RPC over stdio) | JS/TS only (Bun runtime) | | Process isolation | Yes | No | | Portability | Claude Code, Cursor, Zed | OpenCode only | | Agent orchestration | Single agent, sequential | Multi-agent, parallel spawning | MCP is the right model for tools. Standalone utilities that do one thing, work across editors, and benefit from isolation. A hotel scraper, a session sharer, a code search engine. Write it in any language, ship it everywhere. OpenCode's model is the right one for agent systems. When you need hooks, state, parallel agents, deterministic routing, and deep integration with the runtime. When the plugin isn't a utility but a workflow engine. The problem is that Claude Code only offers MCP. If you want to build anything beyond a simple tool, you're fighting the architecture. You can't observe, you can't intercept, you can't orchestrate. You get a pipe and a JSON schema. OpenCode only offers direct plugins. If you want a Python tool, you're wrapping it in a subprocess yourself. If you want cross-editor portability, you're out of luck. The ideal platform would offer both. MCP for portable tools, direct plugins for deep integration. Neither does that today. The code is at [github.com/wunderlabs-dev/claudebin](https://github.com/wunderlabs-dev/claudebin). Stay curious ☕ --- ### References

[1] JSON-RPC 2.0 Specification - jsonrpc.org/specification

[2] MCP Specification: Transports - modelcontextprotocol.io/specification/2025-11-25/basic/transports

[3] MCP Specification: Lifecycle - modelcontextprotocol.io/specification/2025-11-25/basic/lifecycle

[4] MCP Specification: Tools - modelcontextprotocol.io/specification/2025-11-25/server/tools

[5] Claudebin source code - github.com/wunderlabs-dev/claudebin

[6] JavaScript's internal character encoding: UCS-2 or UTF-16? - mathiasbynens.be/notes/javascript-encoding

[7] RFC 8628: OAuth 2.0 Device Authorization Grant - datatracker.ietf.org/doc/html/rfc8628

--- ### vibe-infer: Learning GPU Programming with Claude Code - **URL**: https://blog.vtemian.com/post/vibe-infer/ - **Published**: February 18, 2026 - **Word Count**: 1559 words - **Tags**: gpu programming, ai tools, webgpu, wgsl, vibe-coding, learning - **Summary**: Everyone talks about AI-assisted learning. Here's what it actually looks like. From zero WebGPU knowledge to a working MNIST classifier in 155 messages. Everyone has a story about how AI helped them learn something. "I asked ChatGPT to explain monads and it finally clicked." But these stories are almost always sanitized after the fact. You get the polished takeaway, but you lose the "messy middle"—the confusion, the corrections, and the specific failure modes that actually lead to understanding. This post is the opposite of the "vibe coding" trend where the AI writes the code and you just ship it. I call this repo [vibe-infer](https://github.com/vtemian/vibe-infer) as a piece of intentional irony. I wrote every single line of GPU code. The AI was my tutor, not my ghostwriter. This is the receipt: 155 messages, captured in full, from zero WebGPU knowledge to a working MNIST classifier running compute shaders in the browser. ## Why GPU Programming? GPU programming requires a fundamentally different mental model from writing regular application code. On a CPU, you think sequentially: fetch data, process it, return a result. On a GPU, thousands of threads execute the same instruction simultaneously across different data. You stop thinking about loops and start thinking about thread indices, workgroups, and memory barriers. [WebGPU](https://developer.mozilla.org/en-US/docs/Web/API/WebGPU_API) is the modern browser API for GPU compute. Not just graphics rendering, but general-purpose parallel computation. The shader language is [WGSL](https://www.w3.org/TR/WGSL/), which looks like Rust but with its own type system and execution model. Buffers are raw bytes. There are no arrays in the JavaScript sense. You manage memory layout, type alignment, and dispatch dimensions manually. I wanted to build a neural network from scratch on the GPU. Not by importing a framework, but by writing every compute shader by hand. Matrix multiplication, ReLU activation, softmax normalization, forward pass chaining. The goal was understanding the machinery, not just producing a working demo. ## The Approach: Claude as Tutor I used [Claude Code](https://claude.ai/code), Anthropic's agentic coding tool, as a pair programming partner. But the first thing I established was the dynamic: Claude guides, I write. It explains concepts, reviews my code, catches mistakes. It does not write GPU code for me. (I've written a deeper dive into why Claude Code's choice of [MCP for its plugin architecture](https://blog.vtemian.com/post/mcp-is-great-for-tools-terrible-for-agents/) makes it great for tools but presents challenges for more complex agentic behaviors). With the ground rules set, three dynamics emerged. **I write the GPU code, Claude reviews it.** Every compute shader, every buffer allocation, every dispatch call was written by my hands. Claude's role was code reviewer: catching syntax errors, pointing out API misuse, explaining GPU-specific type requirements. When I used `0` instead of `0.0` in a WGSL shader, Claude didn't just flag it. It explained why WGSL enforces strict typing and what happens when type coercion fails on the GPU. **Claude handles the boilerplate.** HTML scaffolding, CSS layout, the canvas drawing interface. None of that teaches GPU programming. Offloading it meant I could spend 100% of my mental energy on the hard, interesting parts: how workgroups distribute threads, why uniform buffers need 16-byte alignment, how reduction operations work in softmax. **My pace, my depth.** Unlike a course with a fixed curriculum, I controlled when to go deeper and when to move on. If workgroup scheduling fascinated me, I could ask Claude to explain dispatch dimensions three different ways. If I already understood buffer creation, I could skip ahead. The curriculum emerged from the conversation, shaped by my curiosity and my gaps. ## The Journey: 8 Lessons from Zero to Inference The session naturally organized itself into eight progressive lessons, each building on the last until I had a full neural network running on the GPU. ### **The Softmax Wall: A Deep Dive into WGSL** The hardest part wasn't the matrix multiplication; it was Softmax. Unlike a simple ReLU, where every thread is independent, Softmax requires every output element to know about every *other* element in the vector. On a CPU, this is a simple loop. On a GPU, you have to worry about numerical stability. If you naively exponentiate a value like `80`, you get a number larger than the maximum value for a 32-bit float. The entire network collapses into `NaN`. I had to implement a **numerically stable Softmax**, which involves finding the maximum value in a vector first, subtracting it from every element, and *then* exponentiating. This is how the actual WGSL kernel looks: ```wgsl @compute @workgroup_size(64) fn main(@builtin(global_invocation_id) global_id: vec3) { let i = global_id.x; if (i >= params.input_size) { return; } // 1. Find max (for numerical stability) var max_val = -3.402823466e+38f; // min float for (var k = 0u; k < params.input_size; k = k + 1u) { max_val = max(max_val, input[k]); } // 2. Compute sum of exponentials var sum = 0.0f; for (var k = 0u; k < params.input_size; k = k + 1u) { sum = sum + exp(input[k] - max_val); } // 3. Normalize and output output[i] = exp(input[i] - max_val) / sum; } ``` This isn't the most optimized way to do it (a parallel reduction would be faster), but it's the version where I finally understood how GPU memory barriers and float limits work. Claude didn't just give me this code; it watched me struggle with `NaN` outputs for thirty messages until I understood *why* I needed the `max_val` subtraction. It started with **WebGPU Bootstrap**: acquiring a GPU adapter and device. The "hello world" of GPU programming. Then a **First Compute Shader** that added two numbers on the GPU. Simple, but it proved the entire pipeline worked: buffer creation, shader compilation, command encoding, dispatch, and result readback. **Matrix Multiplication** was where things got real. This is the core operation of neural networks, and on the GPU, each thread computes a single element of the output matrix. You index into flat memory using `row * numColumns + col` because GPU buffers have no concept of rows or columns. Just raw bytes. **ReLU Activation** was a welcome breather. An "embarrassingly parallel" operation where every element is independent. No coordination between threads, no shared state. Just `max(value, 0.0)` across thousands of elements simultaneously. **Softmax** was the hardest shader. Unlike ReLU, every output element depends on every other element. You need three passes: find the maximum (for numerical stability), compute exponentials and their sum, then normalize. Getting this right meant understanding reduction operations, race conditions, and why naively exponentiating large numbers produces infinity. The **Forward Pass** chained all kernels together. The critical insight here: keep data on the GPU between operations. Reading results back to JavaScript after each step would kill performance. One command encoder, multiple compute passes, data flowing between buffers without ever touching the CPU. **Real Weights** brought the system to life. A Python script trained a 784-to-128-to-10 network on MNIST (97.5% accuracy), exported the weights as raw binary, and loaded them into GPU buffers. The final lesson was an **Interactive Canvas Demo**: draw a digit, watch the GPU classify it in real-time. Here's what the learning actually looked like. I implemented the matrix multiplication kernel, and Claude reviewed my code line by line. Buffer sizes in bytes, not elements. Uniform buffers need 16-byte alignment. Missing builtin parameters for thread identification. These are GPU-specific gotchas that a textbook mentions in a footnote. A tutor catches them in your actual code and explains why they matter. ## What Made This Different Two things separated this from reading documentation or watching a YouTube tutorial. **The learning was personalized.** I didn't follow someone else's idea of what order to learn things. When Claude introduced softmax and I didn't fully grasp why subtracting the maximum before exponentiation prevents overflow, I asked it to explain numerical stability from first principles. We went down that rabbit hole until it clicked. When I already understood ReLU from prior ML experience, we spent thirty seconds on it and moved on. No filler, no padding, no waiting for the rest of the class. This is the part that surprised me most. I could verify my understanding in real-time. The loop looked like this: I write a shader, Claude reads it, finds the bugs I can't see yet, and explains not just *what's* wrong but *why* GPU programming works that way. It's not Q&A. It's closer to a code review where the reviewer happens to be infinitely patient and knows every corner of the WebGPU spec. The difference from vibe coding is worth stating plainly. In vibe coding, the AI writes the code and you ship it. Here, I wrote the code and the AI made sure I understood it. Every bug Claude caught was a concept I internalized. Every correction was a lesson I wouldn't need to learn again. The understanding compounds in a way that copy-pasting from an AI chat never does. ## The Result And then it worked. Digit 7, classified at 99.95% confidence. Eight lessons of compute shaders, buffer management, and type wrestling, running entirely in the browser. No backend, no ML framework, no CUDA. Just WebGPU, WGSL, and a neural network I built from scratch. You can [try the live demo](https://vtemian.github.io/vibe-infer/) or browse [the source code](https://github.com/vtemian/vibe-infer). Every GPU line was written by hand. The repo includes a [LESSONS.md](https://github.com/vtemian/vibe-infer/blob/main/LESSONS.md) file summarizing what each lesson covered. The entire 155-message session is captured and browsable at [claudebin.com](https://claudebin.com/threads/jmdbMowNTz). I built [claudebin](https://github.com/wunderlabs-dev/claudebin.com) with [Balaj Marius](https://balajmarius.com/) as an open-source tool for sharing Claude Code sessions. If you're using Claude Code and want to share a session with your team or publicly, it's one command to publish and get a shareable link. Stay curious ☕ --- ### Parallel Transaction Execution in Blockchains - **URL**: https://blog.vtemian.com/post/parallel-transaction-execution-in-blockchains/ - **Published**: October 18, 2024 - **Word Count**: 1746 words - **Tags**: blockchain, parallel blockchains, solana, sui, monad, sei, aptos, block-stm, state of parallel blockchains - **Summary**: How Solana, SUI, and Aptos execute transactions in parallel. Deterministic vs optimistic approaches, conflict resolution, and the trade-offs each design makes. In the last couple of years, we have seen the rise of [infrastructure](https://www.shoal.gg/p/parallel-execution-the-next-generation) projects in the blockchain space. One exciting development is [parallel blockchains](https://www.recvc.com/part-i-design-space-for-parallel-blockchains/), specifically those that execute a batch of transactions in parallel. ## Concurrency at Scale When studying those chains, you can see some parallels to classical [concurrency patterns](https://en.wikipedia.org/wiki/Concurrency_control), but those are hard to implement at scale. Why is that? In addition to executing transactions in parallel (happening at the VM layer), we also need to consider replicating their output across the network. Let's explain the problem in a little more detail. We have this distributed state across multiple nodes. We don't sync this state as it is but use state transitions. We need to [replicate state transitions](https://en.wikipedia.org/wiki/State_machine_replication) over the network. ![replicated-state-machine](replicated-state-machine.png) Those state transitions take the form of transactions bundled together in a [block](https://en.wikipedia.org/wiki/Blockchain#Blocks). A node bundles that block, then distributed and validated by other nodes. Once validated, each node mutates its local state by running [transactions](https://en.wikipedia.org/wiki/Transaction_processing). We'll have a blockchain if we chain those bundles of state transitions. Those state transitions can be quite complex since we can run arbitrary code using arbitrary input data and generating arbitrary output data. ![blockchains](blockchains.png) We must be careful which transactions can be executed in parallel because we want to execute multiple transactions in parallel and replicate them across the network. Multiple transactions can be executed in parallel if they don’t mutate the same state in the same execution cycle. ![tx-parallel](tx-parallel.png) Given this, state access will be the key to running transactions in parallel safely. ## Deterministic vs Optimistic There are two principal schools of thought when discussing state access and parallel blockchain execution: [deterministic](https://en.wikipedia.org/wiki/Deterministic_algorithm) vs [optimistic](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) execution. ### Deterministic Execution In deterministic execution, we expect the same result if we run multiple parallel transactions multiple times in different orders. We call this process deterministic because we expect the same result each time. How can this be achieved? We need to know before execution what data each transaction will use. It adds multiple data overhead since we might need to write our smart contracts so that the used data is described in the transaction’s header. Also, we might de-normalize the data, handling complex data structures that might be harder to scale. ![parallel-queues](parallel-queues.png) ### Solana Each Solana transaction contains the [“accounts”](https://solana.com/docs/core/accounts) field, in which an array of accounts used while processing transactions is specified. Accounts are data slots separate from the [program](https://www.quicknode.com/guides/solana-development/getting-started/an-introduction-to-the-solana-account-model) (smart contract) and owned by an address. The entire state comprises ownable accounts, and programs are isolated and don’t contain any account data. That’s different from EVM, where your smart contract has the business logic and the storage used. It contains the state and state transitions, all in one piece. ![solana-accounts](solana-accounts.png) Once we know what data will be used, we can group transactions that do not use duplicate accounts and run them in [parallel](https://medium.com/solana-labs/sealevel-parallel-processing-thousands-of-smart-contracts-d814b378192). Another strong point for Solana when executing transactions is the use of [pipelining](https://medium.com/solana-labs/pipelining-in-solana-the-transaction-processing-unit-2bb01dbd2d8f). The Sealevel VM can Fetch, Verify Signatures, Bank, and Write in parallel. Other blockchains have also implemented this architecture. ![sealevel-vm](sealevel-vm.png) Of course, it is all about trade-offs. There’s no silver bullet. This accounting approach adds overhead for transaction building, validating, and executions. Also, the overall complexity of data management is increasing exponentially. | | *Solana* | | --------------- | ------------- | | Parallelization | Deterministic | | State Access | AccountsDB | | Virtual Machine | SVM Multi-threaded | | Max Recorded TPS| 7229 tx/s | | Max Theoretical TSP| 65000 tx/s | ------------------------------------ ### SUI Instead of using this replicated state machine, SUI thinks slightly differently about state management. For SUI, everything related to data is an ownable [object](https://docs.sui.io/concepts/object-model). If you want to mutate data, you can simply change its ownership. Even smart contracts are objects that can manipulate other objects. ![sui-accounts](sui-accounts.png) [Transactions](https://docs.sui.io/concepts/transactions) on SUI take some objects as input and produce other mutated or fresh objects. Each object knows the last transaction that made them, forming a [DAG](https://docs.sui.io/concepts/object-model#the-transaction-object-dag-relating-objects-and-transactions) (direct acyclic graph) where each transaction is a node and directed edges are between transactions if the output object of the source transaction is the input object of the destination transaction. ![sui-transactions](sui-transactions.png) We can run multiple transactions in [parallel](https://blog.sui.io/parallelization-explained/) without conflicts because there is already a clear separation between states that need to be updated, and each object is independent of the others. | | *Sui* | | --------------- | ------------- | | Parallelization | Deterministic | | State Access | RocksDB | | Virtual Machine | Move Multi-threaded | | Max Recorded TPS| 800 tx/s | | Max Theoretical TSP| 297000 tx/s | ------------------------------------ ## Optimistic Execution Optimistic parallelization is a straightforward and naive approach to the problem: try to execute multiple transactions simultaneously. Pick a winner and re-try the rest if some touch the same memory zone. Repeat until all transactions are processed. ![occ](occ.png) One approach for this optimistic parallelization is an [Optimistic Concurrency Control](https://en.wikipedia.org/wiki/Optimistic_concurrency_control) mechanism, similar to how databases handle transactions. You can implement a transaction statement for each blockchain transaction, execute all state transitions, and validate that another transaction did not alter the data used inside this transaction before committing to the new state. ### SeiV2 SeiV2 is [implementing](https://blog.sei.io/sei-v2-the-first-parallelized-evm/) this OCC mechanism for its EVM transactions. This mechanism draws inspiration from databases, where transactions are processed optimistically, assuming that most will not conflict. In this context, transactions are executed simultaneously, and conflicts are resolved after execution, ensuring that no two transactions modify the same state simultaneously. ![sei_v2](sei_v2.png) How OCC Works in SeiV2: * **Parallel Execution**: Multiple transactions are executed in parallel without initially checking whether they conflict. This "optimistic" approach assumes that most transactions will not overlap in the state they modify. * **Conflict Detection**: After the transactions are executed, SeiV2 checks for conflicts. If two transactions have modified the same state (i.e., touched the same memory zone), one is chosen to be committed, and the others are retried. * **Transaction Rollback**: If a conflict is detected, conflicting transactions are rolled back and re-executed until all transactions are successfully processed without conflict. This can lead to retries but allows for increased parallel throughput. * **State Validation Before Commit**: SeiV2 validates that no other transaction has altered the data involved in a given transaction before committing the state changes to the blockchain. If the data has changed, the transaction is retried. | | *SeiV2* | | --------------- |---------------------| | Parallelization | Optimistic | | State Access | SeiDB | | Virtual Machine | EVM Single-threaded | | Max Recorded TPS| 256 tx/s | | Max Theoretical TSP| 12500 tx/s | ------------------------------------ ### Monad Monad is another example of L1 that implements optimistic techniques. Their approach is very similar to SeiV2's but not the exact implementation. Besides the optimistic concurrency control and their re-trying mechanism, Monad also has a larger view of how concurrency should behave. Before block execution, transactions are [statically analyzed](https://docs.monad.xyz/technical-discussion/execution/parallel-execution#scheduling), and feature dependencies can be detected. ![monad_vm](monad_vm.png) Besides those mechanisms, Monad adds more optimizations to the [execution](https://docs.monad.xyz/technical-discussion/execution/parallel-execution), consensus, and [storage](https://docs.monad.xyz/technical-discussion/execution/monaddb) layers. Once you examine how the Ethereum block time is split, you see that much time is spent on [consensus](https://docs.monad.xyz/technical-discussion/consensus/deferred-execution). From a total of 12s block time, we have 120ms on execution. The entire consensus and execution process runs synchronously, which can be optimized. ![eth_block_time](eth_block_time.png) After the consensus phase is finished, you can immediately start with the next consensus phase while, in parallel, running the execution phase. In this way, you can save those wasted 120ms. Now, the trick doesn’t consist of those saved 120ms, but it consists of extending the execution block to match the consensus block, thus allowing more transactions to get executed in the same block time. ![monad_execution](monad_execution.png) Monad adds excellent [pipelining](https://docs.monad.xyz/technical-discussion/concepts/pipelining) features like Solana, allowing for a more streamlined execution. You can use different hardware to prepare the execution part. Instead of waiting for all parts of the execution (FETCH, DECODE, EXECUTE, WRITE) to finish, you can use different cores and hardware to execute them in parallel. ![monad_pipeline](monad_pipeline.png) | | *Monad* | | --------------- |---------------------| | Parallelization | Optimistic | | State Access | MonadDB | | Virtual Machine | EVM Single-threaded | | Max Recorded TPS| 0 tx/s | | Max Theoretical TSP| 10000 tx/s | ------------------------------------ ## Block-STM One of the most significant breakthroughs in optimistic parallelization was the introduction of [block-stm](https://arxiv.org/abs/2203.06871). Block-STM takes its roots in [Software Transactional Memory](https://groups.csail.mit.edu/tds/papers/Shavit/ShavitTouitou-podc95.pdf) (STM), a concurrency technique that makes parallel execution look like sequential execution for developers. You write code that executes in parallel without orchestrating anything. No locks, no semaphores. It adds execution overhead since you need to track all memory access, usually using a multi-version data structure in which you keep all memory’s different states across execution. This allows for fast validation and easier conflict detection. ![stm](stm.png) ### Aptos Aptos is one of the [pioneers](https://x.com/Aptos/status/1839379657122345248) who implemented those STM mechanisms at the execution VM layer. Aptos can build a more efficient re-execution queue using the [multi-version](https://x.com/Aptos/status/1846310188120424899) data structure and OCC since, if you know that one transaction is written in the exact memory location as before, you can stop it and dynamically mark it for retry. They also introduced optimization for scheduling validation and execution tasks on multiple threads using a collaborative scheduler. ![aptos-execution](aptos-execution.png) Aptos’ multi-version data structure stores multiple versions of values written by different transactions. When a transaction reads data, it retrieves the highest version written by any preceding transaction in the block. This ensures that reads and writes do not interfere with each other across concurrent transactions. Suppose a transaction needs to be re-executed due to validation failure. In that case, the previous version can be used as an estimate, optimizing dependency tracking and minimizing unnecessary aborts and re-execution. ![multi-version](multi-version.png) | | *Aptos* | | --------------- |---------------------| | Parallelization | Optimistic | | State Access | RocksDB | | Virtual Machine | Move Multi-threaded | | Max Recorded TPS| 10734 tx/s | | Max Theoretical TSP| 160000 tx/s | ------------------------------------ ## Trade-offs After exploring the concepts of parallel blockchains and the different approaches to parallelization, it's clear that there are trade-offs and considerations to be made in implementing efficient and safe parallel execution of transactions. First, deterministic execution, as seen in Solana, requires careful management of data overhead and complex data structures, but it enables running multiple transactions in parallel by grouping those that do not use the same accounts. On the other hand, SUI's approach to deterministic parallelization emphasizes the clear separation between states that need to be updated, allowing multiple transactions to run in parallel without conflicts. Optimistic parallelization presents a more straightforward approach, but it requires mechanisms like Optimistic Concurrency Control to handle potential conflicts and ensure data consistency. SeiV2 is implementing such a mechanism for its EVM transactions, drawing inspiration from how databases handle transactions. Each approach has its trade-offs and implications for data management, transaction validation, and overall complexity. As parallel blockchains continue to evolve, striking a balance between scalability, data integrity, and execution efficiency will remain a crucial consideration. | | *Solana* | *Sui* | *SeiV2* | *Monad* | *Aptos* | | --------------- | ------------- |---------------|---------------------|---------------------|---------------------| | Parallelization | Deterministic | Deterministic | Optimistic | Optimistic | Optimistic | | State Access | AccountsDB | RocksDB | SeiDB | MonadDB | RocksDB | | Virtual Machine | SVM Multi-threaded | Move Multi-threaded | EVM Single-threaded | EVM Single-threaded | Move Multi-threaded | | Max Recorded TPS| 7229 tx/s | 800 tx/s | 256 tx/s | 0 tx/s | 10734 tx/s | | Max Theoretical TSP| 65000 tx/s | 297000 tx/s | 12500 tx/s | 10000 tx/s | 160000 tx/s | --- ### Healthy Python Codebase - **URL**: https://blog.vtemian.com/post/healthy-python-codebase/ - **Published**: September 21, 2020 - **Word Count**: 1521 words - **Tags**: python, guide, refactoring - **Summary**: Guidelines for maintaining a healthy Python codebase: consistency, explicit code, fail-fast patterns, testing strategies, and practical refactoring examples The code is a living entity. For the majority of the time, it stays in silence and it's doing its job, without complaining. But, there are these creatures, humanoid, that from time to time, will change it. Will try to fix it, adapt, or completely remove parts of it. Those creatures tend to conserve energy and they are using pattern recognition to do so. They observe patterns and take fast and easy decisions based on those patterns. Because of that, this code entity needs to be structured and behave in certain patterns. ## Consistency Those patterns are grouped together in a collection called code style. Any change that doesn't follow the code style, introduces inconsistency, and should be treated as a systemic disease. Maybe is a little harsh, but any code change should avoid inconsistency. Even if the current code base has a "special" code style, even if some bad decisions were made, introducing inconsistency is even worst. You'll have to maintain two or more styles, multiply the decisions and patterns, thus adding extra effort. So, when reviewing any code, one should take into account the current code style. Following the current patterns and maintain consistency is more valuable in the long run. If the current code base is not consistent, I suggest solving this issue first. It should be an easy refactoring exercise and shouldn't impact the business logic. Use formatting and linting tools and setup a CI pipeline that will run on every code change. You can even run them as a pre-commit hook. Black for formatting and pylint / mypy for linting are the most popular tools. From a behavioral point of view, make sure that the entities are doing what their name implies: fetch_users should return a list of users or an empty list. It shouldn't compute anything or return false, true, int, or string. Be consistent with names and obey components to respect their contract. If a component violets its contract, fail fast and visible. Don't be afraid to throw exceptions or stop the business logic. Failing fast and visible will help you catch bugs and solve inconsistency issues. ## Explicit is better than implicit Python is magic and allows you to abuse it. Besides that, when designing certain components, make sure that those are well defined and their behavior is properly exposed and expected by their clients. Avoid using generic components like Manager, Service, Data since they have a tendency of hiding complexity. Use specific and meaningful terminology for components that are responsible for doing just one thing. Favor small and well-defined logical units over complex ones (simple is better than complex). Avoid implicit defaults and don't be afraid of exposing some implementation details: ```python def compute_interval(start): return time.now() - start def compute_interval(start, end=None): if end is None: end = time.now() return end - start ``` Leverage Python's functional tooling over in-place processing: ```python def filter(items): new_items = [] for item in items: if item.get("property"): new_items.append(item) new_items.sort() # faster, in-place return new_items def filter(items, filters=None): if not filters: filters = [lambda item: item.get("property")] items = [item for item in items if all([filter(item) for filter in filters])] return sorted(items) # slower, creates a new list ``` Keep the levels of indirection small, avoid abusing metaclasses and complex OOP design: ```python # abstractisation just for the sake of it adds more complexity than solves real issues class RequestValidator: def __init__(self, request, validators=None): self.request = request self.validators = valiadtors or [ lambda request: "foo" in request.POST.get("arg") ] def validate(self): return all([validator(self.request) for validator in self.validators]) if RequestValidator(request).validate(): process() # simple and concise, solves the issue is easy to read and to maintain if "foo" in request.POST.get("arg", ""): process() ``` Avoid using classes as logical namespaces: ```python class Refresh: @classmethod def create_token(cls): ... @classmethod def invalidate_token(cls, token): ... @classmethod def refresh_token(cls, token): ... ``` ## Fail fast and visible Breaking the production can be terrifying and sometimes, we tend to "hide" or "swallow" user-facing errors just to avoid showing weakness and to give our user a sense of shaky/unfinished product. Having bugs and errors in your codebase is a natural thing. That's how software grows and evolves. The catch is to see those bugs and errors as soon as possible and to fix them. The scariest bugs always starts with This thing just doesn't work. It says nothing. Avoid catching all exceptions. Instead, try to handle as many exception paths as possible. ```python try: content = requests.get() except Exception: pass try: content = requests.get() except ConnectTimeout: ... except RequestException: ... except RetryError: ... ``` Failing fast and visible allows you to identify the problem and fix it. But you'll need visibility over those exceptions. Just failing fast may not be enough. An exception without a traceback may not be so useful to find the root cause. [Sentry](https://sentry.io/) is the way to go. Other logging and monitoring tools will help as much: [Datadog](https://www.datadoghq.com/), [Google Operations](https://cloud.google.com/products/operations), or [NewRelic](https://newrelic.com/) are just a bunch of observability tools that will make your life easier. ## End-to-End > Integration > Unit tests Testing is your safety net. It allows you a free state of mind in which changes can happen easily, without regression. In theory, at least. Most of the time, we trust too much this safety net. A close to 100% code coverage doesn't really mean that your changes are not breaking the current behavior. Some code paths can be purely understood and tested. Multiple testing methodologies will create different safety nets. A really simple and fast to obtain is one using unit tests. Testing small and well-contained pieces of logic is easy and fast. The downside comes when those pieces are interacting with other parts and the interaction may be broke. Some assertions about those interactions can be tested using mocks, but those are our interpretations and assertion about the behavior and not actual behavior. Avoid trusting only unit tests and mock only if necessary. Mock data over behavior. Instead of mocking behavior, use a production-ready setup and try to test multiple components together. This kind of testing can be named integration tests. Use dependency injection over global services. The setup may cost you more than the setup for unit tests, but in the end, it's closer to production behavior. Usually, a small integration test is more powerful than multiple unit tests. Even in integration, tested behavior is limited to the components tested. A wider testing methodology would be to test the entire stack, a request lifecycle from client to server and back. This will cost you the most, but having just one simple e2e (end-to-end) test can pay the bill big time. This type of test usually involves spawning a production-like environment, close to production data. Favor End-to-End tests over Integration tests. Favor Integration tests over Unit tests. Mock data over behavior. ## Deployable Writing code is easy. But those written symbols are usually meant to run on different machines. They interact with other symbols, alone, without your help. Even if may not be a big concern, deploying software is a big part of it. When writing code, starting with the mindset that it should get deploy will help you with design decisions. Always start with deploying in mind. How should data migration be handled? Does it involve downtime? Should I announce my users? How to deprecate API versions? Beside actually releasing software, thinking about packaging and portability is also important. Dependencies should be locked and build reproducible. A clear and consistent pipeline helps users use your software and allows developers to contribute, with minimum pain. Don't be afraid to use helper scripts, Dockerfiles, Makefiles, or other tools to automate your job. Software is always changing, so allow for this pipeline/procedure to be changing as well. ## Refactoring Example Those were some small guidelines from my experience. In the end, a healthy codebase is a codebase that is easy to change. Making a codebase healthy is not a one-time thing. Is a long and tedious process, with small incremental progress. Small deployable changes are more sustainable, less invasive, and easier to understand. ```python import requests import os from django.conf import settings import time def a(b): s = time.now() try: c = requests.get(b) except: return if time.now() - s > settings.TIME: return "it took too much" else: return c.content, c.status_code ``` ```python # Follow PEP-8 guidelines and sort imports. import os import time import requests from django.conf import settings from requests.exceptions import RequestException def validate_url(url: str) -> bool: if not url or not isinstance(url, str): return False rules = [ lambda url: url.startswith("https://"), lambda url: settings.VALID_HOST in url, ... ] return not all([rule(url) for rule in rules]) # Naming should be meaningful and consistent. def try_fetch(url: str) -> str: # Type-hinting allows tools like mypy to detect bugs. # Fail fast and visible. if not validate_url(url): raise ValueError(f"Invalid or missing url: {url}") start = time.now() # Avoid swallowing exceptions. If not possible, log them. response = response.get(url, raise_for_status=True) # Group conditions and their context request_duration = time.now() - start if request_duration > settings.MAX_REQUEST_DURATION: # Add more context to errors. raise RuntimeError(f"Request duration excided maximum duration: {request_duration}.") # Check invalid usecases. if response.status_code != 200: raise RuntimeError(f"Service responded with {response.status_code} instead of 200.") return str(response.content) # Be consistent with returning type ``` Cheers 🍺! --- ### Building a serverless hosting platform - **URL**: https://blog.vtemian.com/post/serverless-hosting-platform/ - **Published**: May 18, 2020 - **Word Count**: 5705 words - **Tags**: knative, kubernetes, serverless - **Summary**: Building a serverless hosting platform using Knative, Tekton, and Kubernetes on bare metal - from cluster setup to automated CI/CD deployments Deploying a 3-tier application (with the presentation layer, business logic, and storage) can get a little tricky these days. Let's say that we have a simple Django application, [poll's app](https://github.com/vtemian/simple-django-app) from the tutorial. It runs perfect on our local machine, we added a requirements.txt to hold our dependencies. As for the database, we can use SQLite, since we're developing only locally. The purpose of this project is to build a system that will allow us to push on a branch and deploy our changes in a separate environment, giving us a unique URL, to check them. Similarly to how [now.sh](http://now.sh) and [heroku.com](http://heorku.com) are doing. We'll need a mechanism that will package our code and dependencies and will deploy it, but also it needs to consider multiple versions, upgrades, load-balacing, scaling and our stateful part (database). [Introduction]({{< ref "#introduction" >}}) [Serving component]({{< ref "#serving-component" >}}) * [Packet]({{< ref "#packet" >}}) * [Kubernetes on bare-metal]({{< ref "#kubernetes-on-bare-metal" >}}) * [MetalLB]({{< ref "#metallb" >}}) * [Istio]({{< ref "#istio" >}}) * [First Knative service]({{< ref "#first-knative-service" >}}) * [ElasticSearch and Kibana]({{< ref "#elasticsearch-and-kibana" >}}) * [Autoscaling]({{< ref "#autoscaling" >}}) * [Mysql]({{< ref "#mysql" >}}) [CI/CD]({{< ref "#ci-cd" >}}) * [Tekton]({{< ref "#tekton" >}}) * [How does Tekton work?]({{< ref "#how-does-tekton-work" >}}) * [Pipeline Setup]({{< ref "#pipeline-setup" >}}) * [Github webhook trigger]({{< ref "#github-webhook-trigger" >}}) [Conclusions]({{< ref "#conclusions" >}}) ## Introduction In order to achieve that, we'll need two main components: one component that will take our code and prepare it to be published, namely the CI/CD component, and another one that will expose the changes to the Internet, namely the serving component. We can add a third component to hold some state for our application, like database and storage, but we'll add it to the serving component. ![/](/Untitled.png) ## Serving component For the serving component, we can use [Knative](https://knative.dev/). It leverages Kubernetes and integrates components that are already built on top of Kubernetes. At it's very basic, it runs and exposes a Docker image to the Internet, without any fuss. You'll just have to define a `service` that describe your image and its environment and Knative will take care of everything else (from routing, logging, monitoring to managing different versions of your application and autoscaling, including 0 scaling for no use). As you can imagine, [Knative](https://knative.dev/) is way more complex than it can be described in a paragraph and currently, we'll not dissect it. ### Packet In order to move forward with Knative, we'll need a Kubernetes cluster. For the sake of over-engineering it and trying something new, let's try to install Kubernetes on bare-metal. It sounds a little overwhelming, but in the end, it is way simpler than anticipated. I've always wanted to try [packet.com](https://www.packet.com/), since they have automated their deployment (it can be controlled via an API, thus allowing tools like Terraform to shine), they have a marketplace on which you can bid for machine's usage per hour (called [Spot Market](https://www.packet.com/developers/docs/getting-started/deployment-options/spot-market/), accessible via their API) and neat networking features (like [BGP](https://www.packet.com/developers/docs/network/advanced/local-and-global-bgp/) - Border Gateway Protocol, which will need further). We can choose from 3 deployment types: on-demand, reserved and spot. Let's try the spot instances since those can be really cheap. ![/Screenshot_2020-05-09_at_12.19.28.png](/Screenshot_2020-05-09_at_12.19.28.png) Once a spot market request was created, it will check for available machines that comply with your bid, and start provisioning them. For a max bid of $0.10 / h, we get a [c1.small.x86](https://www.packet.com/cloud/servers/c1-small/) instance, with 4 physical cores running at 3.4Ghz ([Intel E3-1240 v3](https://ark.intel.com/content/www/us/en/ark/products/75055/intel-xeon-processor-e3-1240-v3-8m-cache-3-40-ghz.html)), 32GB RAM, 2 x 120GB SSD and 2 Gigabit NICs. ![/Screenshot_2020-05-09_at_12.33.00.png](/Screenshot_2020-05-09_at_12.33.00.png) I've updated the hostname for each of one and now we're ready to install Kubernetes. ![/Screenshot_2020-05-09_at_13.05.54.png](/Screenshot_2020-05-09_at_13.05.54.png) ### Kubernetes on bare-metal There are tons of guides out there on how to install Kubernetes on bare metal, from installing all the components manually to using scripts or other tools. The most popular choices are [kops](https://github.com/kubernetes/kops), [kubeadm](https://github.com/kubernetes/kubeadm) and [kubespray](https://github.com/kubernetes-sigs/kubespray). I went with kubespray since, for me, it was easier to understand and it was the path with the least resistance to follow since I have some ansible experience. [Here](https://github.com/kubernetes-sigs/kubespray/blob/master/docs/comparisons.md) you can find a small comparison between kops, kubeadm, and kubespray. Kubespray is easy to install and to use. We just need to clone the [repository](https://github.com/kubernetes-sigs/kubespray) and install it using ```bash sudo pip3 install -r requirements.txt ``` We can also install it in a separate virtual environment if we have different versions of ansible running on your machine. Next, we need to define an inventory of servers. It comes with pre-defined inventory examples. We can use Packet's API to list all your servers, but I decided to use a static one. Just copy the `sample` inventory into a separate one (I've called it `rabbit`). ```bash cd kubespray cp -R inventory/sample/ intentory/rabbit ``` Now add our servers in `inventory.ini` ``` [all] rabbit-1.vtemian.com ansible_host=147.75.84.27 ansible_user=root ip=10.80.204.129 etcd_member_name=etcd1 rabbit-2.vtemian.com ansible_host=147.75.100.161 ansible_user=root ip=10.80.204.131 etcd_member_name=etcd2 rabbit-3.vtemian.com ansible_host=147.75.100.215 ansible_user=root ip=10.80.204.133 etcd_member_name=etcd3 [kube-master] rabbit-1.vtemian.com [etcd] rabbit-1.vtemian.com [kube-node] rabbit-2.vtemian.com rabbit-3.vtemian.com [calico-rr] [k8s-cluster:children] kube-master kube-node calico-rr ``` Because when I was setting up my cluster, kubespray didn't fully supported Ubuntu 20.04, I had to update the tasks a little bit. I've replaced `python-minimal` with `python2-minimal` and install Docker from Ubuntu 19.10 (Eoan) repositories. Next, we just need to run ansible and let it do the magic. ```bash ansible-playbook --become -i inventory/rabbit/inventory.ini cluster.yml ``` If everything worked as intended, we'll have a new cluster, up and running. In order to access it, we can grab the admin credentials, from the kube-master node. ```bash scp root@rabbit-1.vtemian.com:/etc/kubernetes/admin.conf . ``` Next, add those into our local kubectl config (usually located at `~/.kube/config`) and we'll be able to access the cluster, using `kubectl`. ```bash ╰─>$ kubectl get pod --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system calico-kube-controllers-5679c8548f-rffvp 1/1 Running 0 2m46s 10.80.204.133 rabbit-3.vtemian.com kube-system calico-node-6wt2p 1/1 Running 1 3m12s 10.80.204.129 rabbit-1.vtemian.com kube-system calico-node-98cnq 1/1 Running 1 3m12s 10.80.204.131 rabbit-2.vtemian.com kube-system calico-node-kh9k8 1/1 Running 1 3m12s 10.80.204.133 rabbit-3.vtemian.com kube-system coredns-76798d84dd-75tz6 1/1 Running 0 2m21s 10.233.82.1 rabbit-1.vtemian.com kube-system coredns-76798d84dd-bqt66 1/1 Running 0 2m17s 10.233.80.1 rabbit-3.vtemian.com kube-system dns-autoscaler-85f898cd5c-nskgf 1/1 Running 0 2m18s 10.233.82.2 rabbit-1.vtemian.com kube-system kube-apiserver-rabbit-1.vtemian.com 1/1 Running 0 4m58s 10.80.204.129 rabbit-1.vtemian.com kube-system kube-controller-manager-rabbit-1.vtemian.com 1/1 Running 0 4m58s 10.80.204.129 rabbit-1.vtemian.com kube-system kube-proxy-4ktbs 1/1 Running 0 3m34s 10.80.204.131 rabbit-2.vtemian.com kube-system kube-proxy-kd6n2 1/1 Running 0 3m34s 10.80.204.133 rabbit-3.vtemian.com kube-system kube-proxy-ts8nw 1/1 Running 0 3m34s 10.80.204.129 rabbit-1.vtemian.com kube-system kube-scheduler-rabbit-1.vtemian.com 1/1 Running 0 4m58s 10.80.204.129 rabbit-1.vtemian.com kube-system kubernetes-dashboard-77475cf576-7sdr6 1/1 Running 0 2m15s 10.233.83.2 rabbit-2.vtemian.com kube-system kubernetes-metrics-scraper-747b4fd5cd-k96pn 1/1 Running 0 2m15s 10.233.83.1 rabbit-2.vtemian.com kube-system nginx-proxy-rabbit-2.vtemian.com 1/1 Running 0 3m35s 10.80.204.131 rabbit-2.vtemian.com kube-system nginx-proxy-rabbit-3.vtemian.com 1/1 Running 0 3m36s 10.80.204.133 rabbit-3.vtemian.com kube-system nodelocaldns-9l6vf 1/1 Running 0 2m17s 10.80.204.133 rabbit-3.vtemian.com kube-system nodelocaldns-blbcb 1/1 Running 0 2m17s 10.80.204.131 rabbit-2.vtemian.com kube-system nodelocaldns-vrspt 1/1 Running 0 2m17s 10.80.204.129 rabbit-1.vtemian.com ``` ### MetalLB Going further, we should be able to install Knative. A big step in Knative's installation is the routing component. It supports multiple networking layers (Ambassador, Contour, Gloo, Istio, and Kourier). The only problem is that those layers need a load balancer that will be exposed to the Internet (an external [LoadBalancer](https://kubernetes.io/docs/tasks/access-application-cluster/create-external-load-balancer/)). Kubernetes doesn't have native support for that. Basically, the current implementations are vendor-specific (AWS, GCP, Azure etc.) and because we're on bare-metal, we can't afford the luxury of using one of those. Luckily, there's an implementation for bare-metal, called [MetalLB](https://metallb.universe.tf/). It can do that in two ways: at [layer 2](https://metallb.universe.tf/concepts/layer2/) using ARP/NDP or by leveraging [BGP](https://metallb.universe.tf/concepts/bgp/). Because Packet has support for [BGP](https://www.packet.com/developers/docs/network/advanced/local-and-global-bgp/) and they also provide a useful example on how to configure [MetalLB](https://github.com/packet-labs/kubernetes-bgp), we'll give them a try. The instructions from Packet's BGP - Kubernetes [integration](https://github.com/packet-labs/kubernetes-bgp#calico) are well documented and easy to follow. We just need to be careful with the IPPools. Before defining them, I've configured 2 sets of elastic IPs: A global IP `147.75.40.130/32` and a Public IPv4 `147.75.80.160/30`. ![/Screenshot_2020-05-09_at_16.27.30.png](/Screenshot_2020-05-09_at_16.27.30.png) [For security reason](https://www.packet.com/developers/docs/network/basic/elastic-ips/)s, you'll need to manually configure the IPs, for each server. Its fairly easy to do it and well documented. For each server, attach them an IP from the `Network` section: ![/Screenshot_2020-05-09_at_16.36.03.png](/Screenshot_2020-05-09_at_16.36.03.png) And that, on each server manually (or via ansible), an example for Ubuntu/Debian, if you just want to play around with, run: ``` sudo ip addr add dev lo ``` To make it permanent, we'll need to edit `/etc/network/interfaces` ```bash auto lo:0 iface lo:0 inet static address netmask 255.255.255.255 ``` Continuing with the IPPools configuration, for `metallb-ewr1-public` will have `147.75.80.160/30`, for `metallb-global-ips` will have `147.75.40.130/32` and for `metallb-private` will have our private nodes subnet, which in the current case should be `10.80.204.128/29`. You can play around with the node's private ips and a CIDR-IP conversion [tool](https://www.ipaddressguide.com/cidr). For each calico peer config (worker), we'll put node's private IP. Next, we'll install the latest metalLB manifest: ``` kubectl apply -f https://raw.githubusercontent.com/google/metallb/v0.8.1/manifests/metallb.yaml ``` Followed by the metalLB's config map, in `metallb-system` namespace: ```yaml apiVersion: v1 kind: ConfigMap metadata: namespace: metallb-system name: config data: config: | peers: - peer-address: 127.0.0.1 peer-asn: 65000 my-asn: 65480 address-pools: - name: ewr1-public protocol: bgp addresses: - 147.75.80.160/30 - name: ewr1-private protocol: bgp addresses: - 10.80.204.128/29 - name: global-ip protocol: bgp addresses: - 147.75.40.130/32 ``` We can check if everything is configured correctly, by running `calicoctl node status` in our master node: ```bash root@rabbit-1:~# calicoctl node status Calico process is running. IPv4 BGP status +----------------+-------------------+-------+----------+-------------+ | PEER ADDRESS | PEER TYPE | STATE | SINCE | INFO | +----------------+-------------------+-------+----------+-------------+ | 147.75.100.215 | node-to-node mesh | up | 13:46:38 | Established | | 127.0.0.1 | global | up | 13:51:44 | Established | | 147.75.100.161 | node-to-node mesh | up | 13:47:27 | Established | +----------------+-------------------+-------+----------+-------------+ ``` And other kubectl commands: ```bash ╰─>$ kubectl get pod -n kube-system -o wide | grep calico-node calico-node-479fz 1/1 Running 0 8m25s 10.80.204.133 rabbit-3.vtemian.com calico-node-846gr 1/1 Running 0 7m18s 10.80.204.131 rabbit-2.vtemian.com calico-node-tpnjc 1/1 Running 0 8m8s 10.80.204.129 rabbit-1.vtemian.com ``` ```bash ╰─>$ kubectl get pod -n metallb-system -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES controller-6bcfdfd677-nxnw8 1/1 Running 0 5m4s 10.233.65.193 rabbit-3.vtemian.com speaker-d6kks 1/1 Running 0 5m4s 10.80.204.131 rabbit-2.vtemian.com speaker-kk85w 1/1 Running 0 5m4s 10.80.204.133 rabbit-3.vtemian.com speaker-p4lc7 1/1 Running 0 5m4s 10.80.204.129 rabbit-1.vtemian.com ``` ### Istio Now that we have the MetalLB up and running we can continue with the last routing component. Between all those networking components that Knative supports, I've chosen [Istio](https://istio.io/), because it is the only one compatible with the Knative operator (which will be mention further). We just need to follow the instructions from the main install [page](https://knative.dev/development/install/installing-istio/#downloading-istio-and-installing-crds) and if everything worked, we'll have a load balancer, with an external IP. ```bash ╰─>$ kubectl get service --all-namespaces NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE default kubernetes ClusterIP 10.233.0.1 443/TCP 101m istio-system istio-ingressgateway LoadBalancer 10.233.24.125 147.75.80.160 15020:30935/TCP,80:31380/TCP,443:31390/TCP,31400:31400/TCP,15029:31350/TCP,15030:31699/TCP,15031:32315/TCP,15032:31519/TCP,15443:32542/TCP 55s istio-system istio-pilot ClusterIP 10.233.48.55 15010/TCP,15011/TCP,8080/TCP,15014/TCP 55s kube-system coredns ClusterIP 10.233.0.3 53/UDP,53/TCP,9153/TCP 98m kube-system dashboard-metrics-scraper ClusterIP 10.233.61.223 8000/TCP 97m kube-system kubernetes-dashboard ClusterIP 10.233.16.174 443/TCP 97m ``` ### Knative We're ready to install Knative. I found that the easier path is to install the common operator that will further install all the components. I've tried installing each component manually, but it can get really tricky. For now, we need to install the operator in the `default` namespace, since it will look for a ConfigMap called `config-loggin` in the `default` namespace. ```bash ╰─>$ kubens default ╰─>$ kubectl apply -f https://github.com/knative-sandbox/operator/releases/download/v0.14.1/operator.yaml ``` Once the CRDs are installed and the operator's pods are running ```bash ╰─>$ kubectl get pods NAME READY STATUS RESTARTS AGE knative-eventing-operator-5847fcc5d5-d4cb4 1/1 Running 0 53s knative-serving-operator-587dcd9f85-zlx7v 1/1 Running 0 53s ``` We can create the `KnativeServing` and `KnativeEventing` resources: ```bash ╰─>$ cat <<-EOF | kubectl apply -f - apiVersion: operator.knative.dev/v1alpha1 kind: KnativeServing metadata: name: ks EOF ╰─>$ cat <<-EOF | kubectl apply -f - apiVersion: v1 kind: Namespace metadata: name: knative-eventing --- apiVersion: operator.knative.dev/v1alpha1 kind: KnativeEventing metadata: name: ke namespace: knative-eventing EOF ``` New pods and resources are being installed in the `default` and `knative-eventing` namespaces ```bash ╰─>$ kubectl get pods --all-namespaces -o wide NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES default activator-65fc4d666-7bwst 1/1 Running 0 39s 10.233.125.68 rabbit-2.vtemian.com default autoscaler-74b4bb97bd-ghj59 1/1 Running 0 38s 10.233.65.195 rabbit-3.vtemian.com default autoscaler-hpa-594f68d5c4-8qtg4 1/1 Running 0 30s 10.233.65.198 rabbit-3.vtemian.com default controller-6b6978c965-rqb2z 1/1 Running 0 37s 10.233.65.196 rabbit-3.vtemian.com default istio-webhook-856d84fbf9-wvpph 1/1 Running 0 26s 10.233.125.71 rabbit-2.vtemian.com default knative-eventing-operator-5847fcc5d5-d4cb4 1/1 Running 0 3m18s 10.233.125.67 rabbit-2.vtemian.com default knative-serving-operator-587dcd9f85-zlx7v 1/1 Running 0 3m18s 10.233.125.66 rabbit-2.vtemian.com default networking-istio-6845f7cf59-bsqc2 1/1 Running 0 26s 10.233.125.69 rabbit-2.vtemian.com default webhook-577576647-wrw56 1/1 Running 0 36s 10.233.65.197 rabbit-3.vtemian.com istio-system istio-ingressgateway-75694cd848-l6zfh 1/1 Running 0 64m 10.233.125.65 rabbit-2.vtemian.com istio-system istio-pilot-576d858689-zxv76 1/1 Running 0 64m 10.233.65.194 rabbit-3.vtemian.com knative-eventing broker-controller-854447b8d7-vdmdz 1/1 Running 0 18s 10.233.65.200 rabbit-3.vtemian.com knative-eventing broker-filter-b54b58854-w9jvw 1/1 Running 0 17s 10.233.125.72 rabbit-2.vtemian.com knative-eventing broker-ingress-75b6b8df8d-mlppj 1/1 Running 0 16s 10.233.65.201 rabbit-3.vtemian.com knative-eventing eventing-controller-694594fdd7-gj2br 1/1 Running 0 26s 10.233.125.70 rabbit-2.vtemian.com knative-eventing eventing-webhook-6c6b675b6f-t4ntx 1/1 Running 0 26s 10.233.65.199 rabbit-3.vtemian.com knative-eventing imc-controller-7bb9bd7c6d-q2tsz 1/1 Running 0 10s 10.233.125.73 rabbit-2.vtemian.com knative-eventing imc-dispatcher-6cc5c74c7f-kdj7v 1/1 Running 0 10s 10.233.125.74 rabbit-2.vtemian.com knative-eventing mt-broker-controller-75ddc75d57-rg6jd 1/1 Running 0 15s 10.233.65.202 rabbit-3.vtemian.com knative-eventing v0.14.0-upgrade-4sv89 0/1 Completed 0 9s 10.233.65.203 rabbit-3.vtemian.com ``` Before we actually test it, let's configure the DNS component. We'll want to have a unique URL generated each time a new deployment is created. Knative can do that using [xip.io](http://xip.io) and we just need to create a job (we'll need to install it in the `default` namespace): ```bash ╰─>$ kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/serving-default-domain.yaml ``` ### First Knative service Within our initial application, I've created a simple [Dockerfile](https://github.com/vtemian/simple-django-app/tree/docker): ```bash FROM python:3.7-slim WORKDIR /app COPY requirements.txt ./ RUN pip install -r requirements.txt COPY app ./ CMD exec gunicorn app.wsgi --bind :$PORT --workers 1 --threads 8 --timeout 0 ``` And published the image, publicly, under `vtemian/simple-django-app`. ```bash ╰─>$ docker push vtemian/simple-django-app The push refers to repository [docker.io/vtemian/simple-django-app] 7aa16540cfca: Pushed 2e02cc50aabc: Pushed 768f0318f857: Pushed 663045c38f65: Pushed 715414420313: Mounted from vtemian/helloworld-python dba4fa00b93a: Mounted from vtemian/helloworld-python 9f690547ed37: Mounted from vtemian/helloworld-python 6376837eded8: Mounted from vtemian/helloworld-python c2adabaecedb: Mounted from vtemian/helloworld-python latest: digest: sha256:78799d85949e31728c70ef3dbf3a492d932fc94c140cf1047d948c89141f55ab size: 2205 ``` To publish it on our Knative installation, we just need to define a service: ```yaml apiVersion: serving.knative.dev/v1 kind: Service metadata: name: simple-django-app namespace: default spec: template: spec: containers: - image: docker.io/vtemian/simple-django-app ``` Aaaaaand `kubectl get ksvc`: ```bash ╰─>$ kubectl get ksvc NAME URL LATESTCREATED LATESTREADY READY REASON simple-django-app http://simple-django-app.default.147.75.80.160.xip.io simple-django-app-hc2qv Unknown RevisionMissing ``` Going to the generated URL ![/Screenshot_2020-05-09_at_18.28.18.png](/Screenshot_2020-05-09_at_18.28.18.png) Now this...this is pretty damn cool! There's no database and we still need to build our containers, but it looks pretty damn cool! ### ElasticSearch and Kibana Before we move further to test it more, let's configure some observability tools, like ElasticSearch + Kibana for logs and Prometheus + Grafana for metrics. Let's start with the metrics component. We'll follow the [guide](https://knative.dev/development/serving/installing-logging-metrics-traces/) and we'll just need to edit the `config-observability` config map. It already provides us with an config example, we'll be using it. Just unindent the exemple, for now. Next, we'll need to create the `knative-monitoring` namespace, and apply the manifests: ```bash ╰─>$ kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/monitoring-metrics-prometheus.yaml ``` The pods should be up and running in the `knative-monitoring` namespace: ```bash ╰─>$ kubectl get pod -n knative-monitoring -o wide NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES grafana-c9c94bdff-5f77v 1/1 Running 0 2m3s 10.233.65.210 rabbit-3.vtemian.com kube-state-metrics-b6bcff8f4-tvp46 1/1 Running 0 2m7s 10.233.65.209 rabbit-3.vtemian.com node-exporter-9wkpn 2/2 Running 0 2m4s 10.80.204.131 rabbit-2.vtemian.com node-exporter-lfjss 2/2 Running 0 2m4s 10.80.204.129 rabbit-1.vtemian.com node-exporter-zjl7b 2/2 Running 0 2m4s 10.80.204.133 rabbit-3.vtemian.com prometheus-system-0 1/1 Running 0 2m1s 10.233.65.211 rabbit-3.vtemian.com prometheus-system-1 1/1 Running 0 2m1s 10.233.125.75 rabbit-2.vtemian.com ``` By default, Grafana comes with some really nice dashboards and with Prometheus configured as a data source. The only problem is that the Prometheus configured, is not the currently running service. We'll need to get all currently running services and check Prometheus service name, which in this case is `prometheus-system-discovery`. ```bash ╰─>$ kubectl -n knative-monitoring get service NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE kube-controller-manager ClusterIP None 10252/TCP 5m36s kube-state-metrics ClusterIP 10.233.56.244 8080/TCP,8081/TCP 5m41s node-exporter ClusterIP None 9100/TCP 5m38s prometheus-system-discovery ClusterIP None 9090/TCP 5m36s ``` We'll have to edit Grafana's config map and replace Prometheus' URL with `[http://prometheus-system-discovery.knative-monitoring.svc:9090](http://prometheus-system-discovery.knative-monitoring.svc:9090/)`. ```bash ╰─>$ kubectl -n knative-monitoring edit cm grafana-datasources apiVersion: v1 data: prometheus.yaml: | datasources: - name: prometheus type: prometheus access: proxy org_id: 1 url: http://prometheus-system-discovery.knative-monitoring.svc:9090 version: 1 editable: false ``` Delete the current running Grafana pod ```bash ╰─>$ kubectl delete po -n knative-monitoring --selector=app=grafana pod "grafana-c9c94bdff-rkvrg" deleted ``` Wait until a new pod is started and you can port-forward it ```bash ╰─>$ kubectl port-forward --namespace knative-monitoring \ (kubectl get pods --namespace knative-monitoring \ --selector=app=grafana --output=jsonpath="{.items..metadata.name}") \ 3000 Forwarding from 127.0.0.1:3000 -> 3000 ``` ![/Screenshot_2020-05-10_at_13.29.25.png](/Screenshot_2020-05-10_at_13.29.25.png) All of those default dashboards are interesting, but I found the most useful the `Knative Serving - Revision HTTP Requests`, that describes current running applications. ![/Screenshot_2020-05-10_at_15.47.39.png](/Screenshot_2020-05-10_at_15.47.39.png) And the `Kubernetes Capacity Planning` that gives an overview over the entire cluster. ![/Screenshot_2020-05-10_at_15.48.07.png](/Screenshot_2020-05-10_at_15.48.07.png) Moving to logs, we'll need to configure ElasticSearch and Kibana. We'll need to edit the `config-observability` ConfigMap and set the `logging.request-log-template` to ```bash ╰─>$ kubectl edit cm config-observability logging.request-log-template: '{"httpRequest": {"requestMethod": "{{.Request.Method}}", "requestUrl": "{{js .Request.RequestURI}}", "requestSize": "{{.Request.ContentLength}}", "status": {{.Response.Code}}, "responseSize": "{{.Response.Size}}", "userAgent": "{{js .Request.UserAgent}}", "remoteIp": "{{js .Request.RemoteAddr}}", "serverIp": "{{.Revision.PodIP}}", "referer": "{{js .Request.Referer}}", "latency": "{{.Response.Latency}}s", "protocol": "{{.Request.Proto}}"}, "traceId": "{{index .Request.Header "X-B3-Traceid"}}"}' ``` Apply the manifest ```bash ╰─>$ kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/monitoring-logs-elasticsearch.yaml ``` We'll set `[beta.kubernetes.io/fluentd-ds-ready="true"](http://beta.kubernetes.io/fluentd-ds-ready=%22true%22)` label for our nodes ```bash ╰─>$ kubectl label nodes --all beta.kubernetes.io/fluentd-ds-ready="true" node/rabbit-1.vtemian.com labeled node/rabbit-2.vtemian.com labeled node/rabbit-3.vtemian.com labeled ``` And check if the fluentd daemon set is running on our nodes ```bash ╰─>$ kubectl get daemonset fluentd-ds --namespace knative-monitoring NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE fluentd-ds 2 2 2 2 2 beta.kubernetes.io/fluentd-ds-ready=true 5m37s ``` In this point, on each node a Fluentd daemon is running, collecting logs and send them to ElasticSearch. Furthermore, we'll need to configure Kibana to access those logs. We'll start the local proxy ```bash ╰─>$kubectl proxy ``` And visit [Kibana UI](http://localhost:8001/api/v1/namespaces/knative-monitoring/services/kibana-logging/proxy/app/kibana). If the service doesn't start, you can create one with the following configuration ```yaml apiVersion: v1 kind: Service metadata: name: kibana-logging namespace: knative-monitoring labels: app: kibana-logging kubernetes.io/cluster-service: "true" kubernetes.io/name: "Kibana" spec: ports: - port: 5601 protocol: TCP targetPort: ui selector: app: kibana-logging ``` Create a new index and wait until is processed. ![/Screenshot_2020-05-10_at_16.27.12.png](/Screenshot_2020-05-10_at_16.27.12.png) Set it as the default index ![/Screenshot_2020-05-10_at_16.45.24.png](/Screenshot_2020-05-10_at_16.45.24.png) And the logs should flow ![/Screenshot_2020-05-10_at_16.59.57.png](/Screenshot_2020-05-10_at_16.59.57.png) ### Autoscaling Now that we can really see what is happening in the cluster, let's configure the autoscaling and 0 scaling. For that, we'll need to edit the `config-autoscaler` ConfigMap. All options are already described in the comments, and for testing purpose, this is the configuration I'm using: ```yaml activator-capacity: "100.0" container-concurrency-target-default: "100" container-concurrency-target-percentage: "70" enable-graceful-scaledown: "true" enable-scale-to-zero: "true" max-scale-down-rate: "2.0" max-scale-up-rate: "1000.0" panic-threshold-percentage: "20.0" panic-window-percentage: "5.0" pod-autoscaler-class: kpa.autoscaling.knative.dev requests-per-second-target-default: "20" scale-to-zero-grace-period: 30s stable-window: 60s target-burst-capacity: "10" tick-interval: 2s ``` All those options are explained in the [docs](https://knative.dev/docs/serving/configuring-autoscaling/), but maybe what we're most interested are the 0 scaling ```yaml # specifies the time an inactive revision is left running before it is scaled to zero (min: 6s). scale-to-zero-grace-period: 30s # enables scale to zero enable-scale-to-zero: "true" ``` And the autoscaling trasholds ```yaml # defines how many concurrent requests are wanted at a given time (soft limit) and is the recommended configuration for autoscaling. container-concurrency-target-default: "100" ``` Those are the configuration applied for each revision, but you can control independently, using annotations. Let's configure the Horizontal Pod Autoscaler to follow the CPU metric and scale if the current consumed CPU is 30% of the limit. ```yaml apiVersion: serving.knative.dev/v1 kind: Service metadata: name: simple-django-app namespace: default spec: template: metadata: annotations: autoscaling.knative.dev/metric: cpu autoscaling.knative.dev/target: "70" autoscaling.knative.dev/class: hpa.autoscaling.knative.dev spec: containers: - image: docker.io/vtemian/simple-django-app resources: requests: cpu: 100m ``` Let's start a curl in background ```bash ╰─>$ watch -n 0.1 curl -SI http://simple-django-app.default.147.75.80.160.xip.io/polls/ ``` And we have 2 running pods ```bash ╰─>$ kubectl get po -l serving.knative.dev/service=simple-django-app NAME READY STATUS RESTARTS AGE simple-django-app-g9zf5-deployment-5b76fdf7fc-mtlwt 2/2 Running 0 3m25s simple-django-app-mg96q-deployment-7db5bb6b9c-29ffw 2/2 Running 0 4m18s ``` Let's go further and start a [Locust](https://locust.io/) test. We'll follow the instructions from [zalando-incubator](https://github.com/zalando-incubator/docker-locust) and start for replicas that will hit our service: ```bash _________________________________________________________________________________ L O C A L - D E P L O Y M E N T _________________________________________________________________________________ Target url: http://simple-django-app.default.147.75.80.160.xip.io/polls Where load test script is stored (e.g. https://raw.githubusercontent.com/zalando-incubator/docker-locust/master/example/simple.py): https://raw.githubusercontent.com/zalando-incubator/docker-locust/master/example/simple.py Number of slave(s): 4 Run type [automatic/manual]: manual ---------------------------------------------- VARIABLES ---------------------------------------------- TARGET_URL: http://simple-django-app.default.147.75.80.160.xip.io/polls LOCUST_FILE: https://raw.githubusercontent.com/zalando-incubator/docker-locust/master/example/simple.py SLAVES NUMBER: 4 RUN_TYPE: manual || automatic=false NUMBER OF USERS: HATCH_RATE: DURATION [in seconds]: COMPOSE: false SEND_ANONYMOUS_USAGE_INFO: true ---------------------------------------------- ``` And the results are pretty cool ```bash ╰─>$ kubectl get po -l serving.knative.dev/service=simple-django-app NAME READY STATUS RESTARTS AGE simple-django-app-ns6fm-deployment-85cff985d5-249rj 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-2c6m9 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-2m6kk 2/2 Running 0 86s simple-django-app-ns6fm-deployment-85cff985d5-2mm7t 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-2q7f8 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-5xcxf 2/2 Running 0 71s simple-django-app-ns6fm-deployment-85cff985d5-6jxfw 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-77v6w 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-8qk5s 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-9n4h6 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-b466k 2/2 Running 0 7m57s simple-django-app-ns6fm-deployment-85cff985d5-b8qbf 2/2 Running 0 25s simple-django-app-ns6fm-deployment-85cff985d5-bkt66 2/2 Running 0 71s simple-django-app-ns6fm-deployment-85cff985d5-bxbzf 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-d5xt5 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-jrchv 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-mtrvh 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-mzz7g 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-p7wvx 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-pbmzb 2/2 Running 0 25s simple-django-app-ns6fm-deployment-85cff985d5-pzb92 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-pzkrr 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-qhjxq 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-rc2xx 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-s7lzm 2/2 Running 0 25s simple-django-app-ns6fm-deployment-85cff985d5-sdpmf 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-ss66c 2/2 Running 0 6m27s simple-django-app-ns6fm-deployment-85cff985d5-ssrzg 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-t424m 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-tjlsz 2/2 Running 0 71s simple-django-app-ns6fm-deployment-85cff985d5-tzcjw 2/2 Running 0 56s simple-django-app-ns6fm-deployment-85cff985d5-w2tsp 2/2 Running 0 71s simple-django-app-ns6fm-deployment-85cff985d5-x9626 2/2 Running 0 41s simple-django-app-ns6fm-deployment-85cff985d5-xm5pk 2/2 Running 0 86s simple-django-app-ns6fm-deployment-85cff985d5-xv9sw 2/2 Running 0 56s ``` Requests leaving the local machine ![/Screenshot_2020-05-11_at_19.23.16.png](/Screenshot_2020-05-11_at_19.23.16.png) Requests for this current revision ![/Screenshot_2020-05-11_at_19.28.30.png](/Screenshot_2020-05-11_at_19.28.30.png) Resource consumption ![/Screenshot_2020-05-11_at_19.25.23.png](/Screenshot_2020-05-11_at_19.25.23.png) For now, we have a running Kubernetes cluster, on bare-metal (on top of Packet), with 3 nodes, a running Knative installation that serves and scales Docker images. ### Mysql Finally, let's add some state to this setup. At [Presslabs](https://www.presslabs.com/), the company I'm currently working for, we've built an operator for M[y](https://github.com/presslabs/mysql-operator)SQL. It takes care of replication, backups, and other tedious operations. The installation and its configuration are fairly straight forward, but first, we need to configure some persistent volumes: ```yaml apiVersion: v1 kind: PersistentVolume metadata: labels: type: local name: rabbit-1.vtemian.com spec: accessModes: - ReadWriteOnce capacity: storage: 11Gi hostPath: path: /mnt/data type: "" nodeAffinity: required: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/hostname operator: In values: - rabbit-1.vtemian.com persistentVolumeReclaimPolicy: Retain storageClassName: standard volumeMode: Filesystem ``` Let's create one for each node: ```bash ╰─>$ kubectl get pv NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE rabbit-1.vtemian.com 11Gi RWO Retain Available standard 2m58s rabbit-2.vtemian.com 11Gi RWO Retain Bound default/data-mysql-operator-0 standard 3m9s rabbit-3.vtemian.com 11Gi RWO Retain Available standard 3m19s ``` We now can continue with mysql-operator: ```bash ╰─>$ helm repo add presslabs https://presslabs.github.io/charts ╰─>$ helm install presslabs/mysql-operator --name mysql-operator --set orchestrator.persistence.storageClass=standard ``` Furthermore, we'll need a secret with the credentials we want for our mysql cluster ```yaml apiVersion: v1 kind: Secret metadata: name: my-secret type: Opaque data: ROOT_PASSWORD: bXlwYXNz DATABASE: cmFiYml0Cg== USER: cmFiYml0Cg== PASSWORD: bXlwYXNz ``` And create the cluster with 2 replicas ```yaml apiVersion: mysql.presslabs.org/v1alpha1 kind: MysqlCluster metadata: name: my-cluster spec: replicas: 2 secretName: my-secret ``` Now we have our 2 replicas: ```bash ╰─>$ kubectl get po -l app.kubernetes.io/name=mysql NAME READY STATUS RESTARTS AGE my-cluster-mysql-0 4/4 Running 0 3m11s my-cluster-mysql-1 4/4 Running 0 4m37s ``` And a service on which we can connect: ```bash ╰─>$ kubectl get service -l app.kubernetes.io/name=mysql NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE my-cluster-mysql ClusterIP 10.233.50.17 3306/TCP 10m my-cluster-mysql-master ClusterIP 10.233.29.255 3306/TCP 10m ``` At this point, the serving component is up and running and tested with a dummy application. Let's move further with the building component. ## CI/CD ### Tekton Knative used to have a build [component](https://github.com/knative/build/), which now is [deprecated](https://github.com/knative/build/issues/614) in favour of [Tekton](https://tekton.dev/). There are some nice guides on how to configure Tekton and integrate it with Knative, but first, let's install it: ```bash ╰─>$ kubectl apply --filename https://storage.googleapis.com/tekton-releases/pipeline/latest/release.yaml ``` Finally, we just need to edit the `config-artifact-pvc` ConfigMap, in order to allow Tekton to save artifacts in a PVC. ```yaml data: size: 5Gi storageClassName: default ``` Taking a look at Tekton pod's we can see that it's running properly: ```bash ╰─>$ kubectl get po -n tekton-pipelines NAME READY STATUS RESTARTS AGE tekton-pipelines-controller-5c44bcfc44-gfhdx 1/1 Running 0 85m tekton-pipelines-webhook-7bd568f6c6-vll6v 1/1 Running 0 85m ``` ### How does Tekton work? Before setting up the pipeline, let's explore Tekton a little bit. Tekton leverages CRDs and allow us to describe pipelines by defining Kubernetes resources. I'll resume the information from [this guide](https://www.alibabacloud.com/blog/first-knative-attempt-a-quick-guide-to-continuous-integration-and-continuous-delivery_595803) and their official [docs](https://tekton.dev/docs/pipelines/). [Tasks](https://tekton.dev/docs/pipelines/tasks/) are a template for defining an actual working unit. It's like defining a function, with its parameters and behavior. It defines one or more steps and at each step, a container is executed. Example from [https://github.com/knative-sample/tekton-knative](https://github.com/knative-sample/tekton-knativehttps://github.com/knative-sample/tekton-knative/blob/master/tekton-cicd/tasks/deploy-using-kubectl.yaml) ```yaml apiVersion: tekton.dev/v1alpha1 kind: Task metadata: name: deploy-using-kubectl spec: inputs: resources: - name: git-source type: git params: - name: pathToYamlFile description: The path to the yaml file to deploy within the git source - name: imageUrl description: Url of image repository - name: imageTag description: Tag of the images to be used. default: "latest" steps: - name: update-yaml image: alpine command: ["sed"] args: - "-i" - "-e" - "s;__IMAGE__;${inputs.params.imageUrl}:${inputs.params.imageTag};g" - "/workspace/git-source/${inputs.params.pathToYamlFile}" - name: run-kubectl image: registry.cn-hangzhou.aliyuncs.com/knative-sample/kubectl:v0.5.0 command: ["kubectl"] args: - "apply" - "-f" - "/workspace/git-source/${inputs.params.pathToYamlFile}" ``` A [TaskRun](https://tekton.dev/docs/pipelines/taskruns/) is a running instance of a Task. It executes all the steps of a task, in order, until all of them are completed. Example from [https://github.com/knative-sample/tekton-knative](https://github.com/knative-sample/tekton-knativehttps://github.com/knative-sample/tekton-knative/blob/master/tekton-cicd/tasks/deploy-using-kubectl.yaml) ```yaml apiVersion: tekton.dev/v1alpha1 kind: TaskRun metadata: name: source-to-image spec: taskRef: name: source-to-image params: - name: pathToContext value: "${params.pathToContext}" - name: imageUrl value: "${params.imageUrl}" - name: imageTag value: "${params.imageTag}" resources: inputs: - name: git-source resource: git-source ``` A [Pipeline](https://tekton.dev/docs/pipelines/pipelines/) allows us to define multiple tasks. Using TaskRun we could run only one task. Each Task in a Pipeline executes as a Pod. Example from [https://github.com/knative-sample/tekton-knative](https://github.com/knative-sample/tekton-knativehttps://github.com/knative-sample/tekton-knative/blob/master/tekton-cicd/tasks/deploy-using-kubectl.yaml) ```yaml apiVersion: tekton.dev/v1alpha1 kind: Pipeline metadata: name: build-and-deploy-pipeline spec: resources: - name: git-source type: git params: - name: pathToContext description: The path to the build context, used by Kaniko - within the workspace default: src - name: pathToYamlFile description: The path to the yaml file to deploy within the git source - name: imageUrl description: Url of image repository - name: imageTag description: Tag to apply to the built image tasks: - name: source-to-image taskRef: name: source-to-image params: - name: pathToContext value: "${params.pathToContext}" - name: imageUrl value: "${params.imageUrl}" - name: imageTag value: "${params.imageTag}" resources: inputs: - name: git-source resource: git-source - name: deploy-to-cluster taskRef: name: deploy-using-kubectl runAfter: - source-to-image params: - name: pathToYamlFile value: "${params.pathToYamlFile}" - name: imageUrl value: "${params.imageUrl}" - name: imageTag value: "${params.imageTag}" resources: inputs: - name: git-source resource: git-source ``` Similar to TaskRun, [PipelineRun](https://tekton.dev/docs/pipelines/pipelineruns/) executes all the tasks defined in a Pipeline. Example from [https://github.com/knative-sample/tekton-knative](https://github.com/knative-sample/tekton-knativehttps://github.com/knative-sample/tekton-knative/blob/master/tekton-cicd/tasks/deploy-using-kubectl.yaml) ```yaml apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: generateName: tekton-kn-sample- spec: pipelineRef: name: build-and-deploy-pipeline resources: - name: git-source resourceRef: name: tekton-knative-git params: - name: pathToContext value: "src" - name: pathToYamlFile value: "knative/helloworld-go.yaml" - name: imageUrl value: "registry.cn-hangzhou.aliyuncs.com/knative-sample/tekton-knative-helloworld" - name: imageTag value: "1.0" trigger: type: manual serviceAccount: pipeline-account ``` [PipelineResources](https://tekton.dev/docs/pipelines/resources/) allows us to define objects that are used by tasks' inputs and outputs. Example from [https://github.com/knative-sample/tekton-knative](https://github.com/knative-sample/tekton-knativehttps://github.com/knative-sample/tekton-knative/blob/master/tekton-cicd/tasks/deploy-using-kubectl.yaml) ```yaml apiVersion: tekton.dev/v1alpha1 kind: PipelineResource metadata: name: tekton-knative-git spec: type: git params: - name: revision value: master - name: url value: https://github.com/knative-sample/tekton-knative ``` ### Pipeline setup Those are all the major components that we'll play with. Let's create a new namespace called `ci` and install the above manifests, adapted for our needs. I've commited the changes in the example [app](https://github.com/vtemian/simple-django-app/tree/tekton). ```bash ╰─>$ kubectl get po NAME READY STATUS RESTARTS AGE tekton-simple-django-app-1-deploy-to-cluster-982xv-pod-kkmpw 0/3 Completed 0 3m18s tekton-simple-django-app-1-source-to-image-8c47t-pod-ccc44 0/3 Completed 0 3m44s ``` ```bash ╰─>$ kubectl get pipelinerun NAME SUCCEEDED REASON STARTTIME COMPLETIONTIME tekton-simple-django-app-1 True Succeeded 2m14s 95s ``` ### Github webhook trigger Right now, we manually have to trigger the build by deleting and re-creating the `Pipelinerun` resource. Let's try to automate it, by configuring a Github webhook that will ping the building process each time a new commit is made. The setup for that is not too complex, nor too simple. When a github hook arrives, it lands in an `[EventListener](https://tekton.dev/docs/triggers/eventlisteners/)` pod (that will need to be exposed to the Internet via Istio). From its payload, we'll need to extract relevant parameters, like commit information. For that, we'll be using [TriggerBindings](https://tekton.dev/docs/triggers/triggerbindings/). The parameters are then used by [TriggerTemplate](https://tekton.dev/docs/triggers/triggertemplates/) to generate our pipeline run. The following configurations are inspired by [@nikhilthomas1](https://medium.com/@nikhilthomas1/cloud-native-cicd-on-openshift-with-openshift-pipelines-tektoncd-pipelines-part-3-github-1db6dd8e8ca7). ![/Untitled%201.png](/Untitled%201.png) Let's create the a role, service account and the role binding for this process. ```yaml --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role rules: - apiGroups: - triggers.tekton.dev resources: - eventlisteners - triggerbindings - triggertemplates - pipelineresources verbs: - get - apiGroups: - triggers.tekton.dev resources: - pipelineruns - pipelineresources verbs: - create - apiGroups: - "" resources: - configmaps verbs: - get - list - create - update - delete --- apiVersion: v1 kind: ServiceAccount metadata: name: tekton-triggers-sa --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: tekton-triggers-rolebinding subjects: - kind: ServiceAccount name: tekton-triggers-sa roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: tekton-triggers-role⏎ ``` TriggerTemplate is very basic. It describes some parameters that can be used, from the binding and it patches them together with PipelineRun and other resources: ```yaml apiVersion: triggers.tekton.dev/v1alpha1 kind: TriggerTemplate metadata: name: tekton-triggertemplate spec: params: - name: gitrevision description: The git revision default: master - name: gitrepositoryurl description: The git repository url - name: namespace description: The namespace to create the resources - name: gitrepositoryname description: The name of the deployment to be created / patched resourcetemplates: - apiVersion: tekton.dev/v1alpha1 kind: PipelineResource metadata: name: source-repo-$(params.gitrepositoryname)-$(uid) namespace: $(params.namespace) spec: type: git params: - name: revision value: $(params.gitrevision) - name: url value: $(params.gitrepositoryurl) - apiVersion: tekton.dev/v1alpha1 kind: PipelineRun metadata: name: teokton-build-$(params.gitrepositoryname)-$(uid) namespace: $(params.namespace) spec: pipelineRef: name: build-and-deploy-pipeline serviceAccountName: pipeline-account resources: - name: git-source resourceRef: name: source-repo-$(params.gitrepositoryname)-$(uid) params: - name: pathToContext value: "" - name: pathToDockerFile value: Dockerfile - name: pathToYamlFile value: knative.yaml - name: imageUrl value: docker.io/vtemian/$(params.gitrepositoryname) - name: imageTag value: latest ``` Our TriggerBinding will also be pretty simple. Just a mapping from Github's payload to the parameters used in TriggerTemplate ```yaml apiVersion: triggers.tekton.dev/v1alpha1 kind: TriggerBinding metadata: name: tekton-pipelinebinding spec: params: - name: gitrevision value: $(body.head_commit.id) - name: namespace value: default - name: gitrepositoryurl value: $(body.repository.url) - name: gitrepositoryname value: $(body.repository.name) ``` Finally, we'll need the EventListener, with binds a TemplateTrigger with a TemplateBinding ```yaml apiVersion: triggers.tekton.dev/v1alpha1 kind: EventListener metadata: name: el-tekton-listener spec: serviceAccountName: tekton-triggers-sa triggers: - bindings: - name: tekton-pipelinebinding template: name: tekton-triggertemplate ``` ```bash ╰─>$ kubectl get service | grep tek el-tekton-listener ClusterIP 10.233.47.3 8080/TCP 114m ``` Now that we have the service, we'll just need to expose it using Istio's primitives. Let's use Tekton's tools for that, using a separate service account: ```yaml kind: Role apiVersion: rbac.authorization.k8s.io/v1 metadata: name: tekton-triggers-createwebhook rules: - apiGroups: - "" resources: - secrets verbs: - get - list - create - update - delete - apiGroups: - tekton.dev resources: - eventlisteners verbs: - get - list - create - update - delete - apiGroups: - extensions resources: - ingresses verbs: - create - get - list - delete - update --- apiVersion: v1 kind: ServiceAccount metadata: name: tekton-triggers-createwebhook --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: tekton-triggers-createwebhook subjects: - kind: ServiceAccount name: tekton-triggers-createwebhook roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: tekton-triggers-createwebhook⏎ ``` Following by the task itself: ```yaml apiVersion: tekton.dev/v1beta1 kind: Task spec: params: - description: The external domain for the EventListener name: ExternalDomain type: string - description: The name of the Service used in the VirtualService name: Service type: string - description: The service port that the VirtualService is being created on name: ServicePort type: string steps: - args: - -ce - | set -ex cat << EOF | kubectl create -f - apiVersion: networking.istio.io/v1alpha3 kind: Gateway metadata: name: $(inputs.params.Service)-gateway spec: selector: istio: ingressgateway servers: - port: number: 80 name: http-$(inputs.params.Service) protocol: HTTP hosts: - $(inputs.params.ExternalDomain) --- apiVersion: networking.istio.io/v1alpha3 kind: VirtualService metadata: name: $(inputs.params.Service)-virtual-service spec: hosts: - $(inputs.params.ExternalDomain) gateways: - $(inputs.params.Service)-gateway http: - route: - destination: host: $(inputs.params.Service) port: number: $(inputs.params.ServicePort) EOF command: - sh image: lachlanevenson/k8s-kubectl:latest name: create-istio-gateway-virtualservice resources: {} volumes: - emptyDir: {} name: work ``` And ending with it's initialisation: ```yaml apiVersion: tekton.dev/v1beta1 kind: TaskRun metadata: spec: params: - name: ExternalDomain value: simple-django-app-event-listner.default.147.75.80.160.xip.io - name: Service value: el-tekton-listener - name: ServicePort value: "8080" serviceAccountName: tekton-triggers-createwebhook taskRef: kind: Task name: create-istio-gateway-virtualservice timeout: 1h0m0s ``` And let's check the result: ```bash ╰─>$ kubectl get VirtualService NAME GATEWAYS HOSTS AGE el-tekton-listener-virtual-service [el-tekton-listener-gateway] [simple-django-app-event-listner.default.147.75.80.160.xip.io] ``` Now that we have the tools up and running in our cluster, we can create the webhook. For that, we'll need a Github personal token, stored in a secret ```yaml apiVersion: v1 kind: Secret metadata: name: webhook-secret stringData: #https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line#creating-a-token token: secret: random-string-data ``` The task that will actually create the webhook ```yaml apiVersion: tekton.dev/v1alpha1 kind: Task metadata: name: create-webhook spec: volumes: - name: github-secret secret: secretName: $(inputs.params.GitHubSecretName) inputs: params: - name: ExternalDomain description: "The external domain for the EventListener e.g. `$(inputs.params.EventListenerName)..nip.io`" - name: GitHubUser description: "The GitHub user" - name: GitHubRepo description: "The GitHub repo where the webhook will be created" - name: GitHubOrg description: "The GitHub organization where the webhook will be created" - name: GitHubSecretName description: "The Secret name for GitHub access token. This is always mounted and must exist" - name: GitHubAccessTokenKey description: "The GitHub access token key name" - name: GitHubSecretStringKey description: "The GitHub secret string key name" - name: GitHubDomain description: "The GitHub domain. Override for GitHub Enterprise" default: "github.com" - name: WebhookEvents description: "List of events the webhook will send notifications for" default: '[\"push\",\"pull_request\"]' steps: - name: create-webhook image: pstauffer/curl:latest volumeMounts: - name: github-secret mountPath: /var/secret command: - sh args: - -ce - | set -e echo "Create Webhook" if [ $(inputs.params.GitHubDomain) = "github.com" ];then curl -v -d "{\"name\": \"web\",\"active\": true,\"events\": $(inputs.params.WebhookEvents),\"config\": {\"url\": \"$(inputs.params.ExternalDomain)\",\"content_type\": \"json\",\"insecure_ssl\": \"1\" ,\"secret\": \"$(cat /var/secret/$(inputs.params.GitHubSecretStringKey))\"}}" -X POST -u $(inputs.params.GitHubUser):$(cat /var/secret/$(inputs.params.GitHubAccessTokenKey)) -L https://api.github.com/repos/$(inputs.params.GitHubOrg)/$(inputs.params.GitHubRepo)/hooks else curl -d "{\"name\": \"web\",\"active\": true,\"events\": $(inputs.params.WebhookEvents),\"config\": {\"url\": \"$(inputs.params.ExternalDomain)/\",\"content_type\": \"json\",\"insecure_ssl\": \"1\" ,\"secret\": \"$(cat /var/secret/$(inputs.params.GitHubSecretStringKey))\"}}" -X POST -u $(inputs.params.GitHubUser):$(cat /var/secret/$(inputs.params.GitHubAccessTokenKey)) -L https://$(inputs.params.GitHubDomain)/api/v3/repos/$(inputs.params.GitHubOrg)/$(inputs.params.GitHubRepo)/hooks fi ``` And it's initialization ```yaml apiVersion: tekton.dev/v1alpha1 kind: TaskRun metadata: name: create-api-repo-webhook-run spec: taskRef: name: create-webhook inputs: params: - name: GitHubOrg value: "vtemian" - name: GitHubUser value: "vtemian" - name: GitHubRepo value: "simple-django-app" - name: GitHubSecretName value: webhook-secret - name: GitHubAccessTokenKey value: token - name: GitHubSecretStringKey value: secret - name: ExternalDomain value: http://simple-django-app-event-listner.default.147.75.80.160.xip.io timeout: 1000s serviceAccountName: tekton-triggers-createwebhook ``` ![](/Screenshot_2020-05-16_at_20.23.20.png) Now, each time we push new changes, a new build is being trigger: ```bash ╰─>$ kubectl get po | grep teo teokton-build-simple-django-app-2fcdr-source-to-image-v86-mwxhw 0/3 Error 0 71m teokton-build-simple-django-app-qlw5w-source-to-image-sz2-gpqdm 0/3 Error 0 73m teokton-build-simple-django-app-sl9zf-source-to-image-knl-tzxpk 1/3 Running 0 18s teokton-build-simple-django-app-xh54x-deploy-to-cluster-b-5p7r4 0/3 Completed 0 66m teokton-build-simple-django-app-xh54x-source-to-image-wv5-9bsdt 0/3 Completed 0 66m ``` And the application is being deployed ```bash ╰─>$ kubectl get po | grep simple simple-django-app-cjx8b-deployment-7cd5c5999d-vwjhv 2/2 Running 0 4h3m simple-django-app-d2n6n-deployment-77c664bf4f-pz6hg 2/2 Running 0 4h29m simple-django-app-hcmpl-deployment-7687b96b5f-pv2wz 2/2 Running 0 67m ``` ## Conclusions In the end, we've managed to configure a bare-metal infrastructure, install Knative and have a CI/CD that builds and deploys new versions of our application, on git push. It's a little bit of a hassle and we left behind some details regarding revisions, [routing](https://knative.dev/development/serving/using-subroutes/) and [blue-green deployments](https://knative.dev/development/serving/samples/blue-green-deployment/). Platforms like [Vercel](https://vercel.com/), [Heroku](https://www.heroku.com/), [Google Cloud Run](https://cloud.google.com/run) and [AWS ECS](https://aws.amazon.com/ecs/) are truly remarkable, from an engineering point of view and because they lift the burden of deploying your application and manage your infrastructure. Kudos to Knative and Tekton for bringing such platforms closer to our reach. Cheers 🍺! Thanks [@catileptic](https://twitter.com/catileptic) for the awesome illustrations! --- ### Python Multiple Inheritance - **URL**: https://blog.vtemian.com/post/multiple-inheritance/ - **Published**: May 1, 2020 - **Word Count**: 892 words - **Tags**: python, mro, multiple inheritance, python multiple inheritance - **Summary**: How multiple inheritance works in Python? Inheritance, a simple and evil mechanism for re-using code, can get tricky. Traditionally, you may have encountered inheritance when you wanted to extend or override a class's behavior. ```python class Pet: def walk(self): print("walk") def eat(self): print("talk") def talk(self): print("!@#$") class Dog(Pet): def talk(self): print("Ham!") class Cat(Pet): def talk(self): print("Miau!") ``` Fairly easy to understand and follow. Dog and Cat share most of Pet's behavior, with some "small" particularities. Let's say that we want to isolate particular behavior, for better testing purposes. ```python class Eats: def eat(self): print("eat") class Walks: def walk(self): print("walk") class Talks: def talk(self): print("talk") def Dog(Eats, Walks, Talks): def talk(self): print("Ham!") class Cat(Eats, Walks, Talks): def talk(self): print("Miau!") ``` It may look a little bit weird, but isolating eating, walking and talking, improve re-usability, and may ease testing, but adds complexity and it increases readability efforts. What's Dog's or Cat's behavior? Well, you need to look at its parents. They inherit from Eats, Walks, and Talks. That class is commonly known as Mixins. Using multiple inheritances, you can compose specific behaviors. Think about playing with Lego. Using well defined, independent components, you can build complex constructs. In practice, you may use mixins within models, forms, serializers, views, etc. Multiple-inheritance seems to work for independent components that don't share state or behavior. What's going to happen when you try to combine a Dog and a Cat into a SuperPet? ```python class SuperPet(Cat, Dog): pass super_pet = SuperPet() assert super_pet.talk() == "Miau!" ``` SuperPet will inherit all the methods and properties of Dog and Cat and will respect the Method Resolution Order. More exactly, the C3 Method Resolution Order, which was initially released in a paper in 1996, designed for Dylan, called ["A Monotonic Superclass Linearization for Dylan"](https://doi.org/10.1145/236337.236343). It's implemented in other languages as well, Raku, Parrot, Solidarity, and Perl 5 (as a non-default option). Ok, it starts to look complicated, but basically, it just takes the methods of the left-most class, right? Almost. ```python class GrandParent: def describe(self): print("Grandparent") class Mother(GrandParent): def describe(self): print("Mother, son of") super().describe() class Father(GrandParent): def describe(self): print("Father, son of") super().describe() class Child(Mother, Father): def describe(self): print("I'm child, son of") super().describe() child = Child() child.describe() >>>> I'm child, son of Mother, son of Father, son of Grandparent ``` Introducing super() and we'll discover pretty cool interactions. Going a little deeper, Method Resolution Order (further referring to MRO) represents a set of rules that compute the linearization (a scary term that represents a serial way in which nested classes inherit from each other). Basically, it flattens the graph hierarchy. From: ![](/parents.jpg) to something more like `[Child, Mother, Father, GrandParent]`. The problem above is also known as the [Diamond problem](https://en.wikipedia.org/wiki/Multiple_inheritance). We can easily describe the linearization process as a recursive algorithm: Child + merge(linearization(Mother), linearization(Father)) or Child + the merge of linearization of the parents and the list of the parents. It seems a little trivial, but let's try a more [complex example](https://www.python.org/download/releases/2.3/mro/): ```python class O: pass class F(O): pass class E(O): pass class D(O): pass class C(D, F): pass class B(D, E): pass class A(B, C): pass ``` ![](/complex.jpg) ``` MRO(O) = O MRO(F) = F, O MRO(E) = E, O MRO(D) = D, O MRO(C) = C + merge(MRO(D), MRO(F) + MRO(DF)) = C + merge([D, O], [F, O], [D, F]) = C + D + merge([O], [F, O], [F]) = C + D + F + merge([O], [O]) = C + D + F + O = C, D, F, O MRO(B) = B + merge(MRO(D), MRO(E), MRO(DE)) = B + merge([D, O], [E, O], [D, E]) = B + D + merge([O], [E, O], [E]) = B + D + E + merge([O], [O]) = B + D + E + O MRO(A) = A + merge([B, D, E, O], [C, D, F, O], [B, C]) = A + B + merge([D, E, O], [C, D, F, O], [C]) = A + B + C + merge([D, E, O], [D, F, O]) = A + B + C + D + merge([E, O]), [F, O]) = A + B + C + D + E + merge([O], [F, O]) = A + B + C + D + E + F + merge([O], [O]) = A + B + C + D + E + F + O ``` You can double check it, using the `mro` function: ``` (, , , , , , , ) ``` And, of course, you can have your own MRO, no problem. All you need to do is to define a method called `mro` inside a metaclass. Random MRO is not that smart, but for sure is interesting. ```python import random class GrandParent: def describe(self): print("Grandparent") class Mother(GrandParent): def describe(self): print("Mother, son of") super().describe() class Father(GrandParent): def describe(self): print("Father, son of") super().describe() class RandomMRO(type): def mro(cls): parents = [Father, Mother, GrandParent] random.shuffle(parents) return [cls] + parents + [object] class Child(metaclass=RandomMRO): def describe(self): print("I'm child, son of") super().describe() child = Child() child.describe() >>> [, , ] I'm child, son of Mother, son of Grandparent >>> [, , ] I'm child, son of Father, son of Grandparent ``` Multiple inheritance can get messy and for the full story please take a look at https://www.python.org/download/releases/2.3/mro/. It can be really fun to play with, but in the long run it can be a real trouble maker. Cheers 🍺! Thanks [@catileptic](https://twitter.com/catileptic) for the awesome illustrations! --- ### Interviews - **URL**: https://blog.vtemian.com/post/interviews/ - **Published**: January 3, 2019 - **Word Count**: 739 words - **Tags**: interviewing, plan - **Summary**: A short interviewing statement for 2019 I’m really nervous, only thinking of them. I had only one or two real interviews and conducted a few. Usually, I’m pretty lucky when it comes to job finding. The companies I like or want to work with contacted me, and it’s really flattering. No interview, just a small discussion, a beer and I’ll start next Monday. ![](https://images.unsplash.com/photo-1535515384173-d74166f26820?ixlib=rb-1.2.1&ixid=eyJhcHBfaWQiOjEyMDd9&auto=format&fit=crop&w=1350&q=80) For the past 2.5 years, I’ve started practicing. Practicing maybe it’s too much. I bought the books (like everybody does: [Cracking the Coding Interview](https://www.amazon.com/0984782850-983056789626-Cracking-Coding-Interview/dp/B07F16D3BG/ref=pd_lpo_sbs_14_t_1?_encoding=UTF8&psc=1&refRID=RD27GF1B4JP3R14XH98M), [The Algorithm Design Manual](https://www.amazon.com/Algorithm-Design-Manual-Steven-Skiena/dp/1849967202/ref=sr_1_2?ie=UTF8&qid=1546546500&sr=8-2&keywords=Algorithm+Design) and [Elements of Programming Interviews](https://www.amazon.com/Elements-Programming-Interviews-Questions-Tsung-Hsien/dp/B00C7F0V3W/ref=sr_1_6?ie=UTF8&qid=1546546537&sr=8-6&keywords=elements+of+Programming+Interviews)), paid a monthly subscription to leetcode.com, did some problems out there and that was it. I even told all my friends, and they all supported me. But each time I discovered a hard problem or did a mock interview that went bad, I would really feel ashamed and stopped trying. I would procrastinate and find other things to do (like finishing other work-related tasks or start a side-project in a new language etc.). I wanted to feel safe and comfortable. I felt stupid that I didn’t know how to solve that problem, but I felt smart that I’ve could complete this extra task or did something pointless in a new programming language. Now, this year, I have a terrific incentive to start learning for interviews and start passing them. I’m 25, and I’ve been procrastinating on this goal for over 2.5 years. Since college, I’ve dreamed of working for a big company. Also, last month I refused the best offer I ever had: re-locate to San Francisco and work for a startup there. In Silicon Valley. Living the dream. [Excellent team, awesome people, awesome vibe](https://gorgias.io/). And I said no. I said no mainly for personal reasons that I want to keep them personal and, in some proportion, because I was afraid that I’ll never get to work for a big company. Maybe it’s stupid, and it sounds like problems of a privileged kid that refused a really lovely, 2018 toy, for a cooler and famous 2019 toy. It was hard, the hardest decision I’ve ever taken, but it’s mine, and I stand by it. They were super kind and supportive, and we still work together. ### Some inspiration Inspired by a friend of mine ([Alex Palcuie](https://interviews.palcu.ro/)), I want to start a log and to start working on the following 3 months plan (really ambitious one, but I may alter it a little bit): * build the resume. This also includes to research what a good resume looks like, writing it, asking for feedback, improve it. * read and practice from those 3 holy books: Cracking the Coding Interview, Algorithm Design and Programming Interviews. It’s a little overshot, but since I’ll be working part-time, it’s possible. I need to focus more on it. * practice on [Leetcode](https://leetcode.com/). I have in mind 500 problems (easy and medium) and 250 hard. I know that I can do those 500 in time, the hard problems will be tricky to hit in my timeline. I may extend the timeline or lower the number of problems. * start practicing with other people. This includes re-starting the algorithms meetup, doing mocks online with friends and hopefully, I can get an invite for [interviewing.io](https://interviewing.io/) and start doing mocks with strangers. * read, read, read. Start learning more about databases, about how to think at scale, more about distributed systems, fundamentals about networking, operating systems and security. Pretty much what’s on this [github repository](https://github.com/checkcheckzz/system-design-interview). **This period is not only about interviews. Is a period in which I want to become a better engineer. Right now, I pretty much feel like I’ve learned how to play an instrument, just by myself. Exploring it and discover it’s capabilities by playing and learning new songs for other people.** More like what we call in Romania “lautari” (usually gypsy musicians, but not necessary, who learned to play instruments to [survive](https://www.youtube.com/watch?v=joDtKv62V9c)). Continuing with this analogy, I like playing music, learning new styles and even composing new songs. But I never actually played in an orchestra, on a big stage. That’s nothing wrong with it, maybe is also a really authentic thing, but I crave playing with the big guys. *** Besides the goal of being socially accountable for actually sticking with the plan, I also want to share my thoughts, how I try to solve the problems and maybe some tips regarding the practice and the interviews themselves. For that, I’ll post a log with everything on [interviews.vtemian.com](interviews.vtemian.com). Cheers 🍺! --- ### 2019 - **URL**: https://blog.vtemian.com/post/2019/ - **Published**: January 2, 2019 - **Word Count**: 339 words - **Tags**: blog, 2019, vtemian, confort zone, interviews - **Summary**: 2019 - the beginning So here I am, me, **naked**, starting a blog. I must admit, I’m shy and I don’t have any **courage**. I’ve always been this way. I lack the courage to make decisions, to commit, to trust myself. I’ve always wondered if people will accept me and I’ve tried my best to please and impress people. This led to a stagnation stage. I was too afraid of trying new things, learning, experimenting, trusting. I was, and I still am **afraid of failing**. What my friends will think of me if I say something stupid? What will happen if I’ll say something wrong during a presentation? What if I’ll ask a foolish question, during a presentation? What will the interviewer think if I say or do something wrong? What my friends/parents will think of me if I fail this interview? Why did that driver honk me? Did I make something wrong? What...if...something...stupid? It’s easier to hide, to build a **comfortable space** and to stay there. It’s easier to feel safe and to avoid the harsh truth about yourself. I’m not *smart*. I’m not the *wonder kid I used to think I was*. I’m not *athletic*. I’m not a *leader*. I’m not *funny*. I’m not a *good friend*. I’m not the best *boyfriend*. With this realization, I want to start 2019. My motto this year will be to **“live uncomfortably”**. I’m not what I thought I was, but that’s ok. It’s normal *not always to be the best and the most*.... But that doesn’t mean that I can’t overcome my fears and to live a more **peaceful life**. That’s why this year I want to **live uncomfortably**. I want to start doing the things **I was afraid of**. I want to understand me better and to expose myself to the world. To internalized that **it is normal and ok to fail**. It’s ok to be *sad*, *disappointment*, *jealous*, *slow* or *stupid*. And this is the first step. Exposing myself to the Internet. To the people that I know or don’t know. **Hello!** --- ## Talks & Presentations ### Why I Think Software Development Will Be Automated by 2030 - **URL**: https://blog.vtemian.com/talk/why-i-think-software-development-will-be-automated-by-2030/ - **Date**: March 27, 2026 - **Summary**: My PRO argument from a live debate in Timisoara, fact-checked with verified references. ![Debating at All-Stack Hangout](debate-1.jpeg) Last night I debated "Will AI fully automate the Software Development Life Cycle by 2030?" at [All-Stack Hangout](https://luma.com/hl1u9s3a) in Timisoara, organized by [Victory Square Partners](https://www.linkedin.com/company/victorysquarepartners/). I was on the PRO side, alongside [Alexandru Binzar](https://www.linkedin.com/in/alexbinzar/) (CEO, Hibyte). Against us: [Cristian Toma](https://www.linkedin.com/in/tomacristian/) (20-year software engineer) and [Marius Balaj](https://www.linkedin.com/in/marius-balaj/) (UI engineer turned founder). [Lavinia Cioloca](https://www.linkedin.com/in/laviniacioloca/) moderated. What started as a structured debate turned into a real conversation. This post is my case, expanded and fact-checked. **TL;DR** - AI models are improving on two axes simultaneously: pre-training and reinforcement learning. The rate is accelerating, not plateauing. - The real shift isn't "AI generates code." It's that AI learns to *choose* between tools, patterns, and architectures. That capability generalizes across the entire SDLC. - Products like Lovable and Base44 prove the concept at scale. They're version 1. - The finish line is the closed bugfix loop: detect, diagnose, fix, test, deploy. Pieces of it exist today. - The honest answer: we're closer than most people think, but "fully automated" requires workflow integration that hasn't been built yet. ![Full house at Muzeul National al Banatului](debate-2.jpeg) ## Why AI Keeps Getting Better If you've used AI coding tools in the past year, you've probably noticed: they keep getting better. They "feel" like they understand what you want. You fight them less. This isn't just an impression. Until recently, models improved primarily by learning from text. Reading the entire internet, all the code, all the documentation. That still works. But since September 2024, when OpenAI launched o1, a second path opened up: reinforcement learning (RL). Models learn by doing. They receive a task, attempt it, get feedback, learn from mistakes, and try again. Exactly how a human learns, except thousands of times faster. The two paths compound. It's not one or the other. It's both at the same time. Dario Amodei (CEO, Anthropic) confirmed this on the Dwarkesh Patel podcast in February 2026: "We're seeing the same scaling in RL that we saw for pre-training" [1]. Every major lab (Anthropic, Google, DeepSeek) has demonstrated the same thing. The numbers back this up: | Metric | Then | Now | Source | |--------|------|-----|--------| | SWE-bench Verified (real GitHub bugs) | ~4% (early 2024) | 80.9% (Claude 4.5 Opus) | swebench.com [2] | | Autonomous task duration (METR) | ~1 hour (2025) | ~5 hours, doubling every 4 months | metr.org [3] | | HumanEval (code generation) | 67% (GPT-4, 2023) | 99% (saturated) | EvalPlus [4] | METR's time horizon data is particularly striking. If the doubling rate holds, by 2028 we're talking about week-long autonomous tasks. I'm not saying that's guaranteed. I'm saying the data shows no sign of slowing down. ## Tool Usage: The Real Shift When someone says "AI writes code," yes, it writes code. But that's 2024 thinking. The interesting part is that it's started to *choose*. Give it a problem and it doesn't just produce a solution. It chooses between options. Which API fits better. What data structure makes sense. Which database is right for your use case. You've probably experienced this yourself: you work with a model and it "understands" what you want. It doesn't just execute. It reasons about the right approach. This generalizes. The same capability that picks the right API for a feature can pick the right deployment strategy, the right test framework, the right monitoring setup. It's not magic. It's the same skill applied in different contexts. A concrete example: Rakuten engineers gave an AI agent a task on vLLM, a codebase of 12.5 million lines (Python, C++, CUDA). The agent completed it autonomously in 7 hours with 99.9% numerical accuracy [5]. A task that would take an experienced engineer days. It didn't generate code from nothing. It understood which tools and patterns to use in the context of a massive existing project. I see the SDLC as 6 separate problems: planning, implementation, testing, review, deployment, monitoring. And what I observe is that AI is learning to tackle each one. ## Proof at Scale: Lovable and Base44 Two examples that show this isn't theoretical. **Lovable** takes a sentence in natural language and delivers a working application. Frontend, backend, database, deployment. 8 million users. Over 100,000 new projects every day [6]. **Base44** has an even better story. A single founder, from Israel, bootstrapped, no investors. In 6 months: 5 million applications created by users [7]. What's interesting is that they solve the same problem for different audiences. Lovable is for developers: editable code, React, Supabase. Base44 is for non-technical users: you never see a line of code. Two completely different products, same underlying capability: take what the user wants and transform it into a working application. Neither can build a banking system with compliance and audit trails. They're honest about their limitations. But they're version 1, with one year of existence. ## The SDLC, Phase by Phase Let's look at where we actually stand. **Planning.** AI already takes requirements and breaks them into technical tasks. Google confirmed that over 30% of their new code is AI-generated [8]. **Implementation.** Over 80% on benchmarks. A peer-reviewed study showed developers with AI are 56% faster on coding tasks [9]. 84% of developers use or plan to use AI tools. 51% use them daily [10]. **Testing and Code Review.** AI already writes tests, already catches bugs in review. Not perfect, but useful enough for daily use. **Deployment.** CI/CD has been automated for years. The difference now is that AI can configure the pipeline from scratch, not just run it. **Monitoring and Bugfix.** This is the piece that matters most, and it's not fully solved yet. Think about what happens today when something breaks in production. You get an alert at 3 AM. You open logs. You try to understand what happened. You write a fix. You test it. You send it for review. You deploy it. Hours. Sometimes days. Now imagine the closed loop: AI detects the anomaly, traces the root cause through logs and metrics, generates the fix, runs tests, and if everything is green, deploys automatically. Without waking anyone at 3 AM. Not all of this loop exists today. But pieces of it do. And each piece is improving. I know some will say: "but if you chain 6 automated phases, errors compound." It's a valid point. But that's exactly the problem CI/CD already solves, with feedback loops, quality gates, and automatic rollbacks. It's not a new concept. We're just applying it at a larger scale. When the bugfix loop closes, the SDLC runs without human intervention. That's the finish line. And we're closer than most people think. ![PRO side huddle](debate-4.jpeg) ## What I Actually Believe I don't think the SDLC will be "fully automated" in the sense that zero humans are involved. I think by 2030, a human will describe what they want, and an AI system will deliver working code to production without the human manually writing, reading, or approving code at each step. The human's role shifts from executor to director. Some argue that the gap between "massive automation" and "complete automation" is a difference of nature, not degree. I think it's a difference of time. In 2022, AI couldn't write a correct function. In 2024, it could build features. In 2026, it delivers complete prototypes. Not production systems, not yet. I'm honest about that. But look at the velocity. The question isn't whether the SDLC gets automated. It's whether 4 years is enough. I'd bet on less. If you're curious how I actually use AI coding tools day to day, I wrote about [my harness engineering setup](/post/harness-engineering/) — the configuration, workflows, and boundaries that make it work. ![The audience](debate-5.jpeg) Stay curious ☕ --- [1] Dario Amodei on Dwarkesh Patel podcast, February 2026. [dwarkesh.com/p/dario-amodei-2](https://www.dwarkesh.com/p/dario-amodei-2) [2] SWE-bench Verified leaderboard. [swebench.com](https://www.swebench.com/) [3] METR Time Horizons 1.1, January 2026. [metr.org/time-horizons](https://metr.org/time-horizons/) [4] EvalPlus HumanEval leaderboard. [evalplus.github.io/leaderboard.html](https://evalplus.github.io/leaderboard.html) [5] Rakuten + Claude Code case study, Anthropic 2026 Agentic Coding Trends Report. [claude.com/customers/rakuten](https://claude.com/customers/rakuten) [6] Lovable stats. [techcrunch.com](https://techcrunch.com/2026/03/11/lovable-says-it-added-100m-in-revenue-last-month-alone-with-just-146-employees/) [7] Base44 acquisition by Wix. [techcrunch.com](https://techcrunch.com/2025/06/18/6-month-old-solo-owned-vibe-coder-base44-sells-to-wix-for-80m-cash/) [8] Sundar Pichai, Google Q1 2025 earnings call. [fortune.com](https://fortune.com/2024/10/30/googles-code-ai-sundar-pichai/) [9] Peng et al. (2023), "The Impact of AI on Developer Productivity." [arxiv.org/abs/2302.06590](https://arxiv.org/abs/2302.06590) [10] Stack Overflow 2025 Developer Survey. [survey.stackoverflow.co/2025/ai](https://survey.stackoverflow.co/2025/ai/) --- ### Riding Existing Waves: My Indie Hacking Journey with Sisif.ai - **URL**: https://blog.vtemian.com/talk/indie-hacking-update/ - **Date**: January 22, 2026 - **Summary**: How I went from 0 to growing MRR by abandoning traditional marketing and riding existing distribution waves. A former CTO's lesson in Platform-Led Growth. After 15 years of building production infrastructure—including a stint as CTO at QED (acquired by The Sandbox)—I thought I knew how to build a product. I was wrong. I knew how to build *systems*, but I had zero idea how to build *distribution*. This talk is the story of **sisif.ai**, and how I had to unlearn everything about being a CTO to find my first paying customers. ## The CTO's Unlearning: Distribution > Technicality Coming from a high-scale engineering background, my instinct was to "over-build." I wanted clean code, perfect infrastructure, and a robust CI/CD pipeline. But in the world of indie hacking, **nobody cares about your unit test coverage if you have zero customers.** I had to shift my mindset from **Technical-First** to **Distribution-First**. My "CTO brain" told me to optimize the video rendering engine; my "Founder brain" told me to find a way to get the product in front of people who were already looking for it. The lesson: **Distribution is the only feature that matters when you're at zero.** ## The Marketing Playbook (That Didn't Work) ... I did everything the indie hacking playbook says: - **Twitter/X content** — posted regularly, shared progress - **ProductHunt launch** — prepared the launch, gathered supporters - **SEO** — added llms.txt, optimized pages - **Building in public** — shared the journey The result? - **0** paying customers - **8** followers - ProductHunt: crickets - SEO: too early to tell ### Two Months of Twitter: A Reality Check I committed to **two months of consistent Twitter posting**. Daily updates. Progress screenshots. Building in public threads. Engagement with other indie hackers. The kind of content that supposedly builds audiences. After 60 days: **8 followers**. Not 8,000. Eight. Most of them were bots or other indie hackers doing the same thing. Zero customers came from Twitter. Zero meaningful conversations. Zero inbound interest. The accounts that "blow up" on Twitter? They either got lucky with timing, had an existing audience from somewhere else, or have been grinding for years. There's no shortcut. And for a solo founder with a product to ship, spending 2+ hours daily on tweets that reach nobody is a terrible ROI. ### The ProductHunt Launch That Wasn't I prepared a proper ProductHunt launch. Lined up supporters. Created assets. Picked a launch day. Did everything the guides recommend. Launch day came. And... nothing. No traction. No upvotes from strangers. The supporters I gathered weren't enough to break through. ProductHunt's algorithm buried the launch before it had a chance. Here's what I learned: **ProductHunt is a lottery**. The winners are either products with existing audiences (who bring their own traffic) or products that get lucky with the algorithm. For a new product from an unknown founder? The odds are stacked against you. The harsh reality of indie hacking. Traditional marketing is slow. Twitter takes years to build. ProductHunt is a lottery. SEO needs months to compound. ## The Pivot: Ride Existing Waves I stopped building from scratch. Instead, I asked: **where are the users already?** The answer was **n8n** — the workflow automation tool. Thousands of users building automations, looking for integrations. They don't need to find me. I need to be where they already are. ### The n8n Strategy I published **2 workflow templates** on the n8n creator hub: 1. **TikTok video creation pipeline** — GPT-4o-mini generates the script, Sisif.ai creates the video, posts automatically 2. **Instagram Reels automation** — same stack, different output format ## Platform-Led Growth: The "External SEO" Play The **n8n template strategy** was my breakthrough. Instead of waiting for Google to index a new domain, I used **Platform-Led Growth (PLG)**. I published workflow templates on the n8n creator hub that solved specific problems: TikTok automation and Instagram Reels. This isn't just distribution; it's **External SEO**. n8n already ranks for these keywords. By placing my product *inside* their marketplace, I bypassed the "Google Sandbox" period and started getting thousands of views overnight. ![n8n TikTok workflow](n8n-tiktok-workflow.png) This is the **"Ride Existing Waves"** philosophy in practice. Instead of building an audience from zero, you tap into established ecosystems where your users are already looking for solutions. ... These aren't vanity metrics. n8n users are exactly my target audience: developers and technical founders who want to automate video creation. They're already in a buying mindset — they're looking for tools to plug into their workflows. The conversion funnel is simple: user discovers template → tries it → needs API access → signs up for Sisif.ai. No cold outreach. No content calendar. No algorithm to game. ### Why This Works This is **riding existing waves**. Instead of building an audience from zero, you tap into platforms where your users already hang out: - **n8n** has thousands of users searching for workflow integrations - **Zapier** and **Make** have similar marketplaces - **GitHub** templates get discovered organically SEO? Write for big sites that already rank. Distribution? Let users find you through tools they already use. The math is simple: it's easier to capture 0.1% of 100,000 users than to build 100 users from scratch. ## Pricing Evolution My first pricing was wrong. **$9/month** single plan. Too cheap, wrong incentives. I switched to tiered pricing: - **Alpha Tester**: $10/month (100 tokens) - **Starter Pack**: $50/month (1,000 tokens) - **Pro Creator**: $200/month (5,000 tokens) Higher tiers = higher revenue per customer. The change increased MRR 4x. ![Revenue growth](revenue-chart.png) ## Lessons Learned After months of building and marketing sisif.ai, here's what stuck: **Distribution beats product.** Build where users already are. The best product nobody knows about loses to the mediocre product everyone finds. **Ride existing waves.** n8n, marketplaces, integrations. Don't fight for attention. Go where attention already exists. **Price for value.** Tiered pricing forces you to think about customer segments. Not everyone needs the same thing. Charge accordingly. **Traditional marketing is slow.** Twitter/ProductHunt/SEO didn't work yet. Maybe they will. But I needed results now. Existing platforms delivered. --- Find me at [@vtemian](https://twitter.com/vtemian) or check out [sisif.ai](https://sisif.ai). --- ### Technical Debt is Cheap - **URL**: https://blog.vtemian.com/talk/technical-debt-is-cheap/ - **Date**: October 27, 2025 - **Summary**: Can AI tools tackle a 2,000-line legacy Laravel controller in 3 hours? A case study in AI-driven refactoring and modernizing technical debt. March 17th, 2025. My regular work was done for the day. I had three hours before I needed to stop. A friend had recommended Claude Code a few days earlier. "Think of it like having a really junior developer on your team." I was skeptical. I'd used Copilot for autocomplete, played with ChatGPT. Wasn't sold on Cursor. But I had this project. The one everyone has. Making money for years, users depend on it, and every time someone opens the codebase they quietly close their laptop and find something else to do. So I pointed Claude Code at it. Not because I had to. Not because anyone asked. Just to see if the hype was real. ## The Codebase: A Shared Trauma The project was a classic legacy Laravel app—the kind that makes money, has real users, and keeps you awake at night. Controllers weren't controllers; they were **2,000-line novels**. - **Naming Chaos:** Files literally named `HomeController-2.php` and `HomeController - Copie.php`. - **Linguistic Debt:** Variables in Romanian (`$prenume_utilizator`), comments in English, and error messages in... who knows. - **Frontend Hell:** jQuery nested so deep it felt like a fractal. HTML strings embedded directly in PHP. One missing tag and the entire world explodes. No validation. No tests. Deployed via FTP. I'm not shaming this codebase—it was actually adding real value. But the interest on this technical debt was compounding every month. ## The Three-Hour Transformation I set a timer. Paid $20 for API tokens. No designer, no DevOps team. Just me and an agentic tool. ### 1. **Infrastructure: From FTP to Docker** Something that usually takes a full afternoon was done before I finished my first coffee. Dockerfiles, `docker-compose.yml`, and environment variable mapping appeared in minutes. We moved from "raw files on a server" to a modern, containerized local dev environment. ### 2. **Refactoring: Decomposing the Monolith** That 2,000-line controller? Claude analyzed the business logic, identified the boundaries, and split it into focused controllers and extracted service classes. It was like watching a sculptor carve a statue out of a block of marble. It translated the Romanian variables, added type hints, and generated clean doc blocks. ### 3. **The Economics: Refactoring as a Commodity** This is the real takeaway: **AI has changed the interest rate on technical debt.** If the cost to fix a legacy app drops from "three weeks for a team" to "three hours for one person," then technical debt is no longer a scary liability—it's an asset you haven't processed yet. We can now afford to modernization projects that were previously too expensive to justify. ... ## The "Human-in-the-Loop" Catch You still need to know what you're doing. AI tools amplify what you already know. They take the grunt work—the tedious refactoring, the boilerplate, the repetitive transformations—and compress hours into minutes. But Claude didn't replace my judgment. I made every architectural decision. I validated every change. Without senior Laravel or Docker experience, you'll just get plausible-looking garbage. But with it? You're a force multiplier of one. --- You have a project gathering dust. You know the one. Too scary to touch, too expensive to rewrite. Pick it. Set a three-hour timer. See what happens. Stay curious ☕ ---