How Close Can AI Get To Writing A Techdirt Post?

from the man-vs.-machine dept

I’ve talked on Techdirt about just a few of my AI-related experiments over the past few years, including how I use it to help me edit pieces, which I still write myself. I still have no intention of letting AI write for me, but as the underlying technology has continued to level up, every so often I’ll run a test to see if it could write a better Techdirt post than I can. I don’t think it’s there (and I’m still not convinced it will ever get there), but I figured I can share the process with you, and let you be the judge.

I wanted to pick a fairly straightforward article, rather than a more complex one, just to see how well it works. In this case, I figured I’d try it with the story I published last week about Judge Boasberg ruling against the Trump administration and calling out how the DOJ barely participated in the case, and effectively told him to “pound sand” (a quote directly from the judge).

I know that just telling it to write a Techdirt article by itself will lead to pretty bland “meh” content. So before I even get to the prompt, there are some steps I need to include. First, over time I continue to adjust the underlying “system prompt” I use for editing my pieces. I won’t post the entire system prompt here as it’s not that interesting, but I do use it to make it clear its job is to help me be a better writer, not to be a sycophant, not to try to change things just for the sake of change, and to suggest things that will most help the reader.

I also have a few notes in it about avoiding recommending certain “AI-style” cliches like “it’s not this, it’s that.” Also, a specific one for me: “don’t suggest changing ‘fucked up’ to ‘messed up.’” It does that a lot for my writing.

But that’s not all. I also feed in Techdirt samples, which are a collection of ten of my favorite articles, so it gets a sense of what a “Techdirt article” looks like. On top of that, I give it a “Masnick Style Guide” that I had created after feeding a bunch of Techdirt articles into three different LLMs, asking for each to produce a style guide, and then having NotebookLLM combine them all into a giant “Masnick style-guide.”

Then, I feed it any links, including earlier stories on Techdirt, that are relevant, before finally writing out a prompt that can be pretty long. In this test case, I fed it the PDF file of the decision. I also gave it Techdirt’s previous stories about Judge Boasberg.

Finally, I gave it a starting prompt with a fair bit of explanation of what angle I was hoping to see a Techdirt post on this topic. So here’s my full prompt:

Can you write a Techdirt style first draft of a post (see the attached Techdirt post samples, as well as the even more important Masnick style guide, which you should follow) about the attached ruling in the JGG v. Trump case by Judge James Boasberg. I have also attached a page of previous articles about Judge Boasberg which you should consider, especially as some reference this same case.

You may also want to highlight that Judge Boasberg just was vindicated after the DOJ filed a vexatious complaint against him because of earlier rulings in this case, but that complaint has been easily dismissed. The crux of this article, though, should be on the very strong language Boasberg uses, including the astoundingly candid statement that he felt the government “apparently was no interested in participating in this process” and how he read their response as telling “the Court to pound sand.” There are other parts of the ruling that are useful as well. It should also call out the specific remedy here, and how weak it feels, given the situation explained in the ruling and the DOJ’s actions. Yes, they lost the case, but this still feels like a slap on the wrist with no real consequences for the DOJ.

As always with Techdirt posts, instead of doing in-line quotes, we usually do blockquotes, often of one or even multiple paragraphs from the original sources, so please adhere to that style. Take all of this info and draft the best Techdirt post you can regarding the topic.

And… you can see the results embedded below. I’ve also embedded my original article (though you can just go read it on the site directly). I’m amused that it added a “from the such and such dept” line as well as tags, both of which are core parts of Techdirt posts, but not things I asked it for.

I’m extremely biased, as this is a question of who writes better, me or the machine, and I still think I’m better than the machine. But, I also have to admit it’s not bad. It’s decent. If I had that submitted by another writer, there are definitely changes and edits I’d make, especially to the crappy cliche’d ending. But it’s way better than what most people think of as traditional ChatGPT output, in part because of all the context and detailed instructions I fed it.

Just for fun, I then also asked my editing tool to do its own edit on its own writing (usually not a good idea to let it grade its own homework), but here’s what it had to say:

solid draft. the “pound sand” framing works well and the frustration comes through. few things:

the tension setup: you flag “frustratingly toothless” in graf 3 then say “on one level this is a win” much later. that’s the right editorial instinct—there IS genuine tension here—but you’re burying the lede on your own argument. consider setting up that complexity earlier so readers know they’re getting nuance, not whiplash.

missing a key detail from the ruling: Boasberg notes that “even Plaintiffs’ counsel could not represent how many of the 137 are still in Venezuela and how many wish to proceed with habeas.” that’s devastating. the chaos is so complete that the lawyers literally don’t know where everyone is. worth including.

the numbered remedies block: you cut it off mid-list. the 5th point actually matters—there’s a March 13 deadline for the government to file a Status Report explaining how/when they’ll transport people back. that’s your next milestone. either include it or acknowledge you’re excerpting.

the Venezuela carve-out: you mention it but undersell it. the ruling specifically says it’s “mindful” of the “situation in Venezuela remains in flux” given “the United States’s recent law-enforcement action there.” that’s pretty ominous language from a federal judge about whatever the hell the administration is doing in Venezuela right now. might be worth a parenthetical noting this deserves its own article.

the closer: “I wouldn’t hold my breath” is fine but feels like boilerplate. you’ve got stronger material to end on—maybe the March 13 deadline, or something that points forward rather than just shrugging.

one small thing: “a complaint that was summarily dismissed” in graf 2—might want “recently dismissed” or similar since it was just last week. “summarily” suggests procedure when really the story is how embarrassing the dismissal was.

overall structure is good. the piece does what Techdirt does well: walks through the actual document, quotes liberally, and doesn’t pretend the “win” is bigger than it is.

You’ll note that it also calls out its own weak ending, just like I did. It is kind of wild to see the AI call out its own weaknesses, while making you wonder—if that’s the case—why did it not catch those earlier and fix them? And that then makes you wonder if it’s just coming up with a critique for the sake of giving one (though, that ending really is weak).

The other thing I’ll note is, again, this actually was still a fair bit of work. It was less than writing the whole thing myself, but even just writing out the prompt itself took time, and this piece would still need a fair bit of editing anyway for publication which would probably take away any time benefit.

Overall, though, you can see how the technology is certainly getting better. I still don’t think it can write as well as I do, but there are some pretty good bits in there.

Once again, this tech remains quite useful as a tool to assist people with their work. But it’s not really good at replacing your work. Indeed, if I asked the AI to write articles for Techdirt, I’d probably spend just as much time rewriting/fixing it as I would just writing the original in the first place. It still provides me very good feedback (on this article that you’re reading now, for example, the AI editor warned me that my original ending was pretty weak, and suggested I add a paragraph talking more about the conclusions which, uh, is what I’m now doing here).

I honestly think the biggest struggle with AI over the next year or so is going to be between the people who insist it can totally replace humans, leading to shoddy and problematic work, and the smaller group of people who use it as a tool to assist them in doing their own work better. The problems come in when people overestimate its ability to do the former, while underestimating its ability to do the latter.

Filed Under: ai, prompting, writing

Companies: techdirt