Human vs. AI: Surprising Content Test Results

 
Drew Neisser: Hey, it’s Drew. I’m excited that you’re here to listen to another episode of Renegade Marketers Unite. And if this is your first time listening then welcome. This show is brought to you by CMO Huddles, the only marketing community dedicated to inspiring B2B greatness. And that has a logo featuring penguins. Wait, what? Yeah, well, a group of these curious, adaptable, and problem-solving birds is called the Huddle. And the B2B marketers and CMO Huddles are all that and more, huddling together to heat up the coldest job in the C suite. And now that CMO Huddles has three membership tiers, we’re ready to inspire B2B Greatness at all levels. To learn more, check out CMOhuddles.com. Now before we get to the episode, here’s a shout-out to the professionals at Share Your Genius. We started working with them over a year ago to make this show even better and have been blown away by their strategic and executional prowess. If you’re thinking about starting a podcast or want to turbocharge your current show, be sure to talk to Rachel Downey at shareyourgenius.com and tell her Drew sent you. Okay, let’s get on with today’s episode.

Narrator: Welcome to Renegade Marketers Unite, possibly the best weekly podcast for CMOs and everyone else looking for innovative ways to transform their brand, drive demand, and just plain cut through. Proving that B2B does not mean boring to business. Here’s your host and Chief Marketing Renegade Drew Neisser.

Drew Neisser: Hello, Renegade Marketers! Welcome to Renegade Marketers Unite the top-rated podcast for B2B CMOs and other marketing-obsessed individuals.

Alrighty folks, you’re about to listen to a bonus Huddle, a specially curated Huddle that we run once a month with experts sharing their insights into the topics that are most important to our CMO community. We call them Huddlers. The expert of this particular Huddle was Michael Brenner, founder of the Marketing Insider Group. He joins us to discuss a recent study by his company and CMO Huddles to measure the relative effectiveness of human versus machine-generated content. After you listen to this episode, be sure to visit CMOhuddles.com to get yourself a copy of this groundbreaking man versus machine report. Okay, let’s get to it. Welcome to today’s exclusive interview with the esteemed content marketing expert Michael Brenner. Michael is a keynote speaker, author, and CEO of Marketing Insider Group. As a former Vice President of Marketing and Content Strategy at SAP, Michael has an impressive track record of guiding organizations on their content marketing journeys, and we’re gonna put Michael’s link into chat.

So today, we have a unique opportunity to gain insights from his vast experience and expertise on content in general, and more specifically, the results of our man versus bot contest. And I can’t wait to tell you all about all of this. But Michael, how are you? Nice to see you.

Michael Brenner: I’m good, Drew. Great to be here. Thanks so much for having me.

Drew Neisser: Where are you?

Michael Brenner: I’m in Westchester, Pennsylvania, just outside of Philadelphia.

Drew Neisser: When you and I first talked about doing a man versus bot contest back in March, I think that was about right. We were kind of exploring uncharted territory, talk about the basic test design, and why we ended up pivoting to three content author types from two.

Michael Brenner: I think what we touched base in March, you know, the fun knowing that so many folks were talking about ChatGPT and generative AI, and is it going to take over marketing? And you offered up your Renegade.com website for us to do a test. We decided that we would follow basically a standard process that we do for all of our clients where we do keyword research, trying to understand the audience that comes to Renegade, the topics on the minds, the challenges of typical CMOs that I’m sure all of you would understand. And then we decided we were initially going to do 30 AI and 30 human articles. You brought up, I think just a week or two in, that it’s not a binary decision. Some folks are just trying to do a little bit of both – balancing utilizing ChatGPT, but then human editor. So we created a hybrid category where our writers would spend about half the time that they normally spend generating a human piece of content and then allow AI to take the other half. So yeah, that was it. So we picked 60 different topics – 20 were human-written, 20 were AI-generated, and 20 were a mix of both. We started publishing in early May, I think we did six a week through the beginning of August, I think August 3 was our final date. The whole time we were tracking using a tool called SEMrush, tracking the capability of that tool at least to make it objective – so the average search rankings across those three categories, the visibility, which is a combination of the search ranking as well as the click-through rate or the likelihood someone was going to actually click through on the top search rankings for a keyword. Search rankings, visibility, and estimated traffic based on those two factors. And so those were the three metrics that we were looking to measure.

Drew Neisser: So Renegade used to be an agency. Now that the website Renegade.com exists as a media property, it literally exists to support the CMO Huddles community. And so for that brand to have a lot of content in a short period of time was obviously a windfall. But we’ve been publishing on that website for 15-20 years. And we’ve seen over time that certain content does really well and certain content doesn’t do very well. And we were very specific with our keyword strategy. So it was kind of exciting to just turn over the reins for a moment and say, “Hey, what keywords do you think we could rank for?” Because we know when that happens, typically traffic follows. So okay, you did 20 pieces of content per author, we did it over a 20-week period, talk a little bit about just the content development process for each author type.

Michael Brenner: Yeah, so like I said, it was human, hybrid, and AI. All of the research that we did, all the ideation, the headline development that we came up with was all basically human-developed, if you will. But once we got to that calendar of 60 ideas, we split them up randomly into the three types – human, hybrid, and AI. So for human, it was basically our writers writing an article as they normally would. They Googled the focus keyword, they looked and see what other articles are ranking, they looked at other top sources in the industry. And essentially, they create an outline, they write, and we publish. The average time for each of those was about three to four hours. So 3.5 hours for that. On the AI-generated side, we basically went into ChatGPT and said, “Hey, give me an outline.” ChatGPT can just generate an article, it doesn’t typically get to the length. We tried to standardize it around 1000 words. So we had to play with ChatGPT a little bit with the prompting, but once we figured it out, it took us about an hour. So generally, it was, “Hey, create an outline for this topic. Okay, great. Write section one, section two, section five, whatever.” And then from there, our human team, our editors would essentially just kind of format it into a standard format. So again, trying to minimize the number of factors and variables that were going into the test. So we were really focusing in on who wrote the piece of content. And then with hybrid, we did the same thing that we did with AI-generated content. But our editors went in – in some cases rewrote it, they obviously did the reformatting that they normally do, but in some cases, they rewrote it to a level of quality that they thought would be almost at the level of a human-written piece of content. We never really achieved it, but we kept the amount of time that they would spend on it at about one and a half hours.

Drew Neisser: So we got about three to four hours, we’ve got about one and a half hours for the hybrid, and maybe one hour for the bot only. And I want to do a little editorial on the bot-only content right now. I’ve been talking about using it for newsletter creation. And the most recent time that I tried to do it, I have to say I was so disappointed with what ChatGPT came up with. And the reason was the language was repetitive. And whenever they could use a more complicated word instead of a simple word – great example being “used” or “utilized”, they would always put “utilize”, which to me is always the sign of a bad writer. It’s using longer words instead of being plain and clear. So I struggled with that. So I’m imagining in the hybrid mode, which is the way I look at it a lot, that you could have kept going. If you don’t put limits on it you’re basically going to end up putting a couple hours easily into editing it, right?

Michael Brenner: What they were editing for in the hybrid model was basically factual correctness, relative completeness. So it was like “six things that you needed to know,” they wanted to make sure there were six things and those six things were relatively valid. So it was mostly a quality check. They weren’t changing words like “utilize” to “use.” My name was on all of those pieces of content, and it was on your website, we want to make sure it wasn’t bad or wrong.

Drew Neisser: One thing that you mentioned is that each of these pieces is about 1000 words. And I know you’re a big fan of longer content. For years folks have been saying “short attention spans, keep the copy short.” Why is that 1000-word threshold so important in your planning for programs?

Michael Brenner: Yeah, well we do a lot of research on this. And there’s a lot of research out there. And there is a correlation-causation sort of issue with this. And what I mean by that is you can say that 1000 words or more will get you to rank better on Google search. But you can say, and we’ve proven, that the content that is 1000 words or more is more likely to rank. So the correlation is there. We all have to guess what Google’s reason for that is, but I think it’s pretty clear that longer, more in-depth, more complete answers to a question are going to be more likely to be determined either by the search engines which are placing things in the top rankings. But a lot of those rankings are based on factors like other people linking and the amount of engagement and things like that. So what it means in the end is that longer pieces of content just tend to rank better.

Drew Neisser: I suspect it’s longer, it feels more thorough. If it’s well done, it is more thorough, and that means that if somebody bothers to read it, because I remember all the ad studies way back when people were running print ads, the funny thing was they always said short copy is better. But if you wrote a long copy, if they read the first paragraph, they’d read the whole thing. It was a high percentage, maybe there’s a certain sort of credibility that gets assigned to longer ones or not. I just want to confirm that all the headlines, all 60 of them, just to be clear, we’re human written. And I think that’s an important point to make in this test – that they were human-written, because that was the only way to control the variables, then you just randomly assigned it to all three categories, right?

Michael Brenner: We have not found a tool. And we’ve tried that can do headline development in a way that isn’t so ridiculously boring and generic that we wouldn’t ever suggest it to a client. So we pick the focus keywords with our research, we use our writers and editors to come up with the headlines the way we normally do, all trying to get down to defining that independent variable of the writing process.

Drew Neisser: So we have keywords that we’re trying to target, then we write the 60 Headlines or so that embrace a certain number of those keywords. And they were spread around purposely, then 20 pieces were written by the machines, 20 people were written by humans. And then there was this hybrid version, just to be clear. Okay. We already talked about metrics a little bit in terms of the search, the visibility, and the traffic. Let’s get in some of the results and just start with site traffic. What happened there?

Michael Brenner: Overall site traffic was up, overall search site visibility, search visibility was up, and overall ranking was down, which means it improved. So overall, what we showed was that publishing 60 articles across May, June, July was net total positive for Renegade.com.

Drew Neisser: I think, because it’s so dramatic, it looks like a horse race, maybe if you can pull that chart up. So look at this – AI is blue, human is green, and hybrid is red. And the higher the bar, the more the traffic. Talk a little bit about the headlines here and what we think this set data shows.

Michael Brenner: Overall, the site saw about a 5x increase in traffic from when we started. The first publish date was on May 24th. The last was on August 3rd which is represented by the black line there. When we looked at which one of the author approaches drove traffic the most, you’re right, it was a total horse race. We saw hybrid jumped out to number one, human actually ended up matching it after about three weeks, and AI was basically flat at zero for almost two months of this test. And then you can see there was some variability – hybrid fell off, it fell down to zero a couple of times, human took off a little bit around the 20th of July, and AI started to show some impact on the traffic metric, which is probably the closest thing to something that I would say is a harder metric. Rankings don’t mean anything unless they’re driving traffic, search visibility doesn’t mean anything unless it’s driving traffic. So this was really the key metric, I think. And then you can see, so there’s some variability over the last few weeks of the test. At the end, we ended up with human driving the most significant traffic not only at the end of the test, but also over time if we accumulated it. The hybrid are sort of right around the middle. The content we publish on the site that was written by AI is now at this point producing zero traffic for you.

Drew Neisser: Which is really interesting. And one of the things that I was talking to someone else about is, in theory, there should be some sustaining power here from this. It’s not like you air this, you post this and traffic should die. And what’s so interesting is with the bot, only the AI only traffic just essentially died, whereas the human traffic continues. Overall, you’re looking at two or 3x performance from a traffic standpoint, from human.

Michael Brenner: Yeah, I mean, there’s a lot behind the data here. The headline for AI was a generated a number one ranking on one of the pieces of content, but it also lost a ranking. One of the keywords that we had was a keyword that you did rank for and then you lost it when we published an article on it. So it was kind of like a plus one minus one equals zero. What we saw with human and hybrid is sort of somewhere in the middle. What we saw with human was you actually gained five new rankings, you improved in a couple of rankings, you did still lose and decrease in a few but it was less than the gains. And so I think the general gist of this was the human content generated it was more likely to get a ranking for the focus keywords at a more breadth of focus keyword standpoint than the single winner, single loser we saw with AI. And again, hybrid was kind of in the middle it produced the winners did produce a little bit of loss, but in the end, it ended up somewhat net positive.

Drew Neisser: When I look at content that is purely developed by chat GPT, what I’m seeing is what I think would call average consensus of what other people have already written about the topic. And at least with the human, there’s an opportunity to bring something fresh, some formulation of words, that hasn’t been combined before, that might create some insights in a different way. One is like, by definition, kind of generic, it says it’s see, it’s just average stuff. And in theory, and less, the humans aren’t good professional writers, the humans are going to outperform average. Do you think that’s a reasonable conclusion here?

Michael Brenner: I think that’s part of it. Sometimes I joke that there isn’t an original thought that people can have at this point. What’s really interesting, and I’ll give you a quick story, I was at an Adobe conference a couple of years ago, Drew Brees was the keynote. And I don’t want to go too deep into it. But he was talking and everybody I looked around was on their phone. And he’s like an interesting guy. The facility was asking him questions that were very interesting. And he was answering them in a not very interesting way. And then they asked him a question about how have you been since retiring? And he said something like, Well, you know, this year, I turned 40. And as soon as he said that everybody’s head popped up from their phones. And I was just thinking about, like, why was that the point where people were interested, because a lot of people in the room had turned 40. And they had faced the prospect of getting old? And how to answer that, well, how does it feel to get old, if all of a sudden, it went from being a celebrity discussion to something that everybody could relate to. And I think that’s true with content. Google talks about double E-A-T, which is expertise, authority, and trust, a bot doesn’t have any of those things, it can’t convey any of those things. And even when it can be smart to give you a nice list of six time-saving tips for CMOs. It’s different when it says, I’ve been a CMO. And it’s really hard when you’re stressed and your calendar is booked. How do you find the time to do the right things? Well, these are the tips I found. And ChatGPT can’t say that. So let’s go off the sharing for a second. Lots of questions.

Now. One is, do we have a hypothesis on why we lost rank in some cases, when you create a bunch of content around one keyword, you might put other keywords at risk? I think it’s important to understand what we’re not saying is that the AI content caused the drop in ranking. And I think we need to be really clear about that. Because we wanted to make this an objective test and unbiased as possible. And that’s the truth of it. The fact is, every website is gaining and losing rankings all the time. And more so since the Google helpful content updated in December of last year. And the algorithm updates since we go from 99 and 100,000, ranking keywords down to 40. And back up to 80. That’s just the way that the world works these days is we just don’t see that sort of nice growth chart. Or when we do it’s because you gain two and lose one. That’s what I think was happening. Again, the piece of content where you lost ranking weren’t bad pieces of content. It just meant that in the process of Google trying to figure out what should be a ranking piece of content, you ended up losing one, I think Renegade.com has seen some small decline in rankings in the last three years since you shifted your business model. And that happens one websites all the time. So I don’t think that AI content drove the loss. We saw losses for AI hybrid and human content. It just so happened that the human content was more likely to generate an increased or a new ranking than the AI in the hybrid.

Drew Neisser: Is there another comparison to consider? Like Jasper AI seems to be focused on headlines in search optimization. What do you think in terms of that?

Michael Brenner: We’ve got 35 writers and I’ve been pushing them to test all these tools, because I would love to save 80% of my costs and pass those on to our clients. We’ve tried Jasper writer with the W writer ry Tyr, we’ve tried content at scale, we’ve tried AI SEO, which is a ChatGPT plugin, we’ve tried ChatGPT, we’ve tried almost every tool that I know of out there. And we just haven’t found a tool that would produce a piece of content at the end, that I would feel comfortable could go head to head with a written piece of content by a human. There are some other tests out there. And I’m happy to share some of those results. They’re not very positive for AI-written content. The most recent one showed that AI-written content can generate rankings, and it looked at a lot of variables can generate rankings very quickly and loses those rankings. That’s the consensus of the research shows that it actually can get your ranking quicker. But it often loses those rankings over time, whereas human-written content is kind of a slow and steady increase. I’m not here to disparage any of those tools. They can all be good writing assistance, which is what we’re looking at but from like, hey, do some good inputs and you get an output that you could hit the publish button on. We have not found that to be the case yet.

Drew Neisser: One other question is you and I talked about this at the very beginning because one of my fears early on when we first talked about this test was oh my god we’re gonna put 20 100% AI-generated pieces of content on Renegade.com are we gonna get punished by Google. The question was, do you think search algorithms are aware of content that has been AI-generated, and that ends up not only hurting the ranking of the keyword for that particular content, but could hurt the whole website?

Michael Brenner: Yeah, I’ve done three keynotes in the last two months. And in each one, I use a couple examples that show websites that have gone from high traffic growth to zero. With only AI bot-generated automated publishing, there was a consumer travel site, a large B2B technology company that was a fake company, but it looked like a B2B technology company. And a review site was basically using AI-generated reviews, all three of these examples, the websites were penalized, and at this point are yet to see traffic come back. So there is risk for sure.

Drew Neisser: Yeah, it almost makes me want to go back and have our team go in and edit the AI-only posts that are up there.

Michael Brenner: We made sure we didn’t hurt your site by making sure the content wasn’t wrong. And it didn’t say I’m sorry, as large language model, I can’t answer that question. We made sure that the incorrect hallucinated super poor quality, if we said it was going to be six steps, there were six steps and the six steps were valid.

Drew Neisser: In this test, we didn’t really look at any AI content detection tools, that was not what we were doing. So Okay. All right. Well, I want to share one thing, because this is kind of fun, because there was one post that did better than all the rest. And I just want to share that, which is time management for CMO. Six tips to maximize productivity. There are a couple of classic things in here. It’s a listicle. It’s about productivity and time management. And if you were to ask me, hey, Drew, what is the number one challenge CMOs always talk about? It’s time and lack of time. This felt like a bullseye. And of course, it was just one of the headlines, and then I’ll read the first sentence. As a Chief Marketing Officer, your role is a critical one that often requires you to juggle multiple responsibilities simultaneously. From overseeing campaigns to managing teams, attending meetings and handling day-to-day operations, the list of tasks you’re responsible for can be endless. With so much on your plate, managing your time effectively is essential to ensure you’re able to achieve your goals while maintaining your sanity. All right, I’m gonna stop sharing, and I’m gonna hope that our team can generate the poll. There it is. All right. So your job, guess how was this post created human-only, hybrid, or bot-only. And the votes are coming in quickly like that. All right, we’ve got 16 out of 20 votes. And it’s a tie between hybrid 50% and bot only. Now I’m really dying to know why people thought this was bot-only content not to drag this one on. But this was in fact, bot only. We talked about why this was such an initially effective piece, right? It was the listicle. And it was the topic, and it was one. What was interesting about this post was it killed it, like push, and then it just flatlined. This really begs the question after it was killing it. Did Google say wait a second? We don’t know the answer to that. But do you have any speculation on both? Why this did so well, and then why it tanked?

Michael Brenner: Well, definitely and why it did. So well. It’s going to be a crystal ball on why it tanked. But the things that AI can help with it’s best for blank page syndrome, when they really just don’t know where to start, they found that the tools can be helpful. Is it an evergreen post? Is it something that is relevant over time not newsworthy, and a recent analyst post, those are the two things where AI, at least outlines and support can really be helpful. If it needs that opinion. Obviously, it can help if it’s recent ChatGPT particularly goes to 2021. So obviously, those things are out. So that’s why I think it did well. It’s a list, post ChatGPT did a good job of identifying the right six things. And then why did it tank again, if it was Michael Brenner or Drew sharing our reasons, our experience, the things that we found help CMOs to save time, maybe a little story, maybe an example or a case study, those kinds of things would have certainly helped it, I think to have a little bit.

Drew Neisser:  As we’re talking, I’m definitely rewriting this piece. Because you know, you’ve talked to more CMOs than anyone in the world. Not quite, but we’ve done time management for CMOs two or three times in bonus Huddles. So it is an important topic. And we talk all the time about ways to find time to free up time for strategic thinking, okay, obviously the power of a compelling headline, we got that one of the things that while overall humans outperform human content outperform hybrid content, it is important to say, did it outperform it two to one, and the reason is that hybrid is about half the time so was human clearly the winner or is the future hybrid?

Michael Brenner:  We’re already in a hybrid world. Gmail finishes my sentences. I think that the cat is out of the bag, and I don’t think we’re ever going back to writers who aren’t utilizing the tools that are available. The two-to-one bar I think is a little bit misleading. It was two to one only in the writing portion. And I think we have to keep in mind we had humans doing the keyword research, the headline development, even pushing the article live, those kinds of things. When you add it all up, it was an hour or two different but an overall process that was a little bit longer. But yeah, I mean, I do think hybrid is the future. And like I said, we’re constantly testing the limits of, can we save time? And can we replace human writers with AI? I don’t think we’re there yet. I don’t know if you want to go down the road, or are we ever going to get there? I’ll leave you to kind of answer that. But I keep coming back to my team saying, can we do two posts in the same time, it used to take us to do one and we’re not there yet. And because if we were I could save time, if it was anything less than two to one, I can actually start saving money or passing savings on to our clients. So we’re not there yet. And then again, I’ll defer to you on whether we want to prognosticate on the future.

Drew Neisser: Since we’re talking about that. I am surprised that your team can’t get a two-to-one advantage on that. But I get it you said a second ago was that we’re already in a hybrid world. I mean, I’m sure most of you have Grammarly sitting on top of that, if not any other tools that are checking your spelling that are checking your grammar. And a lot of times I find myself taking those suggestions, or whatever it was some sort of good classic English language structure. And then of course, I find myself rejecting it, because I wanted the stylistic difference from what is classic language. So it’s hard, we’re already using the notion of human 100% human-made, or what I like to call hashtag human-made is gonna really be hard to stick by, right? I mean, we’re there.

Michael Brenner: We use Copyscape and Originality.ai reviews on every piece of content our writers are creating to make sure that they’re not plagiarizing, and using pure copy-paste from AI tools. The reason is, because why can we get to two to one, it’s not that we can’t get to two to one, we this test showed that we can, but you want to go back and rewrite that six tips to time management because it’s not an article you feel great having your name on and our clients refer to it as high stakes content. When Michael Brenner publishes an article on marketing and Senator group, I don’t want it to be from a bot. I’m worried when somebody’s making a $50,000 annual decision on a content program. I want them to not feel like they’re getting content from a bot. Like I want them to know that it’s coming for me. So in the high-stakes world that we’re in, especially in B2B, I don’t think we’re ever going to get there. But yeah, we use those tools. We use Grammarly. We use all those things.

Drew Neisser: First, I want to make sure that it was Copyscape, which is for what and what was the other one?

Michael Brenner:  Copyscape is a plagiarism check as a 90% not-plagiarize score. Again, there is no piece of original content out there. I don’t think I’ve ever seen 100%. And then we use Originality.ai. We’re always looking for tools here. And they’re not always perfect. But we’re actually looking for AI detection to make sure that our writers aren’t getting lazy.

Drew Neisser: Okay, high stakes content, it seems to me that any content that a brand puts on its website, potentially as high stakes content, I get the difference, say between a throwaway post on a social channel, but even then I would argue every bit impacts the brand, but talk about the high stakes content notion a little bit more.

Michael Brenner: The content we’re writing, a lot of our clients are SaaS B2B founders. So they’re technology company founders. And we’re writing on, you know, what is data quality in a hybrid cloud computing environment, how to pick a tool for multi-clouds on the edge, a CIO, a senior director, VP of it reading. And for that company that we’re writing for, it needs to be something that’s adding value that is educating the reader. That is not something thatChatGPT can just spit out. And so it’s high stakes for them. You know, again, the pros and cons are, I think we can get into some of that. But I think all of us would consider the content published on our websites at relatively high stakes, who doesn’t look at the content in an apathetic way, we all care about what’s going to get published. And so it’s all high stakes, the content has to be good.

Drew Neisser: And it also makes me wonder if we should just turn off the AI-created content on Renegade.com. That’s part one of this, which I’m interested in your thoughts and we turn that off.

Michael Brenner:  I don’t think it’s hurting you. Like I said, we made sure it was meeting a level of quality that we didn’t think was going to hurt you. Maybe going back through those 20 pieces and adding a little bit of an example, some experience, I have found and working with 25 clients that this thing applies just a little bit of that I think can help. To be clear, Google has said they don’t care if content is generated by AI, but they do care if it doesn’t have experience, expertise, authority and trust. And so if that article doesn’t convey any of those four things, you most likely will lose the ranking. I don’t think with the 20 piece of content we created at the quality bar. We brought it up to that it’s going to hurt you. But I think it could certainly help you to go back and append expertise into some of that content.

Drew Neisser: And love it and was circling back on one of the things you talked about in this high-stakes world in Huddles, a lot of Huddles, were talking about content and classic term was sales enablement or buyer enablement. And where are we starting to see the conversation? Move to is this notion of value enablement. There is this exchange of value where you’re giving my time to read the whatever it is content and I’m giving In you actual information that is well thought through and have legitimate value. And what we’re just basically saying is at the moment, the AI stuff isn’t at that level. So are there any tools out there similar to ChatGPT that are actually up to date?

Michael Brenner: I know that there is a competitor to ChatGPT. And I don’t remember the name of it. There are two or three. What’s interesting is that, again, unless you’re writing about something that’s recent, like, what’s the impact of Russia’s invasion, Ukrainian or something, most of the B2B content out there is relatively evergreen. I mean, there’s not a lot of stuff that needs to be reflective of recent events. It’s interesting, you can ask ChatGPT like, what’s the future of AI content? Since the launch of Chat GPT to it will say and regurgitate the words? I think my point is, I don’t think that the recency of the training data is the issue. It’s knowing that training data is used to help these programs predict words. And because it used all the content on the web from 1988, or 1965, or whatever, when the web started on through to 2021, or 2023 matters as much as people think the other issues I think that are more important are the fact that AI lies it doesn’t mean to but you need a human editor to make sure that it’s giving you actual factual information. That’s a bigger deal. There’s a huge issue with plagiarism and copyright concerns that are being settled in the courts as we speak. And Sarah Silverman is one that sued open AI because they scraped your book, and they used it in the training model without giving her any rights and other artists and authors that are doing that. So plagiarism copyright concerns, hallucinations, and factuality, I think are bigger issues from a risk perspective than the recency.

Drew Neisser: Just to be clear, because I know that some folks have used that as an excuse to not use it at all. When you’re creating content, a hybrid content, for example, you write the headline, you can give it the six bullet points, you can even give it the quotes. And all that does is say weave it together. Or you could write something yourself. And then you could say, you know what, I’d like to see this but with a different metaphor. It’s still original content. And so I think it would be a mistake to walk away from the saying, we’re not moving into the world of augmentation, even if the New York Times wins their suit against ChatGPT and others, it’s going to be like music industry where these folks like artists are going to get compensated somehow who knows how it is. And these tools will still be available one way or another. Based on the information on the test of the 20 tools you’ve tested, are you less bullish on SDRs being replaced by AI capability? Or will it be more of a hybrid approach there, I could see the use case of AI responding to non-ICP as an ideal customer profile target markets.

Michael Brenner: I don’t think writers are going to be replaced. I think I mentioned you Drew that I talked to a technology CMO a couple of weeks ago who said in two or three years, all of our writers are going to be replaced by AI. And I tried to be gentle in letting them down that I think he had budget dollar signs that he could shift into traditional advertising and golf sponsorships on his mind. But what we’re gonna do is we may shift from 80% writing and 20% editing to 20% writing and 80% editing, I think we’re seeing a lot of that I don’t think STRS are going to be replaced. Julie right. They did a great test where they came out said that conversation agents on their website, increased conversion by like 38%. But this is I think it’s hilarious. It also told people to go check out products that SAP did sell.

Drew Neisser: Not just being generous, they’re adding value.

Michael Brenner: I do think that those are some areas that are STRS are too important. Writers are too important. AES are too important. Social media agencies, I think we’re still gonna need people to push the buttons. But AI is really pretty good at doing social copy, headline development, repurposing existing stuff. ICPs, like personas, and my friend Andy crusted, he did a great article on how to use ChatGPT, to create a really in-depth customer profile. There’s a lot of technology in that space as well. But again, like the thing that touches the customer, I don’t think we’re gonna see AI replacing that actually true. The first time I said, this was in front of you at a Forrester roundtable in Washington DC, the paradox of AI is, the more automation we use, the more human touches are going to be required. And so we’re going to see this healthy balance where we’re going to be using automation and AI, we’re also going to need to make sure that we’re being human, and we’re touching our potential customers and employees with actual people.

Drew Neisser: We have a question about copyright. It’s interesting, because for all the years of the content that we’ve created so many different research studies, when Renegade was an agency, and really I didn’t think about copyright protection as anything necessarily important unless it was like a book. Is this a factor in your mind? I mean, how important is it to your clients that the 1000-word blog post be copyright-protected?

Michael Brenner: The problem is, and again, this is more of an agency client-ish situation, not a like if a client’s using ChatGPT it may not be as much of an issue but our clients are asking us to provide for them or original, bespoke content that they owned. I can’t scrape something from ChatGPT give it to a client and say that they own it. It is actually copyrighted and owned by open AI the owner of ChatGPT. That’s the issue with agencies and clients. Now, if you’re a CMO and a company and you go into ChatGPT, and you want to use it to write a newsletter, I think you’re probably okay. If you want to use it to publish a piece of content that you’re going to put your name on, and you’re going to put it on your company’s website, that’s going to have a little copyright at the bottom, you’re really in the gray, you’re publishing somebody else’s stuff, and calling it your own. So whether somebody’s going to come after you for that is really the issue. I don’t think it’s very likely, but it’s not something you technically own. And so that’s a concern.

Drew Neisser: It’s so funny, because I don’t think while I understand certain copyright things, you want a copyright? Do we worry about copywriting or social posts? We don’t, I guess there’s levels here, where if you’re publishing a book, you better be original content, if you want to copyright it, I get it. And if you’re creating content that is actually worth copywriting, and that you’re striving for that yeah, then it’s got to be human. But not everything that goes out on your website necessarily has to be copyrighted, or it to even do the things that we’re talking about. Okay. Any big surprises from this test? In your mind mean, when we went into this, we had no idea what was going to happen, any surprises for you.

Michael Brenner: Yeah, I mean, I think that the AI article, jumping to number one was a surprise. I was like, wow, like, it’s really going to win, then we kind of looked into it. And we talked a lot about it Drew. And it kind of pulled the covers back a little bit and saw it wasn’t exactly the golden nugget that we thought it was going to be. But that was an early surprise, the amount of traffic or the lack of traffic that AI drove was also a surprise, what I was afraid of, I guess a little bit was that hybrid was going to be the winner, because I thought that was gonna push people into murkier, scarier waters with AI content. You know, in the end, it wasn’t a surprise, it kind of came in right in the middle of just as we kind of thought. So really, to be honest, if you asked me to predict how the test results would go, they probably came in the way I thought they would, with all the things that we knew. But AI did produce a winner. The best quote I’ve heard out there is that companies that learn how to use AI to improve their processes are going to be companies that don’t learn how to use AI. And that’s probably the best takeaway.

Drew Neisser: When you’re looking at it. Even when you say create an outline for me, what you’re getting is, again, the obvious. So I think it’s really important. But stepping back from this test, what was clear to me is that content works, a lot of content over a short period of time on your website, well targeted with good headlines and good copy behind it will drive traffic to your website. And we’ll improve your rankings against the keywords that you care about. So that’s a big, yes, yay, go content, right. There are some little nuances in here in terms of hybrid where, what, how much hybrid, but there’s no doubt at least in the short term that human wins. So final thoughts and recommendations for folks who are listening in terms of their content development plans moving forward.

Michael Brenner: The content works, the frequency and consistency really do work. And as we stopped publishing in August 3, we did see traffic start to decline a little bit. So publishing more, and then stopping can hurt you a little bit. So consistency wins. The second thing is that to really understand what the audience was interested in, where we thought your website could rank, picking topics, is still an art more than a science that I think has to still be human-driven, we have not found any tools that can come close to finding the sweet spots and the opportunity gaps for companies. The third thing we didn’t really talk about, but we alluded to is this whole concept of authorship. Now all of the content that was published had my name on it. But we have heard from Google. And we knew over time, Google is saying that the fact that Michael Brenner has written 3000 articles about content marketing, or marketing strategy, or digital marketing trends, matters. When I go to write another piece of content, I’ve been talking to a lot of CMOs, about what I call employee activation, find the thought leaders and the experts in your organization, and help them build up the content they create the content they share. It’s more effective than when it comes from the brand. Your own employees have more connections than your brand does. They have the expertise and authority and trust that I think audiences are looking for. And so that would be my final thought is that all of the content we publish on your site had my name on it that I think was a factor if we had admin or by Renegade staff or something. I think it may have been slightly different results.

Drew Neisser: Interesting. Well, I hope that those listening right now and those that listen to this later get inspired by a couple of things. There was some fear because Google traffic is down because of Open AI and that content marketing would face a massive decline as well. And I think we’ve shown that content still works and it can work effectively. I think that this hybrid world is an area where it’s how much is too too much What are you gonna get into this copy? When is hybrid copyrightable? When isn’t? And I don’t know if we know the answers to any of that. But anyway, Michael really enjoyed this test spending time with you getting to know you and your organization. Last question, how can people find you and or engage with you?

Michael Brenner: Yeah, please, I would love it if you shared out the LinkedIn profile if people would be willing to connect. And if you have any specific questions for me, and you want to reach out, please do so through their marketing and socrative.com is where we publish our content not driven, not bought created. But if you want to subscribe or check us out over there, that’d be great, too.

Drew Neisser: Awesome. All right. Well, thank you, Michael. And thank you to all the Huddlers. If you’re a B2B CMO, and you want to hear more conversations like this one, find out if you qualify to join our community of sharing, caring, and daring CMOs at cmohuddles.com.
To hear more conversations like this one and submit your own questions while we’re live. Join us on the next CMO Huddles Studio. We stream to my LinkedIn profile, that’s Drew Neisser, every other week.

Show Credits
Renegade Marketers Unite is written and directed by Drew Neisser. Hey, that’s me! This show is produced by Melissa Caffrey, Laura Parkyn, and our B2B podcast partners Share Your Genius. The music is by the amazing Burns Twins and the intro Voice Over is Linda Cornelius. To find the transcripts of all episodes, suggest future guests, or learn more about B2B branding, CMO Huddles, or my CMO coaching service, check out renegade.com. I’m your host, Drew Neisser. And until next time, keep those Renegade thinking caps on and strong!