From A to B

Are synthetic users actually valuable? Or are they just another part of the AI hype cycle?

Nils 'Dolla Billz' Stotz joined this ep to flesh out his research into synthetic users, and I gave him significant pushback. We ended up agreeing on more than I was anticipating tbh, but he is clearly doing the research to see how synthetic users are valuable now! And he shared what he's currently doing today.

We got into:

- Exactly where synthetic users excel and are valuable, but where they hold significant limitations as well

- Holding synthetic users to the standards of humans (and where they are actually creating double standards)

- The issues with 'predicting' in CRO (and why are we too focused on it)

This is the post I referenced in the episode for the homie Craig Sullivan (who also deserves a follow): https://tinyurl.com/FromAtoBCraigSullivan

Go follow Nils on LinkedIn, his content is great and he posts basically every day.

https://www.linkedin.com/in/nilsstotz/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠⁠https://fromatob.beehiiv.com/

What do you .... actually do in your first 30 days as a CRO? Besides like health insurance and get a laptop, and fight with IT to get access to all the tools?

Gertrud “Valhalla” Anirudhan joined this ep to share her wisdom, AND journey, into this, as she recently joined a company and actually went through those 30 days. She's ex-Speero, so you know we talked mad shit about Ben Labay ;)

We got into:

- What you should aim to accomplish in 30 days, tactically (not just 'build a culture of experimentation)

- We debated a bit of 'tactics' vs. 'politics' you should balance in 30 days (I won obviously, but she made a strong case)

- She shared really cool stories of her experiences, one of which is ... making a case for copying competitors? (I totally agreed with her, which for y'all who know, I hate copying competitors.

Banger episode, even cooler human being. Go follow Gertrud on LinkedIn, she's amazing.

https://www.linkedin.com/in/gertrudvahtra/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

This is a bit of a unique episode - not your usual "how to build a culture of experimentation" type episode. And you shouldn't skip this episode because 'it's not experimentation so i don't care' - this episode is SUPER important for anyone in an individual contributor role to hear her wisdom.

Mallory "Take the Mantle" Randall is a seasoned veteran who's gone through all of the different stages in her career. And she's learned super valuable career growth things, moving from individual contributor, to managing a small team, to managing MANY teams as VP.

So many CROs and experimentation people fixate too much on their program, they don't realize there are people outside of the organization with different goals, trying to accomplish different things. This is why her POV is so important.

We got into:

- When you start your job, understanding your peers is just as important (if not more important) than actually getting your program off the ground

- Peers need you to push back - understanding HOW to push back, thoughtfully (and respectfully) is key. And picking and choosing your battles will help you grow in your career and get more things done

- There are limitations to 'being right' - just because you're right, doesn't mean people will listen to you.

Go follow Mallory on LinkedIn (we told her she's going to post more though leadership now, so stay tuned for amazing content coming from her):⁠

https://www.linkedin.com/in/mallorysiler/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

We are temporarily renaming this podcast From D to A - From Data to Action. Because data is almost useless unless you do something meaningful with it.

Josh "One of a Kind" Silverbauer hopped on the pod to dig deeper into the 'analytics' and 'data' part of product building. Identifying better ways to tell the 'data' story, and better ways to communicate that data so you actually can get shit done.

We got into:

- Josh's Pro Tip for consultants and agencies (it blew my mind, I'm serious)

- How 'data' isn't enough, and how presenting trust and authority is critical to how you communicate with data

- What data engineers critically need to do when communicating data

- His honest views on dashboards (is it beyond theater...?)

Go follow Josh on LinkedIn:⁠

https://www.linkedin.com/in/jsilverbauer/

AND HIS BAND....?!

https://open.spotify.com/artist/7ek3eQ8HOLZgtE8FgAy0G2

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

In this episode, Shiva Manjunath and Slobodan Manić go over the important lessons the Knicks (winning in 5) taught us about B2B.

It's a super serious episode and we go in depth with a LOT of important things you need to learn, and know, about B2B.

Are fake door tests ethical? We are super purposefully kinda misleading people to see if something is worth building.... (cue Wile E. Coyote reference).

Jon "Metal as Fuck" Crowder had a SUPER interesting post about the ethics of fake door testing, which I had to have him on the pod to discuss this topic. But we also got into a more broader conversation around ethics of A/B testing too!

We got into:

- Are fake door tests truly ethical? (Or legal...?)

- Should there be a 'board of ethics' for A/B testing? (yes. kinda)

- Jon's philosophy around CRO (and how it's influenced by his ethics)

Go follow Jon on LinkedIn:⁠

https://www.linkedin.com/in/jc-awip/

And check out his website: https://www.anotherwebispossible.co.uk/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Juliana "Run the Jewels" Jackson is someone who I consider a thought leader in the AI/digital marketing space, and I needed her POV on how to understand the impact of LLMs to user behavior.

So I had her on the pod. And she shared some insightful information into how LLMs actually work, how it's meaningfully (or maybe not so meaningfully) changing behavior, and what you should be doing about it.

(PS vote for me for a thought leadership award for Kameleoon https://tinyurl.com/voteforshiva)

We got into:

- How LLMs are evolving user behavior on your site, and what you really should be focusing on (spoiler: it's not what you think...)

- Are we putting too much confidence in AI? (and how do we right-size it)

- How LLMs are free research for you (and how to utilize it better)

Go follow Juliana on LinkedIn:⁠

https://www.linkedin.com/in/juliana-jackson/

And subscribe to her newsletter (it's amazing): https://julianajackson.substack.com/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Are manipulation and bribery the keys to building a culture of experimentation?

No.

(Well, maybe)

Jenni "Broadway" Bruckman (Director of Digital Product at YETI) joined this week to help broad the topic of building "cultures" which foster data-driven thinking and experimentation. It's different than focusing on explicitly building a culture of experimentation, in that your goal is to basically build a culture of fun, transparency, and engagement. The testing and data comes naturally from there!

We got into:

- Tips on navigating stakeholders (especially in a remote-first workplace)

- Better ways to run meetings and tell stories with data

-The value of your personal brand in scaling testing programs

Go follow Jenni on LinkedIn:⁠

https://www.linkedin.com/in/jennibruckman/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

A/B testing tools - all they do is randomize traffic. So why are we spending thousands of dollars for it?

Jonas "Elvis" Alves (of AB Smartly) joined the pod to discuss some of the major challenges in building A/B testing tools. It kinda started in a way I had no idea it would - that you're not buying a testing tool, but buying a... trust platform?

Jonas throws some shade at other testing tools, but I bleep it out, don't worry ;) . (And thankfully, he doesn't sales pitch at all. )

We got into:

- Jonas makes the case for BOTH building in-house and buying an external tool

- How to earn back trust once it's broken, through data or experimentation

- Booking.com ran A/B tests on A/B testers (it's super meta and super cool)

Go follow Jonas on LinkedIn:⁠

https://www.linkedin.com/in/jonasalves/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Is 'ethics' something we've undervaluing when it comes to experimentation? Should we actually care about this nerd stuff of 'ethics'?!

Well, yes. Obviously. AJ "Does No Wrong" Long joined Shiva to chat about this specifically, as someone who works in the fantasy sports realm of things, ethics can be a contentious subject.

We got into:

- Why "users first" is really the only way to foolproof ethnical experimentation (and low-key, the only great way to run a business long term)

- How PrizePicks fosters 'responsibility' in fantasy sports (really cool things they're doing there)

- How to set (and enforce) guardrails in programs. Which is stupidly important in experimentation, when people want to break it down and hack it as much as possible

Go follow AJ on LinkedIn:⁠

https://www.linkedin.com/in/angusjlong/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Emotion in data analysis, from what we've been told, should never happen. But the kicker - humans are humans. We use emotions in literally everything we do. So how do we channel it best to make sure we are positively impacting the business?

Gerda "The koala" Thomas joined Shiva to chat about this topic. Where does it make sense to channel emotion? In getting EXCITEMENT about testing, and ideas! But maybe we park it when actually analyzing tests, because that can lead to bad analysis.

We got into:

- The good (and bad) about data. Because sometimes a hyper-fixation on certain data leads to dark patterns and bad UX

- Emotion can actually be GOOD in building a culture of experimentation (well, some emotions....sometimes)

- Is 'hiring culture' really THAT important to getting people involved in data? (yes....yes it is)

Go follow Gerda on LinkedIn:⁠

https://www.linkedin.com/in/gerdathomas/

Go check out her company, Koalatative:

https://koalatative.com/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Slobodan Manić and Shiva Manjunath join another episode of #CRO after dark to discuss their trends in 2026, what will die, what will live, and how Better Call Saul is definitely better than Breaking Bad

Go check out his second best podcast No Hacks (https://www.nohackspod.com/)

Don't know what the hell CRO and experimentation is? Trying to understand the basic concepts and process?

We gotchu. Rommil "The Thrill" Santiago and Shiva Manjunath went into all the common topics, and we can even help you explain this to others who might not fully understand what you do in CRO.

Please passive aggressively share this episode to people who don't understand CRO. I will fall on my sword for you.

We got into:

- Why CRO is seen as a Hail Mary (and why it SHOULDN'T be....)

- CROs shouldn't exist. Because everyone should be doing CRO in some way

- What the hell MDE really even is? And do you even need it? (Yes....yes you do)

Go follow Rommil on LinkedIn:⁠

https://www.linkedin.com/in/rommil/

Go check out his book:

https://www.experimentnation.com/prove-it-or-lose-it

Sign up for Experiment Nation (I'm there... he's there too I guess):

https://www.experimentnation.com/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Have you ever struggled to get buy in from senior leadership? Never in your entire life, right?

Well I've faced it. A lot. And I wanted to get in the mind of a super seasoned CMO to truly understand what makes them tick. What they care about. What they DON'T care about.

Who better than Kelly Hopping, CMO at Demandbase. And y'all, this one is CHOCK FULL of amazing insights from her to better understand how to communicate what you're doing to senior leadership.

This was the first episode I listened to twice, just to make sure I got everything down.

We got into:

- How to translate OKRs into what you do in experimentation / product (to help you actually get buy in)

- What CMOs (and senior leadership) ACTUALLY cares about (and what you're messaging to them is irrelevant to them)

- AMAZING career advice from her on how to pivot in your career appropriately

- Why leadership challenges A/B testing

Go follow Kelly on LinkedIn:⁠

https://www.linkedin.com/in/kellyhopping/

Go check out her podcast "NextGen CMO":

https://www.demandbase.com/podcasts/nexgen-cmo/

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

Finding clients is hard enough - finding GOOD ones? Feels damn near impossible.

I enlisted the help of Tas Bober to help navigate this complex field of getting client work, and explored topics related to CRO/client work as well!

We got into:

- Bringing confidence to the client relationship (you don’t always have to cave in to EVERY little freaking thing they want you to do)

- Running from bad clients when they don’t pass the vibe check (and how to find shitty clients BEFORE they pay you)

- How to talk to clients (and stakeholders) appropriately without talking DOWN at them (hint: treat them like they’re children. It works.)

Timestamps:

00:00 Episode Start

4:12 Detecting Client Red Flags BEFORE Signing A Contract With Them

8:46 What To Do When A Client Gives You The ‘Ick’

16:07 Importance of Setting (And Enforcing) Boundaries

20:30 Setting Success Metrics for Engagement Upfront

29:46 When “Leading With Data” Doesn’t Make Make Sense…?

46:20 Tas’ BIGGEST Pro Tip (Don’t Just Skip To This Timestamp Ya Leetches)

Go follow Tas Bober on LinkedIn:⁠

https://www.linkedin.com/in/tasbober/ ⁠

Go check out her podcast "Notorious B2B":

https://www.linkedin.com/company/notorious-b2b/posts/⁠

Also go follow Shiva Manjunath on LinkedIn:

⁠⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠⁠https://fromatob.beehiiv.com/

People are finding websites more and more manipulative. And we don't mean this word lightly.

Why is this the case? In large part, it's because of the shitty tactics people are using on sites. But really, it's because websites care far more about short term sales and metrics instead of focusing on LTV. And that leads to hacks, not long term gains.

Made With Intent released a REALLY great report which went into detail on this, and I had the man, myth, and legend David "Zedd" Mannheim on the pod to chop it up to discuss that report (among other things).

We got into:

- Why People Feel Like Websites are Manipulative

- How Popular 'Best Practice' Tactics Are Probably Hurting You Way More Than Helping

- Tips and Tricks on Maximizing Your Popups (WITH DATA)

Timestamps:

00:00 Episode Start

5:05 People Feel Websites Are Too Manipulative

11:12 Websites Seem to Prefer Sales Over CX

14:40 What Are Pop Ups Getting Wrong

19:02 Marketers Are More Focused On Numbers Than Humans

26:56 Does Social Proof Still Work In 2025

31:18 Scarcity Messaging

35:55 Discounting Isn’t A Strategy

42:02 Segmentation Is One Of The Most Important Things To Focus On

Go follow David Mannheim on LinkedIn:

https://www.linkedin.com/in/davidleemannheim/

Download the full report here (it's a REALLY good read):

https://www.madewithintent.ai/the-intent-gap-report

Also go follow Shiva Manjunath on LinkedIn: ⁠⁠https://www.linkedin.com/in/shiva-manjunath/⁠⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

⁠https://fromatob.beehiiv.com/

We back out here with the amazing Emma Travis to dig into 'quality vs. quantity' when it comes to experimentation programs. The conversation Ben Labay keeps trying to force, but a bit more nuance to it.

There is a time and place for quality. But there is a time and place for velocity and scaling your program up in terms of number of tests run. How do you know when to do it? And what are triggers to get you .... ahem.... From A to B.

Emma "Biscuits and Gravy" Travis joined me on this episode to go DEEP into this topic.

Emma was sick and still trooped through this episode - give her all the flowers!!!

We got into:

- When to focus on velocity vs. when to focus on ‘quality’ of experiment

- Tips on auditing your program to know if you’re too focused on quality vs. quantity

- Smart ways to focus on research when resources are limited

Timestamps:

00:00 Episode Start

3:57 Is Velocity Appropriate In Tracking Metric For Experimentation Maturity?

9:23 Velocity Increase Doesn't Always Mean A Decline In Quality

11:51 Experimentation Velocity Is Like Flying A Plane

19:19 When To Scale Velocity Vs. When To Focus On Quality

24:33 Run A Heuristic Audit Of Your Cro Program (*Bonus Points For Doing It In Miro)

28:00 What Defines "Quality" In A Test?

36:35 How Do You Communicate A Decline In Velocity Knowing You're Increasing Quality?

40:35 Good Ways To Use Ai To Help You Speed Up Research

Go follow Emma Travis on LinkedIn: https://www.linkedin.com/in/emma-travis/

Also go follow Shiva Manjunath on LinkedIn: ⁠https://www.linkedin.com/in/shiva-manjunath/⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

https://fromatob.beehiiv.com/

How many of you have seen a website teardown and thought “Dude you literally have 0 data, this is just a vibes analysis?”

I have that thought regularly. And it got me thinking about heuristic audits as a whole, and trying to understand what makes a good heuristic audit good, and a bad heuristic audit bad. I enlisted the help of my mortal enemy Rishi Rawat to explore this.

Vibes were immaculate, and he actually said some smart things in this episode (shocker - I know).

We got into:

- The critical missing ingredients to take your heuristic from ‘shit’ to not only trustworthy, BUT valuable

- Where many heuristic audits fail to be meaningfully impactful (despite being positioned as such)

- An exploration of “expert” vs “non-expert” heuristic audits (and why both are incredibly valuable)

Timestamps:

00:00 Episode Start

2:08 What Is a Heuristic Audit (With Help from Ryan Thomas)

8:53 Good Heuristic Audits Are Systematic in Nature

12:22 The "Goal" of the Heuristic Audit May Be More Important Than the Process

16:20 The Importance Of Data to Informing The Heuristic Audit

22:01 Rishi Actually Says Something Smart

27:07 Heuristic Audits Should Highlight BOTH The Good AND The Bad

35:29 Many Heuristic Audits Don't Simulate Emotion (Yet They Should....)

Go follow Rishi Rawat on LinkedIn: https://www.linkedin.com/in/rawatrishi/

Go check out the homie Ryan Thomas:

https://www.linkedin.com/in/ryancharlesthomas/

Also go follow Shiva Manjunath on LinkedIn: ⁠https://www.linkedin.com/in/shiva-manjunath/⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

https://fromatob.beehiiv.com/

And go get your free ticket for the Women in Experimentation - you might even be entered to win some From A to B merch! : https://tinyurl.com/FromAtoB-WIE

Problematic stakeholders could be convinced if they... just got data. Right?

Nope. Sometimes it's deeper than that. You gotta understand what makes them tick. Is it data? Is it optics? Maybe you should be running heuristic audits on your stakeholders instead of your website...

In the second time EVER, someone else has changed my mind on something. Leave it to "Kingpin" Finn McKenty, who's apparently gone from Punk Rock MBA to Finn McKenty PhD, to change my mind.

We got into:

- How YouTubers are more data driven than your own CEO (lol)

- Why you should be running heuristic audits not JUST on your website, but on the stakeholders you interact with (and tips to do so effectively)

- Finn gives some general life advice on learning to let go (important when many product and CROs don't have autonomy to actually impact anything)

Timestamps:

00:00 Episode Start

2:46 The analogy of CRO and "Gym" goes so deep

6:25 Even YouTubers are data driven

11:20 People who don’t buy into “experimentation” just optimize for different metrics than you

14:28 Psychology of UXers vs. Product/CRO (Finn low key is a psychologist now)

20:01 Running heuristic audits on… stakeholders? (yes - it’s a good idea)

25:07 Optimization sometimes means optimizing for ‘helping people’ (not metrics)

30:16 Sometimes, CROs gotta play the politics game

35:13 Finn offers sage advice in learning how to let go (CROs need to hear this)

49:06 Preach: Samuele Mazzanti

Go follow Finn McKenty on LinkedIn: https://www.linkedin.com/in/finnmckenty/

And go subscribe to his newsletter:

https://finnmckenty.beehiiv.com/

Go check out Samuele Mazzanti's post too:

https://tinyurl.com/FromAtoB-Samuele

Also go follow Shiva Manjunath on LinkedIn: ⁠https://www.linkedin.com/in/shiva-manjunath/⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

https://fromatob.beehiiv.com/

And go get your free ticket for the Women in Experimentation - you might even be entered to win some From A to B merch! : https://tinyurl.com/FromAtoB-WIE

Do you REALLY know what cookies are? Like really, REALLY know? What about GDPR? What about PII?

I know the words. But what do they REALLY mean? I enlisted the help of Eddie "The Techie" Aguilar to help me simplify some of these complex topics, and help me create meaningful next steps on how to address PII concerns and other marketing-related issues in data collection.

We got into:

- Simplified definitions of cookies, data collection, GDPR, etc. (I'm stupid and like hearing things simplified from smart people)

- First vs. Third part cookies (and what it means to your marketing program)

- A/B testing and the importance of NOT collecting PII in your testing tools

Timestamps:

00:00 Episode Start

2:31 What is a Cookie?

7:41 How Cookies Have Been Used Maliciously (Lack of Consent)

9:51 First Party vs. Third Party Data

13:11 Opting Out of Cookies (Explained)

14:45 GDPR

28:20 A/B Testing and Cookies

37:30 PII and A/B testing

Go follow Eddie Aguilar on LinkedIn: https://www.linkedin.com/in/whoiseddie/

Also go follow Shiva Manjunath on LinkedIn: ⁠https://www.linkedin.com/in/shiva-manjunath/⁠

Subscribe to our newsletter for more memes, clips, and awesome content!

https://fromatob.beehiiv.com/

And go get your free ticket for the Women in Experimentation - you might even be entered to win some From A to B merch! : https://tinyurl.com/FromAtoB-WIE