While still in Beta, it looked promising, so I thought I’d give it a go. Although, I have been accused of actually “trying to break it”, which isn’t necessarily far from the truth. So, what’s the verdict?
First of all, it clearly states “Your chat activity may be used to improve the product …”, so I stayed clear of using our clients’ GA4 properties and only tested this for one of our own. Unfortunately, there doesn’t seem to be a way to opt out of this feature for those who are concerned about the use of their data.
Now would actually be a good time (if you haven’t done so already) to check that personal data in GA4 has been redacted, which can easily be done in the admin section.
Although this was never going to be my main use case I started off asking some simple yet specific performance-based questions. The results were a mixed bag, but I expected that.
What I liked was that Analytics Assistant shows you the reasoning steps which gives you transparency into the logic behind each answer and helps you spot mistakes or faulty assumptions, which in turn can help you write better prompts moving forward.
For some reason the AI chatbot really struggled to return any event-based results e.g. based on Default Channel Grouping despite me explicitly asking for it. This seems strange considering that the default attribution model is DDA and as the name “Default Channel Grouping” suggests this should be the default. Results based on First User Default Channel Grouping or Session Default Channel Grouping were returned without any issues.
Although I assume the main purpose of the tool is to answer natural language questions by providing numbers, tables and quick visualisations, this wasn’t what I wanted to use it for.
I was hoping it could guide users step-by-step on how to build their own explorations in an effort to squash an otherwise steep learning curve for new or less confident and experienced users.
To put this to the test I explicitly included in my prompt to provide me with step-by-step instructions on how to build an exploration to answer a specific question.
For some simple questions I received some reasonable instructions, but those were typically at a level that most users were already familiar with and could build without help.
It soon however became apparent that the Analytics Advisor is applying synonyms to naming conventions that should not be changed such as buttons in the UI for example the instructions told me to click “Import” or “Add” when the button in the interface is in fact called “Confirm”.
I’m also not sure if the tool is trained on old documentation or if this again is due to using synonyms but it told me to add “Session conversion rate” despite the fact that the metric is called “Session key event rate”.
I experienced even bigger issues when it came to applying filters.
In one scenario the step-by-step instructions told me in step 1 to select certain dimensions and metrics and then in step 4 to apply a certain dimension as a filter, however this wasn’t a dimension I was asked to add in step 1.
In another case I asked “I need to compare the Total Revenue and Key Event Rate for users coming from Email and Organic Social channels, segmented by Device Category (Mobile vs. Desktop). I want to see this data for the last 60 days to evaluate the profitability of these channels. Please provide step-by-step instructions on how to build a free-form Exploration for this.” The majority of the instructions the Analytics Advisor returned were quite helpful. However, to filter by the two channels I asked about the AI chatbot told me to apply an OR filter. This is not something that’s possible in GA4. When I fed that back, the tool suggested I use the “matches one of” filter option, which also doesn’t exist in GA4. I was expecting the Advisor to supply me with a RegEx I could copy and paste as a filter, but instead it provided me with instructions on how to create a segment. While this solved the problem, it required several extra steps which could be quite frustrating for inexperienced users.
Those tests (+ the concerns about data security) have shown that GA4 Analytics Advisor isn’t quite fit for purpose yet and that it needs to be trained on GA4’s naming conventions and functionality before anything else.
My recommendations moving forward are:
- Keep an eye on official announcements from Google, especially regarding clarity around data security and privacy and how the model is trained before using this for sensitive or competitive data.
- Test and review this regularly to see if the model has improved.
- Thoroughly check every output.
- Establish best practice by developing lists of reliable and tested prompts.
- Provide Google with feedback but don’t share confidential information.
So really, just like humans, GA4 Analytics Advisor needs to learn about the GA4 interface, terminology and functionality. No shortcuts here.