It’s that time of year again, Cyber Week 2017. Black Friday and Cyber Monday are a few days away and Retailers have been scrambling for months to make sure everything goes right this week. We will be providing this live blog like we did in 2016, 2015 and 2014. Through the course of this week we will be looking at which retailers are providing the best performance, some of the techniques and best practices they use and the challenges they face.
Table of Contents:
- Cyber Week Kickoff: Monday, Nov 20th, 4:30pm EST
- Pre Black Friday Carnage: Tuesday, Nov 21st, 9:45am EST
- Visually Complete, What You Need to Know: Tuesday, Nov 21st, 8:00pm EST
- Conversion is the Name of the Game: Wednesday, Nov 22nd, 10:00am EST
- Performance Metrics, Which Ones are You Using?: Wednesday, Nov 22nd, 6:00pm EST
- You Have to be Monitoring from an End User Perspective: Thursday, Nov 23rd, 10:30am EST
- Understanding How Performance and Errors Impact Bounce Rates: Thursday, Nov 23rd, 12:00am EST
Understanding How Performance and Errors Impact Bounce Rates: Thursday, Nov 23rd, 12:00am EST
Here is our first comparison for Cyberweek 2017. Apple, Neiman Marcus and Amazon are providing some of the fastest Visually Complete times for shoppers. When compared to some of the slower retailers these fastest retailers are offering more than 4X faster experience for shoppers.
Let’s have a look at some real user traffic data from a major retailer. In this example, we see that performance across the US is consistent for the past 6 hours. Unfortunately, it is currently consistently poor according to the APDEX score.
Now for this retailer this is actually good ”-ish” news (I am a glass half full guy) because if we look at their APDEXfor their entry actions (where shoppers enter the site) from the previous week (below) we can see it improved from unacceptable to poor over the past 6 hours.
We can see that the past week compared to the previous week saw other improvements. The week before last there was a major issue with JavaScript errors. JavaScript errors occur in the browser can be just as impactful as server side errors to shoppers.
For example, when we look at these errors we can see a direct causation that users were “Bouncing” off the site when they encountered these JavaScript errors. We know this because Dynatrace sees the user behavior for each shoppers user session.
JavaScript errors impact bounce rate, but there are other things which can impact bounce rates. Below is a view showing a histogram comparing the performance impact on the bounce rate for this week to the previous week. We can see in this example that things are headed in the right direction with lower bounce rates as performance improves.
We will provide some more comparisons and real user examples over the next few days.
You Have to be Monitoring from an End User Perspective: Thursday, Nov 23rd, 10:30am EST
Happy Thanksgiving morning everyone. This is an interesting one. Here we see a retailer having ongoing issues and a different issue at the same time of day for the past week.
When we analyze this, we see that during the daily issues there are an excessive number of redirects which are happening (below). This could be caused by a misconfigured network device or an issue with the Content Delivery Network (CDN).
This is a real world issue that is currently impacting shoppers trying to use this retail site. This illustrates that when things go wrong there can be multiple unrelated event some of which might be impacting end users, and others which are not.
That’s where automatically analyzing your traffic using Artificial Intelligence can quickly help you determine which problem needs to be addressed first. It can tell you how many users are being impacted. This allows retailers to address the issues that are most problematic.
Performance Metrics, Which Ones are You Using?: Wednesday, Nov 22nd, 6:00pm EST
As everyone checks out of work and starts their holiday travels let’s have a look at some web performance metrics we will be tracking this Black Friday and Cyber Monday. I am often asked what is the industry standard for web performance metrics? I used to answer this that the wonderful thing about standards in the IT industry is that there are so many to choose from. For years, we have searched for the right metrics to best monitor user experience.
User experience is driven by a combination of design, architecture and delivery. Retailers, regardless of their size, sales or market cap all play on the web on an equal footing. How a retailer chooses to implement their design, architecture and delivery is a choice that they make. I highlight this as these are things retailers can control and that is why we provide these benchmark comparisons.
For example, below is a snapshot of some live data from the last five days for a retailer who has completely optimized their user experience.
The metrics I am looking at are things like:
W3C Request/First Byte time, this is a measure of how fast the servers can respond to a shoppers request. Optimizing this measure not only provides the end user with a faster experience it means that this retailer can actually scale to support more shoppers with less cost because transaction are running faster, consuming fewer server-side resources (like CPU and memory).
First Paint time, this is a measure telling us when something is being painted to the screen. This helps because shoppers want to perceive activity immediately… waiting with a blank screen leads to frustration which can lead to increased rates of bounce (abandonment).
Dom Interactive time, this is a measure which approximated when a user can start interacting with a page. Not being able to click on an item as the page loads leads to shopper frustration which can lead to undesired user behavior.
Speed Index, this is a newer measure which tells us how fast it takes it to load the majority of the viewable window. Shoppers want to see activity when they load a page, the faster this metric is the better the user experience will be from a performance perspective.
Visually Complete, I’ve talked about this at length in this blog, this is the best metric available which gives retailers an understanding as to what shoppers are experiencing when they load pages.
Dom Complete time, this measure tells a retailer when all the dynamic logic for that page has loaded for a shopper. If the retailer is using lots of client side logic through Java Script then they need to pay attention to this metric.
Page Load time, this is when the shopper’s browser/client considers the page to have loaded. This is how most organizations have historically gauged web performance.
Response time, this is the complete “over the wire” end to end response time including any asynchronous activity and beacons which might called. This measure is useful for retailers operations teams to understand if specific IT assets are performing and meeting SLAs (Service Level Agreements).
Below is an example showing the same metrics but for a retailer exhibiting slower performance. To put things in perspective, the fast retailers entire Response time (meaning absolutely everything) is done loading before anything even paints to the screen for the slow retailer.
Don’t believe me? Don’t think that online retail is competitive when it comes to web performance? Have a look at this chart below showing the response times for different retailers. Given a preference which site would you want to visit, a slower or a faster site?
We will start with some more detailed comparisons tomorrow leading into Black Friday. For now, safe travels and Happy Thanksgiving!
Conversion is the Name of the Game: Wednesday, Nov 22nd, 10:00am EST
Over the next few days what every Retailer is looking to do is “convert” shoppers into customers. Tracking conversion is critical for Retailers to understand their business success. Typically, this is done through online Analytics tools like Google Analytics or Omniture. The challenge with these tools is that they are often page focused and the data which they provide is rarely real time… unless you pay extra for it.
Modern applications, specifically what are known as single page applications can introduce challenges in defining meaningful conversion goals. Below is an example of how Dynatrace can define conversion goals on any user action, including XHR actions from click events, like “Click on Proceed to checkout“.This is a much more powerful and easy way to define very granular conversion goals.
Dynatrace allows retailers to capture individual user sessions. It can uniquely identify which retail applications a shopper visits, where they are visiting from and what type of device they are using. Tracking a shopper moving from a mobile device to a desktop to complete a purchase allows retailers to provide more targeted offerings. Below is an view showing how Dynatrace would capture a unique shopper visiting Amazon.com.
Dynatrace enables a retailer to see the details of an individual session including things like their screen size (which, by the way, can impact end user experience which is why retailer should capture Visually Complete times), Device details, ISP and the number page and xhr actions which are observed during the shoppers visit.
Below we can see each specific user action that a shopper does during their visit to the retailers site. You can see that the XHR action “Click on Proceed to checkout” was marked as a conversion goal within this shopper’s visit.
Since the conversion goal is a critical step for a retailer to track, below is a view from Dynatrace showing how that XHR request was executed. Unlike web analytic tools, Dynatrace can automatically trace this transaction all the way back to and through the server side including back into containers and the cloud.
Dynatrace can provide this level of detail on every user session. When a Retailer applies this to thousands or tens of thousands of user sessions they will need help to make the most of this data. That’s where Artificial Intelligence comes in. It automatically analyzes the user sessions to determine if a performance event is impacting shoppers. Below is a view showing an interactive infographic which is determining that shopper from a specific region using Safari mobile are being impacted by an increase in Java Script errors.
Dynatrace analyzes those user sessions to see if the performance problem impacting shopper’s behavior. If it is increasing the bounce rate, specific entry/exit actions and of course conversion goals.
Overall conversion goals are automatically tracked and baselined by Dynatrace. This allows you to compare your current conversion results to historical results.
And of course specific conversion goals can be managed individually.
Retailers need to understand how user experience and performance impacts user behavior and conversion. Retailers can look at undertaking time consuming ELK (Elastic, Logstash and Kibana) projects to marry disparate data sets from their performance tooling and analytics tooling, but this always proves difficult and expensive. Dynatrace provides an all in one solution to do this automatically for retailers.
Visually Complete, What You Need to Know: Tuesday, Nov 21st, 8:00pm EST
I mentioned that this year we will be doing more comparisons based on a new metric called Visually Complete. This is important because it is a better representation of what shopper experience when the view retail web pages. Below is a visualization showing how Visually Complete uses an algorithm to identify when the browser finishes displaying the viewable portion of the screen. This is the “above the fold” view, there may be content which is being loaded “below the fold” that a shopper would scroll down to see, but that is not included in the Visually Complete time.
Dynatrace uses this in both our Synthetic and Real User measurements. Below is a view of a synthetic measurement in Dynatrace showing the visible screen which was measured.
When we look at the details of how the page loads, Dynatrace analyzes a rich set of metrics based on network observations and client side observations. These navigation timing milestones are shown in the context of how the page loads. Depending on how the retail site has been architected these milestones can be faster or slower from one retailer to another.
These are decisions that retailers have to make. Do we want to focus on displaying ads first or do we want the user to be able to interact with options on the page before certain ads or images have loaded? These milestones tell us when items are first visible, when the user can interact with page, when the page appears to be complete and of course when the page is fully loaded.
Some Retailer’s, like Apple have realized that Visually Complete is a metric that needs to be managed. Below we can see how someone at Apple decided that they needed to optimize their site before Black Friday and Cyber Monday to make sure that all customers are getting a consistent and quick Visually Complete experience. We can see from the data below that early in the morning of the 17th Apple pushed a build out that dramatically improved the consistency of their page as measured by our Visually Complete metric.
Visually Complete is gaining traction in the Retail Industry, if you are not managing to it now, you likely will be soon.
Pre Black Friday Carnage: Tuesday, Nov 21st, 9:45am EST
It is starting off as an interesting Cyber Week. Again, it is better to find this out now than on Black Friday or Cyber Monday. Below is a live view of retail transaction which is now failing.
It looks like this retailer has tried to push several builds out into production unsuccessfully in the past 24 hours.
A quick analysis (below) shows that the issue is not a third party or network problem. The issue is not isolated to a particular geography or device type, it is pretty wide spread. One interesting thing to note, while the response time looks like it is improving, this is actually a false indicator. In this case as the Retailer pushes out an automated error message the measured response changes as the error page loads faster than the normal page.
Looking at the details, we see a 503 error. This is a server side error which tell us that something on the backend is causing an issue.
Having shoppers receive a 503 error is clearly a bad thing, but having that message show up for hours is a REALLY bad thing.
I’ve talked in previous years (see last year’s blog) about how complex these retail applications have become. It has gotten to the point that these applications cannot be managed unassisted by humans. We need help. That is where Artificial Intelligence comes. Retailers need to embrace AI to automatically uncover issues like this, and automatically diagnose the root cause of a problem.
Above is a view from Dynatrace showing a problem which is impacting customers. Dynatrace’s AI shows how many customers and which applications are being impacted by a problem and what the root cause of the problem is. Fixing an issue like what we saw from that retailer would take minutes instead of hours.
We will continue to post findings as we see them, but this Retailer’s problem are not unique we are seeing multiple retailers slipping days before Black Friday and Cyber Monday.
Cyber Week Kickoff: Monday, Nov 20th, 4:30pm EST
It’s that time of year again, Cyber Week 2017. Black Friday and Cyber Monday are a few days away and Retailers have been scrambling for months to make sure everything goes right this week. We will be providing this live blog like we did in 2016, 2015 and 2014. Through the course of this week we will be looking at which retailers are providing the best performance, some of the techniques and best practices they use and the challenges they face.
This year I am super excited to be doing some comparison based on new metrics like Visually Complete.
Visually Complete
Businesses and IT need reliable benchmarks to know what a good customer experience looks like. Until now, it’s been difficult to see and capture the visual experience of a real user. With Visually complete and Speed index, Dynatrace provides metrics that are as close to a real user experience as it can get.
Traditional website performance optimization metrics are based on measured time, number of requests and kilobytes. While these numbers are important, they do not reflect on when users are able to see the information they are looking for. Users care more about the time it takes a website to become visually loaded and ready to use.
Visually complete shows you the exact time your real users spent waiting for above-the-fold content to load completely – for each and every location, device, operating system and browser type. It can also be combined with powerful waterfall charts to understand and tune the rendering of a web page. Dynatrace is the only solution that provides these consistent metrics for real user monitoring as well as synthetic tests.
It will be interesting to see how Retailers compare using this new metric.
To set the stage for this week’s coverage, let’s start with a view of a major retail event that happened before Cyber Week.
Singles Day 2017
Singles Day (11/11/17) is now the biggest single day shopping event of the year. According to Forbes, this year’s Singles Day took in $17.8 billion in gross sales. That’s $740 million per hour or $12.3 million per minute, and yup that’s $205k per second. I have to take a pause and say… wow. That’s a lot of money, and that is a lot of sales at risk for Retailers if they don’t get it right.
Below is a view of the digital performance from some of the main Retailer’s in China. Note how one of the retailers had some very poor performance During singles Day (yellow line), which was likely due to an issue on the server side.
The view on the top left tile shows the Visually Complete times for each of the retailers leading into, during and after Singles Day. The average Visually Complete time for China Retailers is 8.92 seconds. The tile on bottom left is Server Time. We measure this through the time to first byte (which tells us how long it takes the server to respond to a request) on the root or base page. The industry average for China retailers is 257 milliseconds on the sever side.
What we see is that Retailers in China are focusing a lot of effort on optimizing the server side of the equation, but are they doing this at the expense of overall digital experience? With an average Visually Complete time of over 8 seconds, shoppers in China are getting some of the worse digital experience when compared to other markets.
I see this as an opportunity for retailers in China. Future Singles Day will see increasing competition and the main way Retailers can differentiate themselves will be by improving digital experience.
Cyber Week 2017 Kickoff
When it comes to Cyber Week, it is way better to fail early than to fail later during the week. Here is an example of a retailer that clearly was looking to push a new build out ahead of Cyber Week, only to have to pull that build back relatively quickly. Below we can see around 4am this retailer pushed a new build for Cyber Week into production. Immediately Dynatrace saw issues from an end user perspective that was impacting a typical retail transaction flow.
When we look at the details around this failure we see Server side response times slowing down and throwing errors. Modern web applications are exceedingly complex and any number of things can cause an issue which can impact end users. It is better to find out issues like this well ahead of time, however sometimes issues can occur at most inopportune times. The faster you can find and fix these issues, the less the impact will be on your bottom line.
This year we have new metrics and will of course be using more Dynatrace AI (Artificial Intelligence) to monitor key retailers.
We will be here all week, check back often to see updates on 2017 retailer web performance. Also feel free to follow my updates on twitter @davidlewsjones or on Linkedin
The post 2017 Cyber Week Coverage / Black Friday and Cyber Monday Retail Digital Performance appeared first on Dynatrace blog – monitoring redefined.