Surveying the Bot Manipulation Landscape Among top US Retailers

Written By International Persuasion Machines

Online retail is one of the largest targets in the cyber-security landscape. The identity theft resource center has logged over 10,000 data breaches, many of them against retailers. Being large stores of individual consumer data, containing potentially sensitive or otherwise valuable demographic and financial data, retailers are already known to be valuable for data exfiltration.

Just one of many examples of tools available to people to exploit popular retailers with automated tools

Data scrapers, account takeovers, card swipers, and other malicious behaviors are often pointed towards retail sites. Evidence for these behaviors abound in news articles covering breaches and all over the margins of the internet. At IPM, the root of our work is grounded in the tools used by these would-be attackers. As a sociotechnical security firm, we identify and validate solutions that are aimed at defeating these types of attackers. If attackers can mimic an individual consumer at scale, with targeted behaviors, they can manipulate markets, steal promo codes, and exploit KYC loopholes to get infinite brownies. We help people verifiably stop these attacks.

Newsweek’s 2022 Best Retailers list is invaluable as a single roadmap for the highest profile retailers, and thus, likely the largest targets for malicious actors. How do these retailers do today in proactively mitigating these actors? We probed the 193 distinct URLs from this list of retailers for their ability to proactively mitigate bot traffic with some form of anti-bot defenses. In total, we delivered 8,492 probes to these the retailers across our 44 bot deployment interfaces, testing whether or not each retailer would determine that our bot traffic was indeed bot traffic.

To make the determination that the retailer determined we were sending bot traffic, we developed an internal tool that interrogates the web response any bot receives to decide whether or not that web response is an anti-bot warning page. We developed a statistical signature of these pages with a machine learning algorithm, cross-validated the model, and determined our tool was ≈96% accurate when labeling these pages.


We used our internal web application firewall / anti-bot machine learning detection algorithm to automatically assess the 8,492 probes we sent to the 193 retailers across our 44 bot deployments. Of the 8,492 probes we sent to the 193 URLs, 61% of our probes successfully reached their targets. While that doesn’t sound like a high percent, we intentionally design bots across the spectrum of sophistication, from basic first-pass direct HTTP requests out to cutting edge bot browsers.

Browser Attack Performance by Browser/Network interface

Bot Deployment Attack Performance Results - Note that most retailers block basic HTTP fairly well, as well as to a lesser extent stock selenium instances - more sophisticated attacks eventually outstrip the capacity of most mitigation techniques.

In the best case, our least-sophisticated bot deployment was successfully mitigated by retailers 91% of the time. In the worst case, our most sophisticated bot deployment penetrated 88% of the retailers. No single bot we deployed worked perfectly against every retailer. Because IPM tests a huge array of bot deployments, however, IPM was able to identify at least one bot deployment that could penetrate every single retailer. In other words, for each of the 193 retailers, at least one of our 44 bot deployment strategies was able to bypass all security measures and access pages intended for human consumers.


Beyond just determining that no retailer can evade all of our bot deployments, we wanted to understand what this data tells us about the latent, nascent industry of defending systems from malicious bots. Given these 193 sites, representing many billions of dollars of revenue, across many retail verticals, can we learn anything meaningful about who mounts strong defenses against malicious bots?

Initially, our hypotheses were relatively straightforward - more high profile companies would mitigate such attacks - perhaps some verticals of industries would deal with these issues better than others (e.g. Nike and Adidas have to contend with people attempting to mass-purchase their shoes as this tweet wonderfully illustrates). In the Newsweek article we sourced these retailers from, there was an associated index score for each retailer which was a composite of various factors of how well the company was regarded - do we see any clear associations with that variable?

In our data analysis, we found no discernible relationships between bot mitigation effectiveness and various measures of company health or retail vertical. Initially, we were disappointed to find that every chart we generated, for every regression we tested, we couldn’t find any compelling associations that told any clear story about which companies excel at bot mitigation and which don’t.

Scatter chart of relationship between percent of tests that were successful and index score of company rating from Newsweek article

Each observation is a company from the Newsweek Best Retailers 2022 article. The X-Axis is the index score that Newsweek compiled for how well-rated the retailer is according to consumers. The Y-Axis is the percent of IPM tests that were successful. The size is the rank of the retailer within each category. Each color represents a distinct top-level retail category (e.g. Food, Clothing, Home Goods).

After some consideration, there’s a reasonable explanation for why there’s seemingly no relationship between how valued a company is and how comprehensive their bot security stance is — retailers are dealing with sociotechnical security issues in a piecemeal fashion. What we see in this chart is evidence of every company idiosyncratically solving the problem according to their own experiences with the problem measured against their risk appetite for the problem. If there were a strong relationship in the above chart, then we’d be able to say that as companies become more sophisticated, they get better at dealing with this problem.

That we don’t see that relationship indicates that companies don’t deal with this in a linear way - it’s not as if companies get better at these things as a matter of course while becoming bigger and better retailers. In reality, they just deal with it when they get hit with a breach, when it just so happens to pop up on their radar, or when they need to in order to address some future due diligence process.

In some sense, IPM’s goal is to make this relationship real - as companies get more sophisticated, dealing with The Bot Problem should become more important - we’re here to help folks through that process.

Finally, we generated a score of our own. To create our own composite score, we factored in the number of bot deployments that are able to break through, the relative exposure of each of those bot deployments as measured by relevant Github activity, the average per-request cost of accessing the sites in question, and the complexity of the HTML structures on the successfully rendered pages. Below, we provide the IPM Best Retailers 2022 Bot Exposure Index for reference - again, no strong relationship here, but a rank-sorted list of the retailers is provided for these companies to start thinking about how to better address sociotechnical security concerns.

IPM Best Retailers 2022 Bot Exposure Index

Rank Parent Category Category Retailer Composite Score
1 Clothing Bridal Fashion Camille La Vie 89.85
2 Clothing Lingerie & Underwear La Perla 89.7
3 Clothing Bridal Fashion Bella Bridesmaids 87.56
4 Home Goods Pet Care Chuck and Don's Pet Food & Supplies 87.16
5 Clothing Second-Hand Clothing & Products Clothes Mentor 85.4
6 Clothing Bridal Fashion Fabulous Frocks 84.89
7 Home Goods Pet Care Petland 84.75
8 Food, Health & Beauty Perfume and Cosmetics LUSH 83.52
9 Clothing Bridal Fashion Lovely 81.45
10 Clothing Lingerie & Underwear Adam & Eve Stores 78.8
11 Clothing Discount Clothing Nordstrom Rack 77.19
12 Clothing Department Stores Nordstrom 77.07
13 Food, Health & Beauty Discount Supermarkets 99 Cents Only Stores 76.27
14 Home Goods Sporting Goods/Equipment (Team sports, Golf etc.) Hibbett Sports 75.52
15 Clothing Second-Hand Clothing & Products Once Upon A Child 75.43
16 Clothing Women's Apparel J.Jill 75.14
17 Clothing Jewelry James Avery 75.1
18 Home Goods Bookstores 2nd & Charles 74.5
19 Home Goods Furniture IKEA 74.05
20 Home Goods Toys Build-A-Bear Workshop 74.03
21 Food, Health & Beauty Chocolate and Candy Stores See's Candies 74
22 Food, Health & Beauty Convenience Stores Pilot Flying J 73.65
23 Clothing Second-Hand Clothing & Products AMVETS Thrift Store 72.97
24 Food, Health & Beauty Convenience Stores Thorntons 72.52
25 Home Goods Furniture La-Z-Boy 72.15
26 Clothing Fashion Accessories Spencer's 71.86
27 Food, Health & Beauty Perfume and Cosmetics L'Occitane 71.85
28 Clothing Outdoor Apparel L.L.Bean 71.65
29 Clothing Purses/Bags Tory Burch 71.57
30 Home Goods Toys LEGO 71.49
31 Home Goods Discount Home Goods dd DISCOUNTS 71.26
32 Home Goods Home Goods & Decor Bed Bath & Beyond 71.17
33 Clothing Jewelry (High-End) Tiffany & Co. 71.11
34 Clothing Purses/Bags Brahmin 70.97
35 Clothing Fashion Accessories Charming Charlie 70.95
36 Clothing Jewelry Diamonds Direct 70.84
37 Food, Health & Beauty Superstores and Warehouse Club Stores Target 70.76
38 Home Goods Toys Disney Store 70.72
39 Food, Health & Beauty Superstores and Warehouse Club Stores Sam's Club 70.6
40 Clothing Footwear Retailers (single-brand) New Balance 70.32
41 Home Goods Sporting Goods/Equipment (Team sports, Golf etc.) Scheels 70.29
42 Clothing Footwear Retailers (single-brand) UGG 70.25
43 Clothing Footwear Retailers (single-brand) Dr. Martens 70.05
44 Clothing Outdoor Apparel Duluth Trading Co. 70.03
45 Clothing Footwear Retailers (single-brand) Ecco 69.96
46 Clothing Luxury Fashion & Accessories Saint Laurent 69.85
47 Home Goods Furniture American Freight 69.82
48 Clothing Women's Apparel White House Black Market 69.78
49 Home Goods Discount Home Goods Tuesday Morning 69.68
50 Home Goods Premium Furniture Ethan Allen 69.6
51 Clothing Purses/Bags Kate Spade 69.48
52 Clothing Fashion Accessories Hot Topic 69.38
53 Clothing Lingerie & Underwear Adore Me 69.16
54 Clothing Second-Hand Clothing & Products Habitat for Humanity 69.11
55 Clothing Footwear Retailers (multi-brand) Shoe Sensation 68.73
56 Clothing Purses/Bags Coach 68.7
57 Clothing Athletic Apparel Fleet Feet 68.65
58 Clothing Outdoor Apparel Patagonia 68.63
59 Clothing Apparel Vineyard Vines 68.49
60 Clothing Fashion Accessories Fossil 68.46
61 Food, Health & Beauty Chocolate and Candy Stores Edible Arrangements 68.44
62 Clothing Discount Clothing Gabe's 68.29
63 Clothing Apparel J. Crew 68.17
64 Clothing Apparel Brooks Brothers 68.13
65 Home Goods Discount Home Goods Homesense 68.12
66 Clothing Jewelry TOUS 68.07
67 Clothing Second-Hand Clothing & Products Uptown Cheapskate 67.99
68 Clothing Lingerie & Underwear Soma 67.77
69 Home Goods Arts and Crafts Old Time Pottery 67.59
70 Clothing Outdoor Apparel The North Face 66.97
71 Home Goods Discount Home Goods Five Below 66.09
72 Clothing Footwear Retailers (multi-brand) Boot Barn 65.93
73 Clothing Premium Apparel Hugo Boss 65.87
74 Home Goods Furniture Aaron's, LLC 65.13
75 Clothing Jewelry (High-End) Harry Winston 65.09
76 Clothing Luxury Fashion & Accessories Chanel 65.04
77 Clothing Luxury Fashion & Accessories Dolce & Gabbana 64.86
78 Food, Health & Beauty Convenience Stores QuikTrip 64.85
79 Food, Health & Beauty Discount Supermarkets Aldi 64.82
80 Home Goods Home Goods & Decor CB2 64.53
81 Clothing Athletic Apparel Puma 64.52
82 Clothing Outdoor Apparel Orvis 64.49
83 Clothing Jewelry (High-End) Van Cleef & Arpels 64.28
84 Food, Health & Beauty Chocolate and Candy Stores Lindt 64.23
85 Home Goods Home Improvement Stores Rural King 63.99
86 Clothing Women's Apparel Cato Fashions 63.53
87 Clothing Baby/Children's Clothing Baby Depot 63.48
88 Home Goods Home Improvement Stores True Value Hardware 63.46
89 Home Goods Camping & Outdoor Gear (Hunting, Fishing) Field & Stream 63.43
90 Clothing Footwear Retailers (single-brand) Vans 63.27
91 Food, Health & Beauty Pharmacies and Drugstores CVS Pharmacy 63.24
92 Clothing Jewelry Kendra Scott 62.93
93 Home Goods Arts and Crafts Michaels 62.77
94 Clothing Women's Apparel Torrid 62.76
95 Food, Health & Beauty Eyewear Warby Parker 62.66
96 Clothing Athletic Apparel Nike 62.61
97 Food, Health & Beauty Pharmacies and Drugstores Giant Eagle Pharmacy 62.2
98 Clothing Footwear Retailers (multi-brand) Shoe Show 62.08
99 Clothing Lingerie & Underwear Victoria’s Secret 61.42
100 Food, Health & Beauty Hearing Care Beltone 61.35
101 Clothing Apparel Uniqlo 61.31
102 Food, Health & Beauty Hearing Care Miracle-Ear 60.99
103 Food, Health & Beauty Supermarkets Wegmans 60.85
104 Home Goods Toys American Girl 60.83
105 Food, Health & Beauty Chocolate and Candy Stores Rocky Mountain Chocolate Factory 60.75
106 Food, Health & Beauty Pharmacies and Drugstores Good Neighbor Pharmacy 60.73
107 Home Goods Auto Supplies Advance Auto Parts 60.62
108 Food, Health & Beauty Supermarkets Whole Foods Markets 60.45
109 Home Goods Pet Care Pet Supplies Plus 60.44
110 Food, Health & Beauty Eyewear Cohen's Fashion Optical 60.2
111 Clothing Fashion Accessories Brighton 60.18
112 Home Goods Furniture Rooms To Go 59.99
113 Home Goods Pet Care Hollywood Feed 59.84
114 Food, Health & Beauty Supermarkets Earth Fare 59.72
115 Food, Health & Beauty Hearing Care Amplifon 59.41
116 Home Goods Home Goods & Decor Crate&Barrel 59.25
117 Clothing Baby/Children's Clothing OshKosh B'gosh 56.67
118 Food, Health & Beauty Superstores and Warehouse Club Stores BJ's 56.31
119 Food, Health & Beauty Discount Supermarkets Trader Joe's 55.99
120 Home Goods Sporting Goods/Equipment (Team sports, Golf etc.) Golf Galaxy 55.51
121 Clothing Footwear Retailers (multi-brand) Journeys 54.96
122 Food, Health & Beauty Supermarkets Sprouts 54.91
123 Food, Health & Beauty Supermarkets The Fresh Market 54.67
124 Clothing Bridal Fashion David's Bridal 53.92
125 Clothing Department Stores Saks Fifth Avenue 53.75
126 Clothing Baby/Children's Clothing Pottery Barn Kids 52.89
127 Clothing Women's Apparel LOFT 52.87
128 Home Goods Sporting Goods/Equipment (Team sports, Golf etc.) Academy Sports + Outdoors 52.8
129 Home Goods Premium Furniture Ballard Designs 52.57
130 Clothing Athletic Apparel Under Armour 52.46
131 Clothing Jewelry (High-End) Bulgari 52.09
132 Clothing Luxury Fashion & Accessories Versace 51.59
133 Clothing Purses/Bags Dooney & Bourke 51.26
134 Home Goods Premium Furniture Pottery Barn 50.89
135 Home Goods Bookstores Half Price Books 50.67
136 Food, Health & Beauty Eyewear Stanton Optical 50.43
137 Home Goods Camping & Outdoor Gear (Hunting, Fishing) Cabela's 50.28
138 Clothing Premium Apparel Tommy Hilfiger 49.75
139 Home Goods Sporting Goods/Equipment (Team sports, Golf etc.) DICK'S Sporting Goods 49.52
140 Clothing Jewelry Pandora 49.38
141 Clothing Department Stores Bloomingdale's 49.02
142 Home Goods Home Improvement Stores Lowe's 48.88
143 Clothing Athletic Apparel Adidas 48.45
144 Home Goods Camping & Outdoor Gear (Hunting, Fishing) Bass Pro Shops 48.06
145 Home Goods Home Goods & Decor At Home 48.04
146 Clothing Baby/Children's Clothing Carter's 47.7
147 Clothing Apparel JoS. A. Bank 46.79
148 Food, Health & Beauty Supermarkets Publix 46.58
149 Clothing Footwear Retailers (multi-brand) Foot Locker 46.14
150 Clothing Premium Apparel Ralph Lauren 45.96
151 Food, Health & Beauty Eyewear LensCrafters 45.6
152 Home Goods Camping & Outdoor Gear (Hunting, Fishing) REI 45.28
153 Clothing Premium Apparel Calvin Klein 45.25
154 Food, Health & Beauty Chocolate and Candy Stores Great American Cookies 45.12
155 Food, Health & Beauty Pharmacies and Drugstores Walmart Pharmacy 45.03
156 Clothing Department Stores Neiman Marcus 44.86
157 Food, Health & Beauty Supermarkets Harps Food Stores 44.02
158 Food, Health & Beauty Perfume and Cosmetics Sephora 43.91
159 Food, Health & Beauty Perfume and Cosmetics Ulta Beauty 43.75
160 Clothing Luxury Fashion & Accessories Giorgio Armani 43.26
161 Home Goods Auto Supplies O’Reilly Auto Parts 42.87
162 Home Goods Bookstores Barnes & Noble 42.86
163 Food, Health & Beauty Pharmacies and Drugstores Walgreens 42.73
164 Food, Health & Beauty Supermarkets Fresh Thyme Farmers Market 42.68
165 Home Goods Camping & Outdoor Gear (Hunting, Fishing) Sportsman's Warehouse 42.53
166 Clothing Apparel Levi's 42.5
167 Home Goods Premium Furniture RH 42.23
168 Clothing Baby/Children's Clothing The Children's Place 42.03
169 Home Goods Home Improvement Stores Tractor Supply Co 41.75
170 Home Goods Pet Care PetSmart 41.49
171 Home Goods Arts and Crafts Hobby Lobby 40.14
172 Home Goods Toys HobbyTown 40.05
173 Food, Health & Beauty Superstores and Warehouse Club Stores Meijer 39.77
174 Food, Health & Beauty Convenience Stores Wawa 39.71
175 Clothing Premium Apparel Salvatore Ferragamo 39.42
176 Food, Health & Beauty Superstores and Warehouse Club Stores Costco 38.81
177 Home Goods Auto Supplies AutoZone 37.34
178 Clothing Department Stores Von Maur 35.71
179 Clothing Apparel Tilly's 35.52
180 Home Goods Home Goods & Decor HomeGoods 35.47
181 Clothing Discount Clothing Marshalls 35.46
182 Clothing Apparel Hollister 35.26
183 Food, Health & Beauty Convenience Stores Sheetz 34.83
184 Food, Health & Beauty Eyewear Pearle Vision 34.44
185 Food, Health & Beauty Supermarkets Kroger 34.39
186 Clothing Discount Clothing TJ Maxx 33.91
187 Home Goods Home Improvement Stores Menards 33.45
188 Food, Health & Beauty Perfume and Cosmetics Bath and Body Works 33.25
189 Clothing Jewelry (High-End) Rolex 26.05
190 Clothing Apparel rue21 24.5
191 Clothing Apparel Guess 24.48
192 Home Goods Discount Home Goods Dollar Tree 24.34
193 Food, Health & Beauty Supermarkets WinCo Foods 24.28
International Persuasion Machines
Previous

Reverse Engineering how WAFs Like Cloudflare Identify Bots
Next

Abusing Promo Code Signup Flows for Reliable Discounts with Major Retailers