d
Topic
Matt T Member
Posts:
5
Last edited about 2 months ago

What's the recommended way of blocking bot spam (from a single source)?

Hello,

Our website is getting hammered with bot spam. It all originates from Boardman, Oregon with the hostname "amazon" -- so it's clearly from Amazon's data center at that location. Thankfully, the bots aren't signing up or checking out, but their direct visits now comprise 50+% (!!!) of our daily traffic. This volume has destroyed the usefulness of our Shopify dashboard/metrics... thankfully, we can filter it out in Google Analytics.

What we're looking for is a way to block this bot spam. What is Shopify's official, recommended means of blocking bot spam, since we don't have access to server files (such as .htaccess) to do so ourselves? Are apps like TrafficGuard or Visitor Blocker the recommended method? We're hoping for a comprehensive answer that can help the community as well as ourselves.

I've seen a few threads asking similar questions, and haven't been impressed by the responses. Please note that we're not looking for advice on how to filter out this traffic in GA, nor are we gathering others' opinions on the harmfulness of bot spam on SEO or SERP rankings. "Just don't worry about it" is not a fix. We're looking for a real solution; if a teenager can fix it in five minutes with .htaccess, then a $16B-market-cap company and its intelligent community can also find a solution.

Thanks!

i
Replies
Posts:
3
26 days ago
g
1
upvotes

I'm afraid we are in the same boat here at Art in Coins and I'm rather disappointed that Shopify doesn't have any server side tools in place to assist.  That's what I would expect with what I pay to use this platform and I'm not at all pleased to find that my only recourse is a very expensive third party app.  (such as Traffic Guard / Visitor Blocker) 

C'mon Shopify, time to get on the ball (please) and get some tools in place to help users block unwanted traffic.

Posts:
2
24 days ago

Hmmm . . . no offical response.

From what I'm seeing elsewhere this started being a problem for a wider audience (not just shopify) as of June 19th.

Disappointing.

Posts:
9
18 days ago

Same problem!

Posts:
3
15 days ago

During my investigation, I have come across the following bash script on Github that can be employed as a server side solution.  I sent the information to Shopify support to investigate and hopefully deploy for their users.  The script is for either Unix or Linux servers so hopefully compatible with Shopify.

AWS-BLOCKER
A simple bash script to block all AWS IP ranges using iptables.
https://github.com/corbanworks/aws-blocker/blob/master/aws-blocker

#!/bin/bash -e
#
# Amazon AWS blocker through iptables.
#
# First we use curl to grab the official list of ranges from Amazon. The -s
# prevents extraneous output from curl, and the -L makes it follow redirects.
#
# The ranges are passed to jq, a JSON parser. The -r makes jq output raw data
# without quotes. We only need the list of prefixes, so we discard everything
# else.

POSITION=1
FILTERS=""
JSON_URL="https://ip-ranges.amazonaws.com/ip-ranges.json"


# Get the line where the jump will be inserted at.
# Useful if you want e.g related / established rules for outgoing traffic.
if [[ -n $1 ]]; then
    POSITION=$1
    shift
fi


##
# Builds region filters based on CLI arguments
#
# Arguments: CLI arguments as passed by $*
#
function build_filters() {
    for arg in ${@:1}; do
        if [[ -n $filters ]]; then
            filters=$filters", "
        fi

        filters=$filters"select(.region | contains(\"$arg\"))"
    done

    if [[ -n $filters ]]; then
        filters=" | "$filters
    fi

    echo $filters
}


##
# Extracts IP ranges from an Amazon JSON file
#
# Arguments:
#     $1 AWS JSON content
#     $2 Prepared filter string
#     $3 Group to extract IP ranges from (e.g. prefixes)
#     $4 Object key for IP ranges (e.g ip_prefix)
#
function extract_ip_ranges() {
    local json=$1
    local filters=$2
    local array=$3
    local prefix=$4

    local group='group_by(.'$prefix')'
    local map='map({ "ip": .[0].'$prefix', "regions": map(.region) | unique, "services": map(.service) | unique })'

    local to_string='.ip + " \"" + (.regions | sort | join (", ")) + "\" \"" + (.services | sort | join (", ")) + "\""'
    local process='[ .'$array"[]$filters ] | $group | $map | .[] | $to_string"

    local ranges=$(echo "$json" | jq -r "$process" | sort -Vu)
    echo "$ranges"
}


##
# Creates the AWS iptables chain if it doesn't exist, then flushes it
#
# Arguments:
#     $1 Version to use. Omit for v4
#     $2 Position to insert chain statement at
#
function create_and_flush_chain() {
    local version=$1
    local position=$2
    local cmd=ip${version}tables

    $cmd -n --list AWS >/dev/null 2>&1 \
        || ($cmd -N AWS && $cmd -I INPUT $position -j AWS)

    $cmd -F AWS
}


##
# Adds an iptables rule for each line in ranges
#
# Arguments:
#     $1 Version to use. Omit for v4
#     $2 Prepared lines
#
function add_iptables_rules() {
    local version=$1
    local cmd=ip${version}tables
    local lines
    local data

    IFS=$'\n' lines=($2)
    unset IFS

    for line in "${lines[@]}"; do
        eval local data=($line)
        local ip=${data[0]}
        local regions=$(echo ${data[1]} | tr '[:upper:]' '[:lower:]')
        local services=$(echo ${data[2]} | tr '[:upper:]' '[:lower:]')

        $cmd -A AWS -s "$ip" -j REJECT -m comment --comment "$regions = $services"
    done
}

# Retrieve IP ranges definition
# Either from an URL or file input (e.g. "< ranges.json")
if [ ! -t 0 ]; then
    JSON=$(cat - <&0)
else
    JSON=$(curl -s -L $JSON_URL)
fi

FILTERS=$(build_filters "$*")


# IPv4
create_and_flush_chain "" $position
V4_RANGES=$(extract_ip_ranges "$JSON" "$FILTERS" "prefixes" "ip_prefix")
add_iptables_rules ""  "$V4_RANGES"


# IPv6
create_and_flush_chain 6 $position
V6_RANGES=$(extract_ip_ranges "$JSON" "$FILTERS" "ipv6_prefixes" "ipv6_prefix")
add_iptables_rules "6" "$V6_RANGES"

Posts:
2
14 days ago

Has anyone made any progress on this?

Posts:
4
12 days ago

Same problem here! Any progress on that topic?

Posts:
2
11 days ago

Doesnt appear so. Its creating malformed URL's on my so I now have a huge amount of 404's.

Posts:
4
11 days ago

We actually have two shops which are affected. Several emails with Shopify, but so far they only referred to 3rd party apps. Absolutely not satisfying. 

Posts:
3
11 days ago

Hello all,

I have been in recent email discussions with one of Shopify's network specialists and they are looking at some options to combat this issue.  One thing I am working on is compiling a list of IP blocks that this Boardman Bot is using to spam my wesbite.

What is see used most is ip address starting with:

18, 32, 34 & 54

If anyone has observed other IP block's please post them here and I'll be happy to communicate them with Shopify.  I'm happy they are taking note and looking to take action either on a case by case basis, or, if this is now widespread, a broad platform solution so we won't have to resort to expensive 3rd party apps.

Speaking of apps.  I use the following:  Back in Stock, Delerious Profit, Mailchimp, Order Printer, Product Reviews and Tawk.to Chat.  I'm taking a little poll to see if perhaps there's an app on the platform that's got a malicious backdoor to it that's directing this bot activity.  Not likely since the Boardman Bot is bombing tons of websites and not just Shopify's, but I am very curious about how this fake search traffic got turned on to my particular store.

Cheers all, light at the end of the tunnel.