d
Topic
Jason Member en.gg
Posts:
24
Last edited 16 days ago
g
1
upvotes

Friday Update to CDN 403 Issue?

I'm not exactly sure what has been done... but it seems something was changed with the CDN server. 

My server is having problems getting the content of IMAGE files stored in the SHOPIFY CDN. It's returning ERROR 403 FORBIDDEN when trying to fetch images via CURL.

Looks like CURL requests now require CURLOPT_USERAGENT and a CURLOPT_REFERER to function properly. This is a breaking change and should have been broadcasted to App developers somehow.

The issue seems to be present with PHP's file_get_contents - its not working with product images on the cdn when using default config settings - file_get_contents($path). Also, its very possible that it is somewhat randomly working/not working.

When I switched from using file_get_contents to a custom CURL function with USERAGENT AND REFERRER included - the problem has been resolved.

 

i
Replies
Jason Shopify Expert freakdesign.com.au
Posts:
9252
Last edited 16 days ago

What API endpoint are you referring to that's not working as you expect? Sounds like this isn't API related or am I missing something?

[edit] I also threw a quick curl command into terminal and got the contents back from an image on the CDN aok. I suppose the options you're talking about could be sent by default, but didn't spot anything unusual.

★ Winning Partner of the Build a Business competition. ★ http://freakdesign.com.au
Jason Member en.gg
Posts:
24
16 days ago
g
1
upvotes

I updated the post to specify a CDN issue, not API. Try running file_get_contents (php) across hundreds of product photos and let me know if you spot any 403s along the way...

My servers process 500K photos per day, I can't tell you exactly what the issue is, only that there is an issue somewhere and I've re-explained above to try to pinpoint it best I can. Will add to it if I spot anything else.

Posts:
1
15 days ago

Seeing the exact same thing from myside, random 403 from cdn. 

Posts:
4
14 days ago

I am also seeing this error when trying to download artwork from Shopify's CDN:

PHP Warning: file_get_contents([Shopigy CDN Link Here]): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden

Although, I note that if the artwork is loaded in a web browser first, then it becomes available to PHP's file_get_contents function.

Posts:
3
14 days ago

I am also experiencing this same issues. When using PHP or C# to download the byte array from image content I am receiving a 403 response code.

This change has broken my app and I have upset customers. Please be more diligent in providing resolution steps. 

Posts:
4
Last edited 14 days ago
g
1
upvotes

Having investigate the PHP side of this a little further...
PHP 7.2 does not have a default user agent when installed on Ubuntu 18.04. I've tested this on my setup by querying a custom web page I setup to echo the user agent out. I get the following:

"http_user_agent": null,

By setting the user agent explicitly in my PHP script file performing the download, it is possible to have this filled out for HTTP requests from file_get_contents:

ini_set('user_agent', 'Mozilla/4.0 (compatible; MSIE 6.0)');
$file = file_get_contents($url);

Resulting in:
"http_user_agent": "Mozilla/4.0 (compatible; MSIE 6.0)",
 

I believe (but as of yet, have been unable to test as I've had no new Shopify orders with this error since this morning!) that because the user agent string is not specified in my case, Shopify's CDN is blocking the request as forbidden when it is the first request for that content. My current hope is that by explicitly setting a user agent string in my code will solve this problem. I guess I will find out later!
 

Posts:
3
14 days ago

I am going to confirm that explicitly specifying the user agent as seemed to solve the issue. Thank you for the post, Sam. 

Posts:
4
14 days ago

Good to hear Paul. So far I am also having positive experiences setting the "user_agent" string in PHP. I've had 3 orders since my last posting, all of which have downloaded absolutley fine after setting the user_agent.

I can also confirm my theory was correct, once a request has been made to the CDN with a user agent string, my old code without the user agent string is able to pull the artwork normally. I suspect the CDN has implemented security to stop scraping from bots or a caching server is rejecting pass through of requests for new image content; when a "web browser" access it, the CDN caches the content and then can respond to the requests as normal.

Steps to reproduce the problem (for anyone at Shopify who cares):
1. create a file on the CDN (a new file, specifically an image in my case.) It must not have been loaded in a web browser first.
2. create a PHP file that is simply using file_get_contents() to load the URL:

file_put_contents('output.jpg', file_get_contents('https://cdn.shopify.com/s/files/1/1244/8592/products/s_8451_Gjht8kCdAYDgGVDbwtBT3AeprdetMFNavy.png?v=1502115618'));

3. call the file, I'm using a terminal

php -f download.php

and you'll get the error I posted previously (403 forbidden)

PHP Warning:  file_get_contents('https://cdn.shopify.com/s/files/1/1244/8592/products/s_8451_Gjht8kCdAYDgGVDbwtBT3AeprdetMFNavy.png?v=1502115618'): failed to open stream: HTTP request failed! HTTP/1.1 403 Forbidden

4. Now open a web browser and go to the URL. It'll load fine
5. Re-run the download PHP script. It too will work fine, the downloaded file will be received and created on disk without issue.

For any PHP coders who can't get the "under_agent" to work with file_get_contents, I suggest looking at the Guzzle PHP library (https://github.com/guzzle/guzzle). It gives far more flexibility of setting headers for the request.

It remains unclear if Shopify are aware that this change has taken place and the impact it's had. Hopefully this provides information to anyone experiencing the issue and Shopify so they understand what we are experiencing.

Chew Member ricemill.co
Posts:
1
11 days ago

Got the same 403 issue. I have managed to fix it with this solution: https://stackoverflow.com/a/18910202/10380513

 

Chew from Ricemill, the Inventory Management Platform