d
Topic
Ryan O Shopify Employee
Posts:
233
Last edited 11 months ago
g
2
upvotes

Upcoming improvements to Webhooks This post is outdated

Update Jan 12th below:

The de-bounce change is now live for all shops.  Please keep leaving your questions and concerns around this new improvement in this thread.  Feel free to share some stats as well.

As for other webhook related issues; feel free to keep bringing them up as well, just note that these improvements are solely based on eliminating duplicates and do not alter the contents of any existing webhooks.

 

Update Jan 4th below:

We are going to be starting a tiered rollout early next week for the webhook improvements.

The changes will be what is described in the original post, with the exception that we will exclude any webhook topics that are /add or /create.

As always we look forward to your feedback before and after this feature has rolled out.  Feel free to reply with any questions or concerns.

 

Hey All,

We are introducing a number of improvements to how Shopify delivers webhooks.  Our existing webhooks implementation offers a poor experience for app developers on a number of dimensions:

  1. Flooding.

    • A spike of requests on Shopify's side causes a spike of webhooks to be sent to apps (e.g., a flash sale). Apps have to design against this.

  2. Duplication.

    • Our webhook implementation often sends duplicate webhook payloads in a short period of time. For app developers, this translates to increased hardware requirements for processing as well as additional contention for the affected resources.

We are introducing changes to improve the above issues that will have a twofold impact; a net reduction in webhooks received from Shopify, as well as short delay between an action and the webhook being sent in order to ensure that only the most recent state is communicated to your app.

For example:  The following actions are performed in quick succession;

  1. A product is created

  2. A title is added to the product

  3. A description is added to the product

  4. A price is added to the product

Previously the behaviour would result in a products/create webhook, and then three subsequent products/update webhooks, not necessarily in the order of changes made.

After introducing these improvements, a products/create webhook would be sent, and then a single products/update webhook with the most up-to-date payload would be sent after a short delay.

If you have questions or concerns about these changes, please comment in the thread below so we can address them prior to launch.

 

All the best,

Ryan O

Developer Experience

Shopify

i
Replies
Felix Member looga.io
Posts:
98
12 months ago

Great improvements.

I experienced that for a single shop attribute change, the shop/update webhook gets fired four times (details in this post). Is this going to be improved, so that it gets fired only once, too?

Looga.io
Martin Member
Posts:
4
12 months ago

Awesome! We were looking for updates about address changes. I thought that customer.updated would be that -- but that was firing on every new order. 

Thanks!

Ryan O Shopify Employee
Posts:
233
12 months ago

Hi Felix,

The issue you describe is indeed one of the scenarios this change is aiming to fix.  Thanks for your feedback!

Ryan

Harold Shopify Partner gaze.bo
Posts:
32
12 months ago

Hi Ryan,

That's a great update!

Do all webhook topics get the delays or only the ones that can have an issue like the one you described. I'm most interested in the the orders/create webhook, as a delay on that one would have a very negative impact on loads of apps. :)

And do you have more info on the timeline of the updates?

Cheers,

Harold

Posts:
618
12 months ago

Awesome news!

On a very related note, I'm still trying to figure out, Does the [orders/updated] webhook fire whenever any other [orders/*] webhook fires? No point subscribing to all other topics if that is the case...

Paul Member
Posts:
6
12 months ago

Hi there,
Wonderful!
From this morning we are heading some issues with APP deletion webhook. With some test account works correctly with others not.
It started more or less today, do you think is a conseguence of your webhooks restructuring process?

Ryan O Shopify Employee
Posts:
233
12 months ago

Thanks for all the feedback, please keep leaving questions and comments.  We are reading and discussing all of the feedback we've gotten.

 

Ryan

Ryan O Shopify Employee
Posts:
233
11 months ago

Hey all,

We are going to be starting a tiered rollout early next week for the webhook improvements.

The changes will be what is described in the original post, with the exception that we will exclude any webhook topics that are /add or /create.

As always we look forward to your feedback before and after this feature has rolled out.  Feel free to reply with any questions or concerns.

 

Ryan

 

atradef Member
Posts:
1
11 months ago

Do you have information on how long the delay is after the events occur? 

Posts:
7
11 months ago

Previously, we’ve got the issue that products/update webhook didn’t come to us during the product purchase, and we didn’t have any ability to detect the product count changes in the store.

Are you going to fix this issue during your future webhook changes? Will the products/update webhook always come to us if something changed with the product data?

Posts:
618
11 months ago
g
1
upvotes

And there is also this bug where the product webhook gets fired but doesn't have the latest data.

https://ecommerce.shopify.com/c/shopify-apis-and-technology/t/possible-bug-in-webhook-product-update-381317

Eric Member
Posts:
15
11 months ago

Will /create webhooks eventually be de-duped? Those really are only ones we are about being de-duped.

Ryan O Shopify Employee
Posts:
233
11 months ago

Hi Eric,

Will /create webhooks eventually be de-duped? Those really are only ones we are about being de-duped.

I think you missed a key word in this sentence! However, there is no plan to include changes to /add or /create webhooks in the future.

 

Eric Member
Posts:
15
11 months ago

Why? Those are really the only ones that we care about de-duping.

Ryan O Shopify Employee
Posts:
233
11 months ago

Why? Those are really the only ones that we care about de-duping.

Hi Eric,

Can you elaborate on this? Are you seeing duplicates for /create webhooks? 

 

Ryan

Eric Member
Posts:
15
11 months ago

Yes. Dupe /create webhooks are a huge pain. Mainly for orders and checkouts.

Harold Shopify Partner gaze.bo
Posts:
32
11 months ago

I had a lot of dupes too, made a workaround that worked perfectly fine, but in the end I found a little bug in the queuing system that I use for webhooks.

Since I fixed that a couple of months ago I received 0% dupes on the orders/create webhook, so I don’t think there is a dupes issue on the API side. :)

Posts:
618
Last edited 11 months ago

In the case where we do receive multiple webhooks for the same object, it would be useful to have an incrementing id in the headers so that we know which one is the latest.

It would come in handy in the case when the app is temporary down and Shopify may requeue many webhooks that failed but it is not obvious to the app which one represents the latest payload. Or is the best practice to consume the request with the highest update_at and discard the others?

Ryan O Shopify Employee
Posts:
233
Last edited 11 months ago

In the case where we do receive multiple webhooks for the same object, it would be useful to have an incrementing id in the headers so that we know which one is the latest.

It would come in handy in the case when the app is temporary down and Shopify may requeue many webhooks that failed but it is not obvious to the app which one represents the latest payload. Or is the best practice to consume the request with the highest update_at and discard the others?

Hi Clement,

The intended behaviour would be that for any webhook subject to the de-dupe, /update for example, would only have one retry per resource.  So when your app starts responding again after being down, there would be a single retry queued per resource updated while offline.

Cheers,

 

Ryan

Posts:
618
11 months ago

Thanks Ryan. That should be very helpful to decrease the number of request.

On that same topic, would you know that following? Does the [orders/updated] webhook fire whenever any other [orders/*] webhook fires? I want to catch all orders update events. Is subscribing to orders/updated enough?

Posts:
20
11 months ago
g
1
upvotes

Is there any way in the inventory quantity for the product(s) sold can be returned in the order_creation webhook? Pretty frustrating that the product_update hook doesn't fire when inventory quantity is changed and that it's not returned in the order_creation hook.

Ryan O Shopify Employee
Posts:
233
11 months ago

Hey all,

The de-bounce change is now live for all shops.  Please keep leaving your questions and concerns around this new improvement in this thread.  Feel free to share some stats as well.

As for other webhook related issues; feel free to keep bringing them up as well, just note that these improvements are solely based on eliminating duplicates and do not alter the contents of any existing webhooks.

 

Cheers,

Ryan O

Posts:
618
Last edited 11 months ago

Our app has been experiencing missing webhooks over the past week.

I'm guessing it is related to this change.

Are missed webhooks expected? Are we the only ones experiencing this?

Ryan O Shopify Employee
Posts:
233
11 months ago

Hi Clement,

Missing webhooks are not intended as part of this change.  Webhooks are a best-effort delivery system though, so it does happen.

If you do see a pattern of consistent misses, can you share any more information on this?

 

Ryan

Posts:
618
11 months ago

Thanks Ryan.

After digging through the particular scenario with Shopify support, it appears to be an existing webhook bug that pre-dates the debounce release.

(If anyone is interested, the bug occurs when a refund is actioned but the gateway doesn't support refunding through Shopify. In this case, the order isn't actually refunded and the items are only restocked and returned into their inventory. However, the bug is that no webhook is issued when this happens. I'm waiting for the support to complete their investigation).

It would be great if you could expand a little regarding the best-effort delivery. Given that Shopify retries each webhook a number of times, it sounds like it would be pretty unlikely that they don't get delivered at all (assuming the app is behaving well). Or are there cases where a particular webhook wouldn't get delivered to the app at all?

Posts:
30
10 months ago

Hi, my app is using order/create webhook to work but suddenly not work recent day.

I've checked and the webhook still on webhook list but not work when I placed some test order, even only this simple code not work:
$db->query("insert into test set test = 'test'");

https://www.upwork.com/freelancers/~01519d714ba4d67667
Ryan O Shopify Employee
Posts:
233
10 months ago

Hi Thanh,

I checked for your shop, and your webhooks are failing because you haven't passed SSL verification.  As per this post: https://ecommerce.shopify.com/c/api-announcements/t/changes-to-webhook-ssl-certificate-verification-on-shopify-456661 any SSL failures will result in undelivered webhooks.  There was plenty of warning, and we even extended the date.  Once you pass SSL verification, your webhooks will be successfully delivered.

 

Ryan

Posts:
6
Last edited 10 months ago
g
1
upvotes

My app has also started to fail webhooks with a 'Error: unable to get local issuer certificate' for all stores. Within the same timeline as @ThanhBT mentioned. (Only started in the past day or two).

Could this be related to the changes?

I am using a LetsEncrypt SSL certificate for my app. It shows valid in the Google Console.

I am going to try and manually refresh it and see if this resolves the issue.

 

Edit: Followed these instructions and ensured I was using my FullChain.pem https://community.letsencrypt.org/t/some-browser-say-certificate-is-not-trusted/28766 Now the SSLChecker is showing that the intermediate cert is good. So I am hoping my webhooks stabilize.

Posts:
30
10 months ago

I just fixed the ssl issue on my site and it seem work, thank you so much!

https://www.upwork.com/freelancers/~01519d714ba4d67667
Ryan O Shopify Employee
Posts:
233
10 months ago

Glad it worked Thanh!

Cole, do you have any luck after updating your SSL certificate?

 

Ryan

Modules4U Shopify Partner www.modules4u.biz
Posts:
7
10 months ago

Hi Ryan,

not sure if this is related but since a few hours my checkouts/update webhook is not being called anymore for my app when the customer reaches the payment step.

This is a big problem and I would highly appreciate any insights.

Posts:
4
10 months ago

I have the same problem.

Ryan O Shopify Employee
Posts:
233
10 months ago

Hi Modules, Sergey,

In the future these threads should only be used for discussion around the listed changes.  Though I checked quickly, and on my test shop checkouts/update webhooks are working as intended.  I see that you have an open ticket already Modules, if you could provide some more info in there about the specifics we could assist more.

 

Ryan

Ryan Alyea Shopify Partner rofish.net
Posts:
26
Last edited 9 months ago

Hey there,

Is de-bouncing turned on or even working? I'm getting multiple orders/updated calls for the same order in the same second. What's even worse is that not everything is updated on some so for example if I update a something over the API, I will get TWO orders/updated calls. AND they're different! Here's an exact example I pulled from a log:

  1. Update a note_attribute of an order from the API call.
  2. Get TWO webhooks with an updated_at at the SAME SECOND:
  3. The only change on the first webhook is just the updated_at time.
  4. The second webhook updates both the updated_at time and the note_attribute.

The way my webhook logic is such that if it sees a webhook of the same time or earlier as the current updated_at time, it is ignored. If it sees a newer one it's updated.

However, since 1) I'm erroniously being sent a bad webhook and 2) the second webhook only has time resolution of the second, I cannot properly resolve the race condition. Since webhooks are sent out of order, I cannot tell which one is most recent data for that second.

In short you could:

  1. Fix your webhooks from firing out just a simple updated time without the updated data.
  2. Provide a stronger resolution of time, even if it's exclusive to the webhook in something like a X-Shopify-Webhook-Sent header.
  3. Get de-bouncing working again; I'm getting multiple orders/updated webhooks for the same second on the same order ID. At least delay webhooks by a couple of seconds until it's finally done processing data.

Store: fangamer.myshopify.com

Plan: Shopify Plus

App ID: None, using the webhooks from the Notifications in Settings.

Posts:
618
9 months ago
g
1
upvotes

I have to agree. I also suggested above that it would be useful to have an incrementing number/timestamp in the headers so that we know which one is the latest.

Personally I'd probably favour an integer, like Shopify-Entity-VersionId or something.

Ryan O Shopify Employee
Posts:
233
9 months ago

Hi Ryan,

Debouncing is only enabled on API Clients, webhooks created in the Notifications section are not included in this.  Also, I will pass along your and Clement's concerns about adding a ID on webhooks as I believe that is a valid concern.

Cheers,

Ryan

Ryan Alyea Shopify Partner rofish.net
Posts:
26
9 months ago

Does API clients include private applications? Or do I need to have a “public” API key?

Due to what I think is Shopify Plus’s rules, I try to keep things in private apps due to API limits.

Ryan O Shopify Employee
Posts:
233
9 months ago

Private applications would be debounced as well.

Ryan Alyea Shopify Partner rofish.net
Posts:
26
9 months ago

Cool. One last question: Is there any other differences between Notification webhooks and API webhooks? I want to be doing The Right Thing™️.

Ryan O Shopify Employee
Posts:
233
9 months ago

Nope, just that they are scoped individually.  API clients can only interact with webhooks they created, and the Admin doesn't show webhooks created by API clients.

Radhesh V. Member
Posts:
5
4 months ago

Hello,

I am using product/update webhook. When I make product/update API call using permanentToken
shopify gives me product/update webhook. Is this an expected behaviour ? if yes than can you
please tell me how to identify that the webhook call is because of our own product/update call ?