Tuesday, December 31, 2013

Management of Cache Defeating URLs

What is server side cache?

Server side cache is a repository where dynamic content is rendered and stored as static content.  The example we will use in this article is a landing page for email campaigns that pulls in the current sale items for the week.  Since the content changes each week can save the cached version for a week before discarding it.

You develop your content and products for the page and it has a URL of http://www.awesomesite.com/week-1-sales.html. When you call this URL for the first time, it takes 2 seconds to generate page and place it into server cache.  The first run has to go to the database and find all the information on the products part of this weeks sale.

Once the static content is assembled the cache on the server side makes an entry for /week-1-sales.html and stores it in an easy to retrieve fashion.  The next time you call that same URL it loads nearly instantly in lets say 500ms which shaves off 75% of the time to build that page.   

This is an excellent example of how Cache in Magento, Demandware, Drupal, Varnish and nearly any other caching mechanism works.

What is a cache defeating URL?

A cache defeating URL is a unique URL to an otherwise static resources.  This may sound complicated but it is a very easy concept to understand.  Marketing campaigns by far generate the most amount of cache defeating URLs.  So how do they do it?

When someone sets up a marketing campaign they always want to know how many people opened the email, clicked on the link and perhaps even purchased a product.  Most bulk email providers can track that through the entire process including the sale down to the individual.  This allows the business unit the insight to know who are their best customers and how effective a campaign is based on the user demographics.

Gathering this information is done by the email containing a unique identifier to that customer.  In some cases it is the users email address.  When you hover over a link in your email you may see something like http://cc.mailprovider.com/clickThrough?cID=HDY7S&email=user@isp.com. You can see that the URL is not going to www.awesomesite.com yet.  It is in fact going to the mail provider to track that you opened the email.
Once you arrive at the mail provider’s website you will be redirected to your actual landing page.  The URL will often look something like this


You can see that it passed along the campaign ID (cID) the users email, and what group (grp) it went out in.  Javascript embedded on your website and in your PHP code picks up on these values and stores them into a cookie or session.   This is used while you visit the website to track anything from what you viewed to what you purchased.

So how does this affect the cache?

Remember that our cache was built off of the URL


However, the mail provider requested


We know by looking at it that it simply adding on the tracking information to end.  We know that despite the additional parameters the actual content on the page will not change at all.  

The web application and the web cache have no way to know if those new parameters will result in different content.  The result is that the page will be re-generated taking a full 2 seconds with those new parameters.  The content will look exactly the same however it will now have a a copy in cache with those parameters.  If we send the same email campaign to 5 users the cache will now look like this:

http://www.awesomesite.com/week-1-sales.html?cID=HDY7S&email=user@isp.com&grp=3 http://www.awesomesite.com/week-1-sales.html?cID=HDY7S&email=abcr@isp.com&grp=3

The amount of CPU time to create all 6 pages is 12 seconds.  In contrast, if those same users all went to the http://www.awesomesite.com/week-1-sales.html only, the total CPU time would be 4.5 seconds.  Imagine the amount of time it would take if an email blast to 20,000 visitors resulted in 500 people opening the email at the same time.  With 64 cores on the box you would have far more requests than the server could process.  These tracking URLs defeat the cache and eventually will cause the server to stop honoring requests.

Mail providers are not the only cause of cache defeating URLs.  Referrals can be another source.  If your site participates in referrals then you will often see a refID parameters in your URLs.   One example might be http://www.awesomesite.com/week-1-sales.html?refID=298374

Managing the problem

The largest challenge in managing this problem is understanding why it happens.  Hopefully after reading this far you now understand how important web cache is.  So what can we do now to mitigate the issue. Magento carts have a big issue with this in particular.

Know your traffic

First do an analysis on your web logs for a specific URL.  A sysadmins can easily take a few weeks of web logs and tell you what parameters they see coming across for a particular URL.  Once you have the list find out who is sending it and why.  This audit will take a little bit of time but can be valuable in keeping your website running after a great marketing campaign.

Throttle Traffic

This is a simple and effective way to prevent a server crash.  It won’t prevent unnecessary cache build up but spreading a 20,000 user email blast over 2 or 3 hours will keep your site running.  This is not always a viable option when you have a time sensitive promotion on your site.

Static Landing Pages

If your marketing blast or referral tracking numbers rely only on JavaScript to read the parameters you can create a static .html file with that JavaScript on the static page.  Once the cookies are set via JavaScript the user can proceed to another page without issue.

Page Cache Filter The last part is filtering out at a code level the parameters sent. This means often tinkering with the page cache itself and the way it is called. The company I work for Lyons Consulting Group has developed just this type of Page Cache filter for Magento. This allows the Magento cart to operate with virtually any mail vendor (Listrak, etc.) to submit parameters that can removed from the cache.

Sunday, March 10, 2013

Getting to the bottom of Bus Errors

Last week I ran into a fatal PHP error which Apache reported as a Bus Error (7).  I've been working in PHP and Apache now for over 10 years and I have never seen PHP just die and report a Bus Error.  So through some research I found out what this bizarre error means and how you can find out what is causing it.  You're going to need to have a dedicated box to make all this happen.

First Bus Error = Core Dump.  This was annoying because had I known it was a core dump from PHP I would have known where to start.  When you google Bus Error 7 you get a lot of bug reports and other garbage but not a lot of information on where to go to figure this issue out.   The key here is that you need the version of PHP and Apache with the debugging symbols in it.  Thats not the out of the box package.

PHP Website talks about recompiling PHP and Apache to get the debugging turned on but I was able to find a better solution.

CentOS has a debugging repo so we found packages here http://debuginfo.centos.org/6/x86_64/php-debuginfo-5.3.2-6.el6_0.1.x86_64.rpm .  You'll need the Apache ones as well.  Once you get these installed you're ready to figure out what is causing your headache.

What you're going to want to do is follow the instructions over on the PHP website. https://bugs.php.net/bugs-generating-backtrace.php This will get PHP configured and setup to generate the core dumps.

Once you have them then it's time to work some magic.

You'll start up GDB just like it shows on the PHP link above and run the "bt" command.  I got this out of the output.

#0  lex_scan (zendlval=0x7ffff7fcce08) at Zend/zend_language_scanner.c:2117
#1  0x00007fa0a14d0aa0 in zendlex (zendlval=0x7ffff7fcce00) at /usr/src/debug/php-5.3.3/Zend/zend_compile.c:4942
#2  0x00007fa0a14bb2e7 in zendparse () at /usr/src/debug/php-5.3.3/Zend/zend_language_parser.c:3282
#3  0x00007fa0a14c5f32 in compile_file (file_handle=0x7ffff7fcd130, type=) at Zend/zend_language_scanner.l:354
#4  0x00007fa09b0cc721 in phar_compile_file (file_handle=0x7ffff7fcd130, type=2) at /usr/src/debug/php-5.3.3/ext/phar/phar.c:3393
#5  0x00007fa09f13bda6 in ?? () from /usr/lib64/php/modules/ioncube_loader_lin_5.3.so
#6  0x00007fa0a14c57de in compile_filename (type=2, filename=0x7fa0aef2aa78) at Zend/zend_language_scanner.l:397
#7  0x00007fa0a15159d8 in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER (execute_data=0x7fa0acccfa58) at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:22432
#8  0x00007fa0a150f400 in execute (op_array=0x7fa0acc6b898) at /usr/src/debug/php-5.3.3/Zend/zend_vm_execute.h:107
#9  0x00007fa0a14e0285 in zend_call_function (fci=0x7ffff7fcd4e0, fci_cache=) at /usr/src/debug/php-5.3.3/Zend/zend_execute_API.c:963
#10 0x00007fa0a15003f7 in zend_call_method (object_pp=0x7fa0acd948a0, obj_ce=, fn_proxy=0x7fa0acd94898, 
    function_name=0x7fa0acca8f98 "aitoc_aitsys_model_rewriter_autoload::autoload\005", function_name_len=, retval_ptr_ptr=0x7ffff7fcd648, param_count=1, 
    arg1=0x7fa0ad68eaa8, arg2=0x0) at /usr/src/debug/php-5.3.3/Zend/zend_interfaces.c:97
#11 0x00007fa0a14044d6 in zif_spl_autoload_call (ht=, return_value=, return_value_ptr=, this_ptr=, 
    return_value_used=) at /usr/src/debug/php-5.3.3/ext/spl/php_spl.c:406
#12 0x00007fa0a14e032e in zend_call_function (fci=0x7ffff7fcd890, fci_cache=) at /usr/src/debug/php-5.3.3/Zend/zend_execute_API.c:985
#13 0x00007fa0a14e094e in zend_lookup_class_ex (name=0x7fa0aee5b9e0 "XXX_Catalog_Model_Inventory_Stock_Item", name_length=38, use_autoload=1, ce=0x7ffff7fcd9b0)
    at /usr/src/debug/php-5.3.3/Zend/zend_execute_API.c:1120
#14 0x00007fa0a14fb4b0 in zif_class_exists (ht=, return_value=0x7fa0aee1a470, return_value_ptr=, this_ptr=, 
    return_value_used=) at /usr/src/debug/php-5.3.3/Zend/zend_builtin_functions.c:1217

Now while this may look like complete garbage it actually tells you quite a bit.

First, it's clear it died in lex_scan which at this point you know something other than "Bus Error".  The key here though was what came at line 10.  "aitoc_aitsys_model_rewriter_autoload" here was the problem.  Something in this was causing the issue.   This module uses some derived code from the Ioncube loader.

Now while I don't have a solution for this piece of code I know what at least was causing the problem and where.  If you look at line #13 you'll see the Magento module where this was being called.   So again we at least know where this issue is happening.

Standard debugging here would be normal where you can isolate the issue.  What we knew going into this is that Aitoc has serious licensing issues.  When developers in Magento use their modules there are constant complains about Aitoc and their licensing.  I found from personal experience their 24 hour once a day response to your email is not worth it.  I wouldn't use their software if I didn't have to.

The point of this whole post is simply to give you some hope when you run into this Bus Error with Apache and PHP.  You can get to the bottom of it with these simple tools.  I was not able to find a way to get to the bottom of the Bus Error without a lot of research on the internet.  This post should now help someone else quickly get to the bottom of a bus error without having to research for hours like I did.