Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

outputcache max-age incorrect #330

Closed
mofee opened this issue Oct 26, 2016 · 30 comments
Closed

outputcache max-age incorrect #330

mofee opened this issue Oct 26, 2016 · 30 comments

Comments

@mofee
Copy link

mofee commented Oct 26, 2016

a simple controller like below:
public class DetailController : Controller { [OutputCache(Duration = 300, VaryByParam = "id")] public ActionResult Index(int id) { return View(); } }

and the view
`@{
Layout = null;
}

<title>Index</title>
@DateTime.Now.ToString()
`

when i first request the url http://localhost:80/Detail/Index?id=3, the max-age in response headers is correct (max-age=300), then i refresh the page (ctrl + f5), the respones state is 304, but the max-age in response headers is incorrect (max-age=63613066544) , and this cause the url link click request read the response from the browser's disk cache ( 200 ok from disk cache).

the issue only happen in .net framework 4.6.2, not happen in 4.6.1 or early version.

@albigi
Copy link

albigi commented Oct 28, 2016

Same issue occurring to a customer of mine.
I can reproduce the issue very easily by setting the Mvc OutputCache attribute in a boilerplate ASP.NET 4 app.
From some initial investigation the problem doesn't seem to be related to 4.6.2 only, at least in my case, since downgrading the framework is not helping.
Apparently, the wrong value is only applied when the System.Web.Mvc.OutputCacheAttribute overridden methods are not invoked, that is when the response is coming from the cache.
On top of that I could verify from some live debug the HttpCachePolicy is properly populated:
image
Will run some further investigation on this.
Additionally, there's also a SO thread related to this: http://stackoverflow.com/questions/40218428/iis7-5-max-age-issueasp-net-mvc-output-cache

@albigi
Copy link

albigi commented Oct 28, 2016

Where I could get so far is that apparently the issue is occurring in System.Web.HttpCachePolicy, more specifically in the method
image

In this method we do:

image

the issue comes when age is set to a negative value due to _utcTimestampRequest being null
image

I couldn't pinpoint why it comes that's null as I assumed it was taken by HttpResponse.Context.UtcTimestamp

Perhaps we are calling a Reset() too early?
It seems this happens all the time we call System.Web.UI.Page.InitOutputCache and in the response the HttpCachePolicy object "Cache" is instantiated for the first time.

@mofee
Copy link
Author

mofee commented Oct 29, 2016

in our production env, we only uninstall 4.6.2 and install 4.6.1, and the issue not occur now.
it is not effact to modiffy targetFramework in web.config if you install 4.6.2

@DazedLead
Copy link

DazedLead commented Nov 1, 2016

We are also getting this issue.
on Microsoft Azure Web App

@albigi
Copy link

albigi commented Nov 3, 2016

The issue has been identified.

We are not setting any more the _utcTimestampRequest value in HttpCachePolicy.ResetFromHttpCachePolicySettings

In previous versions apparently we used to set it tot he actual timestamp of the inbound request.

As a result, we are calling HttpCachePolicy.GetHeaders with a _utcTimestampRequest object with a dateData value set to 0.

Hence when we do age = _utcTimestampRequest - _utcTimestampCreated; we get an absurd value such as the ones in the repro.

I cannot think about any immediate code workaround since the issue appears to happen all within System.Web

@albigi
Copy link

albigi commented Nov 7, 2016

Just to provide an update on this: a fix is being tested and it is bound to be released with the 4.6.3 update or earlier.

@OnTheMike
Copy link

Is there already news on a date for the release of the fix?

@mortb
Copy link

mortb commented Nov 24, 2016

This issue is quite severe, we're about to release an application using asp.net mvc 4.6.2
We rely on cache with a time set to reduce network traffic, we'd rather not remove the OutputCache attribute as that would lead to unnecessary increases in response time / traffic.
Is there anything we can do to correct this before the fix?

@albigi
Copy link

albigi commented Nov 24, 2016

Unfortunately, there are no updates on the official release date for the fix yet. This is still being tested and it's yet to agree on whether shipping this with 4.6.3 or earlier.

Please note that the bug is affecting the way the max-age field is populated only. This can have an impact mainly for caches or CDNs that are reading the max-age attribute in order to manage the content freshness and distribution.
The actual caching is still working as expected and the webserver will not send stale content after the refresh interval has expired.

Currently, the only recommended workaround is to downgrade to 4.6.1 by removing KB151864.

If you are unable to, there might be one hack possible which would use reflection to change the internal private field _utcTimestampRequest and populate it with a proper TimeDate value. This should happen when the EndRequest event is called. This hasn't been tested and it is NOT RECOMMENDED.

Further updates will be made available on this thread.

@kria
Copy link

kria commented Nov 24, 2016

Are the Azure people looped in? This is getting to be a big problem on there.

@Wesley-GONG
Copy link

This bug is very very very serious, please fix it ASAP, thanks!

@mortb
Copy link

mortb commented Nov 25, 2016

No, it is not only an issue for CDN:s.
We set e.g. [OutputCache(Duration=30)] on a asp.net mvc controller method that via ajax gets a list of news items in our application.
First time this resource is requested we get the 200 OK Cache-Control:maxage=30 but if we re request again the url we get a 304 NotModfied with maxage=3615676478 (roughly 110 years...) In some cases I've seen the high maxage on 200 OK responses as well.
Chrome seems to add this to the cache and subsequent calls in Chrome seems to all be resolved by the client's disk cache (you can see this in Developer tools/F12) I've waited for ten minutes and calls are still resolved by the cache (the timeout should be 30 seconds as stated above). I have not yet waited for 110 years to see if the cache is released...

Behavior between Chrome and IE really isn't the same when it comes to caching.

Now our application is for intranet usage only, so damage is limited if we release it with the cache bug, but would this be a public site that would seem superbad actually...

@ttqaserver
Copy link

ttqaserver commented Dec 5, 2016

Can't use CDN's with such bug. How to downgrade to 4.6.1 for Azure WebApp`s?

@Slowacki
Copy link

Slowacki commented Jan 2, 2017

Is there any progress on this one?

@agrinei
Copy link

agrinei commented Jan 20, 2017

This is critical! A bugfix should be released even before .Net 4.6.3

@skimonkey
Copy link

skimonkey commented Feb 23, 2017

I am getting this exact same issue on a .Net 4.5.2 website hosted on Azure (IIS8).
Setting the outputcache on the website main page controller Index method as
[OutputCache(Duration = 86400 )]

On first request
Cache-Control: public, max-age=86400
on all subsequent requests
Cache-Control: public, max-age=63623538558

On subsequent requests
Chrome sends an If-Modified-Since header so I get a 304 and html is loaded from browser cache so less of an issue here
IE and Firefox dont send any such header so I get a 200 and full response.

Dosent look like this is a 4.6.2 only issue...

Any update on a fix..? Any advice..?
this is costing money as Azure charges on output bandwidth

@kria
Copy link

kria commented Feb 26, 2017

@skimonkey it doesn't matter what version you target, it's still going to run using the current Azure .NET Framework version (4.6.2).

@richlander
Copy link
Member

/cc @glennc

@OnTheMike
Copy link

Seems to be fixed in 4.7, surely took some time https://github.com/Microsoft/dotnet/blob/master/releases/net47/dotnet47-changes.md#aspnet

@NickCraver
Copy link
Member

We discovered this at Stack Overflow as the root cause to many problems. This should absolutely be a hotfix to existing .NET 4.6.2 installs. For many .NET 4.7 will not be an option for some time, and this is a critical bug. I'm honestly shocked that this went unsolved for months, we would never had expected such a known issue to stay open for this long. So many people on 4.6.2 will not ever even realize this is the root cause of some of their issues. We didn't until it was too late. The cache poisoning downstream has already caused severe harm.

As a result of this bug, we have tens of millions of downstream requests that are cached in proxies we don't control for over 2000 years. Only manual purges or volatile cache competition will purge them, which results in impossible to debug or remotely solve scenarios as our users report stale content. This is a huge, HUGE bug. It needs a fix for all ASAP. Though we're willing to deploy .NET 4.7 on a whim, very few people are in that position. They need a hotfix, ASAP.

To work around this at all we have effectively disabled all OutputCache directives by stripping cache control at our proxy layer (Fastly) and removed it from anything not behind Fastly. This costs us in bandwidth, latency, and overall slower loads from uncached local resources for users. No workaround is desirable since this bug is so deep in the framework where you cannot safely solve it through user code. This is costing us dollars and performance every minute we have the workaround active.

@MichalGrzegorzak
Copy link

@NickCraver - I couldn't agree more with you. This is really huge issue, and it`s disappointing that MS seems to not realizing it.

I'm the (un)lucky one on Azure, anyone knows when 4.7 will be on Azure?

@richlander
Copy link
Member

Update on this issue:

Apologies that the fixes are not coming sooner. We're currently testing the 4.6.2 fix right now. We're working with @NickCraver to get some testing in his environment before going broader.

@MichalGrzegorzak Will get back to you on when either 4.7 or this fix for 4.6.2 will be in Azure.

@richlander
Copy link
Member

@MichalGrzegorzak can you tell which Azure service you use? Azure app services deploys .NET Framework differently than Azure VMs, for example.

@richlander
Copy link
Member

Fix is coming, so closing.

@mortb
Copy link

mortb commented Apr 27, 2017

Few things are so out of your control as a cache you cannot purge as a developer. Maybe it would be time for MS to put some effort into writing serious unit tests that detect a problem like this before anything is released?

@NickCraver
Copy link
Member

Since I hit this several times when verifying changes, posting here: curl -I (capital I) to view headers will not reproduce it. Because that makes a HEAD request, it's never a full/cached response to trigger the output cache. You need to make a GET request to verify it's fixed when patches land, like this:

curl -XGET -I http://myserver/test-route

@richlander
Copy link
Member

Fixed: https://blogs.msdn.microsoft.com/dotnet/2017/05/17/net-framework-may-2017-preview-of-quality-rollup/

@NickCraver
Copy link
Member

For others ^ we've deployed this across the Stack Overflow web tiers and haven't found any ill effects. The issue is now resolved here.

@MichalGrzegorzak
Copy link

MichalGrzegorzak commented Jun 12, 2017

@richlander We are using WebApp's. Still can`t find information when Azure will be patched. Any idea?

@offirpeer
Copy link

offirpeer commented Jul 19, 2017

@richlander

This release is no longer recommended / available.
For the link you provided.

@NickCraver we encountered the same problem with our site(eCommerce) and customers were asking for special offers/sales that are no longer exists because they saw them online and we must provide it to them or else they may have a reason to sue us. This bug cost us a lot of money!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests