Intermittent 502 response from Trac

[btlogy]

As mentioned in #4176, some users have confirmed receiving the following answer from ​Trac intermittently:

502 Bad Gateway
nginx/1.10.3 (Ubuntu)

Now the disk space issue has been fixed, it seems like refreshing the page usually works around this 502 problem.

However, as the checks from the Upttime monitoring seems to show, there are cases where a user could not get a proper answer from the server even after 3 retries!

Here are the details about the incidents reported for yesterday 20th of May:

• ​Wiki is down at 07:27 UTC #424

  Checking https://tahoe-lafs.org/trac/tahoe-lafs/wiki
  Result from test 502 0.028272
  Result from test 502 0.016042
  Result from test 502 0.016079
  

• ​Web is down at 08:06 UTC #425

  Checking https://www.tahoe-lafs.org/
  Result from test 502 1.064882
  Result from test 502 0.13437
  Result from test 502 0.13468
  

• ​Wiki is down at 08:21 UTC #426

  Checking https://tahoe-lafs.org/trac/tahoe-lafs/wiki
  Result from test 502 0.169718
  Result from test 502 0.131224
  Result from test 502 0.094317
  

• ​Wiki is down at 10:22 UTC #427

  Checking https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TitleIndex
  Result from test 502 0.063878
  Result from test 502 0.063225
  Result from test 502 0.07016
  

• ​Tickets is down at 10:56 UTC #428

  Checking https://tahoe-lafs.org/trac/tahoe-lafs/query
  Result from test 502 0.652738
  Result from test 502 0.068121
  Result from test 502 0.068075
  

• ​Wiki is down at 18:06 UTC #429

  Checking https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TitleIndex
  Result from test 502 0.05107
  Result from test 502 0.050887
  Result from test 502 0.050857
  

• ​Web is down at 20:21 UTC #430

  Checking https://www.tahoe-lafs.org/
  Result from test 502 1.079979
  Result from test 502 0.031663
  Result from test 502 0.031868
  

• ​Wiki is down at 20:34 UTC #431

  Checking https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TitleIndex
  Result from test 502 0.079088
  Result from test 502 0.065165
  Result from test 502 0.079371
  

• ​Web is down at 21:01 UTC #432

  Checking https://www.tahoe-lafs.org/
  Result from test 502 1.622923
  Result from test 502 0.106893
  Result from test 502 0.160037
  

There are likely more cases which have not raised incident like the following one where one of the retries get the proper answer:

Checking https://tahoe-lafs.org/trac/tahoe-lafs/wiki/TitleIndex
Result from test 502 0.083277
Result from test 502 0.097025
Result from test 200 13.603759

Hopefully those timestamps could help to spot the root cause in the logs of the server.