id summary reporter owner description type status priority milestone component version resolution keywords cc launchpad_bug 4097 1.19.0 node connection issues. tlhonmey "I recently decided to update my grid. It was running a mix of 1.14, 1.15, and 1.17. I had upgraded one of the nodes to 1.19 and it started complaining about SSL bad certificate issues when trying to communicate with other nodes. After some discussion with meejah on IRC, it seemed like the best way to deal with the certificate mismatches was to just rebuild the grid, and then copy in the old storage folder. After rebuilding the grid, things are... Strange. The introducer node, can talk to everyone. That's good. Node No. 1, which is running on the same machine as the introducer, with a different port, can talk to everyone as well. That's good. All the other nodes in the grid can only talk to one or maybe two different nodes, and that doesn't necessarily include themselves for some reason. What's more, the helpful connection error report on the web status page has been replaced with opaque stack traces -- without even any line breaks -- like: {{{ failure: [Failure instance: Traceback: : /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:916:errback /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:984:_startRunCallbacks /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:1078:_runCallbacks /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:1949:_gotResultInlineCallbacks --- --- /home/annie/.local/lib/python3.12/site-packages/twisted/internet/defer.py:1078:_runCallbacks /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:809:convertCancelled /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:292:_cancelledToTimedOutError /home/user/.local/lib/python3.12/site-packages/twisted/python/failure.py:481:trap /home/user/.local/lib/python3.12/site-packages/twisted/python/failure.py:505:raiseException /home/user/.local/lib/python3.12/site-packages/twisted/internet/defer.py:1999:_inlineCallbacks /home/user/.local/lib/python3.12/site-packages/twisted/python/failure.py:519:throwExceptionIntoGenerator /home/user/.local/lib/python3.12/site-packages/allmydata/storage_client.py:1348:_pick_server_and_get_version /home/user/.local/lib/python3.12/site-packages/allmydata/storage_client.py:1338:get_istorage_server ] }}} The stdout of the half-connected nodes contains nothing but messages about factories being started and stopped, with no real indication about why. Meejah seemed to think this may have something to do with GBS. I'd be happy to do some diagnostics if there's some way we can coax something useful out of the system. " defect new normal undecided unknown n/a