allow automatic use of pyOpenSSL 0.14

[daira]

For Tahoe-LAFS 1.11 we decided to fix the pyOpenSSL version requirement to == 0.13 [not quite; see comment:4 below], in order to mitigate problems with building the cryptography library that is a dependency of pyOpenSSL >= 0.14 (see #2193 for details).

This ticket is for a longer-term solution to allow use of pyOpenSSL >= 0.14 without causing build/install regressions (such as #2217, or the requirement to manually install libffi on some platforms).

[glyph]

For what it's worth, you don't need to manually install libffi on the mac, at least; it's bundled with the platform.

[daira]

The solution we eventually arrived at for 1.11 (for all platforms) was to attempt to import the OpenSSL module at build time and check its version; if 0.14 or above is already installed then we allow it, otherwise we use 0.13 or 0.13.1.

This is not entirely satisfactory; it's complicated and means that we don't get pyOpenSSL security fixes unless 0.14+ is installed manually.

[daira]

​https://caremad.io/2014/11/distributing-a-cffi-project/ explains what is wrong with cffi from a packaging point of view. Until those design issues are fixed, we can't make much progress on this ticket.

[glyph]

That post includes workarounds for all of the issues described; it also says that the cryptography project has now already worked around all the issues described therein. Given that Tahoe doesn't make use of cffi directly, it's not clear to me why those particular issues would prevent it from being used indirectly in the project that all the workarounds come from.

That said, certainly the requirement to install libffi is still there.

[daira]

Sorry, I missed the fact that they were already in cryptography (probably because I'd seen some of these problems when building previous versions of it). Which is the first version that has all of these fixes?

[daira]

Also, is it possible to build a cryptography egg that allows building without a compiler or any manually installed dependencies on Windows?

(Note: we can't use wheels because we're stuck with zetuptoolz for the time being.)

[glyph]

I don't know what version the fixes appeared in; they have all trickled in over time. I'm not sure if there's even a complete release with all of them.

I assume it would be possible to build an egg, since the metadata provided is the same as for wheels, but eggs are pretty strongly deprecated at this point, and I'm not sure if you could convince the cryptography team to spend much time investigating it if it doesn't work :-\. It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

[zooko]

Replying to glyph:

It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

Last time I considered it, which was one and a half years ago, we had made seven patches to our fork, zetuptoolz, and I wasn't sure if those issues had also been fixed in setuptools or not. Nowadays I guess I would be unsure if those issues would regress if we switched from zetuptoolz to wheel…

[dstufft]

For what it's worth, cffi is being worked on which fixes most (or all) of the issues in my post natively within cffi instead of relying on the hacks that we have in place in cryptography.

[glyph]

Replying to zooko:

Replying to glyph:

It seems like eliminating zetuptoolz would be a fruitful avenue to pursue; why are you still stuck with it?

Last time I considered it, which was one and a half years ago, we had made seven patches to our fork, zetuptoolz, and I wasn't sure if those issues had also been fixed in setuptools or not. Nowadays I guess I would be unsure if those issues would regress if we switched from zetuptoolz to wheel…

My understanding is that the context of those patches was a fairly unresponsive setuptools development team who wasn't fixing things. The Python Packaging Authority is substantially more responsive to bug reports, and users frequently get updates. Ironically zetuptoolz is now creating the exact situation it was created to avoid :-).

The only setuptools bug referenced by the linked comment is ​http://bugs.python.org/setuptools/issue54, which, if it hasn't been fixed (I think maybe it has? maybe dstufft can opine directly) has been mostly obviated by the ecosystem's consensus converging on virtualenv and related toolchains to produce isolated environments.

I want to say more things about this but I can't even find zetuptoolz's source code; all the old links are broken. Where does it live now?

[zooko]

The links in the comment mentioned above are not broken! They still work. The tahoe-lafs trac appears to be responding slowly though… give them a minute to load?

[daira]

glyph: it's in the Tahoe source tree: ​https://github.com/tahoe-lafs/tahoe-lafs/tree/master/setuptools-0.6c16dev6.egg

Last time I looked, the main change that upstream didn't have was the Windows script support that doesn't require an executable blob.

[glyph]

What do you mean by "require an executable blob"?

[glyph]

Also, I apologize for cluttering up this ticket's comments, we're getting pretty far off track. Should we make a new "eliminate zetuptoolz" ticket?

[daira]

There used to be (don't know whether there still is) a 32-bit Windows executable in the setuptools *source*, with no instructions on how to rebuild it. There was no corresponding 64-bit executable, and the 32-bit one was buggy; it failed to pass on Unicode arguments to the Python process correctly, for example.

[zooko]

Replying to glyph:

Also, I apologize for cluttering up this ticket's comments, we're getting pretty far off track. Should we make a new "eliminate zetuptoolz" ticket?

Here are all of the related tickets.

There are no fewer than six open tickets for wholesale replacement of zetuptoolz with something else. I must confess I no longer understand — if I ever did — which of the following tools are alternatives vs. complementary, which ones are currently maintained, etc.:

• #1582 setuptools delenda est
• #2044 investigate setuptools 0.7+
• #2055 Building tahoe safely is non-trivial
• #2020 write requirements doc for what we want from a package installer
• #2077 pip packaging plan
• #2255 Experiment with virtualenv based packaging in the petmail style.

[zooko]

P.S. Since I no longer understand these issues, I would welcome someone who does, such as dstufft, glyph, daira, warner, etc., taking the lead on proposing a path forward.

[glyph]

I think #1582 is probably the right place to continue discussion, so I will move there. Thanks for the overview.

[rvs]

FWIW, I'm having some problems with installing tahoe with --single-version-externally-managed while using pyOpenSSL 0.15.1: it still complains "pkg_resources.DistributionNotFound?: The 'pyOpenSSL<=0.13.1,>=0.13' distribution was not found and is required by the application"

logs are here:

• ​https://copr-be.cloud.fedoraproject.org/results/vrusinov/sundry/fedora-rawhide-x86_64/00166429-tahoe-lafs/build.log.gz
• ​https://copr-be.cloud.fedoraproject.org/results/vrusinov/sundry/fedora-rawhide-x86_64/00166429-tahoe-lafs/root.log.gz (this one lists all installed packages)

Here's the spec file I try to build: ​https://github.com/vrusinov/copr-sundry/blob/tahoe-lafs/third_party/subtrees/tahoe-lafs/tahoe-lafs.spec

Note that I'm doing something weird before install:

rm -rf setuptools-*.egg
mkdir -p setuptools-0.0.0.egg

But it the only way I was able to build it avoiding weird setuptools version conflicts.

I can live without it now, but it may quickly become very annoying.

[warner]

We no longer pin the pyOpenSSL version as we used to do, nor do we use zetuptoolz. Tahoe now uses the current version of everything.

We currently have multiple pathways that depend on pyopenssl:

• a direct dependency that requires >=0.14
• one through Twisted[tls] that wants >=0.13
• one through Foolscap that doesn't declare a version
• one through Foolscap that wants Twisted[tls]
• one through service-identity that wants pyopenssl >=0.12

After this release, we'll probably get rid of one on service-identity (which is no longer used by twisted; tahoe was depending on it just out of paranoia). We might also get rid of the direct dependency, unless there's something special about 0.14 that we care about more than Twisted or Foolscap.

But in any event, 0.14 is fine, and in fact modern builds will use the current release (which is "16.0.0"). So I'm closing this one out.

[glyph]

For what it's worth, service_identity is still totally imported directly by Twisted. It's part of twisted[tls] though, so the direct dependency shouldn't be necessary.

[warner]

Oops, it's characteristic that's no longer used by Twisted. But yeah, we'll also remove our dependency on service-identity and allow twisted[tls] to be responsible for it.