Downloading pysqlite takes quite a long time

[zooko]

NODA, Kai reported to the list:

​http://allmydata.org/pipermail/tahoe-dev/2010-January/003657.html

His full message is appended below.

http://pysqlite.org is still unreachable at this time. I don't think there is a good solution to this problem in Tahoe-LAFS's packaging, though. The only good solution that I can think of is for the code which downloads dependencies to try all sources in parallel instead of in series, so that if one of the sources waits for minutes before timing out, the download can proceed from the other sources.

So that would be an issue for the developers of setuptools and/or "distribute" (the Python packaging module) to address.

Oh, there is another solution that we could implement in terms of Tahoe-LAFS packaging: host copies of all our dependencies on a Tahoe-LAFS grid which is highly available! :-)

On machines with Python 2.4.x (CentOS servers,) setup.py wants to
download pysqlite>=2.0.5 as an external package.
My problem is that installation take as long as twenty minutes.
I must endure so many times of timeouts during name resolution of
"pysqlite.org" which seems expired.

One way to circumvent this annoyance is installing pysqlite beforehand
by easy_install.
To my surprise, installation in this way takes less than two minutes,
because timeout of resolving "pysqlite.org" occurs only once.

My question is that what causes this difference, and whether there are
any ways to skip "pysqlite.org" during installation by setup.py .

Thanks,
Kai

P.S. Lengthy logs are pasted below:

=== 1. USING SETUP.PY ===

$ uname -a
Linux [snip] 2.6.18-128.el5 #1 SMP Wed Jan 21 10:41:14 EST 2009 x86_64 x86_64 x86_64 GNU/Linux
$ cat /etc/redhat-release
CentOS release 5.4 (Final)
$ python -V
Python 2.4.3
$ python setup.py build
[snip]
Processing dependencies for allmydata-tahoe==0.0.0
Searching for pysqlite>=2.0.5
Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-sdists/
Reading http://allmydata.org/source/tahoe/deps/tahoe-dep-eggs/
Reading http://pypi.python.org/simple/pysqlite/
Reading http://pysqlite.sourceforge.net/
Reading http://pysqlite.googlecode.com/
Reading http://code.google.com/p/pysqlite/downloads/list
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://initd.org/tracker/pysqlite/wiki/PysqliteDownloads
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://initd.org/pub/software/pysqlite/releases/2.3/2.3.3/
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://initd.org/pub/software/pysqlite/releases/2.3/2.3.4/
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://oss.itsystementwicklung.de/trac/pysqlite
Reading http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.0/
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://oss.itsystementwicklung.de/download/pysqlite/2.4/2.4.1/
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://initd.org/pub/software/pysqlite/releases/2.4/2.4.0/
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://initd.org/pub/software/pysqlite/releases/2.3/2.3.5/
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Best match: pysqlite 2.5.6
Downloading http://pysqlite.googlecode.com/files/pysqlite-2.5.6.tar.gz
Processing pysqlite-2.5.6.tar.gz
Running pysqlite-2.5.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-3hF-OG/pysqlite-2.5.6/egg-dist-tmp-JWPadv
  File "build/bdist.linux-x86_64/egg/pysqlite2/test/py25tests.py", line 48
    with self.con:
            ^
SyntaxError: invalid syntax
zip_safe flag not set; analyzing archive contents...
  File "/var/tmp/k_noda/tahoe-HEAD/support/lib/python2.4/site-packages/pysqlite-2.5.6-py2.4-linux-x86_64.egg/pysqlite2/test/py25tests.py", line 48
    with self.con:
            ^
SyntaxError: invalid syntax
Adding pysqlite 2.5.6 to easy-install.pth file

Installed /var/tmp/k_noda/tahoe-HEAD/support/lib/python2.4/site-packages/pysqlite-2.5.6-py2.4-linux-x86_64.egg
[snip]
python setup.py build  78.14s user 7.64s system 6% cpu 21:42.50 total

=== 2. USING EASY_INSTALL ===

Here ~/vt is a symlink to /var/tmp/k_noda/ .

$ cat ~/.pydistutils.cfg
[install]
install_lib = ~/vt/local/lib64/python2.4/site-packages
install_scripts = ~/vt/local/bin
$ sh setuptools-0.6c11-py2.4.egg
[snip]
$ time easy_install pysqlite
Searching for pysqlite
Reading http://pypi.python.org/simple/pysqlite/
Reading http://pysqlite.sourceforge.net/
Reading http://pysqlite.googlecode.com/
Reading http://code.google.com/p/pysqlite/downloads/list
Reading http://pysqlite.org/
Download error: (-3, 'Temporary failure in name resolution') -- Some packages may not be found!
Reading http://initd.org/tracker/pysqlite/wiki/PysqliteDownloads
Reading http://initd.org/pub/software/pysqlite/releases/2.3/2.3.3/
Reading http://initd.org/pub/software/pysqlite/releases/2.3/2.3.4/
Reading http://oss.itsystementwicklung.de/trac/pysqlite
Reading http://oss.itsystementwicklung.de/download/pysqlite/2.5/2.5.0/
Reading http://oss.itsystementwicklung.de/download/pysqlite/2.4/2.4.1/
Reading http://initd.org/pub/software/pysqlite/releases/2.4/2.4.0/
Reading http://initd.org/pub/software/pysqlite/releases/2.3/2.3.5/
Best match: pysqlite 2.5.6
Downloading http://pysqlite.googlecode.com/files/pysqlite-2.5.6.tar.gz
Processing pysqlite-2.5.6.tar.gz
Running pysqlite-2.5.6/setup.py -q bdist_egg --dist-dir /tmp/easy_install-YdjN2D/pysqlite-2.5.6/egg-dist-tmp-BTWdyF
  File "build/bdist.linux-x86_64/egg/pysqlite2/test/py25tests.py", line 48
    with self.con:
            ^
SyntaxError: invalid syntax
zip_safe flag not set; analyzing archive contents...
Adding pysqlite 2.5.6 to easy-install.pth file

Installed /var/tmp/k_noda/local/lib64/python2.4/site-packages/pysqlite-2.5.6-py2.4-linux-x86_64.egg
Processing dependencies for pysqlite
Finished processing dependencies for pysqlite
easy_install pysqlite  1.70s user 0.33s system 2% cpu 1:09.45 total

[zooko]

Ah, the idea of hosting dependencies on a Tahoe-LAFS grid is #566 (host tahoe dependencies).

[davidsarah]

Duplicate of #566.