id summary reporter owner description type status priority milestone component version resolution keywords cc launchpad_bug 1696 attempting more than four simultaneous put uploads seems to break client mirimir somebody "Our test client behaves oddly when doing more than four simultaneous uploads. Our test grid (all 1.9.1) is on a private VPN. A remote dedicated server with one 10Mbps port hosts our OpenVPN server and a storage node, with helper enabled. The introducer, test client and nine other storage nodes are VMs (some local and some remote VPS). Although bandwidths in both directions are typically 1.5-2.5Mbps, latencies are large, with RTT averaging 800ms. The client VM has one CPU, 512MB memory and 1.3GB swap. Versions are: allmydata-tahoe: 1.9.1, foolscap: 0.6.3, pycryptopp: 0.5.25, zfec: 1.4.10, Twisted: 10.1.0, Nevow: 0.10.0, zope.interface: unknown, python: 2.6.6, platform: Linux-Ubuntu_10.10-i686-32bit_ELF, pyOpenSSL: 0.10, simplejson: 2.1.1, pycrypto: 2.0.1, pyasn1: unknown, mock: 0.8.0rc2, sqlite3: 2.4.1 [sqlite 3.7.2], setuptools: 0.6c16dev3 For single puts, the client uploads to the helper at ~70KBps, and the helper pushes to storage servers at ~200KBps nominal (660KBps total). For four simultaneous (forked) puts, uploading plateaus at ~200KBps, and pushing plateaus at ~250KBps nominal. However, for five or more simultaneous puts, something breaks. Although all ten shares of all files do get uploaded, some files don't get linked to the grid directory, and their upload operations are missing from ""Recent Uploads and Downloads"". To create ten 1MB test files, we paste this into terminal: {{{ dd if=/dev/urandom of=1MB1 bs=1K count=1K & dd if=/dev/urandom of=1MB2 bs=1K count=1K & dd if=/dev/urandom of=1MB3 bs=1K count=1K & dd if=/dev/urandom of=1MB4 bs=1K count=1K & dd if=/dev/urandom of=1MB5 bs=1K count=1K & dd if=/dev/urandom of=1MB6 bs=1K count=1K & dd if=/dev/urandom of=1MB7 bs=1K count=1K & dd if=/dev/urandom of=1MB8 bs=1K count=1K & dd if=/dev/urandom of=1MB9 bs=1K count=1K }}} To simultaneously put four files, we paste this into terminal: {{{ ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB1 tahoe:1MB/1MB1 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB2 tahoe:1MB/1MB2 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB3 tahoe:1MB/1MB3 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB4 tahoe:1MB/1MB4 }}} We get this output (excerpted): {{{ 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 }}} Browsing to tahoe:1MB, we see all four files: Tahoe-LAFS Directory SI=foobar {{{ Type Filename Size Times FILE 1MB1 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB2 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB3 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB4 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM }}} To simultaneously put five files, we paste this into terminal: {{{ ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB5 tahoe:1MB/1MB5 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB6 tahoe:1MB/1MB6 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB7 tahoe:1MB/1MB7 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB8 tahoe:1MB/1MB8 & ~/allmydata-tahoe-1.9.1/bin/tahoe put --dir-cap=[Directory writecap] ~/1MB9 tahoe:1MB/1MB9 }}} We get this output (excerpted): {{{ 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 201 Created URI:CHK:...:3:10:1048576 }}} Browsing to tahoe:1MB, we only see one new file (not five): Tahoe-LAFS Directory SI=foobar {{{ Type Filename Size Times FILE 1MB1 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB2 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB3 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB4 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM FILE 1MB8 1048576 lcr: 2012-03-27_HH:MM:SS.MMMMMM lmo: 2012-03-27_HH:MM:SS.MMMMMM }}} " defect new major undecided code 1.9.1 integrity preservation reliability scalability error