Sendpayment crashes with "ValueError: Calculated transaction fee is too large for our inputs;"

[AlexCato]

Needed several attempts to do a sendpayment.py, because the coinjoin always crashed when it tried to create the transactions. On the console it reports:
"ValueError: Calculated transaction fee of: (...) is too large for our inputs;Please try again."

This seems to be the relevant part of the log:

''''
TIMESTAMP_0000000000104 [MainThread ] [DEBUG] got estimated tx bytes: 6388
TIMESTAMP_0000000000105 [MainThread ] [DEBUG] Based on initial guess: 10000, we estimated a fee of: 106430
TIMESTAMP_0000000000106 [MainThread ] [DEBUG] CRASHING, DUMPING EVERYTHING
''''
Here's the full log: sendpayment crash

I only could fix this by manually selecting the parties to coinjoin with. I selected those with high fees, because my guess why this happens is: the cheaper joiners have dozens of dozens of super-small outputs to spend and need many of them to create the desired coinjoin-output-size (in my case, about 2 BTC). This creates a huge transaction with way larger fees than estimated (even setting the tx_fee in joinmarket.cfg to 25 did not help here).
More expensive counterparties probably have less "business" and less scattered mixdepths with bigger concentrations. But there's no way for a taker to know this in advance, he just has to eat those tx-fees unless he bumps into a hardcoded maximum like I did, right?

I dont mind this problem myself, but believe that the default settings should work for the average user. We should make it as convenient and easy as possible for them... maybe let makers also send the single biggest output they could spend to give takers a chance to find makers with fewer required outputs, thus creating a smaller transaction, reducing fees and helping the bitcoin network in the process?

[chris-belcher]

Yeah the miner fees part of joinmarket is a mess :(
The orders should advertise their fee/kb and pay more when they have lots of UTXOs.

[AlexCato]

Two thoughts about this:

An easy fix for the taker scripts: when receiving the transaction data for each maker, calculate the size and appropriate miner fee for that size. Reject all maker-offers which do not add enough miner fees.
Reasoning: this makes sure that per default the taker part works and it's in everyones interest to reduce on-chain data usage (currently makers dont care and takers just pay, they dont even get the miner fee info displayed before agreeing).
On average, the makers are savvy enough to adjust to this change, because everyone will have to add miner fees and increase cj fees to cover for it. Takers can more easily choose the truly cheaper offer. Currently, cj fees dont play any role in any join <10 BTC anyways, because miner fees are much higher than the cj fees.

Second thought: might this be the reason why I see so many big transactions fail in my yield generator log? There's tons of < 0.1 BTC joins and most of them go through just fine (guess 80-90% success rate after a taker initializes a join). The few bigger joins from 3-10 BTC seem to have a much lower success rate: less than half of them are being broadcast to the network after the coinjoin-part is completed successfully, i.e. I see the "potentially earned = XXX" line in the debug log, but nothing happens after it. Guess they might run into the problem described here.

[AlexCato]

If the "taker pays for all"-model should be maintained, the script probably should not crash but restart and blacklist the makers which had too many UTXOs.

That would serve three purposes: better user experience (no crash obviously), reduce overall coinjoin costs (since TXFEEs >>> cjfees for probably 99% of all coinjoins) and help the bitcoin network (smaller on-chain transactions).

Edit: I'm not good with python (coming from the C++ side of things) but would try to implement this in a PR if you agree one of the high-level ways you prefer to see this handled?

[chris-belcher]

Probably the easiest thing now is to have the taker count the UTXOs, that makes #156 worse because the taker obtains UTXOs without using them. But until we update the protocol it's probably the best way.

We should also write a test for #414 and merge it so makers can have the number of their UTXOs reduced automatically.

[AlexCato]

#552 fixes this.