Let’s create a routing LND node, part 4 - Unsuccessful looping and rebalancing

Series: Let’s create a routing LND node

Supporting this series

If you want to support this series, you can:

  • Use tippin.me to send me a tip.
  • Use Tootmoney to ask for a Lightning invoice. On Mastodon, just send a toot like ‘@OpinionatedGeek #TootMoney #send 10000’ to have it reply with an invoice QR code.
  • Send bitcoin to this address: bc1qsaampnjm5ykcpd40pstcewt75xrtfy5cxpnpdf

Rebalancing and looping

OK, now we’re getting to the interesting stuff. Now that I have some channels open I need to balance them so there’s liquidity on both sides.

Right now the channels I opened have 100% liquidity on my side and 0% liquidity on the other side. That means I can transfer funds out from my node to the other nodes but I can’t accept anything coming in on those channels. In order to provide a routing service I’d like to have bidirectional routing so I can transfer funds in either direction.

For now I’m aiming for a balance of 50% on both sides. I may be able to make smarter decisions about this for certain nodes in the future (some nodes may be particularly popular as destinations, some particularly popular as sources) but for now 50% seems like a decent starting point.

How do I get a channel to 50% on both sides without just giving 50% of the funds away? There are two common options available to me:

  • Lightning Labs’ Loop
  • Sending funds on a ‘circular route’

Loop

Loop is a component and on-chain service provided by Lightning Labs to allow you to take funds out of your channels. (The ability to add funds to a channel is in-progress but it’s not available right now.) There’s quite a bit of detail to dig into if you like.

The big downside to loop (as I understand it) is the fees. Loop requires some on-chain transactions and also can tie up some of Lightning’s capital in timeouts so it’s not a cheap option.

Here are the current terms:

$ loop terms Loop Out -------- Amount: 250000 - 1000000 Fee: 7810 + 0.0000 % (16006 prepaid) Cltv delta: 50 blocks Loop In ------ rpc error: code = Unknown desc = function not available on mainnet

I plan on using loop, but it's not my first-choice tool. If I can manage things without on-chain transactions that should be cheaper overall.

But for now I have a lot of open channels that are full, so let’s give loop a go.

Let’s start with BitRefill. I have 4million sats on one side and 0 sats on the other.

✅ 👪 Bitrefill.com: 631589165900038144 (Last update: 12:50 UTC on Friday, 3 May 2019) Pubkey: 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f Fees: 1,000+1/40 / 4,000+40/144 Balances: [🔷🔷🔷🔷🔷🔷🔷🔷🔷🔷] 3,990,856 / 0 [capacity: 4,000,000]

$ loop out --channel 631589165900038144 --amt 1000000 Max swap fees for 1000000 Loop Out: 46858 CONTINUE SWAP? (y/n), expand fee detail (x): y Swap initiated ID: 03de8f654e7dc866e0a7c0fc796a05bdda005a625afd2c2bd453ebf98fec372d HTLC address: bc1qtzl03jzvqfz4f38tvq2rhwejjgtn6svw8yk3kv6t3w7qq07jwsfq3ntmrz Run `loop monitor` to monitor progress.

The cost of using loop for 1,000,000sats was (up to) 46858sats - or about $2.65 at today’s conversion rate. (For reference, opening a channel today cost about $0.50.)

$ loop monitor 2019-05-03T16:20:24Z LOOP_OUT INITIATED 0.01 BTC - bc1qtzl03jzvqfz4f38tvq2rhwejjgtn6svw8yk3kv6t3w7qq07jwsfq3ntmrz 2019-05-03T16:22:21Z LOOP_OUT FAILED 0.01 BTC - bc1qtzl03jzvqfz4f38tvq2rhwejjgtn6svw8yk3kv6t3w7qq07jwsfq3ntmrz

Well that’s not good. Here’s what the logs say:

2019-05-03 16:20:24.269 [INF] LOOPD: Loop out request received 2019-05-03 16:20:24.274 [INF] CLIENT: LoopOut 0.01 BTC to bc1ql5dasgqmhs9grk37f5mnjc6xn4e8xnc0253ash (channel: 0xc00016af90) 2019-05-03 16:20:24.279 [INF] CLIENT: Initiating swap request at height 574454 2019-05-03 16:20:24.933 [INF] CLIENT: 03de8f Htlc address: bc1qtzl03jzvqfz4f38tvq2rhwejjgtn6svw8yk3kv6t3w7qq07jwsfq3ntmrz 2019-05-03 16:20:24.937 [INF] CLIENT: 03de8f state Initiated 2019-05-03 16:20:24.937 [INF] CLIENT: 03de8f Sending swap payment lnbc9931700n1pwvc6jgpp5q00g7e2w0hyxdc98cr78j6s9hhdqqknztt7jc275204lnrlvxuksdq8wdmkzuqcqzryxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z96xqqqg9cqqyqqqqlgqqqqqqgq9qrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z9aryqq3fqqqqqqqqlgqqqqqqgq9qrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z9t3vqq8dsqqqqqqqlgqqqqqqgqjqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z9aryqq38qqqyqqqqlgqqqqqqgq9q49c8x2zm2mm3ruy25c0z3zfqghze6ga524vyppqknqlsg3ed9n9j75frlfds8nulkdyjlsm7zqcn5jj436th3mal8xtr5d493r9302gqalz388 2019-05-03 16:20:24.937 [INF] CLIENT: 03de8f Sending prepayment lnbc133390n1pwvc6jgpp57z0p09eh0w06s4kcwdn7j5cdlds3a5ngte35s4nz3newarq9edzsdq2wpex2urp0ycqzryxq97zvuqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z96xqqqg9cqqyqqqqlgqqqqqqgq9qrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z9aryqq3fqqqqqqqqlgqqqqqqgq9qrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z9t3vqq8dsqqqqqqqlgqqqqqqgqjqrzjqgwf02g2gy0l9vgdc25wxt0z72wjlfyagxlmk54ag9hyvrdsw37s6z9aryqq38qqqyqqqqlgqqqqqqgq9qs98eyl3uvzjk3hjkzzzttr6wxl0s8tgudcmthaftglznyyhgy8wz4hy0p8j5rrx9vg9evuwusuc4tahunc2q2r5f67teqjw5mkp7s3sp59lv06 2019-05-03 16:20:24.938 [INF] CLIENT: 03de8f Register conf ntfn for swap script on chain (hh=574454) 2019-05-03 16:20:24.938 [INF] CLIENT: 03de8f Checking preimage reveal height 574484 exceeded (height 574454) 2019-05-03 16:20:24.938 [INF] CLIENT: 03de8f Waiting for either htlc on-chain confirmation or off-chain payment failure 2019-05-03 16:20:32.826 [INF] LOOPD: Monitor request received 2019-05-03 16:21:27.759 [WRN] LNDCLIENT: Payment 03de8f654e7dc866e0a7c0fc796a05bdda005a625afd2c2bd453ebf98fec372d failed: payment attempt not completed before timeout of 1m0s 2019-05-03 16:21:27.760 [INF] CLIENT: 03de8f Failed swap payment: payment attempt not completed before timeout of 1m0s 2019-05-03 16:21:27.760 [INF] CLIENT: 03de8f Wait for server pulling off-chain payment(s) 2019-05-03 16:22:21.043 [WRN] LNDCLIENT: Payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: payment attempt not completed before timeout of 1m0s 2019-05-03 16:22:21.043 [INF] CLIENT: 03de8f Prepayment failed: payment attempt not completed before timeout of 1m0s 2019-05-03 16:22:21.043 [INF] CLIENT: 03de8f Swap completed: FailOffchainPayments (final cost: server 0 BTC, onchain 0 BTC) 2019-05-03 16:22:21.049 [INF] CLIENT: 03de8f state FailOffchainPayments

Hmm. I don’t know what happened there. It tried to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45, which failed, so let’s see what LND says about that payment:

$ docker logs --tail 2000 lnd_lnd_1 | grep f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 2019-05-03 16:20:29.345 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:31.121 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:32.244 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate)(0xc0028600b0)({ 2019-05-03 16:20:33.405 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate)(0xc002860000)({ 2019-05-03 16:20:34.676 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate)(0xc002860000)({ 2019-05-03 16:20:36.273 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate)(0xc002860000)({ 2019-05-03 16:20:38.151 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:40.012 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:42.433 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:44.786 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:45.704 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: TemporaryChannelFailure(update=(*lnwire.ChannelUpdate)(0xc002860000)({ 2019-05-03 16:20:48.796 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) { 2019-05-03 16:20:51.198 [ERR] CRTR: Attempt to send payment f09e1797377b9fa856d87367e9530dfb611ed2685e634856628cf2ee8c05cb45 failed: FeeInsufficient(htlc_amt==13340013 mSAT, update=(lnwire.ChannelUpdate) {

Looks to me like there were 13 different attempts to send the payment to the swap server and they all failed, with either TemporaryChannelFailure or FeeInsufficient.

TemporaryChannelFailure happens all the time as far as I can tell. It usually means one of the hops on the route was offline or had some other problem, and the payment couldn’t get past it.

FeeInsufficient is maybe a bit more interesting. It shouldn’t happen because the nodes all declare the fees they charge as part of their channel policies, but it seems pretty common for this data to become outdated. If someone updates the fee policy on their channel, that update has to make it out across all the nodes on the network and that can take time. In the meantime nodes that haven’t received the updated policy are calculating the fees they need to pay based on old data, and (as in this case) don’t send enough to cover the required fees.

At least I think that’s what’s going on - do you know different?

Circular routes

So how can I try to maintain channels with a balance of inbound and outbound liquidity if I don’t want to pay the on-chain costs of loop? That’s easy in principle and hard in practice.

The idea is that you pay money out on a channel of yours that has too much liquidity, and have it come back to you through a channel with too little liquidity. You build a ‘circular route’ for the money to flow so that all of it (apart from routing fees) comes back to you.

This is great because all you pay are routing fees and they’re usually much cheaper than on-chain transaction fees.

The problems with this are:

  • You need a channel with too much liquidity.
  • You need a channel that can accept enough of that liquidity.
  • LND doesn’t provide any native support for calculating circular routes. (It does provide sendtoroute which can send the funds on the circular route, but it doesn’t help with figuring out what that circular route is.)

Good news though! Someone opened a channel with me! I’ve no idea who it was, or if it’s an automated thing, but I now have this channel:

✅ 🚶 03f175627ecd2b736fdd: 631605658598703105 (Last update: 14:20 UTC on Friday, 3 May 2019) Pubkey: 03f175627ecd2b736fdd29f40414c02169716f884b4b45762483142ed81834141c Fees: 1,000+1/40 / 1,000+1/40 Balances: [🔸🔸🔸🔸🔸🔸🔸🔸🔸🔸] 0 / 1,135,116 [capacity: 1,144,265]

It seems a pretty new node - it hasn’t made it to 1ml.com yet, and it’s not even in my own channel graph when I try to shift some balances around:

$ lntoolkit move-balance --from 631562777641943040 --to 631605658598703105 --amount 500000 2019/05/03 15:29:36 🚨 Exiting with error: finding 'To' edge: could not find ChannelEdge with channel ID '631605658598703105'

So, not a resounding success today. I failed to ‘loop out’ from multiple channels, and I failed to move balances because the only channel I have that could accept liquidity seems to only partially exist.

It looks like I have quite a few issues to work through now! Any advice?

Also, if you want to open a channel with my node, it is:

023c5b5667b16cd7fcca5591a8c0f47beb76c9405e16a4f2d6b42c7b9904a7f0e6@95.179.191.59:9735

Here’s the latest report on my node:

OpinionatedGeek ⚡ - synced: ✅

Total Balance: 49,878,288 satoshi, Wallet: 37,966,360 satoshi (37,966,360 satoshi/0 satoshi)

Fees: 0 this month, 0 this week, 0 today

Channels: 7 active, 2 inactive, 0 pending, containing: 11,911,928 satoshi

Version: 0.6.0-beta commit=v0.6-beta-dirty

URI: 023c5b5667b16cd7fcca5591a8c0f47beb76c9405e16a4f2d6b42c7b9904a7f0e6@95.179.191.59:9735

Pending:

None.

Unbalanced - High:

⭕ 👪 Bitrefill.com: 631589165900038144 (Last update: 21:45 UTC on Friday, 3 May 2019)

Pubkey: 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f

Fees: 1,000+1/40 / 4,000+40/144

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 3,990,856 / 0 [capacity: 4,000,000]

✅ 👪 powernode.io: 631589165899972608 (Last update: 17:44 UTC on Friday, 3 May 2019)

Pubkey: 02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84

Fees: 1,000+1/40 / 0+100/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 1,988,309 / 0 [capacity: 2,000,000]

⭕ 👪 BitMEXResearch: 631655136611991552 (Last update: 08:20 UTC on Saturday, 4 May 2019)

Pubkey: 0395033b252c6f40e3756984162d68174e2bd8060a129c0d3462a9370471c6d28f

Fees: 1,000+1/40 / 1+10/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 991,192 / 0 [capacity: 1,000,000]

✅ 👪 ln1.satoshilabs.com: 631652937637363712 (Last update: 00:00 UTC on Saturday, 4 May 2019)

Pubkey: 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4

Fees: 1,000+1/40 / 0+1,000/144

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,335 / 0 [capacity: 1,000,000]

✅ 👪 CoinGate: 631562777641943040 (Last update: 09:05 UTC on Friday, 3 May 2019)

Pubkey: 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 👪 rompert.com🔵: 631590265456623616 (Last update: 07:12 UTC on Saturday, 4 May 2019)

Pubkey: 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b

Fees: 1,000+1/40 / 0+1/144

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 👪 lightning-roulette.com: 631563877105729536 (Last update: 21:45 UTC on Friday, 3 May 2019)

Pubkey: 031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 👪 LightningPowerUsers.com: 631590265456689152 (Last update: 06:44 UTC on Saturday, 4 May 2019)

Pubkey: 0331f80652fb840239df8dc99205792bba2e559a05469915804c08420230e23c7c

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

Unbalanced - Low:

✅ 🚶 03f175627ecd2b736fdd: 631605658598703105 (Last update: 14:20 UTC on Friday, 3 May 2019)

Pubkey: 03f175627ecd2b736fdd29f40414c02169716f884b4b45762483142ed81834141c

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥕🥕🥕🥕🥕🥕🥕🥕🥕🥕] 0 / 1,134,097 [capacity: 1,144,265]

All Channels:

⭕ 👪 Bitrefill.com: 631589165900038144 (Last update: 21:45 UTC on Friday, 3 May 2019)

Pubkey: 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f

Fees: 1,000+1/40 / 4,000+40/144

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 3,990,856 / 0 [capacity: 4,000,000]

✅ 👪 powernode.io: 631589165899972608 (Last update: 17:44 UTC on Friday, 3 May 2019)

Pubkey: 02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84

Fees: 1,000+1/40 / 0+100/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 1,988,309 / 0 [capacity: 2,000,000]

⭕ 👪 BitMEXResearch: 631655136611991552 (Last update: 08:20 UTC on Saturday, 4 May 2019)

Pubkey: 0395033b252c6f40e3756984162d68174e2bd8060a129c0d3462a9370471c6d28f

Fees: 1,000+1/40 / 1+10/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 991,192 / 0 [capacity: 1,000,000]

✅ 👪 ln1.satoshilabs.com: 631652937637363712 (Last update: 00:00 UTC on Saturday, 4 May 2019)

Pubkey: 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4

Fees: 1,000+1/40 / 0+1,000/144

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,335 / 0 [capacity: 1,000,000]

✅ 👪 CoinGate: 631562777641943040 (Last update: 09:05 UTC on Friday, 3 May 2019)

Pubkey: 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 👪 rompert.com🔵: 631590265456623616 (Last update: 07:12 UTC on Saturday, 4 May 2019)

Pubkey: 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b

Fees: 1,000+1/40 / 0+1/144

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 👪 lightning-roulette.com: 631563877105729536 (Last update: 21:45 UTC on Friday, 3 May 2019)

Pubkey: 031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 👪 LightningPowerUsers.com: 631590265456689152 (Last update: 06:44 UTC on Saturday, 4 May 2019)

Pubkey: 0331f80652fb840239df8dc99205792bba2e559a05469915804c08420230e23c7c

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 988,309 / 0 [capacity: 1,000,000]

✅ 🚶 03f175627ecd2b736fdd: 631605658598703105 (Last update: 14:20 UTC on Friday, 3 May 2019)

Pubkey: 03f175627ecd2b736fdd29f40414c02169716f884b4b45762483142ed81834141c

Fees: 1,000+1/40 / 1,000+1/40

Balances: [🥕🥕🥕🥕🥕🥕🥕🥕🥕🥕] 0 / 1,134,097 [capacity: 1,144,265]

Tags: Lightning
Created by on Logo15659OpinionatedGeek Ltd.Logo15659