Let’s create a routing LND node, part 13 - Loop problems

Series: Let’s create a routing LND node

Supporting this series

If you want to support this series, you can:

  • Use tippin.me to send me a tip.
  • Use Tootmoney to ask for a Lightning invoice. On Mastodon, just send a toot like ‘@OpinionatedGeek #TootMoney #send 10000’ to have it reply with an invoice QR code.
  • Send bitcoin to this address: bc1qsaampnjm5ykcpd40pstcewt75xrtfy5cxpnpdf

Loop problems

Some Loop Outs were still sitting at ‘INITIATED’ this morning, and I was seeing some odd behaviour in some channels:

⭕ 👪 LightningTo.Me: 632050960802447360 (Last update: 23:45 UTC on Thursday, 9 May 2019)

Pubkey: 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d

Fees: 0+10/40 / 1,000+1/40

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 489,395 / 0 [capacity: 1,000,000]

A local balance of 489,395 sats and a remote balance of 0 sats, but with a capacity of 1,000,000 sats? Hmm. The logs showed the outgoing prepayment had failed (with an UnknownPaymentHash error...) but the main transfer seemed to be stuck in limbo.

I restarted the Loop server, and when I next ran loop monitor some of the INITIATEDs had changed to FAILED. I’d hoped this would unfreeze things and allow the main payment to roll back too but no - now the channel was force-closing!

I really don’t know what’s going on there.

If it was just this channel I’d think it might just be the other party forcing the channel closed, but I have two channels force-closing now and another one with the frozen balance like the above.

So I raised an issue about this on Github. There’s lots of log details there if you’re interested. I’d love to know what was going on.

Part of the problem is it’s hard to trace what has happened. I just don’t have enough experience tracing such things yet, but also it is awkward for the uninitiated. For instance, when you loop out you specify a channel ID like 632050960802447360. The logs don’t show this though. Logs show the hex ID like 0xc00050b4d0 (or sometimes the channel point) so searching for the channel ID you used is useless.

The logs show the full payments that were sent, so you can decode them. But they don’t appear in the list of payments (at least as far as I can tell). Should they be there? Well, one has failed, so maybe not. The other has not yet succeeded or failed (I think), so maybe not. But I don’t know for sure that they shouldn’t appear here, and if they don’t appear here I’m not sure where else I can look for them.

I need more practice at this.

So I did spend some of today trying, and I learned where more things were. But not enough to help me figure out what’s going on.

I’m reluctant to do too much else with my channels right now. I’m not sure what’s happening with some of the Loop Outs and I don’t want to confuse things further.

I did update the lntoolkit report command to retrieve and show the forwards each channel has done that month/week/day. I like the idea of this but I don’t think I’ve got the formatting right yet.

Here’s the latest report:

OpinionatedGeek ⚡ - synced: ✅

Total Balance: 46,768,989 satoshi, Wallet: 17,474,906 satoshi (17,474,906 satoshi/0 satoshi)

Fees: 10 this month, 6 this week, 0 today

Channels: 17 active, 3 inactive, 0 pending, containing: 29,294,083 satoshi

Version: 0.6.1-beta commit=v0.6.1-beta-dirty

URI: 023c5b5667b16cd7fcca5591a8c0f47beb76c9405e16a4f2d6b42c7b9904a7f0e6@95.179.191.59:9735

Pending - Open:

None.

Pending - Close:

None.

Pending - Force Close:

⭕ 🚶 LightningTo.Me: 0 (Last update: 00:45 BST on Friday, 10 May 2019)

Pubkey: 03bb88ccc444534da7b5b64b4f7b15e1eccb18e102db0e400d4b9cfe93763aa26d

Fees: --+--/-- / --+--/--

Forwards: None

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 489,395 / 0 [capacity: 1,000,000]

⭕ 🚶 ln.BitSoapBox.com: 0 (Last update: 13:55 BST on Monday, 13 May 2019)

Pubkey: 021607cfce19a4c5e7e6e738663dfafbbbac262e4ff76c2c9b30dbeefc35c00643

Fees: --+--/-- / --+--/--

Forwards: None

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 370,312 / 0 [capacity: 1,000,000]

Pending - Waiting To Close:

None.

Unbalanced - High:

✅ 👪 ln.taborsky.cz: 632189499363622912 (Last update: 20:00 BST on Thursday, 9 May 2019)

Pubkey: 0265fae305778b7cb157365f70cf3a2047d2cad5c1ccc5f550c6d8a033084a8ea5

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 VIX18: 632187300238983168 (Last update: 05:21 BST on Thursday, 9 May 2019)

Pubkey: 03f8ab9b7496be92ce370e0224661d5d2de80bca8f505d66f1226d0e6002007d2a

Fees: 0+10/40 / 1,000+100/144

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 ECRYPTO[LND]: 632502860122357761 (Last update: 04:45 BST on Sunday, 12 May 2019)

Pubkey: 03d40870a931fe2e978f01b26117b64d15d68119f2840c30f1b69b59955b1e7846

Fees: 1,000+1/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 BitMEXResearch: 631655136611991552 (Last update: 11:54 BST on Sunday, 12 May 2019)

Pubkey: 0395033b252c6f40e3756984162d68174e2bd8060a129c0d3462a9370471c6d28f

Fees: 0+10/40 / 1+10/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 SLL: 632187300239048704 (Last update: 16:23 BST on Monday, 13 May 2019)

Pubkey: 039401f72bc0d40efb58b01de15527a2a5ae1943d7c29067b725a1467a93c7e66f

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 hirish-ln: 632189499363557376 (Last update: 01:54 BST on Monday, 13 May 2019)

Pubkey: 036d81bb87af589f949dec53af1b3121c84048d0913eb56945cbbb62c5cb4fcb6f

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

⭕ 👪 1ML.com node ALPHA: 632050960770990080 (Last update: 07:02 BST on Monday, 13 May 2019)

Pubkey: 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266

Fees: 0+10/40 / 1,000+1/144

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 979,875 / 0 [capacity: 1,000,000]

✅ 👪 rompert.com🔵: 631590265456623616 (Last update: 05:55 BST on Monday, 13 May 2019)

Pubkey: 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b

Fees: 0+10/40 / 0+1/144

Forwards: Monthly In: 362,540 [13], Out: 96,415 [1], Weekly In: 362,540 [13], Out: 0 [0], Daily In: 55,761 [2], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥕] 948,114 / 34,539 [capacity: 1,000,000]

✅ 👪 021c97a90a411ff2b10d: 632366520641650689 (Last update: 23:27 BST on Thursday, 9 May 2019)

Pubkey: 021c97a90a411ff2b10dc2a8e32de2f29d2fa49d41bfbb52bd416e460db0747d0d

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 362,540 [13], Weekly In: 0 [0], Out: 362,540 [13], Daily In: 0 [0], Out: 55,761 [2]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥕🥕🥕] 13,264,155 / 3,495,713 [capacity: 16,777,215]

✅ 👪 LightningPowerUsers.com: 631590265456689152 (Last update: 13:44 BST on Monday, 13 May 2019)

Pubkey: 0331f80652fb840239df8dc99205792bba2e559a05469915804c08420230e23c7c

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 289,984 [1], Out: 0 [0], Weekly In: 289,984 [1], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥕🥕🥕] 780,578 / 202,075 [capacity: 1,000,000]

Unbalanced - Low:

⭕ 🚶 03f175627ecd2b736fdd: 631605658598703105 (Last update: 15:20 BST on Friday, 3 May 2019)

Pubkey: 03f175627ecd2b736fdd29f40414c02169716f884b4b45762483142ed81834141c

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 96,416 [1], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥕🥕🥕🥕🥕🥕🥕🥕🥕🥕] 96,416 / 1,027,723 [capacity: 1,144,265]

All Channels:

✅ 👪 021c97a90a411ff2b10d: 632366520641650689 (Last update: 23:27 BST on Thursday, 9 May 2019)

Pubkey: 021c97a90a411ff2b10dc2a8e32de2f29d2fa49d41bfbb52bd416e460db0747d0d

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 362,540 [13], Weekly In: 0 [0], Out: 362,540 [13], Daily In: 0 [0], Out: 55,761 [2]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥕🥕🥕] 13,264,155 / 3,495,713 [capacity: 16,777,215]

✅ 👪 Bitrefill.com: 631589165900038144 (Last update: 04:16 BST on Monday, 13 May 2019)

Pubkey: 030c3f19d742ca294a55c00376b3b355c3c90d61c6b6b39554dbc7ac19b141c14f

Fees: 0+10/40 / 4,000+20/144

Forwards: Monthly In: 0 [0], Out: 289,982 [1], Weekly In: 0 [0], Out: 289,982 [1], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥕🥕🥕🥕] 2,700,600 / 1,282,053 [capacity: 4,000,000]

✅ 👪 powernode.io: 631589165899972608 (Last update: 16:18 BST on Monday, 13 May 2019)

Pubkey: 02809e936f0e82dfce13bcc47c77112db068f569e1db29e7bf98bcdd68b838ee84

Fees: 0+10/40 / 0+100/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥕🥕🥕🥕🥕🥕] 990,592 / 992,061 [capacity: 2,000,000]

⭕ 🚶 03f175627ecd2b736fdd: 631605658598703105 (Last update: 15:20 BST on Friday, 3 May 2019)

Pubkey: 03f175627ecd2b736fdd29f40414c02169716f884b4b45762483142ed81834141c

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 96,416 [1], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥕🥕🥕🥕🥕🥕🥕🥕🥕🥕] 96,416 / 1,027,723 [capacity: 1,144,265]

✅ 👪 VIX18: 632187300238983168 (Last update: 05:21 BST on Thursday, 9 May 2019)

Pubkey: 03f8ab9b7496be92ce370e0224661d5d2de80bca8f505d66f1226d0e6002007d2a

Fees: 0+10/40 / 1,000+100/144

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 ECRYPTO[LND]: 632502860122357761 (Last update: 04:45 BST on Sunday, 12 May 2019)

Pubkey: 03d40870a931fe2e978f01b26117b64d15d68119f2840c30f1b69b59955b1e7846

Fees: 1,000+1/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 BitMEXResearch: 631655136611991552 (Last update: 11:54 BST on Sunday, 12 May 2019)

Pubkey: 0395033b252c6f40e3756984162d68174e2bd8060a129c0d3462a9370471c6d28f

Fees: 0+10/40 / 1+10/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 SLL: 632187300239048704 (Last update: 16:23 BST on Monday, 13 May 2019)

Pubkey: 039401f72bc0d40efb58b01de15527a2a5ae1943d7c29067b725a1467a93c7e66f

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 hirish-ln: 632189499363557376 (Last update: 01:54 BST on Monday, 13 May 2019)

Pubkey: 036d81bb87af589f949dec53af1b3121c84048d0913eb56945cbbb62c5cb4fcb6f

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

✅ 👪 ln.taborsky.cz: 632189499363622912 (Last update: 20:00 BST on Thursday, 9 May 2019)

Pubkey: 0265fae305778b7cb157365f70cf3a2047d2cad5c1ccc5f550c6d8a033084a8ea5

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 982,654 / 0 [capacity: 1,000,000]

⭕ 👪 1ML.com node ALPHA: 632050960770990080 (Last update: 07:02 BST on Monday, 13 May 2019)

Pubkey: 0217890e3aad8d35bc054f43acc00084b25229ecff0ab68debd82883ad65ee8266

Fees: 0+10/40 / 1,000+1/144

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 979,875 / 0 [capacity: 1,000,000]

✅ 👪 rompert.com🔵: 631590265456623616 (Last update: 05:55 BST on Monday, 13 May 2019)

Pubkey: 02ad6fb8d693dc1e4569bcedefadf5f72a931ae027dc0f0c544b34c1c6f3b9a02b

Fees: 0+10/40 / 0+1/144

Forwards: Monthly In: 362,540 [13], Out: 96,415 [1], Weekly In: 362,540 [13], Out: 0 [0], Daily In: 55,761 [2], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥕] 948,114 / 34,539 [capacity: 1,000,000]

✅ 👪 LightningPowerUsers.com: 631590265456689152 (Last update: 13:44 BST on Monday, 13 May 2019)

Pubkey: 0331f80652fb840239df8dc99205792bba2e559a05469915804c08420230e23c7c

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 289,984 [1], Out: 0 [0], Weekly In: 289,984 [1], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥕🥕🥕] 780,578 / 202,075 [capacity: 1,000,000]

✅ 👪 Sagittarius A: 632050960773152769 (Last update: 16:55 BST on Monday, 13 May 2019)

Pubkey: 03a503d8e30f2ff407096d235b5db63b4fcf3f89a653acb6f43d3fc492a7674019

Fees: 0+10/40 / 0+1/30

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥕🥕🥕🥕] 682,652 / 300,001 [capacity: 1,000,000]

✅ 👪 CoinGate: 631562777641943040 (Last update: 15:54 BST on Sunday, 12 May 2019)

Pubkey: 0242a4ae0c5bef18048fbecf995094b74bfb0f7391418d71ed394784373f41e4f3

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 10,003 [3], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥕🥕🥕🥕🥕] 500,597 / 482,056 [capacity: 1,000,000]

✅ 👪 ACINQ: 632050960761552897 (Last update: 15:06 BST on Friday, 10 May 2019)

Pubkey: 03864ef025fde8fb587d989186ce6a4a186895ee44a926bfc370e2c366597a3f8f

Fees: 0+10/40 / 1,000+100/144

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥕🥕🥕🥕🥕🥕] 496,928 / 485,725 [capacity: 1,000,000]

✅ 👪 kornpow: 632376416256983040 (Last update: 02:59 BST on Monday, 13 May 2019)

Pubkey: 03077d02d11d2ade200c7fc5ba4fc66c1c599424fb945e88b3896fee6eedc07147

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥕🥕🥕🥕🥕🥕] 496,927 / 485,726 [capacity: 1,000,000]

✅ 👪 blackgold: 632112533487681536 (Last update: 12:32 BST on Friday, 10 May 2019)

Pubkey: 02f40890af885da4673f0ee9725ee74bb2c66d6491cc4334056a2701057993e61d

Fees: 0+10/40 / 250+10/40

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥝🥝🥝🥝🥝🥝] 492,800 / 0 [capacity: 1,000,000]

✅ 👪 ln1.satoshilabs.com: 631652937637363712 (Last update: 03:52 BST on Monday, 13 May 2019)

Pubkey: 0279c22ed7a068d10dc1a38ae66d2d6461e269226c60258c021b1ddcdfe4b00bc4

Fees: 0+10/40 / 0+1,000/144

Forwards: Monthly In: 0 [0], Out: 0 [0], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥕🥕🥕🥕🥕🥕] 490,098 / 492,555 [capacity: 1,000,000]

⭕ 👪 lightning-roulette.com: 631563877105729536 (Last update: 13:42 BST on Monday, 13 May 2019)

Pubkey: 031678745383bd273b4c3dbefc8ffbf4847d85c2f62d3407c0c980430b3257c403

Fees: 0+10/40 / 1,000+1/40

Forwards: Monthly In: 0 [0], Out: 10,000 [3], Weekly In: 0 [0], Out: 0 [0], Daily In: 0 [0], Out: 0 [0]

Balances: [🥝🥝🥝🥝🥕🥕🥕🥕🥕🥕] 477,827 / 502,058 [capacity: 1,000,000]


Tags: Lightning
Created by on Logo15659OpinionatedGeek Ltd.Logo15659