Post A5yjF93VlFWWf77Mps by alex@gleasonator.com
(DIR) More posts by alex@gleasonator.com
(DIR) Post #A5yirGilbrXXUC1u1g by ngp@mastodon.sdf.org
2021-04-06T18:49:50Z
1 likes, 0 repeats
@alex do you run Tribe? I think I broke it :)
(DIR) Post #A5yirH6ACsFsekyaUi by alex@gleasonator.com
2021-04-06T18:50:11.615459Z
1 likes, 0 repeats
@ngp Uh oh. What happened?
(DIR) Post #A5yjF8hB6HevXqfX1c by ngp@mastodon.sdf.org
2021-04-06T18:52:12Z
0 likes, 0 repeats
@alex Unclear. Billing tabs takes me to an "Internal server error" page and my actual tribe host gives a "PR_CONNECT_RESET_ERROR" on Firefox.
(DIR) Post #A5yjF93VlFWWf77Mps by alex@gleasonator.com
2021-04-06T18:54:29.886815Z
0 likes, 0 repeats
@ngp The billing tab is just broken. You won’t get billed more than the $1 you paid. The Postgres error is another problem. What’s the domain? I can try to see what’s happening.
(DIR) Post #A5yjjKEskJKo2BpMoq by ngp@mastodon.sdf.org
2021-04-06T18:57:20Z
0 likes, 0 repeats
@alex tribe.packetlost.devPotentially related: .dev domains require TLS. If tribes.tribe.host isn't on TLS it won't work.
(DIR) Post #A5yjjKhb1YIrTFGIZk by alex@gleasonator.com
2021-04-06T18:59:56.760852Z
0 likes, 0 repeats
@ngp TLS is provisioned automatically but I don’t even see that domain in the list… 🤔 Did you create this within the past 24h?
(DIR) Post #A5yk18TFEsnikR2QTo by ngp@mastodon.sdf.org
2021-04-06T19:01:57Z
0 likes, 0 repeats
@alex yeah, about 15 min ago. I can try deleting it and trying again.
(DIR) Post #A5yk18zVIwbaMU8BlI by alex@gleasonator.com
2021-04-06T19:03:09.521698Z
1 likes, 0 repeats
@ngp Don’t. I see it now, it’s in “deploying” state
(DIR) Post #A5yk5LduCxPUl527pg by alex@gleasonator.com
2021-04-06T19:03:55.695811Z
1 likes, 0 repeats
@ngp It can take up to 20 minutes because of my stupid storage solution: https://gitlab.com/tribes-host/tribes/-/issues/66
(DIR) Post #A5ykF8ZzdBwv8is4VU by ngp@mastodon.sdf.org
2021-04-06T19:05:30Z
1 likes, 0 repeats
@alex ah, that makes sense. Perhaps a small note on the dashboard or something to give the status would be helpful. Or at least saying "it can take up to x minutes for a Tribe instance to become available"
(DIR) Post #A5ykOeFrEUPVz7zK2C by alex@gleasonator.com
2021-04-06T19:07:25.072707Z
1 likes, 0 repeats
@ngp This is something else anyway. It’s not connecting to the k8s server. 🤦♂️ From the job queue: 3069 | discarded | k8s | TribesDeploy.DeployWorker | {"domain": "tribe.packetlost.dev", "user_id": "A5ygv2RHHTqapn62d6", "instance_id": "A5yh5KEvx73swGEBxw"} | {"{\"at\": \"2021-04-06T18:32:16.277297Z\", \"error\": \"** (Oban.PerformError) TribesDeploy.DeployWorker failed with {:error, {:error, [error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}]}}\", \"attempt\": 1}","{\"at\": \"2021-04-06T18:34:25.595178Z\", \"error\": \"** (Oban.PerformError) TribesDeploy.DeployWorker failed with {:error, {:error, [error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}]}}\", \"attempt\": 2}","{\"at\": \"2021-04-06T18:36:34.701088Z\", \"error\": \"** (Oban.PerformError) TribesDeploy.DeployWorker failed with {:error, {:error, [error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}]}}\", \"attempt\": 3}","{\"at\": \"2021-04-06T18:38:49.857586Z\", \"error\": \"** (Oban.PerformError) TribesDeploy.DeployWorker failed with {:error, {:error, [error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}]}}\", \"attempt\": 4}","{\"at\": \"2021-04-06T18:41:15.018432Z\", \"error\": \"** (Oban.PerformError) TribesDeploy.DeployWorker failed with {:error, {:error, [error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}, error: %HTTPoison.Error{id: nil, reason: :timeout}]}}\", \"attempt\": 5}"} | 5 | 5 | 2021-04-06 18:30:24.138123 | 2021-04-06 18:39:22.857535 | 2021-04-06 18:39:22.93544 | | {tribeshost-deployment-855ff8f8dd-cpxn6,k8s,j184o810} | 2021-04-06 18:41:15.018407 | 0 | {} | {} | It will keep retrying. Give it an hour or so and it will likely work itself out.This is a top priority to fix.
(DIR) Post #A5ykW8JRsOz3EdEk3E by alex@gleasonator.com
2021-04-06T19:08:46.192719Z
2 likes, 0 repeats
@ngp I think you could say there was a… packet lost.
(DIR) Post #A5ykjlnSO44VSOm8dk by seanking@traboone.com
2021-04-06T19:11:15.413019Z
1 likes, 0 repeats
@alex @ngp I swear to God, man. Fucking Kubernetes. A possible alternative solution could be some way to create a template container VM and then deploying it via Proxmox VE once a new instance is needed. IDK. Just a thought to get around them.
(DIR) Post #A5ykmh7EFQQzo8Pz5U by ngp@mastodon.sdf.org
2021-04-06T19:11:22Z
2 likes, 0 repeats
@alex packet lost is actually for for Packet Lost & Found. Where else would all the dropped packets end up?Thanks for the quick response!
(DIR) Post #A5ykoPCBqRO0oDYRXM by ngp@mastodon.sdf.org
2021-04-06T19:11:53Z
2 likes, 0 repeats
@seanking @alex as someone who's been fighting a deployment on K8s since *Thursday*, yeah...
(DIR) Post #A5ynMwnxaooonfotW4 by alex@gleasonator.com
2021-04-06T19:40:43.646324Z
1 likes, 0 repeats
@ngp Deploying an update with a longer timeout on the HTTP requests. Pray for me. 🙏
(DIR) Post #A5yrDgOvbzsv8RwuX2 by ngp@mastodon.sdf.org
2021-04-06T20:16:21Z
0 likes, 0 repeats
@alex still doesn't seem to be up. Is the issue some external service being down?
(DIR) Post #A5yrDh4l6TLTEBWJjU by alex@gleasonator.com
2021-04-06T20:23:52.073138Z
0 likes, 0 repeats
@ngp I really don’t get what’s happening. Is the billing page fixed at least now?The HTTP requests to the K8s API server just aren’t reaching. I’m really confused.
(DIR) Post #A5yrK5L5nzhuDWCxAe by alex@gleasonator.com
2021-04-06T20:25:01.086109Z
1 likes, 0 repeats
@ngp I know the API server is working because I can use kubectl on my laptop with the exact same config. But the requests from Tribes time out. It never had this problem before.
(DIR) Post #A5ytyRWmvZmYWe3B7w by ngp@mastodon.sdf.org
2021-04-06T20:28:43Z
0 likes, 0 repeats
@alex that's interesting. It could be a networking related problem that's not related to your infrastructure/code.Do you have the ability to SSH into the Tribes server and attempt kubectl from there to rule out routing/networking issues?
(DIR) Post #A5ytyRzVCokbxhU6sq by alex@gleasonator.com
2021-04-06T20:54:42.275683Z
1 likes, 0 repeats
@ngp Indeed, kubectl hangs inside the container.
(DIR) Post #A5yuACFdYGDocYdH6W by alex@gleasonator.com
2021-04-06T20:56:50.746294Z
1 likes, 0 repeats
@ngp WTF. It was the firewall.
(DIR) Post #A5yuZvcpzLV8aOkpCy by ngp@mastodon.sdf.org
2021-04-06T20:58:22Z
1 likes, 0 repeats
@alex that would certainly do it
(DIR) Post #A5yuZw5uFGkm2YM2W8 by alex@gleasonator.com
2021-04-06T21:01:29.808866Z
1 likes, 0 repeats
@ngp How did this EVER work??
(DIR) Post #A5yveLrmT0IX5FPuDo by alex@gleasonator.com
2021-04-06T21:13:30.168669Z
3 likes, 0 repeats
@ngp Your site is now online and the billing page is fixed.Thank you for accompanying me on this turbulent journey.I promise it will get better over time.
(DIR) Post #A5yvzjuoFUbfkd232O by ngp@mastodon.sdf.org
2021-04-06T21:16:57Z
3 likes, 0 repeats
@alex no worries, overall this has been a pleasant experience 😄
(DIR) Post #A5ywV6IR5Qb28r2Ml6 by seanking@traboone.com
2021-04-06T21:23:04.245189Z
0 likes, 0 repeats
@alex @ngp A firewall of all things causing the issues. Those motherfuckers can be real tricky, especially if you're dealing with multiple layers of them.
(DIR) Post #A5yyCXTnzkrUr7dFXk by sullybiker@sully.site
2021-04-06T21:42:05Z
1 likes, 0 repeats
@alex @ngp Container networking and firewalls is a major arseache