Post A4yjVMmIndzT6VfsIa by utzer@social.yl.ms
(DIR) More posts by utzer@social.yl.ms
(DIR) Post #A4yjVLIqHgpaWqspk0 by utzer@social.yl.ms
2021-03-01T13:55:08Z
0 likes, 0 repeats
@heluecht MariaDB [friendica]> select `backend-class`, `backend-ref` from photo where id = 888;+---------------+-------------+| backend-class | backend-ref |+---------------+-------------+| | 670476 |+---------------+-------------+1 row in set (0.001 sec)
(DIR) Post #A4yjVLfAwehBe7KfYG by heluecht@pirati.ca
2021-03-01T14:19:10Z
0 likes, 0 repeats
Please check if the "data" field contains a value there. If not then you have got a problem. The backend-class should only be empty when you still store your pictures in the database.Please check if you have got a field in the storage table with the id "670476". Also check if in the storage folder there is some file that contains this id. I don't know the exact format of the stored pictures, so I can't give you a hint for the exact filename.
(DIR) Post #A4yjVM1VbcYmlNmVMW by admin@friendica.utzer.de
2021-03-01T14:29:39Z
0 likes, 0 repeats
Please check if the "data" field contains a value there. If not then you have got a problem. The backend-class should only be empty when you still store your pictures in the database.Please check if you have got a field in the storage table with the id "670476". Also check if in the storage folder there is some file that contains this id. I don't know the exact format of the stored pictures, so I can't give you a hint for the exact filename.I missed that comment. Need to figure out how to check that in mysql cli.
(DIR) Post #A4yjVMNqGaQNseELAm by heluecht@pirati.ca
2021-03-01T14:45:34Z
0 likes, 0 repeats
select id from storage where id=670476;
(DIR) Post #A4yjVMmIndzT6VfsIa by utzer@social.yl.ms
2021-03-01T14:48:29Z
0 likes, 0 repeats
@heluechtResults of select id from storage where id=670476;: MariaDB [friendica]> select id from storage where id=670476;+--------+| id |+--------+| 670476 |+--------+1 row in set (0.001 sec)
(DIR) Post #A4yjVN8dSbr4Dm7i6q by heluecht@pirati.ca
2021-03-01T14:56:42Z
0 likes, 0 repeats
Interesting. We possibly can repair this, but I need some time.
(DIR) Post #A4yjVNUGAD9VIqEyoa by admin@friendica.utzer.de
2021-03-02T17:34:59Z
0 likes, 0 repeats
Did you find out what the problem is? I was thinking when I should switch to the RC, will this be fixed in RC or only in a future release? Is this a problem specific to my system?
(DIR) Post #A4yjVNqwnrIgRCr6B6 by heluecht@pirati.ca
2021-03-02T18:34:39Z
0 likes, 0 repeats
I never saw this problem anywhere. Can you remind me at the weekend when I haven't reacted until then?
(DIR) Post #A4yjVOBrY61xU4dnmK by admin@friendica.utzer.de
2021-03-06T07:01:39Z
0 likes, 0 repeats
Wochenende... 😉 If you need any info/action from my side let me know.
(DIR) Post #A4yjVOWQJeTeVqGDpI by heluecht@pirati.ca
2021-03-06T15:28:40Z
0 likes, 0 repeats
Please try:update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';
(DIR) Post #A4yjVOqd6WdlWViMK0 by admin@friendica.utzer.de
2021-03-06T19:16:11Z
0 likes, 0 repeats
update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 1625 rows affected (0.302 sec)Rows matched: 1625 Changed: 1625 Warnings: 0done
(DIR) Post #A4yjVPBXqlN2ZNV3vE by admin@friendica.utzer.de
2021-03-07T11:54:06Z
0 likes, 0 repeats
@heluecht I think I can execute this again and again and again and it will always find new rows. My uneducated guess would be that the move does not set this and the move is still ongoing. See here: MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 500 rows affected (0.077 sec)Rows matched: 500 Changed: 500 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 245 rows affected (0.059 sec)Rows matched: 245 Changed: 245 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 89 rows affected (0.052 sec)Rows matched: 89 Changed: 89 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 73 rows affected (0.049 sec)Rows matched: 73 Changed: 73 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 77 rows affected (0.044 sec)Rows matched: 77 Changed: 77 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 67 rows affected (0.049 sec)Rows matched: 67 Changed: 67 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 86 rows affected (0.051 sec)Rows matched: 86 Changed: 86 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 67 rows affected (0.048 sec)Rows matched: 67 Changed: 67 Warnings: 0MariaDB [friendica]> update photo inner join storage on storage.id = photo.`backend-ref` set `backend-class` = 'Database' where photo.data = '' and `backend-class` = '' and `backend-ref` != '';Query OK, 87 rows affected (0.048 sec)Rows matched: 87 Changed: 87 Warnings: 0
(DIR) Post #A4yjVPXAYMfTeRcKcy by heluecht@pirati.ca
2021-03-07T12:00:31Z
0 likes, 0 repeats
Interesting. Sadly I don't know that photo class part very well. For some reason your system doesn't write the backend-class".
(DIR) Post #A4yjVPsRHHgKiPZJmS by admin@friendica.utzer.de
2021-03-07T13:32:28Z
0 likes, 0 repeats
I think some pictures were lost also... for example the old profile photo of this account is gone. Also another question, can I somehow initiate that the server pulls all remote profile information for all known contacts? I mean all contacts that are somehow connected to users on this server.
(DIR) Post #A4yjVQHFn1WzxNB8SW by heluecht@pirati.ca
2021-03-07T13:41:37Z
0 likes, 0 repeats
The systen is doing so anyway over time. That's the "UpdateContact" class.
(DIR) Post #A4yjVQcAXGGH0Exq3k by admin@friendica.utzer.de
2021-03-07T13:44:01Z
0 likes, 0 repeats
is that the setting in /admin/site "Auto-discovered contact directory" - "Days between enquiry"? Maybe I will set this to 2 days to force it to happen?
(DIR) Post #A4yjVQyrAuPS8bZxQG by heluecht@pirati.ca
2021-03-07T13:50:10Z
0 likes, 0 repeats
No, there is no setting for this. Every contact with a post on the system is checked once a month or once a week if not failed. And if update_active_contacts isn't set, then also every other known contact is checked every 6 month or 1 month if not failed.
(DIR) Post #A4yjVRP5bNORRxquJM by admin@friendica.utzer.de
2021-03-07T13:57:28Z
0 likes, 0 repeats
hmm ok, so until this happens the profile pictures will be missing. Just in case this would be easy to trigger, such a trigger seems to be helpful to be added to bin/console.
(DIR) Post #A4yjVRkMKIPIVvntSq by heluecht@pirati.ca
2021-03-07T14:10:58Z
0 likes, 0 repeats
Your case is really special and we yet haven't found the reason for this problem. So some manual updating won't fix the issue but would just disguise it.Would be great if @fabrixxm could step in, since he wrote that class 😀
(DIR) Post #A4yjVS4D8UHpVV5kPI by fabrixxm@social.gl-como.it
2021-03-07T20:45:46Z
0 likes, 0 repeats
Ah, like i remember what I wrote ... :)
(DIR) Post #A4yjVSTNcuQ4lYrqdc by heluecht@pirati.ca
2021-03-07T21:04:01Z
0 likes, 0 repeats
I have full trust in you 😁
(DIR) Post #A58FHenDyBojsUFfUm by admin@friendica.utzer.de
2021-03-07T14:19:10Z
0 likes, 0 repeats
also agree that there is something strange here.The contact update was something I wished to have before, not for this problem. I think that sometimes it just takes to long for profile changes to arrive, so I sometimes wished I had a manual way to fix this.
(DIR) Post #A58FHf9Yd9gKzkhVJ2 by admin@friendica.utzer.de
2021-03-07T20:06:19Z
0 likes, 0 repeats
stagely it will only find 500 rows maximum, each time I run it after some time, ie. some hours, it will find 500 maximum, a few on the second time I execute it and now there after. Query OK, 500 rows affected (0.063 sec)Rows matched: 500 Changed: 500 Warnings: 0
(DIR) Post #A58FHfU7Oi821WJvM0 by admin@friendica.utzer.de
2021-03-07T20:57:30Z
0 likes, 0 repeats
again me... I don't know why I never did that before, but now after I switched to the RC and ran mysql_upgrade (seems that is sometimes needed after mariadb updates, I fired up mytop when I restarted the move process and from time to time I see this:and then also this: So it seems this is what is needed, right?
(DIR) Post #A58FHfogAGZj3HwLOy by admin@friendica.utzer.de
2021-03-07T21:04:46Z
0 likes, 0 repeats
or does it - SET `backend-class` = '' - in fact set an empty backend class? Commit UPDATE `photo` SET `backend-class` = '', `backend-ref` = 37368503, `data` = '' WHERE (`id` = 577936)
(DIR) Post #A58FHgAIrrsA8M3c6i by fabrixxm@social.gl-como.it
2021-03-07T21:33:59Z
0 likes, 0 repeats
Are you running bin/console storage move?To which backend are you moving to?
(DIR) Post #A58FHgWzVW1LGifjTE by admin@friendica.utzer.de
2021-03-07T21:35:20Z
0 likes, 0 repeats
From filesystem to database.
(DIR) Post #A58FHguk5D1GSNmhUW by fabrixxm@social.gl-como.it
2021-03-08T07:51:49Z
0 likes, 0 repeats
mmh..what $ bin/console config storage name says?can you try to change in src/Core/StorageManager.php in line 314:'backend-class' => $destination to 'backend-class' => $destination::getName() ?
(DIR) Post #A58FHhJYawrvhLOWAa by admin@friendica.utzer.de
2021-03-12T08:53:15Z
0 likes, 0 repeats
did you already create a pull request for this in the current RC?
(DIR) Post #A58FHhhJAdrqt0VUBs by fabrixxm@social.gl-como.it
2021-03-12T10:24:23Z
0 likes, 0 repeats
no, sorry, I've been drifted away by other things... I saw your commit, and from what we tested, looks like it's something needed for sure (and the code is more explicit, which imho is always a win).I'm for a little pull request with that patch 👍I saw in your screenshot the query INSERT INTO storage (data) VALUES ('') which looks like is inserting empty data for photo. I don't know if the source file was empty, the log you get there is truncated because value of data was too big (but I don't think it's the case) or if the filesystemstorage backend failed in get() (which I don't think, otherwise it would have failed to fetch data for every images)... looking again at the code in Friendica\Core\StorageManager::move() it could be that it was caused by the same bug: with the bug data was fetch from Filesystem, saved in Database and the backend was saved wrongly as "" (nothing). On the next run the same row has been processed again (as the backend saved is not the destination backend), but as now has empty 'backend-class' data was loaded from the photo table, which was empty, and a new row is added in 'storage' with empty data.It could be that the photos you see missing are in reality in 'storage' table, but 'photo.backend-ref' is pointing to the row with empty data.Problem is you probably never manage to find which "orphan" photo data in 'storage' goes with which row in 'photo' table, because there is no filename in 'storage' table, only data...
(DIR) Post #A58FHi4LmyIc2THt6e by admin@friendica.utzer.de
2021-03-12T10:45:02Z
0 likes, 0 repeats
It could be that the photos you see missing are in reality in 'storage' table, but 'photo.backend-ref' is pointing to the row with empty data.Problem is you probably never manage to find which "orphan" photo data in 'storage' goes with which row in 'photo' table, because there is no filename in 'storage' table, only data...Meaning, you don't think it can be fixed?
(DIR) Post #A58FHiS6MfIXE8Or7w by fabrixxm@social.gl-como.it
2021-03-12T10:52:50Z
0 likes, 0 repeats
we can do a query looking for rows in storage table not referenced in photo and attach tables, and we could get data and try to show it as images, but then someone should look at the images and find to which row in photo belong, by hand.Because we have file metadata (filename, mimetype, ect) in photo table with a link to storage table where there is the actual data, but we have no means to automatically get back from storage to photo.
(DIR) Post #A58FHion0JRiMV0yUS by utzer@social.yl.ms
2021-03-12T11:06:54Z
0 likes, 0 repeats
@fabrixxm hmm I think by hand is no an option, I should not look at users photos, maybe there is private photos there. Also I am not sure how many files it would be, so it could be just a few or really alot. Do you know why the move works in some case (social.yl.ms) and not in others (friendica.utzer.de) and I think @heluecht also wrote he was moving storage (but maybe from database to filesystem.
(DIR) Post #A58FHjBpcdsTVxnNPE by heluecht@pirati.ca
2021-03-12T11:12:49Z
0 likes, 0 repeats
Whatever happened there, there had been exactly one bug report so far. I really would like to know the origin of the problem.