https://groups.google.com/a/chromium.org/g/ct-policy/c/PCkKU357M2Q/ [groups_48d]Groups Conversations All groups and messages [Search conversations][ ] Send feedback to Google Help * Account * Search * Maps * YouTube * Play * News * Gmail * Meet * Chat * Contacts * Drive * Calendar * Translate * Photos * Duo * Chrome * Shopping * Finance * Docs * Sheets * Slides * Books * Blogger * Hangouts * Keep * Jamboard * Earth * Collections * Arts and Culture * Google Ads * Podcasts * Stadia * Travel * Forms More from Google Sign in Groups Certificate Transparency Policy Conversations About Privacy * Terms Yeti 2022 not furnishing entries for STH 65569149 22403 views Skip to first unread message Andrew Ayer's profile photo Andrew Ayer unread, Jun 30, 2021, 10:22:17 AM (4 days ago) Jun 30 Reply to author Sign in to reply to author Forward Sign in to forward Delete Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to ct...@digicert.com, Certificate Transparency Policy Yeti 2022 has produced the following STHs: { "tree_size": 65551225, "timestamp": 1624968008035, "sha256_root_hash": "QLoUSs76wgHT7RyyJDdJPbPokyQnNwsVBOMq/gIuEMk=", "tree_head_signature": "BAMASDBGAiEAgpQ9zzpRo8NZHfw/ lXk0g9YvvaNtALinJoaqxK+tUosCIQDYsuAJbqiTv5/ CZechrVOjk3C1SxURGlNibEKW5iQ0zA==" } { "tree_size": 65569149, "timestamp": 1624971609346, "sha256_root_hash": "ogxKC1kdkMoDxTYnEkdHVt/UotIeRpip7x6QljoOGuY=", "tree_head_signature": "BAMASDBGAiEA7gQwSrLcn6JzY9USfUyQjzdifF6ojGztYaCvcYWZFLcCIQDMlxZitnji0l5mcclFCS0C6FpFEITWOqJYEJCnBB6rIg ==" } Although Yeti 2022 can produce a valid consistency proof between these two STHs, the entries in the range [65551225, 65569149) returned from get-entries produce the root hash KTFRWeiy7n+HvsK6lZ7AM1RInOUeYHBhFHMJRO5iGcs= when they are appended to the tree with root hash QLoUSs76wgHT7RyyJDdJPbPokyQnNwsVBOMq/gIuEMk=. I've attached a list of the leaf hashes of the log entries in this range which Yeti 2022 furnished to my monitor. A similar problem previously occurred with Nimbus 2018: https://bugs.chromium.org/p/chromium/issues/detail?id=780654#c10 Regards, Andrew yeti_leaf_hashes.csv Devon O'Brien's profile photo Devon O'Brien unread, Jun 30, 2021, 4:47:04 PM (4 days ago) Jun 30 Reply to author Sign in to reply to author Forward Sign in to forward Delete You do not have permission to delete messages in this group Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to Certificate Transparency Policy, Andrew Ayer, ct...@digicert.com Andrew, thank you as always for your timely and detailed posts to ct-policy@ regarding observed issues with CT Logs. We can confirm that we've observed unexpected behavior from Yeti2022 today as well and are continuing to investigate the scope of this issue. DigiCert CT Ops -- Can you please look into this matter and report your findings and fix/timeline on this thread? -Devon Andrew Ayer's profile photo Andrew Ayer unread, Jun 30, 2021, 5:43:45 PM (4 days ago) Jun 30 Reply to author Sign in to reply to author Forward Sign in to forward Delete Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to Certificate Transparency Policy, ct...@digicert.com The problem seems to be with entry 65562066. If you download this entry with get-entries (https://yeti2022.ct.digicert.com/log/ct/v1/get-entries?start= 65562066&end=65562066) you get an entry with leaf hash ty69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8=. However, if you try to request an inclusion proof for this leaf hash (https://yeti2022.ct.digicert.com/log/ct/v1/get-proof-by-hash?hash= ty69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8=&tree_size=65569149), you get an error: { "error_message": "The leaf hash was not found", "error_code": "hash unknown" } Regards, Andrew Jeremy Rowley's profile photo Jeremy Rowley unread, Jul 1, 2021, 11:29:12 AM (3 days ago) Jul 1 Reply to author Sign in to reply to author Forward Sign in to forward Delete You do not have permission to delete messages in this group Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to Andrew Ayer, Certificate Transparency Policy, ct...@digicert.com I can confirm that the log is not operated correctly. The last good treehead was signed on June 29th around noon GMT. Somehow entry 65562066 shifted one bit after the June 29th signing, which is causing the issue. If you shift the bit back, the tree signs correctly. We are still investigating why the first happened and have only hit deadends so far. The cert is logged in other DigiCert logs. We replicated the timestamp in our dev environment (with a separate private key). Neither seem to be causing the error. Jeremy -- You received this message because you are subscribed to the Google Groups "Certificate Transparency Policy" group. To unsubscribe from this group and stop receiving emails from it, send an email to ct-policy+...@chromium.org. To view this discussion on the web visit https:// groups.google.com/a/chromium.org/d/msgid/ct-policy/ 20210630174304.133c9a1dc9263cd84e568165%40andrewayer.name. Andrew Ayer's profile photo Andrew Ayer unread, Jul 2, 2021, 2:30:01 PM (2 days ago) Jul 2 Reply to author Sign in to reply to author Forward Sign in to forward Delete Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to Jeremy Rowley, Certificate Transparency Policy, ct...@digicert.com It looks like the leaf hash added to the Merkle tree is ti69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8= - a one-bit difference from the correct hash of ty69nN62xOf6NHpRjmsw7GxIa3IfKVUK/2EBQDUsNU8=. I think the most likely explanation is that this was a hardware error caused by a cosmic ray or the like, rather than a software bug. It's just very bad luck :-( Unfortunately, it's not possible for the log to recover from this event. In the case of Nimbus 2018, the log operator was able to fix the get-entries response to return entries matching the STH. In this case, it can't be done because it would require breaking SHA-2's preimage resistance to find the preimage of the bit-flipped hash. Yeti 2022 will never be able to return a get-entries response that matches STHs with tree_size > 65562066. Additionally, the SCT issued for entry 65562066 can't be audited, because the client will attempt to audit the correct leaf hash, which is not in the Merkle tree. There are still entries being added to Yeti 2022. CAs should immediately cease using this log, and ideally it would be made read-only. Regards, Andrew > https://groups.google.com/a/chromium.org/d/msgid/ct-policy/ CAFK%3DoS8jrOyxtnndoJ55pO-pxE0pAtvV8jY2p98tXt862fQ18A%40mail.gmail.com . Devon O'Brien's profile photo Devon O'Brien unread, Jul 2, 2021, 3:49:43 PM (2 days ago) Jul 2 Reply to author Sign in to reply to author Forward Sign in to forward Delete You do not have permission to delete messages in this group Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to Certificate Transparency Policy, Andrew Ayer, Certificate Transparency Policy, ct...@digicert.com, Jeremy Rowley This bit flip does appear to be an unfortunate random event, almost certainly caused by hardware and outside the control of the Log Operator (DigiCert). That said, the inability to recover this entry will continue to interfere with monitoring and auditing Yeti2022 for as long as it remains Qualified, Usable, or Read-Only, and will be making a policy announcement to that effect very soon. In the meantime, as Andrew mentioned, if DigiCert could configure Yeti2022 to stop accepting new entries as soon as possible, it will help minimize the impact to the ecosystem in the interim. I'd also like to reiterate that we do not believe that this failure is the fault of DigiCert, and we've long speculated that extremely low chances of hardware bit flips, combined with the append-only nature of CT Logs might eventually manifest in a situation like this at some point. We would gladly accept (and encourage!) the application of a new 2022 shard in place of Yeti2022. -Devon Jeremy Rowley's profile photo Jeremy Rowley unread, Jul 3, 2021, 1:56:39 AM (2 days ago) Jul 3 Reply to author Sign in to reply to author Forward Sign in to forward Delete You do not have permission to delete messages in this group Link Report message as abuse Sign in to report message as abuse Show original message Either email addresses are anonymous for this group or you need the view member email addresses permission to view the original message to Devon O'Brien, Certificate Transparency Policy, Andrew Ayer, ct...@digicert.com Thanks Devon. We've moved the log to read-only mode. No additional certs can be logged to the Yeti 2022 shart. Reply all Reply to author Forward 0 new messages Search Clear search Close search Google apps Main menu