https://meta.stackexchange.com/a/390023/6212 Stack Exchange Network Stack Exchange network consists of 181 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers. Visit Stack Exchange [ ] Loading... 1. + Tour Start here for a quick overview of the site + Help Center Detailed answers to any questions you might have + What's Meta? How Meta is different from other sites + About Us Learn more about Stack Overflow the company, and our products. 2. 3. current community + Stack Exchange + Meta Stack Exchange help chat your communities Sign up or log in to customize your list. more stack exchange communities company blog 4. 5. Log in 6. Sign up Meta Stack Exchange is a question and answer site for meta-discussion of the Stack Exchange family of Q&A websites. It only takes a minute to sign up. Sign up to join this community [ano] Anybody can ask a question [ano] Anybody can answer [an] The best answers are voted up and rise to the top Meta Stack Exchange 1. Home 2. 1. Public 2. Questions 3. Tags 4. Users 5. Unanswered 3. Teams Stack Overflow for Teams - Start collaborating and sharing organizational knowledge. [teams-illo-free-si] Create a free Team Why Teams? 4. Teams 5. Create free Team Teams Q&A for work Connect and share knowledge within a single location that is structured and easy to search. Learn more about Teams June 2023 Data Dump is missing Ask Question Asked 2 days ago Modified today Viewed 15k times 184 The bounty expires in 7 days. Answers to this question are eligible for a +200 reputation bounty. Resistance Is Futile wants to reward an existing answer. The data dump usually get refreshed the first weekend of the month, every 3 months. The current Data dump is still from March. Is there just a problem and it's delayed like in the past? * support * status-planned * data-dump Share Improve this question Follow edited 4 hours ago Rosie's user avatar RosieStaffMod 10.6k99 gold badges3131 silver badges5757 bronze badges asked 2 days ago Data Dude's user avatar Data DudeData Dude 60122 gold badges22 silver badges66 bronze badges New contributor Data Dude is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. 17 * 21 If the data was published it might make it easier to debunk the nonsense... - Rory Alsop 2 days ago * 5 I don't think it was ever done on time. It's a very complex process, hence always failing at some point. I consider it a miracle when it's actually working and there is actually a data dump. (Kind of like launching a spaceship.) - Shadow Wizard Strikes Back yesterday * 15 @ShadowWizardStrikesBack rude...I disagree that it was never done on time. :) - Taryn yesterday * 10 @ShadowWizardStrikesBack I can confirm it has been frequently late since Taryn left. The upload to the Archive would frequently hang and need baby sitting. That said, we actually resolved that recently, so the data dump wouldn't be late anymore. - AMtwo yesterday * 19 I'm hoping someone from the Company gives an official response soon. Having been recently laid off, I'm not in a position to give a complete/official answer here... But if there's no official response soon, I'll jump in with what I can offer. - AMtwo yesterday * 3 @RoryAlsop the data dump is just a full export of SEDE. You're limited by the tooling for timeouts, and doing the analysis in SQL Server, but there's at least some alternative - AMtwo yesterday * 1 @AMtwo hello! What about Aaron? In all recent cases he was the one to save the day. Well, the dump. :D - Shadow Wizard Strikes Back yesterday * 8 Aaron just took the metaphorical slaps for me, by handling the public comms. It was the other redundant DBRE who got the process running smoothly, and me pointing and crying when I had to manually upload files after it got hung. - AMtwo yesterday * 2 An uncharitable observer might conclude the data contradicts the company's narrative. I'm not feeling particular charitable right now. - Gloweye 12 hours ago * 7 Interestingly, the public availability of the data was a main reason I choose to contribute to stack exchange in the first place. If that's not just a delay, I might request deletion of all my contributions. - miku 8 hours ago * 12 @miku You don't have an option to delete your contributions, you've already licensed them out (and the terms don't give you any option for revocation). Future contributions you can decide on, though. - Bryan Krause is on strike 8 hours ago * 2 @miku The closest you can come is a GDPR Data Deletion Request. Some more info & discussion here - AMtwo 7 hours ago * 8 next step is going to be company rename to Stack Experts-Exchange I guess - gnat -on strike- 6 hours ago * 9 Jody has responded in an answer below. I've updated the status to planned. - Rosie StaffMod 4 hours ago * 3 Can we get the official answer accepted, please? - wizzwizz4 2 hours ago | Show 2 more comments 6 Answers 6 Sorted by: Reset to default [Highest score (default) ] 159 DISCLAIMER: I was recently impacted by the Company's layoff. I am going to carefully respond in a way that ensures I don't reveal anything the Company may feel is confidential--particularly with regard to strategy, or future plans. Any knowledge I have on strategy or future plans is both dated and confidential, and thus it would be irresponsible for me to say more. As a result, this answer may feel incomplete. I suspect that the CM team is rather busy this week with other topics. I'm offering what I can to uphold the Company's values of Transparency & being Community-centric. The upload to the Internet Archive has been disabled. The job that uploads the data dump to Archive.org was disabled on 28 March, and marked to not be re-enabled without approval of senior leadership. Had it run as scheduled, it would have completed on the first Monday after the first Sunday in June. I mention the timing, as this change long pre-dated the current moderator strike and related policy changes. Some comments have suggested otherwise, so I thought it an important detail. Is it going to stay that way? Hopefully the Company will provide an answer that includes this. Edit: The following is an excerpt from a different answer provided by Jody Bailey, CTO of Stack Overflow: We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. We are working to design and implement appropriate safeguards and still sorting out the details and timelines. We will provide regular updates on our progress to this group. How can I access that data? Stack Exchange Data Explorer (aka SEDE) contains a subset of all data for all sites, with PII removed. The same data available in the data dump is also available on SEDE. SEDE is updated via a weekly full refresh (every weekend). The Data Dump that is uploaded to Archive.org is a dump of the SEDE databases. The weekly SEDE refresh runs, then the data is dumped to XML & 7zipped, then the 7z files are uploaded to the Archive. SEDE can't address all the use cases of the Data Dump, nor vice versa. However, there is overlap, and the data is at least queryable. --------------------------------------------------------------------- Hopefully any questions around "why" or "what's next" will be addressed by an official response. Share Improve this answer Follow edited 3 hours ago Random Person's user avatar Random Person 5,7191010 silver badges5050 bronze badges answered 9 hours ago AMtwo's user avatar AMtwoAMtwo 1,61622 gold badges1616 silver badges2222 bronze badges 31 * 13 Well that sucks... - Nick is tired 9 hours ago * 34 I'm sad to see that this has happened. Thank you for providing an answer. - Taryn 9 hours ago * 94 The data dump is also part of the commitment the company has had with the community for years. I would guess this is part of their master plan to sell data but some heads up would have been nice. - Journeyman Geek Mod 9 hours ago * 7 @JourneymanGeek Asking for forgiveness is easier than asking for permission amirite? - Data Dude 8 hours ago * 14 I see next step is going full Reddit and asking payment for API usage... - Mast - on strike 8 hours ago * 14 If they're doing this because they plan to sell access to the data, that would make some business sense except that people who are using this data to, say, train AI models, currently believe that doing so is fair-use, and they can do so by crawling the data publicly available. Anyone with a financial interest big enough to spend money on the data is going to be just as capable of obtaining the data without the dump, so this only affects people like academic researchers. - Bryan Krause is on strike 8 hours ago * 52 I would never have thought I'd see the day, but this is the end for me on SO. The unwritten contract between me and this platform was that the knowledge remains freely accessible and archiveable while I contribute. Since this didn't happen anymore, well after 12 years of membership, that's it. I'll leave and contribute elsewhere. Sorry, but this is a point where no compromise is possible. - Trilarion 8 hours ago * 36 @AMtwo I admire your commitment to the community even after the company has thrown you under the bus. - Emil Jerabek 8 hours ago * 29 The Data Dump was literally one of the safeguards in case the company became evil! And I do mean literally - Jeff or Joel said that, in the early days of Stack Overflow. - S.L. Barth 8 hours ago * 26 This is deeply concerning. If the next step is making all API access paid, I'm going to resign. Numerous community-driven tools, including in moderation, rely heavily on on the API to function. Forcing all those projects or even just individuals making QOL tools to pay for something that benefits the platform is outright despicable and self-destructive, and is not the SO I signed up for - Zoe stands with Ukraine 7 hours ago * 18 Just sorta stating the obvious here, but the timing of this is unbelievably terrible; I actually can't fathom a worse time for this call to be made than in light of this week. - zcoop98 7 hours ago * 15 @zcoop98 Well, the call was made 3 months ago and hidden from the community. It's just exploding today. - Restore the Data Dumps 7 hours ago * 11 So the company unilaterally changed the policy and didn't even tell us about it? Who could have seen this coming? - kaya3 7 hours ago * 9 Here is where it was announced 14 years ago: stackoverflow.blog/ 2009/06/04/... RIP SE - Bob 6 hours ago * 9 The timing means only one thing, that this is all planned. Which makes the company evil and greedy, not caring about the users even a tiny bit. Shame on the management, wish they would all get kicked out with disgrace, ASAP. - Shadow Wizard Strikes Back 6 hours ago | Show 16 more comments 57 Few things have made me more livid than what we've just learned through the answer given by AMtwo. The data dump is one of the main reasons why I'm still participating to this day and has been part of the core values of this community. Please restore it, or consult with us to establish some equivalent way to protect and export our contributions as a whole. I would like to bring the following pieces of history to the foreground. This is an excerpt of Joel Spolsky speaking on the Stack Overflow podcast #84 Oh, expropriation of community content that... We created Stack Overflow to be against it. If there's anything that's more in the DNA of Stack Overflow than that, I don't know what it is. That's one of our most core things. You can see this all over the place in the design of Stack Overflow. First of all, from day one, we use the CC-wiki license. And it's basically a license, it says that we don't own the content that's on there, which is why we make those database dumps that are available. Because we wanted to make sure that if no matter what happens, literally no matter who we sell to, or raise money from, or turn the site over to, and even if they take Stack Overflow, and make it an evil site where you have to pay to look at things and there's pop-up ads and pop-under ads, and you know, dancing chariots of fire that cross the screen and punch the monkey, and, man, I can take so many evil things anyway. And it just becomes a big gigantic spam site. Doesn't matter because just take the latest CC-wiki download that we provided and go start your own site saying, you know what, this is gonna be the clean version. And I think a lot of people will follow you. We very, very deliberately built Stack Overflow in a way that there wouldn't be any chance of locking and we're pretty much doing the same thing with Stack Exchange. Also, on the blog, written by Jeff Atwood in 2009: The community has selflessly provided all this content in the spirit of sharing and helping each other. In that very same spirit, we are happy to return the favor by providing a database dump of public data. Share Improve this answer Follow edited 4 hours ago answered 4 hours ago curious's user avatar curiouscurious 2,92922 gold badges1616 silver badges2121 bronze badges 1 * 5 love you for posting this, really - Vickel 3 hours ago Add a comment | 15 Stack Overflow senior leadership is working on a strategy to [S: protect:S] be able to solely monetize Stack Overflow data [S:from being misused:S] by charging companies building LLMs extortionate amounts of money. While working on this strategy, we decided to stop the dump until we could put guardrails in place. Once again ignoring the community instead of being proactive and informing them regarding this. We are working on setting up the infrastructure to do this correctly in the age of LLMs --- where we continue to be open and share the data with our developer community but work to set up a formal framework for large AI companies that want to leverage the data. We're quickly scrambling because this would be an amazing opportunity for us to make some more money off your backs. We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. We are working to design and implement appropriate safeguards and still sorting out the details and timelines. We're probably going to degrade the experience for people within the community who used to use these dumps because we need to somehow get companies to pay us to use this data in a more friendly way. As for this this line: We will provide regular updates on our progress to this group. Group? Do they mean group as in group of users interested in the dumps? Or, are they that detached with the community that they call these sites groups like you have WhatsApp groups? Nevertheless, I can't believe this was actually unironically written when they didn't even bother to tell the community that this was happening. I guess this is what comes from having employees that no longer interact with the community. Anyone who seriously trusts SO after all they've done deserves to be fooled. --------------------------------------------------------------------- "imagine if stackexchange went from free to paid..." All of the content posted by users on Stack Exchange network is licensed under CC-BY-SA. If SE ever blocks free access to the content, anyone can host it elsewhere. Pulling EE-like shenanigans is not practical in SE. Source @JCL1178: I agree except the data is open source. If EE dies, the data is gone, especially if the past happens again. Source --------------------------------------------------------------------- ^Technically speaking, this is an answer, you might not like it or think that it's not a great answer but it doesn't make it any less of an answer. So, if you happen to feel particularly strongly about it, feel free to downvote but I personally don't think it deserves deleting. But hey, what do I know? -\_(tsu)_/- Share Improve this answer Follow edited 2 hours ago answered 2 hours ago Script47's user avatar Script47Script47 14.9k99 gold badges3737 silver badges5555 bronze badges 4 * "Anyone who seriously trusts SO after all they've done deserves to be fooled." - starball 2 hours ago * @starball the link isn't loading for me. I was so ready to see what you linked. - Script47 2 hours ago * 1 (the angry face meme) - starball 2 hours ago * 2 Re "group": It was copy pasted from the slack chat where they notified the employees. Guarantee it. - Restore the Data Dumps 1 hour ago Add a comment | 12 We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. That ship has sailed. OpenAI's models famously have a knowledge cut-off date of 2021, and few people care. Stack Exchange data is gathered, and fed into these models, via the Common Crawl project. Nobody is using Stack Exchange Data Dumps to train AI models. Even if you were willing to violate the CC BY-SA licenses (of various versions) under which we contributed all that we have contributed, and release the data dumps under some restrictive license, it would achieve nothing, because these companies are already not respecting the restrictions of the licenses. Rightly or wrongly (imo, wrongly), All Rights Reserved is being interpreted as legal permission to perform bulk statistical analyses that launder away any hope of attribution, then justify the actions by the fact that attribution becomes technically impossible. Our CC BY-SA license doesn't stand a chance. If Prosus wants a return on its investment in Stack Overflow, it should consider putting its weight behind getting the "SA" part of the license enforced against the likes of OpenAI. (I'd happily waive 20% of my part of the settlement from a collective-action lawsuit, if you want to profit more directly, and I'm happy to negotiate something higher.) What you're doing now is just... destroying the network. For no good reason. I've stopped recommending people to Stack Overflow - when normally I would be teaching entire classes full of students how to use it. I am now actively avoiding contributing Q&A, which is a shame, because I've got 7 or 8 Q&A pairs tucked away in my drafts folder, meant for Stack Overflow, that I may never have the motivation to polish up now. What you hope to achieve here is unachievable. Ask your lawyer. Heck, ask on Law Stack Exchange. What you're doing is not helping, and you do not know that you do not know that. --------------------------------------------------------------------- Side note: SEDE is already gated behind a Google ReCAPTCHA for not-logged-in users. That's to prevent people taking down your servers with a trillion high-intensity unauthenticated requests, but it serves this purpose quite competently, too. (If you want to make it slightly stricter, gate un-ReCAPTCHA'd SEDE behind the 15-reputation barrier used for upvoting: we have solid empirical evidence that that's high enough to make abuse detectable and manageable.) Share Improve this answer Follow answered 1 hour ago wizzwizz4's user avatar wizzwizz4wizzwizz4 10.7k55 gold badges2525 silver badges3939 bronze badges Add a comment | -2 They're gone, and SO considers your data to be their data, for only them to misuse. Who's going to start NewSO and import the last available datadump? Let me know. Share Improve this answer Follow answered 2 hours ago DimeCadmium's user avatar DimeCadmiumDimeCadmium 12822 bronze badges New contributor DimeCadmium is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. 1 * 5 I understand being frustrated - even angry. I understand making ultimatums, at this point, because it is very serious. But I don't think this answer adds anything. - wizzwizz4 2 hours ago Add a comment | -21 Stack Overflow senior leadership is working on a strategy to protect Stack Overflow data from being misused by companies building LLMs. While working on this strategy, we decided to stop the dump until we could put guardrails in place. We are working on setting up the infrastructure to do this correctly in the age of LLMs --- where we continue to be open and share the data with our developer community but work to set up a formal framework for large AI companies that want to leverage the data. We are looking for ways to gate access to the Dump, APIs, and SEDE, that will allow individuals access to the data while preventing misuse by organizations looking to profit from the work of our community. We are working to design and implement appropriate safeguards and still sorting out the details and timelines. We will provide regular updates on our progress to this group. Share Improve this answer Follow answered 4 hours ago Jody Bailey's user avatar Jody BaileyStaffJody Bailey 27111 silver badge33 bronze badges New contributor Jody BaileyStaff is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. 34 * 22 In order to prevent companies building LLMs from getting our data (which they already have, and have had for years) your plan is to keep all of us from getting our data, am I reading that right? - Restore the Data Dumps 4 hours ago * 28 Just as context for casual readers since it may not be obvious, Jody is our CTO. (I am not commenting on the matter at hand, just providing this info.) - balpha StaffMod 4 hours ago * 33 Why are you not doing this openly? This concerns the data provided by the community; why are you not letting the community have a say in how this data is protected against misuse? Why don't you use our knowledge for this purpose? Excluding us from the process only brings anger, negative speculations, and mistrust. We get so much negative by doing it this way, when we could've cooperated, and gotten so much good from it instead. What is your motivation for not including us in the process? Why is there no transparency in this? - Andreas detests censorship 4 hours ago * 47 Why was this not stated in March, when the decision was made to turn off future data dumps? Or, even better, why was this not discussed to get feedback from impacted members of the community to better assess the risks of turning off the data dump versus not turning off the data dump? - Thomas Owens 4 hours ago * 25 So you could say that. You say that now only because you were caught red handed, that's what I think. - Shadow Wizard Strikes Back 4 hours ago * 26 Also, it's not "Stack Overflow data". The content here is our data that we have chosen to license. So perhaps the creators and owners of that data should be more involved in making decisions about how their data is distributed. - Thomas Owens 4 hours ago * 26 This question has been sitting on Meta for two days. This answer would look more credible if it was the first one posted. Now it just looks like poor damage control. And even then there is still a question why has this not been communicated before? - Resistance Is Futile 4 hours ago * 30 "misuse by organizations looking to profit from the work of our community" - I hope this means you won't start charging for API access, because most of us would consider that a misuse by an organisation looking to profit from our work. - kaya3 4 hours ago * 20 "organizations looking to profit from the work of our community" -- that's funny when you say it like that, you know :) - Levente 4 hours ago * 28 How can data licensed under the CC-BY-SA licenses that SE content is licensed under be "misused"? The license explictly allows others to do essentially anything they want with the data as long as attribution is given, in particular profit off of it. It can't be "misused*, the users can only fail to give attribution or follow the share-alike requirement. It is entirely unclear in what ways restricting access to the data would improve the situation with respect to attribution. - ACuriousMind 4 hours ago * 10 As other comments said... First, why waiting (again) for someone to "catch you red handed". Second, how this fits in with the question Is SE (....) selling our content for AI model training? And what exactly does "reinvest back into our communities" mean?. Does this imply a change of plan or should be interpreted as an action to avoid others getting for free the data you plan to sell? - SPArcheon 4 hours ago * 36 I really, really want to take this at full face value. I also think the Company really needs to understand how important this data dump is to the community, and how easy it is to read this as a message that the dumps are "temporarily postponed" indefinitely, and will never return. I don't think the Company gets to have the benefit of the doubt with messaging like this, this week of all weeks... I beg you bear in mind how the messaging comes across, and how easy it feels, from the outside, to read this as a way to misdirect and soften the blow of doing away with the data dumps for good. - zcoop98 3 hours ago * 5 Please just release it and let those AIs choke on it. Your estimates in the data post are off by orders of magnitude. AI posts are way more rampant than you publicly dare to admit. Let them feed that AI-generated nonsense back into their algorithms and watch the trend burn, please. - CodeCaster 3 hours ago * 12 What happened to at least giving heads-up that something like this was going to happen? That's what community used to mean. This new way of just doing stuff without any communication isn't the best community experience - vbnet3d 3 hours ago * 44 Hey, welcome to SO! You're new here, but someone should have told you: traditionally we used the Reddit dumpster fires as examples of what not to do. - Shog9 2 hours ago | Show 19 more comments You must log in to answer this question. Not the answer you're looking for? Browse other questions tagged * support * status-planned * data-dump . Welcome! Welcome! Meta Stack Exchange is intended for bugs, features, and discussions that affect the whole Stack Exchange family of Q&A sites. About Help * Featured * Statement from SO: June 5, 2023 Moderator Action * We are graduating the updated button styling for vote arrows Visit chat Linked -2 When does the SE "senior leadership" plan to re-enable the uploads of the data dump to Archive.org? 379 What Experts-Exchange thinks of Stack Overflow -375 Statement from SO: June 5, 2023 Moderator Action -35 2023 Layoff impacting 58 SO/SE staff (2023-05-10) 78 Is SE [going to be] selling our content for AI model training? And what exactly does "reinvest back into our communities" mean? 60 All Stack Exchange data dumps 17 Waiting for March 2022 dump 78 Will SE shut down the free API for good? 12 Could someone legally replace Stack Exchange in posting the data dumps to the Internet Archive? 11 Form to request erasure under GDPR doesn't indicate that attribution will be removed from all posts, a possible license violation See more linked questions Related 13 Create a data dump RSS feed 372 Database schema documentation for the public data dump and SEDE 639 Formatting Sandbox 733 How do I get attention for one of my own questions without a good answer? 610 How does the bounty system work? 12 Release schedule for the data dump? 12 Data dump for the month of March 2012 25 Options for community data dump 1011 Firing Community Managers: Stack Exchange is not interested in cooperating with the community, is it? Hot Network Questions * Visualizing point attribute (int) with multiple dots / SVGs in QGIS * .exe with Digital Signature, showing SHA1 but the Certificate is SHA384, is it secure? * Flow Traders math test * Why is there current if there isn't any potential difference? * Longest English homophone/heteronym chain * How many points should we remove so that there are no n points lying on a line? * US tourist visa for as yet unknown short notice trips * C++ auto keyword when using initialize list * Pushing object backwards means no work is done? * Replicating a chronodex in TikZ * How to remove the vertical spacing in align * Paper with potentially inappropriately-ordered authors, should a journal act? * What bread dough is quick to repair and requires no kneading or much skill? * Is there a word that's the relational opposite of "Childless"? (Specifically for when trying to categorize an adult) * I gave items to a Charity Shop to sell, and signed up for Gift Aid on the donated items when sold. Should I put this on my Self-Assessment? * Why is Nietzsche an important philosopher? * A* algorithm implementation in modern C++ * Why give gifts AFTER the cup of parting? * Ordering on two columns is less efficient than order on only the first, why? * Why does Ash say "I choose you" instead of "I chose you" or "I'll choose you"? * How could a sapient creature lose its intelligence? * Why isn't the druid affected by her enchanted cuffs? * Why do we need to make a tensor for the electromagnetic field? * It it possible to have every app have a different IP address more hot questions Question feed Subscribe to RSS Question feed To subscribe to this RSS feed, copy and paste this URL into your RSS reader. [https://meta.stackex] * Meta Stack Exchange * Tour * Help * Chat * Contact * Feedback Company * Stack Overflow * Teams * Advertising * Collectives * Talent * About * Press * Legal * Privacy Policy * Terms of Service * Cookie Settings * Cookie Policy Stack Exchange Network * Technology * Culture & recreation * Life & arts * Science * Professional * Business * API * Data * Blog * Facebook * Twitter * LinkedIn * Instagram Site design / logo (c) 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. rev 2023.6.8.43486 Your privacy By clicking "Accept all cookies", you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy. Accept all cookies Necessary cookies only Customize settings