[HN Gopher] Generate Synthetic Data in 3 Lines of Code
___________________________________________________________________
Generate Synthetic Data in 3 Lines of Code
Author : mmegger
Score : 45 points
Date : 2022-09-07 16:11 UTC (6 hours ago)
(HTM) web link (gretel.ai)
(TXT) w3m dump (gretel.ai)
| johnwatson11218 wrote:
| Does anyone know if there are deep learning libraries that can
| model the relations between table based data? I see many that
| work on a table or a data frame but where I work our db is over
| 1000 tables and nobody can understand it. I feel like the next
| frontier is a tool that you point to your oracle or sql server
| and it compresses the table space. Whether you consider it a kind
| of PCA dimensionality reduction or the logical extension of the
| db "normal" forms ... it is just compression.
| andrewnc wrote:
| We have some preliminary work in this direction
| https://github.com/gretelai/multi-table
|
| I love the idea of "table space" though. It would be fun to
| traverse this space and output a new database at each step,
| like a VAE.
| rockemsockem wrote:
| I believe that random forests are still the primary way data
| scientists work with tabular data. Deep learning hasn't cracked
| tabular data like it has other areas.
| johnwatson11218 wrote:
| Everytime I look into this stuff it is just one
| table/dataframe. Nobody is modelling the relations between
| things. I think there is a huge opportunity for a product
| that can look at Customer, Items, Orders, Returns,
| Payment_Methods etc. and first of all show me when things
| tend to co-occur, so that it could generate synthetic
| customers that have the statistically correct number of
| registered payment methods where those payment details are
| also synthetically generated.
|
| Next would be the ability to decompose or factor my entire db
| into the subcomponents and make suggestions for combining
| tables. The use case would be a legacy enterprise system that
| has grown so complex and tangled that devs are afraid to do
| basic db refactoring. From where I'm sitting and working this
| is the next gold rush, apply DL methods to the bread and
| butter computing, log flow analysis, etc.
| cush wrote:
| There's a company called Tonic.ai that reminded me of your
| comment - they deeply analyze a db, including the data, and are
| able to find and strip out all of the PII, while keeping the
| various relationships in tact. Super interesting podcast on it
| here: https://softwareengineeringdaily.com/2021/09/29/faking-
| data-...
|
| They mention in the podcast that most customers end up finding
| relationships in their tables that they didn't know they had -
| that weren't explicitly in schema
| chiarabeth wrote:
| Thanks for the shoutout, cush! And yup, our platform Tonic
| enables developers to realistically de-identify their data
| while preserving relationships and consistency across tables
| within their DBs, to optimize dev and test with real fake
| data. You can sign up for a sandbox here:
| https://www.tonic.ai/
|
| We've also recently released a new platform called Djinn that
| is specifically designed for data science workflows. It
| enables you to query from tables across your DB to build
| customized views of only the data you need and synthesize
| high-fidelity data based on models trained on those views.
| Relationships are fully preserved and no external scripting
| is required. You can create an account and take it for a spin
| here: https://djinn.tonic.ai/?signup
|
| Full disclosure, I'm Chiara Colombi, Product Marketing
| Manager at Tonic.ai. Cheers!
| andrewnc wrote:
| We've been working hard to make it super simple to get started
| with useful synthetic data.
|
| If you want to know how you would go and use this for your own
| problems check out some of our other posts
|
| https://gretel.ai/blog/how-to-safely-work-with-another-compa...
| niviksha wrote:
| My use case is generating a very high rate (10k e/s up to 100k
| e/s) of JSON-NL events from samples of JSON-encoded log data
| (JSON-NL to be exact). Is this supported in OSS Gretel?
|
| FYI, I'd built a hand-crafted generator using JSONNet templates
| and Golang, but I really wanted something that could model
| source data distributions accurately. The use case is large-
| scale load testing of customer workloads without requiring
| actual data.
| andrewnc wrote:
| We're currently beta testing something that fits this use
| case directly. The models we have today are really great at
| capturing the original distribution, but they're not always
| the fastest. This new stuff will change that, feel free to
| reach out (maybe on our slack?) and we can see if we can get
| something working
| andrewnc wrote:
| Blog post is out now https://gretel.ai/blog/introducing-
| gretel-amplify
|
| They get 43,300 records per second on this example, which
| seems to the right order of magnitude for you
| jmole wrote:
| ...and you too can learn the biases and weights matrix of our
| synthetic data generator!
| BobbyJo wrote:
| Models are trained on input data to generate synthetics data
| similar to the input. It's not so much 'a' synthetic data
| generator. It's more like a platform for creating your own
| synthetic data generators using your own data and ML.
| niviksha wrote:
| Wow, this is great. I built my own synthetic time series data
| generator for benchmarking, could have saved myself a bunch of
| trouble with this.
___________________________________________________________________
(page generated 2022-09-07 23:02 UTC)