[HN Gopher] A Critical Field Guide for Working with Machine Lear...
___________________________________________________________________
A Critical Field Guide for Working with Machine Learning Datasets
Author : Amorymeltzer
Score : 76 points
Date : 2023-02-16 12:58 UTC (1 days ago)
(HTM) web link (knowingmachines.org)
(TXT) w3m dump (knowingmachines.org)
| clircle wrote:
| 100 years ago statisticians starting working on the theories of
| survey design and experimental design to clarify what types, how
| much, what mixture, and how to collect data. Machine learning
| people could learn a lot from these old, uncool theories if they
| want to approach data with rigor. That one can approach data
| collection and usage in a systematic way is totally lost now,
| it's just web scraping.
| kordlessagain wrote:
| A system for this would allow individual collections of
| documents with discussions attached. Without people, the
| machines won't know where to put attention. That may change
| when they have individual identity, but for now the work
| probably needs to be done cooperatively.
| steppi wrote:
| This hasn't been my experience. I don't think this knowledge
| has been lost and it's often still applied when incentivized,
| such as when quality output of machine learning models serves
| an immediate business need, or in scientific applications of
| machine learning. By your reference to web scraping I assume
| your referring to datasets for large language models, but I
| don't think the kitchen sink approach is applied out of
| ignorance, but out of a principled decision to forgo data
| hygiene in order to collect datasets large enough to feed such
| large models. Also, strictly speaking, one only needs to apply
| the critical approach when constructing and using validation
| datasets, and it's typical for larger noisier training datasets
| to outperform smaller more rigorous ones when models trained on
| both are tested on carefully constructed validation datasets.
| razor_router wrote:
| I definitely agree that taking a systematic approach to data
| collection and usage is incredibly important. However, I think
| it is impossible to simply dismiss the ways that modern machine
| learning has revolutionized the way we understand and interact
| with data. There is no question that rigorous survey design and
| experimental design theories still have merit and should be
| taken into account, but I think machine learning has allowed us
| to delve deeper into data analysis and uncover patterns and
| insights that would not be possible without its advancements.
| KRAKRISMOTT wrote:
| Machine learning (or more specifically _deep learning_ ) is
| shaking the world precisely because the models at the highest
| end are able to take in mostly unlabeled garbage and spit out
| useful results in an unsupervised manner. The world is not full
| of i.i.d. spherical cows.
|
| 100 years of statistical advances have not been able to solve
| image recognition at scale until convolutional neural networks.
| Large language models may be _somewhat_ Bayesian but they do
| not resemble any traditional Bayesian techniques except at the
| highest level. Clean data and good sampling are incompatible
| with the real world outside of laboratory conditions (or large
| budgets).
| avs733 wrote:
| That is a fairly brave statement given some of the research
| looking at how failed attempts at ML modeling fall apart [1]
|
| [1] https://ai.googleblog.com/2021/06/data-cascades-in-
| machine-l...
| wongarsu wrote:
| Some of them do, but others depend on labeled and augmented
| data. For example training facial recognition starts with
| images labeled by who appears in them, then you run a model
| to find the face in the image and crop to that, then another
| image tells you where the eyes are and you use that to rotate
| the image to make the eyes level, and that's the input to the
| actual facial recognition neural network. And that's for the
| "naive" approaches that don't have special handling for faces
| don't look at the camera. And the field still routinely has
| problems because it turns out that somebody's data set only
| contained certain lighting conditions, or severely
| underrepresented certain races, etc.
| KRAKRISMOTT wrote:
| Not sure about image recognition but for the language
| models and audio space, semi/self-supervised methods are
| currently doing quite well.
| dachryn wrote:
| you are too pessimistic. Senior profiles definitely dived into
| the statistics aspect of it all. Experiment design is often a
| big part of modern business cases, because naturally it is part
| of the cost to obtain data.
|
| It is still taught at university if you do a master degree or
| PhD at a decent place.
|
| But I agree that many juniors have no clue. Especially those
| who are self taught and join in from a different field
| m00dy wrote:
| terrible layout to read..
| peanutcrisis wrote:
| Note that the term "Critical" here refers to the likes of
| "Critical Theory", "Critical Social Justice", "Critical Race
| Theory" etc. It's biased from the offset so I'd take anything
| from this site with a pinch of salt.
| jna_sh wrote:
| I love the faux spreadsheet presentation of this guide.
| firatsarlar wrote:
| +1
| r3trohack3r wrote:
| I didn't realize it wasn't a "real" spreadsheet application
| until your comment.
|
| The design gave me a bit of anxiety, it was so convincing I was
| worried about clicking into one of the cells!
|
| Anxiety included - I also love the layout.
| kordlessagain wrote:
| Interesting it doesn't cover technologies for storing data. It
| does mention analytical operations though.
|
| Anyone have a suggestion for storing large amounts of ML data,
| training sets or otherwise? I've been using FeatureBase and
| weaviate, and would be interested in learning about other
| solutions.
| kacperlukawski wrote:
| If you want to perform vector search over your data, then
| Qdrant (https://qdrant.tech) is worth checking out.
| gschoeni wrote:
| We've been working on an open source tool called Oxen to help
| store large ML datasets. It's optimized for large sets of
| unstructured data ie images, video, audio, text, as well as
| parquet or arrow style DataFrames.
|
| Would love to get some feedback on it!
|
| https://github.com/Oxen-AI/oxen-release#-oxen
| wongarsu wrote:
| "ML data" could be just about anything: images, sounds, graphs,
| tabular data, vectors. Depending on what your data is, you
| might end up with very different storage solutions.
|
| Though I think one common approach is to just dump most data in
| an S3-compatible datastore of your choice (there's seaweedfs or
| ceph, or the cloud provider of your choice). Specialized
| databases make a lot of sense when you need features like
| vector similarity search though.
| fzliu wrote:
| Vector databases in general are good for storing large amounts
| of unstructured data by first converting them into embeddings
| via ML models. There's also feature stores, which store and
| organize features for later use in model training or predictive
| analytics. Feature stores generally come in _before_ models get
| trained, while vector databases generally come _after_ (i.e.
| they use trained models).
|
| Milvus (https://milvus.io) and Feast (https://feast.dev/) are
| two of the most well known vector databases and feature stores,
| respectively.
___________________________________________________________________
(page generated 2023-02-17 23:02 UTC)