https://github.com/dylan-profiler/visions Skip to content Sign up * Why GitHub? + Features + Mobile + Actions + Codespaces + Packages + Security + Code review + Issues + Integrations + GitHub Sponsors + Customer stories * Team * Enterprise * Explore + Explore GitHub + Learn and contribute + Topics + Collections + Trending + Learning Lab + Open source guides + Connect with others + The ReadME Project + Events + Community forum + GitHub Education + GitHub Stars program * Marketplace * Pricing + Plans + Compare plans + Contact Sales + Education [ ] * # In this repository All GitHub | Jump to | * No suggested jump to results * # In this repository All GitHub | Jump to | * # In this organization All GitHub | Jump to | * # In this repository All GitHub | Jump to | Sign in Sign up {{ message }} dylan-profiler / visions Public * Notifications * Fork 11 * Star 114 * Type System for Data Analysis in Python dylan-profiler.github.io/visions/visions/getting_started/usage/ types.html View license 114 stars 11 forks Star Notifications * Code * Issues 15 * Pull requests 0 * Actions * Projects 0 * Wiki * Security * Insights More * Code * Issues * Pull requests * Actions * Projects * Wiki * Security * Insights This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository. develop Switch branches/tags [ ] Branches Tags Could not load branches Nothing to show {{ refName }} default View all branches Could not load tags Nothing to show {{ refName }} default View all tags 7 branches 21 tags Code Latest commit @sbrugman sbrugman ci: update dependencies ... 8c8f4cf Jan 30, 2022 ci: update dependencies 8c8f4cf Git stats * 1,023 commits Files Permalink Failed to load latest commit information. Type Name Latest commit message Commit time .github ci: disable pytest-spark Oct 26, 2021 docs [skip ci] Updating documentation Sep 27, 2021 docsrc Master -> Dev: Docs update (#180) Sep 15, 2021 examples Remove summary application (#120) May 5, 2021 images Readme Updates (#193) Jan 10, 2022 paper Update paper.md Apr 11, 2020 src/visions ci: dependency update Dec 6, 2021 tests WIP: Tests/python backend (#189) Nov 9, 2021 .gitignore lgtm checks and check-manifest Oct 14, 2020 .pre-commit-config.yaml ci: update dependencies Jan 30, 2022 LICENSE Update LICENSE Feb 10, 2020 MANIFEST.in Readme Updates (#193) Jan 10, 2022 Makefile feat: spark standard set (#173) May 18, 2021 README.md Readme Updates (#193) Jan 10, 2022 make.bat feat: spark standard set (#173) May 18, 2021 pytest.ini feat: spark standard set (#173) May 18, 2021 requirements.txt deps: bottleneck not py39 compatible Sep 27, 2021 requirements_dev.txt ENH: Typeset plot enhancements (#137) Oct 10, 2020 requirements_spark.txt feat: spark standard set (#173) May 18, 2021 requirements_test.txt spark requirement + complex fix (#186) Nov 9, 2021 setup.cfg rtd wants a contents.rst May 20, 2019 setup.py Readme Updates (#193) Jan 10, 2022 tox.ini works in tox Nov 28, 2019 View code The Semantic Data Library Installation Quick Start Guide Supported frameworks Contributing and support Acknowledgements README.md [visions] And these visions of data types, they kept us up past the dawn. [6874747073] [6874747073] [6874747073] [6874747073] [6874747073] [6874747073] The Semantic Data Library Visions provides a set of tools for defining and using semantic data types. * [*] Semantic type detection & inference on sequence data. * [*] Automated data processing * [*] Completely customizable. Visions makes it easy to build and modify semantic data types for domain specific purposes * [*] Out of the box support for multiple backend implementations including pandas, spark, numpy, and python * [*] A robust set of default types and typesets covering the most common use cases. Check out the complete documentation here. Installation Source code is available on github and binary installers via pip. # Pip pip install visions Complete installation instructions (including extras) are available in the docs. Quick Start Guide If you want to play immediately check out the examples folder on [6874747073]. Otherwise, let's get some data import pandas as pd df = pd.read_csv("https://raw.githubusercontent.com/datasciencedojo/datasets/master/titanic.csv") df.head(2) +-------------------------------------------------------------------------------------------+ |PassengerId|Survived|Pclass| Name | Sex |Age |SibSp|Parch|Ticket| Fare |Cabin|Embarked| |-----------+--------+------+---------+------+----+-----+-----+------+-------+-----+--------| | | | |Braund, | | | | |A/5 | | | | |1 |0 |3 |Mr. Owen |male |22.0|1 |0 |21171 |7.2500 |NaN |S | | | | |Harris | | | | | | | | | |-----------+--------+------+---------+------+----+-----+-----+------+-------+-----+--------| | | | |Cumings, | | | | | | | | | | | | |Mrs. John| | | | | | | | | |2 |1 |1 |Bradley |female|38.0|1 |0 |PC |71.2833|C85 |C | | | | |(Florence| | | | |17599 | | | | | | | |Briggs | | | | | | | | | | | | |Thayer) | | | | | | | | | +-------------------------------------------------------------------------------------------+ The most import abstraction in visions are Types - these represent semantic notions about data. You have access to a range of well tested types like Integer, Float, and Files covering the most common software development use cases. Types can be bundled together into typesets. Behind the scenes, visions builds a traversable graph for any collection of types. from visions import types, typesets # StandardSet is the basic builtin typeset typeset = typesets.CompleteSet() typeset.plot_graph() [6874747073] Note: Plots require pygraphviz to be installed. Because of the special relationship between types these graphs can be used to detect the type of your data or infer a more appropriate one. # Detection looks like this typeset.detect_type(df) # While inference looks like this typeset.infer_type(df) # Inference works well even if we monkey with the data, say by converting everything to strings typeset.infer_type(df.astype(str)) >> { 'PassengerId': Integer, 'Survived': Integer, 'Pclass': Integer, 'Name': String, 'Sex': String, 'Age': Float, 'SibSp': Integer, 'Parch': Integer, 'Ticket': String, 'Fare': Float, 'Cabin': String, 'Embarked': String } Visions solves many of the most common problems working with tabular data for example, sequences of Integers are still recognized as integers whether they have trailing decimal 0's from being cast to float, missing values, or something else altogether. Much of this cleaning is performed automatically providing nicely cleaned and processed data as well. cleaned_df = typeset.cast_to_inferred(df) This is only a small taste of everything visions can do including building your own domain specific types and typesets so please check out the API documentation or the examples/ directory for more info! Supported frameworks Thanks to its dispatch based implementation Visions is able to exploit framework specific capabilities offered by libraries like pandas and spark. Currently it works with the following backends by default. * Pandas (feature complete) * Numpy (boolean, complex, date time, float, integer, string, time deltas, string, objects) * Spark (boolean, categorical, date, date time, float, integer, numeric, object, string) * Python (string, float, integer, date time, time delta, boolean, categorical, object, complex - other datatypes are untested) If you're using pandas it will also take advantage of parallelization tools like swifter if available. It also offers a simple annotation based API for registering new implementations as needed. For example, if you wished to extend the categorical data type to include a Dask specific implementation you might do something like from visions.types.categorical import Categorical from pandas.api import types as pdt import dask @Categorical.contains_op.register def categorical_contains(series: dask.dataframe.Series, state: dict) -> bool: return pdt.is_categorical_dtype(series.dtype) Contributing and support Contributions to visions are welcome. For more information, please visit the community contributions page and join on us on slack. The github issues tracker is used for reporting bugs, feature requests and support questions. Also, please check out some of the other companies and packages using visions including: * pandas profiling * Compressio * Bitrook If you're currently using visions or would like to be featured here please let us know. Acknowledgements This package is part of the dylan-profiler project. The package is core component of pandas-profiling. More information can be found here. This work was partially supported by SIDN Fonds. [SIDNfonds] About Type System for Data Analysis in Python dylan-profiler.github.io/visions/visions/getting_started/usage/ types.html Topics python data-science spark numpy pandas data-analysis type-system type-inference hacktoberfest Resources Readme License View license Stars 114 stars Watchers 4 watching Forks 11 forks Releases 15 visions v0.7.5 Latest Dec 5, 2021 + 14 releases Packages 0 No packages published Used by 2.5k * @ygonzalez * @nram812 * @calpis10000 * @jpvazquezz * @wasuratme96 * @Geodego * @avpresbitero * @ypandya4 + 2,503 Contributors 9 * @sbrugman * @ieaves * @actions-user * @dah33 * @dependabot-preview[bot] * @arfon * @ecederstrand * @dependabot[bot] * @cmmm976 Languages * Python 94.0% * HTML 1.6% * TeX 1.5% * Makefile 1.2% * Jupyter Notebook 1.1% * Batchfile 0.6% * (c) 2022 GitHub, Inc. * Terms * Privacy * Security * Status * Docs * Contact GitHub * Pricing * API * Training * Blog * About You can't perform that action at this time. You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.