[HN Gopher] Obsidian Dataview: Turn Obsidian Vault into a databa...
___________________________________________________________________
Obsidian Dataview: Turn Obsidian Vault into a database you can
query from
Author : superasn
Score : 186 points
Date : 2022-05-17 09:09 UTC (13 hours ago)
(HTM) web link (github.com)
(TXT) w3m dump (github.com)
| kepano wrote:
| Dataview is incredible. Being able to query plain text notes and
| their metadata has been transformative for my workflow. It has
| allowed me to move virtually all my thought processing work into
| Obsidian. I am constantly amazed by how performant Dataview is.
|
| I sometimes wonder if learning how to use Dataview inside of
| Obsidian is too technical to get mass adoption, compared to
| WYSIWYG tools like Notion that let you build databases with nice
| UI filtering. But the performance of Dataview is so much better,
| and the control you have over queries so much more granular, that
| it's worth the learning curve. The Obsidian community is also
| incredibly good at helping new users learn, which makes me feel
| optimistic.
|
| To help make things a bit more user-friendly I have been building
| an Obsidian theme called Minimal[1] which allows you to take
| Dataview tables and display them in a card layout[2]. I really
| enjoy that layout for certain types of information, while having
| all the underlying power of Dataview.
|
| [1]: https://github.com/kepano/obsidian-minimal
|
| [2]: https://minimal.guide/Block+types/Cards
| jonnydubowsky wrote:
| Minimal theme w/ Dataview card layout is awesome! I'm just
| digging into it, but it does make the queries more accessible.
| Thanks!
| rpdillon wrote:
| If you like this idea, but prefer something more self-contained,
| Tiddlywiki is an excellent option, as it runs in-browser from a
| single HTML file. Its core approach is to treat notes as a
| queryable database, and its markup supports dynamic content based
| on queries of that DB. Because its UI is constructed from the
| same DB, the same techniques that allow you to construct a
| dynamic view from the DB also allow you to construct custom UIs
| for that view, including changing properties/attributes of items
| in the DB.
|
| That's all pretty abstract. A simple example: it allows you to
| construct a tabbed view of several todo lists based on tags and
| fields and adjust priority and status from that single UI.
|
| It's been available for many years, but has never been trendy,
| despite its unique design and strengths. I recommend giving it a
| spin!
| RyEgswuCsn wrote:
| I use tiddlywiki (with the nodejs setup) regularly. It's great
| until your wiki grows too large, at which point you need to
| wait ~10s for it to load even with modern hardware.
| rpdillon wrote:
| I'd be very curious to hear about the sizes and use cases
| where you're seeing this. My understanding was that single
| file wikis could scale to a hundred megabytes and beyond, and
| I thought the nodejs version was more scalable than that due
| to lazy loading tiddlers. My wikis tend to be less than 20mb
| since I mostly use plain text for everything.
| RyEgswuCsn wrote:
| I do make use of the `backlinks` query extensively, and I
| use saqimtiaz's wonderful `streams` [1] plugin to recreate
| a Roam-like experience. This setup leads to a large amount
| of small-sized tiddlers (2.5k and counting), which might
| add to the overhead.
|
| Not to mention downloading and parsing 20mb-worth of
| html+js each time you open up the wiki can be quite slow.
|
| As for lazy loading --- when enabled, it breaks the
| indexing (e.g. finding backlinks) and searching because
| these are done by the js inside the browser too. I only
| lazy-load images and binaries because they are not
| indexable anyway.
|
| [1]: https://github.com/saqimtiaz/streams
| pqs wrote:
| Can Tiddlywiki be used by a team with simultaneous edits?
| RyEgswuCsn wrote:
| You can. But you need to host it with a nodejs server.
| rpdillon wrote:
| Not at all, AFAIK. Completely single-user in my experience.
| lr1970 wrote:
| > It's been available for many years, but has never been
| trendy, despite its unique design and strengths. I recommend
| giving it a spin!
|
| I was a heavy TiddlyWiki user in the past but moved away from
| it precisely because of its singe-HTML file design. Modern
| browsers like chrome made it virtually impossible to save an
| edited page. Not TW's fault per se, but ruined my experience
| with it. There is a way to install TW-5 with node.js as a
| directory tree but my ship has sailed. Now I am a happy
| obsidian user.
| rpdillon wrote:
| Yes, that aspect of the design makes it feel like a toy,
| which is a shame. I've written my own tooling for saving to
| bridge this gap. It's a Python script that provides a
| response to an OPTIONS call so Tiddlywiki's native WebDAV
| support allows seamless saving. It also provides automatic
| backups and an index page to serve multiple wikis. I've used
| it for my last two jobs and in my personal life and I'm
| really happy with it. I hope to make it available publicly in
| case others find it useful, just haven't gotten around to it.
| slightwinder wrote:
| Dataview is pretty nice, but last time I played with it, I missed
| a bit interactivity. By which I mean working with the results on
| the fly, like sorting them inplace, selecting individual entries
| and move them to new results. Is this possible now, or in
| progress?
| EleanorKonik wrote:
| This one is still in the manual-install phase, but you might
| like this plugin: https://github.com/tomaszkiewicz/obsidian-
| database-plugin
| slightwinder wrote:
| This looks really nice. Is there a timeline when it will be
| installable through obsidian?
| brimwats wrote:
| should be pretty soon tbh, code review is almost done
| rrherr wrote:
| You can sort Dataview results using this plugin:
| https://github.com/alexandru-dinu/obsidian-sortable
| slightwinder wrote:
| Ah, thanks. Makes sense that other extensions will also work
| for this.
| mpalmer wrote:
| Moving dataview entries between data views is fundamentally at
| odds with the paradigm in which dataview operates. The point is
| that it's ephemeral data that's entirely derived from the state
| of your notes.
|
| Maybe there are some basic queries that could support "two-way
| binding" of this sort, but it would be fiendishly complex and
| ultimately better solved by a different tool altogether
| slightwinder wrote:
| > Moving dataview entries between data views is fundamentally
| at odds with the paradigm in which dataview operates.
|
| It doesn't need to be between dataviews. Just temporary
| filter the view, or export it in a regular static table. This
| is just about enhancing the View, not changing the concept.
|
| Which opens the question, can dataview also take tables as
| data-source?
|
| > Maybe there are some basic queries that could support "two-
| way binding" of this sort
|
| I'm not talking about Two-way-bindings. Though this would be
| awesome. And I don't think it would be that hard to implement
| in general, depening on how exactly obsidian works internally
| with dataview.
| lfkdev wrote:
| Thats a really cool idea
| EleanorKonik wrote:
| Honestly this is one of the handful of plugins I hate trying to
| live without.
|
| Tag wrangler is the other.
| TuringTest wrote:
| Hierarchical notetakers/outliners are becoming a new data
| structure for end users, specialized in knowledge management. A
| kind of generalized, expandable spreadsheet.
|
| I expect these tools to include more and more tools for
| automation based on a reactive programming style (more suited to
| non-programmers), expanding its function as a warehouse of ideas
| into a more active role for user-created applications. The NOCODE
| camp is growing from its VB6-like visual-style roots into
| environments that look more like an IDE in the style of the
| browser's "Inspect" window.
|
| The key development that I miss is that of _reifying_ query
| results, allowing the user to use the data generated by a search
| as an object in itself. I even have a name for that reified
| object - a _wit_ would be a collection of data compiled with a
| common criteria (representing a _minimum unit of meaning_ , is a
| wordplay on _bit_ being the minimum unit of knowledge), similar
| to a spreadsheet range. When tools like Obsidian or Logseq start
| allowing the user to use query results as first-class data, I
| predict that we will see a revolution in end-user process
| automation.
| greggsy wrote:
| I'm using obsidian, but I'd move back to OneNote (or even Apple
| Notes) if it offered inclusions and more flexible tag
| management and reporting. I feel like MS could do that
| somewhere down the line.
| stjohnswarts wrote:
| Can't stand the idea of my personal notes being in the form
| of a proprietary format though. That's the key advantage for
| me.
| slightwinder wrote:
| > Hierarchical notetakers/outliners are becoming a new data
| structure for end users, specialized in knowledge management.
|
| Outliners are used in this area for decades. What we have now
| is just a new hype-cycle, with new apps, new technologies and a
| new mix of features.
|
| > When tools like Obsidian or Logseq start allowing the user to
| use query results as first-class data, I predict that we will
| see a revolution in end-user process automation.
|
| So, SQL?
| quaunaut wrote:
| > Outliners are used in this area for decades. What we have
| now is just a new hype-cycle, with new apps, new technologies
| and a new mix of features.
|
| To say that what we have now is nothing but a new hype cycle
| while the Zettelkasten method was only widely exposed within
| the last couple of decades, and was not utilized in the
| design of past Outliners, shows you don't really understand
| what's actually changed.
| slightwinder wrote:
| Zettelkasten is nothing new and just hype. Only shows that
| you just don't know this space, and it's history. Obviously
| there is improvement, but it's not in methods or data-
| structures, it's with technology and implementations. We
| are now at the point where we can move fast with
| implementations and glue much more features together.
| quaunaut wrote:
| Then please, illuminate me. Because the implications of
| Luhmann's use of the Zettelkasten weren't widely
| available till the last decade, and weren't available in
| software until the same period.
|
| So what exactly don't I know? Genuinely.
| slightwinder wrote:
| Luhmann methods are known since the 80s, at least in
| europe. And I think I saw the first digital
| implementation around 2006 or 2008. Looking it up, the
| first(?) Digital Zettelkasten after Luhmann method was in
| 2004 already at version 2.5, so it's likely quite older.
| But dates are not important here.
|
| Luhmanns method is simply not that special, nor unique.
| Many scientists and artist used similar methods and
| tools. He is mostly just replicating what was common at
| the time, but with some twist. And in the 80s or 90s is
| would be really awesome, probably even somewhat
| groundbreaking. But today? It's just another graph-system
| which you find in wikis or hypertext for decades now. And
| most people are not even using the benefits of his
| number-system, just calling everything zettelkasten which
| is just a wiki otherwise, leaving even less of his
| originality. Though, in digital space it make less sense
| anyway, so not a big lose.
|
| But for whatever reason, there now this hype from people
| who seem to have know knowledge about those space or it's
| history, or even really understand the terms, thinking
| Luhmanns method is called Zettelkasten...
| ryanjamurphy wrote:
| > What we have now is just a new hype-cycle, with new apps,
| new technologies and a new mix of features
|
| You contradict yourself. Is it "just a new hype-cycle," or is
| it "new apps, new technologies, and a new mix of features?"
| If it's the latter... isn't that substantively different than
| what we had before?
|
| Outliners have quite the fandom, but my issue with the
| traditional outliner is that they often trap data in outline
| form. Sure, most good outliners offer export options.
| Unfortunately, the second you've taken data from your
| outliner and exported it, you now have two versions of your
| data, and you've lost.
|
| Logseq (and Obsidian) have changed this. Finally, a single
| document _is_ an outline, a backlink, a data source, a
| webpage, a presentation, and so on... The only thing the user
| has to do is work with that single page of information.
|
| It's more than hype for me.
| slightwinder wrote:
| > You contradict yourself. Is it "just a new hype-cycle,"
| or is it "new apps, new technologies, and a new mix of
| features?"
|
| The concepts are old and have a new hype. The apps are new.
| But generally, it doesn't matter much for the user whether
| you have some old-school desktop-app with gtk or Qt, or a
| "modern" electron app, or some webapp on your own server or
| some service in the cloud. At the end it's all the same
| ideas and concepts in different colors and flavors.
|
| > If it's the latter... isn't that substantively different
| than what we had before?
|
| That entirely depends on who is "we". We which are all
| users? Or we the old users? Or we the young users? Or we
| the new users? I'm not saying that there is not development
| or change, just that this is nothing new, or even a new
| development. This has going for a long time, it just
| reached a new cycle and and a new generation of users.
|
| > Outliners have quite the fandom, but my issue with the
| traditional outliner is that they often trap data in
| outline form. Sure, most good outliners offer export
| options. Unfortunately, the second you've taken data from
| your outliner and exported it, you now have two versions of
| your data, and you've lost.
|
| This is a fundamental problem of every app and space. But
| interoperability does exist here and there, it's not just
| exports.
|
| > Logseq (and Obsidian) have changed this.
|
| Not really? They only established a common foundation for
| interoperability. Which is also what other apps are doing
| to some degree. But you still have limitations by
| everything which is not covered by this foundation. For
| example, can Logseq handle dataview-syntax? Or generally
| any obsidian-plugin has it's own special syntax? What about
| other markdown-editors? How much do they support the
| special syntax used by obsidian or it's plugins?
|
| True, the markdown-space has now a healthy movement toward
| interoperability and gains many nice functionalities. But
| it's not like this is especially new. And all those
| features also existed before Markdown in some app, in some
| way or another. Nothing is really new here. What's
| happening is that a new platform is growing, collection
| nice stuff from the collective pool of abilities, and a new
| consolidation is happening. This happens all the time.
| Sometimes it works, sometimes it dies.
|
| Funny enough, the WWW was one of the first and most
| successful of this movements. And markdown grew from this
| as a simplification, to now grew into the same space where
| WWW started and obsidian now moves toward too, just in a
| different flavor. This whole "new" movement is just
| replicating where others were already before in different
| cloths.
|
| > Finally, a single document _is_ an outline, a backlink, a
| data source, a webpage, a presentation, and so on...
|
| So has Org-mode 10 years ago, XML 20 Years ago. HTML 30
| years ago. More or less, the numbers may be a bit off.
|
| > It's more than hype for me.
|
| Good for you. I'm not question that. Just saying this
| generation is not the first one having that.
| TuringTest wrote:
| I'm afraid you're entirely missing the relevance of the
| core fact that the syntax and runtime execution of
| programming languages are humongously difficult to
| understand for the large majority of computer users.
| Every time you create a system to automate some part of
| computer usage that doesn't rely on complex syntax and
| which executes in a declarative way, the same end result
| is created through a new concept, even if there existed a
| different process before that could also create it.
|
| HTML was a step in the right direction (and XML a
| misstep); before it, you needed to program a GUI API to
| lay out content intermixed with interactive components.
| Saying that the web brought nothing new because BBSs
| existed before it is missing out on the importance of the
| details in achieving that result. Precisely, "different
| cloths" is what makes the movement relevant to make it
| accessible to new users who could never have used the
| previous system.
|
| You're right in pointing out that interoperability is a
| key factor in the new batch of tools. The key point that
| neither Org-mode nor SGML-derived codes had is is to make
| user-entered content accessible to transformation tools
| without the need to code the transformation in an
| imperative language. Spreadsheets had this feature as
| well, sure, but spreadsheets didn't easily allow for
| hierarchical content.
|
| The main new aspect that logseq and Obsidian bring to the
| table is the possibility to backlink to any part of the
| hierarchical structure. The tools automatically compile
| all those links as a new data object, which _the user can
| process without creating a script to gather all that
| data_. HTML had the potential to popularize that feature
| 30 years ago (with a slightly more awkward syntax than
| Markdown), but browsers never got around to it.
|
| Previous tools would require users to move content to a
| different tool (say, by copy-pasting the content of your
| files to a spreadsheet to apply formulas), destroying the
| possibility of having a central personal knowledge
| repository. In logseq / Obsidian, you can keep adding
| multimedia content to your structured knowledge base, and
| exploit that content without coding at all, or at most by
| creating simple declarative expressions. What tool before
| these ones had that possibility before? With
| _hierarchical_ structures, not just flat structures like
| the spreadsheet?
| slightwinder wrote:
| > I'm afraid you're entirely missing the relevance of the
| core fact that the syntax and runtime execution of
| programming languages are humongously difficult to
| understand for the large majority of computer users.
|
| No, I know that, I just don't think it's a relevant
| problem for the user group we are talking about.
|
| > Saying that the web brought nothing new because BBSs
| existed before
|
| I did not say that? And in the first place, BBS competes
| with other networks like the Internet, not a service
| running in such a network.
|
| > The main new aspect that logseq and Obsidian bring to
| the table is the possibility to backlink to any part of
| the hierarchical structure.
|
| No, they do not. Linking is a fundamental concept of
| hypertext. And easy linking inside your dataspace, as
| also automatically listing backlinks, was already
| established with wikis 20+ years ago. Even hierarchies
| exists in wiki space for a long time now in several
| different ways.
|
| > The tools automatically compile all those links as a
| new data object, which the user can process without
| creating a script to gather all that data.
|
| What kind of low-level-tools have you used till now that
| this is your level of knowledge here??
|
| > In logseq / Obsidian, you can keep adding multimedia
| content to your structured knowledge base, and exploit
| that content without coding at all
|
| Yes, because someone else does the coding, as always.
| This is the benefit of a popular ecosystem. But this is
| not the point of having a scriptable and programmable
| app. The scripting just enable you to personalize on
| small levels. The programmability should enable the user
| and community to extend the program with new abilities,
| as dataview here is doing it. There are other extensions
| who enable scripting to certain degrees for the user, as
| obsidian is not really adding this on it's own.
|
| No app will ever be perfect for everyone. That's why
| extensions and scripting are beneficial, especially for
| this space.k
|
| > What tool before these ones had that possibility
| before? With hierarchical structures, not just flat
| structures like the spreadsheet?
|
| The constraints again. Why does it matter that
| obsidian&co. can do some more tricks than their
| ancestors, like playing videos, when the fundamental
| purpose and handling is still the same? That's like the
| old joke of taking something old and just add a clock, to
| sell it as new. More bells and whistle don't make a new
| church.
| TuringTest wrote:
| _> No, I know that, I just don 't think it's a relevant
| problem for the user group we are talking about._
|
| What user group are _you_ talking about? I 'm
| specifically talking about the large majority of users
| who could never learn to program in a syntax-heavy
| general-purpose programming language! X-D Those people
| nevertheless have a need to build their own userland
| applications, and no current tool serves them well; but a
| Nocode programmable knowledge database could do it.
|
| _> No, they do not. Linking is a fundamental concept of
| hypertext. And easy linking inside your dataspace, as
| also automatically listing backlinks, was already
| established with wikis 20+ years ago._
|
| And as I explained, no end-user tool has exploited that
| possibility to allow end users to access those automated
| links programatically with nocode tools (like for example
| with spreadsheet-like declarative expressions). The
| browser missed the chance by having no persistent storage
| and only allowing javascript as their programming
| language, and wikis never were nocode-friendly. Yet
| modern tools are just beginning to do that on top of a
| personal knowledge base.
|
| _> What kind of low-level-tools have you used till now
| that this is your level of knowledge here??_
|
| I beg you pardon? I've programmed all the way down from
| building my own flip-flops with electric transistors up
| to non-linear mathematical solvers for combinatorial
| optimization. I don't see the relevance, since low-level-
| tools will never be adequate development platforms for
| end-users.
|
| _> Yes, because someone else does the coding, as
| always._
|
| No. Logseq / Obsidian have tools that allow end users to
| build their own coding through templates and queries. The
| kind of coding is different from scripting, though; it's
| more like creating semi-automated workflows that put all
| relevant information in front of the user, which then
| proceeds to transform it with direct manipulation. What
| tool in the past has allowed end-users to create that
| kind of workflows from end-user templates and arbitrary
| queries, other than non-hierarchical spreadsheets?
|
| _> The constraints again. Why does it matter that
| obsidian &co. can do some more tricks than their
| ancestors_
|
| The constraints are there for a reason, and you don't
| seem to grasp why they are important. The spreadsheet ,
| and spreadsheets are typically not automated through
| scripting and extensions - using those is normally
| considered a failure of the spreadsheet model, because
| you're processing data outside the mental model of the
| spreadsheet as a programming environment i.e. cheating.
| Most End-User Development tools rely on the assumption
| that users can't program at all, so all automation
| concepts must be presented to them through alternate
| syntax and interactions, quite different from classic
| coding.
|
| A spreadsheet-like development environment based on
| outliners as its data structure, rather than the tables
| of a spreadsheet, should also be programmable by end
| users without relying on imperative scripts external to
| the tool. Logseq is creating the basis for that storage
| model, which some people are starting to exploit the kind
| of semi-automated workflows I talked about above.
|
| _> like playing videos_
|
| That's a red-herring, not a main point. I mentioned
| multimedia merely because programmable tools tend to be
| text only, forcing users to use separate tools for rich
| content, or import specific libraries to handle
| multimedia data as programmable objects (rather than
| editing them as native interactive content like they
| would on the GUI). The point is that end users should be
| able to treat all kinds of content equally with their
| tools, not being tied to the Unix shell constraint that
| all content is moved between processes as text streams or
| binary blobs.
| slightwinder wrote:
| > What user group are you talking about?
|
| The kind of people who use such tools. The kind of people
| who can lookup DSLs to create dataviews, or copy&paste
| snippets to make up stuff. Something like a knowledge-
| system is not used by children or someone similar.
|
| > Those people nevertheless have a need to build their
| own userland applications,
|
| No, they do not. Not every user around the world has this
| demand. Most user don't have this demand. We are talking
| here about a minority of quite special usecases. Most
| people are not even able to mange their own files, and
| you mean they will use a complex knowledge-managment-
| system? People are even challenged with a simple todo-
| list or note-app. No way will they use something more
| complex.
|
| > and no current tool serves them well; but a Nocode
| programmable knowledge database could do it.
|
| No, it's not. Nocode is not magic. You still need a
| certain level of competence, understanding and motivation
| to make something with it. And whatever interface you
| present to the users, be it GUI or text or whatever,
| there always will be a certain level of complexity that
| not any random user can master. And this is good, we
| already see with apple and google what harm we have from
| too much simplification to cater to the uninteressted.
|
| > No. Logseq / Obsidian have tools that allow end users
| to build their own coding through templates and queries.
| The kind of coding is different from scripting, though;
| it's more like creating semi-automated workflows that put
| all relevant information in front of the user, which then
| proceeds to transform it with direct manipulation. What
| tool in the past has allowed end-users to create that
| kind of workflows from end-user templates and arbitrary
| queries, other than non-hierarchical spreadsheets?
|
| Many? What? This is also a kind of scripting, and there
| is an endless amount of tools enabling people to do that
| style of scripting. Starting with shellscripts,
| batchscripts, all the strange scripting-engines. Heck,
| emacs and has this style as their main purpose
| established. The data science-sector has a very long
| history of building such tools on a more professional
| level and integrating data-views. Wikis and databases
| have a long history of scripting in the knowledge
| managment-space. The likes of obsidian are just the next
| iteration, there are not fundamentally new.
|
| > and spreadsheets are typically not automated through
| scripting and extensions - using those is normally
| considered a failure of the spreadsheet model
|
| No, they are? What? People have build all kind of crap
| with VBA for a very long time. There is a reason why this
| garbage is still around, getting ported to everywhere.
| And let's not start with the plugins or how extensions
| for MS Office are called these days.
|
| > using those is normally considered a failure of the
| spreadsheet model, because you're processing data outside
| the mental model of the spreadsheet as a programming
| environment
|
| Maybe that's our bubbles understanding. Seems the rest of
| the world doesn't give much here. Limiting yourself to a
| model is nonsens anyway. No model is ever perfect and
| satisfying every user.
|
| > That's a red-herring, not a main point. I mentioned
| multimedia merely because programmable tools tend to be
| text only, forcing users to use separate tools for rich
| content.
|
| It's not like obsidian is doing more with them, just
| because it can play them. Until it understand them can
| make them accessable for the rest of the system it's
| remains some fancy gadget.
| TuringTest wrote:
| _> Outliners are used in this area for decades._
|
| What outliners do you know which allow end-users to feed
| their data into formulas for processing it without using
| general-purpose programming languages? I'm genuinely
| interested, as I've been exploring these tools for years and
| I haven't found any that meets my requirements.
|
| _> So, SQL_
|
| How many end-users without any programming experience are
| able to create moderately complex SQL queries?
|
| With a data-centric outliner tool, they'll be able to
| decompose those into several simple graphical queries.
| slightwinder wrote:
| > What outliners do you know which allow end-users to feed
| their data into formulas for processing it without using
| general-purpose programming languages?
|
| Bit of a pointless constraint, the talk is about outliners,
| not no-code-datamangment. Which tool today does this even
| offer on a useful level?
|
| But you can look at leo editor (https://leoeditor.com),
| which is active for 20+ years, fully scriptable and
| extendable. Though, it's a hot piece of garbage for laymen.
| It's offers a bunch of features and plugins even for non-
| coders, but I'm not sure it would satisfy you for this
| area, if you can't code. Obsidian is probably better than,
| even if it has more limitations at the moment.
|
| But I'm not sure if there ever is a tool which will satisfy
| everyone with just a no-code-approach.
| TuringTest wrote:
| _> Bit of a pointless constraint, the talk is about
| outliners, not no-code-datamangment._
|
| The point I'm making is that those are both aspects of
| the same substrate, and modern knowledge tools are
| exploring the gap between those two - as well as
| expanding them in new directions.
|
| Yes, certainly Leo is one of the closest tools on which
| to build a system like that, although it still relies too
| much on Python programming (and its usability leaves a
| lot to be desired, even if the concepts are solid). I
| expect that the next popular tool in that space will grow
| either from online notebooks like Jupyter or from desktop
| linked note-takers like logseq.
|
| (BTW I can code, but I don't _want_ to; remembering the
| syntax details of the needed APIs does not come natural
| to me, so it 's usually not worth the effort). I still
| believe that it is possible to grow a programming tool
| similar to Excel but with structured hierarchical
| content, without requiring coding more complex than
| spreadsheet formulas and filters.
| slightwinder wrote:
| > and modern knowledge tools are exploring the gap
| between those two - as well as expanding them in new
| directions.
|
| Do they? As I see it, they make it worse. We have all
| kind of specialized languages. Each function coming with
| it's own limited DSL and API. Often with poor
| documentation. Is this actually better?
|
| Yes, each language has on it's own less complexity, but
| all in all they have more problems and limitations than a
| proper general purpose language with a good API. I don't
| think that this is really a good future. But it's true
| that it might lower the bar for new people.
|
| > BTW I can code, but I don't want to; remembering the
| syntax details of the needed APIs does not come natural
| to me
|
| So the problem is not the language, but the support? More
| autocompletion, documentation, automation, etc. That's a
| concern of mine too. Things are complex, but there is no
| support for the casual users. Actually I think if
| obsidian had been build ontop of VS Code and gained
| instant support from all the fancy developer-tools, it
| would gain significant benefits.
|
| Another solution could be integrating an interface like
| Node-RED offers it. This seems to be quite accessible for
| casual experts.
|
| > I still believe that it is possible to grow a
| programming tool similar to Excel
|
| And Excel has good access for new users?
| TuringTest wrote:
| _> Do they? As I see it, they make it worse._
|
| That may be because your development needs are well
| served already if you're a developer, so you won't see
| the value in such system. For people who rely on others
| to build every single piece of automation they need,
| being able to create their own small automatisms would be
| priceless.
|
| _> all in all they have more problems and limitations
| than a proper general purpose language with a good API_
|
| End users don't need a general purpose language with a
| good API - at least not as someting that they will handle
| directly. They need small, self-contained domain-specific
| languages that they can expand in areas where they fall
| short, but without the full power of a general
| programming system (which would overcomplicate the
| development environment and provide too many
| opportunities to shoot themselves in the foot).
|
| _> So the problem is not the language, but the support?_
|
| For me yes, partially, because I already know
| programming. I could work with a programming environment
| with complex syntax that autocompleted the required
| syntax to match my thoughts (something like GitHub
| copilot works in that direction). But that's still two
| points of complexity above the system I envision as
| possible.
|
| Something like the classic Hypercard could be a good
| starting point for the mental model of such tools. A
| simple data model, integrated storage, and interactive
| processes using a programming language with minimal
| syntax. Expand it with hierarchical storage, expanded
| integration with other systems and a modern declarative
| language, and you're set.
|
| Someone today pointed me to a system like this[1],
| Lepiter, which looks a lot like the solution end users
| could manage to create on their own on top of their
| knowledge systems: like a browser with separate cells,
| where each cell can run a different runtime, hyperlinked
| and with a simple storage mechanism.
|
| [1] https://lepiter.io/feenk/introducing-lepiter--
| knowledge-mana...
|
| _> More autocompletion, documentation, automation, etc._
|
| Those are good things but not enough. What end users are
| lacking the most is the possibility to create their own
| automated processes; most tools only allow them to choose
| among a library of predefined ones, without allowing them
| to grow the library.
|
| _> Another solution could be integrating an interface
| like Node-RED offers it. This seems to be quite
| accessible for casual experts._
|
| Unfortunately, graph-based flow visual tools have proven
| to be useful only for a small subset of automation tasks
| largely consisting on data transformations using pre-
| defined functions. The most promising results in End-User
| Development point out that reactive, rule-based runtime
| environments like the spreadsheet seem to be better
| abstractions for allowing users to define their own
| computations.
|
| _> And Excel has good access for new users?_
|
| Yes, spreadsheets have a low learning curve and high
| ceiling. And compared to imperative programming
| languages, they're like walking up the slope of your
| house vs. extreme rock climbing.
| kall wrote:
| OmniOutliner has had some kind of formulas for a very long
| time, however they work like an actual spreadsheet, where
| each outline row can have additional number and formula
| columns.
|
| I don't think it's all that useful, outside of some basic
| "making a shopping/packing/project list" type calculations.
| It's worth checking out if you're interested in the history
| though.
| TuringTest wrote:
| Thanks, I'll take a look at it.
| qwertyzxcvmnbv wrote:
| Check out https://lepiter.io/feenk/introducing-lepiter--
| knowledge-mana...
| TuringTest wrote:
| Where has this tool been my whole life? :P Thanks for the
| link, I didn't know that one.
|
| Yes, this https://youtu.be/Pot9GnHFOVU is the kind of
| programmable system I'm talking about, creating behavior on
| top of your stored content with methods specific to the
| particular collections of data already in your system, and
| ideally with minimal syntax for transferring data among the
| different tools.
| ryanjamurphy wrote:
| I agree with your issue of reification and I like the concept
| of a "wit."
|
| I think we're getting there, too. See, for instance,
| https://github.com/nothingislost/obsidian-query-control -- it
| renders the results of an embedded Obsidian query and you can
| interact with checkboxes. Likewise, a collaboration between
| dataview and the Task Collector plugin might let us manipulate
| different task statuses from a dataview query:
| https://github.com/ebullient/obsidian-task-collector/issues/...
|
| Exciting times!
| TuringTest wrote:
| _> I agree with your issue of reification and I like the
| concept of a "wit."_
|
| Thanks! I had it as a vague idea for several years, though it
| has been getting a lot more concrete as I've seen new tools
| adopting similar approaches - but without any of them taking
| the final step of converting a search result collection into
| a user-accessible object ready for direct manipulation.
|
| And now that I see tools adopting that style of data
| management, the concept allows me to see the commonalities
| among tools with quite different approaches - be it tabular
| Nocode storage like Airtable or outliners like Roam,
| Workflowy and the like. Whenever the tool returns a
| collection from a data gathering process, I see a wit that
| should be able to be extracted and further processed by the
| user.
|
| BTW, the main difference between a query result and a _wit_
| is that you can store its items in the same structure as the
| rest of your content in the outliner, and send them to a new
| process (like spreadsheet cells, but with hierarchical data).
| Usually, query results are presented through a different
| interface than the rest of content, but a 'wit' should be
| _homoiconic_ like Lisp.
|
| (Errata to my grandparent post: a 'bit' is a "minimum unit of
| data", of course, not of knowledge).
|
| _> Exciting times!_
|
| You bet! :-D
| blacksmithgu wrote:
| Whoa, very weird to see my plugin hit HN! Dataview was initially
| borne as "Notion tables for Obsidian", but quickly became a way
| to do flexible data annotation + querying after I realized how
| powerful metadata can be. Viewing your notes as "text + data that
| can easily be accessed" can really alter how you think about note
| taking and knowledge management.
|
| There is a fair bit left to do for the plugin (export to raw
| Markdown, section-level metadata), but I hope you find it useful!
| _bramses wrote:
| thank you for your work on Dataview! I use it as a read later
| list in combination with a Mac Shortcut I wrote +
| https://github.com/hauselin/obsidian-alfred
|
| It has completely replaced Pocket/Instapaper for me.
| chrisweekly wrote:
| Obsidian is amazing. Dataview is transformative, but it's not
| alone; the Excalidraw plugin is also a game-changer in its own
| right, enabling notes in drawings and drawings in notes... just
| incredibly powerful for more visually-oriented users.
| sofixa wrote:
| I've been using it for a few months and it's pretty cool. It
| allows me to do stuff like:
|
| * list all unfinished tasks grouped by some custom metadata (
| project, etc.)
|
| * have note templates which contains a dynamic query for the
| history of the same type of note for historical context
|
| * use it in conjunction with Obsidian charts to make custom
| charts ( e.g. number of meetings with tag/metadata/mention of X)
| pqs wrote:
| How would you manage tasks with this?
|
| Currently, I'm using Evernote and its new Tasks feature is very
| good, even though it could be better.
|
| I cannot move from Evernote to Obsidian if I don't solve the
| issue of managing tasks effectively from with Obsidian, as I
| currently do with Emacs.
| greggsy wrote:
| I add a tag, which filters it into a table. Pretty
| rudimentary but it meets my needs.
| davish wrote:
| I moved from Emacs org-mode to Obsidian with the help of the
| obsidian-tasks plugin, which lets you write queries for tasks
| similarly to Dataview. I used a few of those queries to build
| up an Agenda view that matched my custom agenda buffer in
| org-mode. I put together a little write up here:
| https://davi.sh/blog/2022/01/obsidian-one/
| Macha wrote:
| I use the Tasks plugin for this. Tasks are created by
| creating markdown lists. You can put tags in with hashtags
| and due dates in with (and this bit I dislike, even though
| there's a command for it) the calendar emoji followed by a
| date.
|
| E.g. (note HN strips out emoji) - [] Write
| documentation for Foobar #docs #personxyz (cal emoji)
| 2022-05-22
|
| You then write a query in a markdown code block with a custom
| query language.
|
| E.g. ```tasks not done
| description does not include #deferred due before
| today path does not include Templates ```
|
| For recurring tasks, I combine with the periodic notes
| plugin, I put the tasks in my daily/weekly (I don't use
| monthly but it is an option) with labels like #wednesday if
| it needs to done on a specific time. Then I include
| appropriate queries in my daily notes to bring up the
| relevant tasks, and I also have a Todo note with various
| longer term/backloggy tasks.
|
| There are other plugins you can use, like Dataview which is
| the subject of this post (but means each Todo needs to be its
| own note), or the Todoist plugin, but Tasks is the one which
| works for me.
| RyEgswuCsn wrote:
| Before Obsidian there was ConntextedText [1]. Few people knew it
| even back then, but it is quite powerful, packed with common wiki
| features like linking, attributes, querying (also display results
| as a table), transclusion, page graph visualisation, python
| integration, etc.
|
| [1]: https://www.connectedtext.com/
| Nuzzerino wrote:
| Just a shameless suggestion: if anyone is concerned about the
| security implications of using the Obsidian Sync paid addon, I've
| been having good results with Syncthing instead. It supports
| peer-to-peer sync and is FOSS.
| amon22 wrote:
| what's wrong with git?
| 369548684892826 wrote:
| I've done this too, but on iOS Syncthing doesn't work so well.
| Resorted to running a Windows VM as a bridge between iCloud and
| Syncthing.
| singhrac wrote:
| I have just used iCloud for everything, and it works great,
| even on Windows (note that you cannot use the Microsoft Store
| app, which is garbage).
|
| I guess I fit solidly inside the 5GB free tier and probably
| will for quite a while.
|
| I would use Dropbox but the app feels quite heavy and it
| wasn't very easy to set up on iPhone.
| 369548684892826 wrote:
| Sadly no iCloud client for Linux though, so that's where
| Syncthing comes in!
| dariusj18 wrote:
| So many great features, but all these systems are missing the one
| true feature I really need, Link types. I see that Dataview gives
| some ability to have them, but it requires a very specific format
| that is pretty distinct from how the notes would be taken.
| alfonsodev wrote:
| can you elaborate? How the ideal implementation would look
| like.
| dariusj18 wrote:
| My use case would be more simple than others, but I am not
| super well versed in the actual current syntax of Obsidian. I
| like the idea of this syntax https://forum.obsidian.md/t/add-
| support-for-link-types/6994/... as it would be most flexible.
| But a simple [[Page Name]]{type:child} could be good, all I
| really need is a means of filtering linked documents by their
| link type (in a graph view/visualization) and ideally a
| method to query and display those. Dataview gives the second,
| though the first seems to require the link type to be a first
| class citizen.
|
| I understand that it is a complex thing though, and not
| something easy to just throw in
| brimwats wrote:
| you can use the supercharged links plugin and careful
| control of it's class types to achieve that, it's how I did
| it. I'm largely using schema.org types
| qwertyzxcvmnbv wrote:
| Logseq allows query via datomic:
| https://discuss.logseq.com/t/advanced-queries-and-the-databa...
| hipitihop wrote:
| Minor correction, I don't think Datomic is used. I believe it
| uses a variation of DataScript and in turn DataLog for queries.
| Whereas Datomic is a DB which is also based on and uses DataLog
| queries.
|
| Either way, super powerful.
|
| Refs: https://en.m.wikipedia.org/wiki/Datalog
| https://github.com/tonsky/datascript https://www.datomic.com/
| kall wrote:
| It's great, but I wish there was some more interesting stuff
| you could do with the results of that query, other than show an
| outline or table.
|
| The inline programmability of DataView and the Grid views etc
| are pretty killer. Unfortunately Obsidian is not for me, so I
| hope Logseq catches up.
___________________________________________________________________
(page generated 2022-05-17 23:01 UTC)