[HN Gopher] Spreadsheet "breaks" Apple Vision Pro eye-tracking
___________________________________________________________________
Spreadsheet "breaks" Apple Vision Pro eye-tracking
Author : Topfi
Score : 85 points
Date : 2024-02-09 19:00 UTC (4 hours ago)
(HTM) web link (kguttag.com)
(TXT) w3m dump (kguttag.com)
| orenlindsey wrote:
| To be fair, this is a 1.0 version of brand new software and
| hardware. By the time we get to visionOS 3.0, and maybe even
| before that, this will be gone.
| sneak wrote:
| I don't think this will ever improve significantly (maybe a
| little bit with software tweaks) on AVP1 hardware.
|
| The current algorithm is not terrible; it may be possible to
| improve the artifacting a bit but ultimately the rendering
| setup exists to cope with the resolution of the displays. The
| state of the art in cost-effective HMDs will need to advance a
| bit for this to really get "fixed", as TFA notes.
|
| It is telling that it took a very contrived test (a white grid
| of fine lines on black) for this to be obvious. Most things
| displayed by most users won't run into this problem.
|
| I agree that this thing is not yet monitor-replacement-level
| yet. I mostly just want it for watching movies in bed or on
| planes.
| MBCook wrote:
| All the "it's not good enough" posts amaze me. As you said
| we're still on 1.0. It hasn't even been a week. I bet Apple can
| improve it without further hardware, let alone future versions.
|
| However I was wondering just how accurate this could get
| yesterday. Because of saccades the eye never really stops and
| just looks at one thing right? Which would mean we can never
| tell what someone is looking at, only the area of the eye is
| heavily focused around. Sort of a probability field.
|
| Just how small can we make that field and have it still feel
| accurate? Or does this introduce something of the "thumb"
| problem on touchscreens where you have to just do your best to
| interpret what the user probably wanted?
| crooked-v wrote:
| So far with mine, it feels like fingertip size (subjectively
| speaking) is about the smallest close-up 'eye target' I can
| use and still have 100% accuracy.
|
| I would bet that whatever standard sizes they're using for
| buttons/targets is the basis of window resizing with
| push/pull movement, so that the accuracy rate remains the
| same for farther-away virtual screens.
| threeseed wrote:
| This is simply how foveated rendering works. Nothing is broken
| here.
|
| It's an optimisation technique that is needed now because we
| don't have the resources to handle the full resolution. But this
| will change over the coming years.
|
| For now maybe just increase the font size on the spreadsheet.
| crooked-v wrote:
| The "broken" part is, outside of the visible difference in the
| foveation, the "fizzing" effect from whatever's going on with
| the antialiasing being weird with the zoomed-out grid.
| Havoc wrote:
| Not sure it matters at this stage?
|
| One day excel gang will no doubt be using this type of VR setup.
| It makes for a very poor early adopter crowd though - Excel users
| need keyboards not current finger gesture setup. They're very
| reliant on muscle memory, shortcuts and predictable low latency
| interactions with literally a single piece of software.
|
| Unless you want to pay your accountant twice as much and get 1/10
| of the output...they're probably going to be late adopters by
| virtue of bad fit
| chaostheory wrote:
| Keyboards and a trackpads work with the Vision Pro
| FirmwareBurner wrote:
| But if you're gonna be tied to a keyboard and mouse on a desk
| then why do you need a AVP and not a highdpi monitor and
| spend the rest on a vacation to Hawaii/Ibiza?
|
| The whole point of Vision Pro like devices is you wear your
| computer on your face always with you and the only
| peripherals you need are your eyes and hands also always on
| you.
| martin_ wrote:
| in part, it may serve as a substitute to highdpi monitor
| while you're in Hawaii
| wkat4242 wrote:
| It's not highdpi. It's only 4K per eye and that's over
| the entire field of vision.
|
| Highdpi would be 4K on a 24" screen a foot or two away.
| The vision pro spreads those pixels out over a much
| bigger angle.
|
| It's got enough pixels to equal a few 1080p monitors
| though.
| pjerem wrote:
| Dozens of Excel spreadsheets all across the room. Jira
| boards to the left, performance dashboards to the right,
| Gantt diagrams all over the ceiling.
|
| That's what some people are dreaming at night.
| FirmwareBurner wrote:
| Will that _really_ make you more productive? Or do you
| just imagine it would?
| brokencode wrote:
| Have you ever tried using more than one monitor? It helps
| productivity.
|
| And have you ever upgraded your monitor to a larger one
| so you could fit more on the screen at once? That helps
| productivity too.
|
| Imagine being able to make as many monitors and in as
| many sizes as you want. And imagine being able to take
| them with you as a replacement for your laptop monitor.
| FirmwareBurner wrote:
| _> Have you ever tried using more than one monitor? It
| helps productivity. _
|
| I have and I haven't been delivering code/tasks 50%
| faster when my monitor count increased by 50% from 2 to
| 3. Then went back to one monitor with virtual workspaces
| as that made me more productive than having to constantly
| move my neck eyes across a sprawl of windows. Plus having
| to manage windows around and tile them "just right"
| across several monitors always interrupted my flow, and
| worse, was costing me even more time when those windows
| were moving around and they weren't in the place where I
| remember I left them last time.
|
| _> Imagine [...]_
|
| But I'm not asking what you imagine it might do. I'm
| asking if it actually measurably improves your
| productivity, not if you imagine it could, because those
| are two very different things.
|
| I also imagined that having as many monitors as an
| operator from The Matrix would make me more productive
| but it hasn't. My coding and task delivery speed has
| stayed the same or even gone down inversely proportional
| to the number of monitors.
| crooked-v wrote:
| For somebody who's legitimately doing complicated
| business management stuff and already has two monitors
| covered in documents all the time, of course it would. I
| don't do that myself, but I've worked at multiple
| companies with people whose job includes trying to
| synthesize all that stuff together into useful business
| decisions.
| corry wrote:
| Nah, I disagree. If you ever work on a very large
| spreadsheet that spans several monitors' worth of space,
| both up and down, you quickly realize that multiple
| monitors are not it (the seams break everything).
|
| Can't do it with today's Vision Pro (I believe) but massive
| spreadsheet windows is coming and IMO the Excel gang is
| going to love it.
| threeseed wrote:
| Some of us have to work on our holidays or travel
| extensively.
|
| So being able to bring us the same setup anywhere is a huge
| win.
|
| I think the future of Vision Pro will centre around two use
| cases: (a) content consumption like an iPad and (b) virtual
| monitors with a keyboard/mouse like a Mac.
| FirmwareBurner wrote:
| _> So being able to bring us the same setup anywhere is a
| huge win._
|
| So like ... a laptop? Plus maybe an extra portable USB-C
| display or your iPad as an external monitor?
|
| For your statement to be correct, it means that someone
| would have to be using the APV as their daily driver
| default setup for work in order to take it with them
| wherever they go, and not just use it as a secondary
| travel setup in parrales to their mains etup.
|
| Curious how many people will actually daily drive the AVP
| for 8 hour workday as their default work setup long term,
| versus the ones who will give up on it as soon as the
| hype/honeymoon period is over and then only use it
| sporadically.
| cmcaleer wrote:
| If you're going for months at a time and you're used to a
| couple of large monitors/ultrawide, a USB-C display is a
| grim downgrade and sidecar is really unreliable.
| Currently the Vision Pro doesn't beat my current solution
| cost-wise (just buy some monitors there and flip them for
| cheap before I leave) until like, 15 trips and the
| quality of it as a solution for doing what I want seems
| roughly similar so I'm avoiding it for now. If it was
| $1200 or so it would be a no-brainer for me.
| FirmwareBurner wrote:
| _> If you're going for months at a time _
|
| But could you use the AVP for months at a time for work?
| That's the main question everyone's avoiding to answer.
|
| So far, everyone just plays with it for a couple of hours
| to post on social media "wow that's so cool, it's the
| future", but so far I haven't seen anyone saying they
| actually use it for work 8h/day 5 days/week long therm.
| mepian wrote:
| A keyboard and a mouse are much more portable than any
| monitors, and you don't necessarily need a proper desk to
| use them for most applications.
| elicash wrote:
| There are some differences, but it's useful to think of the
| iPad introduction in January 2010. Apple didn't create a
| keyboard for it until November 2015 with the "Smart
| Keyboard" and then nearly five years later, the "Magic
| Keyboard" that brought cursor support.
|
| Apple _marketing_ is focusing on using hands, sure, but
| that they were so quick to make it compatible with
| keyboards /trackpads is revealing.
| empath-nirvana wrote:
| There's probably just some UI elements people are going to have
| to avoid -- small, closely spaced text seems like it's likely to
| be a problem..
| crooked-v wrote:
| The bare minimum usable font size feels to be about 20-24pt for
| me (judging off a virtual screen placed about the same distance
| I'd hold a laptop), so I think the only time people are even
| going to run into it is if they're doing something like zooming
| in/out an entire game UI at once.
| meindnoch wrote:
| The problem is simply that their downsampling is not gamma-
| corrected, so overall brightness is not preserved.
|
| http://www.ericbrasseur.org/gamma.html
| 01HNNWZ0MV43FF wrote:
| > If you want to give it a try, scale it down 50% using your
| best software.
|
| Happy to report that in the last few years since 2007, GIMP did
| fix their default colorspace, I can scale the Dalai Lama down
| even with the Linear filter and it doesn't turn grey.
| spookie wrote:
| Also the reason why linear gradients aren't done properly in
| CSS, and W3C is aware of this.
|
| https://erikmcclure.com/blog/everyone-does-srgb-wrong-becaus...
| hedgehog wrote:
| There's some pretty glaring aliasing and moire issues. Gamma
| correction might matter once they try to come up with better
| shaders but that's not the main issue. Rendering fine detail
| like a spreadsheet without major artifacts is an interesting
| problem, burden is mostly on Apple to figure out a solution for
| app devs. Higher resolution screens won't really help that
| much.
| miiiiiike wrote:
| And eating popcorn breaks the pinch detection.
| operator-name wrote:
| Honestly, this is a really cool demonstration of how their
| foveated rendering works.
| MPSimmons wrote:
| This isn't breaking, it's just testing the bounds. Still
| interesting, but not groundbreaking.
| mayoff wrote:
| Watch the second half of WWDC 2023 session "Explore rendering for
| spatial computing" to see what Apple wants to tell developers
| about foveated rendering. Ivan never actual uses the term
| "foveated rendering" though.
|
| https://developer.apple.com/wwdc23/10095
|
| There is also an article, "Rendering at Different Rasterization
| Rates":
|
| https://developer.apple.com/documentation/metal/render_passe...
| losvedir wrote:
| A follow-up comment by the author indicates much of this is a bug
| having to do with Excel specifically, since a lossless PNG
| screenshot doesn't exhibit this behavior.
| pixelesque wrote:
| The use-case of viewing images is possibly (likely?) special-
| cased to be mip-mapped for correctly filtering / downsampling
| them, whereas generic applications which draw arbitrary content
| to the frame buffer likely don't have this?
| solardev wrote:
| Is this really some unsolvable problem? Seems like a cool
| opportunity for UX innovation. That shimmer looks kinda like a
| moire effect to me, and probably isn't an issue at higher zoom
| levels? I wonder if there an opportunity here to explore other
| ways of navigating a spreadsheet, like maybe panning and zooming
| the UI like a map, almost. The longer you stare at an area, the
| more it zooms in so you can easily see just a few cells at a
| time. You can still have a minimap nearby to see the big
| overview, but your main focus would be on just a few cells.
| exitzer0 wrote:
| This does not strike me as a productivity device. I doubt many of
| the target demographic will actually notice this issue.
| notdisliked wrote:
| I noticed the same behavior in my Vision Pro the other day. The
| conclusion I came to was actually that I was not seeing the
| boundaries of foveated rendering, but a UI element. When staring
| at touch targets, it projects an overlay over the target with a
| slightly bright large circle around your eyes, fading further
| away from your eyes. As you move your eyes around, the circle
| follows them. This is particularly noticeable where a very large
| touch target (like a large text box) is present, you can easily
| see the entirety of the circle overlay, whereas with smaller
| targets it's cut off on the edges of the target. It also happens
| to align with the foveated rendering downsampling circle because
| of course they both follow your eyes. I'm guessing this is what
| is happening here. To test this myself, I looked around on a
| large, lighter colored page in safari with no large touch
| targets, and could not see the same circular glow.
___________________________________________________________________
(page generated 2024-02-09 23:00 UTC)