Vision Pro(posals)

A couple of weeks ago, at the end of July, I booked a slot to try out the Apple Vision Pro.

It has been available for months in the USA, and might already be in the ‘trough of disillusionment’ there already – but I wanted to give it a try nonetheless.

I sat on a custom wood and leather bench in the Apple Store Covent Garden that probably cost more than a small family car, as a custom machine scanned my glasses to select the custom lenses that would be fitted to the headset.

I chatted to the personable, partially-scripted Apple employee who would be my guide for the demo.

Eventually the device showed up on a custom tray perfectly 10mm smaller than the custom sliding shelf mounted in the custom wood and leather bench.

The beautifully presented Apple Vision Pro at the Apple Store Covent Garden

And… I got the demo?

It was impressive technically, but the experience – which seemed to be framed as one of ‘experiencing content’ left me nonplused.

I’m probably an atypical punter, but the bits I enjoyed the most were the playful calibration processes, where I had to look at coloured dots and pinch my fingers, accompanied by satisfying playful little touches of motion graphics and haptics.

That is, the stuff where the spatial embodiment was the experience was the most fun, for me…

Apple certainly have gone to great pains to try a and distinguish the Vision Pro from AR and VR – making sure it’s referenced throughout as ‘spatial computing’ – but there’s very little experience of space, in a kinaesthetic sense.

It’s definitely conceived of as ‘spatial-so-long-as-you-stay-put-on-the-sofa computing’ rather than something kinetic, embodied.

The technical achievements of the fine grain recognition of gesture are incredible – but this too serves to reduce the embodied experience.

At the end of the demo, the Apple employee seemed to be noticeably crestfallen that I hadn’t gasped or flinched at the usual moments through the immersive videos of sport, pop music performance and wildlife.

He asked me what I would imagine using the Vision Pro for – and I said int he nicest possible way I probably couldn’t imagine using it – but I could imagine interesting uses teamed with something like Shapr3d and the Apple Pencil on my iPad.

He looked a little sheepish and said that wasn’t probably going to happen but sooner with SW updates, I could use the Vision Pro as an extended display. OK- that’s … great?

But I came away imagining more.

I happened to run into an old friend and colleague from BERG in the street near the Apple Store and we started to chat about the experience I’d just had.

I unloaded a little bit on them, and started to talk about the disappointing lack of embodied experiences.

We talked about the constraint of staying put on the sofa – rather than wandering around with the attendant dangers.

But we’ve been thinking about ‘stationary’ embodiment since Dourish, Sony Eyetoy and the Wii, over 20 years ago.

It doesn’t seem like that much of a leap to apply some of those thoughts to this new level of resolution and responsiveness that the Vision Pro presents.

With all that as a preamble – here are some crappy sketches and first (half-formed) thoughts I wanted to put down here.

Imagining the combination of a Vision Pro, iPad and Apple Pencil

Vision Pro STL Printer Sim

The first thing that came to mind in talking to my old colleague in the street was to take some of the beautiful realistically-embedded-in-space-with-gorgeous-shadows windows that just act like standard 2D pixel containers in the Vision Pro interface and turn them into ‘shelves’ or platens that you could have 3D virtual objects atop.

One idea was to extend my wish for some kind of Shapr3D experience into being able to “previsualise” the things I’m making in the real world. The app already does a great job of this with it’s AR features, but how about having a bit of fun with it, and rendering the object on the Vision Pro via a super fast, impossibly capable (simulated) 3d printer – that of course because it’s simulated can print in any material…

Sketch of Vision Pro 3d sim-printer
(Roughly) Animated sketch of Vision Pro 3d sim-printer

Once my designed objected had been “printed” in the material of my choosing, super-fast (and without any of the annoying things that can happen when you actually try to 3d print something…) I could of course change my scale in relation to it to examine details, place it in beautiful inaccessible immersive surroundings, apply impossible physics to it etc etc. Fun!


Vision Pro Pottery

Extending the idea of the virtual platen – could I use my iPad in combination with with Vision pro as a cross-over real/virtual creative surface in my field of view. Rather than have a robot 3d printer do the work for me, could I use my hands and sculpt something on it?

Could I move the iPad up and down or side to side to extrude or lathe sculpted shapes in space in front of me?

Could it spin and become a potter’s wheel with the detailed resolution hand detection of the Vision Pro picking up the slightest changes to give fine control to what I’m shaping.

Is Patrick Swayze over my shoulder?

Vision Pro + iPad sculpting in space.

Maybe it’s something much more throw-away and playful – like using the iPad as an extremely expensive version of a deformed wire coat-hanger to create streams of beautiful, iridescent bubbles as you drag it through the air – but perhaps capturing rare butterflies or fairies in them as you while away the hours atop Machu Picchu or somewhere similar where it would be frowned up to spill washing-up liquid so frivolously…

Making impossible bubbles with an iPad in Vision Pro world

Of course this interaction owes more than a little debt to a previous iPad project I saw get made first hand, namely BERG’s iPad Light-painting

Although my only real involvement in that project was as a photographic model…

Your correspondent behind an iPad-lightpainted cityscape (Image by Timo, of course)

Pencils, Pads, Platforms, Pots, Platens, Plinths

Perhaps there is an interesting little more general, sober, useful pattern in these sketches – of horizontal virtual/real crossover ‘plates’ for making, examining and swapping between embodied creation with pencil/iPad and spatial examination and play with the Vision Pro.

I could imagine pinching something from the vertical display windows ion Vision Pro to place onto my ipad (or even my watch?) in order to keep it, edit it, change something about it – before casting it back into the simulated spatial reality of the Vision Pro.

Perhaps it allows for a relationship between two realms that feels more embodied and ‘real’ without having to leave the sofa.

Perhaps it also allows for less ‘real’ but more fun stuff to happen in the world of the Vision Pro (which in the demo seems doggedly to anchor on ‘real’ experience verissimilitude – sport, travel, family, pop concerts)

Perhaps my Apple watch can be more of a Ben 10 supercontroller – changing into a dynamic UI to the environment I’m entering, much like it changes automatically when I go swimming with it and dive under…

Anyway – was very much worth doing the demo, I’d recommend it, if only for some quick stretching (and sketching) of the mindlegs.

My sketches in a cafe a few days after the demo

All in all I wish the Vision Pro was just *weirder*.

Back when it came out in the US in February I did some more sketches in reaction to that thought… I can’t wait to see something like a bonkers Gondry video created just for the Vision Pro…

Until then…

He’s not there – notes from “Jony Ive: The Genius Behind Apple’s Greatest Product”s by Leander Kahney

Just finished reading “Jony Ive: The Genius Behind Apple’s Greatest Products” by Leander Kahney, which is mainly fascinating because of the abscence of it’s subject. Ive has said so little in public (aside from in corporate pr films) that the book paints a detailed picture of everything around him – the design culture he was raised in, both in education and industry, the design group and wider engineering/manufacturing culture at Apple – right down to gems like this:

“Enter the need for so-called friction stir welding (FSW), a solid-state welding process invented in 1991. It’s actually less of a weld than a recrystallization, as the atoms of the two pieces are joined in a super strong bond when a high-speed bobbin is moved along the edges to be bonded, creating friction and softening the material almost to its melting point.”

Needless to say I really enjoyed it – but Ive is just the hook the book hangs off. It wouldn’t exist or sell as a book without him, although it’s full of fascinating detail about how Apple products are designed and made.

The little you do learn about Ive as a design leader is good. A little hagiographic, but hey. I’d recommend it more for the insights into the design, making and manufacturing approach at Apple than the man at the centre of it however.

‘In America, on the other hand,’ Milton explained, ‘designers are very much serving what industry wants. In Britain, there is more of the culture of the garden shed, the home lab, the ad hoc and experimental quality. And Jony Ive interacts in such a way … [he] takes big chances, instead of an evolutionary approach to design – and if they had focus-grouped Ive’s designs, they wouldn’t have been a success.’

If the education system in America tended to teach students how to be an employee, British design students were more likely to pursue a passion and to build a team around them.

‘As an industrial designer, you have to take that great idea and get it out into the world, and get it out intact. You’re not really practising your craft if you are just developing a beautiful form and leaving it at that.’

I can’t have people working in cubicle hell. They won’t do it. I have to have an open studio with high ceilings and cool shit going on. That’s just really important. It’s important for the quality of the work. It’s important for getting people to do it. – ROBERT BRUNNER

He wanted a ‘small, really tight’ studio. ‘We would run it like a small consulting studio, but inside the company,’ he said. ‘Small, effective, nimble, highly talented, great culture.’4 Setting up a consultancy inside Apple seemed in line with the company’s spirit: unconventional, idea driven, entrepreneurial. ‘It was because, really, I didn’t know any other way,’ Brunner explained. ‘It wasn’t a flash of brilliance: that was the only thing I knew how to do.’

In 1997, English contributed photos to Kunkel’s book about the design group, AppleDesign, but he also worked with a lot of other design studios in the Valley. To his eye, Apple seemed different. It wasn’t just the tools and their focus; the place was rapidly populated with designer toys, too, including spendy bikes, skateboards, diving equipment, a movie projector and hundreds of films. ‘It fostered this really creative, take-a-risk atmosphere, which I didn’t see at other firms,’ said English.

Brunner also made about half a dozen of the designers ‘product line leaders’ (PLLs) for Apple’s major product groups: CPUs, printers, monitors and so on. The PLLs acted as liaisons between the design group and the company, much in the way an outside design consultancy would operate. ‘The product groups felt there was a contact within the design group,’ Brunner said.

Brunner wanted to shift the power from engineering to design. He started thinking strategically. His off-line ‘parallel design investigations’ were a key part of his strategy. ‘We began to do more longer-term thinking, longer-term studies around things like design language, how future technologies are implemented, what does mobility mean?’ The idea was to get ahead of the engineering groups and start to make Apple more of a design-driven company, rather than a marketing or engineering one. ‘We wanted to get ahead of them, so we’d have more ammunition to bring to the process.’

In hindsight, Brunner’s choices – the studio’s separation from the engineering groups, its loose structure, the collaborative workflow and consultancy mind-set – turned out to be fortuitous. One of the reasons Apple’s design team has remained so effective is that it retains Brunner’s original structure. It’s a small, tight, cohesive group of extremely talented designers who all work on design challenges together. Just like the designers had done at Lunar, Tangerine and other small agencies. The model worked.

‘Bob did more than lay the foundations for Jony’s design team at Apple – he built the castle,’ said Clive Grinyer. ‘After Bob, it was the first time that an in-house design team was cool.’

Jony was looking for the Mac NC’s ‘design story’. As his dad, Mike, had instilled in him, developing the design story was an essential first step in conceiving something entirely new. ‘As industrial designers we no longer design objects,’ Jony said. ‘We design the user’s perceptions of what those objects are, as well as the meaning that accrues from their physical existence, their function and the sense of possibility they offer.’

‘When you see the most dramatic shift is when you transition from an abstract idea to a slightly more material conversation,’ Jony said. ‘But when you made a 3-D model, however crude, you bring form to a nebulous idea, and everything changes – the entire process shifts. It galvanizes and brings focus from a broad group of people.

Though Jobs rejected all five names, Segall refused to give up on iMac. He went back again with three or four new names, but again pitched iMac. This time, Jobs replied: ‘I don’t hate it this week, but I still don’t like it.’43 Segall heard nothing more about the name from Jobs personally, but friends told him that Jobs had the name silk-screened onto prototypes of the new computer, testing it out to see if he liked the look. ‘He rejected it twice but then it just appeared on the machine,’ Segall recalled. He came to believe that Jobs changed his mind just because the lower-case ‘i’ looked good on the product itself.

Boxes may seem trivial, but Jony’s team felt that unpacking a product greatly influenced the all-important first impressions. ‘Steve and I spend a lot of time on the packaging,’ Jony said then. ‘I love the process of unpacking something. You design a ritual of unpacking to make the product feel special. Packaging can be theater, it can create a story.’

‘Innovation,’ he wrote, ‘is rarely about a big idea; more usually it’s about a series of small ideas brought together in a new and better way. Jony’s fanatical drive for excellence is, I think, most evident in the stuff beyond the obvious; the stuff you perhaps don’t notice that much, but which makes a difference to how you interact with the product, how you feel about it.’

‘Apple designers spend ten percent of their time doing traditional industrial design: coming up with ideas, drawing, making models, brainstorming. They spend ninety percent of their time working with manufacturing, figuring out how to implement their ideas.’

On iPhone launch day, Jobs turned to Kay and casually asked, ‘What do you think, Alan? Is it good enough to criticize?’ The question was a reference to a comment made by Kay almost twenty-five years earlier, when he had deemed the original Macintosh ‘the first computer worth criticizing’. Kay considered Jobs’s question for a moment and then held up his moleskin notebook. ‘ “Make the screen at least five inches by eight inches and you will rule the world,” he said.’

‘I have literally seen buildings where as far as the eye can see, where you can see machines carving, mostly aluminium, dedicated exclusively for Apple at Foxconn,’ said Guatam Baksi, a product design engineer at Apple from 2005 to 2010. ‘As far as the eye can see.’

Unibody represents a giant financial gamble by Apple. When it started investing seriously around 2007, Apple contracted with a Japanese manufacturer to buy all the milling machines it could produce for the next three years. By one estimate, that was 20,000 CNC milling machines a year, some costing upward of $250,000 and others $1 million or more. The spending didn’t stop there, as Apple bought up even more, acquiring every CNC milling machine the company could find. ‘They bought up the entire supply,’ said one source. ‘No one else could get a look in.’

Apple spent $9.5 billion on capital expenditures, the majority of which was earmarked for product tooling and manufacturing processes. By comparison, the company spent $865 million on retail stores. Thus, Apple spent nearly eleven times as much on its factories as on its stores, most of which are in prime (that is, expensive) real estate locations.

Enter the need for so-called friction stir welding (FSW), a solid-state welding process invented in 1991. It’s actually less of a weld than a recrystallization, as the atoms of the two pieces are joined in a super strong bond when a high-speed bobbin is moved along the edges to be bonded, creating friction and softening the material almost to its melting point. The plasticized materials are then pushed together under enormous force, and the spinning bobbin stirs them together. The result is a seamless and very strong bond. In the past, FSW required machines costing up to three million dollars apiece, so its use was confined to fabricating rocket and aircraft parts. More recent advances allowed CNC milling machines to be retrofitted to perform FSW at a much lower cost. In addition to its other advantages, FSW produces no toxic fumes and finished pieces that require no extra filler metal for further machining, making the process more environmentally friendly than traditional welding.

‘That’s probably the single greatest effect, that we nowadays expect many things to have better designs. Because of Apple, we got to compare crappy portable computers versus really nice ones, crappy phones versus really nice ones. We saw a before-and-after effect. Not over a generation, but within a few years. Suddenly 600 million people had a phone that put to shame the phone they used to have. That is a design education at work within our culture.’

Lost futures: Unconscious gestures?

Lamenting lost futures is not that productive, but it doesn’t stop me enjoying it. Whether it’s the pleasure of reading Ellis’s “Ministry of Space” and thinking “what if?” or looking through popculture futures past as in this Guardian article – it’s generally a sentimental, but thought-provoking activity.

Recently, though, I’ve been thinking about a temporarily lost future that’s closer to home in the realm of mobile UI design. That’s the future that’s been perhaps temporarily lost in the wake of the iPhone’s arrival.

A couple of caveats.

Up until June this year. I worked at Nokia in team that created prototype UIs for the Nseries devices, so this could be interpreted as sour-grapes, I suppose.. but I own an iPodTouch, that uses the same UI/OS more-or-less, and love it.

I spoke at SkillSwap Bristol in September (thanks to Laura for the invite) and up until the day I was travelling to Bristol, I didn’t know what I was going to say, but I’d been banging on at people in the pub (esp. Mr. Coates) about the iPhone’s possible impact on interface culture, so I thought I’d put together some of those half-formed thoughts for the evening’s debate.

The slides are on Slideshare
(no notes, yet) but the basic riff was that the iPhone is a beautiful, seductive but jealous mistress that craves your attention, and enslaves you to its jaw-dropping gorgeousness at the expense of the world around you.

skillswap250907

This, of course, is not entirely true – but it makes for a good starting point for an argument! Of course, nearly all our mobile electronic gewgaws serve in some small way or other to take us away from the here and now.

But the flowing experience just beyond Johnny Ive’s proscenium chrome does have a hold more powerful than perhaps we’ve seen before. Not only over users, but over those deciding product roadmaps. We’re going to see a lot of attempts to vault the bar that Apple have undoubtedly raised.

Which, personally, I think is kind-of-a-shame.

First – a (slightly-bitter) side-note on the Touch UI peanut gallery.

In recent months we’ve seen Nokia and Sony Ericsson show demos of their touch UIs. To which the response on many tech blogs has been “It’s a copy of the iPhone”. In fact, even a Nokia executive responded that they had ‘copied with pride’.

That last remark made me spit with anger – and I almost posted something very intemperate as a result. The work that all the teams within Nokia had put into developing touch UI got discounted, just like that, with a half-thought-through response in a press conference. I wish that huge software engineering outfits like S60 could move fast enough to ‘copy with pride’.

Sheesh.

Fact-of-the-matter is if you have roughly the same component pipeline, and you’re designing an interface used on-the-go by (human) fingers, you’re going to end up with a lot of the same UI principles.

But Apple executed first, and beautifully, and they win. They own it, culturally.

Thus ends the (slightly-bitter) side-note – back to the lost future.

Back in 2005, Chris and myself gave a talk at O’Reilly Etech based on the work we were doing on RFID and tangible, embodied interactions, with Janne Jalkanen and heavily influenced by the thinking of Paul Dourish in his book “Where the action is”, where he advances his argument for ’embodied interaction’:

“By embodiment, I don’t mean simply physical reality, but rather, the way that physical and social phenomena unfold in real time and real space as a part of the world in which we are situated, right alongside and around us.”

I was strongly convinced that this was a direction that could take us down a new path from recreating desktop computer UIs on smaller and smaller surfaces, and create an alternative future for mobile interaction design that would be more about ‘being in the world’ than being in the screen.

That seems very far away from here – and although development in sensors and other enablers continues, and efforts such as the interactive gestures wiki are inspiring – it’s likely that we’re locked into pursuing very conscious, very gorgeous, deliberate touch interfaces – touch-as-manipulate-objects-on-screen rather than touch-as-manipulate-objects-in-the-world for now.

But, to close, back to Nokia’s S60 touch plans.

Tom spotted it first. In their (fairly-cheesy) video demo, there’s a flash of something wonderful.

Away from the standard finger and stylus touch stuff there’s a moment where a girl is talking to a guy – and doesn’t break eye contact, doesn’t lose the thread of conversation; just flips her phone over to silence and reject a call. Without a thought.

Being in the world: s60 edition from blackbeltjones on Vimeo.

As Dourish would have it:

“interacting in the world, participating in it and acting through it, in the absorbed and unreflective manner of normal experience.”

I hope there’s a future in that.