This past week we (Lunar Energy) sponsored a social event for an energy industry conference in Amsterdam.
There were the usual opportunities offered to ‘dress’ the space, put up posters, screens etc etc.
We even got to name a cocktail (“lunar lift-off” i think they called it! I guess moonshots would have been a different kind of party…) – but what we landed on was… coasters…
Paulina Plizga, who joined us earlier this year came up with some lovely playful recycled cardboard coasters – featuring interconnected designs of pieces of a near-future electrical grid (enabled by our Gridshare software, natch) and stats from our experience in running digital, responsive grids so far.
The inspiration for these partly reference Ken Garland’s seminal “Connect” game for Galt toys – could we make a little ‘infinite game’ with coasters as play pieces for the attendees.
If you’ve ever been to something like this – and you’re anything like me – then you might want something to fidget with, help start a conversation… or just be distracted by for a moment! We thought these could also serve a social role in helping the event along, not just keep your drink from marking the furniture!
I was delighted when our colleagues who were attending said Paulina’s designs were a hit – and that they had actually used them to give impromptu presentations on Gridshare to attendees!
So a little bit more playful grid awareness over drinks! What could be better?
“Say that A and B, both fluent speakers of English, are independently stranded on two uninhabited islands. They soon discover that previous visitors to these islands have left behind telegraphs and that they can communicate with each other via an underwater cable. A and B start happily typing messages to each other.
Meanwhile, O, a hyperintelligent deep-sea octopus who is unable to visit or observe the two islands, discovers a way to tap into the underwater cable and listen in on A and B’s conversations. O knows nothing about English initially but is very good at detecting statistical patterns. Over time, O learns to predict with great accuracy how B will respond to each of A’s utterances.
Soon, the octopus enters the conversation and starts impersonating B and replying to A. This ruse works for a while, and A believes that O communicates as both she and B do — with meaning and intent. Then one day A calls out: “I’m being attacked by an angry bear. Help me figure out how to defend myself. I’ve got some sticks.” The octopus, impersonating B, fails to help. How could it succeed? The octopus has no referents, no idea what bears or sticks are. No way to give relevant instructions, like to go grab some coconuts and rope and build a catapult. A is in trouble and feels duped. The octopus is exposed as a fraud.”
He goes onto talk about his experiences ‘managing’ a semi-self driving car (I think it might be a Volvo, like I used to own?) where you have to be aware that the thing is an incredible heavy, very fast toddler made of steel, with dunning-kruger-ish marketing promises pasted all over the top of it.
“You can’t ever forget the self-driver is like a 4-year-old kid mimicking the act of driving and isn’t capable of thinking like a human when it needs to. You forget that and you can die.”
That was absolutely my experience of my previous car too.
It was great for long stretches of motorway (freeway) driving in normal conditions, but if it was raining or things got more twisty/rural (which they do in most of the UK quite quickly), you switched it off sharpish.
I’m renting a tesla (I know, I know) for the first time on my next trip to the states. It was a cheap deal, and it’s an EV, and it’s California so I figure why not. I however will not use autopilot I don’t think, having used semi (level 2? 3?) autonomous driving before.
Perhaps there needs to be a ‘self-driving test’ for the humans about to go into partnership with very fast, very heavy semi-autonomous non-human toddlers before they are allowed on the roads with them…
“LLMs are like being at the pub with friends, it can say things that sound plausible and true enough and no one really needs to check because who cares?”
Ben Terrett was the first person I heard quoting (indirectly) Mitchell & Webb’s notion of ‘Reckons’ – strongly held opinions that are loosely joined to anything factual or directly experienced.
Send us your reckons – Mitchell & Webb
LLMs are massive reckon machines.
Once upon a BERG times, Matt Webb and myself used to get invited to things like FooCamp (MW still does…) and before hand we camped out in the Sierra Nevada, far away from any network connection.
While there we spent a night amongst the giant redwoods, drinking whisky and concocting “things that sound plausible and true enough and no one really needs to check because who cares”.
It was fun.
We didn’t of course then feed those things back into any kind of mainstream discourse or corpus of writings that would inform a web search…
In my last year at Google I worked a little with LaMDA.
The main thing that learned UX and research colleagues investigating how it might be productised seemed clear on was that we have to remind people that these things are incredibly plausible liars.
Moreover, anyone thinking of using it in a product that people should be incredibly cautious.
That Google was “late to market” with a ChatGPT competitor is a feature not a bug as far as I’m concerned. It shouldn’t be treated as an answer machine.
It’s a reckon machine.
And most people outside of the tech industry hypetariat should worry about that.
And what it means for Google’s mission of “Organising the worlds information and making it universally accessible’ – not that Google might be getting Nokia’d.
The impact of a search engine’s results on societies that treat them as scaffolding are the real problem…
My shallow technoptimism will be called into question if I keep going like this so let’s finish on a stupid idea.
British readers of a certain vintage (mine) might recall a TV show called “Call my bluff” – where plausible lying about the meaning of obscure words by charming middlebrow celebrities was rewarded.
Here’s Sir David Attenborough to explain it:
Attenborough on competitive organic LLMs as entertainment
Perhaps – as a public service – the BBC and the Turing Institute could bring Call My Bluff back – using the contemporary UK population’s love of a competitive game show format (The Bake Off, Strictly, Taskmaster) to involve them in a adversarial critical network to root out LLMs’ fibs.
The UK then would have a massive trained model as a national asset, rocketing it back to post-Brexit relevance!
I’m stealing Matt Webb’s “filtered for” format here – for a bunch of more or less loosely connected items that I want to post, associate and log as much for myself as to share.
And – I’ll admit – to remove the friction from posting something without having a strong thread or thesis to connect them.
I’ve pre-ordered “No miracles needed” by Mark Jacobson – which I’m looking forward to reading in February. Found out about it through this Guardian post a week or so ago.
The good news below from Simon Evans seems to support Prof Jacobson’s hypothesis…
The world has reached peak fossil fuels for electricity, thanks to the rapid growth of wind & solar
Wind & solar are the cheapest sources of electricity for 96% of the world's population
“I found this sort of approach really interesting but mostly I like the small scale of it yes I like the fact that it’s you know it’s something that you could imagine just proliferating as a standard component that’s attached to sort of Street Furniture or things around the house or whatever it is you might put them on your windowsill because they’re quite small and they just generate like enough power to make a sensor work or a light or something and yeah it’s this this alternative future to the big powerful set piece green Energy Future that’s obviously being pushed and should continue to be pushed because that’s competing against the big Power and the fossil fuel future but I like this idea of like the smaller cuter weirder fluttery imagine it’s quite fluttery yeah so yeah so this is this is Breeze Punk everybody…”
I like the idea of it being a standard component – a lego. A breezeblock?
I would like it to become… a thing. I suppose that’s why I’m writing this.
Used to be how you made things become things.
It’s probably not how you do it now, you need a much larger coordinated cultural footprint across various short-form streaming formats to make a dent in the embedding space of the LLMs.
Mind you, that’s not the same as making it ‘real’ or even ‘realish’ now is it.
A bit vogue-ish perhaps, to prove a point I asked ChatGPT what it knew about Breezepunk.
It took a while, but… it tried to turn into the altogether less satisfying “windpunk”…
I like making the cursor blink on ChatGPT.
The longer the better. I think it means you’re onto something.
Or maybe that’s just my Bartle-type showing again.
The production design of the recent adaptation of William Gibson’s The Peripheral seemed “fluttery” – particularly in it’s depiction of the post-jackpot London timeline.
Or perhaps the aesthetic is much more one of ‘filigree‘.
There’s heaviness and lightness being expressed as power by the various factions in their architecture, fashion, gadgets.
It’s an overt expression of that power being wielded via nanotechnology – assemblers, disassemblers constructing and deconstructing huge edifices at will.
It’s the (last?) book in Adrian Tchaikovsky’s “Children of…” series – an eon-and-galaxy-spanning set of stories where uplifted descendants of earth creatures interact with the remains of humanity on (generally) badly-terraformed worlds.
Without giving too much away, one of the uplifted life forms is a race of corvids – known as the Corvids, who exist as bonded pairs.
They are a kind of organic GAN or generative-adversarial network, constantly dismantling everything around them – learning and bickering their way toward incredibly effective solutions that other species miss – and leading to the other species in the book to speculate on their sentience in much the same way as many in the last year or two have around GPT-n – including an advanced AI based on an uploaded human (who runs on a computational substrate made of ants, by the way…)
Hear are a few passages from late in the book where that AI questions them around their apparent sentience:
Strutting around and shaking out their wings. Through all the means available to her, she watches them and tries to work out what it must be like to be them. Do they understand what has happened to them? They say they do, but that’s not necessarily the same thing.
She thinks of problem-solving AI algorithms from back in her day, which could often find remarkably unintuitive but effective solutions to things whilst being dumb as bricks in all other respects. And these were smart birds, nothing like that. She wanted them to drop the act, basically. She wanted them to shrug and eye each other and then admit they were human-like intellects, who’d just been perpetrating this odd scam for their own amusement. And yet the birds mutter to one another in their own jabber, quote poetry that predates whole civilizations, and refuse to let her in.
The two birds stand side by side, stiff as parade ground soldiers. As though they’re about to defend their thesis or give a final report to the board. ‘We understand the principles you refer to,’ Gothi states. ‘It was a matter that much concerned our progenitors on Rourke, after diplomatic relations were established between our two houses both alike in dignity.’ Word salad, as though some Dadaist was plucking ideas at random from a hat and ending up by chance with whole sentences. ‘Sentience,’ adds Gethli. ‘Is what is a what? And, if so, what?’ ‘You think,’ Kern all but accuses them. ‘You’d think we think,’ he either answers or gives back a mangled echo. ‘But we have thought about the subject and come to the considered conclusion that we do not think. And all that passes between us and within us is just mechanical complexity.’ ‘We have read the finest behavioural studies of the age, and do not find sentience within the animal kingdom, save potentially in that species which engineered us,’ Gothi agrees. ‘You’re telling me that you’re not sentient,’ Kern says. ‘You’re quoting references.’ ‘An adequate summation,’ Gethli agrees.
‘The essential fallacy,’ Gothi picks up, ‘is that humans and other biologically evolved, calculating engines feel themselves to be sentient, when sufficient investigation suggests this is not so. And that sentience, as imagined by the self-proclaimed sentient, is an illusion manufactured by a sufficiently complex series of neural interactions. A simulation, if you will.’ ‘On this basis, either everything of sufficient complexity is sentient, whether it feels itself to be or not, or nothing is,’ Gethli tells her. ‘We tend towards the latter. We know we don’t think, so why should anything else?’ ‘And in the grander scheme of things, it’s not important,’ Gothi concludes imperiously.
Children of Memory, Adrian Tchaikovsky
Wonderful stuff. Hugely recommended.
Does any one know if Mr Tchaikovsky has commented on what approaches a keen-eyed (magpie?) satire in his work of current AI hype?
I was fortunate to be invited to the wonderful (huge) campus of TU Delft earlier this year to give a talk on “Designing for AI.”
I felt a little bit more of an imposter than usual – as I’d left my role in the field nearly a year ago – but it felt like a nice opportunity to wrap up what I thought I’d learned in the last 6 years at Google Research.
Below is the recording of the talk – and my slides with speaker notes.
The excellent talks of my estimable fellow speakers – Elizabeth Churchill, Caroline Sinders and John can be found on the event site here.
Video of Matt Jones “Designing for AI” talk at TU Delft, October 2022Slide 1
Hello!
Slide 2
This talk is mainly a bunch of work from my recent past – the last 5/6 years at Google Research. There may be some themes connecting the dots I hope! I’ve tried to frame them in relation to a series of metaphors that have helped me engage with the engineering and computer science at play.
Slide 3
I won’t labour the definition of metaphor or why it’s so important in opening up the space of designing AI, especially as there is a great, whole paper about that by Dave Murray-Rust and colleagues! But I thought I would race through some of the metaphors I’ve encountered and used in my work in the past.
The term AI itself is best seen as a metaphor to be translated. John Giannandrea was my “grand boss” at Google and headed up Google Research when I joined. JG’s advice to me years ago still stands me in good stead for most projects in the space…
But the first metaphor I really want to address is that of the Optometrist.
This image of my friend Phil Gyford (thanks Phil!) shows him experiencing something many of us have done – taking an eye test in one of those wonderful steampunk contraptions where the optometrist asks you to stare through different lenses at a chart, while asking “Is it better like this? Or like this?”
This comes from the ‘optometrist’ algorithm work by colleagues in Google Research working with nuclear fusion researchers. The AI system optimising the fusion experiments presents experimental parameter options to a human scientist, in the mode of a eye testing optometrist ‘better like this, or like this?’
It makes the human the ineffable intuitive hero, but perhaps masking some of the uncanny superhuman properties of what the machine is doing.
The AIs are magic black boxes, but so are the humans!
Which has lead me in the past to consider such AI-systems as ‘magic boxes’ in larger service design patterns.
How does the human operator ‘call in’ or address the magic box?
How do teams agree it’s ‘magic box’ time?
I think this work is as important as de-mystifying the boxes!
Lais de Almeida – a past colleague at Google Health and before that Deepmind – has looked at just this in terms of the complex interactions in clinical healthcare settings through the lens of service design.
How does an AI system that can outperform human diagnosis (Ie the retinopathy AI from deep mind shown here) work within the expert human dynamics of the team?
My next metaphor might already be familiar to you – the centaur.
Gary Kasparov famously took on chess-AI Deep Blue and was defeated (narrowly)
He came away from that encounter with an idea for a new form of chess where teams of humans and AIs played against other teams of humans and AIs… dubbed ‘centaur chess’ or ‘advanced chess’
I first started investigating this metaphorical interaction about 2016 – and around those times it manifested in things like Google’s autocomplete in gmail etc – but of course the LLM revolution has taken centaurs into new territory.
This very recent paper for instance looks at the use of LLMs not only in generating text but then coupling that to other models that can “operate other machines” – ie act based on what is generated in the world, and on the world (on your behalf, hopefully)
And notion of a Human/AI agent team is something I looked into with colleagues in Google Research’s AIUX team for a while – in numerous projects we did under the banner of “Project Lyra”.
Rather than AI systems that a human interacts with e.g. a cloud based assistant as a service – this would be pairing truly-personal AI agents with human owners to work in tandem with tools/surfaces that they both use/interact with.
We worked with london-based design studio Special Projects on how we might ‘unbox’ and train a personal AI, allowing safe, playful practice space for the human and agent where it could learn preferences and boundaries in ‘co-piloting’ experiences.
For this we looked to techniques of teaching and developing ‘mastery’ to adapt into training kits that would come with your personal AI .
On the ‘pedal-assist’ side of the metaphor, the space of ‘amplification’ I think there is also a question of embodiment in the interaction design and a tool’s “ready-to-hand”-ness. Related to ‘where the action is’ is “where the intelligence is”
In 2016 I was at Google Research, working with a group that was pioneering techniques for on-device AI.
Moving the machine learning models and operations to a device gives great advantages in privacy and performance – but perhaps most notably in energy use.
If you process things ‘where the action is’ rather than firing up a radio to send information back and forth from the cloud, then you save a bunch of battery power…
Clips was a little autonomous camera that has no viewfinder but is trained out of the box to recognise what humans generally like to take pictures of so you can be in the action. The ‘shutter’ button is just that – but also a ‘voting’ button – training the device on what YOU want pictures of.
There is a neural network onboard the Clips initially trained to look for what we think of as ‘great moments’ and capture them.
It had about 3 hours battery life, 120º field of view and can be held, put down on picnic tables, clipped onto backpacks or clothing and is designed so you don’t have to decide to be in the moment or capture it. Crucially – all the photography and processing stays on the device until you decide what to do with it.
This sort of edge AI is important for performance and privacy – but also energy efficiency.
A mesh of situated “Small models loosely joined” is also a very interesting counter narrative to the current massive-model-in-the-cloud orthodoxy.
This from Pete Warden’s blog highlights the ‘difference that makes a difference’ in the physics of this approach!
And I hope you agree addressing the energy usage/GHG-production performance of our work should be part of the design approach.
Another example from around 2016-2017 – the on-device “now playing” functionality that was built into Pixel phones to quickly identify music using recognisers running purely on the phone. Subsequent pixel releases have since leaned on these approaches with dedicated TPUs for on-device AI becoming selling points (as they have for iOS devices too!)
And as we know ourselves we are not just brains – we are bodies… we have cognition all over our body.
Our first shipping AI on-device felt almost akin to these outposts of ‘thinking’ – small, simple, useful reflexes that we can distribute around our cyborg self.
And I think this approach again is a useful counter narrative that can reveal new opportunities – rather than the centralised cloud AI model, we look to intelligence distributed about ourselves and our environment.
A related technique pioneered by the group I worked in at Google is Federated Learning – allowing distributed devices to train privately to their context, but then aggregating that learning to share and improve the models for all while preserving privacy.
This once-semiheretical approach has become widespread practice in the industry since, not just at Google.
My next metaphor builds further on this thought of distributed intelligence – the wonderful octopus!
I have always found this quote from ETH’s Bertrand Meyer inspiring… what if it’s all just knees! No ‘brains’ as such!!!
In Peter Godfrey-Smith’s recent book he explores different models of cognition and consciousness through the lens of the octopus.
What I find fascinating is the distributed, embodied (rather than centralized) model of cognition they appear to have – with most of their ‘brains’ being in their tentacles…
And moving to fiction, specifically SF – this wonderful book by Adrian Tchaikovsky depicts an advanced-race of spacefaring octopi that have three minds that work in concert in each individual. “Three semi-autonomous but interdependent components, an “arm-driven undermind (their Reach, as opposed to the Crown of their central brain or the Guise of their skin)”
I want to focus on the that idea of ‘guise’ from Tchaikovsky’s book – how we might show what a learned system is ‘thinking’ on the surface of interaction.
We worked with Been Kim and Emily Reif in Google research who were investigating interpretability in modest using a technique called Tensor concept activation vectors or TCAVs – allowing subjectivities like ‘adventurousness’ to be trained into a personalised model and then drawn onto a dynamic control surface for search – a constantly reacting ‘guise’ skin that allows a kind of ‘2-player’ game between the human and their agent searching a space together.
This is CavCam and CavStudio – more work using TCAVS by Nord Projects again, with Alison Lentz, Alice Moloney and others in Google Research examining how these personalised trained models could become reactive ‘lenses’ for creative photography.
There are some lovely UI touches in this from Nord Projects also: for instance the outline of the shutter button glowing with differing intensity based on the AI confidence.
Finally – the Rubber Duck metaphor!
You may have heard the term ‘rubber duck debugging’? Whereby your solve your problems or escape creative blocks by explaining out-loud to a rubber duck – or in our case in this work from 2020 and my then team in Google Research (AIUX) an AI agent.
We did this through the early stages of covid where we felt keenly the lack of informal dialog in the studio leading to breakthroughs. Could we have LLM-powered agents on hand to help make up for that?
And I think that ‘social’ context for agents in assisting creative work is what’s being highlighted here by the founder of MidJourney, David Holz. They deliberated placed their generative system in the social context of discord to avoid the ‘blank canvas’ problem (as well as supercharge their adoption) [reads quote]
But this latest much-discussed revolution in LLMs and generative AI is still very text based.
What happens if we take the interactions from magic words to magic canvases?
Or better yet multiplayer magic canvases?
There’s lots of exciting work here – and I’d point you (with some bias) towards an old intern colleague of ours – Gerard Serra – working at a startup in Barcelona called “Fermat”
So finally – as I said I don’t work at this as my day job any more!
I work for a company called Lunar Energy that has a mission of electrifying homes, and moving us from dependency on fossil fuels to renewable energy.
We make solar battery systems but also AI software that controls and connects battery systems – to optimise them based on what is happening in context.
For example this recent (September 2022) typhoon warning in Japan where we have a large fleet of batteries controlled by our Gridshare platform.
You can perhaps see in the time-series plot the battery sites ‘anticipating’ the approach of the typhoon and making sure they are charged to provide effective backup to the grid.
And I’m biased of course – but think most of all this is the AI we need to be designing, that helps us at planetary scale – which is why I’m very interested by the recent announcement of the https://antikythera.xyz/ program and where that might also lead institutions like TU Delft for this next crucial decade toward the goals of 2030.
“Here was a man who played spiritual, cosmic music, from whom I wanted to know the secrets to the universe. But he was more interested in being in the moment and recognising the power of being in the moment. He showed me that connecting with the great beyond is sometimes about the simplest things.”
“Jim attributed his great old age to long daily walks – he lived to 103 and right up to the end his mind was very vivid. I joined him sometimes wandering through his grounds, where he’d let Gaia have its will. He had a cat and once the cat sat on my shoulder through the entire walk.”
I often say the most mind-bending fact about climate is that half of all emissions came in the last 25 years. Maybe even more startling: the weight of that carbon is more than the total mass of everything ever built by humans and still standing on earth. https://t.co/BOaPHQW0cr
Petafloptimism 