By Ahti Heinla, Co-Founder and CTO of Starship Technologies
I see robots every day. I see them sliding down the sidewalks at pedestrian speed, stopping to make sure it is safe to cross the road. Sometimes I even catch them talking to pedestrians. It’s a glimpse into the fantasies of technologists – an AI wonderland. But this is not a hallucination, no dream, it is a reality that our team of dedicated visionaries has built over the past 5 years; we have brought the future to the present.
Just a few years ago, these robots needed a little human support and were accompanied on their journeys, much like the format followed by self-driving car makers, who test their cars in public with “ safety conductors ”.
Starship became the first robotics team to start operating regularly in public spaces about 18 months ago, without the use of safety drivers; we let our robots explore the world on their own. Today we operate our network of robots every day in cities around the world, bringing people their meals, packages and groceries.
Shared knowledge is acquired knowledge
It’s exciting to be the first.
When I was a founding engineer at Skype, we were the first to make VoIP conveniently accessible; we are now working to do the same with robots in public spaces. For the past four years, our engineering teams have been working behind closed doors on what has been a significant breakthrough and an incredible experience.
I want to share with you some of the details of our technical journey. Over the coming weeks and months, other members of the Starship engineering team will also share aspects of their journey.
Throughout this journey, we have worked with computer vision, path planning, and obstacle detection – well-studied topics in the field of academic robotics. Indeed, Starship started as a research project, but quickly evolved into a functional and practical delivery operation.
This means that in addition to fine-tuning the Levenberg-Marquardt algorithm for nonlinear optimization, we had to develop software for:
- Automatically calibrate most of our sensors – after all, we don’t want to spend hours calibrating them by hand; we have manufactured hundreds of robots and we are now preparing for a larger scale operation.
- Predict how much power each trip will draw from a robot’s battery – so we can orchestrate which robot to send, based on battery status.
- Predict how many minutes it takes for a restaurant to prepare food – for the robot to appear just in time!
Most of the autonomous robots that exist in the world today are expensive, they are built as technology demonstrators or research vehicles, and are not used for commercial operations. A single set of sensors for a stand-alone device can cost over $ 10,000. It just won’t work in the delivery space, it’s not a luxury industry where you can charge a premium.
Autonomous driving research vehicles often have 3 kilowatts of computing power in the trunk; inconvenient for a small, safe delivery robot. Therefore, part of our engineering journey has been to design for a lower unit economy. Here are some topics we had to consider:
- Advanced image processing on a low-end computing platform.
- Work around hardware problems in software.
- Track how often robots need maintenance and why.
- Develop advanced route planning systems, to ensure that we use our robot network efficiently.
It was quite a journey in visual design too, involving hundreds of sketches, drawings and surveys before we made our robot’s first plastic body.
Back in the days when we were still in stealth mode, we didn’t want to reveal what our robots looked like. Regular public testing required the creative use of a trash bag, taped to the robot’s body as a disguise!
The construction of practical robotics is a mixture of science, systematic engineering and hacking. This mixture of various disciplines is the fundamental characteristic of Starship. Nothing is ever easy in robotics. All of your situational awareness is probabilistic; all sensors have failure modes and faults, and even a seemingly simple task such as make the robot stop at obstacles can become its own little research project.
Starship is a fast-growing start-up, and it’s important not to become just a research project. Engineers who get excited about Starship are often not pure scientists, nor pure hackers, nor pure engineers; they have many of these characteristics and can use them as appropriate for the task at hand. We need complicated technical solutions that can be implemented quickly and within the resource constraints of low cost hardware.
Ingenuity and resourcefulness are appreciated skills.
A week is a long time at Starship
At the start of the week, our team will be implementing a new algorithm to detect point cloud borders and have it tested again against a full test case database overnight, they will have it tested. live on our private testing ground by the end of the week.
It will be on the streets next Monday, with the team already reporting on their progress at our engineering meeting on Monday. On most Mondays, some members of the engineering team report a gain of over 300% on at least one of the metrics achieved, just the week before.
Data as a result and facilitator of scale
Metrics and data have become an important part of Starship’s engineering.
You see, when we started out, we didn’t have any data – we hadn’t driven much yet. Every day we modified our robot (yes, just the one back then), took it to the sidewalks and saw how it worked. We now have a lot of them, driving autonomously every day – too many for engineers to observe directly.
Thanks to the data, we can now see the performance of our robots, hundreds of them. We can run weekly ‘data diving’ seminars, where engineers share their findings and watch random deliveries to stay in touch with their work in action.
When we work on making our robots more fluid, we analyze the data in the table of “acceleration events” in our data warehouse; there are at least 1 billion rows in this table. Other tables include “road crossing events”, our maps, every order every robot has ever received from our servers, and of course the data collected with every delivery they make.
Four years ago we had none of this. Back when we were just starting out – and we weren’t doing commercial deliveries yet – I often had to convince people that robotic delivery really worked. People found it hard to believe and were quick to point out various reasons.
Do skepticism and fear always accompany new technologies?
Several years ago, I landed at JFK Airport in New York with a robot in my luggage. The customs officer obviously asked, “What is this thing?” I explained it was a sidewalk delivery robot, to which he replied, “Dude, this is New York! It will be stolen in minutes! “
Indeed, at the time, almost everyone thought these robots would be stolen – I’m sure they probably will (postal delivery vans are stolen, although rarely). To date, our robots have traveled over 200,000 km (130,000 miles) and we have yet to see this problem.
There are of course some security features in place. The robot has a siren and 10 cameras, it is constantly connected to the Internet and knows its precise location with an accuracy of 2 cm (thanks to the Levenberg-Marquardt algorithm mentioned above, and the 66,000 lines of C code ++ automatically generated which allows our robots to use it).
People also believed that pedestrians might be afraid of robots on the sidewalk or not accept their presence. Will people call the police? To be honest, we weren’t sure either! However, once we put one of the robots on the sidewalk, we were surprised.
What happened next surprised us: people just ignored it. The vast majority of the public paid no attention to the robots, even those who saw them for the first time, and people certainly weren’t afraid. Others would pull out their phones and post on Instagram how they saw the future.
And that’s what we wanted.
We want people to pay as much attention to our robots as they do to their dishwashers. This pattern of silent acceptance of robots as if they had always been with us has been repeated in every city in the world in which we have operated.
It gets better. Once people learn that these robots are providing a useful service to the neighborhood, they develop an affinity with them. Kids even write letters to thank the robots, we have a “thank you letter wall” to prove it!
Automating last mile delivery was never going to be easy and we knew it would be a bold move. We also knew from the start that there would be more than one basic roadblock to resolve – it turned out there were hundreds of roadblocks! But we realized a long time ago that all of these problems could be solved – they just required ingenuity and persistence.
Some startups start out as running a sprint, launching a minimum viable product in 3 months. For Starship, it feels more like a marathon – constant big effort is required, but the end result brings huge benefits to the world.
Last mile delivery is one of the global industries that has seen little technological disruption since the adoption of the automobile. The Starship team is looking to change that, and with over 20,000 deliveries to our credit, we’re on the right track.
If you’d like to learn more, check out our second engineering blog post on Neural Networks and how they power our robots here – https://medium.com/starshiptechnologies/how-neural-networks-power-robots-at-starship-3262cd317ec0