Boston Dynamic's robots are works of art - the pinnacle of engineering - but its all designed movement. By this I mean the control systems, their movement plans - it is built and designed by experts in their field. It's not quite as simple as "go from A to B and do some parkour on the way". There's a very large gap between "what is mechanically possible to do" and "Just let the robot figure out how to do that".
Mechanically we're ahead of software for manipulation and kinodynamic planning.
I'm actually working on this problem right now for my master's capstone project. I'm almost done with it; I can have it generating a series of steps to try and fetch me something based on simple objectives like "I'm thirsty", and then in simulation fetching me a drink or looking through rooms that might have a fix, like contextually knowing the kitchen is a great spot to check.
There's also a lot of research into using the latest advancements in reasoning and contextual awareness via LLMs to work towards better more complicated embodied AI. I wrote a blog post about a lot of the big advancements here.
Outside of this I've also worked at various robotics startups for the past five years, though primarily in writing data pipelines and control systems for fleets of them. So with that experience in mind, I'd say we are many years out from this being in a reasonable product, but maybe not ten years away. Maybe.
State of the art in LLMs + Robotics 2023
tldr I write about some of the more interesting works that shaped my understanding of applying LLMs for AI agents and robotic applications. Introduction What is this LLMs as a fad - a caveat Are LLMs actually going to be useful for robotics? Instruct based Benchmarking LLM basics n-shot and rea...
I was aiming to use LLMs with robotics in an upcoming project, and needed to first verse myself in what is the current must-know techniques in the space. To that end I read a ton of papers and wrote this article to try and suss out the best parts of current state of the art.
I hope this helps people; I'd be thrilled to discuss much of this as well!
tldr I write about some of the more interesting works that shaped my understanding of applying LLMs for AI agents and robotic applications. Introduction What is this LLMs as a fad - a caveat Are LLMs actually going to be useful for robotics? Instruct based Benchmarking LLM basics n-shot and rea...
I was aiming to use LLMs with robotics in an upcoming project, and needed to first verse myself in what is the current must-know techniques in the space. To that end I read a ton of papers and wrote this article to try and suss out the best parts of current state of the art.
I hope this helps people; I'd be thrilled to discuss much of this as well!
You're going to need to share a bit more information about your vendetta against Cloudflare; especially considering their prevalence.
Using RLHF to train champion-level drone racing agents
An autonomous system is described that combines deep reinforcement learning with onboard sensors collecting data from the physical world, enabling it to fly faster than human world champion drone pilots around a race track.
A cool application of RLHF (Reinforcement Learning w/ Human Feedback - the same approach as what OpenAI used to train ChatGPT).
The authors trained an agent to fly FPV drones at a level surpassing world champions.
A light week for me, mostly going through some more ROS2/webots tutorials where I can. If anyone has good resources to recommend, lemme know!
RoboAgent: A universal agent with 12 Skills
Towards Sample Efficient Robot Manipulation with Semantic Augmentations and Action Chunking
This isn't mine, it's just an interesting blogpost I came across. Nor am I arguing that it should replace a robotics engineer.
My main thought, not fully represented in the post, is that LLMs can act as a context engine for high level understanding of instructions + spatial awareness, and then apply it to actuation. This is somewhat touched upon in the article.
I do think that there is some interesting work in LLM powered task level planning. I'm hoping to find the time put together a good example of this, utilizing the ability for LLMs to make logical leaps based on instruction. In the article, it took the command "I'm thirsty" to mean move to a drink. In a more applicable application, we can use LLMs to identify that a room with multiple identified objects (refrigerator, oven, stove, cabinets, etc) is in fact a kitchen. Then, from there, determine that "I've seen a room I've identified as a kitchen - I can navigate there to attempt to find a drink".
Oh yeah, MATLAB is painful. I get why you use it at first - it's great for handling derivations for you when looking at control code, and handles matricies well enough when learning kinematics. But once my homeworks started to demand animations and complex processing I yearned for a language with classes or any advance features at all. Still, managed to make some cool stuff - like this RRT path planned transmission removal 😄
What startup (assuming you're out of stealth mode?) Good luck with the jump over to a startup. It's rough but hopefully you knock it out of the park.
As for code deploy - I've worked on the problem at two startups now. I can probably advise you on some stuff to look into, but would need to know more about the problem space you're specifically looking at. Though I'm hesitant to mention full obfuscation if you're not delivering a finished product but rather a module to the end customer.
ChatGPT for Robotics: Design Principles and Model Abilities
We use ChatGPT to allow a user to intuitively control multiple robots such as drones, robot arms, home assistant robots using only language.
I have argued for awhile now that the probabilistic nature of LLMs can represent a form of context that, when applied, can be utilized for robotic applications.
Seems I'm not the only one that had this idea. While simple, Microsoft researched applied a high level control library to demonstrate LLMs (ChatGPT) developing robotic task code.
For the self driving car, are we talking about hobbyist size (ie Donkey car), add on tech (ie Comma OpenPilot) or full on autonomous vehicle? Sounds really cool.
Upgrading from an old tech stack to a newer one is always a pain, especially in Python - one of the reasons I hate that it's so widely adapted for deep learning, CV, and robotics.
Hopefully you'll regale us with details, this sounds fascinating.
From the Desks of ROS Maintainers: A Survey of Modern & Capable Mobile Robotics Algorithms in the Robot Operating System 2
ROS maintainers discuss popular robotics control and navigation algorithms in use within ROS2. The associated discussion can be found here.
If you're looking for what to study or try applying in your own projects next, this is worth a look.
Last week I started going through ROS2 lessons online in order to familiarize myself on it for some upcoming projects.
First I spent time working on utilizing vagrant, a tool by Hashicorp for building "repeatable" (debatable overuse of that term, but I'll digress) dev environments and VM images to quickly set up versioned ROS environments for me. This was actually pretty easy and after a few hours I had a setup I liked. I will report that I do have some issues running Gazebo in VM on the laptop (to be expected) though it's smooth on the beefier desktop. I am still suffering from occasional complete VM freeze ups - irrecoverable, though the host machine shows no lag or issues there. I think it'll still work for a quick setup of ROS2 for a project team.
Now I'm going through the nav2 stack in ROS and trying to familiarize myself with it. I'm not sure what the scope of the upcoming project is going to be (it's the capstone team project for the entirety of my Masters, so there's a bit of time before decisions have to be finalized). Once that's done I'll probably dive into Webots simulator (especially since my own Gazebo is proving unstable).
What are you working on this week? (Aug 14 23)
This is the routine thread where we discuss what you'll be working on this week! A cool robot? A computer vision project? Something cool in reinforcement learning? 3d printing a drive train? Let us know!
Maybe instead you're studying something, or reading a paper that just came out? Post about it!
It’s also okay to say “nothing” too - it’s great for your mental health to take a break!
Looking for help? Ask a question! See someone working on something cool? Ask them about it! No project is too small or too "newbish"!
How're you liking the ID4? That and the ioniq5 are looking pretty good ATM... Though I wish the Honda E or ID3 was sold in the US...
Is anyone else encountering issues w/ being insta-killed out of wild shape?
Playing multiple co-op campaigns, and every druid across multiple campaigns experience, randomly, unexpected deaths when being knocked out of wild shape. Not death saving throws either - straight into full blown d-e-a-d.
Per the rules, it should go from wild shape -> spill over damage being removed from the remaining HP of the druid. Reading the combat logs we see that while enough damage is done to knock the druid out of their wild shape, the remainder is not nearly enough to put them into death saving throws, let alone outright kill them.
Is anyone else encountering this issue?
vagrant ended up being way easier than I had anticipated, with just minor issues. I even got a premade image published so that teammates don't have to sit through the long install process, just pull a good base image.
Given that I don't know yet (nor do I have access to a machine to test) Docker solutions w/ GUIs for Windows/Mac I'm going to just stick to the VM approach for now unless I have a strong driver to switch it up.
I'm working on updating my knowledge of ROS/ROS2. I've done some tutorials/mini projects in ROS, but wanted to refresh for ROS2 for an upcoming large project I'm doing. Since I'm the most experienced programmer in the group and have a strong background in backend tech and dev tools I'm trying to discover the easiest way to have repeatable ROS2 environments such that we can eliminate that pain point from the project entirely.
Currently the front runners are vagrant (neat little image builder and dev env manager by Hashicorp). There's always Dockerfiles w/ the container approach, but I'm the only native Linux user - I'm not sure if X11 pass through can be done at all in the VM's running Docker in Windows/MacOS.
Once this is solved we're going to explore simulation options for the project. Since we're a group of 5 people spread across 3 time zones hardware is a tough sell, so we'll likely be working mostly in simulator. Currently looking into either Webots or Gazebo, but not married to either at all.
As for what the project is - no idea yet. We don't need to know that yet, and I figured having a strong base is going to help irregardless of what we focus on.
One addition- sunlight is very bad for resin printers, so if your garage has windows or you routinely keep your garage doors open keep this in mind.
Ah - I thought the image was in addition to the URL, when it overwrote it? Consider me embarrassed and a lemmy newb on this one. Fixed for now.
I just finished my first long technical blog post - blogging is a relatively new habit I'm trying to build up. I posted the blogpost already to the robotics community so I'll just crosspost it with this link. TLDR it's the trial and tribulations I had while trying to use PPO to train a robotic arm to do pick and place.
I'm putting a hold on the other project I was burning out of - something I called coppermind (yes I'm a Sanderson fan) that handled chatbot memory/applications. I have an old version running on a digital ocean droplet plugged into Twilio, but recent degradations in ChatGPT 3.5 model response quality + that version not having my knowledge/vector database features means it's becoming a bit repetitive on its responses.
Right now with my masters in robotics starting up again in late August, and this being the project for the entire thing, I'm spending some time working on tooling to get myself up to speed on ROS2 and get repeatable environments up and running for it so I can quickly fix/deploy sim robots for the project. I have some ideas I'm toying with there. Unfortunately no idea what the project will be yet.
Using Reinforcement Learning (PPO) to try and teach a robotic arm to pick and place
In my first attempt at a long form technical post, I talk about a project where I had to use deep reinforcement learning to try and solve a robotics application. It worked ok - the post talks about my struggles, solutions, and what I'd change up in the future.