72 posts tagged with "ai"

In my latest experiment with AI-powered robotics, I decided to take all the API help specs from the Spike Prime 3 user guide and load them into Cosmos DB as a knowledge base. The goal was to use this data to generate better, more targeted Python code for Spike Prime robots using a LangChain RAG integration.

Here's the plan I followed:

1. Extracting API Data: I pulled all the relevant API definitions and code snippets from the Spike Prime user guide. This included functions related to movement, turning, and sensor control.
2. Cosmos DB as a Knowledge Base: I uploaded the extracted API documentation into Cosmos DB, setting it up as a searchable knowledge base. This would allow us to easily retrieve API definitions and related examples.
3. LangChain Integration: The idea was to use LangChain's search capabilities to query Cosmos DB. The chatbot would then fetch relevant Python API definitions and example code snippets directly from the knowledge base.
4. Generating Python Code: Once the relevant information was fetched, the chatbot would combine this with user input to generate Python code that could control the Spike Prime robot.

However, I ran into an unexpected issue: Despite having all the Spike Prime API information in Cosmos DB, the chatbot still ocassionally attempted to rely on external Python libraries from the internet instead of using the custom API definitions I had loaded. This led to code that wasn't compatible with the Spike Prime system, as those external libraries were not designed for LEGO robots.

This is a challenge I need to address. The chatbot is prioritizing internet-based solutions over the internal knowledge base, which defeats the purpose of having a custom-built, Spike Prime-specific AI assistant.

Next steps? I plan to tweak the LangChain logic to ensure the chatbot prioritizes fetching from Cosmos DB and only uses the API definitions we've provided. This will help ensure that the Python code generated is compatible with the Spike Prime robot. Stay tuned for updates on how I solve this!

Recently, I came across the Microsoft Azure OpenAI Hackathon and couldn't resist giving it a go! With the help of the detailed online instructions, I was able to follow along easily and dive into some really exciting AI technologies. The step-by-step tutorial provided a smooth learning experience, making the whole process engaging and informative.

During the hackathon, I gained valuable knowledge on several key technologies:

• Cosmos DB: I learned how this scalable database can store and manage massive amounts of data, which is perfect for AI applications.
• LangChain: This framework helped me understand how to integrate language models into real-world applications, particularly with the help of APIs and external data.
• RAG (Retrieval-Augmented Generation): This was one of the most interesting parts! I built a chatbot that retrieves information from a knowledge base to give more accurate and relevant responses, blending both AI generation and data retrieval.

The experience got me thinking: how can I apply a similar approach to improve AI coding for LEGO Spike Prime robots using Python? Could we potentially use tools like Cosmos DB, LangChain, and Retrieval-Augmented Generation to enhance the way robots process commands and interact?

Imagine a chatbot-like interface for programming robots, where the AI retrieves the most efficient and specific commands from a knowledge base to guide the robot's movements and tasks. It opens up a world of possibilities! Could we fine-tune a model that combines language generation with real-time data, and use it to make robotics coding smarter and more intuitive?

The hackathon has definitely sparked some new ideas for how we can push the boundaries of AI-powered robotics programming. I'm eager to explore these possibilities further and see how we can take our LEGO robot AI coding to the next level!

Update: got my completion badge!

I'm excited to share that today I had the incredible opportunity to attend the Global Azure Sydney 2024 user group session at the Microsoft Reactor today! Even more thrilling, I got to present my experiences with AI and coding during the event. It was an amazing moment to share my journey, including how I've been exploring AI model fine-tuning and integrating it with robotics like Spike Prime. Feeling grateful for the chance to connect with so many passionate people in the field and exchange ideas!

Here's a quick overview of my talk and some key takeaways:

• The Power of AI in Code Generation: I dive into how I've been using Azure's fine-tuning to create more efficient, domain-specific code for robotics.
• From Generic Python to Robot-Specific Commands: I shared my experience building models that turn general Python code into Spike Prime-specific instructions.
• Azure Endpoint Integration: I walked through how to set up and connect an AI model to a local environment, demonstrating real-time interactions with the Spike Prime robot.

The feedback I received was beyond amazing! Attendees from all over the world were interested in how AI can assist with robotics programming and coding efficiency. The response from the team at Microsoft was also incredibly positive, sparking some great conversations on future collaborations and projects.

I had a fantastic day at Microsoft Reactor in Sydney, a space filled with innovation and learning. The energy was contagious, and I left with so many new ideas and connections.

You can watch the full session here:

Watch My Azure Global Sydney 2024 Talk

We've made great progress in our journey to fine-tune an AI model for Spike Prime robots. With the training dataset and validation dataset ready, it's time to take the next step: setting up an Azure endpoint and integrating it with our local Python script.

First, we configured the Azure endpoint to allow us to send and receive data. This will enable our fine-tuned model to process instructions in real-time. By doing this, we can interact with the model directly from our local machine, making it easy to integrate AI-powered code suggestions into our Spike Prime project.

For the robot connection, we used a serial port interface. This allowed us to send commands from the Python script running on our local machine to the robot in real-time. This setup ensures that we can test and execute movements and commands on the fly.

The results look promising so far. The fine-tuned model has demonstrated an ability to stick closely to the instructions we provided. It seems to understand the specifics of the Spike Prime movement commands, offering more accurate code suggestions than a generic model.

Here's a glimpse of what we've tried so far:

• Moving forward: The model was able to generate precise movement commands using Spike Prime-specific syntax.
• Turning: We tested multiple turning scenarios, and the model successfully adjusted the turning angle based on the instructions.
• Combining movements: The fine-tuned model was able to chain together commands, making the robot perform complex maneuvers smoothly.

The integration between Azure and our local environment is working well, and the fine-tuned model is proving to be a valuable tool in generating Spike Prime-specific Python code. As we continue to refine the model, I'm excited to see how far we can take it!

Over the last 12 months, I've been amazed by all the developments happening in the AI world. From advancements in natural language processing to AI-driven coding tools, it's clear that AI is transforming the tech landscape faster than ever. With all the buzz, I decided to set a challenge for myself: to build an AI app from scratch.

I've heard a lot about is Azure's fine-tuning capabilities. The idea behind fine-tuning is to take a generic AI model and specialize it for a particular task or domain. In my case, I want to see if I can fine-tune a model to improve the accuracy of Python coding snippets, specifically for Spike Prime robots.

Spike Prime is a versatile LEGO robotics kit that allows users to program movements and interactions. While generic Python code can be used to program the robots, I'm curious if fine-tuning a model on Azure can make the code more optimized for Spike Prime's specific needs.

This will be my first time building a fine-tuned model on Azure, so I want to make the process as smooth as possible. Instead of trying to fine-tune every Python function related to Spike Prime, I've decided to narrow the scope and focus only on a few scenarios, such as robot movement and turning.

The fine-tuning process for this project looks like this:

1. Dataset preparation: Gather Python code snippets that are specifically tailored for Spike Prime, focusing on movement and turning functions.
2. Fine-tuning: Use Azure's fine-tuning service to train the model with this specialized dataset.
3. Testing: Evaluate the fine-tuned model by inputting generic Python code and observing how it converts the code to a more Spike Prime-specific version.

This is just the beginning, and I'm excited to see what kind of improvements we can achieve. Stay tuned as I dive deeper into the world of AI fine-tuning and robotics!

Join me for an exciting session where we'll have some fun with Lego SpikePrime Robot and AzureOpenAI. I'm proud to be presenting at Global Azure Sydney 2024 - Microsoft Reactor Sydney @ 20/04/2024.

📌 Session: Fine-tune GPT-3.5 model to control Lego Robot using Azure OpenAI & Python 🗣️ Speaker: Daniel Fang

Session details: https://sessionize.com/view/rjfzv8k0/GridSmart?format=Embed_Styled_Html&isDark=False&title=Global%20Azure%20Sydney%202024 Reserve your spot: https://www.tickettailor.com/events/azuresydneyusergroup/1193113

Recently, I found myself with some free time, and it brought back a memory from our robotics team. I remember one of the girls on the team trying to use ChatGPT to help her write code for our Masterpiece Season mission. She was hoping to get some assistance resolving a tricky problem with our robot. Unfortunately, even though the code it generated looked solid, it didn't run correctly when we tried it out on the robot.

Out of curiosity, I decided to give it a go myself to see what might have gone wrong. Here's an example of a Python snippet that looks fine at first glance:

import hub
from spike import PrimeHub, Motor, MotorPair

hub = PrimeHub()

motor_pair = MotorPair('A', 'B')

motor_pair.start(50, 0)  # Move straight at 50% speed

The syntax is correct, and the code seems reasonable for controlling a Spike Prime robot. But when you upload this code to the robot, something weird happens: the robot doesn't quite behave the way it should. It might not move at all, or it could respond unpredictably.

So, what's going on here?

Well, after some quick troubleshooting, I realized there's a bit more to getting a Spike Prime robot working smoothly than just correct Python syntax. The code must align perfectly with the robot's specific hardware configuration, sensor inputs, and timing mechanisms. It's not just about writing Python - it's about writing the right Python for this context.

Looks like I've got an investigation on my hands. Time to dive deeper into how the Spike Prime API works and figure out why ChatGPT's code, while syntactically correct, doesn't hit the mark. It's a reminder that while AI tools can be incredibly useful, robotics is a nuanced field where hardware, software, and context all come into play.

If I crack the mystery, I'll be sure to share what I learn!

We are thrilled to share some exciting updates about Fresh, our AI-powered hobby-sharing app. Over the past few months, we've been working tirelessly to enhance your experience by incorporating valuable feedback from tech experts and our vibrant community.

One of the significant improvements is the expansion of our hobby directory. We've added a plethora of new hobbies to ensure that everyone finds something that resonates with their interests. Whether you're into pottery, urban gardening, or drone racing, Fresh has got you covered.

Our chatbot has also received a significant upgrade. It now suggests nearby events based on your current location, making it easier for you to connect with like-minded enthusiasts in your area. Simply tell the chatbot what you're interested in, and it will provide personalized recommendations for events and meetups happening near you.

We've also produced new tutorial videos to help you get the most out of Fresh. These guide you through the app's features and show you how to make the most of our platform.

Moreover, we've made significant strides in enhancing the app's accessibility. Fresh is now fully functional on both desktop and mobile devices, ensuring you can stay connected to your hobbies anytime, anywhere.

We couldn't have achieved these improvements without your invaluable feedback. Thank you for being a part of our journey. The girls are committed to continually enhancing Fresh to make it the best platform for hobby enthusiasts worldwide.

The team is struggling with writing Python for the first time! It's so hard, and we don't know where to start. Drag-and-drop coding in the code blocks was so easy, but now we need to learn Python and figure out what to do. The Spike Prime has some examples, but they're not so easy to understand for us.

We turned to ChatGPT for help. It does output lots of code, but it doesn't run on the app. Why? We asked our coach for help and learned that the Spike Prime only uses a specific set of Python, and the robot only knows some of the functions but not all.

It turns out that Spike Prime uses MicroPython, which is a simplified version of Python designed for microcontrollers and embedded systems. This means that not all standard Python libraries and functions are available. Some of the code that ChatGPT provides uses functions that the Spike Prime doesn't support.

This was a bit disheartening at first. We thought that learning Python would open up new possibilities, but instead, we hit a wall. However, we realized that this is just another challenge to overcome. We're now focusing on learning the specific commands and functions that are available in Spike Prime's MicroPython environment.

It's still tough, but we're making progress. We're finding tutorials and resources specifically for Spike Prime's Python coding. We're also experimenting with small bits of code to see what works and what doesn't. It's a learning process, and while it's frustrating at times, it's also exciting to see our robot respond to the code we've written.

In the end, we're hopeful that with perseverance and a better understanding of Spike Prime's Python capabilities, we'll be able to code like pros. For anyone else struggling with this, know that you're not alone, and don't hesitate to seek help and keep experimenting!

We've decided to bring our idea to life by building an AI app for hobby sharing called FRESH. The inspiration behind FRESH is to create a platform that helps people find nearby hobbies and workshops where they can learn new things and connect with others who share similar interests.

FRESH isn't just a directory; it's a comprehensive resource with a vast database of hobbies available to assist people in exploring new activities. Whether you're interested in pottery, coding, photography, or any other hobby, FRESH aims to make the search easy and engaging.

What sets FRESH apart is its clever chatbot assistant. Powered by AI technologies like ChatGPT and enriched with a collection of documents we've assembled, the chatbot can communicate with users to understand their likes and preferences. Based on this interaction, it offers personalized hobby suggestions drawn from our extensive backend knowledge base.

The chatbot doesn't just list options; it engages in a conversation to help users discover hobbies they might not have considered. It's like having a friendly guide who's always ready to inspire and inform.

We can't wait to see the potential of FRESH to help people discover new passions and build a community around shared interests. By combining arts, technology, and AI, we're creating an innovative way for people to enrich their lives and connect with others.